Category Archives: experimental research

Core Concepts for Experimentation

This post will explore several core concepts that are related to experimentation in research. These concepts include

  • Randomization
  • Replication
  • Blocking

Randomization

Randomization involves making sure that the order of the individual runs of the experiment are determined by chance. The main reason for this is to ensure that observations and error are independently distributed random variables themselves. Spreading out all variables in a similar manner helps with the validity of the results. This is because the error is averaged out among all variables and not only one.

Many computer software will automatically randomize the runs of an experiment for you. Such a process helps to eliminate any accidental patterns that may arise if you try to randomize yourself. A common mistake people make when doing experiments is to let convenience determine the run order. For example, if it is hard to set up equipment that can be used as an excuse to run the experiments in a way that is most convenient but may also influence the results.

There are times in which complete randomization is not possible. There are ways to address this statistically, as we will see in the future.

Replication

A replication is a repeated run of a particular factor combination. For example, let say you are looking at the role of gender (two levels) and class level (four levels) affects quiz score. One replication would be to have at least two female freshmen take the quiz.

The benefits of replication include the ability to estimate error and a more precise measurement of the mean for that particular combination of factors.

Another term confused with replication is repeated measurement. They are the same thing with the exception that repeated measurement leaves out randomization. In other words, with replication, the measurement is not consecutive but spread out, while with the repeated measurement, you would measure your variable repeatedly in a row.

Blocking

Blocking is used to improve the measurement accuracy of experiments by blocking the effect of nuisance factors. Nuisance factors are factors we do not care about. For example, if you are trying to assess the impact on quiz scores but do not care whether the quizzes are in the morning or afternoon, you can block for the time of day. You then randomly assign people to each block and rn the experiment.

The goal is to create blocks that are as homogeneous as possible, which means only afternoon people in the pm block and only morning people in the am block. Doing this helps to control for the influence of time of day.

Conclusion

The topics discussed here are foundational to experimental design. However, we don’t want to give the impression that this is all there is that you need to know. Instead, what is discussed here serves as a guide concerning other topics that need to be investigated.

Types of Experiments

This post will provide some basic ideas for developing experiments. The process of doing valid experiments is rather challenging as one misstep can make your results invalid. Therefore, care is needed when attempting to set up an experiment

Definition

An experiment is a process in which changes are made to input variables to see how they affect the output variable(s). The inputs are called controllable variables, while the outputs are called response variables. Other variables that cannot be controlled are called uncontrollable variables.

When developing an experiment, the experimenter’s approach or plan for experimenting is called the strategy of experimentation. Extensive planning is necessary to conduct an experiment, while the actual data collection is often not that difficult.

Best Guess Approach

There are several different strategies for experimentation. The best-guess approach involves manipulating input variables based on prior results from the output variable. For example, if you are teaching a math class and notice that students score better when they work in groups in the morning compared to working in the afternoon. You may switch to group work in the morning and see if lectures may further increase performance.

This guesswork can be highly efficient if you are familiar with the domain in which you are doing the experiments. However, if the guess is wrong, you have to continue guessing, and this can go on for a long time.

One-Factor-At-A-Time

Another strategy of experimentation is the one-factor-at-a-time (OFAT) approach. You begin by having a baseline for each factor (variable) and then vary each variable to see how it affects the output. For example, you can switch whether students study in the morning or even and see how it affects performance. Then you might test whether group work and individual work affect scores.

The biggest weakness with this is that you can see interactions between variables. Interactions are an instance in which one factor does not produce the same results at a different level of another factor. Interactions can be hard to understand, but sometimes when two factors are mapped at the same time with the response variable, the lines cross to indicate that there is an interaction.

Interaction

Factorial Experiments

Factorial experiments involve varying factors together. For example, a 2^2 factorial design means four combinations of experiments with two variables are varied, and one response variable with four possible combinations of experiments. Often these types of experiments are drawn as a square, as shown below.

Factorial Design

Each point represents a different combination of the two factors. The calculation of this involves subtracting the means of the variable or factor on the x-axis. If we run each combination twice, we would calculate the difference, as shown below.

The more significant this difference, the more likely there is a strong effect based on the independent variables in the model.

When the number of combinations becomes large and complicated to manage, it may not be practical to run all possible combinations. In this situation, an experimenter will use a fractional factorial experiment in which only some of the combinations are used. For example, if 32 experiments are possible (2^5), maybe only 12 of them are conducted. The calculation is the same as above, just with more groups to compare.

Conclusion

Experiments are a practical way to determine the best combination of factors or variables for a given output variable(s). The majority of the time is spent planning and designing the experiment, with the actual data collection being straightforward.