Key Principles of Experimentation

When researchers are faced with a question, their response is often to collect data so they can understand answers to that question. While this seems like a great idea, without a plan, it is likely to be inefficient and may lead to erroneous conclusions. Experimental design is the process of creating a plan, implementing that plan, analyzing the results, and making proper conclusions. This plan creates an efficient and effective road map that will help the researcher better understand the problem and the variables that they have questions about. A well-designed experiment will allow the researcher to minimize the effect of variables that are not of interest and reduce unwanted variability, making it easier to better understand the effect of variables that are of interest.

This textbook will explore the necessary steps to consider as you plan and perform an experiment. These steps include:

  1. Understand the objectives of your research. What questions should be answered?
  2. Determine the measurements that will be needed to help answer the questions of interest.
  3. Determine which variables, which we will call factors, are the sources of variation in the measurements. Determine which of these factors you will be testing and decide which levels of those factors will be included in the study. Determine which of these factors you will hold constant.
  4. Determine how sampling and treatment assignments will be done and then collect the data.
  5. Identify the analysis technique that will be applied, including how to test hypotheses
  6. Describe the data using numerical and graphical summaries.
  7. Perform the analysis and make sure that the analysis is appropriate.
  8. Make conclusions.

This textbook will further discuss each of these steps in detail and offer examples. Steps 1-4 will be discussed in the “Basics of Design” section. Steps 5, 7, and 8 will be discussed in the “Specific Designs” and “Broad Topics” sections. Step 6 will be discussed in the “R Instructions” section.

Before detailing each of these steps, it is important to understand the differences between observational studies and designed experiments. In observational studies the researchers do not control the conditions of the study. The data are collected by observation. In this case, the researchers cannot assign the subjects to the conditions or treatments.

An example of this would be studying to see if there is a difference in the amount of wind at the three Brigham Young University (BYU) campuses. In this study the factor is campus. The campus factor has 3 levels: BYU, BYU-Idaho, and BYU-Hawaii. The measurement taken would be the average wind speed for a particular day. The researchers cannot assign the particular days to a specific campus. Instead, they may select random days and then collect the data for those days. This data would be collected through observation, not through experimentation.

Designed experiments are studies in which the researchers control the conditions in which the study is performed. In designed experiments the researchers assign subjects to conditions. The method used in making those assignments will be discussed further in the “Design” section of a Specific Design’s page. An example of this would be determining which of four different toothbrush types are better at reducing plaque. The researchers would randomly assign each subject to one of the toothbrush types and then carry out the experiment.

Experiments have the advantage over observational studies in isolating a factor’s effect on a response, and thereby proving causality. Sometimes however it is not ethical to assign someone to a condition that is of interest, for example you should not assign anyone to experience the effects of smoking - even if you wanted to study the impact of smoking. In some cases, it may be impossible to assign someone to a condition, for example gender. Furthermore, an observational study may be preferred because it may be a more realistic view of how something will truly play out “in the real world”, rather than in a contrived lab experiment.

Although both types of studies can be used to better understand and answer research questions, this textbook will mostly focus on the steps needed to best design an experiment. The toothbrush study just introduced above will be used to help illustrate how to work through the process of designing an experiment.