On this page
- Why do research on your own farm?
- Tips on making your on-farm trial comparisons fair
- Interpreting the results of your experiment
Why do research on your own farm?
Experiments are hard work and can be difficult to design and analyze. The University of Minnesota conducts large numbers of agronomic experiments as do many other public institutions and private sector ag companies. With this abundance of information, why should you bother with experiments on your farm?
There are several reasons why you may want to conduct your own on-farm experiment (trial). You may want to test agronomic practices or products that you cannot find adequate information on. You may have some unique condition on your farm that you think would produce atypical results. You have a healthy skeptical attitude and need to see things for yourself. For example, you might want to compare yields of corn hybrids, a foliar fungicide compared to an untreated, or the effect of two tillage systems on soybean yield.
This article provides some advice in planning and analyzing your experiments.
Tips on making your on-farm trial comparisons fair
Where to place your trial
Figure 1. Yield variability in a soybean field due to soil conditions. Soil type and other factors should influence plot placement and be considered when understanding average yield.
Some care should be taken when selecting the areas of the field to place the blocks or strips of your trial treatments (things you are trying to compare).
Avoid, as much as possible, areas of the field with obvious different areas of drainage, topography/soil type, and recent cropping histories.
Try to select areas that are similar in yield potential and any other effect you are trying to test with your treatments, crop pests for example. In the soybean yield map shown in Figure 1, notice the variable yield due to soil conditions. If treatments were placed in this field as indicated by the red boxes, yield differences because of field conditions are likely to overwhelm any treatment comparisons.
Additionally, individual fields can vary greatly in cropping and management histories. As a result, it is risky to compare two fields, one treated with insecticide and one untreated for example, and draw an accurate conclusion on the treatment’s effectiveness.
Keep things consistent
Unless they are what you are trying to compare, keep as many things as possible between your treatments the same. Use the same seed lot, planting date, equipment, and non-tested additives for all treatments. When comparisons are influenced by variables other than the one you are testing, the data are said to be confounded. It is difficult, if not impossible, to determine what the real treatment effects were in your trial if there were confounding factors affecting the results.
For example, if you are running a ground sprayer as part of your experiment, run the sprayer with the booms off in the untreated treatment areas. This will keep things fair in case you harvest any wheel track areas in your sprayed areas. Comparing a treatment containing nitrogen fertilizer with one that does not may confound your results if the fertilizer effects crop growth. Differences in yield when comparing treated and untreated seed from different seed lots might be due to the seed treatment or to the seed itself.
Don't rely on too few measurements
Whether measuring weights of a group of people or crop yields in a field, few, if any, measurements are exactly the same.
Figure 2. The normal distribution. There are values above and below the mean or average (0 in this example). The percentages shown (confidence limits) represent 1, 2 and 3 standard deviations from the mean.
The previous yield map example shows that if you take multiple yield measurements, you are likely to end up with multiple estimates of yield for the field. The more measurements or samples you take, the more closely you are likely to estimate the actual yield for the field. If you take enough samples, you will know the average (or mean) yield for the field. You will also know how variable the yield within the field is. You could diagram the yield in the field similar to the graphic of normal distribution (the bell-shaped curve as shown in Figure 2). You would have approximately the same number of sampled yields that are above the mean and as sampled yields that are below. When you are comparing experimental results for two or more treatments, you are comparing the means of the treatments! The variability of those two means determines how easy it is to make those comparisons and detect differences. When the data do not follow this symmetrical normal relationship, correct analysis is more difficult, but fortunately, yields are typically normally distributed.
The need to compare things more than once
Figure 3. A representation of two different treatment mean yields and distributions, along with a yield sampled from each treatment. Is the difference 40 bushels or is it 10 bushels? Depending on which samples are taken, treatment differences in a trial may appear larger than they actually are or differences found that are not real. This is called a Type I error.
We mentioned that provided they are taken well, the more samples you take, the better you can estimate variability of the sample data and thereby the true mean.When you set up an experiment in a field, you are only obtaining estimates of yield from part of the field. If you compare a single strip with fungicide to a strip without, you have two yield samples but no estimate of how variable the measurements are within the field and within treatment. In statistical terms, you have a single replication of each treatment. Depending on how and where the samples were taken, they could be representative of the results or they could be on one tail or another of the curve (as shown in Figure 3).
Figure 4. If samples from the two treatments are taken from the area where the two curves overlap, the means would appear the same, even though they are really different. This is a Type II error.
On the other hand, your single samples may both represent an area where the tails of the two treatments overlap and would indicate yields are the same (see Figure 4). How would you know?
Taking more samples within a treatment area will give you an estimate of variability within that treatment. However, simply taking more samples within a treatment will not improve your ability to measure variability across the field.
Repeating (replicating) treatments by replicating plots or strips provides the ability to assess the variability and estimate the mean across the field. More replications will improve your ability to accurately estimate means, variability and differences between treatments. Three replications of each treatment should be considered a minimum, and practical considerations of space will limit the number of treatment replications possible with farm scale equipment.
The need to randomize
Figure 5. A hypothetical small plot, two treatment (A & B) experiment with four replications of each treatment. The purple single row lines in each treatment area represent single row subsamples within each replicate. These plots appear to be placed in regular rather than randomized order.
Because of within-field variability and inconvenient statistical analysis assumptions on randomness and equal probabilities, you need to pay some attention to how your treatment replications are placed in the field. Obviously, placing all of the replications for one treatment at one end of the field and all of the replications of the other treatment at the other end could lead to some erroneous conclusions and, in fact, could mean you have multiple samples for a single replication of each treatment. Regular alteration of plots should be avoided when possible. This is because soil type and other variability within the field are not regularly or uniformly placed. Regular alteration might be unavoidable with only a few couple treatments in a field (Figure 5). Placement of your plots or strips can be randomized with a calculator function or a coin flip if going low-tech.
Beware of replication that isn't
Multiple samples within a plot or a strip are known as subsamples; they are not replicates. Subsamples can improve the accuracy estimates of a plot but not between plots. Analyzing these subsamples as individual plot means or replicates is known as pseudo replication and should be avoided.
Interpreting the results of your experiment
You have designed your trial, added your treatment with the comparisons you want to make to your plots, and . . . now what?
Just like deciding on and placing your treatments, how you interpret the results can make a great deal of difference in the usefulness of the conclusions you draw from your on-farm experiment.
Do you remember the bell-shaped curve? For this discussion, we will assume that we are comparing yield between two or three treatments and the yields follow the normal distribution. The shape of this curve will determine how easy it will be to draw good conclusions from your data. If the distribution of the data has narrow tails, most of the samples you take will be close to the true mean. If the data are more variable because of inconsistent responses to your treatment or uncontrollable variables such as soil type, the tails of the curve will be spread out further from the true mean.
If you conduct a field experiment comparing, for example, yield of corn from multiple treatments, you only have yield from the replications. This is only a subset of data representing the possible yield in the field (or other fields) for each treatment and thus you do not know the true mean of each treatment. In other words, you are comparing sample means to estimate the true means.
It is easier to make accurate estimates of the true treatment means for data that is less variable. You are less likely to obtain a similar treatment mean when you take additional samples for an experiment where the data is highly variable. The lack of precision allowed by highly variable data can lead to much different, less accurate mean estimates if the experiment is resampled or repeated. A way to improve accuracy is to take more samples (replications).
It is easy to estimate the treatment means for your experiment – you simply average the replicated samples for each treatment. Variability can be estimated by calculating the standard deviation of the mean. The number of replicated samples in your mean is included in the calculation of the standard error of the mean (SEM):
SEM=Standard deviation ÷(√number of samples-1).
One simple way to compare two treatments is to compare the two means as mean +/- standard error. If the means are different by more than 1 standard error the true means are probably different. If they are different by 2 SEM, they are even more likely to be different.
How confident are you?
Confidence intervals can be calculated to describe how confident you can be of the accuracy of your mean estimation and comparisons between means. A 90% confidence interval (C.I.) means that there is a 90% probability that the true mean lies within your range of estimated values. Remember the normal distribution curve? The 90 % C.I. means 45% of the possible sample results would be greater than the average and 45 % below. It also means you have 10% chance that your results are outside those limits (5% above and 5% below). This 10% chance is usually referred to as alpha or α (least significance difference/LSD value of .10 or 10% values are similar). Other alpha values and C.I. probabilities can be calculated, 80% being used in some variety testing evaluation. 95% is commonly used as a standard for academic research with some requiring 99%, or even 99.9999% when NASA wants to land a rover on Mars.
Figure 6. Two treatments have sample means for Trt 1 and Trt 2 of 284 and 275, respectively. The vertical error bars show the impact of increasing the confidence interval on making a decision on whether means are different.
As exemplified in Figure 6, selecting 80% or 95% would indicate that the means being compared are different while selecting the very strict 99% would indicate they are not.
Unless we sampled all treatment combinations in all fields, we do not know what the true mean is. It is your decision on which type of error you are more willing to risk, saying the means are different when not (Type I) or saying means are the same when they are different (Type II).
So what does this mean for interpreting your experiment? Choosing a low C.I., like 80%, means that you are more likely to find means different but are less confident that they actually are different. Choosing a high C.I., like 90%, increases the probability you are making the correct decision on whether two means are different but also increases the chances that you will miss real treatment differences. If you are choosing between treatments that have little downside you might select a lower confidence interval for your test. Examples might include, did an insecticide provide a yield benefit in your field or which hybrid yielded more? If there is a significant economic, safety or health risk, use the opposite approach. Statewide pesticide recommendations, comparing the stability of explosive formulations, and pharmaceutical safety testing would benefit in interpreting comparisons with a higher degree of confidence (e.g. 95% or more) and testing under more environmental conditions.
We have prepared an EXCEL spreadsheet that allows you to compare two or three treatments using a 2-tailed t-test. It allows you to select the precision (80%, 90%, or 95% C.I.) for your comparison and allows to enter 3 to 8 replicated samples for each treatment. These are for real replicates and not pseudo replicates that we mentioned earlier.
Experiment with different replicate sample averages and variabilities, replicate numbers, and confidence intervals. The spreadsheet can be used to compare yield, plant populations, percent control for a crop protection chemicals, and other treatments.
You can download a copy of the software here:
Tips for on-farm research
- Do not try to compare too many treatments in one trial. If you are trying to compare more than 3 or 4 treatments, then you should seek help from a statistician.
- Make sure you have a control treatment. Include a common treatment or untreated plots in your comparison.
- Avoid placing treatments in locations where cropping history or soil factors would affect treatments differently.
- Include replications (more than one plot for each treatment). Three or more replicates are needed to determine the consistency of results. Multiple samples from the same plot or strip is not replication (they are pseudo replications).
- Randomize the order in which your treatment plots/strips occur in the field (planting order of varieties, which plots receive which varieties).
- Understand how variability influences your ability to draw conclusions. Do you want to risk calling treatments different that are essentially the same or do you want to risk not finding real treatment differences?
- Don’t over-extrapolate and assume the rest of your experiment has to be valid for other fields.
For more information
Another, more comprehensive publication on on-farm research can be found at:
For more crops information, visit http://www.extension.umn.edu/crops/