Testing the science of direct marketing
One of the important advantages of direct marketing as opposed to advertising and other kinds of marketing is the ability to test—copy, messaging, offers, and creative. At Marketing General Incorporated, we always try to improve our most successful promotions and better our clients’ results through continuous and rigorous testing.
Sometimes analogies are the most appropriate learning tools. Recently, MGI Account Supervisor Carol Cohen had market testing on her mind as she was helping her 12-year-old son Matthew with a school science project. She discovered that her son was actually learning the scientific method of hypothesis proof and the principles of experimental design.
The Chocolate Chip Cookie Hypothesis
Matthew set about to test whether adding baking soda would have an effect on the chocolate chip cookies he was baking, and if there was an impact, how much.
“In direct marketing we call them tests instead of experiments,” Carol said, “but they are essentially almost the same thing. And to do it right, we need to understand and follow scientific methodology.”
Here’s how both professional marketers and 12-year-old science students go about applying the scientific method.
The recipe for testing success
First we’ll review hypothesis testing, and then we’ll apply them to our projects.
| Step 1.0 |
Make a Hypothesis, which is nothing more than an educated guess. For example, “I think my direct mail effort will produce more members if I offer a $25 discount on dues.” No matter how sure you are that your hypothesis is correct, it’s still just a guess until you’ve tested it. |
| Step 2.0 | Design Your Test. This is the meat of the scientific method. First you must identify the dependent and independent variables. By definition, variables vary. The difference is what causes the variance. An independent variable is one that you manipulate, such as the membership offer. A dependent variable changes as a result of how you manipulated the independent variable and this is what provides your measurable data. In direct mail, the dependent variable is usually the response rate. |
| Step 3.0 | Keep a Control. It’s necessary to have a baseline to measure your test results, so you will need to establish a control. That requires assigning a portion of your marketing campaign to the independent variable, which remains unchanged. The only difference between your control and your test is the one independent variable that changes in the test. The control results are predictable based on prior results, in this instance the $25 discount on dues. |
| Step 4.0 | Measure and Record. Plan how you will know the source of responses to your marketing efforts (usually using source codes), so you can identify which come from the Test and which from the Control. Keep track of all this data and record it. |
| Step 5.0 | Draw Conclusions. Do your results prove or disprove your hypothesis? |
| Step 6.0 | Repeat. Ideally every direct marketing test should be retested once to validate the original results. Only when the test is repeated with the same outcome can we be certain of the results. |
Hypothesis testing
Let’s see how the scientific method applies in two very different situations—a sixth-grade science experiment and an association membership direct mail campaign.
| The Effect of Baking Soda on Chocolate Chip Cookies | The Effect of a $25 Dues Discount on Direct Mail Response Rates |
|
| Hypothesis | Increasing the amount of baking soda used will increase the height and width of homemade chocolate chip cookies. | Offering a $25 dues discount will result in increased direct mail response rates for ABC Association. |
| Test Design | Bake a batch of cookies with an extra teaspoon of baking soda.Independent Variable – The amount of baking soda. Dependent Variable – The height and width of the cookies. (Everything else – other ingredients, mixing and baking times, oven temperature, etc. – remains the same as the Control.) |
Mail ABC Association’s existing direct mail package with a $25 dues discount offer. Independent Variable – The offer price. Dependent Variable – The response rate. (Everything else – lists, copy, design, mail date, etc. – remains the same as the Control.) |
| Control | Bake a batch of cookies following the recipe exactly without the baking soda. | Mail ABC Association’s existing membership package with the regular dues price. |
|
Measure |
Measure the height and width of both batches of baked cookies. The test cookies’ width increased, but the height decreased. The hypothesis is partially correct. |
Measure response rates to both the Control and Test offers. The test package’s response rate was higher. The hypothesis is correct. |
| Repeat | Repeat Steps 1.0 – 5.0. Are the results the same? | Repeat Steps 1.0 – 5.0. Are the results the same? |
What have we learned?
The goal of testing in direct marketing, just like tweaking your favorite cookie recipe, is to improve results—to get more members, more revenue, or both. So don’t think of it as a one-time experiment. Keep developing new hypotheses and testing them using the scientific method. This disciplined approach is the best way to continue to learn about your market and optimize results. And, it also makes the best cookies!
Do you have questions?
If your organization would benefit from a discussion about MGI’s performance measures through scientific methodology, you can find out more by contacting John Sample, Senior Relationship Director, email at JSample@marketinggeneral.com, or call 703-706-0346