As analytics capabilities have evolved across businesses and geographies, it has been observed that business managers expect analytics to provide insights into numerous questions. “Do our customers love a free gift more than a discount?” “Do our customers respond to advertising that contain the picture of a sports icon?” so on and many more… – These are examples of questions that a marketing manager will expect the analytics team to answer. As an analyst it will require us to delve deep into the data to find answers, using all the tools and techniques that we have at our disposal. But what if we do not have the data? If the company has never leveraged a sports icon for advertising or if it has never offered a free gift, then how will data help us answer the question? The situation where relevant data is unavailable is quite common. And when we encounter such a situation, we either have to take help of expert judgment, or try to identify suitable proxies or “ask the customer”. Once we are able to ask the customer, we are able to create the relevant data to answer the question of interest. The process of “asking the customer” entails creating experiments or tests where one is able to read the result and obtain answers to the questions of interest.
The concept of testing (A/B, Split-run, Flip-flop and test vs. control)
A/B testing, split run testing or test vs. control comparison are common methodologies that are adopted to understand the impact of single factor on customer behaviour.
Split run testing
In order to test the effectiveness of a marketing communication (mostly pint advertisement), one can either use a “split run” testing or a “flip flop” testing. Split run testing is by far the most effective way of testing a print advertisement. For running a “split run” testing two different versions of the same advertisement, each with a different identification number, are placed in the publication as a split insertion on the same date. This will ensure that exactly half of the publications will carry version one of the advertisement and the other half will carry the second version. Hence, the results of the split run test can be thought of as two advertisements run on a random sample of the publication. The way the advertisements are inserted ensures that the samples are absolutely random in every respect. A very similar concept can be used for testing website banner advertisements as well.
Flip flop testing
In case, a magazine does not offer the flexibility of running a split run campaign but has a separate regional publication for various regions, then one can use the region-1 publication for one version of the advertisement and the region-2 publication for the other version of the advertisement. This form of testing is called flip flop testing and is an approximation of a split run testing. The biggest shortcoming of this testing is that the two samples are not random and hence, there can be an inherent regional bias in the test results.
Test vs. Control
A control group is defined as a group of customers who are identical to the customers who are eligible for a campaign or any other targeted marketing action but are not subjected to the action under consideration. The behaviour of customers in the control group is compared with the behaviour of customers who are subjected to the marketing action. This comparison provides a good understanding of the impact of the marketing action in question.
Problems with traditional testing
The testing methodologies mentioned above provide robust answers about the incremental impact of a single marketing intervention (or factor) one at a time. However, in case the factors are too many in number? Then one needs to conduct a large number of such tests to ascertain the impact of each intervention (or factor). It is well know that it takes time and money to read the results of a test and infer the trends. Hence, in case one needs to test the impact of multiple factors, one needs to do something different so as to ensure that one can generate all the required learnings within the limited budget that is available. What does one need to do differently? Let’s find out using an example.
The concept of Design of Experiments
Marketers often need to test the impact of a wide range of targeting, advertising, promotion, pricing and product options so as to find out the optimal combination of factors to obtain all the desired results at the minimum possible cost. But testing has a cost associated with it. Hence, most companies set aside a testing budget that would allow them to test and learn. However, as the budget is always limited, it is never possible to test all combinations of every marketing parameter. Therefore, marketers often build a testing framework which helps them in identifying the critical few learning that they would like to derive out of the limited test budget that is available. In many cases, the concept of design of experiments is widely used in building the testing framework. Design of experiments or DoE is a common analytical technique to design the right testing framework. Though intuitively everyone understands the concept, yet a formal understanding is required to harness the maximum benefits of such a concept.
To illustrate the use of design of experiments, one could consider web banner advertising. There are multiple factors that affect the successes of a banner advertisement. However, before discussing the same, it is important to quantify the “success metric” for a banner advertisement. The most common success metric that is used is called the Click Through Rate (CTR). Click through rate is a very simple metric which is calculated as: Number of visitors clicking the link in the advertisement divided by the number of visitors who are exposed to the advertisement. The success of a banner advertisement depends on numerous factors. Some of the factors are: the website where the advertisement is displayed (possibly the most important), content of the advertisement, the placement of the advertisement on the website etc. In a digital world, it is possible to accurately measure the impact of all possible factors accurately; it is also feasible to ensure that a particular visitor is only exposed to a particular combination of the advertising variables. Hence, the concepts of DoE can be very accurately applied and measured in this scenario.
As stated above, the parameters that are of interest can be quite numerous. However, for the purpose of illustration, a simple example has been considered in this article. For simplicity, one can consider an advertisement, which consists of the following:
Marketers often need to test the impact of a wide range of targeting, advertising, promotion, pricing and product options so as to find out the optimal combination of factors to obtain all the desired results at the minimum possible cost. But testing has a cost associated with it. Hence, most companies set aside a testing budget that would allow them to test and learn. However, as the budget is always limited, it is never possible to test all combinations of every marketing parameter. Therefore, marketers often build a testing framework which helps them in identifying the critical few learning that they would like to derive out of the limited test budget that is available. In many cases, the concept of design of experiments is widely used in building the testing framework. Design of experiments or DoE is a common analytical technique to design the right testing framework. Though intuitively everyone understands the concept, yet a formal understanding is required to harness the maximum benefits of such a concept.
- A picture
- A text message about the offer and product
- A link which will take the visitor to the web page of the advertiser. The link which will take the visitor to the web site of the advertiser is termed “Call to Action”
This example will involve the following parameters.
- Position of the picture: Left, Right, Middle
- Position of the Call to Action link: Top and Bottom
- Presence of animation or movement in the picture: Yes, No
- Position of the banner advertisement on the web page: Left and Right
The parameters (mentioned above) are also referred to as factors, and the values that a parameter or factor takes is often referred to as levels or attributes. For example “Position of the picture” is a parameter or factor, and the values that it takes i.e. “Left”, “Right” and “Middle” are levels/attributes. Figuie-1 illustrates the combinations (other than the presence or absence of animation).
Figuire-1: Depiction of the parameters of banner advertisement
In order to ascertain the effectiveness of all these components, it is critical to conduct experiments where visitors are exposed to all possible combinations of the above, and the effect of the same is measured on the click through rate.
Table-1 depicts the total possible combinations. The cells marked in grey are the ones which take a value of zero for that particular combination. For example:
- The combination C1 involves:
- Position of picture: left
- Position of call to action link: top
- Presence of animation: yes
- Position on website: left
Table-1: All possible combinations of the parameters
It can be observed that are 3 possible positions of the picture, 2 possible positions of the call to action link, 2 configurations with regards to animation (presence or absence) and 2 possible placements on the web site (left or right). Hence there will be 3*2*2*2 = 24 combinations that one could have; this is a large number of possible combinations to explore individually. Marketers have used concepts of design of experiments to limit the number of combinations (out of the set of all possible combinations) that need to be tested to make meaningful inferences. In understanding how design of experiments can help one in limiting the number of combinations that need to be tested, one needs to understand the effects of each attribute or level separately and the effect of these attributes acting in tandem.
Design of Experiments without Interaction Effects
The levels of a particular parameter or factor are used as variables for constructing the response function for each combination listed in Table-1. For example the factor “Position of picture” comprises of 3 levels. Therefore, due to degree of freedom constraints, it would require two variables to construct the response equation; any two of the levels can be used as binary variables. In case of position, one can use “Left” and “Right” as two binary variables. If the picture position is on the left then the binary variable “Left” takes the value of 1 otherwise it takes the value of 0. If the picture position is on the right then the binary variable “Right” takes the value of 1, otherwise it takes the value of 0. If the picture position is in the middle, then both the variables “Left” and “Right” takes the value 0.
Similarly one would use 1 variable each for the other parameters (as all the other parameters consists of two levels each). If one assumes no interaction effect between the factors then the generic response function can be written as:
Ln(CTR/(1-CTR)) = α + β1(Position of picture is left) + β2(Position of picture is right) + β3(Position of call to action link is top) + β4(Presence of animation is yes) + β5(Placement on web site is left)
In this expression “CTR” represents the probability of response or the click through rate. The β’s represent the effect of each attribute or level on probability of response.
Based on past experience, it has been found that in most cases response can be predicted by using a logistic function. The generic response function needs to be applied to each design combination. The resulting function for each design combination is depicted in Table-2.
Table-2: The Response Equation for all Possible Combinations of the Parameters
From the table, it can be observed that if one tests combination C4 (ln(CTR4/(1-CTR4))=α+β1 +β3 ) and C23 (ln(CTR23/(1-CTR23))=α +β5), then one could easily estimate the click through rate for combination C3 (ln(CTR3/(1-CTR3))=α+β1 +β3 +β5). It can be seen that:
This feature is the key benefit of a properly designed experiment or test. By performing a limited number of tests it is possible to infer the results of some of the combinations, which have not been tested. The importance of this result lies in the fact that marketers always have a limited budget for testing and hence it is always crucial to identify the limited number of tests that needs to be performed to the derive maximum amount of learning.
(ln(CTR4/(1-CTR4) ) + (ln(CTR23/(1-CTR23) ) = (ln(CTR3/(1-CTR3) )
That is interesting isn’t it? Let’s take a pause here and ask – “If we want to know the effect on CTR if position of picture is in the right (i.e. find out β2) then which results do we need?” Hope everyone has got it right. So let’s move on and explore a bit more complex ideas.
A case, where one tests all the combinations involved is referred to as “full factorial design”. On the other hand, as mentioned above, if the marketer is able to eliminate certain combinations, and test a limited set of combinations, then the same is referred to as “partial factorial design”
Table-3 illustrates how a limited set of experiments that can be used to compute all the required test results.
Table-3: The Partial Factorial Design
The analytical objective involves estimating the coefficients α, β1, β2, β3, β4, β5. The following combinations can be used to estimate the coefficients:
- Estimating α: If one has results of experiment C24 one will be able to ascertain the value of α
- Estimating β3: If one has the results of C4 and C8 then one can obtain the value of β3.
- Estimating β2: If one has the results of experiment C12 then one can plug in the values of α and β3 to obtain β2
- Estimating β4: If one has the result of C10 one could use the values of α β2 and β3 to obtain β4
- Estimating β1: The value of β4 can then be plugged into the result of C6 to obtain β1
- Estimating β5: The value of β5 can be obtained by plugging in the value of β2 into the result of experiment C15
It can be observed that by conducting only 7 experiments (C4, C8, C12, C14, C6, C10 and C15), one can obtain all the information that can be obtained by conducting 24 experiments. Hence, the concept of design of experiments can be used to reduce the experiments from 24 to 7.
The property mentioned above, is the major benefit of a partial factorial design, whereby one can obtain the required learning without conducting all the possible experiments. However, as mentioned earlier, this approach assumes that there exists no interaction between the factors. It will be a worthwhile exercise to find out the minimum number of experiments that one will have to perform if presence of interaction is considered.
Design of Experiments with Interaction Effects
As a critic of the partial factorial approach, one could argue that the combination of an animation and a placement of the advertisement to the right of the website would be more effective in conjunction, because most viewers tend to focus on the right side of the screen. This implies that the interaction between placement and animation needs to be taken into account. Hence the generic response function would take the following form:
- Ln(CTR/(1-CTR)) = α + β1(Position of picture is left) + β2(Position of picture is right) + β3(Position of call to action link is top) + β4(Presence of animation is yes) + β5(Placement on web site is left) + β10(Placement on web site is left & Presence of animation)
It would be worthwhile to find out the minimum number of experiments that one will have to conduct if one assumes the presence of interaction effects. It can be easily seen, that it is difficult to limit the number of experiments or tests that needs to be conducted if there are significant number of interactions. To generate the maximum learning from any test program, it is best to adopt a full factorial test design whereby all the possible combinations are tested; however, because of cost constraints a partial factorial design is often favoured. While adopting a partial factorial design, appropriate assumptions about interaction effects need to be put into place to limit the number of experiments that one needs to conduct. Based on prior business knowledge one can eliminate certain interactions, thereby reducing the number of tests that should be performed. In this case, if one assumes that the only interaction effect that exists is between the placement of the advertisement and animation, then it will be interesting to find out the number of tests that needs to be conducted to estimate all the coefficients involved.
