To know the effect of a campaign it is important to measure the change in key revenue and profitability metrics for the customers who are part of the campaign. One possible way of measuring the effect is to find the change in key metrics before and after the campaign. This is called pre-VS post analysis where the effect of the campaign is measured on the customers who are part of the campaign only.
But this approach suffers from a serious limitation; this approach tells us the effect of the campaign but does not tell us what would have happened if the campaign was not carried out. For example, if one considers a campaign where bonus reward points are awarded to credit card customers for shopping above a particular threshold during the Christmas shopping time then one would of course observe an increase in sales before and after the campaign however it is difficult to say if the increase would have naturally occurred because of the festive shopping season or was the increase due to the offer.
To tide over this problem a control group is used to compare the performance of the customers who are part of the campaign (also referred to as the test group). The control group is created in such a way that these customers look exactly like the campaign or test group in the period before the campaign is rolled out. Hence after the campaign is rolled out the difference between the test and the control group provides the incremental effect of the campaign.
The control group needs to be designed in such a way that the test and control groups are exactly similar in the pre campaign period. For this purpose, the control group is selected randomly out of the eligible population after all filters are applied. After selecting the control group key revenue and profitability measures are tested across the test and control group to ensure that there is no significance difference between the test and control group. Table 1 illustrates the same:
Table 1 – Ensures that the test and control groups are similar before executing the campaign.
Similarly after the campaign is rolled out key metrics between the test and control groups are compared to see if there exists a significant difference between the two groups in the post campaign period. In case a significant difference exist only then can it be concluded that the campaign has an incremental effect otherwise any change in key metrics cannot be attributed to the campaign. The table 2 illustrates the test VS control comparison in the post campaign period.
Table 2 – Test vs. control analysis post campaign execution to understand if the campaign has made a significant difference to the test group
Some companies use a universal control group which is not part of any campaign or customer management activity, for each campaign the eligible population which is part of the universal control group is kept aside as the control group. However, a possible limitation of this approach could be that in some cases the eligible group might have very few customers from the universal control group, in that case the control group might become too small.
Creating a synthetic control group
In some cases designing the control group might be more difficult, the following two examples illustrates the same.
- Example-1: Above the line campaign: If one considers an above the line campaign like a campaign in restaurants which says that customers using their credit cards in the restaurant will receive 10% off on their bills. Now while rolling out the campaign it is impossible to know which customers will see the offer and hence it is impossible to select a control group beforehand. Only after the campaign is over and some customers have responded to the offer it is possible to know who all responded and then it is important to answer what is the incremental effect of this offer on their behaviour
- Example-2: Control group which looks like customers who respond to the offer: If one considers any direct marketing offer then a very small percentage of customers who are made the offer respond to it. The control group which is designed before the roll out of the offer looks exactly like the overall population which was targeted in the campaign. But the group which responded to the offer is a small, biased sub-sample of the population which is targeted. Now if one wants to answer how the campaign effects the responders only then one needs to curve out a group out of the control group which looks exactly like the responders. The overall control group which looks exactly like the overall campaign (test) population is not suitable for this purpose.
In both the above examples it is necessary to curve out a control group from the overall population (in the first case) or from the bigger control group (in the second case) which looks like the population of interest. A stratified sampling technique or a nearest neighbour technique is used for this purpose. The following example illustrates the stratified sampling technique for creating a desired control group.
If one considers two dimensions on which the control group needs to be like the test group and the two dimensions have three levels each namely High, Medium, and Low then the distribution of the test group is first checked along these two dimensions (depicted in table 3).
Table-3: Create a distribution of the test group which is used as a guide to create the synthetic control group.
Now the control group must also have the same distribution with respect to these two dimensions. If one wants to have a control group of size 1000, then the distribution of the control group should be as depicted in table 4.
Table 4 – Replicate the same distribution to create a synthetic control group.
Now in the group High-High the desired control group size is 50 hence out of the 500 customers in the control group for this group 50 needs to be selected randomly. Now if one looks at the Medium-High and Medium-Medium groups then one would find that the required number of customers in these two groups is more than the available number of customers hence this is clearly not possible. So given the control group that is there it is not possible to create a control group of 1000 hence one has to be satisfied with a smaller control group size. If one wants to create a control group of size 800 then all the proportions are less than 1 and hence it is feasible to create a control group of size 800 (the table 5 illustrates the same).
Table-5: The distribution of customers in the synthetic control group.
The above observation illustrates the limitation of approaches of creating control groups, in case there are many dimensions along which the test and control group needs to be identical then one might run into a sample size issue where one will not be able to find enough observations to choose from.
Per VS Post analysis
In cases where it is impossible to have a control group then one need to perform a pre VS post analysis on the campaign group itself for understanding the effect of the campaign. Though a pre VS post analysis does not provide the real impact of the campaign yet in the absence of a control group it is the best possible alternate that one can have. Similar to the test VS control analysis described earlier one can perform a hypothesis, the null hypothesis being that the pre and post metric are equal. Sometime one might use a percentage difference between pre and post as a measure as well. This assumes importance when the campaign needs to be compared with agreed benchmarks. For example a campaign might promise a 5% lift in minutes of usage (MOU). In this case one can calculate percentage lift = (MOU Pre – MOU Post) / MOU Pre and the lift needs to be compared to the benchmark set prior to the campaign. In the ideal case if a control group would have existed one should not only see the lift but would also examine incremental lift (Lift in test – Lift in control).
