Skip to Content

A Retrospective Investigation of Test & Learn Business Experiments & Lift Analysis

Parijat Rai, Manya Mittal, Snigdha Bhargava, Matthew A. Lanham
Purdue University Krannert School of Management
rai21@purdue.edu; mittal46@purdue.edu; bharga16@purdue.edu; lanhamm@purdue.edu

Introduction

Retailers often run promotions to increase demand for their products. The goal is often to design and run these promotions to maximize profit margins. Often, a category manager might run a promotion without expected knowledge of how such an intervention would perform. Some retailers run ‘test & learn’ experiments, meaning they try to employ a design of experiments (DOE) approach to the rather chaotic and often uncontrollable world of business. The motivation behind this research is to identify the products on which the promotion should be applied to increase revenue. Next, identify products that have similar sales patterns, and find the associated increase/decrease to validate the profit/loss margins, when the intervention (i.e. promotion) occurs. This allows the retailer to estimate the impact of the promotion, often referred to as ‘lift’. An illustrative example is shown below:

Estimating Promotional Lift

The problem is many retail promotions are not designed in a ‘test & learn’ fashion, making it challenging for the category manger to identify what the affect of the promotion really was. Thus, our research question in collaboration with a major retail partner follows: Is there a way to retrospectively estimate the effect of the promotion ran and estimate the corresponding lift?

Methodology

Data Sources

The scope of the data provided:

  • Spanned 3 years with over 5 million rows and 30 attributes
  • Contained 6 major categories/classes of products for one US market
  • Measured 30 attributes containing information such as product hierarchy (classes, subclasses, SKUs), sales dollars, SKU costs, and when the SKU was on and not on promotion

The overall methodology conducted can be seen below:

Exploratory Data Analysis

The analysis started by descriptive statistics to gauge the variables present in the dataset. We built different distributions between independent and dependent variables to understand if any trends or patterns seemed apparent, and assessed correlation matrices to check relationships between numeric variables. We also analysed the distribution of sales units and sales dollars, laying the foundation for the baseline log-log model.

Data Preparation

The next step involved preparing the data for conducting t-tests and building the log-log models. We noticed that there are certain negative sales dollar values, which were those items returned by a customer. To adjust for this, we added a constant (the same sales dollar value) to the sales dollar value (of when the item was bought and when the item was returned). According to the published research, only the intercept in the model is affected and the coefficients of the model remain the same. Log transformations were applied to improve normality and for variance stabilization.

Figure of methodology

One-side t-test

Compute the average lift change of each product: A one-sided t-test hypothesis test was used to identify whether or not the application of promotional activity brought about a conjoint change in the average units sold and dollar value associated with them.

𝐻0: Promotional activity does not affect sales units & dollars

𝐻𝑎: Promotional activity does affect sales units & dollars

Model Building

To determine the promotional price sensitivity (i.e. elasticity) of each item, a log-log model was employed which allowed us estimate the percentage change in demand for a percentage change in price and while being on promotion.

formula for log-log model

Time-Series Intermittent Forecasting

Predictive modelling techniques such as intermittent time-series forecasting was used to estimate the baseline sales, thereby accounting for the seasonal effect which was critical for the lift analysis of incremental sales. We employed the R imputeTS library to generate these forecasts.
Next, to answer the business problem at hand, a drilled down analysis of calculated elasticity versus the estimated lift was conducted, thereby enabling the decision makers to see the relative effect of pricing promotion in terms of dollar value, units and profit margins. 

Results

The hypothesis testing performed for the 50 unique items present under one selected class of products indicated that there is a need for a change in the inner workings of the company to achieve its end goal of profit maximization. The results to study the impact of promotion applied at a 95% confidence interval suggests –

Data table: impact of promotion applied

The log-log model, a famous econometric model to measure the constant demand elasticity, further underscored the following result. Out of the 50 items tested for, it is visible that only 14% of them are clearly price elastic and thereby should be the focus of the firm to provide more promotional offers on them as they generate high revenue.
The coefficient in the log-log model related to the ‘Promotional Flag’ variable gives the % change in demand of sales units if the promotion is applied.

This figure below shows the % change of demand of SKUs where there were promotions applied at some given point of time & the promotion is significant to change in sales units.

Figure: % change of demand of SKUs

The deep-dive analysis at an item level confirmed the hypothesis that promotions are indeed effective in producing a lift in unit sales. It was assumed that in absence of promotions the total sales are the baseline sales. The baseline sales for the weeks where promotion was applied were forecast using intermittent time-series forecasting techniques. The method used for forecasting was seasonally decomposed missing value imputation which removes the seasonal component from the time-series and adds it back once the imputation has been performed The incremental sales were hence estimated by taking the difference of the total sales and the estimated baseline sales.
The graph on the left depicts the intermittent time series for a product where the missing data indicates the weeks where promotion was applied. The graph on the right depicts the baseline sales (red) and total sales (green). For this particular product an average lift of 17% per week was observed in weeks where a promotion was applied. The analysis can be replicated to other products and categories enabling better item specific decisions.

Graphs: The intermittent time series for a product where the missing data indicates the weeks where promotion was applied; and the baseline sales (red) and total sales (green)

Conclusions

The analysis provided an overall confirmatory expectation that these promotions were indeed effective, but also identified products where certain promotions should not be run going forward, as they created no extra profit margins in the long run. A deep-dive analysis of effect of promotion on incremental sales, price, and profit margins of a product can help stakeholders make better decisions. We believe the approach is both a novel and practical solution to retrospectively understand promotional effects that all retailers could implement to help improve their revenue management.

Acknowledgements

We thank our Professor Matthew Lanham for constant guidance and the Fortune 500 Retailer for giving us the opportunity to work on the problem. The Purdue BIAC partially funded this project.