Analytics Strategist

March 16, 2009

the new challenges to Media Mix Modeling

Among many themes discussed in the 2009 Digital Outlook report by Razorfish, there is a strand linked to media and content fragmentation, the complex and non-linear consumer experience, interaction among multiple media and multiple campaigns – all of these lead to one of the biggest analytics challenge: the failure of traditional Media Mix Modeling (MMM) in searching of a better Attribution Analytics.

The very first article of the research and measurement section is on MMM. It has some of the clearest discussion of why MMM failed to handle today’s marketing challenge, despite of its decades of success.  But I believe it can be made clearer. One reason is its failure to handle media and campaign interaction, which I think it is not the modeling failure but rather a failure for the purpose of attribution ( I have discussed this extensively in my post: Attribution, what you want may not be what you need).  The interaction between traditional media and digital media however, is of a different nature and it has to do with mixing of both push and pull media.  Push media influence pull media in a way that render many of the modeling assumptions problematic.  

Here’s its summary paragraph:

” Marketing mix models have served us well for the last several decades. However, the media landscape has changed. The models will have to change and adapt. Until this happens, models that incorporate digital media will need an extra layer of scrutiny. But simultaneously, the advertisers and media companies need to push forward and help bring the time-honored practice of media mix modeling into the digital era.”

The report limit its discussion to MMM, the macro attribution problem.  It did not give a fair discussion of the general attribution problem – no discussion of the recent developments in attribution analytics ( called by many names such as Engagement Mapping, Conversion Attribution, Multicampaign Attribution etc.).  

For those who interested in the attribution analytics challenges, my prior post on the three generations of attribution analytics provide an indepth overview of the field. 

Other related posts: micro and macro attribution and the relationship between attribution and  optimization.

Advertisements

March 10, 2009

fairness is not the principle for optimization

In my other post, what you want may not be what you need, I wrote about the principle of optimization. Some follow up questions I got from people made me realize that I had not done a good job in explaining the point. I’d like to try again.

Correct attribution provides business a way to implement accountability. In marketing, correct attribution of sales and/or conversions presumably help us optimize the marketing spend. But how?  Here’s an example of what many people have in mind:  

    Suppose you have the following sale attributions to your four marketing channels:
             40% direct mail
             30% TV
             20% Paid Search
             10% Online Display
    then, you should allocate future budget to the four channels in proportion to the percentage they got.

This is intuitive, and perhaps what the fairness principle would do:  award according to contribution.  However, this is not the principle of optimization. Why?

Optimization is about maximization under constraints.  In case of budget optimization, you ask the question of how to distribute the last (or marginal) dollar more efficiently.  Your last dollar should be allocated to the channel with the highest marginal ROI.  In fact, this principle dictates that as long as there is difference in marginal ROI across channels you can always improve by moving dollars around.  Thus with true optimal allocation, the marginal ROI should be equalized across channels.

The 40% sale/conversion attribution to Direct Mail is used to calculate the average ROI.  In most DM programs, the early part of the dollar goes to the better names in the list, which tends to contribute to higher ROI; on the other hand, the fixed cost such as cost incurred for model development effort etc. will lower the ROI for the early part of the budget.  ROI and marginal ROI are variable functions of budget, and the marginal ROI in general is not equal to the average ROI.  There are different reasons for every channel with similar conclusion.  This is why those attribution percentages do not automatically tell us how to optimize. 

You may ask that, assuming all the marginal ROI are proportional to the average ROI, are we then justified to use of attribution percentages for budget allocations?  The answer is no.  If your assumption is right you should give all your dollars to one channel with the highest ROI, but not to all channels in proportion to the percentages.

This is an example of macro attribution. The same thinking applies to micro attribution as well.  Attribution is seen as linked to accountability and further more to operation and/or budget optimization.

We used an example of macro attribution to illustrate our point; same thinking applies to micro attribution as well.  Contrary to commonsense that regards attribution as the foundation for accountability and operation optimization, attribution percentages should not be used directly in optimization. The proportional rule or the principle of fairness is not the principle for optimization.

March 2, 2009

the first 3 insights on attribution analytics

Looking at micro attribution from a conversion modeling framework, there are a few insights we can contribute right away without getting into the details.

1)  The sampling bias

If your attribution analysis used only data from convertors, then you have an issue with sampling bias.

As a first order question for any modeling project, understanding the data sample, therefore the potential sampling bias, is crucial.  How is this relevant to the attribution problem?

Considering a hypothetical, but commonly asked type of, question:

What is the right attributions to banner and search given that I know the conversion path data:
    Banner -> conversion:   20%
    Banner -> search -> conversion: 40%
    Search -> conversion: 25%
    Search -> Banner -> conversion:  15%

Well, what could be with the question?  A standard setup for a conversion model is to use conversion as the dependent variable for the model with banner and search as predictors. The problem here is, we only have convertor cases but no non-convertor cases.  We simply can’t perform a model at all.  We need more data such as the number of click on banner but did not convert.

The sampling bias issue is actually deeper than this.  We want to know if the coverage of banner and search are “biased” for the data we are using, an example is when banner were national while search were regional. We also need to ask, if the future campaigns will be run in ways similar to what happened before – the requirement of modeling setup mimicking the applying context.

2) Encoding sequential pattern

The data for micro attribution is naturally in the form of collection of events/transactions:
User1:
    banner_id time1.1
    search_id time1.2
    search_id time1.3
    conversion time1.4
User2:
    banner_id time2.1
User3:
    search_id time3.1
    conversion time3.2

Some may think that this form of data makes predictive modeling infeasible. This is not the cases.  There are many predictive modeling are done with transaction/event type of data: fraud detection, survival model, to name a couple.  In fact, there are sophisticated practice in mining and modeling sequential patterns that are way beyond what being thought about in common attribution problem discussion. The simple message is:  this is an area that is well researched and practices and there have been great amount of knowledge and expertise related to this already.  

3) Separating model development from implementation processes

Again, the common sense from the predictive modeling world can shed some light on how our web analytics industry should approach attribution problem.  All WA vendors are trying to figure out this crucial question: how should we provide data/tool service to help clients solve their attribution problem. Should we provide data, should we provide attribution rules, or should provide flexible tools so that clients can specify their own attribution rules. 

The modeling perspective says that there is no generic conversion model that is right for all clients, very much like in Direct Marketing we all know there is no one right response model for all clients – even for clients in the same industry. Discover Card will have a different response model than American Express, partly because of the differences in their targeted population, their services, and partly because of the availably of data.  Web Analytics vendors should provide data sufficient for client to build their own conversion models, but not building ONE standard model for clients (of course, they can provide separate modeling services, which is a different story). Web Analytics vendors should also provide tools so that clients’ modeling can be specified/implemented once it’s been developed.  Given the parametric nature in conversion models, none of the tools from current major Web Analytics vendors seem sufficient for this task.

That is all for today. Please come back to read the next post: conversion model – not what you want but what you need.

February 27, 2009

it is about who get the job

If you ever wonder what all my recent attribution posts are about … it is about who get the job to handle the problem. Statisticians and modelers should be the ones — it is their jobs and I am speaking on their behalf.

If in fact the solution to attribution analytics is conversion modeling, then why it seems like everyone is talking about it like everything else but conversion modeling?  

Well, in my humble opinion, it is a sign of the lack of involvement, or lack of engagement, from statisticians, modelers and data miners. Today’s web analytics certainly got a lot of hammers and power tools; however, attribution may just be a different kind of problem.

Stay tune for the next post on the first 3 insights of the conversion modeling perspective

February 26, 2009

micro attribution analytics is conversion modeling

If you are surprised by the title statement, you are in the majority.  

This is actually a very strong statement and I did not make it lightly. It is saying that micro attribution is an area of data analytics that can be defined and studied with rigorous statistical methodologies. In short, it is more of a science than art or common sense.  Micro attribution problem is more like a response modeling or risk modeling problem than the problem of finding out a fair rule for distributing year-end bonus.  

Does this sound the same as how others describe attribution problem and solutions? 

It is certainly different from those who think the solution to attribution problem is about tracking.  Tracking is important because it provides you the data, but in itself they do not tell you what factors or customer experience have more or less influence on conversion.

It is also different from many who think about “last click”, “first click” etc. when they speak about attribution models.  Those are not data analytics models or statistical models that I was referring to.  One is about intuition-based smart rule; the other is about data-driven behavioral modeling.  The smart rule vs. modeling debate was long over in Direct Marketing, but it is just beginning in web analytics and online, right here in the micro attribution problem.   

It is also different from the many who think this is all about metrics (because of the claim that there is no right solution to attribution :).  It is not about averaging the attribution of first-click and last click. It is not about using engagement metrics as a proxy either. 

It is definitely not the same as those who think we need to wisdom-of-the-crowd type of solution.  The percentage of you who think early keywords should get 15% attribution for “assist” maybe right, but it has no bearing to me.  I do not believe that there is an average truth in any of these, for reasons that I do not believe one retailer’s offer-X response model shouldn’t be used for a loyalty campaign of a telecom company.

It is categorically different from those who hold that there is no right answer to the attribution problem. I agree that there is no perfect model that has no model prediction errors, but that is not a refutation for statistical modeling.  Statistics is founded on imprecision in data and never afraid of counter examples.

It is an approach of simplification, not of complication and certainly not a proposal to bring in psychology, media-logy or astrology into the picture.  In that regard, it could be a spoiler for the fun party we had so far.

Still, it is really just a claim at this point.  Please come back to read the next post: (TBD)

February 25, 2009

micro and macro attribution

What is attribution analytics? I was thinking of quoting a definition from Wikipedia but could not find it. So here’s my version of it:

Attribution analytics is a set of data analytics techniques used to determine the proportional contribution of marketing campaigns to conversions.  

There are two general types of attribution problem:  micro attribution focus on attribution of individual level conversion to marketing campaign touch points; macro attribution focus on attribution of aggregated level sale/conversion to marketing campaigns activities. Both types of attribution are trying to answer the question of how much credit each marketing campaigns should get; the data analytics behind them are in fact very different.

Marketing Mix Modeling (MMM) is one type of macro attribution analytics.  

The data used for macro attribution are commonly time-series data:  sale volume over time and marketing campaign spend data over the same time periods (may also include a few periods prior to that).  MMM is typically statistical/econometric model that use marketing spend data, and others, to predict the sale volume. The estimated relationship will then applied back to the data to figure out the proportional contribution of each marketing campaigns to total sale (or total incremental sale). MMM is a well developed analytics technique that has been around and practiced for decades. Its pros and cons are also well understood.

Macro attribution is mostly used for the purpose of evaluating the effectiveness of marketing/advertising channels, budget allocation and optimization.

Micro attribution on the other hand, uses individual level event data.  When individual is exposed to multiple marketing events (touch points), which events should get credit and how much? The explosive growth of individual level event data and cross channel matching methodology/technology is perhaps the major factor behind the rapidly growing interest in micro attribution. There has been a great deal of misunderstanding regarding to what the right analytics framework for this, and what are appropriate analytics tools for it. Some think that it is all about coming up with smart business rules and protocols while others believe that more rigorous statistical modeling is needed.

Micro attribution is mainly used for tactical campaign optimization.  However, when multiple channels/media events are involved (such as email, banner, ppc and seo etc..) its implication on macro level budget allocation is inevitable. Hence the organizational complications come with it as well. 

Individual event level data is clearly a better source for attribution – even if the interest is only at the aggregate level.  Multichannel marketers have been wrestling with attribution problem for a long time.  Many of the innovation are in the data capturing area, such as custom 1-800 numbers and custom URL etc.. Those are important measurement processes, designed to capture micro event level data to avoid having to do the less precision and less effective macro attribution.  However, in most cases only the last touch point is recorded – because it often happens at the same time as conversion event.  It is a situation that fit neither the micro nor the macro attribution problem above.  From the micro attribution perspective, because it has only one (last) touch point, it is simply a dummy LAST touch point attribution rule.  From the macro attribution perspective, because we know which touch point a sale/conversion was attributed, there is no need for any modeling at all.  Hence the popularity of the use of these types of attribution measurement devices.

The calling for using micro attribution is due to its optimization potential for business operation.  If attribution for the purpose of attribution, i.e. to satisfy political or procedural reporting need, then the last touch point attribution is a perfect solution.  However, it fails mainly because of its inability to capture and understand today’s consumer experience – that is multi-channel, multi-touch in nature.  One touch point analytics paradigm is outdated for understanding today’s consumers. The future is multi-touch, the future is micro attribution.  Hence it is extremely important to understand the right analytics framework for micro attribution.

Please come back to read the next post: micro attribution analytics is conversion modeling.

February 24, 2009

attribution problem is a data analytics problem

Is there anyone out there as frustrated as me with the many different terms and concepts around “attribution”?  For those who haven’t thought about this yet, here’s a sample of the terms related to the discussion:  

attribution management, attribution protocol, attribution problem, multiple touch point attribution, online marketing attribution, multiple attribution protocol, attribution modeling, marketing mix modeling, last-touch attribution, equal-attribution, impression attribution, attribution theory, online-offline attribution, attribution rules …

In this and a few follow up posts I will discuss a few topics that I hope will bring some clarity to this.  

Let me be upfront with my main point: attribution problem is a data analytics problem. I know that few people would argue with me on this, but I think few people have taken this seriously all its implications.  

Since it is fundamentally a data analytics problem, we should start with data.  What is the underline business question requiring an attribution analytical solution? What kinds of data we have, or we need to have, to answers attribution question?  How to translate the business questions into a data analytics questions that match the type of data we have.  What questions are not answerable given the limitation of data, or available analytics tools?  How rigorous is the proposed data analytics strategy:  a heuristic, a rule of thumb, a well-specified model? Are we over or under in our use of data?  Are we over design the analytics and making it more complex than necessary? What are all the limitations and disclaimers associated with an approach?

My sense is that we have not taken a serious look of the attribution problem from a data analytics perspective yet. We know the business problems, but most of us are not expert in data analytics methodology. 

Any comment?

Please come back to read my next post on micro and macro attribution.

February 23, 2009

attribution is the hottest topic these days …

Why?  there are a few signs:

If web analytics is your profession or your passion, considering these:

    Avinash Kaushik has several posts in recent month:  Measuring Value of “Upper Funnel” Keywords,   Measure Latent Conversions & Visitor Behavior  and this one from almost a year ago: Conversion / ROI Attribution.

    Eric Peterson  has been busy with webinars and presentations, all over attribution problem.  He use a new metric as a foundation for attribution analysis: “Appropriate Attribution Ratio”.  

We all know that all the major web analytics vendors are working hard trying to figure out the right attribution modeling tool to offer.  My recent meeting with a major web analytics vendor also convinced me; it is all about attribution data service and attribution modeling.

Attribution is also the most talked about topic in SEM/SEO today. I can’t think of a search conference does not have sessions focus on attribution; and every SEM tool makers are tooting its solution for atrribution management

From ad-servers: Atlas’s Engagement Mapping and DoubleClick’s Exposure to Convertion Report.

Another variation of name is Multiple Attribution Protocol.

Forrester Research, who lead research in perhaps everything online marketing, has three researches on attribution within the last six months: Attribution by Emily Riley, Search and Attribution by Evan Andrews and Multicampaign attribution by John Lovett et al.  

It is a hot topic and an area not short of differing approaches.  Anyone who is seriously interested should also read Barry Parshall’s fighting post on the right and wrong approaches for attribution modeling.      


Create a free website or blog at WordPress.com.