Analytics Strategist

September 17, 2012

The state of the attribution business (in marketing/advertising)

Filed under: misc — Tags: , , — Huayin Wang @ 8:01 pm

I had some chats with friends and colleagues lately over the topic of attribution model; the shared feeling is that there is a glaring contrast between the growing number of vendors and the utterly lack of clarity and consensus on approaches. To put this more concretely:

We all know that last click was wrong, but do not know what to do with it.

We do not know what’s the right approach and why (others are not) – is it algorithm attribution? Or is it experimental design?

Why does the right or wrong attribution model matter and to whom? What does it have to do with optimization, and how?

Very confused about Marketing Mix Modeling vs Attribution – is one more right than the other – is one enough?

And finally: who should build attribution services and who should own it?

If you think the current state of attribution is clearer than the above picture, please make you voice heard.  If you know the answer for any of the above questions, even better! You are more than welcome to share your thought and enlighten us, right here!

Thanks!

 

 

June 6, 2012

The professed love of data science

Filed under: Business — Tags: , — Huayin Wang @ 8:01 pm

It seems everyone fall in love with Big Data, Data Science and Data Scientists lately;  not a lot good stories outside the few poster-child start-ups.  It reminds me of an ancient Chinese story and a well known idiom:  Professed Love of What One Really Fears

In the spring and autumn period (770-476bc), there lived in Chu a person named Ye Zhuliang, who addressed himself as “lord Ye”.

It’s said that this lord ye was very fond of dragons – the walls had dragons painted on them, the beams, pillars, doors and the windows were all carved with them. As a result, his love for dragons was spread out.

When the real dragon in heaven heard of lord Ye, he was deeply moved. He decided to visit lord Ye to thank him. You might think lord Ye was very happy to see a real dragon. but, actually, at very the sight of the creature, he was scared out of his wits and ran away as fast as he could. From then on, people knew that lord Ye only loved pictures or carvings which look like dragons, not the real thing.

May 24, 2012

The Principles of Attribution Model

Filed under: attribution analytics — Tags: , , — Huayin Wang @ 7:36 pm

(Disclaimer:  some questions and answers below are totally made up,  any resemblance to anything anyone said is purely coincidental)

How do we know an attribution model, such as Last Click Attribution, is wrong?

  • it is incorrect surprise surprise, a lot of people just make the claim and be done with it
  • it does not accurately capture the real influence a campaign has on purchase – but how do you know it?
  • it only credit the closer – isn’t this just a re-statement of what it is?
  • it is unfair to upper funnels and only awards to lower funnel – are you suggesting that it should award to all funnels, why?
  • it leads to budget mis-allocation so your campaign is not optimized – how do you know?
  • it is so obvious, I just know it – what?

How do we know an attribution model, such as a equal attribution model, is right?

  • it is better than LCA – intuition?
  • it gives out different credits than LCA so you can see how much mis-allocation LCA does to you campaign – different from LCA is not automatically right
  • we tested and it generate better success metrics for the campaign – sound good, how?
  • it is fair – what does that mean?

How do we find the right attribution model?

  • try different attribution models and test the outcome – attribution model does not general outcome to campaigns directly
  • play with different models and see which one fit your situation better – how do I know the fitness?
  • use statistical modeling methodology to measure influence objectively – what models? conversion models?
  • use predictive model for conversion – why predictive models? what models? how to calculate influence and credit from the models?
  • test and control experiment – how many test and control, what formula to use to calculate credit?
  • you decide, we allow you to choose and try whatever attribution weights you want – but I want to know what’s the right one?
  • the predictive models help you with optimization, once we get that, you do not care about attribution – but I do care …
  • shh … it is proprietary: I won’t tell you or I will kill you! – ?

The Principle of Influence

Three principles are often implicitly used:  the “influence principle”,  the “fairness principle” and the “optimization principle”.

The influence principle works like this: assume we can measure each campaign’s influence on a conversion, the correct attribution model will give credit to campaigns proportional to their influence.  The second principle is often worded with “fairness”, but very much the same as the first principle:  if multiple campaigns contribute to a conversion, giving 100% credit to only one of them if “unfair” to others.  The third principle, the optimization principle, in my understanding, is more about the application of attribution (or the benefit of it) and not about the principle of attribution.

The principle of influence is the anchor of three; the fairness and optimization principles are either a softer version or a derivative of it.

Now we have our principle, are we close to figuring out the right approach to attribution model?  We need to get closer to the assumption of this principle.  Can we objectively measure (quantify)  influence?  Are there multiple solutions or just one right way to do this?

If influence principle is the only justification of attribution models, then quantitative measurement methodology such as probabilistic modeling, some time it is called algorithmic solution which I think is a misnomer,  will be the center technology to use.  It leave no room for arguing just on the ground of intuition alone.  Those who offer only intuition and experience, plus tools for clients to play with whatever attribution weights are not attribution solution provider, but merely a vendor of flexible reporting.

Those of the intuition and experience school like to frame attribution model around the order and position of touch points:  the first/last/even and the introducer/assist/closer. (how many vendors are doing this today?)  They have troubles in providing quantitative probabilistic solution to attribution issue.  The little known fact is that it is analytically flawed:  the labeling of “last touch” and “closer” are only known post-conversion, and therefore not usable inside probabilistic modeling framework.  In predictive modeling and data mining lingo, this is known as the “leakage problem”.  (search on Google, or read Xuhui’s article that mentioned this).

Unfortunately, we have a problem with the data scientist camp as well but of different nature; it is the lack of transparency with metrics, models and process details.  Some vendors are unwilling to open up their “secret sauce”.  Perhaps, but is that all?  I will try to demystify and discuss the “secret sauce” of attribution modeling.


May 17, 2012

Attribution Model vs Attribution Modeling

Attribution is a difficult topic, growing into a mess of tangled threads.

I hope this post, and subsequent ones, will help to untangle the messy threads.  I like to start with simple stuff, be meticulous with the use of words and concepts and be patient; after all, haste makes waste.

When an advertiser records a conversion or purchase, some times there are multiple campaigns in the touch point history of the conversion; how do we decide what campaign(s) responsible for the conversion and how should the conversion be credited to each of these campaigns?  This is the attribution problem; a practical issue first raised in digital advertising but in itself a general analytical challenges.  It is applicable to many marketing/advertising contexts, cross channels or within a particular channel for example.

Micro vs Macro

Notice that Attribution is a “micro” level problem: it dealt with each individual conversion event.  In contrast, Marketing Mix Model (or Media Mix Model) deals with “macro” level problem: crediting conversion volume to each channel/campaigns in aggregate.  There are similarity between the two when viewed from the business side; they are quite different analytic problems, different in all major aspects of the analytic process: from data to methodology to application.

Attribution Model vs Attribution Modeling

Advertisers implement business rule(s) to handle this “attribution”, or credit distribution, process.  These rules are generally called “attribution rule” or “attribution model”; examples of it are Last Click Model, First Click Model, Fractional Attribution Model etc..  Rules and models are interchangeable in this regard, they serve as instruction set for the execution of attribution process.

There are no shortage of attribution rules or models being discussed. Anyone can come up with a new one, as long as it does partitioning credit .  The challenge is finding the right one, to choose from too many of them.  In other words, it is the lack of justification for the approach, process and methodology of Attribution Rules/Models that is a problem.

Now comes the Attribution Modeling – a statistical model-based approach to quantify the impact of each campaigns on each individual’s conversion behavior.  It is a data-driven algorithmic approach; it is hot and cool, with an aura of objectivity around.  It is often treated as the secret sauce to unlock attribution and optimization and covered with a proprietary blackbox.

Let me slow down a bit.  I have discussed two important concepts here: Attribution Model and Attribution Modeling. The former refers to the attribution rules; the later refers to the process of generating/justifying the rules. I understand that everyone do not agree with my use of the words, or the distinction between the two; but I think this is a critical distinction, for untangling the threads in attribution discussion.

Domain Expert vs Data Scientist

There are generally two camps when it comes to the generation/justification of attribution model.  The first is the “domain experts” and the second the “data scientists”.  Domain experts take issues with attribution models by pointing out the good and bad, arguing on behalf of common sense, experience and intuition; it is qualitative, insightful and at times interesting but generally pessimistic and fall short when it comes to build a rigorous data-driven solutions. The general principle for justifying attribution is one of two: the influence and the fairness.  The influence principle attributes credit based on the influence of the campaign on conversion, whereas the fairness is often stated generally.

The fairness principle is not a concern for the data scientists camp; in fact, it is all about modeling/quantifying the impact or influence. After all, if you can do the attribution based on precise measurements of influences of each touch points, what other principle do you need? Of course, the problem is often about the promise and less about the principle.  In contrast to the domain experts, the data scientists approach is quantitative, rigorous and data driven. You can argue with the choice of a specific modeling methodology, but the resulting model itself does not require common sense or past experience to justify.

Principle of Attribution:  Influence, Fairness, Optimality

A third principle in picking the right attribution model is optimality, for lack of better word.  Do right attribution models lead to optimal campaign management?  Some argue yes.  Does the reverse statement true? Can optimality be a principle in choosing or justifying attribution model?  These are some of things I will discuss and debate about in my next writeup.

Thanks for reading!

April 23, 2012

Attribution Model and Attribution Modeling do not mean the same thing

Filed under: misc — Tags: , — Huayin Wang @ 9:27 pm

With great frustration (to myself and many others who speak about attribution model before), I am making a plea here:  please make it clear what do you mean when you write or speak about attribution model!  For those who do not have the patient to think over, pick one from the two most common uses of it:

A:  Attribution Model as a reference to the process or rules about crediting marketing/advertising success to individual campaigns. Names for such commonly used credit allocation rules:  Last-Click, First Click and even distribution etc.

B:  Alternatively, people use attribution model to means the statistical modeling methodology and/or processes in producing the credit allocation rules above – this could be all kinds of control/experiment testing, regression modeling, bayesian statistical modeling etc..  There are arguments about whether the right model has to be causal model, explanatory model or predictive models.

A or B, which one are you? In other words, which one do you mean when you utter “attribution model”?

I am A; and I use “attribution modeling” for B.  This is the best I can do, after quite sometime struggling with it.

I believe this is a serious matter.  To quote Confucius: “If language is not correct, then what is said is not what is meant; if what is said is not what is meant, then what must be done remains undone.”

April 11, 2012

Funny analogies of wrong attribution models

Few topics are near and dear to my heart as Attribution Modeling is.  I first bumped into it a more than 4 years ago; and my first written piece on attribution is a linkedin Q&A piece answering a question from Kevin Lee on duplication-rate  (in August 2007).  Since then, my interest in attribution gets real serious, resulting in a dozen’s attribution related blog posts.  The interest never died after that, although I have not written anything the last three years.

I am back on it with a vengeance! Consider this as my first one back.

I want to start on a gentle note though.  I am amused about people still debating about First Touch vs Last Touch attribution as viable attribution models, a bit out of the question in my opinion.  I want to share some funny analogies for what could go wrong with them.

Starting with Last Touch Attribution Model, a football analogy goes like this: “relying solely on a last click attribution model may lead a manager to sack his midfielder for not scoring any goals. Despite creating countless opportunities he gets no credit as his name isn’t on the score-sheet. Similarly a first click attribution model may lead the manager to drop his striker for not creating any goals, despite finishing them. – BrightonSEO presentation slides

There are a lot of good analogies like this that are derived from team sports.  This analogy is applicable not only to Last Touch, but to all single touch point attribution models.  The funniest one I heard is about First Touch Attribution, from none other than the prolific Avinash Kaushik: “first click attribution is like giving his first girlfriend credit for his current marriage.” – Avinash quote

Analogy is analogy, it does not do full justice to what’s been discussed.  However, what we should learn at least this much: if your attribution model is solely based on the sequencing order of touch points, you are wrong.  Those who propose Last, First, Even, Linear or whatever attribution models, watch out!

A good attribution model needs a disciplined development process, and better yet, a data-driven one.  The less the assumptions made about the values of touch points the better – we should learn to let empirical evidence speak for itself.

Do you have any interesting analogy, or thought?

November 21, 2011

Ad exchange as matchmaker

Filed under: misc — Tags: , , , , — Huayin Wang @ 9:52 pm

Looking at ad exchange from the matching game perspective can be interesting.  My early post tried to draw some insights from game theory into ad exchange design and practice.

Ad exchange is a platform of matching advertiser’s ads to publisher’s slots for each audience-impressions.  The massive scale of its operation, the technological challenge and sophistication are unprecedented; and yet the basic matching mechanism is under-designed in comparison to traditional matchmaking. There are so many things that publisher care about which ad will be shown on their sites, not just what’s is the highest bid price; yet, the within current ad exchange design, there is no opportunity for publishers to express their preferences fully.

If ad exchange were a matchmaker, what it would be like?  Imaging publisher as bride and advertisers as potential grooms – our ad-exchange-matchmaker goes to a bride and say,

M:  Let’s put it out there and see who will be the richest man coming here for you

B:  But I am scared of marrying one I do not know”

M:  Well, you can give me a black list and tell me what you do or do not like as filters

B:  Can I pick it myself?

M:  No, I will pick one for you –  based on wealth

Now you get the idea of the frustrations of our publisher/bride.  There need to be a process for publishers to perform data-driven ads evaluation based on their objective and value, not just bid price.  About publisher concerns of ad exchange and RTB,  Brian O’Kelley discussed here on ClickZ.  A better designed matching mechanism should handle publisher’s preference properly, which may ease some of the publishers’ concerns (channel conflict, data leak and brand safety) if not solving them.

November 16, 2011

Ad exchange, matching game and mechanism design

Over the years, I have learned some interesting things in this new ad:tech industry, particularly around the Ad Exchange and RTB ad auction market model.  I want to share some of my thoughts here and hope you find them interesting to read.

Ad Exchange is not like a financial exchange

The “exchange” in the name is suggestive of a financial stock exchange market, and interesting observations can be made based on this analogy.  However, there are some fundamental differences such as the lack of liquidity in ad impression and information asymmetry.  Jerry Neumann has blogged about this topic profusely;  it is still a topic of great interest today, as seen in a recent article by Edward Montes.

In fact, the differences are easy to understand. The harder part is, like Jerry asked,  If the ad exchange aren’t exchanges, what are they?  or I should add, what should they be like?

Publisher’s preference is the missing piece

The analogy with financial exchange (stock and future) is not a good analogy partly because of its inability to fully model advertiser preference. Not all impressions are of the same value to an advertiser and not all advertisers give the same value to an impression. The commodity auction model as embedded in ad exchange does better, because it allows advertisers to bid based on any form of evaluation – a chance for advertiser to fully express its preference over audience, contextual content and publishers’ brand.

Still, there is a problem for the current auction model: after collecting all the bids from advertisers, it takes the highest bidder to be the winner, as if price is the only thing publishers care about.  In reality, not all bids with the same price are of the same value to a publisher.  Publishers care about brand safety and contextual relevancy as well; in fact, the quality of user experience may mean more to publishers than advertisers!  In sum, publishers care about the quality of the ads above and beyond the bid price.  Unfortunately, the current ad exchanges lack the proper mechanism allowing publishers to articulate their full preferences.  This results in lost of market efficiency and lost of opportunities to remove transaction frictions.  This is a design flaw.

Display marketplace is still far from perfectly efficient and this design flaw does not help.  The recent developments of Private Marketplace are piecemeal attempts to overcome this design issue.  Some market movements in late merge and acquisition attempts can be understood from this angle.

Where can we look for design idea on how to handle this issue?  – paid search and game theory!

The quality score framework from paid search

In many ways, paid search is just like ad exchange, with Google plays one of a few “publisher” roles.   In both markets, advertisers are competing for ad-view through auction bidding;  if we equate audience in display to keywords in search, then the bidding processes is quite the same:  search advertisers do extensive keyword research, look at past performance along side of other planning parameters such as time of the day etc. to optimize their bids;  similarly display advertisers look at the audience attributes, the site and page contents, past performance and planning parameters as they perform bid optimization.

The bidding processes in both markets are similar;  the differences lie in the post-bidding ad evaluation.

After all bids are collected, ad exchange today simply select the highest bidder.  In case of paid search, bids are evaluated on price, ad relevancy and many other attributes.  Google has mastered the evaluation process with its Quality Score framework.  This difference in having a Quality Score framework vs not is not a small thing.  As anyone familiar with the history of paid search know, the quality score framework played a pivotal role in shaping the search industry when Google introduced it around the turn of the century.  The post-bidding ad evaluation for display may just be a critical piece of technology and have potentially significant impact on the efficiency and the health of the display market.

The need for a non-trivial post-bidding ad evaluation calls for an extra decision process (and algorithm) to be added, either at ad exchange or at publisher’s site, or both.  In this new model and with this extra component, ad exchange will send the full list of bidding to the publisher instead of picking a winner based on price alone.  It is then up to the publisher to decide which ad will be shown.  With millions of publishers, large and small, this seemingly small change may be a trigger for more inside this industry where technology is already orders of magnitude more complex than paid search.

The matching game analogy

With full preferences being taking into account for both advertisers and publishers, ad exchange looks less like a commodity marketplace and more like the matching game.  It will be interesting to look at market efficiency from the perspective of mechanism design in game theory, which is another way of saying operational market process.

Matching advertisers with publishers under a continuous series of Vickrey auctions is our setup for the discussion – the best model I can think of that mimic the matching game setup;  it shouldn’t be too surprising to anyone that matching game is an interesting analogy to ad exchange.  As a game theory abstraction of many practical cases, matching game includes college admission and marriage market.  Let’s take the marriage market as an example.

Using a simplistic description, a marriage market involves a set of Men and a set of Women.  Each man has a preference vector over the set of women (a ranking of women);  similarly each woman has a preference vector over the set of men (a ranking of men).  A matching is an assignment of men to women such that each man is assigned to at most one woman and vice versa.  A matching is unstable if there exist a man-woman pair not currently matched to each other but both prefer match to each other than their current match – the pair as such is called a blocking pair.  When there is no blocking pair exist, a match will be called a stable match.

Clearly, stability of a match is a good quality: a stable match is not vulnerable to any voluntary pairwise rematch (translating into ad exchange language, a stable match is one such that no pair of advertiser – publisher currently not matched to each other have incentive to switch and form a new match).  A matching is male-optimal if no two males have incentive to switch partners. Female-Optimal is defined similarly.  A stable matching that is both male-optimal and female optimal looks like an perfect efficient market; we hope to find a mechanism that lead to the unique stable matching as such – something we can then mimic to implement for a future ad exchange model.

Unfortunately, there is no unique stable matching for matching game in general (in this case, having too many good things may not be a good thing).  There is also no unique optimal matching that is optimal from both men and women’s perspective.  We learned that Male Proposals Deferred Acceptance Algorithm, sort of like the current auction process in ad exchange in which advertisers played the male roles, produce Male-Optimal stable matching.  If we switch the role of men and women, a similar algorithm exists that produces Female-Optimal matching.  The two algorithms/mechanisms lead to two distinctly different results.  You can read more about algorithmic game theorycomputational game theory, specifically on matching game and mechanism design if interested.

So, why are we looking into this and what we’ve learned from it?  Below is my translation, or transliteration to be more appropriate, from game theory speak to the ad:tech domain.

We all like to believe that there is an efficient market design for everything, including the exchange marketplace for ads. Our believe is justified for all commodity marketplace by the general equilibrium theory.  Unfortunately, there is no equivalence of a “general equilibrium” or universal optimal stable match for a marriage market, which implies that there is no universal optimal advertiser-publisher matching in ad exchange.  If this is the case, the search for an optimal market mechanism for ad exchange will be a mission impossible.

However, there exist one-sided optimal condition, advertiser-optimal and/or publisher-optimal matching.  It is also easy to find the corresponding mechanisms that lead to those one-sided optimal stable matching.  The auction market as currently implemented in ad exchanges, with the addition of post-bidding evaluation process, is similar to the mechanism leading to advertiser-optimal matching.

The future seems open for all kinds of good mechanism design.  Still, I believe that there is a “naturalness” in the current style auction market.  It is quite natural for the auction process to start from the publisher side, by putting the ad impression on auction, because it is all start with the audience requesting a webpage – a request send to a publisher. It is not easy to imagine how advertiser can set up a “reverse auction” starting from the demand side, within a RTB context. We can never rule out the possibility, and it may work potentially for trading the ad future.

Conclusion:

I am reluctant to draw any conclusions – these are all food for thought and discussion.  I’d love to hear your comments!

August 26, 2011

What’s wrong with BS-ing?

Filed under: misc — Huayin Wang @ 2:27 pm

Professionals who are BS-ing are lemon professionals.  They are fake or counterfeit of real professionals.

What’s wrong with fake product or any counterfeit?  Read on the market for lemon.

Is BS-ing still “not good, but ok” because a lot of people do it (even worse, relying on it for a living)?

I hope we all get serious about BS-ing, it is not a small thing.

January 2, 2010

a decade in data analytics …

Filed under: misc, Web Analytics — Tags: , , , — Huayin Wang @ 10:53 pm

I was reading an article The Decade of Data: Seven Trends to Watch in 2010 this morning and found it a fitting retrospective and perspective piece.  I have been working in data analytics for the past 15 years, so naturally I went searching for similar articles with more of a focus on analytics, but came back empty handed 😦

I wish I could write a similar post, but feel the task is too big to take.  A systematic review with vision into the future would require much more dedication and effort than I could afford at this point.  However, I do have a couple of thoughts and went ahead to gather some evidence to share.  I’d love to hear your thoughts; please comment and provide your perspectives.

The above chart shows search volume indices for several data analytics related keywords over the last six years.  There are many interesting patterns.  The one caught my eyes first is the birth of Google Analytics: Nov 14, 2005.  No only did it cause a huge spike in the search trend for “analytics”, the first day “analytics” surpass “regression”, it become the driving force behind the growth of web analytics and analytics discipline in general.  Today, more than half of all “analytics” searches are associated with “Google Analytics”.  Anyone who writes the history of data analytics will have to study the impact of GA seriously.

I wish I could do a chart on the impact of SAS and SPSS on data analytics in a similar fashion, but unfortunately it is hard to isolate SAS searches for statistics software vs other “SAS” searches.  When limited to the “software” category, it seems that SAS has about twice the volume of SPSS, so I used SPSS instead.

Many years ago, before Google Analytics and the “web analyst” generation, statistical analysis and modeling dominated the business applications of data analytics.  Statistician and their predictive modeling practice were sitting in their ivy tower.  Since the early years of the 21st century, data mining and machine learning became a strong competing discipline to statistics – I remember the many heated debates between statistician and computer scientists about statistical modeling vs data mining.  New jargons came about, such as decision tree, neural network, association rule and sequence mining.  To whomever had the newest, smartest, most math grade, efficient and powerful algorithm went the spoils.

Google Analytics changed everything.  Along with data democratization came the democratization of data intelligence. Who would’ve guessed that today, for a large crowd of (web) analysts, analytics would become near-synonymous with Google Analytics and building dashboard, tracking and reporting the right metrics the holy grail of analytics?  Those statisticians may still inhabit the ivy tower of data analytics, but the world is already owned by others – the people – as democracy would dictate.

No question about it, data analytics is trending up and flourishing as never before.

comments?  Please share your thought here.

« Newer PostsOlder Posts »

Blog at WordPress.com.