Case Study

The Challenge

With rapid advancements in a broad suite of analytics tools and data sets available to marketers, brand thinking has evolved. It’s no longer:

“Should we utilize an analytics solution?” but instead:

“How do we proactively manage the insights and scope across potentially overlapping or conflicting analyses?”

Since 2012, USAA—the member-based, financial products and services company serving military personnel and their families—has leveraged Neustar’s proprietary Media Mix Modeling (MMM) solution in order to better evaluate the effectiveness of their enterprise and product-level marketing investments. USAA also heavily leverages extensive A/B testing to evaluate media impact on acquisition cost of product sales and other outcomes across the sales funnel.

MMM works differently than A/B testing. MMM econometric regression models incorporate 3+ years of weekly time series data—encompassing media channels, competitive activities, economic factors, and more—into one holistic analyses that measures the impact of marketing activities on sales and other business outcomes. A/B testing, in general, compares a treatment applied by a brand (be it price, promotion, paid media, etc.) against a control group that does not receive that change in treatment. Those groups can be either customer segments, geographies or markets, etc. In USAA’s case, A/B testing often measures a single media channel’s impact on outcomes over a specific time period.

With both analytics solutions heavily in use, amongst others, the USAA analytics group wanted a framework ensuring that business and marketing stakeholders received one harmonized and consistent source for insights about media efficiency and recommended allocation. Different answers from different solutions would impede credibility and create confusion and doubt for both MMM and A/B testing.

To this end, USAA developed an approach it calls Informed Attribution to triangulate across solutions. With the foundational objectives for this program clear, Neustar and USAA co-created a specific process to determine if, how, and when MMM and A/B test results should impact each other. The end benefit would be more confidence from business and marketing teams in results, and a clear roadmap for model governance and comparisons.

The Neustar Solution: A 3-Part Plan for Success

1. Understanding the relative strengths and limitations of MMM and A/B Testing

The Neustar team knew that the key to optimizing USAA’s measurement approach was to first ensure that the strengths and potential weaknesses of MMM and A/B testing were understood.

MMM measures all media types on an even playing field. It controls for non-marketing factors and provides an estimation of impact across a several years, developed using consumer impressions to mimic the nature in which individuals consume media. It incorporates diminishing returns, includes synergistic effects of media, and captures ad stocks (time delayed impact of marketing). This is all built into Neustar MarketShare software to enable what-if simulations and optimizations.

MMM is well-suited for the following use cases:

  • Estimation of total media contribution and ROI
  • Estimation of the relative contribution, CPA, and ROI across media sub-types
  • Right-sizing the media budget, based on insight into incrementality
  • Allocation of portfolio media mix across product and media type
  • Stress-testing of marketing plans utilizing multiple, simultaneous changes
  • Forecasting future business performance

A/B testing, conversely, provides a more focused view. It evaluates just one or two media types at a time, and targets at the geographic or customer-group level.

A/B testing would be best suited for the following use cases:

  • New media sub-types with less than one year of investment history
  • Individual DMA/geography targeting strategies
  • Individual customer-type targeting and segmentation strategies
  • Individual marketing campaigns
  • Tactical level measurement
2. Framework for One Tool Impacting the Other

Neustar’s MMM development process utilizes a framework in which it is possible to ingest exogenous information or sensitivities for select market factors, when that information is of high confidence. This approach can allow for the inclusion of A/B test results in model development, enhancing the broader and more holistic insights from MMM with the targeted, specific insights from A/B testing

USAA and Neustar agreed A/B tests should only selectively be incorporated into MMM in instances where all of the following conditions are true – and after a qualitative discussion of quantitative results between measurement teams:

  • Limited MMM data quality exists for a given media channel or non-media factor
    • This could be in the form of limited history, limited or excessive volatility over time, limited variability across cross section (market, product, etc.)
  • A/B tests show repeatable outcomes that materially differ from MMM
  • A/B test results have high confidence – both from a statistical point of view and qualitative business sense assessment
  • A/B test incrementality-result magnitude is within reasonable ranges for the industry vertical and/or prior measurement experiences
  • All other potentially relevant factors are appropriately controlled for when comparing model results (e.g., time periods, products, geographies, magnitude of investment & source, campaign message, etc.)

Neustar also recommended, and USAA agreed, that MMM results should be used to improve the effectiveness and focus of future A/B testing. MMM’s holistic view of media measurement could be utilized to identify more granular opportunities where A/B tests could fill existing measurement gaps (e.g., emerging media sub-channels). MMM also provides a predictive simulation environment which could be leveraged to establish test parameters and media response expectations.

By aligning MMM and A/B testing, the expectation was that USAA would have a more consistent read on media performance that could be socialized throughout the organization and improve decisions regarding budget allocation.

3. Setting Up a Governance Plan

Once USAA understood the relative advantages of both types of analyses and the best use cases for incorporating A/B testing into MMM, the next step was to develop a governance plan that would allow for consistent alignment on media measurement.

Based on best practices with other clients, Neustar recommended a quarterly review of the latest A/B test findings to coincide with MMM data review meetings covering the most recent historical periods. Importantly, this dialogue takes place before MMM modeling commences, to ensure the right context is applied during that subsequent process.

The teams found the most beneficial setup was a qualitative and quantitative Q&A session with the teams knowledgeable about A/B test results and MMM models. Reporting documentation is helpful and necessary, but insufficient for alignment. Based on those dialogues, with the framework described, the teams jointly decide if and where each model’s results may want to inform the other. Systemizing this coordination ensures regular adoption of best practices described in steps one and two. After several rounds, the teams found they were able to align on areas of opportunity within just one, one-hour session.

The Outcome

The goal of the USAA and Neustar engagement has been, and continues to be, to maintain a big-picture measurement plan that validates and informs overall media performance and planning. The close coordination required to effectively implement a consistent process for incorporating A/B test results into MMM has allowed that goal to be more fully realized.

By leveraging and combining the strengths of both analytic techniques, MMM’s measurement of media effectiveness is enhanced and USAA’s confidence in the results has increased. This, in turn, ensures that MMM remains an effective and robust solution to inform and enable decision-making throughout the organization.

Roadmap for the Future

This approach also lends itself to incorporating multi-touch attribution (MTA) results in the future, as a third leg of the measurement framework. Neustar has similar capabilities to not only ingest A/B test results into MTA, but also to integrate a unified MMM and MTA solution and harmonized insights across the organization. With continued attention to planning and governance, this additional analytic tool will make USAA’s measurement program even more robust and actionable.

View PDF