The Conundrum of Changes in Market Research and Predictive Analytics

As a market research practitioner and a technology industry analyst covering big data and business analytics, I enjoyed listening to other analysts discuss the market research industry in a webinar.  My own research augments and sometimes contrasts with that of the webinar participants.

In the webinar Simon Chadwick spoke about data mining and the need to focus on the analytic skills gap. I’d maintain that businesses need to focus on traditional hypothesis-driven statistics in addition to data mining, especially when we start talking about predictive analytics. While data mining is directed from the data itself, statistics are more hypothesis-driven. If you’re familiar with SPSS tools that I recently assessed, you can see the difference by comparing SPSS Statistics and SPSS Modeler. Market research practitioners have traditionally been trained in latter, but not in the former. I go into more detail in a joint webinar I did with IBM on predictive analytics.

Folks who know data mining are in limited supply. They often come from the data warehouse or business intelligence worlds, which have not traditionally churned out deep analytical expertise. That is often the domain of database administrators or folks who have a good understanding of structured query language (SQL). Some technology companies are trying to fill the skills gap related to big data (read unstructured data) by taking advantage of SQL skillsets. Teradata is moving in this direction with its Aster Data acquisition, and Karmasphere with its toolset, but SQL is a declarative language, and while it fills some gaps in organizations’ ability to access big data, it has limitations.

To bring the point home of how important advanced analytic skillsets are to an organization, our benchmark research shows that companies are more satisfied (70% versus 59%) when their predictive analytics initiatives are led by analytics professionals than by the data warehouse team. (As an aside, our benchmark research into predictive analytics shows it to be a key area where marketing and sales are focusing their efforts right now, with social media analytics, attrition, response and attribution modeling as key components of the strategy.)

Since neither the technology industry nor the market research industry has trained analytics professionals especially well, we have a big shortage of folks who can lead big analytics initiatives. This skills gap is driving funding to companies such as Mu Sigma, Absolute Data and GoodData. Not surprisingly, these companies target the marketing and market research client-side professional. It’s not a big leap for analytics professionals in the market research industry who already know hypothesis-driven statistics to move into data mining and data modeling, but these skills generally exist on the market research side, not inside client organizations themselves. For this reason, I’d argue that companies that are really savvy in advanced analytics, such as Probit Research and Definitive Insights (now YouGov), are in a great position to help companies drive their analytics strategies.

Technologists and market research practitioners have long lived in parallel universes.  In technology, we deal with tables, joins and the ETL process. In market research and analysis, we deal with datasets, merges and data preparation. When you think about it, these are the same things! The subtle difference is that technologists have had a data mining mindset and market researchers have had a hypothesis-driven mindset.

As analytics and data environments come together, market researchers need to get more into data mining and more comfortable with data modeling. At the same time, technologists need to get more into hypothesis-driven analytics. In the webinar, Chadwick mentioned the massive advances in technologies, which are apparent when we look at trends in in-database analytics and embedded analytics. We’re seeing a rise in the availability of complex algorithms and open source languages such as R; nevertheless the three most used algorithms in enterprises today are still the simpler ones –  logistic regression, linear regression and decision trees. These should sound very familiar to those of you in the market research industry.

With respect to digital analytics, a new class of tools beyond cookie-driven web analytics is pulling from machine data, which is exposed in a variety of ways. The net impact is that rather than just looking at numbers of hits or click-through rates or transactions, we're beginning to be able to see into the customer journey – sort of like doing shop-along in retail, but in a digital space. This gives us great insight into the purchase funnel and competitive dynamics, the likes of which we just didn't see before. As we marry this technology with analysis of offline behavior, things get even more interesting, but also more complex. We start to deal with privacy issues unless we invoke faceless types of analysis, but that limits us in our ability to market at an individual level. At the same time, attribution modeling becomes more complex given the increases in the number of both promotional and transaction channels.

Finally, my one disagreement with the webinar speakers is their assertion that there's not enough investment in education going into schools. On the contrary, every major vendor in the technology space talks about the schools they are aligned with, and most of these folks talk about investing in these schools with respect to analytics training. I agree with webinar participant Lenny Murphy that academia is often slow to change; I often compare market research with academia, in fact. But schools are getting more private funding as the government pulls back its spending, and for this reason schools will become more responsive to the needs of private enterprise. It’s here where the skills gap will begin to be filled as schools move away from classical education underpinnings to be more aligned with the needs of the 21^st century enterprise.

Regards,

Tony Cosentino

VP & Research Director