Closing the Analytics Skills Gap

A study by the McKinsey Global Institute published earlier this year suggests a coming shortage of more than 140,000 workers with deep analytical skills and a shortage of more than 1.5 million data-literate managers. I’m not sure how the study defined these roles, but I’d guess that those with deep analytic skills are those folks building the complex models, and the data-literate managers are those executives, middle managers and analysts who interpret the results and use the models to help drive business decisions. In other words, businesses are facing two skills gaps – one related to those producing the analytics, the other related to those using them in some type of discovery or review purpose.

The first skills gap is personified in the so-called data scientist – a creature that’s hard to find in the real world and according to LinkedIn, there are less than 825 in the United States and most are in Silicon Valley working for technology or social media companies. Someone with the job title of data scientist should be able to bridge the divide between  computer science, statistics and particular domain expertise and deliver an integrated systems approach to analytics. To give some context, the primary obstacle to the deployment and use of predictive analytics we identified in our predictive analytics benchmark research is architectural integration. The statistician who builds models rarely has the skill set to code them, which often leads to misalignment between the intent of the model designer and what the model actually does once it is deployed and used within business processes. Even worse is that 83 percent of organizations do not have the skills training to produce their own predictive analytics.

As more BI vendors embed predictive analytics support in their portfolios, and with further adoption of the PMML standard, this architectural shortcoming should become less of an issue. But we still have the challenge of understanding the mathematics which is a challenge in 58 percent of organizations. In the world of big data and Hadoop, multiple companies are working to provide analytics platforms that leverage existing skill sets such as SQL but new skills for supporting technologies related to Hadoop are a larger challenge. As the next generation of tools emerge, the bridge will likely start to form over this first skills gap where automation and integration of technology is more readily available.

The second skills gap relates to those using the models. With this group, we might expect a certain measure of understanding of data manipulation techniques, such as cross-tabulations, what-if scenarios and chart interpretation, but we shouldn’t expect a formal background in advanced statistics or probability theory. Our recent benchmark research into business analytics also points to a skills gap in this area. One interesting finding in our benchmark research, but perhaps not a surprise to anyone who reads our posts regularly, is that spreadsheets still dominate in conducting analytics today. Unfortunately, spreadsheets are not well equipped by themselves to handle the analytics of tomorrow’s organizations, which means managers will need to learn an entirely new set of tools. While the new breed of analytics tools are visual and collaborative rather than tabular and siloed, learning another skill set is seldom a simple task.

Today’s tools vendors and the business leaders must work to create self-guided, closed-loop systems that are intuitive and don’t take an advanced degree to learn. Where a process cannot be completely automated, systems should be able to provide users guidance on what data to pay attention to in order to make decisions. Our research shows that usability is becoming more and more important to business users, and this is where we expect to see work done in terms of simplifying tools. We still have a long way to go, as my colleague Mark Smith pointed out earlier this week.

Given the way people talk about the analytics skills shortage, we might advise all of our children to get a degree in mathematics or statistics. That may be a good idea, but it also seemed like a good idea to go to law school just a few years ago. While mathematics and statistics are important foundational skills, perhaps we should regard them as a means to an end rather than as ends in themselves. The McKinsey research shows a projected skills gap in these areas, but as with any model it rests on underlying assumptions. One assumption is that analytics tools vendors will not quickly fill in the gaps and allow analytics to be consumed at a lower skill level with better wizards, information and suggestions on applying analytics. The ability to monitor and contribute to this progress is a key reason Ventana Research continues its research and education in the area of business analytics.

Regards,

Tony Cosentino - Vice President and Research Director