Evaluating Analytics and BI Software Vendors’ Capabilities

Ventana Research has been evaluating analytics and business intelligence (BI) software for a long time—almost 20 years. Our methodology for these assessments is referred to as a Value Index. We use weightings derived from our benchmark research about how you, as buyers of these technologies, value and evaluate vendors. You can view our 2019 Value Index results here. I am in the process of completing the 2020 evaluation now.

As we consider analytics vendors, we look at over 1,500 criteria for each vendor. One-third of those criteria are related to capabilities of the vendors’ products. My goal here is to give you a perspective on why we evaluate such a broad range of capabilities, and to help your organization understand how it should consider a more robust evaluation of what you need from analytics and BI.

Let’s work backwards through the analytics process to understand the categories we evaluate. Analytics should lead to action in an organization. To drive action, you must not only make decisions, but you must communicate those decisions and the actions they require to the appropriate portions of your organization. Where possible, you should also be able to automate those tasks, activities and processes, including some of the actions resulting from the decisions.

There are many aspects to communication in analytics processes. People consume information on different devices and collaborate across multiple channels. Each type of device should support rich, personalized interactions that are appropriate to the individual and the device. For example, you don’t necessarily need to prepare a completely new analysis from scratch on a mobile phone, but you should be able to ask questions and get answers using natural language processing. We assert that by 2022, one-half of organizations will be using conversational computing in voice and chat methods to simplify interactions with business applications. Organizations also need to communicate their plans, strategy and even vision to make sure everyone is aligned around the same goals and objectives. Without this context, individual analyses may be made in a vacuum and may not support the organization’s mission.

Making decisions requires several steps that go beyond merely looking at historical data in a visualization. History can be informative, but analyses should also include predictive and comparative analyses of alternative scenarios for dealing with what has been observed. One solution for combatting declining sales may be found in a recommendation to increase future advertising and marketing campaigns. An alternative recommendation may be to hire additional inside sales reps to make more outbound calls. A complete evaluation of the alternatives using analytics would compare the projected costs and benefits of each. If increased marketing campaigns are selected, for example, predicting which offers to make to which customer segments will help maximize the effectiveness of the spend.

Not everyone in your organization will understand how to perform these different analytical steps. Guidance is both helpful and necessary. Simple guidance might include a series of related analyses sometimes referred to as a “story” or “presentation.” More sophisticated guidance includes natural language explanations of relationships in the data, and suggested next steps to consider. Providing guidance also ensures consistency throughout the organization.

Analyses should be supported by multiple presentations and ways to interact with the information. In some cases, a simple list or report is enough. In other cases, a dashboard including a variety of metrics may be more appropriate. And in others, interactive visualization and exploration of the data is necessary. With the volume of presentations and information available in organizations, employees need to be able to easily and efficiently search through all the material to find the right information and analyses.

Underlying all of this is the data behind the analyses. There are several key requirements around data processing. First, which sources can be accessed, and how easy is it to access those sources? Second, how does the product support data preparation? These two tasks, connecting to data sources and preparing data for analysis, are the most time-consuming tasks in the analytical process for many organizations as was found in our Data Preparation Benchmark Research.

We also evaluate the richness of the data modeling capabilities. Line-of-business personnel should be able to view the data organized in a way that is familiar to them, including hierarchies and other associations in the data. It should also be easy to create derived metrics as part of the data model since these are generally the basis for key performance indicators in an organization. Throughout all these data processes we also examine the data quality and governance capabilities.

Our criteria may not align entirely with every organization. And no vendor has ever fully met all the criteria we evaluate. In your organization’s evaluations, determine what importance to assign these different categories of capabilities. If you choose to use multiple products to meet your needs, make sure to consider how well the products integrate with one another. Fortunately, vendors are starting to include more embedding and integration capabilities to products, opening up a wider range of options for an organization.

Hopefully, this analyst perspective will help your organization evaluate analytics and BI software vendors capabilities. Organizations should consider the full breadth of criteria highlighted above to insure they can maximize the value of their analytic processes.

Regards

David Menninger