Analyst Perspectives

Natural language processing (NLP) is a field that combines artificial intelligence (AI), data science and linguistics that enables computers to understand, interpret and manipulate text or spoken words. NLP includes generating narratives based on a set of data values, using text or speech as inputs to access information, and analysing text or speech, for instance, to determine its sentiment. There are various techniques for interpreting human language, ranging from statistical and machine learning (ML) methods to rules-based and algorithmic approaches. In this perspective, we will focus on two aspects of NLP: natural language query (NLQ), which offers the ability to use natural language expressions to discover and understand data, and natural language generation (NLG), which uses AI to produce written or spoken narratives from a dataset. NLQ and NLG enable business personnel to communicate information needs with business intelligence (BI) systems more easily.

NLP is one of the especially useful application areas of AI and is currently undergoing rapid evolution as new methods and tools converge. A trending approach is using deep learning to train neural network algorithms to process linguistic information. Deep learning methods have become very efficient over the years and can now solve challenging natural language problems, like sequence-to-sequence prediction, with new modeling approaches. It learns features from the natural language required by the model itself and replaces existing linear models with better-performing models capable of learning and exploiting nonlinear relationships within massive unstructured datasets. Recurrent neural networks (RNN) are a robust type of conventional feed-forward artificial neural network that can work with sequential data or time-series data to solve common temporal problems seen in language translation and speech recognition.

Organizations are increasingly turning to NLP technology to make analytics more easily accessible and broaden use within the organization. Our recent research found that in three-quarters of organizations, 50% or less of the workforce currently use analytics. This shows that there is a need to move away from complex dashboards and data visualization to other more easily understood methods of delivering analytics. Our research also shows that more than two-thirds of organizations consider AI/ML, natural language search and visualization important for analytics. But newer techniques, such as conversational computing and augmented and virtual reality, are still making their way to mainstream BI.

We assert that through 2024, conversational experiences involving analytics will improve organizations' use of data but for three-quarters these experiences will remain primarily text-based rather than voice-based.

NLP and big data analytics can be used together to create systems that can query data at scale and perform various analyses, such as text or speech analysis, to facilitate decision-making. Organizations can use NLQ and NLG to build interactive searches with real-time access to all stored data to generate insights instantly. NLP can ensure that personnel are connected to the latest data and can easily extract relevant data and insights that would have previously been hidden within vast quantities of data and would have taken hours or days to extract.

BI and analytics vendors are continuously adding and improving NLP capabilities to simplify data discovery. However, there are still some challenges to perfecting NLP, such as user intent. End users can say or type a query that is not necessarily what they meant and, as a result, the system can deliver different information from what the user intended. For example, misspelled words or mis-phrased questions can the wrong query that will deliver incorrect results. Ambiguous words such as homonyms can also create a conflict in intent. An effective NLP system must be able to infer the user's intent and deliver relevant results by understanding the variations and context of a query.

BI dashboards have democratized data analytics but require additional AI capabilities to keep up with the quantity and complexity of the data generated today. NLP offers a solution to fill the insights gap between data experts who analyze big data and business personnel who make intelligence-driven decisions. The recent advancement in NLP offers a multitude of applications for organizations in all domains. NLP technology can simplify access to information and deliver insights in a personalized, human-like conversation.

Organizations looking to democratize complex data analysis and enable line-of-business personnel to easily extract information with the help of natural language should consider the NLP offerings of many of the analytics and BI vendors. NLP can facilitate faster workflows and help to bring analytics to a broader portion of the workforce, enabling personnel with a range of technical skills to easily query datasets in various formats and accelerate data discovery.

Regards,

David Menninger