December 13, 2018 - Dale Kim | Big Data Ecosystem

AI and BI Were Made for Each Other

Regardless of your opinion of the term artificial intelligence (AI), there’s no question machines are now able to take on a growing number of tasks that were once limited to humans. These days, AI is commonly discussed in the context of video games and self-driving cars, but it is increasingly becoming relevant in business intelligence (BI) and analytic platforms. AI has the potential to break down some of the major barriers to wider adoption of BI technologies, especially among more casual users. In fact, BI and AI can work together to make both technologies accessible to more people.

First let’s talk about how AI can help BI. BI has long been constrained by the need for users, especially business analysts, to master complex skills, including query languages like SQL. That’s because the tools assume that users know what data is available to them, understand the data, and have a general idea of the answers they expect when they ask a question. Certainly there’s an opportunity here to lower the barrier, especially by promoting a more exploratory process in BI and analytics. (Check out 451 Research’s views on search-based BI as one way AI can help BI.)

Data lakes have been a great technology for eliminating limitations to data analytics access by making it easy for people to query data in its native format. A natural next step for data lakes is using machine learning (a.k.a. ML, a popular discipline of AI) to simplify the analytic process needed to formulate structured queries. Technologies like SQL are restrictive as the primary analytics option because the knowledge needed to apply them to relational tables requires some study time. To be fair, SQL isn’t a difficult language to learn and master, but it does take time. And even if you are well versed with SQL, you still need to understand your data sets to apply SQL to them. What if you could use BI without being an expert in the tools, SQL, and/or your data?

The answer is in search, or in this context, search-based BI. Search-based BI is a key component of the technology category Gartner calls “augmented analytics,” which includes natural language querying to gain insights without advanced skills. Gartner predicts that by 2020, half of all analytical queries will be via natural language processing or automated. This level of usage growth makes sense, as no one needs training to use a search engine, but the technology that’s so much a part of our everyday lives has until recently been unavailable to BI users. That’s changing though. Starting with a basic set of rules about how human speech maps to data, machines can now infer from natural language queries what users want and present them with a best-guess response that serves as a basis for further exploration.

Search engines have evolved into capable query engines in ways we may barely notice. Years ago, a question like “What is the population of Wichita, Kansas?” might have returned a list of documents from the Census Bureau. That’s because the broad technology umbrella known as “search” was more about full-text search, in which terms in the query were matched to terms in documents. The matching process tried to identify the relevance of documents to the given query, which could then lead the user to an answer. The goal was not necessarily about immediately providing an exact answer. Today, search capabilities can serve up specific answers, and for the example above, they can deliver a precise number (population of Wichita: 390,591) along with a map, photos of Wichita points of interest, and information drawn from Wichita’s Wikipedia entry. Search engines not only answer the question but also reference other information the questioner might want to know about Kansas’s largest city. From there, a more detailed exploration process can begin.

Search engines perform this magic working with unstructured data. They use knowledge gathered from analyzing billions of webpages to derive structure, even if it isn’t explicitly defined. The same technology can be used to understand the content of data lakes, so users without specific knowledge of the underlying data, or any BI tool, can now get analytical insights directly from the many data sets in the lake.

That’s why we’re excited about leveraging the power of search to move the data lake to the next level. The search-based BI and analytics capability in our Arcadia Enterprise uses a query processor and open source libraries to both parse conversational sentences and to learn from query behavior over time. The engine can even suggest related questions the user might ask or show type-ahead suggestions the same way today’s Internet search engines do. As more people in the organization take advantage of natural-language search, the algorithms provide better and better contextual responses. For example, sales managers asking about revenue across geographies might be looking for different details than product managers. Over time, the search-based BI and analytics engine will give each group results that best match their areas of interest.

Natural language querying isn’t a replacement for SQL, or even BI interfaces, as it is most useful as a complementary technology. It’s a great starting point for further exploration. Rather than settling for a single answer, natural language querying encourages asking more questions that may lead to insights that weren’t previously investigated. And the results/answers can help set up a dashboard that a large business user community can use to help make important decisions as part of their daily responsibilities.

AI can help the BI and analytics lifecycle in other ways as well. Recommendations on using the right visuals and the right color scheme are also useful. For example, you might get an initial visualization as an answer for a natural language query, but you might wonder if there are other visuals that might be a better representation of your results. You can ask your system to display other ideal visual options on the exact data you just retrieved, not sample data that is unrelated to your immediate use case, so you can more easily find the best visual for your data. We call this capability Instant Visuals. You can also get recommendations on color schemes for your visuals to see make sure the right data points are highlighted.

Adding elements of AI into a BI platform is an example of how AI can make BI more accessible, but BI can also return the favor. Machine learning is an exciting branch of AI that is all but inaccessible to most people. It’s a statistically demanding discipline that is practiced mostly by data scientists who are comfortable working with tabular reports and long strings of numbers. With BI tools, however, the power of machine learning can be made more accessible. That’s because BI tools excel in the use of data visualization to make data meaningful. When dense columns of numbers are transformed into charts and graphs, relationships can leap off the page.

Going back to our Wichita example, suppose a political campaign strategist wanted to know the likelihood of precincts in the city to vote Democrat or Republican. This is a perfect machine learning use case because it involves scouring years of voting data to look for patterns that indicate the probability of different outcomes. BI dashboards can display the results of this query on a heat map using color-coding to designate confidence levels. This gives the strategist immediate insight into where to commit campaign resources. Another good example is customer 360 degree view analysis for industries where the risk of customer churn is particularly high.

If you’d like to see first-hand how our AI/ML techniques in Arcadia Enterprise can help with simplifying your BI and analytics lifecycle and how we can help you make sense of your ML outputs, please let us know.

Related Posts