Companies are modernizing their BI platform based on a massive shift in the big data analytics market which started with the Hadoop ecosystem and continues to evolve. This is not only a shift in technology in response to the scale and growth of data from digital transformation and IoT initiatives at companies, but a shift in end user (e.g., business analyst) preferences. Together, these shape a set of analytics capabilities that are “must haves” for staying competitive today.
Some such as Peter Cohan in Forbes magazine are calling this the “3rd Wave of BI platforms” (Business Objects et al were the “1st”, Tableau et al were “2nd”) but few if anyone other than Howard Dresner, Founder and Chief Research Officer at Dresner Advisory Services has watched the BI market evolve as closely over the past 30 years. In 1989, Howard Dresner coined the term “business intelligence” while working at Gartner (then called Gartner Group). Hence, the “the Godfather of BI”.
Fast forward to today, nearly 30 years later, and the 2018 Big Data Analytics Market Survey was just released. We at Arcadia Data are proud to have been recognized and ranked #1 in this report ahead of stiff competition of 3rd generation and legacy BI players.
This blog post will cover the shifts in the big data analytics market as reflected in Dresner Advisory Services 2018 Big Data Analytics Market Study, as well as how Arcadia Data has been architected and shaped by our customer requirements to be the best BI and visual analytics platform for modern data architectures.
Key Trends in Big Data Analytics in 2018
- This is the 4th year the Big Data Analytics Market Study was conducted and despite the relative backlash in the press around the term “big data,” the study shows that big data is approaching peak interest. Over 85% of respondents say big data is important, very important, or critical (with “critical” as the largest segment at 36.44%) R&D and the business intelligence competency center (BICC) are the greatest functional drivers of big data. And, industry (i.e., technology vendor) sentiment is rising to near all-time-high levels
- As the Dresner study states, “Actual users of big data account for an all-time high of 59 percent of our sample, and fewer than 10 percent have no plans, signaling mainstream adoption.” In my opinion, this is why we are seeing a big shift to business use cases and end user adoption, which is driving the requirement for easier-to-use and more scalable BI technology. Hadoop and big data systems start out in R&D groups with highly-skilled data scientists and data engineers discovering and creating value from the data, but the “last mile” served by BI platforms is providing more business analysts and “casual” users with direct self-service access to data lakes and other modern data architectures. This is where the value of big data increases exponentially.
- As discussed on a recent webinar with Mr. Dresner and Cloudera, we are also seeing that natural language search analytics and processing has increased in priority by 23% from 2017 to 2018, according to Dresner’s survey. I think this feeds back to the need for end users to interact more easily with information to find insights as part of their data discovery process. Search engines like Google are ubiquitous and have conditioned people to seek and consume information this way, which is why Arcadia Data announced the first search-based BI interface native to big data as part of our BI platform, Arcadia Enterprise, which runs directly within Hadoop and cloud architectures. This goes beyond text-based search and provides visual analytic answers to simple questions like, “what are the sales trends in California in 2018 vs 2017?”
How Were Big Data Analytics Vendors Rated?
As stated in Dresner’s study, “In rating vendors for big data analytics, we examined levels of functionality in five categories: infrastructure, data access, search, machine learning, and supported distributions.” For each of these five attributes or categories, here are my main takeaways:
Infrastructure – This category is about supporting the most important infrastructure technologies today. Spark has been important for several years now, but the hot newcomer for this year’s report is Apache Kafka. I believe Kafka is so important today that if your BI tool can’t directly visualize Kafka topics, then you will lose ground to your competitors who use real-time streaming analytics as a competitive weapon. Kafka is a key technology that will let you uncover insights faster, react faster, and make the right business decisions faster.
Arcadia Data leads the industry for real-time BI with its ability to support streaming visualizations on Kafka topics through our partnership with Confluent and support for KSQL – a streaming SQL interface.
Data access – Some of the top methods for big data access in the survey includes Amazon S3, Spark SQL, Hive, and HDFS. Also near the top was Azure Data Lake Store (ADLS) which confirms wide use of cloud object stores for big data storage. As businesses turn more to the cloud, cloud object stores are a great starting point for leveraging the elasticity advantages that the cloud promises.
Arcadia Data runs natively within big data platforms such as Hadoop and the cloud to give extreme scale and speed advantages along with much lower TCO because data does NOT have to be moved to a special-purpose analytics engine, unlike traditional BI tools
Search – Incorporating search technologies like Solr, Cloudera Search, and Elasticsearch is critical for the growing capabilities covered under the BI term. If you’re not taking advantage of these technologies for your unstructured data, then you’re possibly not getting value from a sizable segment of your business data. And as mentioned above, natural language processing goes beyond text-based search to provide search-based BI and easier ways for casual users to visualize and iteratively analyze data and create dashboards. This is a key capability of Arcadia Data which combines a complete BI platform with natural language analytical processing.
Machine learning – Machine learning (ML) together with artificial intelligence (AI) are some of the hottest topics in data today, and frankly, we believe that AI and BI were made for each other. The next wave of BI requires ML/AI techniques to make the analytic process easier for a wider range of users. From search-based BI (a.k.a., natural language querying), to visualization recommendations, to automated BI acceleration, ML/AI will increasingly play an important role in BI.
Arcadia Data was early to incorporate ML into our BI platform. First, on the back-end, we provide a recommendation engine as part of our Smart Acceleration process which enables faster performance and high user concurrency for dashboards. Second, Arcadia Data’s “Instant Visuals” recommend the best way to visualize data based on the dimension, measures, and cardinality of data and learns over time based on user feedback. And, finally, our new search-based BI incorporates ML and AI to extend ease of use by allowing natural language searches to bring back visual analytic answers — learning over time what the most popular visualizations are and recommending those to users. Big data analytics made easy with machine-assisted insights.
Supported distributions – Software distributions that package and support open source software together from the Hadoop ecosystem with other distribution-specific components give customers a common and modern data platform for big data analytics. BI tools that are certified with the broadest set of popular big data and Hadoop distributions give customers the widest choice and helps avoid vendor lock-in. In order, the most popular big data distributions in 2018 were Cloudera, Hortonworks, MapR, and Amazon EMR. Microsoft HD Insights and Google Dataproc had the most plans for support.
Arcadia Data was built from inception to run natively on Hadoop, cloud and other modern data architectures. This is why architecture is important for BI on big data, and companies are choosing a new BI standard for their big data systems such as data lakes. We support a broad and growing ecosystem of next-generation data platforms including systems beyond this market study, such as Snowflake DB, in our latest product release.
Arcadia Data had the highest overall score among these five attributes. Now you might be asking whether this matters to you. Perhaps you already have one of the tools listed in the survey, and perhaps it didn’t rank in the top 10, and maybe instead ranked 16th, but isn’t that good enough? Should you even worry about investigating new BI technologies for your big data initiatives?
Once again that old adage, “use the right tool for the job,” comes to mind. And yes, that implies your #16 tool might not be the right tool for big data. It might be perfect for your traditional analytics environment, and it might be satisfactory for big data, but there are good reasons why a native big data BI platform provides significant advantages for you. It’s very reasonable to add another BI tool to your portfolio, and if big data is important to you (and this report shows that many companies believe it’s important to them), then it’s worth exploring new options. If this feels like you will end up procuring redundant tools, take a look at one cost analysis model that breaks down the advantage of having BI that is native to big data. We believe you will be better off with a distinct set of technologies specifically used to get you the most out of your big data.