BI and Analytics Meet Business Transformation
A Brief History of BI
It may come as a surprise, but the term “business intelligence” was first used in 1865 by Richard Miller Devens in his book Cyclopedia of Commercial and Business Anecdotes. However, it was nearly a century later when it was first adopted by the technical community when IBM researcher Peter Luhn published “A Business Intelligence System” in 1958. Even then, it took until the 1980s for BI to gain visibility. This is when Bill Inmon and Ralph Kimball conceived the first “data warehouse” which brought data from multiple sources, storing the results in a central “system of record” (in their case, a mainframe computer).
Initial data warehouses were tightly managed by corporate management information systems (MIS) departments. They controlled the entire process from end to end—creating, scheduling, and distributing “canned” reports to company’s executives and other line-of-business counterparts. When an individual or group required information that was not provided in those prepackaged documents, they submitted formal requests which took between days, weeks, and in some cases months to fulfill. As time progressed, demand for these ad-hoc reports multiplied in number, resulting in increased frustration and backlogs.
By the early 1990s, companies interested in cutting costs started to migrate their data management operations from mainframes to “client-server” environments using relational databases to house their online transactional processing (OLTP) while introducing the world to online analytical processing (OLAP). As part of this platform shift, the first generation of independent BI companies emerged to simplify report development and distribution. Enterprises often purchased large quantities of these products, but the vast majority of reports were still developed and maintained by MIS, with pockets of non-technical “power users” scattered across their ranks for good measure.
At the same time, use of computer-aided manufacturing (CAM) applications grew, including material resource planning (MRP), supply chain management (SCM), and product data management (PDM) systems. Computer-aided design/computer-aided drafting (CAD) products forever changed how people managed development of new “things.” These “things” eventually would gain “smart” capabilities as well as the ability to connect with each other, which would be referred to as the “Internet of Things.”
While enterprise resource planning (ERP) products were initially developed in the 1980s, they became more popular around this time due to the desire for companies to cut costs through using less expensive UNIX servers instead of mainframes and minicomputers. Further, traditionally non-technical departments entered the information age through the introduction of sales force automation (SFA) and customer relationship management (CRM) applications, transforming operations for their respective organizations as well.
Even though the target audiences for each of these systems were very different, one key detail all of them had in common was rudimentary (or in some cases, non-existent) reporting capabilities. This spawned the beginnings of the “analytical applications” market with both incumbent BI players as well as startups racing to reverse engineer these systems to address this glaring oversight.
In the mid to late 1990s, companies began to transform the Internet, which previously was mostly limited to government, academia, and increasingly, high tech companies, into a communications channel as well as a source for generating revenue. BI embraced this new approach, offering “zero footprint” BI clients that were accessible by a desktop web browser. When used in conjunction with “analytical applications,” this helped set the stage for the advent of the enterprise performance management (EPM) market.
In the early 2000s, companies that were unable to invest in the human, hardware, and software capital associated with deploying traditional enterprise resource planning (ERP) and CRM systems on their own, finally were given a way to do so with the introduction of application service providers (ASP). Companies like Asera, Corio, and others offered their customers the ability to share a common installation. BI once again followed, which laid the seeds for the “Self-Service BI” movement.
Today there is an increasing number of tools claiming to offer close to real-time analytics for business analysts of all stripes—not just data scientists. But as we will see in later chapters, not all these offerings are created equal.
Wither the RDBMS? Not So Fast…
Where do these latest developments leave the traditional BI world of structured data-driven data warehousing? After all, most of these traditional approaches are relatively expensive. Their ability to discover new patterns and business insights is highly limited to the subset of “normalized” data that has been painstakingly extracted and placed into a data warehouse or mart. Yet there is a rich and broad-based set of tools to very quickly analyze that information — and a great deal of legacy IT expertise to support that environment.
It turns out that most enterprises today are not debating if data warehouses, enterprise data management systems, or even (big) “data lakes” will emerge as their ultimate data repository. They will all co-exist, and will continue doing so for several years to come. As technologies like Hadoop/Spark, NoSQL, NewSQL, and object stores are all complementary with the traditional technologies, organizations should expect to use some combination of these technologies together.
Since the vast majority of new data being created is recognized as “unstructured,” it is highly probable that use of traditional RDBMS systems will wane over time as the use of Hadoop/Spark and the other big data technologies will increase. After all, these big data technologies were designed to elastically accommodate data growth.