It’s not controversial to say that everyone’s chasing big data. But like the joke about the proverbial dog chasing cars, an important question follows: what do you do when you catch it? A survey out this week suggests that if you work for the CIO, your boss isn’t sure you know.
The Bain & Company survey published this week in Forbes researched a few hundred companies to see how well big data is driving them to be data driven. It’s not looking so good.
59% believe they lack the capabilities to generate meaningful business insights from their data. In another Bain survey … 85% said they will require substantial investments to update their existing data platform, which includes consolidating and cleaning data, simplifying access and rights management, and improving access to external data sources.
Just when everyone had graduated from mastering spreadsheets and slide decks, down comes a big vote of no-confidence from the executive suite on really driving the business with data. But is it really not a technology problem? And why does the survey suggest that IT isn’t putting better technology in place to help solve the problem?
The division of labor between IT stewards of data and business consumers of data has introduced real barriers between them. There’s a natural complexity to the multiple markets, processes, and functions that running a customer-facing business requires. Data is at the heart of those barriers, and those barriers emerged for good reasons: security, sustainability, speed of implementation, and architectural soundness, all constrained by the technology available when the systems were built. Also, money.
The journey from ongoing business needs to IT implementation loses something in translation. What makes this worse – both for technologists who process data and analytic users who consume it – is how long it takes and how hard it is to get the data and make sense of it. Big data can make this worse, especially when the tools and expectations both sides bring to the party date back to small-data thinking (and the complex stack of small data tools that many IT organizations still rely on). Just what do we mean by small data?
But big data can also ameliorate the gap between business needs and IT data stewardship. The secret is not in the size, but in the structure and access to the data. A large share of the applications generating big data are relatively new, and are developed with modern styles of agile development practices and iterative refinement of user requirements. As the Bain article in Forbes observes, software development has greatly benefited from agile methodology; it’s proven a better approach to drain uncertainty from the practice of understanding and building apps. And unlike older software engineering ‘waterfall’ practices, it recognizes that development is never really done, but driven at its core by a feedback loop. The key: involving business users in the development process, early and often.
A similar principle of iteration holds true for the way businesses and analysts access to big data. Contrary to what the survey says about CEOs skepticism of technology, data architects have an essential role to play in rethinking how users interact with data to begin with.
First, recognize that a stack made of small data tools and technologies can’t reliably accelerate the process of creating extracts, charts, and reports on really big data. Once that standalone extract of data yields some insights, you need to go back to the well and extract new a new set of data (or worse yet, submit an IT ticket to request a new report).
A better way: feed data directly from the source through to an interactive chart. Where many small-data self-service dashboards let end users pivot and drill within the same standalone extract, Arcadia Data provides interactive visual data applications that are premised on drilling down securely to the raw data. It’s the nature of data-driven inquiry that data doesn’t just provide answers; it creates more questions.
Secure access to raw data is good. Simplifying the creation and dissemination of the visuals is better. Arcadia combines Point-and-click data modeling and a broad set of easy-to-create data visualizations that you build and publish directly in the browser (cloud optional). Using Hadoop’s native security lets Arcadia manage role-based access control; that way, you can share it broadly across any number of users inside and outside the corporate firewall.
But what of all the natural barriers that keep IT and business-end users apart, that have made so many CIOs skeptical of organizational data readiness? Once you get a bigger audience of business users with more secure interactive access to more complete data, IT attention follows, and IT gets better aligned with what the business wants. That’s probably what the boss wants to begin with.