This blog was first published on Forbes.
The amount of data generated by devices and machines continues to rise. Exponential data growth places an enormous amount of pressure on systems that must keep that information moving at high speeds from machines to data lakes or warehouses and into analytics platforms. Data storage and management are only a subset of today’s data-driven requirements — deriving analytical value is also extremely important. That value is difficult to show without systems that can analyze streams of information in real time. In particular, organizations are trying to enable non-technical users to elicit value from their data using off-the-shelf visualization tools.
Enterprises are exploring a variety of architectures and technologies to incorporate real-time analytics on streaming data into their ecosystems. An emerging area is the use of general purpose visualization tools on streaming data. Instead of the traditional approach of custom coding and integrations that apply in only limited situations, this new paradigm simplifies the extraction of value from streaming data. Recent innovations are opening up a new range of capabilities for non-technical users. Such capabilities enable many valuable use cases: predictive maintenance, operations optimization, financial services risk reporting and cybersecurity, among others.
Ultimately, the usefulness of streaming technologies will be measured by the businesses who depend on them for critical capabilities and use cases. Here are a few examples of real-time capabilities and the use cases they enable that are critical for modern enterprises:
Streaming Analytics Capabilities Support Real-World Use Cases
Alerting is an obvious and critical capability for streaming analytics. You can receive automatic feedback when a certain event or trend occurs that indicates the need for attention. While alerting is not unique to streams, the fact that users can receive alerts in real time means that they can respond more quickly. There is no built-in delay due to technology processes like moving data into a large data store. The following use cases exemplify how this might work.
• Streaming analytics can be used in cybersecurity, where anomalous behavior should be immediately flagged for investigation. Cybersecurity environments are increasingly turning to machine learning for identifying anomalous, and thus potentially suspicious, behavior in a network. Using alert-based visualizations in conjunction with machine learning outputs is an ideal way to get a broader community of analysts to help with detecting cyberthreats, so you don’t have to depend only on security experts and developers.
• The retail industry is another example where alerting can be beneficial. Priorities differ from store to store, and you can’t have your information technology (IT) team write custom code for alerts for every possible scenario that needs attention. So instead of writing code to detect events such as low inventory or high customer traffic, alerts using analytical tools can help non-technical staff easily monitor these events so they can respond immediately.
Side-By-Side Historical And Real-Time Analysis
In many situations, it’s important to have historical data alongside real-time data to have a complete picture of your business. An example of where this is useful for financial services organizations is with regard to real-time risk assessment. In this case, the full life cycle of a transaction needs to be considered, which includes past (the trade was agreed to and executed) and present (post-trade events like amendments, transfers, and terminations). A real-time alert of a trade event, although important, is more valuable in the context of the larger historical picture of the entire set of portfolios. To do this at scale is crucial because the domino effect of any series of real-time events on a larger system of trading strategies could be the difference between success or a massive loss that triggers regulatory scrutiny and reputational risk. For example, if a large position (relative to historic norms) is taken shortly after a risk model has been changed to enable it, it is better to investigate that sooner than later.
Stream Data Enrichment
In certain industrial Internet of Things (IoT) environments, all of the records share some of the same information. For example, you need to keep track of the source of the data, but you don’t want every record to have to contain information about the data source. This could be problematic if you are dealing with thousands of data points or more per second. This means a significant amount of your data is redundant.
Instead of storing all that redundant information, you can put one copy of it in a lookup table. By running a real-time join of the stream data with the lookup table, you can generate a complete record on which you can run visualizations.
• For example, a single oil rig will have “static” data such as manufacturer and location. Instead of repeating that data across all data points, simply store that in the lookup table in a database and join it by a key such as “machine_id.” This saves on data storage requirements while still providing full details of the data source to enable granular analytics. In this example, by having information such as location, you can see if the oil rig’s location plays a role in wear and tear on the rig, which can be used as a factor in performing predictive maintenance to minimize non-productive time.
Streaming Data Offers Insights Not Found Elsewhere
We are in a time of unprecedented interest and development around streaming technologies and analytics. This is driven both by technological advancements and the increased realization of business value from streaming analytics and real-time visual analysis. Expect this trend to continue at an even faster pace as we adopt more connected devices and the connected world built on the Internet of Things matures. Businesses looking for the next source of competitive advantage will turn to streaming data to gain insights that they cannot generate from their existing approaches to analytics.