May 31, 2018 - Paul Lashmet | Industry Solutions

Cross-Functional Trade Surveillance: The Keystone to a Holistic Trade Surveillance Strategy

This article was originally posted on the Cloudera VISION site.

Criminals don’t refer to a playbook of best practices to execute a crime. They are creative in their thinking and collaborative in their efforts (including with parties who may not know they are complicit) to obtain their objectives and avoid getting caught. Trade surveillance in the financial services industry must respond to their creativity and the complexities of a global trading network, the myriad of digital and physical communications, as well as social media-savvy players that develop sophisticated schemes, often converging on different forms of financial crime.

Regulators are quite aware of this trend, having prosecuted the same and are now demanding that firms take a proactive and holistic approach to preventing financial crime. This approach also relates to monitoring internal fiduciary risk by tying separate events together, such as a large position (relative to historic norms) being taken immediately after the risk model that would have flagged it was modified in a separate system. Prevention also applies to avoiding corporate embarrassment and reputational risk that can arise when employees who disregard code of conduct standards are not identified early on. All three cases require a “big picture” approach that incorporates new and alternative data sources and cross-functional collaboration throughout the organization not only to identify illegal activities, rogue traders, or personal misconduct but also to provide evidential material that demonstrates a deep understanding of the intent.

This post describes how native visual analytics on a Cloudera Enterprise Data Hub is the keystone that supports a holistic trade surveillance program. This architecture lets you store and process all requisite data types to enable cross-functional teams to identify and investigate events more efficiently while establishing a robust and defensible process across their enterprise.

Let’s start with a $78 million “pump and dump” securities fraud example in which multiple perpetrators conspired to pump up the stock of a little-known coffee company and then dump their own shares on the market, profiting from the artificially inflated price. The small coffee company became a tradable entity when it was merged with a public waste management company, the price of which was inflated through a fraudulent promotion campaign. This example combines three types of unrelated data:

  • Legal entity data: Two companies with completely unrelated business lines (coffee and waste management) merged together;
  • Unstructured data: Fraudulent promotion campaigns took place through press releases and a fake stock-picking robot.
  • Market data: Coordinated trading among multiple parties. You need to make sure no one from your team is part of the scheme.

This example took place in 2015 and was only discovered through extensive cross-jurisdictional coordination among global regulatory bodies. Imagine how much more sophisticated these schemes have become in the last three years considering the pervasiveness of social media and the introduction of new fraud targets such as cryptocurrencies. It is difficult to keep up and the damage is hard to rollback after the fact. This is why regulators are putting more pressure on firms to prevent issues in the first place. Market spikes and fake news help to spot trickery but to prove deception by correlating one with the other cannot happen if surveillance processes are fragmented.

Problem: Fragmented Surveillance

Preventative surveillance requires an iterative, exploratory, and collaborative process to keep up with the sophistication of schemes.

Most organizations are comprised of independent business and functional groups. To respond to their respective business, operational, and regulatory requirements in a complex and competitive marketplace, each group needs its own set of subject matter expertise and best of breed technology platforms. This is also true for the transactions and communications within each group but multiple functional layers lead to a fragmented surveillance environment.

The chart below illustrates a scenario in which distinct surveillance teams, each good at what they do, monitor legal entity data, electronic communications, market activity, audio, server logs/sensors, and video. The problem is that each surveillance team responds to the output of their platform with their own set of tools and so the analysis is done in isolation from the others, limiting the ability to spot sophisticated risk activities that cross-functional areas. Although the teams can collaborate through committees, working groups, and review boards, the sharing of data is often inefficient and a manual process. The pump and dump example described above would be difficult for the markets surveillance team to detect without the direct expertise of the electronic communications and legal entity teams.

Solution: Cross-Functional and Collaborative Surveillance

Below we start with the same foundation of distinct best of breed surveillance analytics, each of which provides data of suspicious and/or other relevant activity. However, in this case, that output is ingested into a data lake. Instead of each group’s tools acting on the output in isolation, they leverage a common visual analytics platform that is native to the lake and uses all of the data without moving it to a separate server. Not moving data mitigates data loss, ensuring data integrity and if the platform security of the data lake is inherited, then the data will only be viewed by those with proper access.

unnamed (1)

By implementing native visual analytics on a data lake, analysts are enabled to visually explore data across surveillance platforms and correlate risk activity patterns that would normally not have been identified when looking at each area in isolation. Each of the investigative teams described above will more easily collaborate as a single investigative group. They can use both predefined dashboards and drill down to the transaction or social interaction data in an ad hoc manner. For example, the group could more easily correlate suspicious corporate mergers and fake news to a market spike, identifying potential risks and/or build a case for investigation.

Going Forward: Improved Economics

Native visual analytics on a data lake improves the economics of a holistic trade surveillance strategy. Costs associated with acquiring and processing data, and the time and personnel needed to put the strategy into action, are critical to its success.

Data lakes store more varieties and process higher volumes of data at a significantly lower cost than traditional data platforms. Native visual analytics provides the subject matter experts with a direct and intuitive interface into the data that enables them to investigate in a timely, secure, and collaborative way. Moving forward on the strategy illustrated above, machine learning and AI-based surveillance processes can be run directly within the lake, enabling the subject matter experts direct access to the results.


Arcadia Data and Cloudera can help you deploy a trade surveillance solution that gives you the “big picture” you need to identify and stop rogue trading activity. Large volumes of disparate datasets can be managed in Cloudera, with Arcadia Data as the powerful analytics platform that helps multiple teams work together to monitor trades.

Arcadia Data partners with Cloudera to realize their shared vision of enabling subject matter experts to gain business insight from modern data platforms. Arcadia Enterprise runs within the Cloudera data platform and enables business intelligence (BI) and rich visual analytic applications to be built for hundreds of business users working on data in Hadoop. Arcadia Enterprise is certified with Cloudera to enable Cloudera Manager-based installation and management and Apache Sentry-based central authorization to reduce administration cost and reduce security risk. Our partnership extends beyond certification into joint engineering and solution development such as the joint cybersecurity solution based on Apache Spot.

Related Posts