How Big Data and AI Have Revolutionized Financial Trading

Many banking and finance companies have already taken advantage of Big Data analytics to simplify the process of personalized offers, targeted cross-sales and to improve their customer service. The term Big Data keeps expanding and today incorporates numerous new meanings, such as Deep Learning, Cluster Analysis, Neuron Networks, and Artificial Intelligence.

Banks are powering their scoring models with social networking, payments, search history and other client’s behavioral data. Big data systems have evolved from being simply passive supplementary technology, once used to provide more accurate targeted information, and have become the deep learning mechanisms we see today, capable of making decisions and taking actions. These technologies allow financial institutions to address much more complicated goals like fraud prevention, process adherence, and regulatory compliance issues worldwide.

AI technologies are already widely used for surveillance tasks by Regulators and Exchanges. They perform the micro-structure analysis of market data, detecting and preventing issues such as insider trading, harmful algorithms, and market manipulation.

In contrast, major quant hedgers have used big data/deep learning for decades, working on neuro networks to build predictive investment models and looking for the most profitable algorithms. According to Bloomberg, Big data-driven machine trading has grown significantly over the past 10 years: estimates hold that it accounts for roughly 40% of trading volume in European equity markets and about 55% in U.S. equity markets. In futures markets the volume of trading of foreign exchange futures has grown to roughly 80%. HFT algorithmic trading firms have in recent years come to dominate U.S. stock trading by using supercomputers to pick off profits across dozens of electronic markets in less than a blink of an eye. Along the way, the shops have also drawn criticism from those who say they have “gamed” the system at the expense of everyone else.

Firms such as Renaissance’s Medallion fund, use algorithms that analyze reams of data to find signals hidden in the noise of markets and predict changes in prices of futures, equities, and currencies. This fund alone demonstrated huge annual returns of more than 70% on average (before taxes) for two decades (1994-2014), which is more than 7 times the average annual S&P gain.

This success has attracted the attention of big money and launched a new investment wave in the use of computer power in trading over the last 10 years. But this fact is also one of the reasons that profitability of quants started to decline, and currently in 2017 many of them even underscored the S&P growth. The reason for this is quite simple – as more players start using machine trading algorithms, the less effective those algos become. To understand this with more depth, we can imagine a group of AI-powered mechanisms sitting around a poker table trying to beat one another, but there are no bad poker players among them anymore. In a similar way to the poker game, the so-called “dumb” money has already been wiped out from the market. In order to gain an advantage over the other players, the more data that can be considered at once, and the more accurate this data is, the better chance you have to beat others. In terms of trading software, only those systems which are capable of learning from information sources and accessing data more quickly than others, can win the party.

It is also worth pointing out that the progress of big data technologies runs in parallel with the development of visualization techniques, like Heatmap, Zooming UI and Augmented Reality where traders can navigate through large volumes of data, which was not possible before in legacy 2D visualization software.

Evgeny Kuznetsov, Director of Product Management, Market Data at dxFeed. He is an IT professional with 15 years of experience in Requirements Engineering, Solution Architecture, Product Marketing and delivery of complex B2B software solutions for Fortune 500 companies.