Financial Services trends seen at Elastic
Financial Services (FS) is an industry that I find exhilarating to work with as a solutions architect here at Elastic. Well, most of the times until you ask dreaded questions like...
“So how long will it take you to get that infrastructure racked and ready to go?”
The responses are mind-blowing, to get a mid-spec’ed server can range anywhere from three months to a whopping nine months plus. People then wonder why there has been a lack of innovation in this FS sector for many years, go figure!
In this post, I am going to take a look at some of the Financial Services technology trends that I think are interesting and worth taking note of.
Who uses Elastic currently and for what?
One aspect that never ceases to amaze me during my time at Elastic is the wide variety of use cases that the Elastic Stack gets used for.
The Elastic Stack is a great logging solution as illustrated by Citi, or for security analytics and threat hunting like that of Barclays and USAA. However Elasticsearch is also great for “You Know, for Search” and is enabling organisations to conform to regulations like PSD2 (Speed Layer) with both Rabobank and Collector Bank using our stack for such a use case.
However did you know that Elastic has become a go-to technology across all Financial Services organisations? Below are just a few examples:
- Goldman Sachs use our components for tracking and analysing stock trades to provide better financial guidance.
- FICO uses the Elastic Stack to help determine and protect your credit score.
- Softbank Payment Service uses our product to monitor transactions and service performance while also monitoring for fraud.
- We also have several investment banks working on building solutions to help mitigate against the new EU CSD Regulation which focuses on the late settlement of trades.
This list goes on and on, but hopefully, you get my point.
What trends do we see on the Solution Architect team?
Anyone who works within Financial Services knows that there is a constant shift in regulation, in the past few years this has stepped up with new regulations like, CSDR, Open banking (PSD2), enhancements to MiFID II and the list goes on. These changes in regulation have forced companies to evolve and adapt to these new requirements. Let me give you an example:
Changes due to the Open Banking Regulation
Open banking, as its name suggests, means that “we” the consumer of financial products (accounts, loans, credit cards, etc. ) have the right to access that data and use it however we see fit. Before open banking, you could only really see your transactional information on the terms provided by your bank, e.g. bank statements received in the post. This transparency then improved with the introduction of online banking and has led to open banking.
One of the common discussion points we are having with many customers is how they offset the needed increase in hardware resources required to power their legacy data stores (mainly the mainframe) to comply with Open Banking Regulation usage requirements. This is where Elasticsearch comes into the picture.
Mainframes have been the de facto system of record for all financial institutes since the ’50s. The problem they face now is that those systems were never architected to power real-time customer interactions, and especially not via dynamic API’s. This new requirement allows all customers of a financial institute to access their data whenever and however they like. One recurring conversation we are having with financial institutes is how many more MIPS they would need to buy in order to handle the new workload that will arise from their Open Banking API’s. With the Elastic Stack, there’s no need to make costly investments in expensive proprietary technology.
Below are a few key points that illustrate why Elastic is a great fit for such a use case:
- API first - The RESTful APIs of Elasticsearch allow for quicker and simpler adoption of PSD2 regulation, as you are not having to retrospectively build functionality into a system that was not built for such a purpose.
- Parallel operation - Traditional systems like mainframes have a place in an organisation (high-speed transaction processing), and will not be removed in the near term. But that does not mean that you can not run both in parallel. We are seeing great success within organisations who are taking real-time feeds from their traditional systems allowing them to innovate and provide newer functionality that can only be provided by using a modern datastore.
- Horizontal scalability - The Elastic Stack can easily grow to any scale, without downtime. This is a key feature of all new systems and Elasticsearch has this capability with abundance, as the core founding principles of our stack is “Speed, Scale and Relevance”.
- Ease of management - IT departments are getting less budget, and being asked to streamline their operations. One key way this can be down without hindering technology adoption is via automation. Elastic Cloud Enterprise and Elastic Cloud on Kubernetes allow an organisation of any size to manage their Elastic Stack deployments with just a small, skilled team. All it takes is a few skilled engineers, and institutions can manage hundreds (or thousands) of Elasticsearch clusters.
As a real-world example, Rabobank has written an excellent article talking about their implementation of Elastic to ensure they can provide and grow with the additional requirements from regulations.
Trade Tracking or/and Central Securities Depositories Regulation
Central Securities Depositories Regulation (CSDR) is a new regulation that has come into play which I have already discussed in much more depth, you can find it in my previous CSDR compliance blog. To extract one key point from the article I wrote was the following:
“Participants are required by CSDR to settle their transactions on the intended settlement date. To provide a strong incentive for timely settlements, CSDs are required to monitor settlement fails and report them to relevant authorities.”
The crucial part of this statement is the implication that all Investment Bank needs to know what is going on internally as they settle a trade. With the usual combination of legacy tech debt combined with acquisition and mergers of organisations, this has lead to a tangle of systems that are all touched in the settlement of a trade. In order to get more visibility of what is going on internally, this means in many cases consume data a wide range of different logs, metrics and audit data in order to make sense of what is going on.
A successful trade tracking system needs to be more than just a system of record. It needs to be fast, powerful, scalable, and smart. And the Elastic Stack is perfect for those requirements:
- Observability - The ability to understand and see what is going on is a key part of this requirement. Elastic and its ability to make sense of not only Logs but Metrics and APM data is fundamental in this endeavour.
- Flexibility - Elasticsearch has incredible flexibility not only take an array of different data sources, but also handle data that comes in many different formats. This provides the foundation upon which teams can iterate upon to build their own common data format or can standardise on top of the Elastic Common Schema.
- Scale - Scale is paramount due to the volume of data that will need to be collected. In order to provide a truly unfiltered view of a trade, you need data from all settlement systems in order to provide that continuity view. This data can quickly mount up, but with Elasticsearch, scaling is not a problem. Don’t just take my word for it, check out this talk from Uber where they discuss how they scale Elasticsearch to handle up to 1.3 million events per second.
- Speed - When you are relying on a data store to influence business-critical actions, speed is of the utmost importance. This is something that Elasticsearch can handle, and this is also illustrated in the above talk by Uber. They run a cluster with more than 1 PB of data and require sub-second response times when scanning up to 4 billion documents per second.
- Machine learning - We are seeing the use of the Elastic machine learning functionality used to provide an early warning system for when settlements deviate away from the norm. This is helping organisations understand there is an issue before it hits the Intended Settlement Date. This allows them time to resolve the problem, even when that problem is early on in the settlement cycle.
Visibility and understanding of internal system steps while settling a trade is precisely what Goldman Sachs set out to rectify as they built their next-generation trade tracking solution. Learn more about their system by watching their presentation from Elastic{ON} 2016.