This article was originally written by Avanade alum Dael Williamson.

I recently worked with Databricks to create a new financial risk management platform in response to a common problem we hear from clients – how can I find, organize and use the data I have so it delivers the greatest value?  Antoine Amend, Technical Director - Financial Services, Databricks captures the key learnings in the blog post below. As a contributing author, I would like to give you a glimpse into the journey.    

Financial Institutions today are still struggling to keep up with the emerging risks and threats facing their business. Managing risk, especially within the banking sector, has increased in complexity over the past several years. 

First, new frameworks (such as FRTB) are being introduced that potentially require tremendous computing power and an ability to analyze years of historical data. Second, regulators are demanding more transparency and explain ability from the banks they oversee.  Finally, the introduction of new technologies and business models means the need for sound risk governance is at an all-time high. However, the ability for the banking industry to effectively meet these demands has not been an easy undertaking. 

Agile approach to risk management
Traditional banks relying on on-premises infrastructure can no longer effectively manage risk. Banks must abandon the computational inefficiencies of legacy technologies and build an agile modern risk management practice capable of rapidly responding to market and economic volatility using data and advanced analytics. 

Our work with clients shows that as new threats emerge, historical data and aggregated risk models lose their predictive values quickly. Luckily, modernization is made possible today based on open-source technologies powered by cloud-native infrastructure, that bring an agile and forward-looking approach to risk management.

Traditional datasets limit transparency and reliability
Risk analysts must augment traditional data with alternative datasets to explore new ways of identifying and quantifying the risks facing their business, both at scale and in real-time. Risk management teams must be able to efficiently scale their simulations from tens of thousands up to millions by leveraging both the flexibility of cloud compute and the robustness of open source computing frameworks like Apache Spark.  

They must accelerate model development lifecycle by bringing both the transparency of their experiment and the reliability in their data, bridging the gap between science and engineering and enabling banks to have a more robust yet agile approach to risk management.

Data organization is critical to understanding and mitigating risk
How data is organized and collected is critical to creating highly reliable, flexible, and accurate data models. This is particularly important when it comes to creating financial risk models for areas such as wealth management and investment banking. In the financial world, risk management is the process of identification, analysis and acceptance or mitigation of uncertainty in investment decisions. 

When data is organized and designed to flow within an independent pipeline, separate from massive dependencies and sequential tools, the time to run financial risk models is significantly reduced. Data is more flexible, easier to slice and dice, so institutions can apply their risk portfolio at a global and regional level as well as firmwide.

Plagued by the limitations of on-premises infrastructure and legacy technologies, banks particularly have not had the tools until recently to effectively build a modern risk management practice. A modern risk management framework enables intraday views, aggregations on demand and an ability to future proof/scale risk management.

Replace historical returns with highly accurate predictive models
Financial risk modeling should include multiple data sources to create more predictive financial models. A modern risk and portfolio management practice should not be solely based on historical returns but also must embrace the variety of information available today. 

For example, a white paper from Atkins et al describes how financial news can be used to predict stock market volatility better than close price. As indicated in the white paper, the use of alternative data can dramatically augment the intelligence for risk analysts to have a more descriptive lens of modern economy, enabling them to better understand and react to exogenous shocks in real time.  

A modern risk management model in the cloud 
Avanade and Databricks have demonstrated how Apache Spark™, Delta Lake and MLflow can be used to organize and rapidly deploy data into a value-at-risk data model calculating how financial institutions can modernize their risk management practices by moving to the cloud and adopting a unified approach to data analytics with Databricks. 

Using the flexibility and scale of cloud compute and the level of interactivity in an organization’s data, clients can better understand the risks facing their business, quickly developing accurate market risk calculations. With Avanade and Databricks, businesses can quickly find out how much risk can be decreased and then accurately pinpoint where and how they can lower risk as quickly as possible. 

For business leaders who want to find out more information about financial risk modeling, explore our Azure Virtual Data Studio.  

For technical professionals, register for the virtual workshop on unifying data analytics.