Inspiration
Financial markets are highly volatile, and sudden price movements can happen with little warning. We wanted to explore how machine learning could help identify potential market disruptions before they happen.
Inspired by the challenge of anticipating unexpected movements in the foreign exchange (FX) market, we developed Storm Radar — a machine learning-powered system designed to detect potential market storms and alert traders to unusual market conditions.
What it does
Storm Radar features a Live Replay function that allows users to replay any trading day using SGX FX one-minute historical data across three currency pairs: EUR/USD, USD/JPY, and USD/SGD.
The system recreates the market as it unfolded in real time, ensuring that the model only has access to information that would have been available at each specific minute. This prevents future data from influencing past predictions and makes the simulation more realistic.
When the model detects a potential market storm, it generates an alert. Each alert is subsequently evaluated and classified into one of two outcomes:
- Storm Followed: A significant market movement occurred after the alert.
- False Alarm: The expected significant movement did not materialise. This allows us to evaluate the model's ability to distinguish genuine market disruptions from normal market fluctuations.
How we built it
We developed Storm Radar using Python, setting up a virtual environment and installing the necessary libraries for data processing, analysis, and machine learning.
Our development process involved several key stages:
Data Collection and Preparation: We loaded, cleaned, organised, and processed the SGX FX one-minute historical data for EUR/USD, USD/JPY, and USD/SGD.
Train-Test Split: We separated the historical data into two periods: July was used as the training dataset, while August was reserved for testing.
Model Training: We trained our machine learning model using July's data to identify patterns and relationships associated with significant market movements.
Incorporating Scheduled Events: To improve the model's ability to anticipate major market movements, we collected scheduled economic events from July and August. These events were incorporated as additional information to help the model recognise periods when market volatility was more likely.
Live Replay and Evaluation: We developed a replay mechanism that simulates market conditions minute by minute. The model generates predictions using only information available at that moment, and its alerts are evaluated against subsequent price movements.
Challenges we ran into
Our biggest challenge was figuring out how to predict sudden and unexpected market movements.
Unlike ordinary price fluctuations, sudden market disruptions can be triggered by unexpected announcements, geopolitical developments, breaking news, or unforeseen economic events. These are difficult to predict because they may not have any identifiable warning signs in historical price data.
We initially addressed this challenge by incorporating scheduled economic events into our model. While this helped the model identify periods of potentially increased market volatility, we realised that scheduled events are not the same as genuinely unexpected events.
The biggest challenge remains finding a way to detect the warning signs of an unpredictable market storm before it happens.
Accomplishments that we're proud of
One of our biggest accomplishments was successfully incorporating scheduled economic events into our machine learning pipeline.
Instead of relying solely on historical price movements, we managed to introduce external information that could influence market behaviour. This gave our model additional context when assessing the likelihood of significant price movements.
We are also proud to have developed the Live Replay feature, which allows us to test the model against historical market conditions in a realistic, minute-by-minute simulation.
What we learned
Throughout this project, we gained practical experience in applying machine learning to financial markets.
Some of our key takeaways include:
- How to prepare, clean, and process financial time-series data.
- How to train and evaluate machine learning models using separate training and testing datasets.
- How to incorporate external information, such as scheduled economic events, into predictive models.
- The importance of preventing look-ahead bias when simulating real-time predictions.
- Why predicting financial market volatility is significantly more challenging than identifying historical patterns. Most importantly, we learned that detecting patterns in historical data is not the same as predicting unexpected events in the real world.
What's next for Storm Rader
Our next goal is to improve Storm Radar's ability to anticipate sudden and unexpected market movements.
We aim to explore additional data sources, such as breaking financial news, geopolitical developments, and real-time market anomalies, to give the model greater awareness of events beyond the economic calendar.
We also hope to improve its ability to distinguish between normal market volatility and the early warning signs of an approaching market storm.
Ultimately, we want to move beyond predicting when volatility is expected and develop a system that can recognise when something unusual is happening in the market — before the storm hits.
