Inspiration
I wanted to move past toy ML projects and actually learn how real-time data systems work under load. Crypto markets offer a continuous, high-frequency data stream for free — no waiting for daily closes, no synthetic datasets. That made it the perfect testbed for building a system that has to ingest, process, and act on data as it happens, not in a batch job. CRY BOT started as an excuse to learn streaming architecture and ended up becoming a full predictive trading pipeline.
What it does
CRY BOT ingests live crypto market data and predicts short-term price direction (up/down) using an ensemble of ML models. Based on those predictions, it can either paper trade — simulating trades against live prices to validate strategy performance risk-free — or execute real trades via exchange API when running in live mode. Every tick, prediction, and trade decision is logged and queryable, so the system's behavior is fully auditable rather than a black box.
How I built it
The pipeline is split into three layers. Live market data streams through Kafka, which decouples data producers from consumers and lets the system absorb bursts without dropping ticks. Storage is split between TimescaleDB, used for transactional and recent-window queries, and ClickHouse, which handles heavier analytical queries over historical data like backtesting and feature aggregation. An ensemble ML model combines multiple signals to predict price direction, reducing the risk of any single model overfitting to noise. A decision layer then turns predictions into either simulated or real trade actions.
Challenges I ran into
Getting the Kafka pipeline fast enough that predictions were still relevant by the time they reached the execution layer was a real constraint — streaming systems fail quietly if latency creeps up. Making the ensemble actually predictive was harder still, since crypto price movement is extremely noisy and it's easy to build a model that just fits that noise instead of beating a naive baseline. On the storage side, balancing TimescaleDB's strength for recent, high-write workloads against ClickHouse's strength for large historical scans meant carefully deciding what data lived where, and keeping both in sync without duplicating logic.
What I learned
Building CRY BOT taught me that most of what people call "ML in production" is actually data engineering — the model is a small piece sitting on top of ingestion, storage, and latency decisions that determine whether predictions are even usable in time. I also learned how differently ensemble models behave on noisy financial data compared to clean benchmark datasets, and how much validation discipline that demands.
What's next for CRY BOT
Next steps include expanding to more data sources and trading pairs, adding stronger risk management and position sizing to the execution layer, and running longer live-testing windows to validate performance beyond backtests.
Built With
- clickhouse
- docker
- ensemble
- kafka
- learning
- machine
- postgresql
- python
- rest
- timescaledb
