This hack is implementing a strategy of arbitrage in futures markets. There are plenty of opportunities to take advantage of and it refers to a trading strategy where the profits are made off of temporary market inefficiencies that result in disparate pricing of investment assets across different retail brokers or in some cases just because some brokers are making temporary heading to play against their clients or just because they want to make more profits and match the orders with theirs clients instead of sending out.
What it does
We created a library with high speed and low latency to stream live market data using ZeroMQ What the library does basically is streaming ticks and order book changes to our platform, once a new data arrives it is automatically processed using kafka and spark streaming to create a pipeline or this data just join to an existing pipeline, this pipeline is analyzed in real time to find opportunities, when we find one, we place buy or sell orders in the market.
How I built it
The project is composed of two parts:
- Library: to connect to the brokers and streaming the data and also placing the orders. It was written using C++ and ZMQ. ( I made it open source, so, you can get this code on github)
- Server: the server was written in scala and spark to achieve maximum performance and analyze the data as fast as possible.
Challenges I ran into
This is not only about the speed but also is about the processing and analyzing a ton of data in real time, so it means every line of your code matters, every nano second matters, trying to be faster and keeping the performance is a huge challenge.
Accomplishments that I'm proud of
- We are processing and analyzing around 82 million bid, asks and traders in an average of 5 minutes
- In our streaming pipeline we are saving offline not only bid, ask and volume, but also every change on the orders book, later we use this data for training our models, this info even if you are willing to buy, if can't find out there.
- We are able to push real time data for zeroMQ, rabbitMQ, PubNub and Kafka to third party software's.
What I learned
Speed matters only if you know how to do with the data.
What's next for hitback.us
We are also really excited about the opportunities on the emerging markers and cryptocurrencies.