Reactive Analytics Platfrom

• 

  Backend System Architecture

Inspiration

I've heard about Quarkus maybe a year ago but never had enough free time to explore it more. I was on vacation when I saw information about this hackathon and I finally had some time to invest to learn more about Quarkus. Generally, I'm interested in microservices, data-intensive systems, and reactive programming, so I got the idea for Reactive Analytics Platform that will fuse all those stuff that I'm interested in.

What it does

Reactive Analytics Platform is a system that tracks user check-ins from specific locations and provides real-time statistics backed by Java, SmallRye, Quarkus, Postgres, Kafka, Kafka-Streams, and RocksDB following reactive principles.

How I built it

The platform is a system based on a distributed microservices architecture. I wanted to incorporate as many as possible real-world patterns common in a microservices architecture like:

1. When and how to use Kafka, Kafka Streams, and Kafka State Stores.
2. How to communicate with other microservices via HTTP in a resilient way.
3. How to compose multiple asynchronous computations and in general how to implement service in a reactive manner from end to end.

The platform consists of multiple microservices:

1. Event Generator is used to produce random events that represent user data. More details here.
2. Ingestion Service is responsible for ingestion and aggregation of user data. More details here.
3. Location Service is a service that contains details about locations. More details here.
4. Query Service is used to query aggregated data and location details. More details here.
5. Dashboad UI can be used to retrieve data from Query Service. More details here

Challenges I ran into

I've never used reactive RDBMS client so I think that will be the biggest challenge, but official documentation is great so I haven't had any major issues. The biggest challenge for me was to build the frontend application with React.js since I'm not a frontend developer and had only one day to learn basics of React.js and implement a solution because I found out about this hackathon pretty late.

Accomplishments that I'm proud of

I've learned the basics of React.js and build frontend application in just a few hours because I needed it for my dashboards app. I'm most proud of the fact that all microservices that I've implemented are reactive from end to end.

What I learned

I've learned a new library for reactive programming - SmallRye-Mutiny and a lot of Quarkus plugins that take advantage of this library. As someone who comes from the Spring ecosystem and no Quarkus experience at all before this hackathon, I must admit that I'm amazed by startup time, how easy is to incorporate reactive programming into Quarkus applications, and generally the huge amount of Qurakus plugins. Working in quarkified environment was pretty productive and fact that how easy is to implement asynchronous HTTP APIs with it is pretty amazing. The most pleasant experience for me, as a every-day Kafka user, is Quarkus level of integration with Kafka and Kafka Streams.

What's next for Reactive Analytics Platfrom

The next step is to apply the outbox pattern on the location-service so every Location update will be published to Kafka topic which will be consumed by query-service and stored in a query-service database. Another improvement would be to implement Kafka Consumer in query-service that will consume check-in-aggregation topic directly and store those data on some distributed database like Cassandra. By doing this, I will completely remove the need for HTTP calls that are used in query-service to fetch data from ingestion and location service, which will make services loosely coupled, more fault-tolerant, and more performant since there is to need to contact external services to retrieve all data that is needed. Besides this, I plan to secure microservices and improve UI of web-ui service.

Built With

• java
• kafka
• kafka-streams
• postgresql
• quarkus
• rocksdb
• smallrye