Kaf-Equa

Inspiration

We are working on Realtime Analytics for Essence which has a event framework which allows us to read realtime data. We wanted to enable Realtime Analytics for another banking product Equation which is hosted on AS/400 or i-series machine.

What it does

This project allows us to generate events based on database updates done by Equation

How I built it

I have used Kafka Connect jdbc source connect framework to connect to AS/400 database. Then I configured it to connect to one of the table and generate avro messages from the table and publish it to Kafka.

Challenges I ran into

Kafka connect works on Linux machine. So I had to setup docker on my windows and configure it work with hyperv. I had to figure out the right drivers and an AS/400 environment that would connect. We also had to get the right jars in place and this link was helpful. Most of our AS/400 environments are Manila based and most of the time we got timeout. For the demo purpose, we also tested it out against Essence Oracle DB which would connect seamlessly.

Accomplishments that I'm proud of

I was able to generate events out of Equation using multiple mode (bulk, incremental).

What I learned

I learned a lot about confluent's kafka connect architecture and how it can be easily used to generate events without touching any of the Equation or Essence source code.