Parrots

• 

  Parrots Configurations

Inspiration

Migrating data from a data store to another is not an easy task specially without downtime. One of the solution to migrate from a data store to anther is to stream the changes from the source database to the destination database in near real-time.

One other use case is having a near real-time analytics from production databases. For example, replicating the production data stored on databases to the data warehouse. There are various tools that helps in replicating data and one of those is Parrots.

What it does

Parrots is a scalable, configurable, lightweight near real-time data replicator CLI (command line interface) that replicates data from different database engines as a source to any database engine as a destination

How we built it

Using Python, MySQL Binlog reader Parrots was built as a proof of concept. It could be extended to have any other source such as Cassandra Commit Logs, PostgreSQL logs, ... etc and SingleStore or any other database engine as a destination.

Challenges we ran into

Handling different database engine log streaming and parsing and the different syntax of the destination databases.

Accomplishments that we're proud of

Having a running proof of concept that is up and running and capable of replicating data from MySQL to SingleStore.

What we learned

MySQL Binlog parsing and events handling as well as SingleStore basic features.

What's next for Parrots

1. Adding more sources and destinations
2. Probably open sourcing

Built With

• mysql-binlogs
• python
• singlestore