On Demand Data Sharding

Inspiration

Data sharding becomes a crucial solution in managing and processing the ever-increasing volume of data. By dividing data into distributed shards, companies can achieve the scalability necessary for their business growth. The ODS (On-demand Data Sharding) platform aims to provide users with an easy-to-use tool for partitioning and storing data in TiDB databases, as well as efficiently running data queries and obtaining accurate results from the database. ODS offers a flexible and effective solution for users seeking to optimize data management and access in a dynamic business environment.

What it does

ODS (On-demand Data Sharding) is a platform designed to facilitate users in data sharding or partitioning to multiple databases, such as TiDB. With ODS, users can store data on several shards distributed within the TiDB cluster. Additionally, ODS enables users to run data queries easily and obtain query results. It provides the flexibility to manage and access data in a dynamic business environment.

How we built it

ODS built in front-end also back-end as a different component. In the front-end side we built it using Vanilla JavaScript also include html and css technology. In the back-end side it was built using Go language by Fiber framework, using Mongodb, Redis, TiDB as the databases to contain sharding results. We uses ChatGPT API also so it can generate query languages and embedded it on the sharding databases for user to query through using natural languages.

Challenges we ran into

In the development of ODS, there has been some challenges that we came through. Implement sharding technique into logic, also send the shard results from request to the services sharding databases. It cause by, to use the cloud services of databases like TiDB, MongoDB, also Redis need to open the port using the right credential or configuration. It kind of challenging also to embedded the ChatGPT API to databases, because it our first experience working with it.

Accomplishments that we're proud of

After completing the ODS project, we were proud because our team successfully created services that might be can grow much larger than before because of its useful, the services feature alone is rarely seen in other company offers.

What we learned

Data sharding is a way to split a large database into smaller pieces. This can be useful for managing large databases and improving performance. This topic is a complex and challenging problem, but we know it will going to be a big project in the future.

What's next for On Demand Data Sharding

In the future, ODS (On-demand Data Sharding) will evolve by enhancing scalability, database integration, query optimization, real-time data processing, data security, cloud-native and multi-cloud support, improved user interface, analytics, machine learning integration, and community collaboration. ODS will become a crucial tool for efficiently managing and processing data with flexibility and scalability.

These advancements will enable ODS to efficiently manage and process large volumes of data, provide solutions that cater to diverse user needs, and become a valuable tool for businesses seeking flexible and scalable data management solutions.