Router Holmes

Route map
Router temperature
Heatmap of number of requests on router
Heatmap of router temperature
Architectural diagram

Inspiration

While reading customer stories of IoT companies we came across services providing a variety of internet-related services. We realized that such companies must be getting data at a high frequency and we also came across several use cases for such data and thought it would be really cool if we can leverage the features of SingleStore for real-time analytics of the internet data received by such companies.

What it does

Consider a company that provides services through several internet routers in a city. Our application takes in the real-time stream of data received by the routers and also some diagnostic details from the router (like the router temperature) and displays analytical details on a map.

How we built it

Data generation

We created data using a python simulation:

Obtained a polygon mapping of "blocks" in New york city and placed a router at the centroid of each block.
Considered an initial population of citizens
Assumed that each citizen travel from destination A to B (through other blocks) and back to A (with a resting time at A and B), to simulate the average citizen that travels from home to work and back.
Assumed a probability for each citizen to send a request to the router at their current block (this probability is high while resting and lower while travelling)
Assumed the rate of outflow of packets in a router and a default queue size
Assumed that the router temperature increases with queue size.

Data ingestion

Using Apache Kafka, we loaded the data into Confluent cloud, we used the SingleStore pipeline to ingest data from Confluent cloud into the SingleStore tables.

Analytics

Using SWR (stale while revalidate package), we poll the SingleStore database for updates and we display the analytics data in a real-time fashion.

Challenges we ran into

We were new to Kafka and data streaming, hence it took us quite a while to wrap our heads around it.
Since SingleStore offers a wide variety of use cases, it was quite challenging to decide between the several ideas that we had. Eventually, after getting reviews from peers, this idea felt the best :)

Accomplishments that we're proud of

Quality of idea

We feel that our idea matches the use case of SingleStore and also has useful applications. We also came across several studies such as this, which talks about using Wifi data to monitor crowds and other applications. Since this data comes at a high frequency and quantity, we feel that SingleStore is a perfect match for this.

User Interface

We took a rather different approach to our user interface this time and we feel good about how it looks :D (Especially how it changes real-time)

Uniqueness

We feel that the idea of analyzing data from network devices such as routers is pretty unique and is still yet to be implemented on a large scale. The studies that we have read do not talk about existing popular implementations of the same.

Simulation

We feel that our simulation was quite accurate and showcased our scenario perfectly.

What we learned

The world of real-time streaming and real-time analytics was new to us so we learned a ton about how all these work. We were also new to concepts such as OLAP, OLTP and HTAP, and we got better insights into what these mean from this hackathon.

What's next for Router Holmes

With a more sophisticated simulation, taking more factors into account, we could come up with more insightful analytical data. It would also be really great to model the application of our system on upcoming smart cities using real data (either from experiments or from actual companies).