Echo

A carbon-aware intelligence layer that moves LLM chat to the edge. Instead of hitting the cloud for every query, Echo uses a Shared Vector Cache in the browser. Zero latency and zero carbon.

Comment

Internal Track:

  • Sustainability

Sponsor Tracks:

  • Redbull Basement - Most Likely to be a Startup
  • Peraton - Best UI/UX

MLH Tracks:

  • Best Use of Google Gemini API

Echo: The Carbon-Aware Hive Mind

Inspiration

Every time a user types "Hello" or asks a common question like "How do I center a div?", massive GPU clusters in data centers burn kilowatt-hours of energy to generate the exact same answer they generated five seconds ago. We realized that while LLMs are brilliant creators, they are terribly inefficient storage devices.

Why burn a forest to answer a question we already know?

I wanted to build a system that treats intelligence as a shared, sustainable resource. A green hive mind if you will. It gets faster, cheaper, and cleaner the more people use it.

What it does

Echo is a carbon-aware conversational AI that moves the heavy lifting of understanding to the edge.

Instead of sending every token to a cloud GPU, Echo uses the user's browser to generate vector embeddings. It then checks a distributed semantic cache, which is synchronized via AWS S3.

  • If the community has asked it before: The answer is returned instantly from memory.
  • If it's new: We call Google Gemini, generate the knowledge, and then save it to the cache storage as a new vector, for everyone else.

How I built it

1. Frontend

I built the frontend using SvelteKit, Tailwind, and borrowed couple of Shadcn UI components. Crucially, inference implemented using @xenova/transformers.

  • We run the all-MiniLM-L6-v2 model directly in the user's browser.
  • This converts user text into a 384-dimensional vector locally, meaning we don't even need to wake up the server to understand the user's intent.

2. Backend

I chose golang for its raw speed, lightweight and concurrency primitives. The backend manages an in-memory MockVectorDB protected by sync.RWMutex for thread safety.

When a query arrives, it performs a cosine similarity search against our knowledge base. The similarity score between the user's query vector and a cached vector is calculated here. If similarity score is >0.9, we consider it a semantic match and serve the cached response.

3. AWS S3

To make the system collaborative, I chose AWS S3 as a global synchronization layer.

  • On startup and periodic intervals, the server downloads the latest cache.json dump from S3.
  • When new response is generated by Gemini, it is batched and uploaded to S3 every 300 seconds.
  • This architecture allows multiple instances of Echo to share knowledge asynchronously without the overhead of a heavy database cluster.

Challenges

  • Initially, in-memory cache was empty every time we restarted the server. Implementing the cache sync with S3 was tricky. I had to handle race conditions where the backend tries to read from S3 while simultaneously writing new cache entries. I solved this using Go's channels and batched uploads.
  • Implementing raw matrix math in Go was a fun challenge. I had to ensure the dot product calculations were optimized to keep latency under 5ms, even as the cache grew.

Accomplishments that I am proud of

  • I built a mathematical model to estimate energy savings that resulted from applying caching method to chat apps.

  • Successfully ran a transformer model in the browser. It proves that the future of AI software is hybrid: +cloud.

  • The difference between waiting 5 seconds for Gemini and 50ms for the cache is significant. It transforms the chat experience.

What I learned

  • I learned that intelligence doesn't always mean generation. Sometimes, intelligence is just memory.
  • The transition from Java/.NET to Go gave me a massive performance boost, especially for the vector search loops. Such a lightweight too.
  • Green software isn't about planting trees; it's about writing efficient code that prevents hardware from waking up unnecessarily.

What's next

The current version is obviously an MVP, but the architecture is ready to scale.

  1. Replacing the in-memory JSON slice with a proper Vector Database (PostgreSQL + pgvector) for scale.
  2. Currently, our similarity threshold is fixed at 0.90. We want to use AI to dynamically adjust this based on the complexity of the topic.
  3. Adding a session layer so the cache can be distinguished between public and private context.
  4. Privacy. User should be able to choose if they want to share the response to cache it, or not. If user forgot to opt out, a method to recognize personal information and scrap the cache must be implemented.

Built With

Share this project:

Updates

Submission history