Social Nutrition Label

Architecture Diagram
Flow Chart
Nutrition Check on Fake Post
Nutrition Check on Factual Post
Database Dashboard

Inspiration

The internet is drowning in noise, and for our team of international students, this problem is deeply personal. During spikes in global political tension, social media became saturated with fear-driven and unverified stories. The anxiety was real enough that people close to us changed plans because they could not tell what was true and what was amplified panic. That constant uncertainty creates mental fatigue. Feeds reward outrage and velocity, not context. We wanted to build something that reduces that pressure in the moment. Instead of hiding posts through blunt filtering, we focused on helping people interpret what they are seeing with clearer context and grounded signals.

What it does

Social Nutrition Label is a Chrome extension workflow that lets a user analyze a questionable post in one click. It extracts the visible post text and image, sends that payload through our analysis pipeline, and returns a clean nutrition-label style result with confidence, summary, category scores, and source grounding when available. The goal is not censorship. The goal is to give users fast, structured context so they can make better decisions before reacting, sharing, or doomscrolling deeper.

How we built it

We built this as an end-to-end asynchronous pipeline. The extension layer handles extraction and submission. content.js captures the active post’s text and image URL, popup.js triggers the flow, and background.js sends the job to Supabase. Supabase acts as the queue bridge using social_posts. New rows enter as pending, then a Node worker claims them, runs analysis, and writes back completed results or errors. The worker uses Gemini multimodal analysis with Google Search grounding through Gemini’s tool capability, then normalizes and validates the output against a strict JSON schema before persisting it. We also added a post-level cache. A normalized post hash is generated from caption and image URL, and repeated scans can reuse prior analysis from post_analysis_cache instead of calling Gemini again. That gave us both speed improvements and lower repeated inference cost.

Challenges we ran into

One major challenge was reliability across the full chain, not just model quality. We had to enforce strict JSON behavior, handle schema validation failures safely, and prevent jobs from getting stuck in processing when downstream calls timed out or failed. We also hit practical data-layer issues around schema cache refresh, migration order, and permissions. Those problems were subtle because parts of the flow could still work while specific columns or cache writes failed. Debugging and hardening those operational edges became a big part of the build.

Accomplishments that we're proud of

We are proud that the multimodal path works in real usage: extension to queue to worker to Gemini to structured result. We are also proud that we did not stop at a demo-only happy path. We added validation, queue-state handling, and repeat-scan caching so the system behaves more like a real product than a single script. Most importantly, we turned a stressful user experience into something calmer and more actionable. Even when certainty is limited, the user gets a structured explanation instead of pure noise.

What we learned

We learned that multimodal analysis is significantly more useful than text-only checks for misinformation-style content, especially when visual context and caption context conflict. We also learned that grounding quality should directly shape confidence behavior, because unsupported certainty is one of the fastest ways to lose trust. From an engineering perspective, we learned how much resilience matters in asynchronous AI systems. Queue discipline, schema contracts, and operational observability ended up being just as important as prompt quality.

What's next for Social Nutrition Label

Our next step is to improve confidence calibration so results feel less flat when grounding is weak while still remaining honest about uncertainty. We also want to improve duplicate detection further and expose cache-hit status directly in the extension UI so users can see when a verdict was reused instantly. Longer term, we want to expand beyond static post checks toward richer media literacy support, including better handling of evolving narratives and features that encourage healthier, more constructive engagement patterns.