Clippy

The Clippy device.
All device internals.
Clippy's app

Inspiration

What was I doing last Thursday at 4pm?

I can never remember the small details when they actually matter, and it drives me crazy.

What it does

Clippy is a small camera that clips onto your clothes and takes photos throughout the day. Those photos get turned into a journal entry for your day, which you can read in the Clippy iPhone app. Forgot what you did last Sunday night? Clippy wrote it down.

Daily recap: a short summary of your day, a timeline of moments, and a question to reflect on.
You can edit anything: rename a moment, write a reflection, or delete a photo. Your changes always override the AI's.
Clarifying questions: if a photo is too unclear, Clippy asks you what was going on. You get three guesses you can tap, or you can type your own answer. The recap updates about a minute later.
Works offline: you can still read your journal without a connection, and your edits sync once you're back online.

How we built it

There were two of us, so we split the work and connected everything through one shared contract.

Hardware: a Raspberry Pi Zero with a camera and a motion sensor (IMU). Python on the device captures photos and uploads them whenever it has a connection.
Contract first: one JSON Schema defines every message between the device, the cloud and the app. We generate the TypeScript and Swift models from it, and the same test fixtures run in Python, TypeScript and Swift, so nothing gets out of sync.
Backend on AWS, written in TypeScript with AWS CDK:
- Cognito for sign-in, and API Gateway with Lambda for the API.
- S3 for photos. The device uploads straight to S3 using presigned URLs, and the server checks each file's hash.
- IoT Core (MQTT) for messages from the device.
- DynamoDB for journals and devices, with version checks so two edits can't overwrite each other.
- Step Functions to run the recap pipeline.
- Bedrock running Claude, which describes each photo and then writes the recap for the day.
- Location Service for place names.
iPhone app: SwiftUI with a simple black-and-white design. It checks every server response against the shared schema.

The pipeline: each photo gets described once and the description is cached. Then one call writes the whole day. The model only writes text. The server builds the moments and links each one to its photo, so the AI can't reference a photo it never saw.

Challenges we ran into

Working in parallel: one of us was on the app and the other on the backend at the same time. Agreeing on the contract first, with fixtures every language had to pass, kept us from stepping on each other.
Bad connections: the device goes offline a lot, so every upload and edit has to be safe to retry. Uploads can't overwrite an existing photo, and every write has a key so retries don't create duplicates.
Prompt injection through the camera: what if you take a photo of a sign that says "ignore previous instructions"? We treat any text in a photo as data only, and we filter the suggested answers so text from a photo can never show up as something you tap.
Privacy: a camera you wear all day is personal. Questions never ask about people, health or relationships. Place names are rough and off by default, and deleting a photo also clears the recap that used it.
Cost and speed: after you answer a question, Clippy waits 60 seconds before rewriting the recap. If you answer n questions in that window, it does 1 rewrite instead of n.
Blurry photos: our first prompts just threw them out. Now a blurry photo still becomes a moment, and Clippy asks you about it.

Accomplishments that we're proud of

It works end to end. A photo on the device turns into a recap on your phone, and a full local test run plus iPhone simulator tests check that automatically.
The whole backend runs on AWS and is defined in code (woop woop IaC), with 31 automated infrastructure checks.
The clarifying questions, where the AI asks and you answer with one tap.
The hardware was difficult to make in such a small time period, including so many sensors on a small enclosure.