• 

  memex architecture diagram

memex (Taskmaster track)

Inspiration

This is a multi-modal, multi-platform capture and ADK agent-assisted PKM system, built for scatterbrained humans, starting with the one who wrote it.

Ideas and thoughts come and go in a busy mind, before having the chance to write them down. And so do Chrome tabs accumulate crowding out links I still needed, then lost when I inevitably declare Chrome tab bankruptcy. Or the folder of screenshots taken with the intention making use of a graphic or data. And the "I should look into that" thoughts that pop into consciousness throughout the day- driving, in the shower, etc.

Any capture app that makes you type, title, tag, and file has a bar that's too high. By the time the form is filled, the thought is gone.

The need is effortless capture, anywhere, into a trustworthy system that proactively simplifies, centralizes, and streamlines the regular noise from daily life.

What it does

memex is a personal capture agent with one inbox and (currently) seven ways into it: a web UI, native Android and iOS apps, an iOS/macOS Shortcut to trigger the 'Action Button', desktop/laptop hotkey triggers, and Chrome extensions that pushes annotated screenshots and Chrome browser tabs that are intended for later follow-up. All are working today, and can be easily adapted to meet other needs. Within seconds, each capture is a searchable note with a summary, tags, and any to-dos it contained. Gemini processes audio and images (screenshots) natively, so there is no separate speech-to-text or OCR.

Routines. Twice a day Cloud Scheduler fires real agent sessions: a morning task review for staleness and duplicates, a nightly digest of themes and open questions. Every run stores its complete tool-call trace, replayable in the UI.

Approvals. Agents can't edit your notes/tasks, just queue 'proposals'- e.g. "these five look like duplicates, drop them?", to which you accept or reject. Enforced by the toolset, not by a prompt- the routine agent has no task-mutation tool to call.

Chat. A sidebar chat runs over the same corpus and streams interaction events, one event per model turn, tool call, and tool result. Chat mutates directly rather than queuing proposals, because your live instruction in the conversation is the approval (and every mutation still lands in the stream as a trace event).

Deep Research. Flag a capture for research and memex starts a Gemini Deep Research job, asynchronously. A job ID is maintained in Firestore so compute resources don't need to sit idle while a report is compiled. Later, when the report is completed, the cited report appears in your feed as a note, linked to note that initiated the research.

How we built it

Memex is "serverless"- it runs what it needs to in Cloud Run, and scales "down to zero" when it's idle.

Cloud Run serves a FastAPI API, the agent (Google ADK + Gemini on Vertex AI), and a React SPA.

Firestore is the system of record across seven ULID-keyed collections; GCS buckets hold raw content (audio and screenshots).

Every path in is event-driven. Text and links enrich synchronously in-request. Audio and images return 202, land in GCS, and the object-finalized event drives enrichment through Eventarc into an OIDC-verified internal endpoint, about ten seconds from speaking to a note in the feed. Cloud Scheduler drives the two routines through the same gate.

Deep Research runs use the Long Running Operation (LRO) pattern: they are durable in Firestore. Cloud Tasks polls for completion.

Analysis runs on Gemini 3.7 Flash, transcription-heavy audio enrichment on Gemini 3.5 Flash-Lite. Clients all authenticate with per-device bearer keys from one Secret Manager secret.

All infrastructure is deployed via Terraform to your project, so you all the resources are fully within your control (never sent to other parties).

Challenges we ran into

At-least-once delivery makes "poll a job" hard. Two deliveries of the same poll task could both see a finished research run and both write a report note. The fix was to stop treating the operation document as a record and start treating it as a lock: every state change is a compare-and-set on status, and the completion path reserves the result note's ULID on the operation before writing the note. Two racing deliveries then write the same document id, and a crash between the note write and the settle replays onto that same id instead of adding a second report.

Real security risk: ensure content is not interpreted as instructions! Three separate channels, each closed differently:

• Research used to trigger on a research tag, which the enrichment model inferred from the text it was reading. A page could therefore drive up surprise bills. It now needs an explicit boolean flag set by the client's own client; content the model reads can no longer authorize a paid run.
• A screenshot's page title is text somebody else wrote, and it arrived in the same prompt as the user's caption. A newline in a title was enough to pose as the next field, the one field the prompt lets ask for a task. Every metadata field is now flattened to one line, and page-supplied fields are ordered before the user's own.
• A link saved with no note of your own now produces no tasks at all, because nothing in a bare URL and a site-chosen title is you asking for anything.

Accomplishments that we're proud of

I actually use this daily because it delivers on the 'effortless capture' goal.

Deliberate AI policies - human-in-the-loop for changes, and ensure auditability.

Agent chat and Deep Research features add significant value to daily workflows.

Broad client integrations: 6 added so that 'effortless capture' applies in a variety of different contexts.

What we learned

A prompt saying "treat this as data, not instructions" is a request; a toolset that contains no mutation tool is a boundary.

Structured output turns "parse the model's answer" into a schema contract.

What's next

Extend the chat tool policy (the risk classification layer is implemented and tested, the ADK confirmation flow is designed and proved, while some policies need to be written).

Inspiration

All code is new for this hackathon, with some inspiration from:

• google/adk-samples: The routine mechanism (scheduler → push endpoint → agent session with tools → stored trace) and the human-in-the-loop approval queue follow the long-horizon-harness pattern from google/adk-samples.
• allenhutchison/obsidian-gemini: The chat tool policy in docs/chat-tool-policy.md was inspired by the CAUTIOUS preset (approve reads, ask for writes/destructive/outbound calls) in obsidian-gemini by Allen Hutchison (src/types/tool-policy.ts).

Built With

• adk
• cloud-run
• cloud-scheduler
• cloud-tasks
• eventarc
• fastapi
• firestore
• gcs
• gemini
• kotlin
• react
• secret-manager
• swift
• terraform