goose drive

• 

  goooose

Inspiration

Google Drive is where real work lives — lecture notes, screenshots, PDFs, notebooks, random zips — and the UI still treats it like a filing cabinet. Folders lie. Filenames lie. Photos have no text. You already know the file exists; you just can’t find it.

We wanted search that felt like walking a map of meaning: similar files sit together, a query lights up a neighborhood, and the answer cites the actual chunks it used. Not another chatbot bolted onto Drive. A place you can roam.

What it does

Sign in with Google (drive.readonly), pick files from /import, then enter /space.

Every file becomes a node in one vector space:

Type What we index Docs / Sheets / Slides / PDF / text / code Extract, cap, chunk, embed Photos + images inside Docs Drive thumbnail + gpt-4o-mini caption Video One thumbnail + duration metadata + optional caption (no Whisper) Jupyter notebooks Flatten markdown + code cells, skip huge outputs Binaries (zip, exe, …) Filename + mime + size + folder only — no byte download Then you:

Walk the corpus in a 3D embedding manifold (clusters, teleport, modality-colored meshes). Search with Elastic BM25 + text-embedding-3-large kNN, fuse with Reciprocal Rank Fusion, then rerank with gpt-4o-mini. Read a cited answer grounded only in retrieved Drive chunks ([Doc 1], [Img 2], …), with follow-up questions. The HUD shows pipeline timings (dense / BM25 / rerank / e2e), rank-delta after rerank, and per-file ingest mode (full / truncated / vision+text / filename-only). Cost caps keep OpenAI spend bounded.

How we built it

Stack: Next.js App Router on Vercel, Auth.js Google OAuth with token refresh, Elasticsearch Cloud (dense_vector 1536, cosine), OpenAI embeddings + vision + synthesis, React Three Fiber for the scene.

Ingest. Client batches one file per request so serverless maxDuration stays safe. A classifier routes by mime/extension. Text is truncated (12k chars default) and chunked (6 chunks/file). Images go through low-detail vision captions. Opaque binaries never download bytes.

Layout. After import we mean-pool chunk embeddings per file, run a 3-axis PCA (power iteration on the Gram matrix) to place nodes in 3D, then k-means and name clusters from folder hints. The scene uses that layout: docs as slabs, images as octahedra, video as cylinders, notebooks as dodecahedra, binaries as wireframe cubes.

Retrieval. Query → embed → Elastic kNN (k=100) and BM25 multi_match on text + filename → RRF (k tied to a theta slider) → LLM rerank to top-5 → cited synthesis. Hits pulse in the manifold; click a node to teleport.

Privacy / cost. Read-only Drive scope, per-user userId filters on every Elastic query, ingest caps (chars, chunks, vision, download bytes) shown in the import UI.

Challenges we ran into

Drive is not a corpus. Docs, photos, .ipynb, and zips cannot share one extractor. Filename-only ingest was the honest fallback so everything still becomes a node. Serverless vs. embeddings. Importing a folder in one request would timeout and blow the OpenAI bill. One-file-per-request + client-side batching + hard caps was the fix. 1536-D → 3D that still looks like meaning. Naive random scatter is pretty and useless. PCA + k-means had to be fast enough to run on-demand after ingest, without a Python sidecar. Hybrid search that you can see. RRF + rerank is usually a black box. We surfaced dense rank, fused rank, rerank score, and rank-delta so judges (and we) could watch a lexical hit get promoted.

Accomplishments that we're proud of

A full multimodal Drive → Elastic → 3D RAG loop in a hackathon weekend, not a mock dataset. Hybrid retrieval that is actually hybrid: BM25 + dense kNN + RRF + rerank, with cited answers. A HUD that treats retrieval like a radar, not a chat box — timings, evidence, teleport, clusters. Cost and timeout discipline so a demo on real Drive files doesn’t melt the API key. What we learned Lexical and dense search fail in different, complementary ways — RRF is not a slogan, it’s how you recover both “exact filename” and “that screenshot of the whiteboard.” Also: if you put embeddings in 3D, people immediately get retrieval. The manifold is the explanation.

What's next

Real UMAP / hierarchical layouts (the UI already has the modes; PCA flattening is the current 2D path). Elastic ELSER sparse retrieval alongside dense + BM25. Streaming ingest and incremental layout so a 10k-file Drive doesn’t wait for a full re-cluster. Optional audio captions for video (still capped). Shareable “rooms” so a team can walk the same Drive space.

What happens when we realized that we ran out of OpenAI API credits

When OpenAI embeddings are available, Drive Spatial uses text-embedding-3-large so every file lands in one 1536-dimensional search space. If the API is out of credits or rate-limited, ingest does not fail: we fall back to a local hashed embedding of the same shape, so files still index into Elasticsearch, appear in the 3D manifold, and can be retrieved with hybrid BM25 + kNN. Vision captions and answer synthesis degrade gracefully too, so the demo keeps running even when the paid model is unavailable.

Built With

• typescript