Inspiration
The physical world contains information web search agents cannot easily access: objects on a desk, text on packaging, a device's position, or details someone knows about an item.
A camera can capture that information, but a photo does not provide reusable object records. We wanted to give agents a structured way to access physical context and connect it to web information.
That became Veyra: an infrastructure layer that turns physical scenes into structured data for web search agents.
What it does
Veyra turns a photo or captured camera frame into a saved scene containing individual object records.
Each record can include a category, colors, visible text, approximate position, attributes, and uncertainty. Users can review object boundaries, correct records, and add known physical properties.
An agent retrieves the objects and fields relevant to a question. When the answer requires external information, Veyra uses that context to guide web research through Telem AI and Tavily.
Answers expose their supporting records, source excerpts, provider activity, tokens, and estimated costs. Follow-up questions use the saved data without sending the image through vision again.
How we built it
We built the interface with React, TypeScript, and Vite, with an Express backend and local JSON persistence.
OpenAI performs the initial scene inventory and checks proposed object regions using enlarged crops. Veyra stores the results as structured records and exposes tools such as list_objects, search_objects, and get_object.
For research, OpenAI interprets the task using selected object fields. Telem AI routes focused queries across search providers and fetches selected pages. Tavily is available through Telem and as a direct retrieval option.
Veyra preserves provider ranks, merges duplicate sources, and applies reciprocal rank fusion before limiting candidates. The agent selects pages to read and explains its choice. Citation checks connect quoted excerpts to retained source text.
Challenges we ran into
Object localization proved harder than object recognition. Models could identify an item while placing its box incorrectly. We added coordinate landmarks, crop-based checks, and manual correction tools.
We also needed to distinguish visible facts from assumptions. Image-relative size does not establish physical dimensions, and a familiar-looking object does not establish its exact brand or model.
Another challenge was merging search results fairly. Keeping the first results could favor one provider, so we preserved rank information and combined candidates before applying limits.
We also worked on making model responses reliable enough to become structured records, while preserving uncertainty instead of filling missing fields with guesses.
Accomplishments that we're proud of
- Built a complete workflow from a physical scene to reusable object data and sourced research.
- Enabled agents to retrieve selected fields instead of receiving the entire scene.
- Integrated Telem auto-routing and page fetching with inspectable ranking and provider provenance.
- Preserved corrections, scene revisions, task history, and usage receipts.
- Added automated tests for retrieval, partial failures, ranking, citation handling, and the research workflow.
What we learned
Useful physical context does not always require an exact identity. “Cat-shaped toy” can be enough to begin a relevant search.
We also learned that reuse has an upfront cost. Initial analysis must be included when evaluating whether structured context improves session economics.
Telem’s routing and normalized results help connect multiple retrieval providers, but the application still needs careful source selection and evidence handling.
Finally, finding an excerpt in a source is different from proving that it supports a claim. Inspectable evidence and human evaluation remain essential.
What's next for Veyra
We’ll focus first on efficiency and performance: reducing processing time, token usage, and cost while improving the quality of object records and answers. We’ll build benchmarks to measure these tradeoffs across detection, retrieval, and grounded answers.
Next, we plan to explore supermarket scenarios, helping agents detect items, structure their attributes, and connect them to relevant web data.
Longer term, we want to extend Veyra to robotics, giving robots a structured interface for understanding physical objects and interacting with web data.
Built With
- node.js
- react
- sharp
- tavily
- telem
- typescript
- zod