Inspiration
這一欄不是單純放功能介紹，而是要寫成一篇完整的 Project Story，至少包含：
靈感來源 → 解決的問題 → 如何打造 → 遇到的挑戰 → 學到什麼 → 最終成果。
下面這版可以直接貼進 Devpost 的 About the project 欄位。內容依照 FORME 目前實際功能撰寫，也清楚區分 GPT-5.6 與 Codex 的角色。
FORME — A Personal Visual Memory for the Objects That Shape Your Life
Inspiration
FORME began with a simple observation: people discover meaningful objects everywhere, but rarely have a meaningful place to keep them.
A chair in a café, a lamp in a film, a camera in a magazine, or a bag in a screenshot may stay in our minds for days. Yet the object itself usually disappears into a camera roll, a browser tab, a note, or an unorganized shopping wish list.
Existing visual-search tools are useful for finding visually similar products, but they often blur two very different questions:
- What is this exact object?
- What other objects might this lead me toward?
That distinction became the foundation of FORME.
We wanted to build a private visual memory that could help users identify an object when reliable evidence exists, remain honest when it does not, and preserve the discovery as part of a longer-term portrait of personal taste.
What it does
FORME is an iOS app that turns a photo, screenshot, or text description into a structured and searchable object memory.
Users can import an image, select the object they care about, and receive grounded information such as its category, color, material, visible attributes, and a focused product-search query.
The results are then divided into two separate experiences.
Find This Item
Find This Item focuses on product identity.
A result may be classified as:
- Exact Item
- Likely This Item
- Exact Item Not Confirmed
FORME does not treat visual similarity as proof. Exact identity requires supporting evidence such as official product pages, model numbers, SKUs, ISBNs, catalog numbers, manufacturer data, or canonical URLs.
Explore More
Explore More focuses on discovery rather than identity.
It may show:
- the same designer
- the same product series
- a similar form
- an adjacent taste direction
These results remain clearly separated from the identity answer, so an alternative is never presented as the original object.
Saving and personal memory
A result can be saved as a real Find, including its image, grounded title, source, search context, match status, and collection membership.
Saved Finds persist after the app is relaunched. Over time, they become more than isolated search results—they form an evolving personal archive of the objects the user notices and values.
The Starting Edition
A new user does not yet have any saved Finds, so FORME begins with a short visual onboarding experience.
Users select a few life worlds and editorial seed objects across:
- Wear
- Space
- Objects
- Sound
- Books
FORME then creates a personalized Starting Edition.
The Starting Edition is not populated with fake products or artificial collection entries. Its silhouettes act only as editorial starting points for exploration.
When a user saves an eligible real Find, FORME can generate a local editorial representation of that object and use it to replace the matching seed.
This creates a visible transition:
from a world based on selected interests to a world increasingly shaped by objects the user has genuinely saved.
How we built it
FORME is built as an iOS application using SwiftUI and a reusable Swift core package, supported by a Python backend.
The main runtime flow is:
Photo, screenshot, or text
→ object selection or intent extraction
→ foreground preparation
→ multimodal object understanding
→ structured object metadata
→ product discovery
→ category filtering and deduplication
→ canonical source-evidence evaluation
→ Find This Item and Explore More
→ Save to FORME
→ Collection and Product Detail
→ persistent personal taste memory
For image-first Capture, the user-selected object is prepared through a replaceable foreground pipeline.
The current architecture supports:
- a SAM 2.1-ready remote provider
- Apple Vision foreground extraction
- a selected-crop fallback
The fallback is important because it guarantees that the user can continue even when automatic segmentation is unavailable or imperfect.
The backend handles product discovery, source-page analysis, image resolution, ranking, deduplication, identity evidence, retries, and cost tracing.
Different object categories also require different forms of identity evidence.
For example:
- cameras may depend on brand and model
- books may depend on title, publisher, edition, and ISBN
- vinyl may depend on artist, album, label, and catalog number
- furniture may depend on designer, manufacturer, series, and model
How we used GPT-5.6
GPT-5.6 acts as FORME’s grounded visual-understanding and intent-structuring layer.
For image-first Capture, the model analyzes the user-selected object rather than making assumptions based on the full scene.
It produces structured information such as:
- a grounded display title
- category
- color
- material
- visible attributes
- a product-search query
The model is explicitly instructed not to invent a brand, model, SKU, or exact product identity based on appearance alone.
GPT-5.6 also helps distinguish between general descriptions and explicit identity constraints. For example, it can recognize the difference between:
black sculptural floor lamp
and:
Isamu Noguchi Akari 10A
However, GPT-5.6 is not the final authority for exact identity.
The final Exact or Likely decision is made using real source evidence. This separation allows FORME to benefit from flexible multimodal understanding while keeping identity claims conservative and trustworthy.
How we used Codex
We used Codex as an engineering collaborator throughout development, not as a runtime feature presented to users.
Instead of asking Codex to generate the entire application in one pass, we divided the project into isolated milestones with:
- clear boundaries
- frozen systems
- explicit acceptance criteria
- regression tests
- required runtime evidence
Codex helped us:
- inspect the existing Swift and Python architecture
- implement focused changes
- create and update regression tests
- diagnose simulator and backend connectivity problems
- validate persistence after relaunch
- improve search-result ranking
- separate identity results from exploratory results
- protect against duplicate or fabricated Finds
- validate onboarding and Starting Edition behavior
- preserve local image references across application-container changes
This process allowed us to move quickly without repeatedly destabilizing previously accepted parts of the application.
Challenges we faced
1. Separating recognition from certainty
The most important challenge was preventing a visually similar result from being presented as an exact product.
Multimodal models are very effective at describing appearance, but appearance alone is not reliable evidence of product identity.
We solved this by separating the system into two layers:
Visual understanding
≠
Verified identity
The model structures what it sees, while a separate evidence system evaluates product sources and determines whether the result is Exact, Likely, or Not Confirmed.
2. Searching across very different categories
A chair, a camera, a book, and a vinyl record cannot be verified using the same identity rules.
We had to design category-aware evidence rather than relying on one universal matching score.
This made the architecture more complex, but it also made the results more credible.
3. Building a reliable image-selection flow
Foreground extraction is not perfect, especially when an object overlaps with furniture, people, shadows, or a visually complex background.
To make Capture resilient, we designed a replaceable preparation pipeline and kept the user-selected crop as a guaranteed fallback.
The principle was:
The automatic result may improve the experience, but it should never prevent the user from continuing.
4. Persistence and local image references
Saved images initially depended too heavily on temporary application-container paths.
Those paths can change after relaunch or rebuild, causing saved objects to lose their images.
We introduced portable local references and validated that saved Finds remained accessible after the app relaunched.
5. Creating personalization without fake data
The Starting Edition needed to feel personal before a user had saved anything.
However, we did not want to populate the account with fake Finds or pretend the user had already collected real products.
The solution was to use deterministic editorial seeds that are visibly different from real saved Finds. As the user saves genuine objects, those seeds can gradually be replaced.
What we learned
The biggest lesson was that AI product experiences need to communicate uncertainty as carefully as they communicate results.
A confident but unsupported answer may feel impressive for a few seconds, but it damages long-term trust.
We learned that a better system does not always provide a definitive answer. Sometimes the most useful answer is:
“This is visually similar, but the exact item cannot yet be confirmed.”
We also learned that personalization should be based on real user actions rather than artificial profiles. A taste system becomes more meaningful when it grows from what a person actually selects, saves, rejects, revisits, and organizes.
From an engineering perspective, we learned that AI-assisted development works best when tasks have narrow boundaries, explicit tests, and verifiable outputs. Codex was most valuable when working inside a clearly defined architecture rather than being asked to redesign everything at once.
What makes FORME different
FORME brings together four experiences that are usually separated:
- visual object capture
- identity-first product search
- long-term personal collection
- taste-based exploration across an entire lifestyle
FORME does not treat every object as a shopping transaction.
It treats objects as parts of a person’s evolving visual and cultural world.
The core principle of FORME is:
Find the item when evidence exists. Be honest when it does not. Then help the user explore what the object means for their broader taste.
Log in or sign up for Devpost to join the conversation.