Main Page (View via laptop)
Room Recorded
Scan Room Page
Sample Living Room
Menu in Each Room
Menu for Remi (Powered by Muse AI)
Interconnected Users
Footer
Instruction
Our Logo (Concept)

RemindMe (Remi) — Giving the Physical World a Memory

Inspiration

Imagine trying to find your keys, glasses, or wallet without being able to see where you last placed them.

For people who are blind or have visual impairments, locating misplaced belongings can be more than a minor inconvenience. It can create barriers to independence and sometimes require assistance from others for everyday tasks.

This was the problem that originally inspired us.

We wanted to explore a simple question:

What if AI could remember what our surroundings look like, so we wouldn't have to rely on sight or memory alone to find our belongings?

Our initial vision was to develop an assistive technology that could help people with visual impairments locate everyday objects through a combination of computer vision, spatial awareness, and natural-language interaction.

But as we developed the concept, we realized something important.

The challenge of remembering where we placed our belongings isn't limited to people with visual impairments. It's something everyone experiences.

Whether it's a student rushing to class, an older adult looking for their glasses, a busy parent searching for a child's belongings, or roommates sharing a living space, we all rely on our memory to navigate the physical world.

That realization changed how we thought about our project.

Instead of building a tool exclusively for one group of users, we wanted to create an inclusive spatial memory assistant, inspired by accessibility needs but designed to help everyone.

That's how Remi was born.

Our goal is to transform the smartphone into an intelligent assistant that remembers where belongings were last seen, understands their surroundings, and helps people retrieve that information naturally.

Because technology designed with accessibility in mind can often make life better for everyone.

What It Does

Remi is an AI-powered spatial memory assistant that helps people find misplaced belongings by giving their surroundings a searchable memory.

Using a smartphone camera, Remi is designed to recognize everyday objects, understand their locations relative to their surroundings, and preserve those observations over time.

When users forget where they placed something, they can simply ask:

"Where did I leave my keys?"

Instead of searching through photos or manually remembering locations, users can retrieve a saved observation with contextual guidance:

"Your keys were last seen on the desk, next to your laptop."

Our system centers around three core capabilities:

Visual Understanding: Detect everyday objects and understand their relationships to nearby furniture and landmarks.
Persistent Spatial Memory: Remember objects, reference images, timestamps, and their last-observed locations.
Intelligent Retrieval and Guidance: Allow users to search through natural language and receive contextual descriptions and visual guidance.

While visual highlights can help sighted users, our accessibility-focused vision also emphasizes voice interaction and descriptive guidance that do not depend entirely on visual information.

Connecting People Through Shared Memory

As we explored the broader applications of Remi, we realized that memory isn't always individual.

In homes, dormitories, and shared living spaces, people constantly help one another remember where things are.

A parent remembers where a child left their backpack. A roommate remembers who last used the charger. A family member helps someone find a misplaced item.

What if AI could make that shared knowledge accessible, even when the person who remembers isn't nearby?

This inspired our vision for collaborative spatial memory.

With permission-based sharing, family members, roommates, and caregivers could contribute observations to a common memory of their environment.

Imagine someone asking:

"Where did my roommate leave the charger?"

RemindMe could retrieve a previously shared observation and provide a useful description of its last-seen location.

Rather than replacing human interaction, we envision AI helping people cooperate, support one another, and maintain independence without requiring everyone to remember everything.

Our goal is not simply to build an AI that recognizes objects, but to create technology that makes everyday spaces more accessible and helps people stay connected through shared knowledge.

How We Built It

Remi combines computer vision, spatial mapping, geometric localization, multimodal AI, and persistent data storage to transform visual observations into searchable spatial memories.

1. Smartphone-Based Visual Capture

We designed a mobile-first web application that uses an iPhone camera to capture images of the surrounding environment.

Accessibility was an important consideration in this decision.

Rather than relying on specialized smart glasses, expensive sensors, or custom hardware, we wanted to explore how far we could go using a device many people already own.

The frontend is designed to support camera capture, object searches, and the presentation of stored memories.

2. Spatial Mapping and Camera Localization

One of our biggest technical challenges was teaching the system to understand not only what an object is, but also where that object is located in the physical environment.

To address this, we designed a spatial mapping pipeline using concepts from Structure-from-Motion (SfM) and Simultaneous Localization and Mapping (SLAM).

Our approach involves:

Extracting visual features from overlapping camera images.
Matching features across different viewpoints.
Applying geometric consistency checks such as RANSAC to reject unreliable correspondences.
Estimating camera poses and reconstructing static environmental landmarks.
Matching new observations against previously recorded views to estimate the user's position relative to the environment.

These techniques provide a foundation for connecting object observations to a consistent spatial reference rather than treating each image independently.

3. Dynamic Object Detection and Filtering

An indoor environment is constantly changing.

Keys are moved, chairs are repositioned, bags are picked up, and objects disappear from their previous locations.

This creates a fundamental challenge for spatial mapping: an object that moves should not necessarily be treated as a permanent landmark.

We therefore designed a pipeline that separates relatively stable environmental features from potentially movable objects.

Static features can support camera localization, while movable objects are preserved as observations that may change over time.

An object doesn't have to remain in the same place to remain part of the system's memory.

4. AI-Powered Object Recognition and Scene Understanding

We incorporated multimodal AI into our architecture to recognize objects and interpret their surroundings.

Object recognition provides information about what appears in an image and where it appears within the frame.

Vision-language understanding adds contextual descriptions, such as:

"The glasses are on the bedside table, beside the lamp."

This distinction is particularly important for accessibility.

Simply returning a bounding box is not enough for someone who cannot easily interpret the image. We want the AI to transform visual information into meaningful language that helps people understand their environment.

5. Persistent Spatial Memory

Instead of treating each camera image as an isolated prediction, Remi is designed to preserve structured object observations over time.

Each memory can include:

Object identity and description
Reference image and bounding box
Room or scene association
Timestamp of the last observation
Spatial relationships to nearby landmarks
Optional 3D position when reliable localization is available

These observations form a searchable record of what the camera has seen.

Importantly, Remi distinguishes between where an object was last seen and where it is guaranteed to be now.

6. Natural-Language Retrieval and Guidance

We wanted users to interact with spatial information as naturally as possible.

Instead of navigating through image galleries or manually searching for object labels, users can ask questions in everyday language.

When a matching observation is available, Remi can retrieve the corresponding memory and explain where the object was last seen.

For sighted users, the application can display the reference image with the object highlighted.

For users who benefit from nonvisual interaction, our goal is to extend the system with spoken descriptions, voice queries, and accessible navigation guidance.

With reliable camera relocalization and 3D object coordinates, the architecture can also support directional guidance from the user's current viewpoint.

Challenges We Faced

1. Recognizing an Object Is Not the Same as Locating It

One of our most important technical realizations was that detecting an object in an image does not tell us where it exists in the physical world.

A bounding box provides image coordinates, but determining an object's physical position requires additional information, such as camera pose, depth, and a consistent spatial reference.

This led us to separate visual object memory from more advanced 3D spatial localization.

By doing so, the system can still provide useful last-seen descriptions even when precise geometric localization is unavailable.

2. Distinguishing Static Landmarks From Moving Objects

Visual mapping relies on features that remain geometrically consistent across observations.

However, indoor environments contain many movable objects.

We explored combining semantic object understanding with geometric feature matching to distinguish reliable environmental landmarks from objects that should be remembered without necessarily being used for camera localization.

3. Recognizing the Same Object Over Time

Detecting a pair of headphones is different from recognizing a specific person's headphones across multiple observations.

This introduced the challenge of object re-identification, where visual similarity, scene context, and observation history become important.

4. Building an Accessible and Inclusive Experience

Our original inspiration raised another important challenge:

How do we design an object-finding assistant that doesn't depend on the very sense we're trying to supplement?

A system that only displays a highlighted picture may be useful for sighted users, but not sufficient for someone who is blind.

This shaped our focus on natural-language descriptions and our plans for voice interaction and nonvisual guidance.

We learned that accessibility cannot simply be added at the end; it should influence the architecture and interaction design from the beginning.

5. Balancing Technical Ambition With Practical Usability

Accurate real-time 3D navigation requires sophisticated mapping, camera tracking, and reliable geometric reconstruction.

Rather than making every feature dependent on perfect localization, we designed Remi so visual memory retrieval can remain useful independently of the most advanced spatial capabilities.

This helped us balance ambitious engineering to build something people could realistically use.

What We Learned

Building Remi changed how we think about both artificial intelligence and accessibility.

We originally approached the project as an assistive technology for people with visual impairments. As our ideas evolved, we realized that many accessibility-driven innovations can solve much broader problems.

Designing for people who face the greatest barriers can lead to better experiences for everyone.

From a technical perspective, we learned how different areas of AI and robotics complement one another.

Computer vision identifies what the camera sees. Spatial mapping helps determine where observations belong. Persistent memory preserves information over time. Multimodal AI makes that information understandable through natural language.

We also learned that a technically impressive system is not necessarily a useful one.

A complex 3D reconstruction may be fascinating, but a simple, accurate description of where someone last placed their keys can have a more immediate impact on daily life.

Perhaps our most important takeaway is that AI should not only make machines more intelligent; it should help people become more independent, confident, and connected.

What's Next for Remi?

Our long-term vision goes beyond simply finding misplaced objects.

We want to explore how persistent spatial memory could make physical environments more accessible, collaborative, and intelligent.

Our future directions:

Collaborative Household Memory: Enabling families, roommates, and caregivers to share selected spatial memories with appropriate permissions and owner identification.
Continuous Spatial Memory: Automatically updating last-seen locations as users move through their environment with a higher frequency of updates.
Real-Time Navigation: Combining camera relocalization and spatial 3D mapping to guide users toward remembered objects.
Wearable Integration: Extending the system to smart glasses and other hands-free camera devices beyond applications on mobile devices.
Privacy-Preserving AI: Exploring on-device processing, user-controlled sharing, and secure storage for personal environments.
Accessibility Testing: Working with people who are blind or have low vision to improve our interaction design to benefit other needs.

We ultimately envision a future where our surroundings can provide helpful, accessible information without requiring people to memorize every detail of their physical environment.

Remi began with a question about accessibility, grew into an idea for universal spatial memory, and continues toward a vision of AI that helps people support one another.

Because technology is most powerful when it doesn't just solve a problem for some people, but opens new possibilities for everyone.