PetKnows.me

PetKnows.me: see the world through your pet's eyes

Comment

PetKnows.me

What inspired me？

I still remember the night I decided to change my life. It was 1:00 AM and I was scrolling through cityscapes of places I’d never been, exhausted from another long day of 72-hour weeks in banking. Something clicked—I wanted a version of myself I hadn't met yet. What I didn't understand then was the price: the 8 TOEFL attempts, the soul-crushing essays, and the loneliness of chasing a dream no one understood. During those breaking points, it wasn't the "grand goal" that kept me going. It was Coco, my British Longhair Silver Shaded cat. His purrs refilled my "emotional bank account" when it was empty. In September 2025, I stood on a campus in Seattle—a stranger in a strange land. Feeling utterly out of place and anxious, struggling to catch every English word, I suddenly saw two fluffy dogs walking toward me. The university president said they were there to help students decompress. In that heartbeat, the thousands of miles across the Pacific vanished. I realized then that no matter where we are, these creatures give us the strength to stand firm. But love is a two-way street. Living with Coco, I often wondered: What do her tiny expressions really mean? How can I make her more comfortable? What does the world look like through her eyes? On a hiking trip, I met my co-founder—an AI scientist from the healthcare industry. We saw a tech world obsessed with cold efficiency and the anxiety of replacement. While AI is often used for logic, human mental health is becoming our most critical frontier. We realized that if we can use AI to monitor complex human health, we can use it to enhance our emotional well-being through the bond we share with our pets. We asked: If we can build the most rational, cutting-edge technology, why can’t we use it to protect our most primitive, softest emotions? When cold algorithms meet warm paws, AI shouldn't be a competitor; it should be a Translator of Love. This is why we started PetKnows.me. A pet’s life spans only a decade or so—roughly the length of one economic cycle. We want to use AI to build a deeper, more harmonious bond between pet parents and their companions. In a world full of uncertainty and economic winters, we want this cross-species understanding to add a little more "balance" to everyone’s emotional account. We cannot control the macro-cycles of the world, but we can protect the warmth in our own homes. PetKnows.me is here to walk alongside every ordinary person, using small moments of healing to build resilience—until the next spring arrives.

How we built it

To transform "emotional healing" into a tangible technical reality, PetKnows.me utilizes a collaborative architecture of Large Multimodal Models (LMM) and Small Language/Specialized Models (SLM). The project is structured into four core functional modules:

  1. The "Cross-Species Vision" Module: Multi-spectral & Perspective Simulation Mission: To bridge the sensory gap between humans and pets, fostering a deep understanding of how our furry companions perceive light, color, and space. Technical Implementation: We employ customized Computer Vision (CV) SLMs. Based on the biological optical traits of different breeds (e.g., dichromatic vision cone cell distribution, tapetum lucidum reflectance gain, and low-level focal geometry), we construct transformation matrices. Through Neural Style Transfer or pixel-level color mapping, standard RGB images are transformed in real-time into "Pet-Vision Spectral Maps" and "Low-Angle Perspective Projections." Value: The AI helps owners identify visual stimuli that are particularly sensitive to pets (such as specific contrasts or motion). It recommends optimal interactions based on chromatic sensitivity. For instance, in a grassy field, the AI may suggest using a blue or yellow frisbee (the most vibrant colors in a pet's spectrum) and recommend high-quality gear to ensure every moment of companionship is perfectly attuned.
  2. The "Harmonious Living Space" Module: Spatial Topology & Optimization Mission: To shift indoor environments from "human-centric" to "human-pet co-living" spaces. Technical Implementation: By integrating SAM (Segment Anything Model) edge detection with Multimodal LMMs, the system performs semantic segmentation and topological relationship modeling on indoor images. It quantitatively evaluates pet activity trajectories, vertical climbing potential, and safety hazards (e.g., fragile decor or toxic plants) to output a comprehensive "Spatial Friendliness Score." Value: The AI identifies layout flaws and provides effortless furniture adjustment suggestions. For low-score areas, it offers actionable upgrades—such as recommending a suction-cup window perch for a sunlit spot that lacks a resting place. Improving their domain is, in essence, an investment in the owner’s own sense of fulfillment.
  3. The "Emotion Translator" Module: Emotion Computing via Verifiable Reasoning Mission: To decode the physiological and psychological motives behind a pet’s subtle micro-expressions. Technical Implementation: Built on Gemini 3, Flask, and Streamlit, this module utilizes a Multimodal Agent framework. We designed a "Two-Stage Decoupled Reasoning (Chain-of-Thought Verification)" loop: Feature Extraction Layer: Identifies physiological feature vectors (e.g., ear angles, pupil dilation, whisker alignment). Semantic Reasoning Layer: Inputs these vectors into a verifiable logic loop, leveraging multimodal cross-referencing to mitigate LMM "hallucinations." Value: It translates fleeting physiological signals into intuitive emotional insights (e.g., Alertness, Comfort, or Entreaty). When the AI identifies states like "seeking attention" or "anxiety," it recommends soothing toys or calming treats. This immediate material feedback serves as a bridge for mutual empathy, allowing owners to feel the profound joy of "being needed."
  4. The "Health Guardian" Module: AI-Driven Non-Invasive Wellness Alerts Mission: To enable low-cost, high-frequency wellness monitoring within the home. Technical Implementation: We adopt a RAG (Retrieval-Augmented Generation) + Fine-tuned SLMarchitecture. Gemini 3 handles the foundational image description, which is then processed by a vertical model trained on Rigid Veterinary Knowledge Graphs (KG). The system performs comparative analysis on pathological features such as corneal clarity, coat gloss, and gum coloration. Value: The AI detects subtle health risks often missed by the human eye, outputting "prevention-first" nursing alerts in plain language. For sub-optimal health states, it provides targeted solutions—such as recommending veterinary-standard teething sticks for a kitten in its teething phase. Watching them thrive under your care is the most stable investment in an owner’s "emotional bank account."

System Architecture: "Minimalist Interaction, Sophisticated Computing" The core logic of the PetKnows.me system architecture focuses on seamless delivery: User Side: Designed for zero-barrier operation. Users simply upload a single image without needing to understand the underlying algorithms. Backend Side: A Routing Agent (powered by Gemini 3) automatically analyzes the image context (Outdoor, Indoor, or Pet Close-up). Like a "Symphony Conductor," it dispatches the data to the appropriate algorithmic modules: Landscape Photos: Triggers the Spatial Transformation algorithms. Indoor Photos: Triggers the Spatial Friendliness analysis. Pet Portraits: Concurrently triggers Emotion Analysis and Health Screening. This "Single-Trigger, Multi-Modal Dispatch" architecture ensures that anyone—from a 10-year-old child to an 80-year-old—can receive professional-grade insights and an optimal user experience in an instant.

Business Model Core: AI Pet Assistant & Smart Distribution for Pet Ecosystem To ensure the long-term emotional connection between owners and their pets, PetKnows.me utilizes a sustainable business model. We use AI to create the shortest path from "Perception → Understanding → Consumption." By solving the "information asymmetry" in human-pet interactions, we turn intangible "emotional healing" into observable and actionable digital solutions. Our platform doesn’t just answer "What is my pet thinking?"; it answers "What should I buy or do for my pet right now?" This prompt-based recommendation engine, driven by physiological and psychological data, gives PetKnows.me the potential to become a major traffic gateway for the pet industry.

  1. The "Perceive-to-Recommend" Loop: Precision E-commerce (The Logic of Conversion) Unlike the "search-based" logic of traditional e-commerce, our project is driven by "diagnostic consumption." (1)Trigger: AI identifies a specific physiological state (e.g., using CV models to detect red or swollen gums). (2)Emotional Lever: Taps into the owner’s desire to care for or comfort their pet. (3)Conversion Path: Instant Consumption: Recommends soothing sprays or stress-relief toys for anxious pets. (4)Preventive Consumption: Recommends pet-specific eye protection lights if the "Cross-Species Vision" module detects poor lighting. (5)Value: It eliminates hesitation by framing a purchase as an "act of love," significantly increasing conversion rates.
  2. The "Experience-to-Stickiness" Loop: Freemium Services (The Logic of Retention) We use high-frequency visual interactions to keep users engaged and professional reports to drive paid subscriptions. (1)Free Tier (Traffic Pool): Includes a daily "Mood Translation" and "Pet-View" camera filters. These features have high social value, encouraging owners to share content on Instagram, X (Twitter), and Reddit. (2)Paid Tier (Profit Pool): Room Optimization: A "Human-Pet Harmony Home Guide" generated using the SAM model (available via one-time purchase or subscription).Digital Health Records: Long-term tracking of fur quality, pupil changes, and walking gait, resulting in a "Quarterly Health Analysis Report." (3)Value: Builds a professional brand barrier, turning users from "tool users" into "data-reliant subscribers."
  3. The "Data-to-Ecosystem" Loop: B2B2C Partnerships (The Logic of Scale) We use the AI’s "early warning" capabilities to capture value at the top of the supply chain. (1)Precision Referrals: When the AI detects health anomalies (e.g., pale gums suggesting anemia), it provides direct booking links for partner vet clinics or pet insurance claim guides. (2)Targeted Marketing: Provides pet brands with anonymized behavioral data (e.g., average activity levels in a specific region) to help with product R&D and inventory planning. (3)Value: Lowers customer acquisition costs for vet clinics while enhancing PetKnows.me’s authority in the industry through professional endorsements.

Challenges we ran into

In the process of bringing the PetKnows.me technical solution to life and deploying it to production, we encountered five core challenges:

  1. Heterogeneous Model Orchestration & Efficiency The Challenge: Since we use an architecture that combines Large Multimodal Models (LMM) and specialized Small Language Models (SLM), the key difficulty is how to coordinate different algorithms, ensure complete information transfer, and maintain high computational efficiency. We need the LMM to act as the "brain" for decision-making while the SLM handles the "heavy lifting" for specific calculations—all without noticeable delay. Example: When a user uploads a video, the system must use SAM (SLM) for frame-by-frame segmentation and Gemini 3 (LMM) for environmental analysis simultaneously. If the scheduling logic is poorly designed, the processing time of the SLM might not align with the LMM's global understanding, causing a 2-3 second "audio-visual sync" lag in the rendered pet-view video, which ruins the immersion.
  2. Agent Workflow & Hallucination Control The Challenge: When dealing with multiple AI Agents working together, a tiny error from the first Agent can be magnified by the next one (error accumulation). Designing effective workflows and prompts that satisfy users while reducing GPU costs and hallucinations is a continuous iterative process. Example: In the "Emotion Translator," the first Perception Agent might misidentify a cat’s grooming action as "anxiety." If the prompt is not strict enough, the second Reasoning Agent will follow this wrong conclusion and suggest a long list of unnecessary soothing advice. We call this "confidently talking nonsense." We must design a self-checking mechanism where the Agent double-checks biological vectors before giving an answer.
  3. External API & MCP Integration The Challenge: AI model outputs are often semi-structured. Converting these suggestions accurately into structured queries for external e-commerce or veterinary systems is a major hurdle. Example: If the AI identifies that a kitten is teething and suggests buying a "vet-approved teething stick," it must connect to an external API via the Model Context Protocol (MCP). If the link is not precise, the system might suggest a generic "plastic teaser wand." This doesn't just fail to provide "emotional value"—it could even lead the user to buy an unsafe product.
  4. Memory Management & Personalization The Challenge: Over time, pet data (age, medical history, preferences) becomes massive. How do we extract only the most relevant memories within a limited Context Window? Managing user memory accurately is what builds long-term product loyalty. Example: A user mentioned three months ago that "Coco has a sensitive stomach." Today, the user uploads a photo of Coco vomiting. If the Memory Agent fails to quickly retrieve the "sensitive stomach" history from the vast archives, the AI might give a generic "dieting tip" instead of a critical warning that this could be a recurring medical issue.
  5. The Conflict: Symbolic AI vs. Connectionism The Challenge: Neural networks (Large Models) are based on probability, but health advice must be based on facts. How do we use "fixed rules" to constrain "uncertain predictions"? Example: If the system detects discharge near a cat's eye, the neural network might predict "infection" based on probability. However, our Veterinary Knowledge Graph (KG) sets a "Bottom Line": without temperature or physical exam data, specific medical diagnoses are forbidden. At this point, Symbolic AI overrides the model's fuzzy guess and guides the output to a safe, compliant suggestion, such as "monitor temperature or consult a vet." For us, this project is more than just writing code; it is about building an AI system that has perception, memory, and clear ethical boundaries. This complexity is exactly what forms the "moat" (competitive advantage) of PetKnows.me.

Accomplishments that we're proud of

  1. Accomplishments: From 0 to 1 with Grit Amidst the relentless pace of full-time work and MS-AI studies, our two-person team completed the entire journey—from field research at pet parents & shelters and technical architecture validation to model training. Today, our four core functional modules are live and accessible for users to try. It has been a difficult road, but we made it.
  2. What We Are Most Proud Of: The Echoes of Ordinary Souls What makes us proudest isn't the code, but the profound trust from our users. Despite our unpolished prototype, users provided us with serious feedback across nearly a hundred questions, sharing their grief and joy with their pets. Through in-depth interviews with over 30 pet parents, we identified more than 100 core needs. This made us realize that we are doing something small, yet deeply meaningful.

What we learned: AI for love.

During the journey of bringing PetKnows.me to life, we realized that our true goal is: "To help every ordinary person around me, and everyone they connect with, feel an 'emotional recharge'." We believe that if advanced technology and smart algorithms only bring more pressure instead of adding love and warmth to our lives, they lose their meaning. We hold onto our original dream: AI for Love. We want to turn the quiet healing we get from our pets into a visible, digital way to support our hearts. Big things often start from small dreams. By using AI to handle the little things in daily life—like understanding a pet’s silent look or making a home more comfortable—we allow love to grow, bit by bit. Just like a butterfly flapping its wings can start a breeze that changes the weather, these small moments of healing can create a storm of love and connection. We want to be there for every ordinary person, helping you through the ups and downs of life and protecting the small lights in the dark. PetKnows.me wants to walk beside you, giving everyone a little more strength to keep going. Let’s let our furry friends stay by our side as we change the world together, one small healing moment at a time.

What's next for PetKnows.me

While our initial version is live, it is far from perfect. We are acutely aware of our current limitations—such as the latency in multi-model orchestration and the simplicity of our current data privacy framework. However, we have a clear roadmap and the determination to solve these technical hurdles.

  1. Mobile-First Experience: Web to App Transition Current Gap: Our current Web-only interface lacks the convenience needed for real-time, "on-the-go" pet parenting. The Plan: We will develop Native Mobile Apps (iOS/Android) featuring seamless "snap-and-upload" camera interactions. Mission Alignment: By moving to the smartphone, we ensure that moments of healing are just a pocket-reach away, allowing users to replenish their emotional accounts the very second they feel the weight of the world.
  2. User-Driven Iteration & Specialized Agents Current Gap: Our current functions are broad; we need deeper vertical expertise to address specific pet needs. The Plan: We will expand user testing to introduce specialized Agents, such as "Pet Weight Management" and "Breed-Specific Nutrition". Mission Alignment: Listening to every piece of feedback allows us to transform PetKnows from a general tool into a personalized companion, acknowledging that every pet’s unique story is a vital part of an owner's emotional resilience.
  3. Multimodal Evolution: From Image to Video Current Gap: Static images are snapshots; they cannot capture the rhythmic behavioral patterns that signal a pet's true well-being. The Plan: We are upgrading to Short Video Analysis to capture dynamic behaviors like gait patterns and play habits. Mission Alignment: Capturing the language of movement allows us to translate the most primitive, softest emotions that static images miss, providing a more profound sense of mutual understanding.
  4. Advanced Memory & Privacy Security Current Gap: Our memory management is in its early stages, and data encryption must be fortified. The Plan: We will enhance the User Memory Bank for long-term/short-term management while establishing Local Encrypted Storage Mechanisms. Mission Alignment: A "Memory Bank" should be a safe sanctuary. By protecting this data, we protect the history of love and companionship, ensuring the "emotional account" remains a secure asset for the user's resilience.
  5. Ecosystem Connection via MCP (Model Context Protocol) Current Gap: There is a disconnect between AI insights and the actual ability to provide tangible care. The Plan: We will deepen our integration with the MCP protocol to connect AI advice directly with third-party veterinary and e-commerce platforms. Mission Alignment: This ensures a "Recommendation of Love" can become an "Action of Care" with one click, turning digital insights into tangible warmth for the home.
  6. Community & Knowledge Expansion Current Gap: Our Veterinary Knowledge Graph (KG) requires more authoritative data to be truly reliable. The Plan: We will open select non-core APIs to invite veterinary experts and developers to co-build our Knowledge Graph. Mission Alignment: By grounding our AI in professional truth, we ensure the "bottom line" of our technology is as safe as it is warm, providing a reliable foundation for every ordinary person to lean on.

The Concluding Bridge: Beyond the Code

Every technical milestone listed above—from optimizing GPU latency to refining video analysis—is ultimately a servant to a single, non-technical goal. We are not building these features to win a race for technological dominance; we are building them to serve as a Translator of Love. As an ordinary person who has felt the deficit of a bankrupt emotional account, I know that resilience is built through small, consistent moments of healing. These future updates are our way of ensuring that PetKnows.me can walk alongside every ordinary person, helping them—and everyone they connect with—to feel their emotional accounts being replenished. We choose this path to guard the faint light in the everyday, ensuring that even as the world feels uncertain, the warmth in our homes remains a constant gift of fate

A Call to Action There is still so much to do. We sincerely invite you to try PetKnows.me and share your thoughts. Whether your feedback is praise or criticism, we welcome it all. With your help, we hope PetKnows can grow to accompany every person and their pets through the long journey of life. Please feel free to reach out directly via email: jennyjingjing525@gmail.com or linkedin: https://www.linkedin.com/in/zhaoyu-li-809385ba/?locale=en_US

Built With

Share this project:

Updates