storepilot

when our team sat down to brainstorm we were determined to build something unique soo we wanted to tackle a problem that went unnoticed, and we initially wanted to implement robotics just cuz someone said it but after brainstorming over 100 ideas we couldnt find a single concept that felt genuinely engaging or practical. until one of our teammates joked, "hena machi tae marketing tae dlala" we all started laughing but then we paused ppl in dlala is one of the oldest forms of marketing. It is active, physical, and highly adaptive. That joke shifted our focus to "physical marketing" soo we realized that in modern retail, the placement of products on a shelf or table is a silent form of physical marketing. It is a science yet retail managers have very little data on how customers interact with physical spaces compared to digital analytics. soo we said building an IoT and AI-driven system to study customer interactions and optimize store layouts in real-time. We called it StorePilot.

StorePilot acts as the "Google Analytics for physical spaces." It maps customer behavior in physical retail stores to help managers make data-backed merchandising decisions.

1. Computer Vision Tracking: Monitors visitor foot traffic using security camera feeds, tracking when customers stand in front of displays.
2. IoT Telemetry: Uses ESP32 PIR motion sensors to detect active presence and RFID scanner nodes to track when specific items are picked up.
3. Heatmap & Designer Layout: Dynamically renders active planograms and spatial blind spots, showing how customers navigate the floor.
4. AI Merchandising Recommendations: Automatically generates strategic recommendations using LLMs based on real time telemetry metrics.
5. Auto-Calibration Workspace: Uses Qwen Vision Language models to auto detect store layout boundaries from a camera feed snapshot, making calibration plug and play.

🛠️ How We Built It

We built StorePilot as a distributed, high performance architecture:

• Ingestion Gateway (FastAPI): Handled high-throughput telemetry events asynchronously. It includes coordinate processing, camera snapshot cache distribution, and the Qwen Vision AI auto-calibration gateway.
• Core Engine (Spring Boot + JPA): A thread-safe event-processing state machine that aggregates sensor hits into logical customer sessions, records dwell times, and calculates metrics.
• Computer Vision Pipeline (YOLOv8 + ByteTrack): Tracks visitors in real-time, matching feet coordinates against database polygons. Runs on PyTorch GPU (CUDA) acceleration.
• Frontend Dashboard (React + Vite): A custom-designed neo-minimalist administration panel with interactive canvases for layout design, spatial telemetry charts, and live LLM recommendation cards.
• Database (MySQL): Standard transactional persistence storing planograms, configs, products, and event timelines.

Challenges We Ran Into

• Camera Hardware Contention: Early on, the OpenCV camera thread locked the system, preventing the FastAPI server from serving screenshots to the web page. We solved this by implementing an asynchronous frame-caching queue in the Python script.
• Startup Race Conditions in Docker: During containerization, Spring Boot booted faster than MySQL could initialize its database tables, causing JPA to crash. We resolved this by implementing Docker container healthchecks and conditional dependency locks (condition: service_healthy).
• Visual Hallucinations in AI Calibration: Getting Qwen to return precise coordinates was difficult because it kept outlining physical furniture (the tops of tables and chairs) instead of the floor space where people stand. We resolved this by prompt engineering Qwen to specifically target the feet standing areas on the floor.

-

Built With

• ai
• api
• bytetrack
• css
• esp32
• fastapi
• hibernate
• httpx
• iot
• java
• jpa
• llama
• mysql
• opencv
• pir
• python
• qwen
• react
• rest
• rfid
• sensors
• spring
• typescript
• vision
• yolov8