CodeShift

Inspiration

CodeShift began with a deceptively difficult challenge: build a conversational shopping agent that does more than produce convincing responses. It had to understand evolving customer intent, ask useful questions, retrieve relevant products, rank them accurately, and meet strict technical benchmarks within ten turns.

Our team consists of five sophomore and junior computer science students. We saw the prompt as an opportunity to build something technically grounded rather than relying on a single black-box model. We wanted every recommendation to come from a pipeline we could inspect, test, measure, and defend.

What it does

Our agent turns an evolving shopping conversation into a structured product search.

It extracts catalog-grounded categories and customer constraints such as material, color, size, style, budget, brand, and use case. It remembers those preferences across turns, detects when the customer changes direction, and selectively removes outdated constraints. At the same time, it continues returning ten product recommendations, so every clarification turn remains an opportunity to find the target.

The result is a deterministic, privacy-conscious agent that combines natural conversation with measurable retrieval performance.

How we built it

We divided the system into five responsibilities, R1 through R5:

R1 developed retrieval, including BM25 search, candidate-pool processing, category constraints, popularity signals, and multi-turn query accumulation.
R2 developed the ranking layer, with a local cross-encoder option and deterministic fail-soft behavior.
R3 built the catalog-grounded dialogue system, including slot extraction, clarification selection, intent classification, and override detection.
R4 integrated the components, enforced timeouts and fallback behavior, and built the evaluation workflow.
R5 implemented intra-session memory and preference-aware retrieval signals while documenting the final architecture.

A shared contract connects these components through a clear pipeline:

dialog → memory → retrieval → ranking → recommendations

Each component has an explicit fallback. If dialogue, memory, retrieval, or ranking fails, the agent still returns a valid response containing unique catalog products. Separate timeout workers also prevent one slow component from blocking the rest of the pipeline.

We evaluated every meaningful change instead of relying on intuition. On the 150-session development evaluation, the integrated pipeline achieved a technical score of approximately 0.774, with 94% Hit Rate@10, 0.546 MRR, and 4.02 mean turns to conversion.

Development tools used

CodeShift was developed in Python with Git and GitHub-based collaboration. We used feature branches, pull requests, focused commits, and careful merges to coordinate work across a fully virtual team. The repository includes a Makefile for common workflows, Python's built-in unittest framework for automated testing, and the challenge-provided deterministic local evaluator for repeated scoring and regression checks.

The repository does not require a specific IDE, notebook platform, container system, or web framework. It can be built and evaluated locally without VS Code, Google Colab, Jupyter, Docker, or a hosted backend.

APIs used

The committed pipeline does not require an external API and is designed to run offline. This keeps the core system reproducible and avoids sending customer messages or catalog data to a hosted service.

An optional OpenAI listwise reranker is included behind an explicit configuration flag and an OPENAI_API_KEY check. It is disabled in the submitted configuration, so the active pipeline does not make OpenAI API calls. Optional Hugging Face model downloads may be used only during setup when enabling local embedding or cross-encoder models; they are not required by the BM25-based core path.

Libraries and frameworks used

The active core uses a deliberately small Python stack:

PyYAML for configuration loading.
NumPy for vector and score-fusion utilities.
Python's built-in SQLite FTS5 support for BM25 lexical retrieval.
Python standard-library modules including unittest, concurrent.futures, dataclasses, json, re, and pathlib.

The repository also supports optional local ML paths using PyTorch, sentence-transformers, and Hugging Face Transformers. These power dense embeddings and a cross-encoder reranker when their dependencies and model artifacts are installed. Each optional ML path fails safely to the deterministic baseline when unavailable.

Datasets and assets used

The project uses the competition package derived from Amazon Reviews 2023, published by the McAuley Lab at UCSD:

A frozen 50,000-product Clothing_Shoes_and_Jewelry catalog.
Structured product metadata, including product titles, categories, features, descriptions, stores, prices when available, and rating information.
A challenge-provided 200-session labeled public development set used for local evaluation.
An organizer-held private evaluation set that is not available to the team.

The agent uses product metadata and anonymized session profiles only. It does not use product images, videos, raw review text, direct user identifiers, raw purchase history, manually created labels, private evaluation data, or externally sourced user data.

Challenges we overcame

Our greatest challenge was coordination. All five team members were balancing busy academic schedules, and the project was completed entirely virtually.

That forced us to treat collaboration as part of the engineering problem. We established clear component ownership, shared interfaces, feature branches, focused commits, and disciplined merge practices. Changes were tested before integration, conflicts were resolved deliberately, and the complete pipeline was evaluated again before being proposed for merge.

We also learned that individual components can appear correct in isolation while still failing to exchange the state they need. The final integration work connected dialogue slots, memory boosts, session identifiers, override signals, retrieval diagnostics, and ranking candidates across the entire pipeline.

Accomplishments we are proud of

We are proud that five student developers working remotely produced a complete, modular search system rather than five disconnected prototypes.

The agent is deterministic, testable, and resilient. It understands multi-turn constraints, handles changes of intent, remembers session preferences, filters against real catalog categories, and always returns recommendations alongside clarification questions. Our test suite grew to more than 100 passing tests, and repeated evaluation runs produced identical metric results.

Most importantly, every architectural choice has a reason behind it and evidence supporting it.

What we learned

This project taught us that strong AI systems are not defined only by the model they use. Retrieval quality, state management, interfaces, failure handling, evaluation discipline, and team coordination can matter just as much.

We also learned to treat benchmark results as diagnostic signals. When an idea failed to improve retrieval or ranking, we kept it optional or disabled it. When multi-turn queries lost earlier context, we fixed the state flow rather than trying to hide the problem with a more complex model.

Working virtually reinforced another lesson: clean Git habits and precise ownership are not administrative details. They are what allow multiple developers to move quickly without destabilizing the system.