Hand

An AI Driven Second-Hand Item Exchanging Platform on Campus

Comment

Inspiration

About a week ago, when I was preparing to move out from my old dorm to a new one, I felt ashamed seeing eight large bags of "trash." Most of these items were impulsive purchases that I used once or twice and then discarded. Since the individual items were not of high value and I was sure I wouldn't use them in the future, I decided to throw them away. I noticed that this is a common issue among my friends as well—everyone has to painfully discard many items when moving. This inspired me to create Hand to Hand, a second-hand goods exchange platform that allows people to meet each other's needs, thereby reducing waste and avoiding unnecessary new purchases.

You might wonder, how is this different from second-hand websites like eBay? The biggest difference lies in its eco-friendly concept. On Hand to Hand, all items are acquired through exchanges instead of money, which significantly reduces unnecessary waste. Additionally, in the main use case of Hand to Hand—campuses—platforms like eBay have several well-known issues:

  1. Students find it difficult to spend a lot of time and effort uploading item information to second-hand websites.
  2. High shipping costs make it impractical to trade items worth less than $20 because the shipping could be more expensive than the item itself.
  3. Ensuring product quality is difficult.

To address these issues, Hand to Hand offers the following features:

For “Sellers”

All you need to do is lay all your stuff on the ground and take a picture. Hand to Hand uses image segmentation and visual understanding models to automatically crop the item images and assess the condition of the second-hand goods. Using Selenium, Hand to Hand scrapes the original prices and images of the items from Amazon and generates a brief description for your items. Remember, all you need to do is take a picture!

During the Exchange

Traditional bartering usually occurs between two people. However, considering the supply and demand in a campus might not always match, I designed an algorithm to facilitate possible multi-person (more than two) exchanges. Simply put, I store all items and users as nodes in a graph and use an optimized Depth-First Search (DFS) to find a complete loop. Since loops formed by multi-person exchanges might be unstable, I trained a machine learning model to determine item similarity, so when one edge of the loop is removed, the most similar item can automatically replace it. This recommendation system greatly enhances the chances of forming multi-person exchange chains.

For “Buyers”

Since users live on the same campus, meeting offline becomes the best way to trade. I integrated Google Calendar to automatically find common free times between users for transactions. To further save time, I designed a feature called “trade along your way.” In simple terms, Hand to Hand automatically invokes the Google Map API and, combined with the schedule in Google Calendar, helps you arrange trades during times when users’ routes intersect. This maximizes time efficiency and minimizes waste.

How We Built It

I will explain my tech stack based on the functionality of the project.

Function 1: Process Item Information with One Click

First, I deployed the YOLO V8 model using OpenVino for instance segmentation, which significantly boosts the speed of visual models on a CPU. After obtaining the segmentation masks, I used OpenCV and Selenium for web scraping. OpenCV detects the search icon in the browser and simulates mouse clicks, while Selenium retrieves specific item information (including name, price, image, category, description, etc.) from Amazon. Then, using GPT-4-O’s visual capabilities, we compare images of new and second-hand items to estimate their condition (Excellent, Like New, Good, Acceptable, Not Very Well) and appropriate price. The processed information is stored in JSON format and sent to MongoDB. For all images, we used Amazon S3 Bucket to handle image and URL conversion.

Function 2: Find Possible Exchanges Between More Than Two People

Here, we store all items and users as nodes, connected by directed edges (if the direction is from user to item, it means User_X owns Item_X; if from item to user, it means User_X wants Item_X). By doing this, we transform the problem into a mathematical one: we need to find a closed loop where users and items are mutually connected. I used a Depth-First Search (DFS) algorithm to find the shortest path loop based on the existing edges, thus forming a feasible trade chain.

Function 3: Smart Replacement

For situations where a loop cannot be formed, similar items can be substituted. For this, I trained an autoencoder model using Pytorch GPU in Intel IDC Jupyter Notebook to determine item similarity. The dataset used is the Kaggle Amazon Sales Dataset, which contains detailed information on 1,466 transactions from Amazon in 2023.

Model Training Steps:

  1. Data Preprocessing: Standardizing numerical features, One-Hot encoding categorical features, and TF-IDF vectorizing text features.

    • Numerical Features (Price): Standardizing item prices to enable comparison across different price ranges.
    • Categorical Features (Category): One-Hot encoding to convert item categories into numerical features for model training.
    • Text Features (Name and Description): TF-IDF vectorization to extract key information from the text.

  2. Autoencoder Training: Comprising an encoder and a decoder.

    • Encoder: Compresses high-dimensional input features into low-dimensional representations (embeddings), extracting main information and reducing noise and redundancy.
    • Decoder: Restores low-dimensional representations to high-dimensional input, ensuring key information from the original data is retained.

  3. Similarity Calculation: Using the encoder to extract low-dimensional embeddings, simplifying item features while retaining key information. Cosine similarity is used to calculate item similarity, normalized to a 0-1 range.

  4. Recommendation Generation: Based on the similarity matrix, generating a recommendation list for each item by finding those with the highest similarity.

After training the model, with Intel IDC’s powerful computing capabilities, I expanded the dataset to include 100,000 shopping records and used Optuna for automatic fine-tuning of model parameters for 100 rounds, resulting in a highly effective similarity and item recommendation model. Additionally, to better understand text information, I experimented with fine-tuning the Llama3 8B model using Intel IDC, but due to time constraints, the fine-tuning is still in progress.

Function 4: Smart Meetup with Automatic Schedule Setup

We simulated two users with different Google Calendar IDs. First, we searched for common free slots shorter than one hour (to avoid disrupting relaxation) in both schedules. Then, by invoking the Google Map API, we obtained the routes of the two users during these time slots. We segmented the routes and looked for locations where the latitude and longitude differences are less than 0.0001 (approximately 14 meters) at the same time. Finally, we added the location and time slots to both users’ Google Calendars.

Other Features

For the front end, I used templates from Figma and Anima tools to automatically convert the designs into HTML and CSS, saving a lot of time.

Challenges we ran into

Although the functional design was established early, we faced numerous technical challenges during implementation. For example, since YOLO V8 is a category-based model, its initial performance in instance segmentation was suboptimal, causing me to waste a significant amount of time. The connection issues with MongoDB were also quite frustrating. When using the OpenAI API, we frequently encountered rate limit issues with base64 image encoding. I resolved this by switching from base64 to using Amazon S3 to convert images to URLs.

Another challenge was managing the task progress in an orderly manner. Since I completed the entire project alone, going thirty consecutive hours without sleep often caused my brain to crash and cease functioning. To address this, I alternated between coding and documentation. After implementing each feature, I immediately wrote the documentation and created the PPT, which effectively alleviated my drowsiness at 3 AM.

Integrating all functionalities also proved difficult. The code directly converted from Figma for the front-end design had serious layout issues in actual use. Additionally, the CORS (Cross-Origin Resource Sharing) settings for the Flask server were unstable, often causing page content to fail to load, which was a constant source of frustration.

Accomplishment that we're proud of

Firstly, my execution capability is something I am most proud of. I am pleased that I was able to implement almost all the initially planned features. Previously, I had only done separate front-end and back-end development and had not truly integrated them. This full-stack development experience has made me very happy and proud.

Most importantly, the features I implemented perfectly addressed the problems faced by eBay in the campus context. Brief Recap:

  1. As students, spending a lot of time and effort uploading item information to second-hand websites is difficult; (✅ Solved by one-click upload of all item information)
  2. High shipping costs make items under $20 not worth trading (since shipping could cost more than the item itself); (✅ Solved by face-to-face transactions, eliminating shipping costs)
  3. Ensuring the quality of items is challenging; (✅ Solved by face-to-face transactions, allowing buyers to inspect item quality in person)

What we learned

I learned to more proficiently call vision models and use the OpenAI API to process images. I tried integrating Google Calendar and Google Maps for the first time. I applied search algorithms (DFS) learned in class to real-world scenarios for the first time. In terms of model training, I discovered a very user-friendly and powerful tool—Intel IDC (better than Colab).

Whats's nest for Hand to Hand

Next, I will seek partners and engineers for collaboration. Additionally, I will implement a more mature backend framework for Hand to Hand and design more aesthetically pleasing front-end interfaces. I plan to promote this project at Stanford University and further optimize it based on market feedback.

Share this project:

Updates