TikTok On The Clock

Inspiration

As an avid TikTok user, one of our team members came up with this idea after struggling to find the exact product through watching TikTok videos or collages.

What it does

TikTok On The Clock is a chatbot that is integrated with TikTok's current Search page. The chatbot is designed to aid users with their search experience and alleviate the workload they need to do. With the help of the chatbot, users can get access to reviewed/vouched for product links with just one inquiry. TikTok On The Clock will also add all the links it provides to a wish list that users can reference at a later time.

How we built it

The chatbot interface is built using React and Tailwind CSS and was built to simulate how the Search page on TikTok looks. Behind the chatbot interface is a ChatGPT wrapper that utilizes the API provided by OpenAI. It is used to find which link best fits the user based on “memory” (provided by a json file) and a description describing each link. Returning the single link that best fits the user. In order to generate the "memory" fed to ChatGPT, we used the Canny algorithm and OpenCV library to apply Image Segmentation on the collages. Products in the collage are individually identified through edge detection and stored for later reverse image searching. After that, each individual images is passed through the Google Lens API provided by SerpAPI, which takes in each product and matches it to Google Search results, outputting retailers and relevant websites for ChatGPT to learn.

Challenges we ran into

The images generated from the Image Segmentation portion is saved locally to the user's device while the Reverse Image Search portion takes in a public URL. This disconnect between the two caused us some difficulties since they needed to piece together for everything to work. We ended up needing to set up an AWS S3 Bucket to upload all the individual images and obtain a public URL through there to feed to the Reverse Image Search segment.

Accomplishments that we're proud of

Overall, we are incredibly proud of our achievements and progress throughout building TikTok on the Clock. Despite all the different components we started with, we were able to seamlessly integrate all the working components into one solid product. The chatbot is able to provide reasonable responses to a user's message and the links generated are relevant to inquiry provided. One of the technical highlights of our project was the implementation of advanced image processing techniques. Using the Canny algorithm and OpenCV for image segmentation allowed us to identify individual products in the collages, which was a huge leap forward, but unfortunately wasn't able to seamlessly integrate with the reverse search. Our solution of utilizing AWS S3 to bridge the gap between local image storage and the URL requirement not only resolved the issue, but also provided a scalable and more stable method for handling the image uploads, ensuring that our system can support future growth. On the other hand, our chatbot interface's was intentionally designed to mimic TikTok's search page. This not only makes it more intuitive and familiar for users but also ensures an ease of navigation and interaction, furthering our commitment to a smooth and robust engagement experience.

What we learned

Since half of our team is located elsewhere, we had to coordinate accordingly in order to ensure that we were all on the same page and that everything was progressing in a timely manner. This meant scheduling additional meetings and communicating more proactively over the communication channels we used.

The whole team also learned so much more about integrating multiple technologies and tools to create a cohesive product. From image processing to leveraging various APIs, each component required us to expand our knowledge and problem-solving skills.

What's next for TikTok On The Clock

Here are some future implementations we have thought of:

Adapt the chatbot’s personality/way of speaking to the user’s watch history
Include TikTok videos in the analysis process and provide links to videos that are most helpful considering the user's inquiry.
Expand beyond gift ideas! We can embed product links directly to any video mentioning items/products