Try It Here
As UT Dallas students, we observed that our advisors were often overwhelmed with answering simple questions that could easily be found by simple Google. This will take away resources from student who required actual advising.
Inspired by the reasoning abilities of PaLM 2 and the expert-level LLM of Med_PaLM 2 in answering complex medical questions, we decided to fine-tune PaLM 2 using Langchain with UT Dallas' public information database to help our advisors. The results from our advisor bot have shown exceptional accuracy and quality compared to the basic Bard model and ChatGPT, and we have received many positive feed back from our early testers.
What it does
CompassUTD can provide answers to a wide range of questions about UT Dallas, offering the most up-to-date and accurate information. Its capabilities include:
- Searching for course information and descriptions using natural language or shorthand names (e.g., "Comp Arch" instead of "CS 2340: Computer Architecture").
- Providing professor ratings from RateMyProfessors.com for UT Dallas faculty.
- Offering information about the majors and minors available at UT Dallas.
- Supplying general information about UT Dallas, such as tech support, parking, and more.
- Providing details about staff, schools, and departments, including contact information, office locations, hours, and more.
- Keeping users informed about the latest news and events at UT Dallas.
How we built it
- CompassUTD utilized Google PaLM 2's reasoning capabilities to generate accurate answers.
- PaLM 2 will ultilize Langchain's tools and the Programmable Search Engine to conduct research and locate the correct information.
- MongoDB is employed to store chat messages, enabling PaLM 2 to retain previous conversations for contextual understanding. It also generate a SessionId token so that the user can save their chat history in the cookies and enable them to see past messages.
The process of sending a message and then arrive at a conclusion follow the paper ReAct. The steps are as follow:
- Users initiate interaction with the chatbot via the web app by either making a new chat or accessing a previous conversation.
- The web app transmits the user's message and a sessionId token to the FastAPI server deployed on GCP's Cloud Run container.
- The FastAPI server checks MongoDB to determine if the user has any previous messages stored in the database.
- Vertex AI's PaLM 2 model receives the current message along with the previous message retrieved from MongoDB using the token.
- Utilizing the current and previous messages, PaLM 2 "reason" on the appropriate course of action to generate a response. This is the "Reason" in "ReAct"
- PaLM 2 "acts" on its plan by utilizing Langchain's microservices deployed on the same Cloud Run instance. This is the "Action" in "ReAct"
- If the initial response is unsatisfactory, steps 5 and 6 (the "ReAct" plan) are repeated until a satisfactory response is obtained or a timeout occurs.
- The response is then sent back to the FastAPI server, allowing the web app to retrieve the result by calling the FastAPI server.
This approach enables the chatbot to achieve high accuracy and retain conversation context, resulting in a more personalized user experience.
Challenges we ran into
- Initially, we planned for a dual-container architecture, with one container housing Langchain's tools and API, and another container hosting Vertex AI's ML serving. However, this setup increased the response time. Consequently, we spent 1-2 days refactoring the code to run everything in a single container.
- During the initial testing phase, some users successfully injected malicious prompts that could compromise PaLM 2. Fortunately, we mitigated this by chaining the prompts using the techniques described in Learn Prompting's defensive measures with Langchain. PaLM 2 can now detect malicious prompts and reject it. This ensure the safety and inclusivity of our app for all UT Dallas students.
Accomplishments that we're proud of
What we learned
- We gained insights into prompt engineering and learned how to create secure, safe, and inclusive prompts for UT Dallas students. Additionally, we mastered the technique of chaining prompts to facilitate the "ReAct" process.
- We acquired knowledge about Google Cloud Platform and MongoDB. Prior to CompassUTD, we developed PromptGPT trained on the Google Cloud platform with data stored in MongoDB. Consequently, we became proficient in several GCP technologies, including Custom Model, Pub/Sub, Cloud Run, Cloud Functions, and Triggers.
- We determined that PromptGPT did not produce satisfactory outputs, leading us to transition to CompassUTD. Recognizing the exceptional reasoning capabilities of Google's PaLM 2 from our experience with Med-PaLM, we decided to utilize PaLM 2 and train it to understand UT Dallas information. The results have been remarkable.
What's next for CompassUTD - PaLM 2 powered chatbot
MongoDB just announced a public review of Atlas Vector Search on July 10th(which is too late). Thanks to this announcement, we hope to use this Vector Search function to find the accurate and correct information. Currently, all the data are being web-scraped in real time using Programmable Search Engine and custom Beautiful Soup script. With Vector Search, we can compile all the resource monthly and have the langchain agents do vector search for the most accurate information.
We are currently in talk with an open-source club on campus called Nebula Labs, a Computer Science club that aim to make tools to improve UT Dallas students' experience. We planning to integrate their Course Catalogs APIs so that the bot can make the students schedule all in CompassUTD, with just a simple chat say: " Tell me about all the available sections" or "Will there be any conflict if I take this class in this section"
Our project will also be pitched at Big Idea Competition, a university sponsored startup event. And we are hoping to continue using Vertex PaLM 2 for our Advising chatbot, as PaLM 2 reasoning capabilities are extremely powerful.
Compass UTD Framework and Backend: https://github.com/DanNguyenN/CompassUTD
Compass UTD Frontend: https://github.com/arihanv/CompassUTD