AnchoredAI - AI Grounded in Truth

Inspiration

We were inspired by the vast amount of AI disinformation and hallucinations to try to find some solutions for this problem.

What it does

We built two features - One that implements knowledge graphs to ground LLMs in truth One that adds a grounding step for text to image generation

How we built it

We built it on top of the Big AGI open source project and added a python backend to enable using the latest and greatest AI/ML functionalities. We also updated the LLM Grounded Diffusion HuggingFace space to process text to image generation

Challenges we ran into

Parsing images from a Gradio client to a Javascript front-end was tricky Matching API schemas between Python and Javascript was also difficult

Accomplishments that we're proud of

Successfully implemented more accurate text to image generation Created a knowledge graph from scratch

What's next for AnchoredAI - AI Grounded in Truth

Run LLMs locally to safeguard confidential data Add Semantic Search for documents Expanding on knowledge graphs to add explainability (like Elon Musk's xAI project) - chain of thought + grounded LLMs

Built With

gradio
nextjs
python
react

Submitted to

SDxAI Hackathon
- Winner Third Place

Created by

I worked on adding a LLM-grounded layout generation step to improve accuracy for text to image generation using Stable Diffusion.

1) The first step was to create API endpoints for the UI to call. There were several challenges passing images from the Gradio app API to a Python FastAPI, then parsing them on the front end.

Gradio deprecated (https://github.com/gradio-app/gradio/issues/2262) outputting base64 strings for images so I needed to implement the base64 conversion in Python.

You can see my deployed changes here: https://huggingface.co/spaces/michaelcreatesstuff/llm-grounded-diffusion/tree/main

2) I added a Python FastAPI backend to run alongside an existing open source Next.js project. This was necessary because the Python AI ecosystem is much better than the Javascript one. This will allow the UI to integrate the latest AI advancements.

3) I modified the existing React/Next.js UI to

a) asynchronously execute the before grounding, layout generation, and after grounding api calls and display them in the UI
b) clean up conversation history context and the token counter stored in localStorage to not be polluted by the custom /imggen chat command and base64 output I added
c) Add functionality to upload base64 images to the app's filesystem

My changes for 2) and 3) are in closed PR #2 in this repo:
https://github.com/SDxAI-Anchored-AI/private-repo/pull/2

4) Integrate the UI with the API endpoints my teammate created for a proof of concept knowledge graph used to ground LLMs in truth

Michael Yang
I worked on creating the knowledge graph enhanced LLM system.

1) Firstly, I researched the various kinds of knowledge graph formats and the libraries that are usually used to construct them. I looked into Cypher, the RDF triple system, Neo4j databases, kglib, and rdflib for my project, and decided that the RDF system combined with rdflib would be the best one for the health and rental chatbot that we were trying to demonstrate.

2) Then I decided to implement a very basic form of a knowledge graph using a simple Subject-Verb-Object format for an apartment rental chatbot. I gave in simple truth statements as the context which the rdflib functions were able to format into triples and decode into context statements when queried.

3) Then, I worked on creating two functions which would be presented during the demo: grounded_llm(prompt), and ungrounded_llm(prompt).
The grounded_llm function queries relevant facts from the knowledge graphs based on the user prompt and adds that as a context to the LLM during inference. The ungrounded_llm function just calls the LLM without any modifications. I played around with different prompts to get familiar with the different ways the LLM would respond in different situations.
4) Then I came up with different interaction scripts to compare the two functions. I discovered that the LLM begins to hallucinate a lot more deep into the conversation with humans, so I decided to have a predefined past interaction instead of having a fresh conversation for the demo. Then, using the queried truth statements, I was able to show that the ungrounded LLM hallucinates a lot lesser in the latter case.

5) Lastly, I worked on the demo video, slides, and helped with integrating the backend python functions with the frontend Nodejs. My changes are available to see on the PR on the BigAGI project on Github

Vishvesh G B

Updates

Michael Yang started this project — Jul 16, 2023 05:29 PM EDT

Leave feedback in the comments!

Log in or sign up for Devpost to join the conversation.