Cloak.AI - Use private LLMs

💡 Why?

Everyone in TreeHacks and beyond is using an LLM right now.

Be it ChatGPT (OpenAI), Claude (Anthropic), Gemini Pro (Google), whenever you click that "send prompt" button, your data is not private anymore. From training new models based on your information to tracking your location and purchasing your data (yeah, read the TOS!), companies are constantly stealing from us!

📧 How does it work?

Cloak.AI stands lightly but firmly on top of your prompt, detecting private information (PII) and removing it before it arrives to the LLM providers.

Users can enjoy top-tier state-of-the-art LLM results, while protecting their privacy.

Data is "masked" before it leaves your laptop, for example:

Your prompt is: "Write an email for Jack, sending back a high-blood pressure test result with a Cholesterol of 160 mg/dL. . ."
Cloak.AI's "masked" prompt is: "Write an email for A person, sending back a health appointment test result with a measurement of 100 mg/dL: Hi Person! . . ."
LLM response: "Here is an email for a person, with a health appointment test measurement of 100 mg/dL: Hi person . . ."
Cloak.AI "unmasked" prompt: "Heres an email for Jack, with a health appointment test result with a Cholesterol of 160 mg/dL: Hi Jack! . . ."

👨‍💻 How we built it

We built a proxy LLM that would be hosted locally for enterprise clients (currently we use ChatGPT 4 as a proof of concept) to hide user data. We then generate zero knowledge proofs and allow an admin to generate proof of compliance (SOC, HIPAA) that private data is not leaving the servers. Finally, we allow the user to choose from a variety of APIs (Intel Cloud, Monster API, ChatGPT) to perform their query, consisting of cloaked (masked) data. Then our proxy LLM receives the response and returns it back to the user with the real data filled in!

🏗 Technologies

We used ReflexAPI for the frontend, Flask and FastAPI for both of our backends, Intel Cloud, MonsterAPI and ChatGPT for our LLM backends, and an enormous amount of water, tea and boba for our human fuel.

✨ What's next for Cloak.AI

We are looking to fine-tune our models to provide better data privacy guarantees, and expand on healthcare, government and compliance sectors to allow everyone to use top tier LLMs without compromising privacy!

Built With

cryptography
openai
privacy
python
reflex
zkps

Submitted to

TreeHacks 2024

Created by

I worked on building zero knowledge proofs for set membership of private words in the prompt and connecting them with our main LLM backend. I also built our slide deck, brand and pitch!

Nicolas Berrios
I fine-tuned the censorship models. Also, I did the backend, specifically the masking of the prompts, local database, and connection with LLMS.

Fernando Smith
I was the main lead for the frontend, developed in Reflex. I created the structure and design for the whole website and connected the frontend and backend together.

SalarC123 Cheema
Jacob Jarvis