Inspiration
LLMs have opened up a new type of privacy risk. Any API call or chat with LLMs now leaks thought patterns and big model providers can develop a cognitive fingerprint of any user, this can be very concerning.
What it does
The product priweight.ai provides private inference, where all the prompts and chats cannot be read by anyone else. It also verfiably proves that the inference is coming from GPT OSS 120b
How we built it
The GPT OSS 120b is hosted inside a GPU TEE. The prompts are encrypted on the client side, sent to the TEE that decrypts it, runs the model and re-encrypts it and strams the token back.
No one can see the plain text ever, it can be crypographically proven that is the case.
Challenges we ran into
High gpu costs
Accomplishments that we're proud of
Building this with not much knoweldge in cryptography
What we learned
We need more GPUs to scale to many users
Log in or sign up for Devpost to join the conversation.