CarpAI

Architecture

Inspiration

LLMs are a very powerful piece of technology. Having the possibility to use an LLM through a job based decentralized fashion is the minimal requirement for anyone to construct APIs and Apps on top of lilypad.

What it does

Here, I built a stack composed of a python script, bundled in a docker container, which is then called by a module on lilypad.

The project does the following:

1) A JSON template prompt is uploaded to IPFS 2) A job using the docker image is send through the lilypad network 3) The job is run on a node, and the resulting inference is uploaded to the IPFS network. 4) The result can be gathered and injected into the JSON template prompt to create a loop and therefore and conversational chatbot.

How we built it

The project was built using python, the Transfomer library from HuggingFace, docker stack, and the lilypad framework.

However, the lilypad framework was not ready for production yet, therefore I spend most of the first day helping the lilypad team to fix their stack.

All the code is available on github: The CarpAI LLM image: https://github.com/Xqua/CarpAI-LLM and a fork of the lilypad project with the module code: https://github.com/Xqua/lilypad

Challenges we ran into

The lilypad stack was not working, and I had to help the team fix their stack such that it would run on any GPU based node. The stack was trying to package nvidia library the wrong way therefore I helped the team create a more robust docker image for their tech such that it is not dependant on weird library mounting hacks.