iOS Pocket LLM

A powerful AI assistant in your pocket—no internet required. This language model runs on your device, giving you instant help anytime, anywhere—even when you’re off the grid.

Comment

Inspiration

I'm inspired by this research Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone.

What it does

We aim to create an iOS app that uses LLM to do on-device processings and provides users with useful answers in seconds without the access of internet, this is especially helpful when a user want to explore the wilderness and needed a hand from AI but they are off the grid or if you just want to keep your convos with AI secure. The app can also act as a better and alternative solution to Apple Intelligence.

How we built it

We utilized the LLM module used for M-series MacBooks to port it into an iOS application, since they are all compiled in arm64, we have no problem porting it.

Challenges we ran into

  1. iOS memory allocation issue, iOS by default only gives an application 25% of the total device ram, we decided to offload the LLM module directly into the Video ram, which is how LLMs are supposed to be configured. This surprisingly bypasses the iOS memory allocation issue as iOS doesn't consider the Video Ram as part of the total ram allocation.

  2. Package size, by incorporating a lower quantization level LLM, we are able to decrease the size of the application into 1.69GB while maintaining the same level of quality.

  3. By default, the LLM is processed by the CPU, which make the process extremely slow, by pushing all requests to Apple's neural engine when available or pushing the process to iGPU can speed up the process significantly.

Accomplishments that we're proud of

  1. Fully offline, secure and fast AI model that is just in the palm of your hand.

  2. Successfully on creating a local AI app that can generate relevant and quality answers.

  3. Accomplishing various problems that are undocumented.

What we learned

  1. iOS hardware allocation scheme

  2. Different types of AI models, specifications.

  3. The amount of processing powers on a phone is enough to run a 3b AI model.

What's next for iOS Pocket LLM

  • The AI model should not be loaded upon start up / The AI model should load 1b if a device have ≤ 4GB of ram to prevent from crashing upon start.
  • When used on an un supported device, it should prompt user a confirmation of proceeding.
  • For ≥6GB ram devices, user have the ability to choose between 1b model, 3b model.
  • For ≥8GB ram devices, user have the ability to import models.
  • The ability to save questions.

Built With

Share this project:

Updates