CardScope

collections page
login bottom sheet
recognition details
scan
match
developer diagnostics
how agent-device was used by codex to auto-test the ML model

Inspiration

Gaden: I play a lot of Pokemon, and at the local league, when we were trading cards, it was hard for us to figure out the value of the cards.

We had to manually go ahead and look up all the prices and determine which cards could be traded with which and what would be a fair deal. It was very tedious doing this manually.

Other apps, they didn't have the European market; they only had the US market when you were scanning cards. So, we wanted to make an app that could really fast scan cards and look up card prices on the European market.

Apps, you could only scan a single card at a time, so we wanted to develop something where it would be easier to scan a lot of cards at the same time as well—just being able to scan your phone across an entire binder and figure out the prices of every single card automatically.

Another thing that is often an issue when you want to scan cards is that you have to upload the entire card to the database that will run the AI model to determine what card it is. So, we wanted to run it all on-device for it to be faster. That way, when you are at card shows with bad internet, it's still gonna work nice.

What it does

Gaden:

Our app detects the cards in real time and gets their bounding boxes.

We then crop those bounding boxes into images, normalize for perspective, etc., and then feed those into an embedding model. The embedding model then turns the crop into an embedding that we can perform a search across in a database that we have constructed containing all Japanese and English Pokemon cards in existence. We have developed this model such that it can run fully on-device, cross-platform.

The app also features prices from all the major sites that have Pokemon card prices, and you can also track your collection, make sub-collections, see your value, etc.

How we built it

For the AI part, we tried running Google SigLip 2 on-device, which worked just fine on newer phone models, but it still heated up the phone over time, so we wanted to make something more performant.

We used Codex to go ahead and research what models would run well on mobile. Then we tried a bunch of different experiments and made a bunch of different ablations on all the different options that we had. What we ended up with is a small convolutional neural network that is using a teacher-student model, where the Google SigLip model is actually teaching our smaller RepViT model how to go ahead and make these embeddings such that we could retrieve the cards. So, it's a distilled version of the SigLip model that essentially runs on-device, super performant, without overheating the phone.

We also have a small convolutional neural network that detects the cards and gets the bounding boxes of the cards. That way, we can, in real time, display an overlay on all the different cards where they are located, and we can then feed those crops to our retrieval model that will make the embeddings and perform a search on the database that we have constructed. Then we can, in real time, look up what cards are currently present in front of your camera.

To begin with, Gaden built an Xcode version of the app, but we quickly realized that we needed to support Android as well because we want to support as many people as possible, not only iPhone users.

So, Mark started turning it into a cross-platform app as well with Expo.

Mark:

I first went ahead and dropped my preferred choice of stack, some other params in tech to use, the zip of pokemon-scanner and effect-smol, some skills I had, etc., into 5.6 sol pro in ChatGPT, which gave us a rundown on how to make the model cross-platform.

I've also had several of these projects cloned locally and vectorized(opencode,t3code, some nitro modules etc) so I could point Codex at these repos for Effect, mobile patterns, Nitro Module patterns, and the prototype pokemon-scanner project Gaden first scaffolded.

Once I got back the findings from 5.6 pro, I went to town after scaffolding Expo. I also asked Gaden to look into what pro recommended on model choices and other optimizations on the ML side.

I had an old Android phone I've used extensively to test the Android version. It was stuck into a MacBook Pro box and pointed at my second screen while Codex was debugging it using agent-device or argent.

Challenges we ran into

Gaden:

It was a challenge getting a, like, somewhat complex neural network to run on-device very performant without overheating the phone as well. It had also been a big task orchestrating Codex to go ahead and make these very long-running jobs because, essentially, what we did was set up, like, the experiments that we wanted it to make, and we made plans along with Codex on what it should try and do. Then we just had long-running experiments running all the time. It was just hard to ensure that we were following the correct paths and making sure that it behaved correctly at all times.

Another silly challenge that we had was that my gaming computer is in my bedroom, so when Codex was running experiments, my graphics card would overheat the room. So, I had to move my gaming computer into my living room instead. I had set it up such that I was using my MacBook to develop. That way, I could easily build the Swift app, and then it would SSH into my Windows computer that had the GPU for running the training and experiments, etc. That is essentially how I solved being able to develop on my MacBook while still being able to have the full processing power of my RTX 5080 on my Windows machine.

Mark:

I had model availability issues, so I set up scheduled tasks for an overnight heartbeat so I could wake up to several finished PRs. The other issue was Codex trying to take over my phone via agent-device to test the iOS app, so I had to make some adjustments there :)

Accomplishments that we're proud of

Gaden:

We have vibe-coded some really nice machine learning models that we have not been able to find online, just with the help of 5.6 sol pro and sol high-max locally. We have constructed something new that may be super useful once we release this, being able to instantly look up cards on your phone with on-device machine learning. That is super nice.

Mark:

Being able to almost e2e finish an Expo app in a span of less than a week, with complex ML models locally, is really crazy. The output has been tremendous.

What we learned

Gaden:

I have learned that the long prompts that I needed to give GPT-5.5 are no longer necessary because the agent can figure out so much by itself. Sometimes, it might even be a bad idea to give it long prompts because maybe I write some things in there, and it will follow it through until, like, it fulfills what I wrote in there. Instead, I should often just let it think for itself.

I have also learned a lot about what machine learning models can run on-device and how they can be bundled to be running on-device.

I have learned how to develop an app. I have not done that before. This was my first app that we, like, fully developed.

Mark:

Being able to let go of the constant handholding of 5.5 was really nice, and I'm proud that I also got there myself, i.e., not needing to babysit 5.6 as much.

I have worked on Expo projects with 5.1 onwards, and it really shows how far it has gotten in long-running tasks.

What's next for CardScope

Polishing up the app and making more performance improvements—thanks, 5.6 sol pro!
Releasing on the Google Play Store and App Store and getting some first users in.

Gaden:

I will start sharing it with all my friends at my local Pokemon League and getting them to test it out. That way, we can further improve it.