APOLLO

Ask Apollo
Summarize and take measurements
Passive metrics are kept for you to see

Inspiration

Finding out something basic about your health usually means having the right device on hand. “Do I have a fever?” needs a thermometer. “Is my heart racing?” needs a watch or a pulse oximeter. “Am I breathing as well as last week?” needs a meter most people have never held. Meanwhile, everyone carries a phone packed with sensors that already measure the physical world: thermistors, a camera and flash, a barometer, accelerometers, a speaker and microphone. The issue is that all these sensors run in the service of the phone, not necessarily in the service of the person holding it.

This was our inspiration for Apollo: We want to help people use the powerful sensors that are already in their pocket to understand their health and act on it.

Building on research produced by the Ubicomp lab here at UW, Apollo leverages the existing sensors on your phone to help you answer questions about your health. We can measure your pulse from the camera and flash, your temperature from the thermistors that usually monitor the battery, and so much more. And Apollo doesn't stop after giving you a reading, it helps you take the next steps: Whether that's planning a doctors appointment to confirm the illness, order over the counter medicine from your local pharmacy, or discovering the answer is "you're probably fine" and enjoying your day.

What it Does

Apollo is an android app built entirely on the phones own sensors. It requires no additional sensors to use, and all of the measurements it provides work without an internet connection. It offers two kinds of measurement, tests you take deliberately and tracking that runs in the background, and then helps you act on what it finds.

Measure Once

The heart of Apollo is a daily check-in: four short tests, each a few minutes or less, with a ring on the home screen that fills as you go. Every test also runs on its own whenever you want it.

For heart rate, you cover the rear camera and flash with a fingertip for twenty seconds. Each heartbeat changes how much light passes through your skin, and the camera sees it as a tiny flicker. Apollo estimates the rate two ways, from the rhythm of the brightness and from the gaps between beats, and shows a number only when the two agree. In our tests it agrees closely with a Fitbit.

For temperature, you hold the phone by its corners and press the screen to your forehead for ninety seconds. Heat flows from your skin into the phone, and Apollo reads how fast the internal thermistors warm. A short calibration teaches it how your forehead warms your particular phone, and a trained model turns that warming into a body temperature. Readings are labeled as estimates, and a fever reading always tells you to confirm with a thermometer.

For a lung check, you blow hard at the screen a few times. The barometer feels the glass flex under the blast, and Apollo tracks your breath strength over time against your own baseline rather than claiming a clinical number.

For calm sensing, the speaker plays a tone you can’t hear and the microphone listens for its echo, like sonar. When your movement spikes above what’s normal for you, Apollo nudges you to take a breath.

Measure Passively

If you opt in, Apollo also watches a few signals without being asked. Steps and flights climbed come from the step counter and the barometer in the background. An eye-tracking prototype uses the front camera to count blinks and gaze shifts, which are early signals of fatigue or feeling overwhelmed. Only the counts are kept; no images are ever stored.

Act on the Information

Ask Apollo is a chat that explains your readings against your own history and points you to the right test. It runs on the phone by default. If you turn on Gemini mode, your chat and a short summary of your readings go to the model, and nothing else does.

Picked for you turns readings into a short shelf instead of a catalog. Apollo learns from your readings, your chats and your searches, and suggests a few items with a reason attached (“Based on your temperature”). Products show real photos, current prices and facts pulled from the FDA drug label. Hiding something teaches it to stop suggesting it. Sometimes the right shelf is empty, and Apollo will say so.

Get it closes the loop. You can order through DoorDash or Uber Eats, in their app or on their site inside Apollo, or open a map of the nearest open pharmacies and clinics with hours and phone numbers. Emergency help stays pinned to the screen.

All of this sits behind a few rules written in plain code that run before any AI and can’t be overridden by it. Emergency or crisis words bring up 911 and 988 immediately, on the phone, and are never used for recommendations. Fever answers always say to confirm with a thermometer. And Apollo never buys anything: every order happens in DoorDash or Uber Eats, with you in control. No accounts, no cloud database; the only things that leave the phone are search words, an approximate location, and the Gemini chat if you’ve opted in.

Challenges

Providing so many sensitive tests in such a short amount of time came with serious challenges. Our first attempt at measuring lung functioning tried to extract a meaningful signal from microphone data. This failed because the microphone was too sensitive to the way that the phone was held, and because the data we received had the peaks of the waveform cut flat. The temperature measurement was also hard to tune - the reading is based on an ODE describing the rate at which the device heats up against the skin, and so we had to ensure the phone starts the experiment at a consistent temperature.

Integrating the features into one app taught us a second set of lessons. Small data overfits: when we let an automated search tune the recommender’s weights, it fit the handful of scenarios we’d written and little else, and only cross-validation caught it, so we kept simpler weights we set by hand. Android has rules: background camera use is only allowed as a visible, user-started service with a permanent notification, and we designed the eye-tracking prototype around that rather than fighting it. And commerce from health data can feel creepy, so we kept the data on the phone, required that every order go through you, and put the safety rules ahead of the AI, so that only data you are comfortable sharing leaves your phone.

What we are Proud Of

Heart rate that agrees with a Fitbit, from nothing but the phone’s camera and flash.
Temperature that calibrates to you and your phone, read from the internal thermistors with no root access and no extra hardware. We found a route to the thermistors that needs only a developer permission granted once over USB.
Breath strength you can track over time, from a sensor nobody thinks of as a health instrument.
A four-test daily check-in that runs on an ordinary phone, with one tap to start, built in Java on the plain Android SDK with the whole UI written in code and the logic tested on a laptop against recorded sensor data.
A shopping assistant that sometimes tells you not to buy anything, with live product, drug-label and map data from sources that need no API keys and a bundled snapshot so it still works offline.
Honest negative results. When an approach didn't look promising, we were honest with ourselves and looked for alternative routes instead of pushing a product we didn't trust.

What's next for Apollo

Several of our measurements can only tell you when you deviate from your own baseline. Calibrating these tests to give you precise numbers would take user data that we did not have time to collect, but could easily be done in a week. After that, agentic checkout through Visa Intelligent Commerce, using agent-specific payment tokens with spending limits you set, so an approved order can complete without handing over a card number. A travel mode would match medicine names across countries (Tylenol is paracetamol in much of the world) and find what’s open nearby. And we want more signals and more phones: breathing rate from the motion sensor, and devices beyond the Pixels we tested on.