SOMA - Subjective Objective Medical Assistant

SOMA helps you understand your health day by day, transforming your personal notes into actionable insights so you know when to see a doctor and walk in fully informed, making your visits smoother.

Comment

Inspiration

Our health is not something to take lightly, yet we know so little about the effects that our actions and environment can have on our body. Many times we're confused about how severe some symptoms could be, or even what kind of treatment we should be looking into. This app, among other things, mainly aims to help you with decisions about your health.

What it does

SOMA works like a diary - take a few notes, and let the rest happen. The system takes your previous entries and pre-existing conditions into account to come up with suggestions for you. The user experience is designed to be as smooth as possible, supporting speech-to-text entries for ease of use. Adding an entry only takes a few moments and suggestions are received automatically when they're ready.

If your condition seems to be worse than expected and you decide to visit a doctor, we want to help you make this experience smoother as well. Doctors usually listen to your subjective symptoms during a visit, but we already keep track of these for you - even things you might have forgotten. If you agree to let doctors see your data, they can use a summary of your recent condition directly in the medical record, giving them precise and structured information.

Doctors can also use a speech-to-text feature during the examination. The app will organize all the information into the SOAP format, which can even be directly used for a medical record.

How we built it

As healthcare related data is specially sensitive, our main goal was to design the project with special care taken about data privacy.

Taking into account the Hungarian healthcare sector specifics while remaining open to the international market, our solution is multilingual, supporting both Hungarian and english speaking users. To achieve this, our stack was built from purely open source software in the form of self-hosted LLMs and speech-to-text models. We think it's important to mention that all the components of our stack run near real-time on consumer hardware.

Our solution has two main use cases: the patient and the doctor side. These views both use the same backend written in Python with Flask. The user interface was made with Flutter.

To achieve near real-time voice transcription we utilized models from the OpenAI Whisper model-family, one of which was finetuned on hungarian corpus. The main LLM we used for the construction of the SOAP notes was a model from the Gemma 3 model-family.

Challenges we ran into

Transcribing medicine related technical terms while maintaining our focus around complete data privacy, especially with keeping the performance of the entire process near-realtime out to be quite a task. It involved extensive testing, benchmarking, and searching for high quality, relevant datasets in order to validate our solution.

Keeping track of users' TAJ number is something that we had to be very careful about, as it's exceptionally sensitive data. In this case, the user has to give explicit consent for handling this data, which we comply with.

Using self-hosted services usually means that our models have lower performance. With enough experimentation however, we were able to find models that according to our experience performed just as good as enterprise level models on our specific tasks.

Accomplishments that we're proud of

Big emphasis on data security: our solution does not use third party hosted services for processing user data. All models (speech-to-text, large-language models) are self-hosted, making sure sensitive data never leaves the application. This architecture is designed with the EU AI Act in mind: as a high-risk AI system under Annex III (healthcare), SOMA incorporates human oversight by design, decision traceability, and full data governance — making it not just a prototype, but a compliance-ready foundation for real-world deployment

What we learned

Building SOMA showed us that in healthcare, trust comes first. Privacy and data security aren’t optional features - they are core product decisions.

When designing and working on the project, we had to really imagine ourselves as the users: both as patients and doctors. How would this feel to use, is there any value to this feature - it was constant thinking and experimenting to get just the right feel for our app.

What's next for SOMA

The next major milestone would be integration with EESZT — Hungary's national e-Health infrastructure. A recent amendment to the Health Data Act (effective January 1, 2026) now conditionally grants developers access to this massive, centralized health database for AI development — making this integration not just feasible, but a no-brainer.

EESZT is a healthcare goldmine: allergies, chronic conditions, medication history, lab results, and full care timelines — all in one place. With direct access, SOMA could auto-populate patient profiles, but more importantly, feed real clinical context into LLM inference to deliver truly personalized health guidance.

Hungary is uniquely positioned here — few countries offer this level of structured, regulation-backed access to national health data for AI development. What we could build on EESZT wouldn't just be a breakthrough for Hungarian healthcare; it's a blueprint that could scale globally as other nations open similar pathways. We have a concrete integration roadmap targeting EESZT's eProfil, EHR Repository, and eRecept services as our first touchpoints.

Share this project:

Updates