StreamCare

CV Demo
Generate Module
Generate Module Post-Droid
Post-Demo Data
Aggregated Participant Data

The problem we face

Currently, a longitudinal study takes hundreds of thousands to over a million human-hours to conduct in its entirety [1]. Not only that, but low participant retention can cause biases in the data and loss of consistency across multi-year time scales. Our project proposes a unique way to tackle this massive human workload by streamlining the process from the researcher’s side while simultaneously removing the barriers that cause participants to drop out or lead to non-response [2]. By connecting participants and researchers in this manner, StreamCare greatly improves the efficiency of longitudinal studies.

Inspiration

Our team was personally inspired by the sheer impact that longitudinal studies have had on shaping the world’s views. One salient example is Harvard’s own “Study of Adult Development” that showed key indicators of life satisfaction were the quality of relationships in an individual’s life, a finding that has fundamentally changed how we see our lives and growth [3]. But amidst all the success that these landmark longitudinal studies have had, there is a huge barrier to entry, logistical complexity, and loss of data consistency that has caused the failure of at-scale longitudinal studies, and some of these studies that have fallen apart may have held similarly earth-shattering results that we won’t be able to obtain. If we can fight back against the roughly 80% loss in participants after 2 years and over 90% loss after 3 years that is common in many longitudinal studies, we can reclaim the statistical power these studies are capable of [4]. A study on retention strategies in longitudinal cohort studies finds that barrier reduction is the most effective method to increase participant retention, so that was our starting philosophy [5].

What it does

StreamCare enables researchers to create a tighter data loop with participants, receiving more data. This data is also more digestible with computer vision statistics. Researchers can combine these statistics with the video and participant form to assess how the participant is doing. This system is also modular, meaning researchers can create new video assessments to assess new sides of the participant and whether they correlate with the participant’s well-being.

For participants, StreamCare provides an easy-to-use system to submit high-quality videos and assessment measures. Before anything is sent, the camera coaches them in real time, much like a banking app scanning a check: "move a little closer," "turn on a light," "keep your whole hand in view." Clinicians stay firmly in the driver's seat, reviewing each result and validating each data point. With these more common check-ins, participants need fewer in-person clinic visits, and each visit can be more informed and specialized to the patient.

How we built it

Live computer vision with Google’s MediaPipe hand tracking. It runs in the browser and gets paired with tracking for capture quality (blur, lighting, framing), so that bad recordings are never uploaded to the system.
Researchers engage in a feedback loop with Droid SDK to describe a medical condition, a task, and the measurements they want to take to determine symptoms. Droid judges whether a phone camera can actually capture each one and suggests rewrites. The researcher can reply and ask again until the plan is accepted, after which it can be added to studies.
Droid drafts a new module that fits our standard format and is instructed to reuse our existing MediaPipe hand-tracking components (allowing for modularity in CV models)
Daytona trains in parallel in isolated sandboxes to tune CV model hyperparameters for the best-performing threshold
Connected Supabase (Postgres, Auth, Storage) as a backend to store a full audit log, migrations, and Droid and Daytona’s progress.

Challenges we ran into

One of the hardest challenges with this project was creating the customizable computer vision models for each researcher's application. We knew that static models taking user input would not scale to real research needs, so we built a generative module pipeline with Factory Droid and Daytona.

We made sure to prioritize clean code quality throughout the project, both in developing the project and within Droid’s Python modules. Code written by an AI agent shouldn't run on our server, so every experiment runs in a disposable Daytona sandbox. We also check the code's structure (the class, its ID, and its required methods) before using it.

Accomplishments that we're proud of

End-to-end implementation, from participants joining with an invite code, live CV recordings in the browser with feedback, uploading to the clinician portal for review, and then exporting data to CSV to better create longitudinal studies.
A working AI pipeline for writing new modules: a researcher can go from a plain-English description of a specialized medical condition to a reviewed, deployable CV module instantly available to be deployed to participants.
Using only open-source CV models as our foundational models to maintain transparency, which is critical in the medical field.

What we learned

Live feedback on home recordings is a must. Our MVP first prompted the participant to upload a video, but about 21% of the pictures in Google's SCIN dermatology dataset failed our quality checks, mostly due to blur, before any model saw them. This applies to videos as well. Checking quality on the device and giving live feedback was instrumental to the user experience (even when testing our product!)
Generic AI isn’t clinical AI. AI LLMs can often give great health advice, but we learned, especially through our conversation with Henry Wei, that entrusting AI to make predictive measures is unsafe and unrealistic. However, continuous feedback loops between a more standardized, general-use AI like Droid and domain knowledge from a clinician can improve the process and allow for data collection that could potentially be used for inference.

What's next for StreamCare

The most powerful part of StreamCare is its modularity and the fact that it becomes nearly exponentially better with scale. Even with the current modularity of the set of computer vision models we have now, researchers are relatively limited to tracking hand movement or grading skin conditions. But, as more researchers use the platform, we can allow for shareability of the computer vision models such that a researcher can whip up their own custom CV modules based on a vast array of pre-made and pre-verified models that can be tuned to pick up on the specific inputs they are looking for. This network of research paper-driven models, known procedures, and easily repeatable experimental design means that every study makes data collection more accessible and transparent.

The end goal of StreamCare is to create a complete toolkit for researchers to manage their data collection, participants, and eventually their procedure and grant writing. By having all the aspects of the research process neatly packaged into one application, researchers are able to dedicate more time to the scientific process instead of having to clean messy data and constantly contact participants for regular check-ins.

The future of accessible medical data collection will take place away from the hospital. And StreamCare makes sure the data collected there is data researchers can trust.

Citations

Caruana EJ, Roman M, Hernández-Sánchez J, Solli P. Longitudinal studies. J Thorac Dis. 2015 Nov;7(11):E537-40. doi: 10.3978/j.issn.2072-1439.2015.10.63. PMID: 26716051; PMCID: PMC4669300.
Mein, G., Johal, S., Grant, R.L. et al. Predictors of two forms of attrition in a longitudinal health study involving ageing participants: An analysis based on the Whitehall II study. BMC Med Res Methodol 12, 164 (2012). https://doi.org/10.1186/1471-2288-12-164
Hart J. Harvard Study of Adult Development: Human Connection is Key to Health and Well-Being. Integrative and Complementary Therapies. 2023;29(3):122-124. doi: 10.1089/ict.2023.29074.jha
Tobler AL, Komro KA. Contemporary options for longitudinal follow-up: lessons learned from a cohort of urban adolescents. Eval Program Plann. 2011 May;34(2):87-96. doi: 10.1016/j.evalprogplan.2010.12.002. Epub 2010 Dec 30. PMID: 21338843; PMCID: PMC3061838.
Teague S, Youssef GJ, Macdonald JA, Sciberras E, Shatte A, Fuller-Tyszkiewicz M, Greenwood C, McIntosh J, Olsson CA, Hutchinson D; SEED Lifecourse Sciences Theme. Retention strategies in longitudinal cohort studies: a systematic review and meta-analysis. BMC Med Res Methodol. 2018 Nov 26;18(1):151. doi: 10.1186/s12874-018-0586-7. PMID: 30477443; PMCID: PMC6258319.