-
-
Synthetic example: 3.5 positive seconds / 8 reviewed = 43.75% occupancy; whole-recording bounds 35–55%, retaining unresolved time.
-
Held-out cup-mask pilot: fine-tuning raised AP but lowered F1 at score 0.5, so we retained the pretrained model.
-
Observe: a research-video workbench for inspecting, correcting and reproducing behavioral observations.
-
Cells: inspect proposed nucleus fragments, source images and recorded coordinates. Linked fragments remain provisional.
Submission update in progress: The video currently shown is an unrelated temporary placeholder, not an Observe walkthrough. [FINAL DEMO VIDEO — pending] [GALLERY IMAGES — pending]. The linked repository is private; judge access is being arranged.
Try Observe: public saved demo — available without an access code until October 25, 2026, 8:24 p.m. Eastern (October 26, 00:24 UTC). Start with cell replays and the exact saved ERK answer, then explore Objects in motion through the sliding-bottle demonstration. This public demo permits viewing saved evidence; new inference, uploads and paid AI generation are unavailable.
Inspiration
A recording is not yet a research dataset. Someone still has to locate objects, inspect uncertain frames, decide which events count, and connect measurements to a question. A convincing overlay can conceal missing observations; a fluent explanation can conceal unsupported assumptions.
We built Observe around the researcher's next decision: where did this number come from, what remains unresolved, and what should I inspect next? Observe brings Cells and Objects in motion into one research workbench. We lead with cell movement, then demonstrate physical-object tracking and measurement using sliding bottles. The broader motivation is inspecting motion in physics experiments and recorded lab processes. Other objects or chemistry workflows would need an appropriate detector, calibration and domain validation; they have not been demonstrated here.
What it does
Observe connects research video to inspectable measurements and source-linked evidence through two workspaces: Cells, followed by Objects in motion. Researchers inspect source frames, select proposed cell fragments or objects, explore measurements, and export evidence.
Cells
Follow a proposed fragment, inspect its recorded coordinates, and return to the source image. Numbered detections and trajectory tables stay connected to their original frames; provisional fragments remain distinct from verified biological identities.
Objects in motion
Track, measure, compare and verify at the source. The current physical demonstration uses sliding Gatorade and Stanley bottles. Its tools separate known-length calibration from travel direction and expose positions, displacement, interval velocity and fitted derivatives. This demonstrates a measurement workflow for these recordings; applying it to another object requires a suitable detector, calibration and validation.
Event review supports object-in-region and hand-near-object proposals. Hand proximity means projected image distance, not physical contact. Researchers can correct boundaries, accept or reject proposals, add missed events, and separately mark scorable, unknown, or unreviewed coverage. Acceptance does not silently review surrounding negative footage. Whole-recording occupancy bounds retain unresolved time; they are not confidence intervals.
Keyboard controls, precise numeric inputs, semantic tables, and readable reports accompany the visual timeline. Camera evidence still needs visual inspection, potentially with a collaborator. We have checked browser and keyboard behavior, but have not demonstrated screen-reader usability or researcher benefit in a user study.
The research workspace takes a question and hypothesis and binds its evidence to the selected measurements. Current demonstrations include explicitly labeled saved Codex interpretations with source citations and an evidence-packet export/import workflow for new questions. Automatic live-provider generation remains a separate completion task.
Application: behavioral research
A clearly labeled simulated trajectory illustrates how interval selection, coordinates and missing observations could support behavioral research. It is an authored workflow illustration; no animal tracking, biological effect or researcher benefit has been validated.
How we built it
The current Observe application is newly authored, informed by requirements and failure cases from an earlier Observe prototype. That implementation history does not establish when the broader project began. External libraries, public datasets, and pretrained weights retain their own provenance.
React, TypeScript, and Vite power the interface; Python and FastAPI handle persistence and deterministic measurement. PyAV/FFmpeg preserve decoded source presentation timestamps and time bases. YOLO segmentation, MediaPipe hand landmarks, StarDist nucleus segmentation, and the VISEM authors' detector serve different tasks. A trained bottle specialist is explicitly selected; it does not turn the system into a universal vision model. The language model interprets supplied evidence rather than calculating measurements.
Immutable inference runs preserve source/model identities and settings. Human changes have revisions; undo adds history. Ordinary analysis exports reference the source video by hash without bundling it, and reproduce saved events, coverage, and motion without neural inference or networking.
The full research workspace keeps review and project storage local. Local inference is available; explicitly selected remote processing transfers recordings or preview frames to the worker. Colab batch mode requires manual source transfer. Cloud failure never silently starts a laptop CNN. The separately hosted public demo serves a fixed saved-evidence collection and blocks uploads, new inference and paid AI generation. Anonymous HTTP checks passed for its ten bottle recordings, three cell viewers and exact saved ERK answer; the public URL also loaded on a phone over cellular. That phone check establishes access, not validation of the full mobile workflow.
Challenges
The hardest problems appeared at the boundary between a plausible measurement and a defensible conclusion. A clipped bottle silhouette cannot supply a complete-object centroid. We therefore retain a separately identified visible-body reference, with its own assumptions, rather than silently treating it as the whole bottle.
Timing and missing evidence also matter. Gaps cannot be filled with invented positions, and fitted derivatives must remain distinguishable from observed coordinate changes. In microscopy, a source field-of-view change breaks fragments instead of implying a cell traveled across it.
Finally, a model can improve one metric while making the workflow worse. We preserved adverse results, alternate settings, and withheld conclusions instead of treating every successful run as validation.
Accomplishments
For microscopy, selecting thresholds on 28 calibration images improved StarDist mask F1 from 79.69% to 83.67% on 14 reserved confirmation images containing 798 reference nuclei, at mask IoU at least 0.50. Weights were unchanged. This separation applies to our threshold experiment; upstream training overlap and specimen independence remain uncertain. False positives fell from 283 to 135, while misses rose from 82 to 127. These are static-mask results, not movie-tracking accuracy.
On one previously inspected image divided into 80 regions, original StarDist settings classified 79 correctly, versus 53 for the fixed CLIP-L/14 ensemble. CLIP's post-hoc best of six tested prompt pairs reached 76; simple Otsu thresholding reached 80. The newer StarDist settings reached 75. StarDist used full-image context while CLIP saw crops. This illustrative comparison supports task-specific evaluation, not generalized superiority.
For the Objects in motion demonstration, the selected bottle pipeline processed all 476 native frames across ten trials of two individual bottles. It retained 447 usable body positions: availability, not localization accuracy. Saved fits use 24 samples per trial and actual source times. They support conditional deceleration estimates; physical assumptions remain unverified, so friction coefficients stay unavailable.
We also completed real Colab fine-tuning: 12 epochs on a T4, with 450 training, 270 validation, and 270 test images separated by physical object identity. The test contained 180 mug and 90 bowl frames, from only two mugs and one bowl. Mask AP50–95 rose from 0.8180 to 0.8592, but F1 at score 0.5 fell from 0.9778 to 0.8802; bowl false-alarm frames rose from 1/90 to 49/90. That candidate was not promoted; the official pretrained nano remains the general upload default.
What we learned
Reliable research software needs explicit boundaries around its claims. Reproducing an export establishes consistency, not biological or physical truth. More usable frames do not establish more accurate positions. Fewer false detections do not establish less human work.
The smooth VISEM replay illustrates another boundary: all 1,470 frames of its 49-fps source were processed, but the sample comes from a training directory. Its locations are detector-box centers, distinct from the nuclear masks in the ERK view. Its reference agreement is a replay check, not held-out generalization or clinical validation. Likewise, source-grounded language can help organize interpretations and follow-up hypotheses without making those hypotheses established findings.
What's next
We are completing genuine live application answers and the final walkthrough. Evaluation must then address the workflow itself: independent source-position checks, representative recordings, event and boundary errors, unresolved coverage, active reviewer effort, and assistive-technology testing with researchers. Additional domains need their own evidence and validation. Our aim is to make adaptation and inspection easier while preserving the distinction between observation, interpretation, and the next question.
Built With
- artificial-intelligence
- computer-vision
- data
- fastapi
- ffmpeg
- google-colab
- human-in-the-loop
- machine-learning
- mediapipe
- modal
- numpy
- opencv
- pyav
- python
- pytorch
- react
- stardist
- tensorflow
- typescript
- video-analysis
- vite
- yolov8