shipcheck

Inspiration

What it does

ShipCheck: Devpost write-up draft Paste each section into the matching Devpost field. Fill the [brackets] before submitting.

Tagline (short description): Pre-judge your hackathon submission before the judges do. Nemotron on Nebius Token Factory checks your repo against that hackathon's own rules and cites file:line for every verdict.

Track: Best Apps and Agents Track

Built with: python, streamlit, nvidia-nemotron, nebius-token-factory, tavily, github-api, pydantic

Links:

Live demo: https://shipcheck.streamlit.app Code: https://github.com/Akul-Sharma009/shipcheck Video: https://youtu.be/ZQZ9ciCOP5Y Inspiration Hackathon entries get disqualified for reasons that have nothing to do with the idea: a LICENSE file GitHub can't detect, a README with no setup steps, a required model that is mentioned but never called, a video that is unlisted or 20 seconds too long. The rules that say all this run to 40,000 characters, and nobody rereads them at 11 PM on deadline night. I wanted the checklist a strict judge would run, available to every team before they submit.

What it does Paste a Devpost hackathon URL and a GitHub, GitLab or Bitbucket repo: the three hosts this hackathon's rules accept. In about 30 seconds ShipCheck:

Reads that hackathon's rules live and turns them into a requirement checklist. It isn't hard-coded to this hackathon: the PayPal AI and Atlassian Codegeist hackathons work unchanged. Checks the repo against every requirement. Facts come from the GitHub API (visibility, GitHub's own license detection, README, commit dates). Everything else gets an AI verdict that must cite the path:line it's based on. Puts failures first, each with one concrete fix, and drafts the missing README section from files it actually read. Optionally checks the Devpost submission too: whether the YouTube video is public and under the time limit, and whether the write-up covers what the rules ask for. Runs in CI too: a GitHub Action writes the report to the job summary, posts it as a pull-request comment that updates on every push, and can fail the build on a failing requirement. Shows progress: rerun after pushing a fix and it marks what's FIXED since the last commit, with a countdown to the deadline in your timezone. Every report has a share link. It's built to avoid the costly error, a false "all clear", which gets you disqualified while you think you're fine. GitHub's facts override the model, and any AI "pass" that doesn't cite a file ShipCheck read is downgraded to "unclear". A README claim can't pass a usage requirement either: if the README says Nemotron but the code calls another provider, that's a fail citing the line.

How we built it All model calls go to NVIDIA Nemotron 3 on Nebius Token Factory through its OpenAI-compatible API, with each job routed to the smallest model that does it well:

Nemotron 3 Super, reasoning off: turns the cleaned rules pages into a structured checklist, and writes README fix drafts. Nemotron 3 Nano, reasoning off: summarizes each relevant file in parallel (about 25 calls per repo): which platforms and models it calls, on which lines. Nemotron 3 Super, reasoning on: the one step that needs reasoning. It decides each requirement from the rules, the GitHub facts and the per-file evidence. Where Token Factory accelerated the workflow:

The OpenAI-compatible endpoint meant the standard openai SDK worked as-is, and swapping models is a config change. Per-request reasoning control (enable_thinking) let me turn reasoning off for extraction. That took the rules step from 69 s to 11 s, and Nano's output from 227 tokens to 25 per call. Fast parallel Nano calls keep a whole-repo check at about 30 s. Tavily Extract reads the Devpost overview and rules pages at runtime. The GitHub API supplies the facts. The UI is Streamlit, and every model output is validated with Pydantic, retried once, and falls back to "unclear" rather than a guess.

Other Nebius services: none. Token Factory is the only one used.

Evaluation I compared ShipCheck with two alternatives on the same data, as planned before building:

the incumbent: paste the repo into one model call a dumb baseline: keyword rules, such as "is there a LICENSE file?" The data was 9 real entries to this hackathon, each pinned to a commit: 63 pass/fail labels (10 true fails), made by independent reviewers that never saw ShipCheck's code or output.

Method Accuracy False-pass rate High-confidence labels (55) ShipCheck 92% 20% 100%, 0 false passes One model call 90% 10% 95% One model call, without GitHub's license field 79% 20% 82% Keyword baseline 81% 30% 85% ShipCheck is the most accurate, and every error it made is on a label the reviewer marked low-confidence (judgment calls like "does the README highlight Token Factory usage"). It does not clearly beat a single model call on false passes: across two runs each scored 10% and 20%, and one item is 10 points. The first run on this set also exposed two ShipCheck bugs; I fixed them and reran everything, and both runs are published. Full method and every error: eval/RESULTS.md.

Challenges we ran into Reasoning tokens silently consume max_tokens. When they run out, the content comes back empty, so a budget problem looks like a broken response. Nano copied example values from the prompt into its answers, claiming files called APIs they never mention. I removed the examples, and any claimed model or service that doesn't appear in the file is now dropped. Rules extraction isn't stable from run to run (16 to 19 requirements from the same page), so rules are cached per hackathon to keep the checklist consistent. Super sometimes ignored an exclusion instruction, so items that can't be checked from the repo (like the track choice) now go through a deterministic filter first. Accomplishments that we're proud of Every verdict carries evidence a person can click and check. The license case: ShipCheck catches exactly the error that a keyword check, and a tired human, both miss. It generalizes: other Devpost hackathons work without code changes. What we learned Use the model for judgement and plain code for facts. Wherever GitHub already knows the answer, asking the model only adds a way to be wrong. Reasoning is a per-step choice, not a global setting. Most of the pipeline is faster and just as good with it off. Decide on the metric and the baseline before building, and grow the test set. On 28 labels ShipCheck looked perfect; on 63 it found two real bugs and showed where a single model call is just as good. What's next for ShipCheck Self-hosted GitLab, and hackathon platforms beyond Devpost. Checking usage by behaviour, not just model IDs and endpoints in the code (e.g. models chosen only through environment variables). Testing the Action end to end on real runners, and comparing reports across a team's PRs. Feedback on Nebius Token Factory and NVIDIA tools Nebius Token Factory

Model IDs are case-sensitive and inconsistent (NVIDIA-Nemotron-3-Nano-30B-A3B vs nemotron-3-super-120b-a12b), and Ultra's ID differed from the announced one. A wrong ID costs a debug cycle. Suggestion: case-insensitive matching, or a "did you mean" in the 404. Reasoning tokens silently use up max_tokens, and the only signal is finish_reason=length with empty content. Suggestion: document it in the quickstart, or add a separate reasoning budget. The reasoning toggle (chat_template_kwargs.enable_thinking) is a big cost and latency win, but it's undocumented. I found it in the model card's chat template. Suggestion: document per-model reasoning controls in the API reference. Reasoning is reported inconsistently. Super returns reasoning_content; Nano returned none but still billed 227 tokens for a one-line answer. Suggestion: add usage.reasoning_tokens. No way to make extraction repeatable: the same page gave 16, 17 or 19 requirements. Suggestion: a seed parameter, or document it if one exists. Positive: Super answered in under a second for short prompts, and all tiers returned valid JSON from prompt instructions alone. Nebius AI Cloud

Not used. ShipCheck needs only hosted inference, not GPUs or training, so Token Factory covered it. No feedback beyond that. NVIDIA Nemotron

Positive: Super cites exact path:line evidence when given line-numbered excerpts, and writes grounded README drafts in about 2 s. Nano (reasoning off) copies example values from the prompt into its answers, which is a silent false-evidence risk. Model cards could warn about this at small sizes. Nano returned prose instead of the requested JSON about once in 30 calls. One retry fixed it every time. Super with reasoning on ignored an explicit exclusion instruction once. Instruction-following on exclusions could be stronger. Tavily

Positive: one Extract call read the overview and the 40K-character rules page, and worked unchanged on other Devpost hackathons. About 6K characters per page was repeated site navigation. Suggestion: a "main content only" extract mode. Did this project exist before the submission period? No. ShipCheck was built from scratch during the submission period: the first commit is on 30 Oct 2026, and the commit history is public.

Built With

• bitbucket-api
• github-actions
• github-api
• gitlab-api
• nebius-token-factory
• nvidia-nemotron
• openai-sdk
• pydantic
• python
• streamlit
• tavily