ForkProbe: AI Skill Selection and Trial-Run Tool

Stop guessing which AI skill works. ForkProbe runs the same task with multiple skills, compares real outputs in a local report, and lets you continue with the winner.

Comment 1

Inspiration

ForkProbe began in June 2026 with a problem we kept encountering in real Agent workflows: there are more AI skills available every week, but choosing the right one is still mostly guesswork.

A skill may have a convincing README, a polished demo, or thousands of GitHub stars. None of those signals can answer the question that matters most to a user: will this skill produce a better result for my task?

Testing candidates manually was slow and inconsistent. We had to discover skills, run them one at a time, find their outputs in different folders, and compare results from memory. By the time we found a good path, we had often spent more effort choosing the tool than completing the task.

That led to the idea behind ForkProbe: do not guess which AI skill will help. Give the same task to several candidates, place their real outputs side by side, and let the user continue with the winner.

ForkProbe was already underway before we joined the hackathon. The hackathon became the moment to push the product further. We asked whether the same comparison experience could work not only for writing, presentations, scientific figures, and research reports, but also for finished webpages. That became the focus of V0.5.

What it does

ForkProbe is an AI skill selection and trial-run tool for Agent workflows.

It turns skill selection into a simple, visible process:

  1. Understand the user's task and recommend a small set of relevant skills or pipelines.
  2. Let the user review and confirm the candidates before any long-running task begins.
  3. Give every confirmed candidate the same input and run them in parallel.
  4. Generate a local HTML report that places the real outputs side by side.
  5. Show an AI Judge recommendation while keeping the final decision with the user.
  6. Continue the Agent workflow using the selected result.

Instead of asking users to trust skill descriptions, ForkProbe lets them compare what the skills actually produce: the writing, the presentation, the scientific figure, the research report, or the webpage they will ultimately use.

Since June, ForkProbe has expanded across several high-frequency scenarios:

  • academic polishing, SCI writing, reviewer responses, and submission materials;
  • naturalization, style rewriting, and reducing AI-flavored writing;
  • finished PPTX deck generation and comparison;
  • paper figures and scientific graphics, including previews and publication files;
  • market research and research reports with sources and supporting evidence;
  • finished webpage and HTML comparison in V0.5.

V0.5 adds a complete webpage comparison experience. When a user asks for a landing page, product site, dashboard, web app, or report page, ForkProbe recommends suitable web skills first. After confirmation, the candidates generate runnable pages in parallel. The report then brings together:

  • desktop and mobile previews;
  • direct links to open each page;
  • editable source files and downloadable source packages;
  • browser quality checks and responsive-layout feedback;
  • generation time and estimated token usage;
  • candidate explanations and AI Judge recommendations.

The user is no longer comparing prompts or screenshots alone. They are comparing complete, inspectable webpage deliverables.

How we built it

ForkProbe is designed as a local-first orchestration layer that works with Agent skills rather than replacing them.

Candidate skills are organized in structured catalogs and selected according to task signals. ForkProbe can work with built-in candidates, local skills, and supported GitHub skills. Each candidate runs in its own workspace with the same task, so the comparison remains clear and the generated files do not interfere with one another.

Different tasks produce very different kinds of results, so we built dedicated artifact pipelines. A writing task produces comparable drafts. A presentation task produces openable PPTX files and slide previews. A figure task produces image previews, editable files, captions, and publication formats. A research task produces reports, sources, evidence tables, and limitations.

V0.5 adds a dedicated webpage artifact pipeline. It packages every candidate as a runnable site, captures consistent desktop and mobile views, checks common browser and responsive issues, and collects the source files. All of that is normalized into the same ForkProbe report experience: review the candidates, understand the trade-offs, choose one, and continue.

The report is generated locally. This keeps the workflow easy to inspect and makes ForkProbe suitable for private writing, research material, internal documents, and unfinished product work.

Challenges we ran into

The first challenge was making comparisons fair. Skills often have different assumptions and working styles, but every candidate still needs to respond to the same user intent and produce a deliverable that can be judged against the same goal.

The second challenge was moving beyond text. Comparing two paragraphs is straightforward; comparing a PPTX, a scientific figure package, a sourced research report, and a runnable webpage requires different previews and quality signals. We had to define what a complete deliverable means for each scenario while keeping the overall experience familiar.

Another challenge was deciding how much authority to give the AI Judge. A score can help users scan several candidates quickly, but quality is not one-dimensional. One user may value scientific rigor, another may value editability, visual style, speed, or cost. We designed the Judge as an adviser, not an automatic decision-maker.

V0.5 also challenged us to make webpage comparison feel like a product workflow rather than a screenshot gallery. A webpage should be openable, responsive, editable, and accompanied by enough context for the user to understand why one candidate may be a better starting point than another.

Accomplishments that we're proud of

We are proud that ForkProbe has grown steadily since June while keeping one clear product idea at its center: compare real results before committing to an AI skill.

The project progressed from text comparison to finished artifact comparison. V0.2 introduced scientific figures, V0.3 added research-report workflows, and V0.4 expanded the anti-AI writing candidate pool. During the hackathon, V0.5 brought finished webpage comparison into the same experience.

What matters most to us is that these are not separate demos. The same product flow now works across writing, presentations, scientific graphics, research, and webpages:

recommend, confirm, trial-run, compare, choose, and continue.

V0.5 also makes the webpage result genuinely useful after the comparison. Users can open the candidates, inspect desktop and mobile versions, review the source, understand the quality checks, and carry the selected version forward as the starting point for the next Agent task.

ForkProbe is open source, local-first, available through GitHub, and packaged as a downloadable skill that users can bring into their own Agent workflow.

What we learned

We learned that the AI skill ecosystem does not only need better discovery. It needs better decision-making.

A directory can tell users what exists. A recommendation can narrow the options. But confidence comes from seeing several skills solve the same real task and understanding the trade-offs between their outputs.

We also learned that users care about the final deliverable, not the pipeline label. A presentation skill should be judged by the deck it produces. A scientific figure skill should be judged by the usable figure package. A web skill should be judged by the page the user can open, inspect, and continue editing.

Most importantly, we learned that keeping the user in the loop makes the workflow stronger. ForkProbe does not try to hide the choice behind automation. It makes the choice easier to understand.

What's next for ForkProbe: AI Skill Selection and Trial-Run Tool

Our next planned scenario is image-generation comparison, where users will be able to review finished visual candidates, generation details, and reusable files in the same report workflow.

For webpages, we plan to add deeper interaction, accessibility, performance, and visual-quality checks. We also want to expand the candidate catalog, make it easier to bring a personal or GitHub-hosted skill into a trial, and create reusable benchmark tasks for more consistent comparisons.

Over time, ForkProbe can become a shared evaluation layer for the growing Agent skill ecosystem: not another place that claims one skill is the best, but a practical way for every user to discover which skill is best for the task in front of them.

GitHub repository · ForkProbe V0.5

Built With

Share this project:

Updates