Inspiration
I spent a lot of time and effort understanding skills and making them work for smoothing out jagged edges and for in-context teaching to handle repetitive tasks. During my time doing this, I regularly checked around for tools to help and ideas surrounding how well models/harnesses can use them. In my searches, there is almost nothing to help with this and no way to know if the model is good at calling them or if your skill is the problem. This is a huge bottleneck in understanding the power of skills. It takes a lot of time and effort to understand how to make them and to figure out if the model is using them correctly. It's hard to tell if you're facing a model quirk, a harness problem, a skill description issue, or a skill body performance issue. Because of this, many people tinker with skills a little, the skills don't work, and they think skills are no good. I want a way to overcome the friction of using skills effectively. This hackathon gives me the motivation to build that.
Ironically, this is likely to be a bunch of skills and a thin website wrapper.
What it does
It evaluates skills and iterates skills for the given model using them until they work. It evaluates models/harnesses ability to actually call skills consistently through out a conversation.
How we built it
I built Skill Issue with Codex and 5.6 sol. Basically everything was done with codex the CLI, the research, the planning, the website, the video. I mumbled into the mic I didn't write the site by hand. Not one line. Most tasks started the same way Codex app voice dictation plus my dictate plan skill. Then deep research and then Codex executing. That was the loop. Deep research uses my own home-rolled skills. 5.6 Sol fans out 5.6 Luna subagents onto narrow research tasks, then aggregates that into one target subject. Harness research for the CLI went through the same path dig into each configuration hard enough that Codex could embed real usage in the tool, then run eval after eval. Codex deep-searched the website design and built it. I just said the style I wanted. 5.6 sol built and deployed the site while I talked into the microphone. The demo video was Remotion and Speechify. Codex did nearly all of it with just my telling it how. The skill-generation loop was really meta. The skill generator was refining the skill, and Codex and 5.6 built me a wrapper harness that refined the refiner. It worked. With a little fiddling.
Challenges we ran into
It was hard to isolate the model and harness, I got it as close as could in the time I had. All harnesses have different configuration and permissions, trying to squash that into the CLI and make it portable was tedious. I started with 10 and ended with 5 because of portability and contamination/permissions issues. I did this in 4 days. I wish i had more time to flesh out the system more.
Accomplishments that we're proud of
I am over the moon at what I got done in 4 days. I answered questions and built something i think is meaningful that fills a gap. I would like to have got more data to come to better conclusions before the dead line, but that is only the start.
What we learned
Models and harnesses are a fickle couple and live in a fragile balance that when just right makes them shine bright or crumble. They have idiosyncrasies and quirks that you need to understand and embrace or you wont be able to make them perform. We see if you want to use skills Codex and 5.6 are unmatched.
What's next for Skill Issue
More data, I need to build up a decent data set to see if patterns emerge, we can see some pairings work and some don't. But it's hard to get much more out of it.
Log in or sign up for Devpost to join the conversation.