Compute

Inspiration This made it difficult for me to evaluate my results. Every time I ran an analysis on the AI, it gave me some numbers but I wasn’t sure whether to believe them. Could this R² be real, or was the model just pattern-matching towards a value that sounded plausible? There’s no way to know without running an analysis myself, which kinda defeats the purpose of why you’re talking to an AI in the first place. It’s also tough when both a correct answer and incorrect one look exactly the same to anyone viewing your chat windows. Rather than focus on the fact I couldn’t trust what I was being told, maybe it’d help if we could get past the whole “plumbing” stage. Imagine having a conversation where each person was sitting in their own room communicating through note-passing. This is more akin to two people sitting at opposite ends of a table sharing views of the same screen. The WebMCP technology enables this since all these tools are on the same page instead of being exposed via an API. Once you understand how to use them (the WebMCP), the rest comes easy, really. What it does In our compute system you are joined by another user (the agent) at some sort of statistical research bench where there’s just as much code as there would be in the lab. You paste your URL into ChatGPT and say “does industry momentum survive out of sample and tx costs?” and the agent loads the data, builds a signal, runs a backtest, then performs a Fama French Regression and writes a little piece with the result. What really matters to me is what goes down when we run that regression — those numbers get sent along to the agent, they render a chart on our end. It’s pretty interesting to watch the process play out as opposed to read a summary of everything afterwards. Everything’s logged. If you want to use the data, you have to reference the steps that led you there. The record_finding tool will check every reference against our logs – we’re not going to let you publish results you’ve never computed! You’ll be able to go back in time while working on your analysis as well. You could drag back the window of dates by two years and hit ‘re-run’. Your agent would make a phone call with this new context (and there wouldn’t be a message, nor a synch!) – but there’d be just one workspace, so no problem here. 5 different datasets in fields ranging from finance to biology, all available through static CSV files. No network requests during runtime, no need for an API key and most importantly everything stays on the page. How we built it Written in TypeScript using React & Vite for the UI with Tailwind CSS styles, then deploying to Netlify as a static site (no backend). This means we have to build our own stats engine, but that’s unavoidable if you’re trying to do anything in the browser! I used Householder QR for least squares regression, Newey-West HAC standard errors for time series regressions, and Benjamini-Hochberg for false-discovery rate control — everything I cross-checked agains SciPy / NumPy for my 166 unit tests. The really neat stuff is how we built up our web layer over WebMCP. Our tools aren’t hard-coded — instead they’re rebuilt based on current state in every workspace change. If you load a dataset, we recompute the schema definitions from what columns are actually in memory at runtime. So you’ll see 4 “cold” tools show-up when the workspace initially loads, more get enabled once datasets have been loaded (thereby enabling 12 tools), even more once we can compute signals (13) and finally 14 possible tools once we’ve defined a backtest. Each tool will return both a compact form of its output that’s used by your agent (capped at 1500 chars), and then a full version designed to be rendered to the human. And we’ll keep that 500-row result from chewing up the model’s context while you get the whole chart! Finally expensive operations require an explicit click as a form of confirmation — trying to do a bootstrap with 2000+ simulations would prompt this confirmation card + wait for the tool call to proceed. Challenges we ran into Firstly we needed the agent to be able to see when there are state changes. In my first iteration the agent was keeping its own cache of the workspace, and the human edits weren’t being applied to their workspaces. As a result everything appeared to be working fine and numbers were silently wrong (which was precisely the failure-mode I designed the system to avoid). So to solve that problem I forced all tools to read the live store at call-time – no exceptions. Secondly provenance that couldn’t be talked-around. When I asked the agent for source references it came up with some pretty nice stuff, including a few things that were made-up. But after I politely asked the agent to explain why it used these citations I realized it wasn’t a mechanism. So now each citation has been matched to an entry in the log, if they don’t match then we won’t allow the write. Thirdly the 1500 character budget. This one proved far more difficult than I thought, in particular as we needed to create test data that would help us find a good set of parameters that worked well together. The initial output format produced huge volumes of numbers and I ended up rewriting how the results were serialized about four different times before the agent was able to correctly infer its reasoning from the input provided. Numerical correctness without any SciPy involved. Doing a Newey-West with the wrong choice of lags was getting me t-stats looking great but being off by 30%. Cost an entire evening. Not understanding what schemas were going to be available. Creating this toolregistry where I didn’t have the faintest idea about all the columns in there meant that my code needed to work on all datasets. Accomplishments that we're proud of Dynamic Tool Surface – Watch your badget rise from 4-12-13 as you work! Citations Enforced! – A little bit o’ code doin’ somethin’ I haven’t seen anywhere else yet… it’s just plain flat out IMPOSSIBLE for someone to fake their results… Stats Engine agrees with SciPy to Numerical Precision – You can run this in your browser tab without needing some fancy-ass server… One of my favorite ones was when the momentum demo concluded that momentum does not survive. I had tons of datasets on which you could apply this strategy successfully. In the end, though, the 2020 crash combined with the 2022 value rotation made all the in-sample benefits vanish into thin air. This was what the tool said after its analysis – and to be fair, that’s exactly how it happened. And if your research bench only tells you about wins, you’re no research bench. What we learned So constraints on the agent matter more than instructing the agent. Everytime we told it how to behave we mostly saw what we wanted. When we could make the bad behavior impossible we always saw that. The two-audience-per-tool-call approach is a really interesting design primitive — once you stop thinking about the tool result as being the same thing as the human-facing output lots of issues around context windows go away. One of the things webmcp unlocks which i underrated was shared mutable state — while i thought the dynamic schema stuff would be interesting, there’s nothing better than having zero protocol. No keys required — browser only! No servers involved! Nothing going on outside this webpage! This is for you if you have some data that you cannot upload to somewhere else. This is your entire product. What's next for Compute Bring Your Own Data – Just drop me a CSV and I’ll do the same thing, because all my stuff is local. There’s no change in the underlying model, it’s just adding a loader and doing some schema inference. More of the Toolkit – Things like panel regressions (fixed effects, IVs), time series diagnostic tests etc. Because this is all driven by an engine that has the architectural capability for it, its really a question of building these components and verifying they work as expected. Exportable / Rerunable Sessions – All the logs contain sufficient info to rerun a session. We’re going to serialise those out too so we can give people runs of findings to execuate. Pre-registration: Make your hypothesis go through the motions first, before seeing data. Now we’ve got the second side of the p-hacking equation (the ‘multiple testing correction’) covered. Now for non-statisticians! All of the above doesn’t depend on any particular property of either finance or scientific research. So long as there’s some sort of trusted party who needs to place their trust in results generated by some sort of agent, they could take advantage of the “shared workspace” and forced provenance.