ByteQL

• 

  Homepage

• 

  Audio play

• 

  Byte provenance

Inspiration

Anyone who works near the wire knows the ritual: you have a binary file—a packet capture, an event log, a MIDI file, some proprietary telemetry—and a simple question about it. Which IP addresses talked the most? Are DNS queries hiding inside TCP streams? What changed between these archives?

The answer is usually a format-specific GUI, a conversion pipeline that strips away the original byte context, or another parsing script that will be thrown away after the investigation. Wireshark is superb for packet captures, but its interaction model does not generalize to every binary format. JSON is convenient, but once the conversion is complete, getting from a suspicious result back to the bytes that produced it is often painful or impossible.

What I wanted was simpler: ask the file a question in SQL, then prove the answer against its source bytes.

Distribution and privacy explain why that tool is difficult to build. Native tools require installation and trust. Cloud applications are easy to access, but DFIR analysts, protocol engineers, and consultants often cannot legally or ethically upload evidence. The browser offers a compelling third option: zero installation, a strong sandbox, local file access, WebAssembly, workers, and origin-private storage. The hard part is making all of those pieces behave like a serious data system.

GPT‑5.6 changed the feasibility of attempting that during a build week. I brought the product thesis, audience, privacy boundary, provenance model, and architecture. Using GPT‑5.6 Sol through Codex, I could turn those decisions into specifications, implementation plans, tests, and a working product without losing the original constraints as the codebase expanded.

ByteQL is the tool I always wanted: SQL for binary files, entirely in the browser.

What it does

ByteQL turns binary files into relational tables you can query with full DuckDB SQL. Today it supports classic pcap, Standard MIDI Files, and structural ZIP analysis. You can open one file or a same-format batch and run queries such as:

  SELECT src_addr, count(*) AS packets
  FROM ip
  GROUP BY src_addr
  ORDER BY packets DESC;

The public application at byteql.dev is a complete local analysis workbench: format detection, streaming intake, table explorer, saved queries, SQL editor, virtualized result grid, row inspector, diagnostics, and a canvas hex pane.

Its signature feature is bidirectional byte provenance. Every projected row carries its source file and exact byte interval. Select a SQL result and ByteQL highlights the bytes that produced it. Select bytes in the hex pane and ByteQL finds the rows that cover them. It is Wireshark’s most powerful trust-building interaction, generalized to other binary formats and connected to relational queries.

Privacy is architectural rather than contractual. Parsing happens in killable workers, DuckDB runs through WebAssembly, and large intermediate tables spill to Parquet in the browser’s origin-private file system. The file is never sent to a server. Automated browser tests assert that no requests occur after application readiness, and the bundle audit rejects remote URLs, CDNs, fonts, analytics, and runtime-loaded code.

MIDI provides the joyful version of the same idea: run a playback query, change its WHERE clause, and hear the result change. SQL becomes something you can listen to.

How I built it

The architecture is a streaming pipeline connected by one central decision: Apache Arrow IPC at every boundary.

A random-access byte source feeds a format-specific container framer. Kaitai Struct-generated parsers and focused wrappers decode individual records inside a worker. A declarative projection engine turns the resulting trees into Arrow record batches, attaches exact provenance, and chains protocol dissectors. Small datasets remain in memory; larger datasets rotate into Parquet files in OPFS. DuckDB-WASM queries both tiers, while the Svelte interface renders the resulting Arrow data.

The format-pack model separates reusable engine behavior from format knowledge. A pack combines a binary grammar, a YAML table-projection specification, canned SQL queries, and a thin façade. The pcap pack declaratively chains Ethernet, IPv4/IPv6, TCP/UDP, DNS, ICMP, ICMPv6, and TLS parsers. Adding another protocol changes the pack rather than the query engine.

My collaboration with Codex began before implementation. I wrote the initial product thesis and used adversarial multi-model critique plus a failure-inversion exercise—assume the product failed, then explain why—to harden it into the PRD. I made the key decisions: browser-only operation, SQL plus provenance, Arrow everywhere, declarative format packs, DFIR as the beachhead, and MIDI as the validating slice.

I then used GPT‑5.6 Sol at high reasoning for the architecture-critical Phase 0 work. Codex helped turn the PRD into approved designs and executable plans, then implemented the first complete path: MIDI framing, byte normalization, parsing, projection, Arrow IPC, DuckDB-WASM, session recovery, the query workbench, and audio playback. Once those contracts were stable, I moved Sol to medium reasoning for narrower, non-critical implementation tasks. Other models and tools later helped with supporting plugin work, after the base architecture and first use case were operational.

I returned to GPT‑5.6 Sol for the final UI/UX, deployment automation, and demonstration workflows. Those tasks benefited from the model’s design judgment and its ability to inspect an entire product experience rather than treating each component in isolation.

The development loop ran through Codex using the Superpowers skills framework:

brainstorm → approved design → implementation plan → test-first build → review → verification

The repository preserves that process in PRD.md, AGENTS.md, committed design records, implementation plans, co-located tests, browser acceptance suites, and post-implementation notes. Codex was especially effective at carrying constraints across phases. My “nothing send to the cloud” requirement became bundle audits, privacy tests, same-origin DuckDB extensions, locked-down database configuration, and guarded release scripts.

I planted the seed and retained product and engineering judgment. Codex made the scope feasible, took the ideas further, and converted promises into executable evidence.

Challenges I ran into

The browser follows performance rules nobody puts in the happy-path tutorial. To keep parsing responsive, I periodically yielded to the event loop. That polite-looking code triggered Chromium’s nested-timer clamp and silently cost roughly forty seconds per gigabyte. The slow part of the pipeline was literally the code doing nothing. Replacing the timer-based yield with an unclamped scheduler reduced the 1 GB parse from roughly 80 seconds to 44.25 seconds.

My privacy requirement repeatedly called my bluff. MIDI playback sounded flat, but realistic soundfont libraries fetched assets from CDNs or would have broken the bundle budget. Codex helped turn the restriction into a local synthesis system: the 128 General MIDI programs map into eight oscillator-based voice families, with separate synthesized percussion routing. No samples, downloads, or new runtime dependencies.

Deployment exposed the same tension. DuckDB normally loads its Parquet extension from an external host, while ByteQL promises no off-origin runtime access. The release path now mirrors the signed extension under the application origin, compresses oversized WebAssembly artifacts, verifies the deployable directory, and only then allows Wrangler to upload it.

The most important challenge was learning what not to trust. An AI can produce excellent code and still confidently report a quality gate as passing when it is not. “Done” is a claim, not a fact. Every major requirement therefore ends in something outside anyone’s confidence: a unit test, browser test, bundle inspection, benchmark, or artifact verifier.

Accomplishments that I'm proud of

• ByteQL is a deployed, runnable product—not a parser demo—with three format packs, same-format multi-file sessions, SQL editing, result inspection, and bidirectional hex provenance.
• A 1 GB packet capture becomes queryable in 44.25 seconds in a browser tab, below the project’s 60-second target.
• A three-column query over a 4 GB capture reads only 1.71% of the original capture through Parquet projection and predicate pushdown.
• The engine performs bounded TCP stream reassembly across out-of-order segments and retransmissions, producing clean rows for multi-segment TLS ClientHello and DNS-over-TCP messages.
• Privacy is a CI-enforced property: automated checks reject external bundle references and assert zero post-readiness network requests.
• Every result can carry file-qualified byte provenance through SQL, the grid, the inspector, and the hex pane.

What I learned

GPT‑5.6 is excellent at the kind of work humans find exhausting: following binary layouts, preserving offset conventions, tracing state through multi-step implementations, and checking many similar edge cases without getting bored.

But models are strongest on the details you make explicit. The sharpest bugs live inside assumptions nobody wrote down.

My first MIDI playback sounded like a funeral dirge because the audio library interpreted a bare number as hertz, while the code was feeding it a MIDI note number. Every song came out more than two octaves too low. The model had implemented the requested transformation; the unwritten unit assumption was wrong.

I also learned to allocate model effort deliberately. Sol at high reasoning was valuable while architecture and invariants were still fluid. Medium reasoning was faster and sufficient once tasks became bounded by stable interfaces and tests. Sol became valuable again when the work required whole-product judgment: UI polish, deployment behavior, and demonstration design.

Most importantly, I learned that provenance may be the feature people remember. Tables are useful. Tables where every result can point back to its exact evidence feel like a superpower.

What's next for ByteQL

The immediate format gap is pcapng, which modern capture tools commonly produce by default. The current networking pack supports classic pcap, so pcapng is the first compatibility problem many users will encounter.

From there, the beachhead is digital forensics. I want ByteQL to parse Windows event logs and run hundreds of Sigma detection rules locally in the browser. Same-format multi-file sessions already work; the next step is mixed-format sessions that can join the packet capture, event log, registry hive, and archive from one incident.

Finally, I want to open the format-pack registry so the community can publish reviewable format definitions built primarily from a grammar and projection specification. Formats that require real code will use the sandboxed component boundary: bytes in, Arrow out, with no network or ambient file-system access.

No cloud upload. No opaque native binary. Query the file, then prove the answer!

Built With

• codex
• duckdb
• gpt-5.6
• kaitai
• svelte
• tone.js
• typescript
• wasm