Scipture

Scripture is an open-source cloud-native streaming system in rust providing the cheapest way to get reliable messaging on cloud object storage with best-in-class durability and provable correctness

Comment

Inspiration

I've been wanting to build something similar to warpstream, open data's log and buffer, confluent's freight, and NATS for years; but only recently have found the specific grounding correctness whitepapers of the overall shape that I want. Confluent's recent Conflux paper really sealed the deal and the latest frontier models are letting me turn my idea into a reality.

What it does

Scripture is designed to be simple to operate and as flexible as it is hardy.

  • You can run a single scripture node or many and they will assemble into a highly-available fleet of scalable workers.
  • You can give scripture one object storage backend or multiple, with the choice to span them or increase durability even further
  • Clients can send data in over http, protobuf, or raw text tcp streams with dynamically-defined parsers. Adding more streaming formats is also designed to be very easy
  • Data streamed in can be automatically landed into parquet on object storage for simple iceberg table creation

How we built it

I used the entire GPT 5.6 family extensively, switching from Sol all the way down to Luna depending on whether I was doing the heavy proof-correctness checking or grunt work of coding from well-written specs.

I started with a deep analysis of the white-paper itself and making tiny probes to test various edge cases until I built-up holylog, an implementation of the correctness kernel described in the Conflux whitepaper, then build scripture to build on top of holylog's primitives.

As I built new things, I used additional codex agents (and other agents) to set up a variety of edge-case testing scenarios on my homelab cluster which includes RustFS running locally but also real R2, S3, and GCS.

I also had to build a completely separate project to help multiple agents on multiple computers tackle the same project in parallel; that project is also open source and is called tracker.

Challenges we ran into

The SETUP for testing at this scale without the resources of a large established company is as much (or more!) work than the project work itself. This is clearly the largest challenge for a project as ambitious as this.

Accomplishments that we're proud of

Genuinely proud of what already works and the correctness testing, especially the formal methods.

What we learned

How capable and concise the GPT 5.6 family is, and how well it pairs with fleets of other coding agents.

What's next for Scipture

More plugins for production and consumption, and of course, more testing!

Built With

Share this project:

Updates