Inspiration

Data pipelines are often rerun on a CSV that only grew a little: a log file, a daily export, a live feed. Pandas reloads and recomputes everything every time, even when 99% of the rows haven't changed. We wanted to keep writing ordinary pandas code and have the rerun cost scale with the new rows instead of the whole file.

What it does

IncPandas is a drop-in replacement for pandas. You change one line, import incpandas as pd, and keep your code as it is.

It records your operations into a plan instead of running them right away.
On the first run it computes the result and saves state to disk.
On a rerun it processes only the rows appended to the source CSV and updates the saved state.
Incremental operations include filters, assigns, renames, drops, fillna/dropna, astype, reductions (sum/mean/min/max/count), groupby aggregations, and left/inner merges.
Anything it doesn't support falls back to real pandas, so results are never wrong.
df.explain() shows which steps ran incrementally and which fell back. df.verify() checks the incremental result against a full pandas run.

How we built it

Plan of nodes. Each operation is a node. A node's stream(ctx) returns (df, reset): reset=False means only the newly appended rows, and reset=True means the complete output. Downstream nodes use this to decide whether to update or rebuild.
Saved state per node. Groupby keeps per-group partial aggregates, reductions keep running totals, and merges remember how many output rows they have produced. State is saved on disk and keyed by a signature of the plan.
Correctness first. Every node can also replay itself in plain pandas. With INCPANDAS_VERIFY=1, every compute is checked against that replay, and the test suite runs this way.
Fuzzing. A registry of every operation is run against random CSVs. The fuzzer appends rows, edits rows in the middle, truncates the file, changes a column's dtype mid-stream, and mutates the merge's right-hand table. Incremental and plain pandas must give the same result, or raise the same exception.
Team split. We split the work into two ownership areas with frozen interfaces, so the two of us could build the core operations and the engine/source layer in parallel.

Challenges we ran into

Never being wrong. The hard part wasn't speed, it was knowing when incremental is safe. A new row can change a column's dtype (an int column gains a NaN and becomes float), a file can be edited in place instead of appended, and a merge's right side can change and give old left rows new matches. Each case needed detection and a safe fallback.
Exact dtype and rounding matches. Pandas keeps float32 as float32, so an incremental float64 mean or a float32 sum with different rounding was wrong. We restricted the incremental path to ints and float64.
Pandas version differences. On pandas 2.2.3 and 3.0.5, an inner merge against a right side with duplicate keys orders rows differently depending on whether you merge everything at once or in chunks. Pandas 3.1 fixed it. We added a one-time probe that detects the behavior and recomputes in full when it's unsafe.
Aliasing bugs. A cached in-memory frame was returned directly, so editing a result silently changed the lazy frame behind it.

Accomplishments that we're proud of

A drop-in API that needs a one-line change.
A test suite of about 950 tests that passes on pandas 2.2.3, 3.0.5 and 3.1.0, plus 2,430 mutation-fuzz cases on the two newest versions.
A correctness safety net: unsupported operations fall back to pandas, and the verification mode compares against real pandas on every compute.
[Add your benchmark here, e.g. "a rerun on N rows with M appended went from X s to Y s". I only know that one df.columns call used to cost 0.26 s on 300k rows by running the whole plan, which is now instant.]

What we learned
Incremental computation is mostly about knowing when not to be incremental. The fallback paths mattered more than the fast paths.
Pandas behavior isn't stable across versions, even for something as basic as row order in a merge, so we tested against all three versions.
Fuzzing against the real library found bugs that hand-written examples would have missed.
Freezing the interfaces early let two people work in parallel without stepping on each other.

What's next for IncPandas

More incremental operations: right/outer merges, sort, drop_duplicates, window functions, and more groupby aggregations.
More file formats beyond CSV, such as Parquet and database tables.
Handling edited and deleted rows, not only appended ones.
Merges where the right side also grows incrementally.
Smarter state storage for very large datasets.
Publishing it as a package.