LedgerNova

• 

  TransactGrid

Inspiration

What it does

How we built it

Challenges we ran into

Accomplishments that we're proud of

What we learned

What's next for LedgerNova

Multi-Region Serverless FinTech Ledger & Payment Gateway What inspired this project Two recurring problems in modern financial systems motivated this design: The unacceptable risk of data loss or inconsistency during cloud-region failures in systems using single-region databases or active-passive failovers. Financial systems need RPO = 0 and strong guarantees that money cannot be duplicated or lost. The increasing requirement for global low-latency payments and multi-currency support, while maintaining strict auditability and regulatory compliance (immutable, chronological transaction trails). I wanted to build a practical architecture that combined serverless scalability, distributed-consensus-backed storage, and reliable event streaming so a global fintech service could accept writes in any region with zero data loss and auditable transactions. What I learned Distributed consensus and serializable transactions solve many correctness problems in global ledgers. CockroachDB’s Raft-based replication and serializable isolation made it straightforward to reason about invariants like “total supply invariant” and per-wallet atomicity. Serverless functions (AWS Lambda) simplify multi-region compute deployment and scale elastically; however, designing idempotency, retries, and cross-region consistency for financial transactions still requires careful protocol design. Event streaming (Kafka/MSK) is invaluable not only for audit logs but also for asynchronous reconciliation, analytics, and eventual consistency jobs—treating the stream as a system of record for events derived from committed transactions. Locality matters: pinning hot-currency balances to regions greatly reduces read/write latency for customers in that region while keeping global correctness via CockroachDB’s geo-partitioning features. Testing distributed systems is hard: fault injection, network partition simulation, and multi-region chaos tests uncover issues not visible in single-region dev setups. System overview Client requests route to the nearest region using Route 53 (latency-based routing) or Global Accelerator. Regionally deployed AWS Lambda functions handle API requests and interact with: CockroachDB (multi-region active-active SQL) for the canonical ledger and balances. Amazon MSK (Kafka) for streaming immutable audit events and driving downstream processes like reconciliation and analytics. Key properties: ACID and serializable isolation at the database layer ensures funds are never duplicated and concurrent transfers are safe. Kafka captures every wallet state change for auditing and supports automated reconciliation consuming the same events. Active-active deployment accepts reads/writes in every region and survives an entire region outage without data loss. How I built it High-level components API layer: AWS Lambda + API Gateway (or ALB with Lambda targets), deployed in each region. Functions validate requests, enforce idempotency, and start transactions. Database: CockroachDB cluster spanning 3+ AWS regions, configured with geo-partitioning and zone configurations to keep data local when possible. Event stream: Amazon MSK cluster (or managed Kafka in each region forwarding to a central topic) that receives audit events after DB commit. Audit service: A lightweight consumer that stores immutable events in a write-once store (e.g., S3 with versioning + object lock) and indexes them for search. Reconciliation: Periodic jobs (Lambda or Fargate) consuming Kafka to compare ledger state with event-derived balances and generate alerts for discrepancies. Observability: Prometheus-compatible metrics, distributed tracing (OpenTelemetry), and structured logs pushed to a centralized monitoring stack. Transaction flow (simplified) Client submits a transfer request to nearest region. Lambda authorizer validates auth, rate-limits, and computes an idempotency key. Lambda opens a serializable transaction in CockroachDB and runs: Read source and target balances (with row-level locality where applicable). Validate sufficient funds and account states. Write debit and credit ledger rows and update balance aggregates. Commit the DB transaction (CockroachDB coordinates Raft across replicas). On successful commit, Lambda publishes an append-only audit event to Kafka containing transaction details and DB commit metadata (transaction timestamp, IDs). The API returns success to the client only after DB commit and Kafka enqueue succeed or are reliably retried (ensuring the audit trail is synchronized with state). Schema design highlights (conceptual) accounts(account_id PK, currency, owner_id, metadata, updated_at) balances(account_id, currency, balance, region_hint) with unique constraint (account_id, currency) transactions(tx_id PK, from_account, to_account, amount, currency, status, created_at, committed_at) ledger_entries(entry_id PK, tx_id FK, account_id, delta, balance_after, timestamp) idempotency_keys(idempotency_key PK, request_hash, result_ref, created_at) Design notes: Use SQL transactions to create ledger_entries and update balances atomically. Persist idempotency records within the same transaction scope or use “upsert-if-not-exists” patterns to avoid double processing. Use range constraints and geo-partitioning to pin hot currency rows to specific regions. Idempotency and retries Require clients to provide idempotency keys OR compute deterministic idempotency from request contents and client credentials. Store idempotency result references (success/failure and tx_id) and return cached result for repeated keys. Implement exponential backoff and bounded retries in Lambdas; rely on CockroachDB’s serializable retries for transient conflicts. Kafka & Audit pipeline Produce an immutable audit event after DB commit. Include the DB transaction timestamp and a strict monotonic sequence (CockroachDB’s commit timestamp helps). Configure Kafka topics with replication across regions (MSK multi-AZ in each region + replication or MirrorMaker for cross-region durability). Archive events to S3 with object-lock enabled for regulatory immutability and store index metadata in a search engine for quick lookups. What I learned while building (concrete takeaways) CockroachDB’s serializable isolation made reasoning about invariants easier, but write latencies can rise when cross-region Raft quorums involve geographically distant nodes; geo-partitioning was essential to keep latencies acceptable. Ensuring end-to-end durability (DB commit + Kafka produce) requires careful ordering and failure handling. The safest pattern is: commit DB first, then produce event; if produce fails, use a reliable local dead-letter queue + background publisher that re-publishes until success, while the transaction remains authoritative. Idempotency must be stored durably and consistently with transactional changes—cheap eventual solutions led to rare doubles during retries. Multi-region chaos testing (simulating entire region loss) validated RPO = 0 and exposed edge cases around DNS failover and client-side retries. Observability is non-negotiable. Traces linking client request → DB transaction → Kafka event helped debug discrepancies. Challenges faced and how I solved them Challenge: Cross-region commit latency Solution: Use geo-partitioning to keep hot state local; only cross-region transfers incur the additional consensus latency. Implement optimistic UI patterns for some reads and surface estimated ETA for cross-region transfers. Challenge: Ensuring audit event persistence tightly follows DB commits (no committed state without an audit event). Solution: Commit-first, then produce to Kafka with synchronous acknowledgement. If produce fails, persist the event in a durable local store and run an at-least-once background forwarder. Include transaction commit metadata so consumers can correlate and deduplicate. Challenge: Idempotency and double-spend risk during retries Solution: Enforce server-side idempotency records stored transactionally and use unique tx_id derived from idempotency keys. Use CockroachDB upsert semantics combined with a uniqueness constraint to guarantee single execution. Challenge: Testing region failures and split-brain scenarios Solution: Build automated chaos tests that: Kill regional network routes and observe failover. Partition nodes and verify Raft leader election and linearizability. Replay large batches of transactions to ensure invariants hold. Use integration tests against a multi-region staging cluster to validate behavior before production rollout. Challenge: Regulatory immutability for audit logs Solution: Archive Kafka topics to S3 with object-lock (WORM) and store checksums. Maintain a compact, signed index (or Merkle tree roots) to prove event lineage and integrity. Metrics, SLOs, and monitoring Availability: 99.999% (multi-region active-active aims for minimal impact during region loss) RPO: 0 (CockroachDB + Raft) Latency SLOs: Regional transfers median < 50ms; cross-region writes median < 200–400ms depending on geography Observability: End-to-end tracing for each transaction (API → DB → Kafka) Metrics for DB commit latency, Kafka produce latency, Lambda cold starts, idempotency cache hit rate Reconciliation discrepancy rate and time-to-rectify alerts Example mathematical invariant checks Use simple checks during reconciliation to detect corruption or duplication. For example, total currency supply invariant: Let accounts_i be balances for currency C. Then at any time:∑i▒  〖"balance" 〗(i,C)=〖"issued" 〗_C-〖"burned" 〗_C Run periodic assertions of this invariant across shards and verify event-stream-derived deltas match DB aggregates. Future improvements Move the audit-event write into a transactional outbox pattern within the DB to avoid commit-first/produce-after logic, using a background forwarder to push outbox rows to Kafka. Implement CRDTs for some eventual-consistency use-cases where strict serializability is unnecessary, reducing global coordination. Add more advanced policy-based geo-routing for currencies and legal compliance per jurisdiction. Explore CockroachDB serverless offerings (if available) for managed operational overhead reduction.

Built With

• and-any-analytics-or-testing-utilities-around-the-ledger-pipeline.-?-sql-for-ledger-modeling
• and-low-latency-handling-matter.-your-prior-project-work-has-repeatedly-centered-on-go-microservices
• and-sql-backed-systems
• automation
• clean-apis
• data-validation
• especially-for-cloud-native-and-interview-ready-architecture-stories.-languages-?-go-for-backend-services-and-microservice-style-transaction-logic
• especially-where-concurrency
• for-this-multi-region-serverless-fintech-ledger-and-payment-gateway
• plus-cockroachdb-and-kafka-for-the-ledger-and-audit-trail.-that-matches-your-broader-project-history-of-go-microservices
• python-automation
• reconciliation-jobs
• the-stack-i-used-was-go/python/sql-at-the-application-and-data-layers
• transactional-reads/writes
• which-makes-it-a-natural-fit-here.-?-python-for-glue-code
• with-aws-for-serverless-compute-and-global-routing