Data Systems at Scale

Why this is hard

The unglamorous research, and the one the rest depends on. When stock moves and a voucher posts in the same transaction, or a million threshold rows are recomputed on every write, correctness under concurrency stops being theoretical. We work on the patterns that hold: transactional boundaries that actually bound, set-based recomputation instead of row-at-a-time, and freshness that does not cost consistency.

Questions we are working on

1. What genuinely belongs inside one transaction, and what is a queue pretending to be one?
2. When does set-based recomputation beat incremental maintenance, and at what row count does that flip?
3. How do you make a batch safe to retry without building a distributed lock nobody can reason about?
4. What is the cheapest architecture that still delivers sub-second freshness without a broker in the path?

What comes out of it

• Idempotent, all-or-nothing batch processing with per-entity deduplication
• Set-based recomputation over millions of rows in place of row-at-a-time maintenance
• Event delivery evaluated inside the writing transaction rather than by a sweeping job
• Column-level lineage and contract tests that fail a pipeline before a dashboard reports a wrong number

Where it ships