Netflix AI团队将Cassandra宽分区读取延迟从秒级降至毫秒级

Netflix’s engineering team published a method for handling wide partitions in Apache Cassandra. The research work targets Netflix’s TimeSeries Abstraction, a platform for temporal event data. TL;DR Dynamic partitioning splits wide Cassandra partitions per TimeSeries ID, asynchronously and transparently, with no application changes. Detection runs on the read path via byte counting and a Kafka event; splitting targets immutable partitions first. Bloom filters (single-digit microseconds) plus a cached wide_row metadata lookup route reads to the smaller child partitions. Checksums, retained original partitions, Data Bridge Spark checks, and shadow comparison guard correctness. Reads improved from seconds to low double-digit milliseconds; tail latency fell to ~200 ms; 500MB+ partitions stayed available. What is Dynamic Repartitioning? Netflix’s TimeSeries Abstraction ingests and queries petabytes of temporal event data with millisecond latency. It uses Apache Cassandra 4.x as the underlying storage. Cassandra was chosen for throughput, latency, cost, and operational maturity. Time-series data is organized into partitions that group events by identifier and time range. These partitions can grow ‘wide’ as events accumulate over time. Dynamic repartitioning splits an oversized partition into smaller child partitions asynchronously. Applications keep querying the same logical partition while the storage layout evolves transparently. Why Wide Partitions Hurt Reads For most datasets, average read latency stays in single-digit milliseconds. When partitions grow too wide, tail read latencies rise into seconds. These slow reads can produce read timeouts. In extreme cases, clusters see Garbage Collection pauses, high CPU utilization, and thread queueing. TimeSeries servers also handle very high read throughput, which compounds the problem. Scaling up the Cassandra cluster is always an option. But Netflix team wanted smarter alternatives than just throwing more money at the problem. The Partitioning Strategy Behind TimeSeries TimeSeries breaks datasets into discrete time chunks: Time Slices, time buckets, and event buckets. This lets Netflix team query and drop data by time without creating tombstones. When a namespace (dataset) is created, users specify anticipated workload characteristics. A provisioning pipeline runs Monte Carlo simulations to pick infrastructure and partition configuration. This up-front approach falls short in three situations. Workload can be unknown or inaccurately estimated early in a project. Workload can also evolve as traffic and product requirements change. Finally, data outliers exist, where a few IDs receive far more events. Discrete Time Slices provide a natural escape hatch for the first two cases. Each new Time Slice can use a different partitioning strategy. But manually tuning thousands of datasets is not sustainable, so automation is required. Solution 1: Time Slice Re-Partitioning Cassandra exposes introspection APIs such as nodetool tablehistograms for partition-size percentiles. A background worker monitors these histograms and exposes them through a Cassandra virtual table. It computes an adjustment factor when partition sizes miss a configured density. That density is often set between 2 MiB and 10 MiB, depending on workload. For example, provisioning once selected 60-second time buckets, producing partitions under 10 KB. That over-partitioning caused high read amplification and thread queueing. The worker updates future Time Slices with a corrected strategy: Copy CodeCopiedUse a different BrowserDynamicTimeSliceConfigWorker: namespace: my_dataset_1 Observed: TimeSlices have p99 partitions below configured target of 10MB. Proposed: time_bucket interval: 60s -> 604800s This reduced read latencies and timeouts caused by thread queueing. But it only helps when most of the table warrants re-partitioning. It does not help when only a percentage of IDs are wide. For those partial cases, Netflix team offers three options: Do Nothing: the right approach when top-level metrics show no impact. Partial Returns: aborts an in-flight request breaching a latency SLO, returning data collected so far. Block IDs: an extreme step for test or spam IDs that destabilize the system. Block IDs is applied through configuration, listing the offending TimeSeries IDs: Copy CodeCopiedUse a different Browserdgwts.config.<dataset>.block.Ids: "<tsid-1>, <tsid-2>, <tsid-3>" These options do not help when valid, important IDs grow wide. Those callers still need every event, which is where Solution 2 applies. Solution 2: Dynamic Partitioning per ID Dynamic partitioning is an asynchronous pipeline that splits wide partitions per TimeSeries ID. It operates at the ID level, not the table level. It has three stages: Detection, Planning & Splitting, and Serving Reads. Detection happens on the read path. Every read tracks bytes read for a partition. When bytes exceed a configured threshold, the server emits an event to Kafka: Copy CodeCopiedUse a different Browser{ "time_slice": "data_20260328", "time_series_id": "profileId:123", "time_bucket": 7, "event_bucket": 2, "immutable": true, "version": "0" } Here, immutable marks a partition no longer receiving writes. The version field is reserved for future use, such as invalidation. Netflix team detects on reads, not writes, because most data never needs splitting. Some reads on wide partitions stay slow for seconds until the pipeline catches up. The initial implementation targets immutable partitions to reduce complexity. Planning reads the entire partition once to compute an accurate split plan. Checkpointing lets failed planning reads resume from the last saved point. The wide_row metadata table stores split states, checkpoints, and routing information. Splitting delegates to a strategy such as EventBucketPartitionSplitStrategy. It assigns more event buckets to the same time bucket. For ultra-wide partitions, it caps event buckets to control read amplification. Spreading across partitions still distributes reads over Cassandra replicas. Validating compares a pre-split checksum against a post-split checksum. A split is marked COMPLETED only when both checksums match. Netflix also tracks pre- and post-split partition sizes to confirm splits are sized well. The Read Path: Bloom Filters and Metadata Routing TimeSeries servers periodically load completed split partition-keys into in-memory Bloom filters. Every read checks the Bloom filter, which responds in single-digit microseconds. That check is small enough to be practically invisible to callers. On a hit, the server reads wide_row metadata to route the query: Copy CodeCopiedUse a different Browser{ "pre_split_data": { "time_slice": "data_20260328", "time_series_id": "6313825", "time_bucket": 0, "event_bucket": 2 }, "post_split_data": { "time_slice": "wide_data_20260328_0", "event_bucket_partition_strategy": { "target_event_buckets": 2, "start_event_bucket": 32 } } } That metadata read is backed by a read-through cache. The existing PartitionReader then serves reads from the smaller split partitions. Reusing the same schema minimizes code changes, and results are merged before returning. The original wide partition is never deleted, providing a safe fallback for partial failures and eventual consistency. Netflix team also verified splits offline using Data Bridge Spark jobs. A phased rollout advanced through read modes as confidence grew. Its Comparison phase compared bytes served by the old and new read paths in shadow mode. Solution 1 vs Solution 2: Comparison DimensionTime Slice Re-Partitioning (Solution 1)Dynamic Partitioning per ID (Solution 2)GranularityTable / Time Slice levelIndividual TimeSeries ID levelTriggerBackground worker on partition histogramsBytes read exceed a threshold on the read pathScope of effectFuture Time Slices onlyExisting immutable wide partitionsCore mechanismAdjust time_bucket interval / target densitySplit into child event-bucket par

原文超出正文长度上限，此处截断——上游还有内容，完整版见上方「原文 ↗」。