Uber has unveiled a novel approach to tackle one of the most stubborn challenges in distributed data processing: running exact COUNT(DISTINCT) queries on high-cardinality metrics that don't benefit from rollup tables. The ride-hailing giant's engineering team detailed the solution in a recent technical post, addressing a pain point that has long plagued analytics engineers working with massive, non-aggregated datasets.

For years, teams have struggled with the trade-off between accuracy and performance when counting unique values across billions of rows. Approximate algorithms like HyperLogLog offered speed but at the cost of precision, while exact counts often resulted in query timeouts or exhausted cluster resources. Uber's new methodology promises to change that equation for good.

Why Exact COUNT(DISTINCT) Is So Hard at Scale

In a distributed data pipeline, COUNT(DISTINCT) requires shuffling all unique values across nodes to eliminate duplicates, a process that becomes prohibitively expensive when cardinality is high and no rollup summaries exist. Rollup tables pre-aggregate data, making counts trivial, but they force teams to sacrifice granularity and flexibility.

Uber's engineers found that for "non-rollup metrics"—those that need to stay raw for deeper analysis—the standard approaches broke down. "The naive implementation would either run out of memory or take hours to complete, which is unacceptable for real-time decision-making," the team noted in their write-up. Their new technique focuses on smart partitioning and data skew handling to keep exact counts both feasible and fast.

The Core Idea: Break the Problem Apart

Rather than treating the entire dataset as one monolithic block, Uber's solution divides data into finer-grained segments based on the distinct key being counted. By pre-filtering and grouping data at the ingestion layer, the system reduces the volume of data that needs to be shuffled during the final count phase.

"We realized that the bottleneck wasn't the counting itself, but the way data was being distributed across the cluster," the engineers explained. "By restructuring how data flows through the pipeline, we can achieve exact results without the usual performance cliff."

The approach also leverages existing infrastructure like Apache Spark and Hive, meaning teams don't need to overhaul their entire stack to benefit from the optimization.

Practical Implications for Crypto and Web3 Data Teams

While Uber's post is aimed at general data engineering, the implications for blockchain and crypto analytics are significant. On-chain data is inherently high-cardinality—think unique wallet addresses, token IDs, and transaction hashes—and often cannot be rolled up without losing critical detail.

  • Wallet analytics: Counting active users per day requires exact distinct wallet counts, which explode in size during market volatility.
  • NFT tracking: Each token is unique, making distinct counts on collections or traits a nightmare for traditional pipelines.
  • DeFi metrics: Liquidity provider positions and swap pairs generate millions of distinct keys that need precise counting for accurate TVL or volume reports.

For projects building on Ethereum or other chains, adopting similar partitioning strategies could mean the difference between near-real-time dashboards and stale, approximate data. Uber's method offers a blueprint that can be adapted to off-chain data warehouses that ingest blockchain data.

How the Technique Works in Practice

The key steps, as outlined by Uber, involve:

  1. Identifying the high-cardinality dimension and partitioning the input data on that key
  2. Pre-aggregating partial distinct counts within each partition
  3. Merging partial counts in a final stage that only handles unique values from each partition

This reduces the shuffle size dramatically, as duplicate keys are already resolved locally before the global merge. The result is a linear scale-out that performs well even with tens of billions of rows.

Performance Gains and Real-World Results

Uber reported that the new strategy allowed them to run exact counts on datasets that previously were impossible to process within their service-level agreements. Although they didn't disclose specific numbers, the team emphasized that the improvement was "orders of magnitude" faster than the previous exact-count implementations.

More importantly, the solution is generic and can be dropped into existing ETL jobs with minimal code changes. This is a boon for data engineers who have been forced to choose between accuracy and speed for years. For a company like Uber, which processes billions of events daily, even small improvements in query efficiency translate to massive cost savings in compute resources.

The open-source community may also benefit, as Uber has a history of contributing back to projects like Apache Hudi and Presto. If the company releases this as a library or plugin, it could quickly become a standard tool in the data engineer's arsenal.

Key Takeaways

  • Exact COUNT(DISTINCT) on high-cardinality, non-rollup metrics is now viable at scale thanks to Uber's partitioning technique.
  • The method avoids the accuracy trade-offs of approximate algorithms and the inflexibility of rollup tables.
  • Crypto and Web3 teams can apply the same principles to on-chain data, improving analytics for wallets, NFTs, and DeFi protocols.
  • Performance gains are significant enough to justify the engineering effort, with minimal changes to existing pipelines.

As data volumes continue to explode across industries, solutions like this are not just nice-to-haves—they're essential for staying competitive. Uber's latest contribution is a reminder that sometimes the biggest wins come from rethinking how we structure data flow, not just throwing more hardware at the problem.