Fast drilldown dashboards from a single Parquet file

Fast drilldown dashboards from a single Parquet file One 40MB Parquet data cube, an 18KB reader, an R2 bucket, and a few unassuming http range requests. Aug 21, 2026 Every month brings a new eruption of clever uses for object storage, easily the most volcanically active corner of non-AI software infrastructure on earth. The most recent lava bomb was Vicent Martí’s writeup of Cursor Origin’s S3 + WAL approach to managing Git repositories at scale. It’s a masterpiece of technical writing, unlike this post. I’ll admit that even before reading it, I was daydreaming about a totally different kind of task where object storage probably just works, this time for customer-facing analytics dashboards. A friend of mine has customer usage data in Iceberg on R2, and wants to show his users some basic charts with filters. He told me he didn’t want to add any more vendors, which ruled out MotherDuck, the cloud-hosted DuckDB database company where I currently work. Well, in analytics, when all you have is object storage, everything looks like a range request. We could probably just roll this kind of data up into a Parquet-backed data cube, and fill out the dashboard with very simple range queries against it, using Hyparquet, a small javascript Parquet reader that runs in the browser. With that, you can serve a real drilldown dashboard with neither a database nor a query engine. The cube could even be tens (or hundreds) of MB, since a correctly laid-out file means you only ever read a few small slices of it at a time. You just need a data pipeline to produce the cubes ~ which is also, it turns out, where all the actual money goes when you do have a real analytical database. The heresy was too good to pass up, since these days I assume DuckDB is the lightweight solution to all my data problems. To test it, I took the well-known NYC 311 service requests dataset I had on my computer ~ about 34 million rows at the request level, 15 or so years ~ and rolled it up into a 40MB Parquet cube with filters for city agency, complaint type, submission type, and borough, plus a single creation-time column for the time series. Then I stuck it on R2. 40MB is big enough to feel the pain of downloading the whole thing. The demo dashboard below reads directly from that file using Hyparquet. The bytes pass through a small Cloudflare Worker on the way, because the free r2.dev URL is rate-limited. The Worker proxies byte ranges and caches them at the edge, which is safe because the file is immutable. To be honest, I was surprised how fast new data loads, given that it forgoes both a real database and a powerful query engine. The UI does all of the actual reading, and it is lightweight enough to embed directly in this post without hurting the page load. The real complexity is almost entirely offloaded to the data cube layout. Try scrubbing the chart or clicking on the rows of the leaderboards. nyc 311 ~ daily requestsall time ~ 0 requests in view0 range requests · 0 KB fetched · 0.0% of the cube so farno filters ~ click a leaderboard row, or brush the chart (click the chart to clear)no filters ~ tap a row or brush the chartby agencyby complaint typeby boroughby channel So, how does this dashboard work? A dashboard like this one is designed to answer a bounded set of analytical questions ~ requests per day, requests per day for one agency, all-time totals by borough. Each question can be answered by GROUP BY queries, so we can precompute them all ahead of time and save each result as its own small table, called a grouping set. Stack all of the grouping sets in one Parquet file, one section per set, and you have a data cube. A grouping set is only useful if it either enables a question to be answered, or reduces the latency of pulling the data. This file has both kinds. The all-time totals feed the leaderboards, and a daily grouping set for every combination of filters provides the data for the line chart. The weekly and yearly grouping sets reduce the number of rows scanned that results from brushing the chart. The same totals could be summed from daily rows, but there are fewer rows to fetch if we precompute by weeks and years.

They are a fixed set of charts ~ events over time, counts or sums by hour or by day, a few filters or leaderboards ~ over data that updates on a coarse schedule rather than in realtime, for their sake as much as yours. From the perspective of latency, the cube size doesn’t matter, but you’ll want it to be somewhat small anyway since you’re regenerating one per customer on a schedule. The time grain is clearly dominant in my example, since the daily sections account for most of the bytes of the file. Cardinality is the other multiplier ~ complaint type has 485 distinct values, and every large section in the diagram contains it. In fact, choosing a daily grain for the line chart made the file about 7x larger than the weekly equivalent (5.6mb). Still, the daily grain did not meaningfully impact the latency of the range requests, since any interaction only ever reads a few row groups. And for this case, it’s nice to see a big single-day spike, since a big uptick in service requests can happen in a single day because of major events like hurricanes or blizzards. Dashboards such as the one above work well for distributive and algebraic aggregations, which can be computed in pieces and then combined before visualizing. Think of sums, counts, maxes, and averages. Making this setup work for holistic aggregations (ones that require knowledge of the distribution before achieving a final filtered aggregate) have both exact and approximate solutions. I’ll leave that as an exercise to the reader and their favorite agent. Range requests over a carefully laid-out file have plenty of prior art. PMTiles packs a tileset into one file that clients read via range requests over http. It works because the tiles are laid out in the file along a Hilbert curve, so the tiles for a given map view sit near each other in the file and can be fetched in a few coalesced range requests. And of course, the well-known SQLite-over-HTTP writeup proved the mechanic works even for B-trees. My favorite part is that this approach shifts the complexity “left” all the way to the data pipeline. The layout is decided beforehand, so by the time a user clicks on a leaderboard or scrubs a time series chart, the client only has to fetch the right rows and sum them. As for the pipeline, for most customer-facing dashboards, a 10mb per-customer cube falls out of a DuckDB GROUP BY GROUPING SETS statement. For my friend, who’s a data engineer, it’s pitch-perfect déformation professionnelle.