ClickBench removes pgrust over benchmark-specific code and caching

ClickBench removed the pgrust entry from its analytical database benchmark after Alexey Milovidov reported benchmark-specific execution code and prohibited result caching. The removal was merged into the ClickHouse-hosted repository on September 22, 2026, following an update to pgrust’s v0.4-preview release.

In the removal proposal, Milovidov said the implementation was tailored to ClickBench’s 43 queries despite being labeled as untuned. He also alleged that repeated executions reused previously computed answers, rather than measuring the work the benchmark was intended to compare. These findings are his published assessment, not an independent verification by this article.

Milovidov cited query-numbered execution routines and benchmark-specific matching logic in support of the decision. He also reported that small changes to some queries were rejected. Importantly, his assessment said the benchmark answers themselves were correct, including differences attributable to valid ordering ties; the objection concerned the execution methods behind the reported timings.

ClickBench’s published rules distinguish between acceptable caching and shortcuts that undermine the comparison. Keeping source data in memory is permitted, but query-result caches must be disabled. Intermediate caches are generally acceptable unless they sit sufficiently close to the final answer to have an effect similar to result caching.

That distinction matters because each benchmark query runs three times. The second and third executions are designated hot runs, allowing systems to benefit from data already loaded into database or operating-system caches. The intention is to measure execution with readily available data, not simply retrieval of an answer computed during an earlier run.

The project also allows benchmark-specific tuning, although it discourages it and requires separately identified vanilla and tuned results. Its documentation presents ClickBench as a reproducible comparison for workloads including web analytics, structured logs and event data. Published scripts cover installation, data loading, query execution and collection of timing results, making the testing process available for scrutiny.

The disputed update had been submitted by the GitHub account malisper with revised scripts and results. In that discussion, the submitter said the source code would be released within weeks. When Milovidov subsequently raised the missing source, malisper agreed that the release could be marked proprietary while the code was being prepared for publication.

The benchmark’s documentation also cautions against treating its rankings as a universal measure of database quality. It uses a single flat table and covers only a subset of possible workloads. Those limitations make the distinction between performance on a representative workload and specialization for the exact published queries particularly important when interpreting results.

A later issue requesting pgrust’s inclusion was closed on October 14, 2026, after Milovidov referred participants back to the removal. He distinguished the disputed update from the earlier submission, which he described favorably and said had outperformed ClickHouse on some configurations. That exchange documented no reinstatement; the published rules remain the framework for assessing any future submission.