Sorry if my spoken English is a bit lacking, so the presentation is a bit difficult to follow. If anything isn’t clear, please feel free to jump in and ask.

Hi everyone, and thank you for having me. I’m Zeyi Xu, and this is joint work with Victor Baranov. Today I’d like to introduce CTIT, the Clang-Tidy Integration Tester. CTIT is a public service that takes a proposed clang-tidy change and runs it on real-world C and C++ projects while the pull request is still in review. So far it serves clang-tidy, but at the end I’d like your opinion on whether it could also help the Static Analyzer or other tools.

Let me start with the problem. A clang-tidy change can break three things. Accuracy, which consists of false positives, false negatives, and fix-its that produce invalid code. Robustness is a check crashing on perfectly valid code. Performance is analysis time that explodes once you run on a whole project. Until now we relied on three defenses: the PR author checking their own patch, reviewers reading the code, and users filing bug reports after a release. The last one works, but it is the slowest and most expensive place to find a bug.

The clang-tidy contributing guide already says: “To test a check, it’s best to try it out on a larger code base.” In reality, some authors don’t do it, partly because it is real work: you need to build LLVM, set up several projects, and generate compilation databases. This screenshot is a real review comment by Victor on the new performance-use-std-move check. The PR’s own tests passed. The crash was found only because a reviewer took the time to run the check on real code by hand.

Why can tests pass while real code crashes? Our main regression tool is the lit test. These tests are small, fast, deterministic, and portable, and that is exactly why we use them. But they only cover what authors and reviewers anticipate, and mock headers are only as accurate as the real headers they imitate. Even check options have gaps: as of February, 92 of 611 clang-tidy checks still lacked comprehensive test coverage for their configuration options.

To measure how often this gap matters, we traced LLVM issues filed from January 2021 to September 2026 that carry the clang-tidy label plus false-positive, false-negative, crash, or invalid-code-generation. That gives 673 reports. In 507 of them, about 75%, the issue showed a valid failure caused by a condition the focused tests did not represent. We see three mechanisms. Ecosystem drift: new language standards and library versions. Unanticipated context, 493 reports and by far the largest: language features, macros, and build setups nobody tested. Project-scale amplification, 6 reports: small defects multiplied by project size. One report can show more than one mechanism, so the rows add up to more than 507.

Here is an ecosystem-drift example. For this issue, nothing changed in clang-tidy. The trigger is a glibc commit. The lit tests use a mock assert.h, so they keep passing. Only a current, real glibc shows the problem.

Next is an unanticipated-context example. readability-non-const-parameter crashed on this tiny piece of legacy C. It is a K&R-style definition, where the parameter type is declared after the parameter list. It is valid C and it still exists in real code, but no test covered it. We see this pattern again and again: once someone hits the case, adding a test is easy. The hard part is knowing the case exists. Real projects give us those cases for free.

Do existing tools already solve this? Two come close. Clippy’s lintcheck is almost exactly what we want: it builds the current Clippy, runs it on a corpus of Rust crates, and compares diagnostics between revisions. But it relies on Cargo, and C and C++ have no common build abstraction, so each project must be configured separately to get faithful compile commands. CodeChecker does use real compilation databases, but its goal is to test the projects themselves. We need the opposite: the linter is the system under test, and the projects are the test corpus.

So we need a new tool with three properties. First, it tests clang-tidy on diverse real-world projects in their native configurations. Second, it is easy to use and maintain, and open to every contributor. Third, it is review-native: everything happens on GitHub, where LLVM review already takes place, and every input, log, and result is public.

Here is the architecture in a simple form. A request names an LLVM PR, a check, and its options. GitHub Actions parses the request and selects a runner. In the middle is the integration test. The LLVM revision and every test project are pinned to exact commits, so runs are reproducible. On the PR path, we apply the PR’s source changes without its tests and docs, build clang-tidy, and analyze every project with only the requested check enabled. If the check already exists, we can first run the same steps on unpatched LLVM as a baseline. Containers and the sccache build cache are shared to keep builds cheap. On the right is triage and reporting. We summarize the logs into warnings, errors, crashes with stack excerpts, and analysis time. The results are uploaded to a public results repository and posted as an issue comment. An optional LLM step suggests false-positive verdicts.

Using it is simple. Open an issue in the CTIT repository with the template. The first line is the PR URL and the check name. Then add one option per line, and optionally /baseline to compare with unpatched LLVM. The cpp label starts the run. CTIT posts a “Run started” comment with a link to the workflow, then replaces it with the report. To protect shared runners, runs start only for people with commit access to llvm-project. Without access, you can still open the issue and ask a reviewer to comment /redo. Because the issue links the PR, it also appears in the PR timeline.

Currently we test against 13 projects. Nine are C++. Four are C. We picked them for diversity: analysis tools, a compiler, frameworks, template-heavy libraries, networking, and so on. Each one is pinned to a commit and configured with CMake and Ninja in its native setup. CTIT can also run with three standard-library profiles: libstdc++ from GCC 14 and 15, and libc++ 21. This targets the ecosystem-drift problem directly.

Is it useful? Here are two real cases. CTIT issue 183 tested bugprone-macro-condition, PR 210768. The first run, on the seven projects we had then, produced more than 3,100 warnings. Most of them are FPs. The focused tests had passed. No reviewer can catch a warning flood like this by reading the diff. The author revised the check, and the later run on all 13 projects reports 98 warnings in total, with five projects clean.

Second, crashes. bugprone-smart-ptr-initialization. The PR’s regression tests passed, but CTIT crashed on six of the seven projects. The author added a regression test and separate fixes, and triggered a rerun. The table on the right is a later run on all 13 projects, and every project passes. That is the full detect, fix, and verify loop inside one review.

CTIT also finds performance problems. Victor found this one while building the daily check-timing workflow, which enables a broad set of checks and records time per check. Two LLVM files, RISCVVEmitter.cpp and Multilib.cpp, each took more than two hours with an assertions-enabled clang-tidy. The problem was bugprone-unchecked-optional-access, which is built on Clang’s dataflow framework. The check logic was fine. Three invariant checks in the watched-literal solver ran on every iteration and rebuilt solver state each time. A fix moved them to expensive-checks builds, and the time dropped to 12 to 14 seconds. LLVM’s own pre-merge CI used the same configuration, so this affected LLVM’s infrastructure too.

But would CTIT have caught bugs that actually reached users? To answer that, I replayed two release cycles, LLVM 20 to 21 and 21 to 22. I built the exact trees at each cycle boundary and ran every new or behavior-changing check of that cycle on all 13 projects. Out of 307 candidate PRs, 163 were in scope, covering 121 new or modified checks. I diffed the results, grouped the new diagnostics by root cause, and reviewed each group by hand. CTIT surfaced 15 false-positive regressions, all confirmed. Users later reported 11 of them, 2 were fixed without an issue, and 2 are still unreported. For the 11 reported ones, CTIT would have flagged them at the introducing PR, a median of 6.2 months before the first user report.

What does it cost? Across 69 successful runs on 33 reviewer-selected PRs, a typical run takes around 20 minutes from start to report. Slow checks take longer, up to about 90 minutes in most cases. For comparison, LFX Insights reports an average merge time of 7 days for LLVM pull requests. Mean feedback time is about 0.3% of that, and about 0.6% even when we count reruns. So CTIT costs less than 1% of the whole review process, and it runs alongside review instead of blocking it.

If you are interested in CTIT, you can find the source code in the main repository and the test results in the CTIT-results repository. The links to both repositories are on this slide.

There is still a lot to do. First, false negatives: diffing cannot tell a removed false positive from a new false negative, so we want real-world cases with known expected diagnostics. Second, LLM triage: we want to test more models against manually reviewed reports, for speed and accuracy. Third, a wider corpus and more configurations: more standard libraries, platforms, and build systems beyond CMake. Fourth, applying fix-its and rebuilding the projects, to catch invalid fixes, which we don’t do yet. And finally, a website to display and compare results.

I’d like to end with three questions for this group. First, would the Clang Static Analyzer be interested in using this tool? The pipeline is not specific to clang-tidy: build a patched tool, run it over real compilation databases, and collect diagnostics, crashes, and timing. I’d like to know what CSA would need from it. Second, which new features should we add first? Third, are there projects you would like us to test, especially ones that stress the analyzer differently from our current corpus? Thank you, and I’m happy to take questions.