Creating a new maintainable debuginfo test suite #936
Description
Proposal
TLDR: This MCP proposes creation of a new debuginfo test suite that would be easier to maintain and easier for contributors to work with than the current one.
Currently, we have a debuginfo test suite for testing debugger visualizations in GDB/LLDB/CDB, but it has two main issues:
- It is not possible to bless the reference debugger output in tests, which leads to a high maintenance burden of the debuginfo tests. This caused the test suite to deteriorate over time, and some parts of it (like the LLDB tests) are not even executed fully on our CI at the moment (#141539).
- By default, it uses whatever ambient debuggers are available in the user's environments, which commonly leads to the test suite failing both locally and on CI (#126092) because of an unsupported LLDB/GDB/CDB version being present in the environment.
There are also lot of other technical details in which the test suite is lacking, e.g. it does not check DWARF/LLVM DI nodes or perform sanity checks for LLDB Python visualizer objects. These are described in 1 and 2.
The current "half-broken" state of the debuginfo test suite blocks work on debugger visualization improvements, which is a shame, because "the debugging experience is not great" is frequently being reported as a top 3 challenge of Rust users (https://blog.rust-lang.org/2025/02/13/2024-State-Of-Rust-Survey-results/#challenges).
Fixing the whole test suite at once is not really feasible, and we also do not yet know what would be the best design to avoid the mentioned issues and make the suite more manageable in the future. In fact, making any changes to the existing test suite at the moment is quite tricky, as we do not even run parts of it on CI and it's not trivial to run it locally.
I thus propose to create a new debuginfo test suite from scratch (e.g. debuginfo2), and incrementally port the existing tests from the old debuginfo suite to it, while also iterating on the design that would resolve the issues of the old debuginfo suite. This is similar to the run-make test suite redesign, which created a new test suite for Rust run-make recipes, in which we iterated on how to best build a test API for writing these tests, and once that API was solid, we incrementally moved all the old Makefile run-make tests to the new suite, and then removed the old test suite.
I do not want to propose specific implementation details of the new test suite here, but I want to describe two primary goals of this redesign.
The first goal is that it should be simple to maintain. That is the number one goal; if we do not manage to fulfill it, then there is no need to continue with this experiment and we could just go back to the old broken test suite.
In order to achieve that, the tests should be blessable, so that when a debugger output changes for any reason (that we don't consider to be a bug on our side), we can bless the debugger output using O(1), rather than O(n), manual work.
The second primary goal is that it should be simple to run the test suite locally, while ensuring that the correct versions of debuggers are present (rather than running with whatever the system has). That might mean that we will have to e.g. build and ship our own versions of debuggers, to ensure that we use the same versions locally and on CI.
Note that given that the MCP doesn't specify implementation details, its intention is essentially to ask whether t-compiler thinks that this is a worthwhile effort, and also to give us a permission to experiment with this in-tree.
I would suggest for the test suite to "start small", e.g. we could focus on a single debugger version and single OS (e.g. LLDB + Linux), to iterate and find what works best.
Ultimately, if we do find a way to build a maintainable debuginfo test suite, we would port all the old debuginfo tests to it, remove the old test suite, and thus also re-enable all GDB/LLDB/CDB debuginfo tests on our CI, to ensure that the quality of our debuginfo doesn't regress over time.
If that succeeds, then there is a number of secondary goals that we could do in the future to improve our debuginfo support, such as testing multiple debugger versions, testing how exactly we generate DWARF or LLVM DI objects directly in the debuginfo tests, or even things like optimizing performance of the visualizer scripts. But I want to clarify that this is a "nice-to-have" thing that we can do as a follow-up, and not within the scope of this MCP (I don't think such changes would even require an MCP though).
Mentors or Reviewers
Walnut356 has been doing a lot of experimentation and work on Rust and LLVM debuginfo, and has a lot of ideas on how to build a better debuginfo test suite.
I would work with Walnut and other interested people on implementing the new test suite and helping with reviews around bootstrap/compiletest integration.
Process
The main points of the Major Change Process are as follows:
- File an issue describing the proposal.
- A compiler team member who is knowledgeable in the area can second by writing
@rustbot secondor kickoff a team FCP with@rfcbot fcp $RESOLUTION.- Refer to Proposals, Approvals and Stabilization docs for when a second is sufficient, or when a full team FCP is required.
- Once an MCP is seconded, the Final Comment Period begins.
- Final Comment Period lasts for 10 days after all outstanding concerns are solved.
- Outstanding concerns will block the Final Comment Period from finishing. Once all concerns are resolved, the 10 day countdown is restarted.
- If no concerns are raised after 10 days since the resolution of the last outstanding concern, the MCP is considered approved.
You can read more about Major Change Proposals on forge.