solana-program-test new features necessary for code coverage functionality

[procdump]

Problem

1. sol_get_processed_sibling_instruction in solana-program-test is unimplemented. here
2. ProgramTest's payer is randomly chosen. here
3. Inability to obtain access to ProgramTest's invoke_context. here
4. Currently it's possible to register a recent blockhash in the solana-runtime::bank::Bank for testing (gated behind feature dev-context-only-utils) but that's missing or it's not exposed from the solana-program-test crate. here

Summary of Changes

1. Modify ProgramTest to support sol_get_processed_sibling_instruction.
2. Make ProgramTest payer configurable while preserving the old logic if not explicitly configured.
3. Modify get_invoke_context to be pub.
4. Add a new method to ProgramTestContext for registering a recent blockhash. Add dev-context-only-utils feature to ProgramTest's dependency solana-runtime so that the Bank's method register_recent_blockhash_for_test can be used.

At first glance, these changes are not directly related to code coverage but in the end unleash us with the ability to get to those statistics. A working example can be checked here. A vigilant reader will recognize the two problems we're facing with this example:

• We're forced to rely on a fork of solana-program-test so that we have the proposed features upper.
• LiteSVM's solana-program-runtime to day is tied to v2.3.

The main goal is to contribute the code coverage tool from the working example to the Solana community.
Ultimately the main idea behind this PR is to have these features here in master and later have them also in the v2.3 branch.

[0xbrw]

@Lichtso when you get a chance, can you take a look

[Lichtso]

We are not really maintaining program-test anymore and generally recommend to use the SVM directly & build your programs as SBPF.

[procdump]

The problem is that you can't get Rust code coverage from compiled SBPF programs, but you can very well get statistics from program-test and that's why we've adopted it.

I hope you reconsider dropping program-test as it would be of no benefit to the Solana community.

This is part of a TypeScript Code Coverage solution directly in Anchor & LiteSVM. You can check what the end result looks like here.

Tagging @Aursen from LiteSVM for his perspective.

[Lichtso]

The problem is that you can't get Rust code coverage from compiled SBPF programs, but you can very well get statistics from program-test and that's why we've adopted it.

Interesting use-case, yes we currently have no direct tooling for that, but it should be possible to develop. One needs the source to instruction mapping in the form of debug info emitted by the compiler and the instruction trace recorded by the VM. Putting those together should allow you to measure code coverage.

Edit: Seems to already be done this way: https://github.com/trailofbits/anchor-coverage

Can you join us on Discord in the virtual-machine channel to brainstorm this?

I hope you reconsider dropping program-test as it would be of no benefit to the Solana community.

It is not that it takes to much effort to maintain, but that the approach of program-test is fundamentally incompatible with the direction we are headed in, e.g. account data direct mapping (SIMD-0219) Thus v3.0 will probably be the last release which supports it.

[procdump]

Interesting use-case, yes we currently have no direct tooling for that, but it should be possible to develop. One needs the source to instruction mapping in the form of debug info emitted by the compiler and the instruction trace recorded by the VM. Putting those together should allow you to measure code coverage.

Edit: Seems to already be done this way: https://github.com/trailofbits/anchor-coverage

The suggested approach seems to produce inaccurate and incomplete results.
Here are sample screenshots of the output of generated coverage from an example vault program.

This is using the instruction mapping approach as you've explained above citing the github link:

It clearly misses the '?' operator in Rust. Also there are lines that are 100% covered but not in the output.
Our observation is consistent with the known problems the author has posted here: https://github.com/trailofbits/anchor-coverage?tab=readme-ov-file#known-problems
It also worths mentioning that I happened to make it work with LiteSVM 0.6.1 on Solana 2.1.20 but for 2.2.x or 2.3.x I didn't have such luck - particularly failing at the DWARF analysis of vault.debug from addr2line.

This is from our approach:

It catches the '?' Rust operator, lines are clearly caught and branching seems to be in working shape.

Can you join us on Discord in the virtual-machine channel to brainstorm this?

I'm on Discord with username procdump_ but I can't post messages in the #virtual-machine channel.

It is not that it takes to much effort to maintain, but that the approach of program-test is fundamentally incompatible with the direction we are headed in, e.g. account data direct mapping (solana-foundation/solana-improvement-documents#219) Thus v3.0 will probably be the last release which supports it.

As a matter of fact, program-test is also very useful for step debugging of programs. Now that they're compiled into SBPF step-debugging isn't nice.
It's a pity you're deorbiting program-test as up to now it's a reasonable testing facility with as it seems lots of use cases.

[Lichtso]

I think the underlying issue is that it currently only resolves at line level, not at basic blocks / conditional branches. That way it misses the ? operator or double counts lines.

It's a pity you're deorbiting program-test as up to now it's a reasonable testing facility with as it seems lots of use cases.

The program-test crate might continue to exist but the path way you are using which bypasses the SBPF VM will be removed. Building and then measuring for the host architecture is fundamentally incorrect because the compiler can generate completely different code paths. Only building and running on SBPF can produce accurate results. We just have to improve the tooling for that.

[procdump]

The program-test crate might continue to exist but the path way you are using which bypasses the SBPF VM will be removed. Building and then measuring for the host architecture is fundamentally incorrect because the compiler can generate completely different code paths. Only building and running on SBPF can produce accurate results. We just have to improve the tooling for that.

The program-test crate is provided by Solana to the community and we've just taken advantage of it having for granted that you've backed some level of parity. There are lots of tutorials on this topic. While fundamentally incorrect testing on the host architecture has its benefits - for example the native Rust debugging features where step-debugging on Rust source lines (not bytecode) work out of the box. This is an invaluable weapon for catching and investigating a family of issues. At any rate, in its core as a mocking testing facility nobody expects from program-test to be perfect after all.

I think the underlying issue is that it currently only resolves at line level, not at basic blocks / conditional branches. That way it misses the ? operator or double counts lines.

Do you have a roadmap for that? When is it expected to land so that the Solana community can take advantage of it? To day this approach isn't usable and as it seems with each version of Solana the emitted DWARF gets changes and hence causes the addr2line crate to fail eventually. Till this is meticulously settled can we have some sort of consensus? The program-test crate is just a testing facility and I doubt these few changes will have any negative impact at all. Then when you announce you're shutting it down we'll have to adapt and we hope you will have backed a good working substitution in advance.

[LucasSte]

To day this approach isn't usable and as it seems with each version of Solana the emitted DWARF gets changes and hence causes the addr2line crate to fail eventually. Till this is meticulously settled can we have some sort of consensus?

Dwarf information must be valid for every compiler release, and we are not changing it. If it is breaking the tool, then there is a bug. There was a relocation problem in dwarf generation that was fixed in platform tools v1.51.

[Lichtso]

Unfortunately, I couldn't get you permissions for Discord, but I started this discussion thread here instead as there are even more people interested in code coverage.