feat(plugins,docs): EIP checklist generator plugin

docs/CHANGELOG.md

@@ -53,6 +53,7 @@ Users can select any of the artifacts depending on their testing needs for their
 - 🐞 Fix bug in ported-from plugin and coverage script that made PRs fail with modified tests that contained no ported tests ([#1661](https://github.com/ethereum/execution-spec-tests/pull/1661)).
 - 🔀 Refactor the `click`-based CLI interface used for pytest-based commands (`fill`, `execute`, `consume`) to make them more extensible ([#1654](https://github.com/ethereum/execution-spec-tests/pull/1654)).
 - 🔀 Split `src/ethereum_test_types/types.py` into several files to improve code organization ([#1665](https://github.com/ethereum/execution-spec-tests/pull/1665)).
+- ✨ Added automatic checklist generation for every EIP inside of the `tests` folder. The checklist is appended to each EIP in the documentation in the "Test Case Reference" section ([#1679](https://github.com/ethereum/execution-spec-tests/pull/1679)).
 
 ### 🧪 Test Cases
 

docs/navigation.md

@@ -21,6 +21,7 @@
       * [Exception Tests](writing_tests/exception_tests.md)
       * [Using and Extending Fork Methods](writing_tests/fork_methods.md)
       * [Referencing an EIP Spec Version](writing_tests/reference_specification.md)
+      * [EIP Checklist Generation](writing_tests/eip_checklist.md)
       * [Testing Checklist Templates](writing_tests/checklist_templates/index.md)
           * [EIP Execution Layer Testing Checklist Template](writing_tests/checklist_templates/eip_testing_checklist_template.md)
       * [Post-mortems](writing_tests/post_mortems.md)

docs/scripts/gen_test_case_reference.py

@@ -43,9 +43,12 @@
     "filterwarnings=ignore::pytest.PytestAssertRewriteWarning",  # suppress warnings due to reload
     "-p",
     "pytest_plugins.filler.gen_test_doc.gen_test_doc",
+    "-p",
+    "pytest_plugins.filler.eip_checklist",
     "--gen-docs",
     f"--gen-docs-target-fork={TARGET_FORK}",
     f"--until={GENERATE_UNTIL_FORK}",
+    "--checklist-doc-gen",
     "--skip-index",
     "-m",
     "not blockchain_test_engine",

docs/writing_tests/checklist_templates/eip_testing_checklist_template.md

@@ -3,6 +3,12 @@
 
 Depending on the changes introduced by an EIP, the following template is the minimum baseline to guarantee test coverage of the Execution Layer features.
 
+## Checklist Progress Tracker
+
+| Total Checklist Items | Covered Checklist Items | Percentage |
+| --------------------- | ----------------------- | ---------- |
+| TOTAL_CHECKLIST_ITEMS | COVERED_CHECKLIST_ITEMS | PERCENTAGE |
+
 ## General
 
 #### Code coverage

docs/writing_tests/eip_checklist.md

@@ -0,0 +1,206 @@
+# EIP Checklist Generation
+
+The EIP checklist feature helps track test coverage for EIP implementations by automatically generating filled checklists based on test markers.
+
+## Overview
+
+When implementing tests for an EIP, you can mark specific tests as covering checklist items from the [EIP testing checklist template](../writing_tests/checklist_templates/eip_testing_checklist_template.md). The framework will then generate a filled checklist showing which items have been implemented.
+
+## Using the `pytest.mark.eip_checklist` Marker
+
+To mark a test as implementing a specific checklist item:
+
+```python
+import pytest
+from ethereum_test_tools import StateTestFiller
+
+@pytest.mark.eip_checklist("new_transaction_type/test/intrinsic_validity/gas_limit/exact")
+def test_exact_intrinsic_gas(state_test: StateTestFiller):
+    """Test transaction with exact intrinsic gas limit."""
+    # Test implementation
+    pass
+```
+
+### Marker Parameters
+
+- **First positional parameter** (required): The checklist item ID from the template
+- **`eip` keyword parameter** (optional): List of additional EIPs covered by the test
+
+Example with multiple EIPs covered by the same test:
+
+```python
+@pytest.mark.eip_checklist("new_transaction_type/test/signature/invalid/v/0", eip=[7702, 2930])
+def test_invalid_signature(state_test: StateTestFiller):
+    """Test invalid signature that affects multiple EIPs."""
+    pass
+```
+
+### Partial ID Matching
+
+You can use partial IDs that will match all checklist items starting with that prefix:
+
+```python
+# This will mark all items under "new_transaction_type/test/signature/invalid/" as covered
+@pytest.mark.eip_checklist("new_transaction_type/test/signature/invalid/")
+def test_all_invalid_signatures(state_test: StateTestFiller):
+    """Test covering all invalid signature scenarios."""
+    pass
+```
+
+## Generating Checklists
+
+### Using the Dedicated `checklist` Command
+
+To generate only checklists without filling fixtures:
+
+```bash
+# Generate checklists for all EIPs
+checklist
+
+# Generate checklist for specific EIP
+checklist --eip 7702
+
+# Specify output directory
+checklist --output ./my-checklists
+
+# Multiple EIPs
+checklist --eip 7702 --eip 2930
+```
+
+### Automatic Generation in Documentation
+
+When building the documentation with `mkdocs`, checklists are automatically generated for all EIPs that have tests with checklist markers. The checklists appear in the test documentation alongside the test modules.
+
+## External Coverage and Not Applicable Items
+
+### External Coverage
+
+For checklist items that are covered by external tests, procedures, or tools (e.g., EELS coverage), create a file named `eip_checklist_external_coverage.txt` in the EIP test directory:
+
+```text
+# tests/prague/eip7702_set_code_tx/eip_checklist_external_coverage.txt
+general/code_coverage/eels = Covered by EELS test suite
+general/code_coverage/second_client = Covered by Nethermind tests
+```
+
+Format: `checklist_item_id = reason`
+
+### Not Applicable Items
+
+For checklist items that are not applicable to a specific EIP, create a file named `eip_checklist_not_applicable.txt` in the EIP test directory:
+
+```text
+# tests/prague/eip7702_set_code_tx/eip_checklist_not_applicable.txt
+new_system_contract = EIP-7702 does not introduce a system contract
+new_precompile = EIP-7702 does not introduce a precompile
+```
+
+Format: `checklist_item_id = reason`
+
+Both files support partial ID matching, so you can mark entire sections as not applicable:
+
+```text
+# Mark all system contract items as not applicable
+new_system_contract/ = EIP does not introduce system contracts
+```
+
+## Output Format
+
+The generated checklist will show:
+
+- ✅ for completed items (either by tests or external coverage)
+- N/A for not applicable items
+- Test names that implement each item
+- External coverage reasons where applicable
+- A percentage of covered checklist items (excluding N/A items)
+- Color-coded completion status: 🟢 (100%), 🟡 (>50%), 🔴 (≤50%)
+
+Example output snippet:
+
+```markdown
+# EIP-7702 Test Checklist
+
+## Checklist Progress Tracker
+
+| Total Checklist Items | Covered Checklist Items | Percentage |
+| --------------------- | ----------------------- | ---------- |
+| 45 | 32 | 🟡 71.11% |
+
+## General
+
+#### Code coverage
+
+| ID | Description | Status | Tests |
+| -- | ----------- | ------ | ----- |
+| `general/code_coverage/eels` | Run produced tests against EELS... | ✅ | Covered by EELS test suite |
+| `general/code_coverage/test_coverage` | Run coverage on the test code itself... | ✅ | `tests/prague/eip7702_set_code_tx/test_set_code_txs.py::test_set_code_txs` |
+
+## New Transaction Type
+
+| ID | Description | Status | Tests |
+| -- | ----------- | ------ | ----- |
+| `new_transaction_type/test/intrinsic_validity/gas_limit/exact` | Provide the exact intrinsic gas... | ✅ | `tests/prague/eip7702_set_code_tx/test_checklist_example.py::test_exact_intrinsic_gas` |
+| `new_transaction_type/test/intrinsic_validity/gas_limit/insufficient` | Provide the exact intrinsic gas minus one... |  |  |
+
+## New System Contract
+
+| ID | Description | Status | Tests |
+| -- | ----------- | ------ | ----- |
+| `new_system_contract/test/deployment/missing` | Verify block execution behavior... | N/A | EIP-7702 does not introduce a system contract |
+```
+
+## Best Practices
+
+1. **Start with the checklist**: Review the checklist template before writing tests to ensure comprehensive coverage
+2. **Use descriptive test names**: The test name will appear in the checklist, so make it clear what the test covers
+3. **Mark items as you go**: Add `eip_checklist` markers while writing tests, not as an afterthought
+4. **Document external coverage**: If items are covered by external tools/tests, document this in `eip_checklist_external_coverage.txt`
+5. **Be explicit about N/A items**: Document why items are not applicable in `eip_checklist_not_applicable.txt`
+6. **Use partial IDs wisely**: When a test covers multiple related items, use partial IDs to mark them all
+
+## Workflow Example
+
+1. **Create test directory structure**:
+
+   ```bash
+   tests/prague/eip9999_new_feature/
+   ├── __init__.py
+   ├── spec.py
+   ├── test_basic.py
+   ├── eip_checklist_external_coverage.txt
+   └── eip_checklist_not_applicable.txt
+   ```
+
+2. **Mark tests as you implement them**:
+
+   ```python
+   @pytest.mark.eip_checklist("new_opcode/test/gas_usage/normal")
+   def test_opcode_gas_consumption(state_test: StateTestFiller):
+       """Test normal gas consumption of the new opcode."""
+       pass
+   ```
+
+3. **Document external coverage**:
+
+   ```text
+   # eip_checklist_external_coverage.txt
+   general/code_coverage/eels = Covered by ethereum/execution-specs PR #1234
+   ```
+
+4. **Mark non-applicable items**:
+
+   ```text
+   # eip_checklist_not_applicable.txt
+   new_precompile/ = EIP-9999 introduces an opcode, not a precompile
+   ```
+
+5. **Generate and review checklist**:
+
+   ```bash
+   checklist --eip 9999
+   # Review the generated checklist for completeness
+   ```
+
+## See Also
+
+- [EIP Testing Checklist Template](./checklist_templates/eip_testing_checklist_template.md) - The full checklist template

pyproject.toml

@@ -93,6 +93,7 @@ checkfixtures = "cli.check_fixtures:check_fixtures"
 check_eip_versions = "cli.pytest_commands.check_eip_versions:check_eip_versions"
 consume = "cli.pytest_commands.consume:consume"
 protec = "cli.pytest_commands.consume:consume"
+checklist = "cli.pytest_commands.checklist:checklist"
 genindex = "cli.gen_index:generate_fixtures_index_cli"
 gentest = "cli.gentest:generate"
 eofwrap = "cli.eofwrap:eof_wrap"

src/cli/pytest_commands/__init__.py

@@ -19,6 +19,8 @@
 fill --help
 # for example, or
 consume engine
+# or
+checklist --help
 ```
 
 They can also be executed (and debugged) directly in an interactive python

src/cli/pytest_commands/checklist.py

@@ -0,0 +1,77 @@
+"""CLI entry point for the `checklist` pytest-based command."""
+
+from typing import List
+
+import click
+
+from .base import PytestCommand
+
+
+class ChecklistCommand(PytestCommand):
+    """Pytest command for generating EIP checklists."""
+
+    def __init__(self):
+        """Initialize checklist command with processors."""
+        super().__init__(
+            config_file="pytest.ini",
+        )
+
+    def process_arguments(self, pytest_args: List[str]) -> List[str]:
+        """Process arguments, ensuring checklist generation is enabled."""
+        processed_args = super().process_arguments(pytest_args)
+
+        # Add collect-only flag to avoid running tests
+        processed_args.extend(["-p", "pytest_plugins.filler.eip_checklist"])
+
+        return processed_args
+
+
+@click.command()
+@click.option(
+    "--output",
+    "-o",
+    type=click.Path(file_okay=False, dir_okay=True, writable=True),
+    default="./checklists",
+    help="Directory to output the generated checklists (default: ./checklists)",
+)
+@click.option(
+    "--eip",
+    "-e",
+    type=int,
+    multiple=True,
+    help="Generate checklist only for specific EIP(s)",
+)
+def checklist(output: str, eip: tuple, **kwargs) -> None:
+    """
+    Generate EIP test checklists based on pytest.mark.eip_checklist markers.
+
+    This command scans test files for eip_checklist markers and generates
+    filled checklists showing which checklist items have been implemented.
+
+    Examples:
+        # Generate checklists for all EIPs
+        eest checklist
+
+        # Generate checklist for specific EIP
+        eest checklist --eip 7702
+
+        # Generate checklists for specific test path
+        eest checklist tests/prague/eip7702*
+
+        # Specify output directory
+        eest checklist --output ./my-checklists
+
+    """
+    # Add output directory to pytest args
+    args = ["--checklist-output", output]
+
+    # Add EIP filter if specified
+    for eip_num in eip:
+        args.extend(["--checklist-eip", str(eip_num)])
+
+    command = ChecklistCommand()
+    command.execute(args)
+
+
+if __name__ == "__main__":
+    checklist()