Add spec references, a mapping of spec to implementation

[jtraglia]

What type of PR is this?

Documentation

What does this PR do? Why is it needed?

This PR adds a directory with specification references. These are used to map specification items (configs, presets, functions, etc) to client implementations (code in Prysm). This is much less invasive than my original idea of adding spec tags all across the codebase. These specification references are meant to (1) help developers keep track of specification changes and (2) make it easier for third-parties (eg EF Protocol Security) to verify clients adhere to the specifications.

Note, we recently did the same thing for Teku. View that PR for more details.

• Add spec references, a mapping of spec to implementation Consensys/teku#9731

Other notes for review

If we decide to merge this PR, we can remove the existing spec-checker tool.

Depends on #15590.

Acknowledgements

• I have read CONTRIBUTING.md.
• I have included a uniquely named changelog fragment file.
• I have added a description to this PR with sufficient context for reviewers to understand this PR.

[prestonvanloon]

Thanks for the PR. I am wondering how we maintain this?

Will it be difficult to update in the future? Perhaps you could share a blurb about how you created this PR so we can refer to it in the future when it is time to update it with new spec changes.

[jtraglia]

Thanks for the PR. I am wondering how we maintain this?

Will it be difficult to update in the future? Perhaps you could share a blurb about how you created this PR so we can refer to it in the future when it is time to update it with new spec changes.

I don't believe it will be difficult to maintain. Creating this was pretty manual & not a good representation of future efforts.

When adding support for a new specification (eg from v1.6.0-alpha.3 to v1.6.0-alpha.4) developers will:

• First update the version in specrefs/.ethspecify.yml.
• Then run ethspecify check --path=specrefs to see if there are new items. For example:

$ ethspecify check --path=specrefs
MISSING: functions.is_data_available#fulu

• Then add an entry to the relevant file (eg specrefs/functions.yml).

- name: is_data_available#fulu
  sources: []
  spec: |
    <spec fn="is_data_available" fork="fulu" />

• Then locate where this is in the codebase and update the sources list.

- name: is_data_available#fulu
  sources:
    - file: beacon-chain/blockchain/process_block.go
      search: func (s *Service) isDataAvailable(
    - file: beacon-chain/blockchain/process_block.go
      search: func (s *Service) areDataColumnsAvailable(
  spec: |
    <spec fn="is_data_available" fork="fulu" />

• Repeat last three steps until the ethspecify check passes.
• Then run ethspecify process --path=specrefs to update spectag bodies.

- name: is_data_available#fulu
  sources:
    - file: beacon-chain/blockchain/process_block.go
      search: func (s *Service) isDataAvailable(
    - file: beacon-chain/blockchain/process_block.go
      search: func (s *Service) areDataColumnsAvailable(
  spec: |
    <spec fn="is_data_available" fork="fulu" hash="2062e9c6">
    def is_data_available(beacon_block_root: Root) -> bool:
        # `retrieve_column_sidecars` is implementation and context dependent, replacing
        # `retrieve_blobs_and_proofs`. For the given block root, it returns all column
        # sidecars to sample, or raises an exception if they are not available.
        # The p2p network does not guarantee sidecar retrieval outside of
        # `MIN_EPOCHS_FOR_DATA_COLUMN_SIDECARS_REQUESTS` epochs.
        column_sidecars = retrieve_column_sidecars(beacon_block_root)
        return all(
            verify_data_column_sidecar(column_sidecar)
            and verify_data_column_sidecar_kzg_proofs(column_sidecar)
            for column_sidecar in column_sidecars
        )
    </spec>

• Run git diff to see what else has changed.
• If you see that an object/function/etc has changed, make the necessary change.
• Lastly run act -j check-specrefs to ensure everything is correct. This will catch several issues, like if a file is renamed and a search condition fails. See this comment for some more examples: Add spec references, a mapping of spec to implementation Consensys/teku#9731 (comment)

[jtraglia]

One more example, see the changes made here & the files below:

• https://github.com/Consensys/teku/pull/9756/files#diff-ec02454b6e3ca728abb1f6b9225ec07d641858f4870bb26547d6be2359d9a4ee

[prestonvanloon]

Great, thanks for this!