protos, python: use betterproto

[woodruffw]

WIP for now.

This replaces protobuf with betterproto, which:

• Generates much nicer, more Pythonic code;
• Is type-checked by default;
• Provides an on-ramp for doing extended validation with Pydantic here (although that isn't presently possible, since betterproto doesn't support Pydantic V2 yet;
• Is consistent with what Sigstore's protobuf-specs does, making the two more compatible.

The main downsides, in turn:

• It's essentially a total rewrite, since the underlying protobuf runtime support library is changing from protobuf to betterproto. That means it'll be backwards-incompatible with previous versions;
• Betterproto's 2.0 release is still in beta, although it has been stable in testing and is currently being used on protobuf-specs;
• The high-level wrappers will need to be rewritten, along with their tests.

Closes #291.

[woodruffw]

(I haven't removed any of the old protobuf-based code yet. This PR just adds the new stuff on top.)

[marcelamelara]

Thanks for getting this started! A couple thoughts.

It's essentially a total rewrite, since the underlying protobuf runtime support library is changing from protobuf to betterproto. That means it'll be backwards-incompatible with previous versions;

I'm not too concerned about this at the moment. Beyond in-toto's own (in-flight) transition to this library in our Python CLI, I don't know of other projects that currently depend on the current version of the Python library.

The high-level wrappers will need to be rewritten, along with their tests.

This is probably going to be the more cumbersome part. But if it's going to make writing future tests and using the library easier in the future, I think we can definitely motivate the re-implementation.

[woodruffw]

I'm not too concerned about this at the moment. Beyond in-toto's own (in-flight) transition to this library in our Python CLI, I don't know of other projects that currently depend on the current version of the Python library.

Sounds good! The only other use I know of is sigstore-python's (also in-flight) use, which is currently ongoing here: #315

This is probably going to be the more cumbersome part. But if it's going to make writing future tests and using the library easier in the future, I think we can definitely motivate the re-implementation.

Makes sense -- I think this will absolutely make future testing and use easier, so I'll start the cumbersome bits now.

[marcelamelara]

This is probably going to be the more cumbersome part. But if it's going to make writing future tests and using the library easier in the future, I think we can definitely motivate the re-implementation.

Makes sense -- I think this will absolutely make future testing and use easier, so I'll start the cumbersome bits now.

Thank you!

[woodruffw]

Just to post a brief update: I've run into some snarls with betterproto's JSON ser/de. This unfortunately looks like a bug in betterproto itself; I'll do some more debugging tomorrow.

[woodruffw]

Blocked on: danielgtaylor/python-betterproto#551

[marcelamelara]

@woodruffw Pinging this PR. Is this still something you expect to get back to? Otherwise, we'll close this until we're ready to pick this back up.

[woodruffw]

I'll get back to it eventually, but I probably won't have enough time in the next month or so. If that timeline is too far out, please close this!

[marcelamelara]

No worries @woodruffw. We'll close this PR for now, but make sure the issue stays open!