[Enhancement]: Improving the speed of the material and product attestor

[matglas]

Describe the solution you'd like:

When hashing 10000's of files small and big for both material an product it takes time. Currently the hashing is done in serial while parallel hashing thru threads would speed it up.

User value:

The amount of time spend 'extra' to do the attestation in a pipeline goes down.

Expected behavior:

Material and product hashing should be fast and have little impact on the overall pipeline that a witness execution is part of.

Proposed solution:

Implement threading and a worker queue to process files at the speed of the amount of processors available.

Anything else you would like to add:

I have done a test earlier that showed a 10x speed increase.

Testing changes required:

TBD

Documentation changes required:

TBD

[adityasaky]

I have done a test earlier that showed a 10x speed increase.

I thought this seemed familiar! Do you have a link handy to that?

[matglas]

Yes I do. https://github.com/matglas/go-hash-walk