util: testing: manifest: shim: Initial commit

[johnandersen777]

This concept of a manifest seems important (see 2ndparty dependency declaration section for background on how this applies to DFFML, it applies to other projects as well.

If we (talking to non-DFFML CI/CD community people here) use a shim layer it sets us up for forward compatibility. We can all work to define manifest formats that work well for our projects. By using the shim we ensure that they share enough properties that we set ourselves up for an easier eventual merging for manifest formats into one common CI/CD manifest format.

From the DFFML side, a manifest can easily be represented as a DataFlow. The manifest is a more user friendly form of a dataflow essentially, something more tailored to a given use case which describes which operations should be preformed, in what order, and what data types are involved. The shim layer allows projects to parse the manifest however they want. It allows them to easily switch what their next phase is at any time. This next phase might be their existing workflow, or it might be a dataflow enabled workflow. If the next phase parser outputs a dataflow when given a manifest we'll ensure auditability and achievability of the subsequent phase processing that will be taken.

The shim layer effectively operates as a trampoline (think UEFI shim, kexec, etc.) bridging the gap between the initial phase of manifest parsing and the next phase (which in DFFML's case is looking like metric collection, ml job execution, and executing our own CI both in the upcoming polyrepo setup as well as enabling local testing).

• https://www.agwa.name/blog/post/ssh_signatures
  • https://github.com/intel/dffml/blob/alice/docs/arch/0007-A-GitHub-Public-Bey-and-TPM-Based-Supply-Chain-Security-Mitigation-Option.rst
  • Architecting Alice: Volume 0: Context: Part 28: Confirmed Decision to use Same Private for X/Ed25519
    • https://github.com/SmithSamuelM/Papers/blob/master/whitepapers/KERI_WP_2.x.web.pdf

[codecov-commenter]

Codecov Report

Merging #1273 (eece55a) into master (e91235f) will decrease coverage by 0.21%.
The diff coverage is 53.62%.

❗ Current head eece55a differs from pull request most recent head 87e8a1e. Consider uploading reports for the commit 87e8a1e to get more accurate results

@@            Coverage Diff             @@
##           master    #1273      +/-   ##
==========================================
- Coverage   84.97%   84.76%   -0.22%     
==========================================
  Files         174      175       +1     
  Lines       10975    11044      +69     
  Branches     1821     1829       +8     
==========================================
+ Hits         9326     9361      +35     
- Misses       1280     1310      +30     
- Partials      369      373       +4     

Impacted Files	Coverage Δ
dffml/util/testing/manifest/shim.py	53.62% <53.62%> (ø)
dffml/df/memory.py	79.48% <0.00%> (-0.26%)	⬇️

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update e91235f...87e8a1e. Read the comment docs.

[johnandersen777]

doctest and Popen.communicate were not playing nice together, so when with pipe for now

[johnandersen777]

schema validation goes here

probably want to add only_schema_check here

[johnandersen777]

TODO validate @context,vocab etc. see w3c/vc-data-model#953 (comment) as a part of DID related quest line

[johnandersen777]

• https://multiformats.io/
• https://github.com/ChainAgnostic/varsig

[johnandersen777]

Could treat . as namespace