The current evaluation workflow operates solely on skills. This mean a test tries to validate a specific skill. It currently doesn't validate E2E when a consumer installs the plugin and uses all the available skills + agents.
That basically means we aren't validating:
- How skills interact with each other
- Custom agents
- If the right skill gets picked via a prompt
- If the plugin itself works correctly (i.e. metadata, has all the intended skills, etc).
cc @JanKrivanek @jeffschwMSFT
The current evaluation workflow operates solely on skills. This mean a test tries to validate a specific skill. It currently doesn't validate E2E when a consumer installs the plugin and uses all the available skills + agents.
That basically means we aren't validating:
cc @JanKrivanek @jeffschwMSFT