Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions .cspell/general-technical.txt
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,7 @@ acunetix
adobe
adrm
adrs
agentic
agentless
agile
anom
Expand Down Expand Up @@ -160,6 +161,7 @@ chatbot
chatbots
chatmode
chatmodes
chatlog
cheatham
checkbox
checkboxes
Expand Down
50 changes: 50 additions & 0 deletions docs/contributing/architecture.md
Original file line number Diff line number Diff line change
Expand Up @@ -177,6 +177,56 @@ This repository will not maintain a separate codebase domain for simulation. Ins
- Designing Isaac Lab environments using Manager-based and Direct workflows
- Connecting simulation outputs to the Synthetic Data and Training domains

## Agentic Tooling

This project uses [GitHub Copilot](https://code.visualstudio.com/docs/copilot/overview) agents, instructions, prompts, and skills to automate development workflows. Tooling comes from two sources: the HVE-Core extension (shared across Microsoft HVE projects) and project-specific artifacts defined in `.github/`.

### HVE-Core Extension

The [hve-core-all](https://marketplace.visualstudio.com/items?itemName=ise-hve-essentials.hve-core-all) VS Code extension provides shared agentic tooling:

| Artifact Type | Count | Examples |
|---------------|-------|----------|
| Agents | 33 | RPI workflow, backlog management, PR creation |
| Instructions | 24 | Coding standards (Bash, C#, Python, Terraform, Bicep), commit messages, markdown |
| Prompts | 27 | ADO work items, GitHub issues, security planning, PR descriptions |
| Skills | 2 | PR reference generation, video-to-GIF conversion |

HVE-Core artifacts are registered via the extension's `package.json` `contributes` section and loaded when the extension activates.

### Project Copilot Artifacts

This repository defines project-specific artifacts in `.github/` that extend HVE-Core with domain knowledge:

| Artifact Type | Count | Purpose |
|---------------|-------|----------|
| Agents | 2 | OSMO training manager, dataviewer developer |
| Instructions | 4 | Copilot instructions, dataviewer conventions, documentation style, shell scripts |
| Prompts | 4 | OSMO training submission, LeRobot pipeline, dataviewer workflows |
| Skills | 2 | Dataviewer interaction, OSMO LeRobot training |

Project artifacts are auto-discovered by VS Code from the `.github/` directory without explicit registration.

Two workflow chains compose these artifacts:

- **OSMO Training Manager**: `osmo-training-manager` agent β†’ `osmo-lerobot-training` skill β†’ training submission prompts
- **Dataviewer Developer**: `dataviewer-developer` agent β†’ `dataviewer` skill β†’ dataviewer instruction conventions

### Artifact Types and Loading

Each artifact type uses YAML frontmatter to declare behavior:

| Artifact | File Pattern | Key Frontmatter | Loading |
|----------|-------------|-----------------|----------|
| Agents | `*.agent.md` | `mode`, `tools`, `description` | Auto-discovered from `.github/agents/` |
| Instructions | `*.instructions.md` | `applyTo`, `description` | Auto-discovered from `.github/instructions/` |
| Prompts | `*.prompt.md` | `mode`, `description`, `tools` | Auto-discovered from `.github/prompts/` |
| Skills | `SKILL.md` | N/A (referenced by agents) | Referenced via `copilot-skill:` URI |

HVE-Core artifacts follow the same patterns but load through extension contribution points rather than workspace auto-discovery.

For the detailed per-artifact inventory and workflow chain diagrams, see [Copilot Artifacts](../reference/copilot-artifacts.md).

## Agent Skills and Specification Documents

Each domain will contain specification documents alongside working examples. These specifications serve as structured inputs for [GitHub Copilot Agent Skills](https://code.visualstudio.com/docs/copilot/chat/chat-agent-mode), enabling customers to adapt this reference architecture to their own codebase and infrastructure.
Expand Down
2 changes: 2 additions & 0 deletions docs/contributing/prerequisites.md
Original file line number Diff line number Diff line change
Expand Up @@ -189,6 +189,8 @@ The workspace `.vscode/settings.json` also configures Copilot Chat to load instr

These paths resolve when hve-core is installed as a peer directory or via the VS Code Extension. Without hve-core, Copilot still functions but shared conventions, prompts, and chat modes are unavailable.

For a complete list of available agents, prompts, and skills, see [Copilot Artifacts](../reference/copilot-artifacts.md).

## Related Documentation

* [Contributing Guide](README.md) - Main contributing guide with all sections
Expand Down
1 change: 1 addition & 0 deletions docs/reference/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -20,6 +20,7 @@ Technical reference documentation for submission scripts, workflow templates, en
|----------------------------------------|--------------------------------------------------------------------|-------------|
| [Script Reference](scripts.md) | Submission script inventory, CLI arguments, and configuration | Available |
| [Script Examples](scripts-examples.md) | Detailed examples for training, inference, and pipeline submission | Available |
| [Copilot Artifacts](copilot-artifacts.md) | Agents, instructions, prompts, and skills inventory | Available |
| Workflow Templates | AzureML and OSMO workflow YAML templates and configuration | Coming soon |
| Terraform Variables | Terraform input variables and outputs reference | Coming soon |
| Environment Variables | Environment variables for training and deployment | Coming soon |
Expand Down
177 changes: 177 additions & 0 deletions docs/reference/copilot-artifacts.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,177 @@
---
title: Copilot Artifacts
description: >-
Inventory and reference for GitHub Copilot agents, instructions, prompts,
and skills configured in this repository.
author: Microsoft Robotics-AI Team
ms.date: 2026-03-11
ms.topic: reference
keywords:
- copilot
- agents
- instructions
- prompts
- skills
- extensibility
---

GitHub Copilot extensibility artifacts provide AI-assisted workflows
for dataset analysis, training job management, and coding standards
enforcement. These artifacts are configured in `.github/` and activate
automatically in VS Code.

## πŸ“‹ Artifact Inventory

| Type | Name | Description | Path |
|-------------|------------------------------|------------------------------------------------------|------------------------------------------------------------------|
| Agent | Dataviewer Developer | Interactive dataset analysis and tool development | `.github/agents/dataviewer-developer.agent.md` |
| Agent | OSMO Training Manager | LeRobot training lifecycle on OSMO with Azure ML | `.github/agents/osmo-training-manager.agent.md` |
| Instruction | Commit Messages | Conventional Commits format for all commit messages | `.github/instructions/commit-message.instructions.md` |
| Instruction | Dataviewer | Coding standards for dataviewer development | `.github/instructions/dataviewer.instructions.md` |
| Instruction | Docs Style and Conventions | Writing standards for all markdown files | `.github/instructions/docs-style-and-conventions.instructions.md`|
| Instruction | Shell Scripts | Implementation standards for bash scripts | `.github/instructions/shell-scripts.instructions.md` |
| Prompt | `/chatlog` | Create and maintain conversation logs | `.github/prompts/chatlog.prompt.md` |
| Prompt | `/check-training-status` | Monitor OSMO training job progress | `.github/prompts/check-training-status.prompt.md` |
| Prompt | `/start-dataviewer` | Launch Dataset Analysis Tool | `.github/prompts/start-dataviewer.prompt.md` |
| Prompt | `/submit-lerobot-training` | Submit LeRobot training job to OSMO | `.github/prompts/submit-lerobot-training.prompt.md` |
| Skill | dataviewer | Dataset browsing, annotation, and export | `.github/skills/dataviewer/SKILL.md` |
| Skill | osmo-lerobot-training | Training submission, monitoring, and analysis | `.github/skills/osmo-lerobot-training/SKILL.md` |

## πŸ”— Quick Reference

| Want to... | Use this artifact |
|-------------------------------------------|----------------------------------------------------------|
| Launch the Dataset Analysis Tool | `/start-dataviewer` prompt β†’ Dataviewer Developer |
| Browse and annotate training episodes | Dataviewer Developer agent |
| Submit a LeRobot training job | `/submit-lerobot-training` prompt β†’ OSMO Training Manager|
| Check training job status | `/check-training-status` prompt β†’ OSMO Training Manager |
| Save a conversation log | `/chatlog` prompt |
| Enforce commit message standards | `commit-message` instruction (auto-applied) |
| Enforce coding standards in dataviewer | `dataviewer` instruction (auto-applied) |
| Enforce markdown writing standards | `docs-style-and-conventions` instruction (auto-applied) |
| Enforce shell script standards | `shell-scripts` instruction (auto-applied) |

## πŸ€– Agents

### Dataviewer Developer

Interactive agent for launching, browsing, annotating, and improving the
Dataset Analysis Tool.

| Property | Value |
|----------|-----------------------------------------------------|
| Handoffs | Start Dataviewer, Browse Dataset, Annotate Episodes |
| Tools | All (no restrictions) |
| Skill | `dataviewer` |
| Prompts | `/start-dataviewer` |

Four-phase workflow: Launch/Configure β†’ Interactive Browsing (Playwright) β†’
Episode Annotation (API+UI) β†’ Feature Development (React+FastAPI).

### OSMO Training Manager

Multi-turn agent for managing LeRobot imitation learning training lifecycle
on OSMO with Azure ML integration.

| Property | Value |
|----------|---------------------------------------------------------|
| Handoffs | Submit Training Job, Check Training Status, Run Inference Evaluation |
| Tools | 11 explicit (run_in_terminal, memory, runSubagent, ...) |
| Skill | `osmo-lerobot-training` |
| Prompts | `/submit-lerobot-training`, `/check-training-status` |

Five-phase workflow: Submit β†’ Monitor β†’ Analyze β†’ Summarize β†’ Inference
Evaluation. Handles VM eviction recovery, CUDA errors, and KeyError
failures.

## πŸ“ Instructions

Instructions activate automatically when files matching their `applyTo`
pattern appear in the chat context.

| Name | Applies To | Purpose |
|----------------------------|---------------------|---------------------------------------------------------|
| Commit Messages | `**` | Conventional Commits format, scopes, line-length limits |
| Dataviewer | `src/dataviewer/**` | SOLID principles, test-first, validation commands |
| Docs Style and Conventions | `**/*.md` | Document hierarchy, tables, voice/tone, frontmatter |
| Shell Scripts | `**/*.sh` | Script template, library functions, deployment patterns |

## ⚑ Prompts

Prompts are slash commands invoked via `/` in the chat input. Each prompt
targets a specific agent.

| Command | Agent Target | Required Inputs |
|----------------------------|-----------------------|-----------------------|
| `/chatlog` | Generic | None |
| `/check-training-status` | OSMO Training Manager | workflowId (optional) |
| `/start-dataviewer` | Dataviewer Developer | datasetPath |
| `/submit-lerobot-training` | OSMO Training Manager | dataset (required) |

## πŸ› οΈ Skills

Skills provide multi-file capabilities with progressive 3-level loading:
discovery (frontmatter only) β†’ instructions (SKILL.md body) β†’ resources
(bundled reference files).

### dataviewer

| Property | Value |
|-----------|--------------------------------------------------------------------------|
| Directory | `.github/skills/dataviewer/` |
| Resources | `references/PLAYWRIGHT.md` (selectors, interaction recipes, API endpoints) |
| Used by | Dataviewer Developer agent |

### osmo-lerobot-training

| Property | Value |
|-----------|----------------------------------------------------------------------------------------------------------------|
| Directory | `.github/skills/osmo-lerobot-training/` |
| Resources | `references/DEFAULTS.md` (env, datasets, GPU profiles), `references/REFERENCE.md` (CLI, inference, AzureML navigation) |
| Used by | OSMO Training Manager agent |

## πŸ”„ Workflow Chains

Agents compose prompts and skills into end-to-end workflows:

```text
OSMO Training Manager (agent)
β”œβ”€β”€ /submit-lerobot-training (prompt)
β”œβ”€β”€ /check-training-status (prompt)
└── osmo-lerobot-training (skill)
β”œβ”€β”€ references/DEFAULTS.md
└── references/REFERENCE.md

Dataviewer Developer (agent)
β”œβ”€β”€ /start-dataviewer (prompt)
└── dataviewer (skill)
└── references/PLAYWRIGHT.md

Standalone:
└── /chatlog (prompt, generic)
```

## βž• Adding New Artifacts

VS Code provides generator commands for scaffolding new artifacts:

- `/create-agent` β€” Create a new custom agent
- `/create-instruction` β€” Create a new instruction file
- `/create-prompt` β€” Create a new prompt file
- `/create-skill` β€” Create a new agent skill

Place new artifacts in the corresponding `.github/` subdirectory and update
this inventory page.

## Related Documentation

For broader project context, see these companion guides:

- [Contributing Guide](../contributing/README.md) β€” Development workflow and coding standards
- [Architecture](../contributing/architecture.md) β€” System architecture and agent skills design
- [Prerequisites](../contributing/prerequisites.md) β€” Required tools and VS Code settings

---

*Crafted with precision by Copilot following brilliant human instruction,
then carefully refined by our team of discerning human reviewers.*
Loading