Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 7 additions & 5 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,7 @@ history and issue-tracking systems. It analyzes commits, pull requests, and issu
notes, making it easy to integrate release documentation into your CI/CD pipelines and documentation workflows.

For detailed documentation, see the [User Guide](https://github.com/demaconsulting/BuildMark/blob/main/docs/user_guide/introduction.md).
For command-line options, see the [CLI Reference](https://github.com/demaconsulting/BuildMark/blob/main/docs/user_guide/cli-reference.md).

## Features

Expand Down Expand Up @@ -160,7 +161,7 @@ Changes, Bugs Fixed, and Dependency Updates sections with pre-wired routing rule
common label and work-item patterns.

For configuration details and examples, see the
[User Guide](https://github.com/demaconsulting/BuildMark/blob/main/docs/user_guide/introduction.md).
[Configuration Guide](https://github.com/demaconsulting/BuildMark/blob/main/docs/user_guide/configuration.md).

### Authentication

Expand All @@ -173,7 +174,7 @@ from environment variables at runtime.
`AZURE_DEVOPS_EXT_PAT`, then `SYSTEM_ACCESSTOKEN` (Azure Pipelines), then
`az account get-access-token` (Azure CLI).

For more detail see the [User Guide](https://github.com/demaconsulting/BuildMark/blob/main/docs/user_guide/introduction.md).
For more detail see the [Authentication Guide](https://github.com/demaconsulting/BuildMark/blob/main/docs/user_guide/introduction.md#with-github-token).

## Self Validation

Expand Down Expand Up @@ -209,8 +210,9 @@ Each test in the report proves:
- **`BuildMark_KnownIssuesReporting`** - Known issues are correctly included when requested.
- **`BuildMark_RulesRouting`** - Rules-based item routing assigns items to the correct report sections.

See the [User Guide](https://github.com/demaconsulting/BuildMark/blob/main/docs/user_guide/introduction.md) for more details
on the self-validation tests.
See the [CLI Reference][cli-ref] for more details on the self-validation tests.

[cli-ref]: https://github.com/demaconsulting/BuildMark/blob/main/docs/user_guide/cli-reference.md#self-validation

On validation failure the tool will exit with a non-zero exit code.

Expand All @@ -220,7 +222,7 @@ BuildMark supports an optional `buildmark` code block in issue and pull request
to control visibility, type classification, and affected-version ranges. Azure DevOps work items
additionally support native custom fields for the same controls.

For details, see the [User Guide](https://github.com/demaconsulting/BuildMark/blob/main/docs/user_guide/introduction.md).
For details, see the [Item Controls Guide](https://github.com/demaconsulting/BuildMark/blob/main/docs/user_guide/item-controls.md).

## Report Format

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -123,8 +123,14 @@ Main entry point. Performs the following steps:
12. Fetch linked work items for each PR via
`GET /git/repositories/{id}/pullrequests/{prId}/workitems` and batch-fetch
work item details via `GET /wit/workitems?ids={ids}&$expand=all`.
13. Collect known issues (open bugs not resolved at the time of the build) via a
WIQL query, applying item controls from description bodies and custom fields.
13. Collect known issues from **all** bugs (resolved and unresolved), via a WIQL
query, applying item controls from description bodies and custom fields.
For each candidate bug:
- If `AffectedVersions` is declared, the bug is a known issue if and only if
`AffectedVersions.Contains(toVersion)` is true, regardless of resolved
state. This covers resolved bugs that were never back-ported to older
branches (LTS back-port gap).
- If no `AffectedVersions` is declared, only unresolved bugs are included.
14. If routing rules are configured, call `ApplyRules` (inherited from
`RepoConnectorBase`) to distribute all collected items into the configured
report sections and populate `BuildInformation.RoutedSections`. If no rules
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -121,8 +121,14 @@ Main entry point. Performs the following steps:
8. Get all commits between the baseline and target.
9. Collect changes and bugs from pull requests merged in the commit range,
applying item controls overrides from description bodies.
10. Collect known issues (open issues not included in this build), applying item
controls overrides from description bodies.
10. Collect known issues from **all** issues (open and closed) by querying GitHub
with `states: [OPEN, CLOSED]` and applying item controls overrides from
description bodies. For each candidate bug:
- If `AffectedVersions` is declared, the bug is a known issue if and only if
`AffectedVersions.Contains(toVersion)` is true, regardless of open/closed
state. This covers closed bugs that were fixed in a later release but were
never back-ported to older branches (LTS back-port gap).
- If no `AffectedVersions` is declared, only open bugs are included.
11. Sort all lists chronologically.
12. If routing rules are configured, call `ApplyRules` (inherited from
`RepoConnectorBase`) to route all collected items into the configured report
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -31,6 +31,13 @@ determines the target and baseline versions, collects changes and known issues,
and returns a fully populated `BuildInformation` record. The logic mirrors the
production GitHubRepoConnector flow but operates entirely on in-memory data.

When collecting known issues, **all** issues (open and closed) are considered:

- If `AffectedVersions` is non-null, the bug is included if and only if
`AffectedVersions.Contains(targetVersion)` is true, regardless of open/closed
state (models a closed bug never back-ported to an older branch).
- If `AffectedVersions` is null, only open bugs are included.

When routing rules have been configured via `Configure`, `GetBuildInformationAsync`
collects all items and passes them to `ApplyRules` (inherited from `RepoConnectorBase`)
to produce the `RoutedSections` list. If no rules are configured, the legacy
Expand Down
24 changes: 21 additions & 3 deletions docs/reqstream/build-mark/build-mark.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -316,16 +316,34 @@ sections:
- BuildMark-Program-Report

- id: BuildMark-Report-KnownIssues
title: The tool shall support including known issues in build notes.
title: >-
The tool shall support including known issues in build notes, using
affected-versions metadata when available and falling back to open/closed
status when it is not.
justification: |
Disclosing known issues in release notes promotes transparency, helps users avoid
known pitfalls, and manages expectations about current limitations. This improves
user trust and reduces support burden.
known pitfalls, and manages expectations about current limitations.

The following rules determine whether a bug is a known issue for a given build:

1. A closed bug with no declared affected-versions is NOT a known issue.
2. An open bug with no declared affected-versions IS a known issue.
3. A bug in any state (open or closed) whose declared affected-versions contain
the build version IS a known issue.
4. A bug in any state (open or closed) whose declared affected-versions do NOT
contain the build version is NOT a known issue.

This matters for LTS branches: a bug may be closed after being fixed in a later
release, but an LTS branch that was cut before the fix still needs to report the
bug as a known issue. The affected-versions field captures that scenario precisely.
tests:
- IntegrationTest_Report_IncludesKnownIssues_WhenFlagIsSet
children:
- BuildMark-BuildNotes-ReportModel
- BuildMark-Program-Report
- BuildMark-RepoConnectors-GitHub
- BuildMark-RepoConnectors-AzureDevOps
- BuildMark-RepoConnectors-Mock

- id: BuildMark-Report-VersionRange
title: The tool shall support filtering build notes by version range.
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -45,13 +45,20 @@ sections:
The primary purpose of the AzureDevOpsRepoConnector is to assemble BuildInformation
from Azure DevOps repository data using the REST API, correctly identifying the current
version tag, the baseline (previous) version tag, and the changes between them.

Known-issues collection queries all bugs regardless of state. When affected-versions
is declared the item state is ignored; only the version range check determines
inclusion. This handles the LTS back-port gap scenario where a bug is resolved after
being fixed in a newer release but was never back-ported to an older branch.
tests:
- AzureDevOpsRepoConnector_GetBuildInformationAsync_WithMockedData_ReturnsValidBuildInformation
- AzureDevOpsRepoConnector_GetBuildInformationAsync_WithMultipleVersions_SelectsCorrectPreviousVersion
- AzureDevOpsRepoConnector_GetBuildInformationAsync_WithPullRequests_GathersChangesCorrectly
- AzureDevOpsRepoConnector_GetBuildInformationAsync_WithOpenWorkItems_IdentifiesKnownIssues
- AzureDevOpsRepoConnector_GetBuildInformationAsync_ReleaseVersion_SkipsAllPreReleases
- AzureDevOpsRepoConnector_ImplementsInterface_ReturnsTrue
- AzureDevOpsRepoConnector_GetBuildInformationAsync_KnownIssues_FilteredByAffectedVersions
- AzureDevOpsRepoConnector_GetBuildInformationAsync_ClosedBugWithMatchingAffectedVersions_IsKnownIssue

- id: BuildMark-AzureDevOps-ItemControls
title: >-
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -27,6 +27,11 @@ sections:
from GitHub repository data. It must correctly identify the current version tag,
the baseline (previous) version tag, and the changes between them, correctly
handling pre-release tags and version selection edge cases.

Known-issues collection considers all issues (open and closed). When
affected-versions is declared the issue state is ignored; only the version range
check determines inclusion. This handles the LTS back-port gap scenario where a
bug is closed after being fixed in a newer release but was never back-ported.
tests:
- GitHubRepoConnector_GetBuildInformationAsync_WithMockedData_ReturnsValidBuildInformation
- GitHubRepoConnector_GetBuildInformationAsync_WithMultipleVersions_SelectsCorrectPreviousVersionAndGeneratesChangelogLink
Expand All @@ -40,6 +45,8 @@ sections:
- GitHubRepoConnector_ImplementsInterface_ReturnsTrue
- GitHubRepoConnector_GetBuildInformationAsync_PrWithSubstringMatchLabel_NotClassifiedAsBug
- GitHubRepoConnector_GetBuildInformationAsync_IssueWithSubstringMatchLabel_NotClassifiedAsKnownIssue
- GitHubRepoConnector_GetBuildInformationAsync_KnownIssues_FilteredByAffectedVersions
- GitHubRepoConnector_GetBuildInformationAsync_ClosedBugWithMatchingAffectedVersions_IsKnownIssue

- id: BuildMark-GitHub-ItemControls
title: >-
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,11 @@ sections:
Tests that exercise the report generation and self-validation logic must run
without external dependencies. The MockRepoConnector provides a fixed,
predictable dataset so tests can assert exact outcomes.

The dataset includes a closed bug (issue 7) that carries an affected-versions
range, allowing tests to verify that the known-issues collection applies the
correct two-tier rule: version-range check for bugs with affected-versions, and
open/closed status fallback for bugs without.
tests:
- MockRepoConnector_Constructor_CreatesInstance
- MockRepoConnector_ImplementsInterface
Expand All @@ -24,3 +29,5 @@ sections:
- MockRepoConnector_Configure_StoresRulesAndSections
- MockRepoConnector_GetBuildInformationAsync_WithRules_ReturnsRoutedSections
- MockRepoConnector_GetBuildInformationAsync_WithoutRules_ReturnsNullRoutedSections
- MockRepoConnector_GetBuildInformationAsync_KnownIssues_FilteredByAffectedVersions
- MockRepoConnector_GetBuildInformationAsync_ClosedBugWithMatchingAffectedVersions_IsKnownIssue
Original file line number Diff line number Diff line change
Expand Up @@ -123,7 +123,7 @@ public override async Task<BuildInformation> GetBuildInformationAsync(VersionTag
lookupData);

// Collect known issues via WIQL query
var knownIssues = await CollectKnownIssuesAsync(restClient, allChangeIds, lookupData);
var knownIssues = await CollectKnownIssuesAsync(restClient, allChangeIds, lookupData, toVersion);

// Sort all lists by Index to ensure chronological order
nonBugChanges.Sort((a, b) => a.Index.CompareTo(b.Index));
Expand Down Expand Up @@ -559,21 +559,26 @@ private static void ProcessPullRequestWithoutWorkItems(
}

/// <summary>
/// Collects known issues (open bugs not resolved) via a WIQL query.
/// Collects known issues via a WIQL query.
/// When a bug declares <c>AffectedVersions</c>, it is a known issue if and only if
/// <c>AffectedVersions.Contains(targetVersion)</c> is true, regardless of its state.
/// When no <c>AffectedVersions</c> are declared, only unresolved bugs are included.
/// </summary>
/// <param name="restClient">Azure DevOps REST client.</param>
/// <param name="allChangeIds">Set of all change IDs already processed.</param>
/// <param name="allChangeIds">Set of all change IDs already processed in this build.</param>
/// <param name="lookupData">Lookup data structures.</param>
/// <param name="targetVersion">The version being built, used for affected-versions filtering.</param>
/// <returns>List of known issues.</returns>
private static async Task<List<ItemInfo>> CollectKnownIssuesAsync(
AzureDevOpsRestClient restClient,
HashSet<string> allChangeIds,
LookupData lookupData)
LookupData lookupData,
VersionTag targetVersion)
{
// Query for open bugs and issues
// Query all bugs and issues — state filtering is applied in code so that resolved
// bugs with a declared affected-versions range are still considered as known issues.
const string wiql = "SELECT [System.Id] FROM workitems " +
"WHERE [System.WorkItemType] IN ('Bug', 'Issue') " +
"AND [System.State] NOT IN ('Done', 'Closed', 'Resolved')";
"WHERE [System.WorkItemType] IN ('Bug', 'Issue')";
Comment thread
Malcolmnixon marked this conversation as resolved.

var queryResult = await restClient.QueryWorkItemsAsync(wiql);
if (queryResult.WorkItems.Count == 0)
Expand All @@ -590,21 +595,26 @@ private static async Task<List<ItemInfo>> CollectKnownIssuesAsync(
{
var workItemId = workItem.Id.ToString(CultureInfo.InvariantCulture);

// Skip items already included as changes
// Skip items already included as changes in this build
if (allChangeIds.Contains(workItemId))
{
continue;
}

// Skip resolved work items (defense in depth)
if (WorkItemMapper.IsWorkItemResolved(workItem))
var workItemUrl = BuildWorkItemUrl(lookupData.OrganizationUrl, lookupData.Project, workItem.Id);
var itemInfo = WorkItemMapper.MapWorkItemToItemInfo(workItem, workItemUrl, workItem.Id);
if (itemInfo == null || itemInfo.Type != "bug")
{
continue;
}

var workItemUrl = BuildWorkItemUrl(lookupData.OrganizationUrl, lookupData.Project, workItem.Id);
var itemInfo = WorkItemMapper.MapWorkItemToItemInfo(workItem, workItemUrl, workItem.Id);
if (itemInfo != null && itemInfo.Type == "bug")
// With affected-versions: include if version matches, regardless of state.
// Without affected-versions: only unresolved bugs are included.
var isKnownIssue = itemInfo.AffectedVersions != null
? itemInfo.AffectedVersions.Contains(targetVersion)
: !WorkItemMapper.IsWorkItemResolved(workItem);

if (isKnownIssue)
{
knownIssues.Add(itemInfo);
}
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -511,7 +511,7 @@ public async Task<List<IssueNode>> GetAllIssuesAsync(
Query = @"
query($owner: String!, $repo: String!, $after: String) {
repository(owner: $owner, name: $repo) {
issues(first: 100, after: $after) {
issues(first: 100, states: [OPEN, CLOSED], after: $after) {
nodes {
number
title
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -156,7 +156,7 @@ public override async Task<BuildInformation> GetBuildInformationAsync(VersionTag
repo);

// Collect known issues
var knownIssues = CollectKnownIssues(gitHubData.Issues, allChangeIds);
var knownIssues = CollectKnownIssues(gitHubData.Issues, allChangeIds, toVersion);

// Sort all lists by Index to ensure chronological order
nonBugChanges.Sort((a, b) => a.Index.CompareTo(b.Index));
Expand Down Expand Up @@ -730,21 +730,50 @@ private static void ProcessPullRequestWithoutIssues(
}

/// <summary>
/// Collects known issues (open bugs not fixed in this build).
/// Collects known issues from the full issue list.
/// When a bug declares <c>AffectedVersions</c>, it is a known issue if and only if
/// <c>AffectedVersions.Contains(targetVersion)</c> is true, regardless of its open/closed
/// state. When no <c>AffectedVersions</c> are declared, only open bugs are included.
/// </summary>
/// <param name="issues">All issues from GitHub.</param>
/// <param name="allChangeIds">Set of all change IDs already processed.</param>
/// <param name="issues">All issues from GitHub (open and closed).</param>
/// <param name="allChangeIds">Set of all change IDs already processed in this build.</param>
/// <param name="targetVersion">The version being built, used for affected-versions filtering.</param>
Comment thread
Malcolmnixon marked this conversation as resolved.
/// <returns>List of known issues.</returns>
private static List<ItemInfo> CollectKnownIssues(IReadOnlyList<IssueInfo> issues, HashSet<string> allChangeIds)
private static List<ItemInfo> CollectKnownIssues(
IReadOnlyList<IssueInfo> issues,
HashSet<string> allChangeIds,
VersionTag targetVersion)
{
return issues
.Where(i => i.State == "OPEN")
.Select(issue => (issue, issueId: issue.Number.ToString(CultureInfo.InvariantCulture)))
.Where(tuple => !allChangeIds.Contains(tuple.issueId))
.Select(tuple => CreateItemInfoFromIssue(tuple.issue, tuple.issue.Number))
.OfType<ItemInfo>()
.Where(itemInfo => itemInfo.Type == "bug")
.ToList();
List<ItemInfo> knownIssues = [];

foreach (var issue in issues)
{
// Skip issues already addressed in this build
var issueId = issue.Number.ToString(CultureInfo.InvariantCulture);
if (allChangeIds.Contains(issueId))
{
continue;
}

var itemInfo = CreateItemInfoFromIssue(issue, issue.Number);
if (itemInfo == null || itemInfo.Type != "bug")
{
continue;
}

// With affected-versions: include if version matches, regardless of state.
// Without affected-versions: only open bugs are included.
var isKnownIssue = itemInfo.AffectedVersions != null
? itemInfo.AffectedVersions.Contains(targetVersion)
: issue.State == "OPEN";

if (isKnownIssue)
{
knownIssues.Add(itemInfo);
}
}

return knownIssues;
}

/// <summary>
Expand Down
Loading
Loading