fix(harvest): retry failed sets serially with backoff (#613 Option C)

[jsboige]

What

Closes #613 (Option C — retry serial). After the parallel harvest loop drains (ContinueOnHarvestSetFailure, #614), re-attempt each failed set serially (degree=1) with configurable backoff. A large set that timed out under high parallelism frequently succeeds when it has CardPen/Playwright to itself.

This is the runtime-resilience gap surfaced during the v0.9.0 release regen (cycle where EnableParallelism=true → large sets timed out → Mismatch throw). The first mitigation was #614 (don't abort the whole harvest on one set failure). This PR adds the second mitigation: recover the failed sets automatically before declaring partial failure.

Changes

WebBasedGeneratorConfig.cs (+17) — 2 new config props with XML doc:

• HarvestSetRetryAttempts (default 1)
• HarvestSetRetryBackoffSeconds (default 30)

HarvestManager.cs (+102) —

• Seam (between the Parallel.ForEachAsync drain and the failedSets.IsEmpty check): gated by HarvestSetRetryAttempts > 0, calls RetryFailedHarvestSetsAsync and replaces the bag with the residual.
• RetryFailedHarvestSetsAsync (private): iterates the failure bag, looks up each CardSetJob by .Name, re-runs ProcessLocalizedHarvest serially per item via the pure RetryAsync helper. Re-renders correctly because a failed set never added its key to the dictionary (ContainsKey guard in ProcessLocalizedHarvest). Returns a residual bag → the existing [HARVEST-PARTIAL] aggregate-error throw then reports only what genuinely could not be harvested.
• RetryAsync (internal static, pure): runs action × attempts, backoff between, swallows the last exception (logs it) and returns bool. Extracted as a pure helper so the retry/backoff contract is unit-testable without a browser (precedent: ComputeExpectedImageCount).

Design notes

• Does not touch ParallelismOptimizer — retry is strictly post-loop, orthogonal to parallelism tuning.
• Reuses the exact same call path (ProcessLocalizedHarvest) — no duplicate harvest logic.
• Opt-in via HarvestSetRetryAttempts > 0 — defaults are conservative (1 attempt, 30 s backoff); set 0 to disable and recover the current behavior.
• Serial by construction — no new concurrency concerns (QuestPDF global lock / Playwright page pool unaffected; the parallel loop has already drained).

Verification

• Build: 0 errors (3 pre-existing warnings SYSLIB0014 + SGEN, unrelated).
• Tests: 570 pass / 5 skip / 1 known-fail. The single failure is OwlE2EGenerationValidationTests.LoadedOntology_RdfTypeAndInScheme_DroppedByOwl2XmlRoundTrip — pre-existing OWLSharp round-trip bug tracked by feat(ontology): Publish OWL ontology to public endpoint #133, unrelated to harvest. No regression.

Known interaction — #630

Bug #630 (Spectre [HARVEST-FAILURE] fatal) short-circuits #614 on a set-level failure path, which would also bypass this retry. This PR is correct and valuable on its own (it fires whenever #614's ContinueOnHarvestSetFailure path is active — the common case), but a full fix for the fatal-path requires #630 to be resolved separately. Documented here for transparency; not a blocker for this PR.

Garde-fou

Pure pipeline code (.cs), no runtime/2sxc/DNN surface — not subject to the #596 runtime-pending gate. Ready for review.

🤖 Generated with Claude Code