Repository navigation
feat(providers): onboard Morph as a catalog provider - #1648
Conversation
|
Important Review skippedAuto reviews are disabled on base/target branches other than the default branch. Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: Repository: juspay/neurolink/.coderabbit.yaml Review profile: CHILL Plan: Advanced Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
✅ Single Commit Policy - COMPLIANTStatus: Policy requirements met • 1 commit • Valid format • Ready for merge 📊 View validation details📝 Commit Details
✅ Validation Results
🤖 Automated validation by NeuroLink Single Commit Enforcement |
Documentation Validation Results🚀 Documentation validation passed!
📦 Build artifact uploaded successfully. Ready for deployment preview. Commit: |
9477bdb to
cbee555
Compare
Tara-ag
left a comment
There was a problem hiding this comment.
Review: Morph catalog onboarding
Verdict: APPROVE — clean catalog-only onboarding; the two "traps" (tool calling 200-with-no-tool_calls, array content → 500) are both correctly reflected as data flags that the existing catalog machinery genuinely enforces (supportsTools off keeps tools off the wire; messageContentFormat:"string" is consumed by the loader + ConfiguredOpenAICompatProvider.adjustRequestBody), with test coverage and nightly env wiring included.
| Severity | File:line | Finding |
|---|---|---|
| MINOR | src/lib/providers/catalog/morph.json:12 |
morph-v3-fast is a code-edit specialist that the PR's own evidence says returns unrelated text when used for chat; it is the secondary fallback/fallbackModelName |
Checked and clean
messageContentFormat:"string"quirk →loader.tsreadsentry.quirks.messageContentFormat→ConfiguredOpenAICompatProvider.adjustRequestBody(src/lib/providers/configuredOpenAICompat.ts). Same path as Cloudflare. ✓tools:false/toolsWithStreaming:false→loader.tsmapscapabilities.tools→supportsTools;executeStream/generate gate onsupportsTools()so notoolsarray ever reaches the Morph wire. ✓errorRules401/400 with{model}/{apiKeyEnvVar}/{setupUrl}placeholders →buildErrorRulesinterpolates them. ✓- Credential/env resolution:
catalogEnvVar("morph", …)→MORPH_API_KEY/MORPH_BASE_URL/MORPH_MODEL, matching the nightlyMORPH_API_KEYwiring. ✓ AIProviderName.MORPH+NeurolinkCredentials.morph?+CatalogProviderName/CatalogCredentialKeyare all additive (rule 5 backward-compatible); no hand registration needed beyond catalog + registry loop.addedInPR:"pending"matches the convention for pre-merge catalog entries (mancer, gmicloud, io-intelligence …). Not a finding.- Generated
index.generated.ts/providerCatalog.generated.ts+docs/api/**are codegen/CI-owned (skipped per repo conventions). - Alias tests for
morph/morphllmin the openai-compat catalog suite match the catalog (URLapi.morphllm.com/v1, defaultmorph-v3-large).
Impact-analysis note: I could not run the code-review-graph blast-radius tools in this environment (the local checkout is not indexed and the sandbox blocked filesystem access outside .yama/). Blast radius was instead assessed from the diff plus remote source inspection via the GitHub API. The change is a purely additive catalog/enum registration touching only the generic catalog+registry machinery, so no out-of-diff callers are expected to break.
Minor doc nit (not blocking): the PR body states all four tool probes ended finish_reason=stop, but morph.json's evidence.liveMatrix.result records morph-v3-fast returning finish_reason=length. Worth reconciling so the evidence stays internally consistent.
Tara-ag
left a comment
There was a problem hiding this comment.
Minor design note on the Morph fallback chain; onboarding otherwise looks correct (verified quirks/capabilities/error-rules wiring against the catalog loader and OpenAI-compat provider code).
Tara-ag
left a comment
There was a problem hiding this comment.
Approving per the full review in the summary comment (<!-- yama:summary -->): clean catalog-only onboarding with the two mechanics flags correctly enforced and test/CI coverage included. The single non-blocking MINOR finding (morph-v3-fast as the chat fallback in morph.json:12) is tracked in the inline thread and does not block merge.
cbee555 to
364c6a6
Compare
Tara-ag
left a comment
There was a problem hiding this comment.
Recurring review — Morph catalog onboarding
Verdict: APPROVE — re-confirmed on current head 364c6a6. The head commit SHA moved since the last review (cbee555 → 364c6a6) but the diff content is identical; the catalog-only onboarding remains clean, additive, and the two "trap" flags (tools:false, messageContentFormat:"string") are genuinely enforced by the catalog loader + ConfiguredOpenAICompatProvider machinery. No new findings.
| Severity | File:line | Finding | Status |
|---|---|---|---|
| MINOR | src/lib/providers/catalog/morph.json:12 |
morph-v3-fast is a code-edit specialist used as the secondary chat fallback — a general-chat request can get unrelated output when the primary model is unavailable |
Retained (inline thread) |
| MINOR | PR body / morph.json evidence.liveMatrix.result |
PR body says all four tool probes ended finish_reason=stop, but the evidence records morph-v3-fast at finish_reason=length; reconcile for internal consistency |
New (doc nit, non-blocking) |
On the retained finding: the inline thread (marker yama:morph-fallback) has no author reply, and the evidence in morph.json still confirms morph-v3-fast "ignores the tools parameter and returns unrelated generated text" and is "purpose-built for applying code edits rather than general chat". So it is kept as a MINOR design note, not elevated — the fallback only engages on invalid-model / discovery-empty paths, defaults resolve to morph-v3-large.
Checked and re-verified clean on this pass
tools:false/toolsWithStreaming:false→supportsTools()gate keepstoolsoff the Morph wire. ✓messageContentFormat:"string"→adjustRequestBodystring-content path (same quirk as Cloudflare). ✓- Additive surface only:
AIProviderName.MORPH,NeurolinkCredentials.morph?,CatalogProviderName/CatalogCredentialKey,MorphModels— no exhaustive switch / hand-registration breakage;detect_changesrisk 0.00, no affected flows. ✓ errorRules401/400 placeholders + credential env resolution →MORPH_API_KEY/MORPH_BASE_URL/MORPH_MODEL. ✓- Alias tests for
morph/morphllmmatch the catalog (URLapi.morphllm.com/v1, defaultmorph-v3-large); nightlyMORPH_API_KEYwired. ✓ docs/api/**,llms-full.txt,index.generated.ts,providerCatalog.generated.tsare codegen/CI-owned (skipped per repo conventions). ✓- Check runs on the current head include the review agent; prior gates (
test:providers-mocked,test:provider-structure, catalog codegen drift, 43/43 alias suite) plusSingle Commit PolicyandGenerated artifacts are currentpass.
Live acceptance run — SDK and CLI, real vendorRun from this PR's worktree at head All four surfaces pass. And a correction to my own earlier comment on the fallback threadWhile running this I also probed the no-model path, and it contradicts what I claimed in #1648 (comment). Details are on that thread; the short version is that the default model resolves to I had read |
`loader.ts` derives a provider's fallback positionally —
`fallbackModelName ?? fallbacks[1] ?? fallbacks[0]` — and nothing ever
checked the result. When that derivation lands on the provider's own
default, the retry path is silently dead: the primary fails, the SDK
retries the identical model, and it fails again for the same reason.
Auditing all 16 shipped catalogs found three in that state, so this is
not a hypothetical:
| catalog | models | fallback == default | avoidable |
| -------------- | ------ | ------------------- | --------- |
| gmicloud | 1 | yes | no |
| inception-labs | 1 | yes | no |
| mistral | 32 | yes | YES |
`mistral.json` ships 32 models and explicitly pinned `fallbackModelName`
to its own default, overriding a perfectly serviceable `fallbacks[1]`
(`mistral-large-latest`). Removing that one line restores a real
fallback. The two single-model catalogs have no alternative to name and
are exempt by construction — the check only fires when the catalog
declares more than one model, so it reports a mistake rather than an
unavoidable fact.
The rule lives in the schema's existing `superRefine`, beside the checks
that already constrain `default` and `fallbacks` to catalog keys, so it
runs wherever catalogs are parsed. That is both required CI jobs:
`test` via `codegen:catalog -- --check` (ci.yml:221) and
`provider-safety-net` via `verify:provider-onboarding` (ci.yml:440),
which imports `parseProviderCatalogJson` directly.
Before, on the unmodified tree:
Error: Invalid provider catalog file mistral.json:
models.fallbackModelName: the fallback model resolves to
"mistral-small-2506", which is also models.default — a fallback
identical to the primary cannot fall back.
exit 1
After:
codegen:catalog complete — 15 providers
verify:provider-onboarding — 8/8 new providers fully onboarded
test:openai-compat-catalog — 41 passed · 0 failed
exit 0
No generated artifacts change: the fallback is resolved at load time,
not emitted into a generated file.
Found while investigating a review thread on #1648 which suspected
Morph's fallback. Morph turned out to be fine — its fallback differs
from its default — and the real defect was in a provider nobody was
looking at, reached by checking the derivation itself instead.
`loader.ts` derives a provider's fallback positionally —
`fallbackModelName ?? fallbacks[1] ?? fallbacks[0]` — and nothing ever
checked the result. When that derivation lands on the provider's own
default, the retry path is silently dead: the primary fails, the SDK
retries the identical model, and it fails again for the same reason.
Auditing all 16 shipped catalogs found three in that state, so this is
not a hypothetical:
| catalog | models | fallback == default | avoidable |
| -------------- | ------ | ------------------- | --------- |
| gmicloud | 1 | yes | no |
| inception-labs | 1 | yes | no |
| mistral | 32 | yes | YES |
`mistral.json` ships 32 models and explicitly pinned `fallbackModelName`
to its own default, overriding a perfectly serviceable `fallbacks[1]`
(`mistral-large-latest`). Removing that one line restores a real
fallback. The two single-model catalogs have no alternative to name and
are exempt by construction — the check only fires when the catalog
declares more than one model, so it reports a mistake rather than an
unavoidable fact.
The rule lives in the schema's existing `superRefine`, beside the checks
that already constrain `default` and `fallbacks` to catalog keys, so it
runs wherever catalogs are parsed. That is both required CI jobs:
`test` via `codegen:catalog -- --check` (ci.yml:221) and
`provider-safety-net` via `verify:provider-onboarding` (ci.yml:440),
which imports `parseProviderCatalogJson` directly.
Before, on the unmodified tree:
Error: Invalid provider catalog file mistral.json:
models.fallbackModelName: the fallback model resolves to
"mistral-small-2506", which is also models.default — a fallback
identical to the primary cannot fall back.
exit 1
After:
codegen:catalog complete — 15 providers
verify:provider-onboarding — 8/8 new providers fully onboarded
test:openai-compat-catalog — 41 passed · 0 failed
exit 0
No generated artifacts change: the fallback is resolved at load time,
not emitted into a generated file.
Found while investigating a review thread on #1648 which suspected
Morph's fallback. Morph turned out to be fine — its fallback differs
from its default — and the real defect was in a provider nobody was
looking at, reached by checking the derivation itself instead.
364c6a6 to
af71220
Compare
Tara-ag
left a comment
There was a problem hiding this comment.
Reviewed Morph catalog onboarding. Clean PR; one MINOR follow-up on the test suite's env-var neutralization. Details in the summary comment.
Verdict: APPROVEMorph is a well-evidenced catalog onboarding. Capability flags are grounded in live wire probes (tools deliberately
Existing thread — accepted (resolved)
Checked and found clean
No CRITICAL / MAJOR findings. |
Tara-ag
left a comment
There was a problem hiding this comment.
Approving per the full review in the summary comment (<!-- yama:summary -->). Clean additive catalog onboarding; the two non-blocking MINOR findings (morph-v3-fast as chat fallback — author-justified, thread resolved — and the MORPH_* env-var neutralize gap) do not block merge.
Review: Morph catalog onboarding — finalVerdict: APPROVE — clean, additive catalog-only onboarding; both "trap" flags (
Checked and clean
Validation note: blast radius assessed from the diff + remote source inspection (code-graph tools unavailable in this sandbox). Change is additive and touches only generic catalog/enum machinery, so no breakage is expected. Note on summaries: this is the single canonical summary for the PR (issue comment). Earlier identical summaries embedded in two review bodies on older commits (cbee555, 364c6a6) are superseded by this one. |
af71220 to
2abc939
Compare
`loader.ts` derives a provider's fallback positionally —
`fallbackModelName ?? fallbacks[1] ?? fallbacks[0]` — and nothing ever
checked the result. When that derivation lands on the provider's own
default, the retry path is silently dead: the primary fails, the SDK
retries the identical model, and it fails again for the same reason.
Auditing all 16 shipped catalogs found three in that state, so this is
not a hypothetical:
| catalog | models | fallback == default | avoidable |
| -------------- | ------ | ------------------- | --------- |
| gmicloud | 1 | yes | no |
| inception-labs | 1 | yes | no |
| mistral | 32 | yes | YES |
`mistral.json` ships 32 models and explicitly pinned `fallbackModelName`
to its own default, overriding a perfectly serviceable `fallbacks[1]`
(`mistral-large-latest`). Removing that one line restores a real
fallback. The two single-model catalogs have no alternative to name and
are exempt by construction — the check only fires when the catalog
declares more than one model, so it reports a mistake rather than an
unavoidable fact.
The rule lives in the schema's existing `superRefine`, beside the checks
that already constrain `default` and `fallbacks` to catalog keys, so it
runs wherever catalogs are parsed. That is both required CI jobs:
`test` via `codegen:catalog -- --check` (ci.yml:221) and
`provider-safety-net` via `verify:provider-onboarding` (ci.yml:440),
which imports `parseProviderCatalogJson` directly.
Before, on the unmodified tree:
Error: Invalid provider catalog file mistral.json:
models.fallbackModelName: the fallback model resolves to
"mistral-small-2506", which is also models.default — a fallback
identical to the primary cannot fall back.
exit 1
After:
codegen:catalog complete — 15 providers
verify:provider-onboarding — 8/8 new providers fully onboarded
test:openai-compat-catalog — 41 passed · 0 failed
exit 0
No generated artifacts change: the fallback is resolved at load time,
not emitted into a generated file.
Found while investigating a review thread on #1648 which suspected
Morph's fallback. Morph turned out to be fine — its fallback differs
from its default — and the real defect was in a provider nobody was
looking at, reached by checking the derivation itself instead.
2abc939 to
c9fb8c9
Compare
63a8a3c to
d3d6557
Compare
Morph's payment wall cleared. The recorded wire contract gives it plain chat, SSE streaming and genuinely strong structured output (json_schema came back as valid schema-conformant JSON), so it earns a catalog entry. Tool calling is declared false on purpose. Morph answers 200 when sent tools but never returns a tool_calls array: morph-v3-large emits the call as literal text in message.content and morph-v3-fast ignores tools and returns Python. Verified across both models at two token budgets. An earlier campaign note claiming Morph tools worked cited models that are no longer in its roster. Message content must be a plain string: array parts return HTTP 500 "text.charCodeAt is not a function". That is expressed the same way cloudflare.json expresses the identical quirk, and a live call proves our client now avoids it. Only the two models the vendor's own error text lists as accepted by chat/completions are catalogued. MORPH_API_KEY is wired into the nightly live matrix.
c9fb8c9 to
6c2595e
Compare
|
Follow-up checked and closed out — nothing to carry to #1728. The remaining item was Tara-ag's doc-consistency note, raised twice (once in the first review, once in the "Recurring review" table as New (doc nit, non-blocking)) and then dropped from the final approval summary: the PR body says all four tool-calling probes ended Checked both at the pinned head. They do contradict — but Recording it here so the discrepancy is on file rather than silently dropped. #1728 carries only the three items that describe durable code. |
Two providers in one commit because #1648 (Morph) was merged into this branch rather than into release, which left feat/novita-catalog carrying both and failing the single-commit policy. The content is unchanged from the two reviewed commits; only their messages are joined. Novita Novita's payment wall cleared, so its wire contract could finally be recorded end to end: 156 models, chat, SSE streaming with a usage chunk, real structured tool calls, json_object and json_schema, and the 401/404 shapes the error rules need. The entry is written from that recording. The declared default is zai-org/glm-5.3-flash rather than a plain instruct model. meta-llama/llama-3.3-70b-instruct returns HTTP 400 for BOTH response_format json_schema ("Supported formats: json_object") and json_object, and the catalog schema can only express structuredOutput per provider, so making it the default would have declared a capability the default caller could never use. It stays as a fallback for plain chat. Vision is not claimed: the vision probe returns 200 with an empty reply. Live proof against the built package: generate and streaming on the default model, a schema call returning parsed structuredData, and tool calling verified at the wire on both models (finish_reason=tool_calls). NOVITA_API_KEY is wired into the nightly live matrix so the entry keeps being checked. Morph Morph's payment wall cleared. The recorded wire contract gives it plain chat, SSE streaming and genuinely strong structured output (json_schema came back as valid schema-conformant JSON), so it earns a catalog entry. Tool calling is declared false on purpose. Morph answers 200 when sent tools but never returns a tool_calls array: morph-v3-large emits the call as literal text in message.content and morph-v3-fast ignores tools and returns Python. Verified across both models at two token budgets. An earlier campaign note claiming Morph tools worked cited models that are no longer in its roster. Message content must be a plain string: array parts return HTTP 500 "text.charCodeAt is not a function". That is expressed the same way cloudflare.json expresses the identical quirk, and a live call proves our client now avoids it. Only the two models the vendor's own error text lists as accepted by chat/completions are catalogued. MORPH_API_KEY is wired into the nightly live matrix.
`loader.ts` derives a provider's fallback positionally —
`fallbackModelName ?? fallbacks[1] ?? fallbacks[0]` — and nothing ever
checked the result. When that derivation lands on the provider's own
default, the retry path is silently dead: the primary fails, the SDK
retries the identical model, and it fails again for the same reason.
Auditing all 16 shipped catalogs found three in that state, so this is
not a hypothetical:
| catalog | models | fallback == default | avoidable |
| -------------- | ------ | ------------------- | --------- |
| gmicloud | 1 | yes | no |
| inception-labs | 1 | yes | no |
| mistral | 32 | yes | YES |
`mistral.json` ships 32 models and explicitly pinned `fallbackModelName`
to its own default, overriding a perfectly serviceable `fallbacks[1]`
(`mistral-large-latest`). Removing that one line restores a real
fallback. The two single-model catalogs have no alternative to name and
are exempt by construction — the check only fires when the catalog
declares more than one model, so it reports a mistake rather than an
unavoidable fact.
The rule lives in the schema's existing `superRefine`, beside the checks
that already constrain `default` and `fallbacks` to catalog keys, so it
runs wherever catalogs are parsed. That is both required CI jobs:
`test` via `codegen:catalog -- --check` (ci.yml:221) and
`provider-safety-net` via `verify:provider-onboarding` (ci.yml:440),
which imports `parseProviderCatalogJson` directly.
Before, on the unmodified tree:
Error: Invalid provider catalog file mistral.json:
models.fallbackModelName: the fallback model resolves to
"mistral-small-2506", which is also models.default — a fallback
identical to the primary cannot fall back.
exit 1
After:
codegen:catalog complete — 15 providers
verify:provider-onboarding — 8/8 new providers fully onboarded
test:openai-compat-catalog — 41 passed · 0 failed
exit 0
No generated artifacts change: the fallback is resolved at load time,
not emitted into a generated file.
Found while investigating a review thread on #1648 which suspected
Morph's fallback. Morph turned out to be fine — its fallback differs
from its default — and the real defect was in a provider nobody was
looking at, reached by checking the derivation itself instead.
…e catalog docs Three catalog providers merged without any user-facing documentation: FriendliAI (#1657), Morph (#1648) and Novita (#1647) each have a catalog JSON file and an AIProviderName member, but no page under docs/getting-started/providers/ and no entry in the README or provider index. Each now has a setup page built only from its catalog entry (base URL, key env var, models, capabilities), wired into the provider index, the README list and the docs-site sidebar. The migration ledger added in #1764 still listed mistral, deepseek and huggingface as pending, although #1781 moved all three to the JSON catalog and removed them from HAND_DESCRIPTORS. They now sit under a Migrated section with the schema fields each one needed. The tier-2 authoring guide described quirks as "two escape hatches" while the schema has five, including replayReasoningContent from #1800. Every quirk is now documented with what it does and when to use it. Provider counts disagreed across the docs: most pages said 40, the README said "the other 23" above a longer list, and the enum has 44 members (excluding auto). Counts now match the code: 44 providers, 27 in the README's secondary list, and 31 native / 3 model-dependent / 10 no-tool support, which sums to 44.
…e catalog docs Three catalog providers merged without any user-facing documentation: FriendliAI (#1657), Morph (#1648) and Novita (#1647) each have a catalog JSON file and an AIProviderName member, but no page under docs/getting-started/providers/ and no entry in the README or provider index. Each now has a setup page built only from its catalog entry (base URL, key env var, models, capabilities), wired into the provider index, the README list and the docs-site sidebar. The migration ledger added in #1764 still listed mistral, deepseek and huggingface as pending, although #1781 moved all three to the JSON catalog and removed them from HAND_DESCRIPTORS. They now sit under a Migrated section with the schema fields each one needed. The tier-2 authoring guide described quirks as "two escape hatches" while the schema has five, including replayReasoningContent from #1800. Every quirk is now documented with what it does and when to use it. Provider counts disagreed across the docs: most pages said 40, the README said "the other 23" above a longer list, and the enum has 44 members (excluding auto). Counts now match the code: 44 providers, 27 in the README's secondary list, and 31 native / 3 model-dependent / 10 no-tool support, which sums to 44.
…e catalog docs Three catalog providers merged without any user-facing documentation: FriendliAI (#1657), Morph (#1648) and Novita (#1647) each have a catalog JSON file and an AIProviderName member, but no page under docs/getting-started/providers/ and no entry in the README or provider index. Each now has a setup page built only from its catalog entry (base URL, key env var, models, capabilities), wired into the provider index, the README list and the docs-site sidebar. The migration ledger added in #1764 still listed mistral, deepseek and huggingface as pending, although #1781 moved all three to the JSON catalog and removed them from HAND_DESCRIPTORS. They now sit under a Migrated section with the schema fields each one needed. The tier-2 authoring guide described quirks as "two escape hatches" while the schema has five, including replayReasoningContent from #1800. Every quirk is now documented with what it does and when to use it. Provider counts disagreed across the docs: most pages said 40, the README said "the other 23" above a longer list, and the enum has 44 members (excluding auto). Counts now match the code: 44 providers, 27 in the README's secondary list, and 31 native / 3 model-dependent / 10 no-tool support, which sums to 44.
Summary
Morph's payment wall cleared. Its recorded wire contract gives it plain chat, SSE streaming and genuinely strong structured output, so it earns a catalog entry — with tool calling deliberately declared false.
Tool calling is false on purpose
Morph answers 200 when sent
tools, which is exactly the trap that makes an optimistic flag easy to write. It never returns atool_callsarray:Verified across both models at two token budgets, all four runs
finish_reason=stopwith notool_calls. An OpenAI-compatible client would surface that as assistant prose. An earlier campaign note claimed Morph tools worked; it cited models that are no longer in the 22-id roster, so it was stale.Message content must be a plain string
Array content parts return HTTP 500
Request processing failed: text.charCodeAt is not a function— the same class of quirkcloudflare.jsonalready handles, and expressed the same way. The vision probe fails identically, so vision is not claimed.A live call proves the fix: it succeeds with no 500, which is only possible if our client sent string content.
Capability flags and their evidence
tools/toolsWithStreamingtool_calls, on both models, at two budgets.structuredOutputjson_schemareturned valid schema-conformant JSON;json_objectaccepted.structuredOutputWithToolsOnly the two models the vendor's own error text lists as accepted by
chat/completionsare catalogued. The other 18 roster ids were never live-verified against chat, so they are omitted rather than invented.Live proof against the built package
Generate returns
ready, streaming yields1 2 3, a schema call returns{"color":"blue"}withstructuredDataparsed, and the declared defaultmorph-v3-largeis confirmed to be the model that actually resolves.MORPH_API_KEYis wired into the nightly live matrix. That was missing from the first cut and would have silently excluded Morph from nightly verification.Gates
check,lint(0 errors),build, catalog codegen drift check (16 providers),test:providers-mocked98/98,test:provider-structure3/3,test:openai-compat-catalog43/43 including the new Morph alias cases,test:error-classifier-contract44/44,verify:provider-onboarding9/9,docs:apiregenerated. Break-one ritual on the new case: inverted reports a failure with a non-zero exit, restored passes.The catalog alias suite is hand-maintained rather than derived from the catalog, so Morph was silently absent from it at first; the cases were added and the suite re-run.