feat(providers): add NEMWeb DISPATCH fallback wholesale-price source (Phase 7 PR-3)#87
Merged
Conversation
…PR-2) Standalone async client for OpenElectricity v4 — not yet wired into the coordinator or config flow (that's PR-2 part 2 / Plan 07-02b). Foundation for Phase 8 Silver-compliance handlers and Phase 9 external-statistics integration. * ``custom_components/pricehawk/providers/openelectricity.py`` — new module exposing ``OpenElectricityPriceSource`` async client + ``WholesalePrice`` frozen dataclass. CC BY-NC 4.0 attribution baked into every result per research §1.3. * ``manifest.json:requirements`` pinned to ``openelectricity>=0.10.1,<0.11`` — minor-bounded per the SDK README's "currently under active development" notice. Audit invariants applied (07-02-AUDIT.md): * M1 — API key never appears in ``__repr__`` output or log messages. ``_scrub()`` filters the full key plus its 8-char prefix from any text passed to ``_LOGGER.warning``. ``test_log_scrubs_api_key_from_sdk_error`` uses ``caplog`` to verify against a leaky SDK exception. * M2 — ``asyncio.timeout(_FETCH_TIMEOUT_SECONDS=30.0)`` bounds every SDK call. Hung connection → ``None`` + WARNING, never an HA-wide stall. * M3 — Missing ``openelectricity`` install raises ``ConfigEntryNotReady`` (HA retries setup), not ``ImportError`` (permanent crash). Lazy import inside the async method. * S1 — Auth-error detection is belt-and-braces: message-string match plus exception-class-name probe. Defensive against the SDK's active-development error-message format drift. * S2 — HTTP 429 rate-limit has a distinct branch: ``None`` + WARNING saying "rate-limited", last-good cache preserved, no spurious ``ConfigEntryAuthFailed``. * S5 — Module is intentionally NOT exported from ``providers/__init__.py``. ``OpenElectricityPriceSource`` is a wholesale-price source, not a ``Provider`` Protocol implementation. Consumers import directly. Implementation diverged from plan pseudo-code: real ``openelectricity==0.10.1`` ``TimeSeriesResponse`` is nested three levels deep (``data`` → ``results`` → ``data``), not flat. ``_extract_latest_for_region`` walks the nested structure and returns tz-aware UTC. We skip ``response.to_records()`` because it emits naive-local timestamps. 12 contract tests (``tests/test_openelectricity_provider.py``) cover AC-1 through AC-7e: happy-path NEM, happy-path WEM, attribution-verbatim, 401 mapping, generic-error cache preservation, empty-data, invalid region, empty API key, timeout, missing SDK, 429 rate limit, repr redaction, log scrub. Test suite: 815 → 827 passed. ``tests/conftest.py`` gained one stub: ``ConfigEntryAuthFailed`` (was only ``ConfigEntryNotReady`` before). Necessary because the new module is the first user of the auth-failed exception in this project. Decisions D-P7-5 through D-P7-8 logged to DECISIONS.md. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…(Phase 7 PR-3) Anonymous public-AEMO wholesale-price source at ``custom_components/pricehawk/providers/nemweb.py``. Same ``WholesalePrice`` contract as PR-2's OpenElectricity client — same return type, same ``asyncio.timeout`` discipline, same last-good cache semantics — but with no API key (no scrubber needed, no ``ConfigEntryAuthFailed`` path). NEM-only. WEM requests raise ``ValueError`` with a message pointing the user at OpenElectricity for WA coverage. Wraps the existing ``aemo_api.fetch_current_rrp`` rather than rewriting it — Flow Power still depends on ``aemo_api`` and the wrapper keeps PR scope tight. Future cleanup PR can migrate ``aemo_api`` to honour AEMO's 2026-04-07 HTTP decommissioning notice (HTTPS-only + case-sensitive URLs). The load-bearing decision: settlement-date parsing is anchored to ``Australia/Brisbane`` (no DST), NOT ``Australia/Sydney`` (DST applied). AEMO publishes NEM dispatch in AEST year-round per their docs. A Sydney-anchored parser would silently 1-hour-shift every dispatch row from October through April. ``test_settlement_date_parsing_summer_no_dst`` is the regression guard — asserts a January 02:30 NEM-time row resolves to 16:30 UTC (offset −10:00), NOT 15:30 UTC. 12 contract tests in tests/test_nemweb_provider.py cover: attribution verbatim, shared contract surface (``nemweb.WholesalePrice is openelectricity.WholesalePrice``), happy path, WEM rejection, cache preservation across aemo_api None returns, invalid-region rejection, winter + summer settlement-date parsing, empty-string and quoted-string parsing edge cases, timeout, malformed settlement date. Test suite: 827 → 839 passed (+12 new). NOT wired into the coordinator or config flow in this PR — that's Plan 07-02b. ``providers/__init__.py`` is intentionally left unchanged (same boundary as PR-2 S5: wholesale-price sources are not ``Provider`` Protocol implementations and don't belong in the package init). Stacked on ``feat/openelectricity-client`` (PR #86). When PR-2 merges to dev, GitHub will auto-retarget this PR's base to dev. Decision D-P7-9 logged. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This was referenced May 21, 2026
Artic0din
added a commit
that referenced
this pull request
May 22, 2026
Narrow scope: only edits dual-loop-review.yml (the Claude AI review workflow). PRs targeting dev now trigger Claude review, which is the core need for the stacked v3.0 PR series. The other 5 workflows (python-ci, pr-checks, lint, docs-check, security-scan) also have the same `[main, develop]` gate but expose pre-existing failures when extended to dev: - python-ci.yml mypy: 27 baseline type errors in tests - pr-checks.yml: CodeRabbit gate still checks for CR approval despite CLAUDE.md saying CR is not used - security-scan.yml gitleaks: flags string-literal test secrets in tests/test_reauth.py + others; needs allowlist for test fixtures - dual-loop-review run for THIS PR hit an upstream tsconfig directory-mismatch bug in the anthropics/claude-code-action@v1 action; transient Each is a separate cleanup PR. Landing this narrow change unblocks the Claude AI review path for the 17 stacked PRs without sweeping unrelated debt under the rug. After merge: the 17 stacked PRs (#87..#102) trigger dual-loop-review on next synchronize (push, rebase, or empty commit). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Artic0din
added a commit
that referenced
this pull request
May 22, 2026
Narrow scope: only edits dual-loop-review.yml (the Claude AI review workflow). PRs targeting dev now trigger Claude review, which is the core need for the stacked v3.0 PR series. The other 5 workflows (python-ci, pr-checks, lint, docs-check, security-scan) also have the same `[main, develop]` gate but expose pre-existing failures when extended to dev: - python-ci.yml mypy: 27 baseline type errors in tests - pr-checks.yml: CodeRabbit gate still checks for CR approval despite CLAUDE.md saying CR is not used - security-scan.yml gitleaks: flags string-literal test secrets in tests/test_reauth.py + others; needs allowlist for test fixtures - dual-loop-review run for THIS PR hit an upstream tsconfig directory-mismatch bug in the anthropics/claude-code-action@v1 action; transient Each is a separate cleanup PR. Landing this narrow change unblocks the Claude AI review path for the 17 stacked PRs without sweeping unrelated debt under the rug. After merge: the 17 stacked PRs (#87..#102) trigger dual-loop-review on next synchronize (push, rebase, or empty commit). Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
# Conflicts: # CHANGELOG.md # DECISIONS.md # custom_components/pricehawk/manifest.json # tests/conftest.py
Artic0din
added a commit
that referenced
this pull request
May 23, 2026
… Jinja (#145) Two real bugs surfaced by a Copilot-CLI retro-review of the 22 merged PRs that the prior @claude batch couldn't reach (OIDC workflow-validation against stale main). statistics.py — external_statistic_id now sanitizes via regex: CDR-derived provider_ids carry the plan id verbatim (e.g. ``agl_AGL-CDR-N0001`` for AGL via CDR), so `.lower()` alone left hyphens that the recorder's ``[a-z0-9_]+`` regex silently rejected. Every CDR user's dual-write would fail and the Energy Dashboard would never receive their cost data — same silent-failure class #107 fixed for uppercase ULIDs. Added ``_STATISTIC_ID_OBJECT_SAFE`` compiled regex that coerces ANY character outside ``[a-z0-9_]`` to underscore. New regression test ``test_cdr_plan_id_with_hyphens_is_sanitized`` + adjusted ``test_entry_id_sliced_to_8_chars`` for the post-sanitization shape. Surfaced by retro-review of PRs #93, #95. blueprints — variables block replaces !input inside Jinja: ``daily_7pm_summary.yaml`` and ``wholesale_spike_alert.yaml`` had templates like ``{{ states(!input today_cost_sensor) }}``. ``!input`` is a YAML tag (resolved at YAML parse time) and is invalid inside Jinja ``{{ }}`` — Jinja parses ``!`` as an invalid operator and the template never renders. Replaced with the standard HA pattern: a ``variables:`` block at action level that binds blueprint inputs as Jinja identifiers (``today_cost_entity: !input today_cost_sensor`` etc.), then ``{{ states(today_cost_entity) }}`` in the message. Surfaced by retro-review of PRs #99, #100. 22 Copilot reviews ran (PRs #85, #87-#101, #104, #105, #108-#111). Most findings were false positives — Copilot flagged ``ServiceValidationError`` and ``async_items`` as broken HA APIs (they're both fine), and several findings duplicated bugs already fixed by codex P0/P1 work (token-in-URL #109, DWT reset #109). Findings library + triage notes archived in ``.planning/copilot-retro/``. Full test suite: 1120 passing.
4 tasks
Artic0din
added a commit
that referenced
this pull request
May 24, 2026
#147) Critical hotpatch. The earlier _LEGACY fix (#107) silenced one cause of NEMWeb listing parse failures but not the actual one. Live UAT against the production NEMWeb directory on 2026-05-24 showed the real shape: <A HREF="/Reports/CURRENT/DispatchIS_Reports/PUBLIC_DISPATCHIS_202605221100_..._.zip"> The prior regex was case-insensitive (via re.IGNORECASE) but required PUBLIC_DISPATCHIS_ to sit immediately after the opening quote — so it matched zero files even though both the case and the _LEGACY-optional fixes were correct. AEMO-Direct DWT and Flow Power's AEMO poll were both silently broken in production since the PR #87 / #107 work. Fix: insert a non-greedy ``[^"]*?`` between ``href="`` and the filename capture group, keeping the captured filename clean so the existing lexical-sort logic in _pick_latest_dispatch_file works unchanged. Verified against the live directory: regex now finds 576 matches in the 2026-05-24 snapshot (was 0). Two regression tests pin the new path-prefix tolerance — one using the exact shape pulled from the live directory, one defensive with mixed case + absolute URLs. Manifest bumped to 1.6.0-beta.3 for HACS-beta re-deploy. Silver-checklist test version assertion updated to match. Full test suite: 1130 passing.
Artic0din
added a commit
that referenced
this pull request
May 24, 2026
…ion) (#151) CRITICAL cost-math bug found via live UAT 2026-05-24. User reported today_cost showing ~$66 for ~12 kWh of consumption. Investigation: Real VIC1 dispatch at 15:40 today (PUBLIC_DISPATCHIS_202605241540...): D,DISPATCH,PRICE,5,"2026/05/24 15:40:00",1,VIC1,...,96.16181,... D,DISPATCH,REGIONSUM,9,"2026/05/24 15:40:00",1,VIC1,...,5738.11,... Parser at aemo_api.py:197 was matching ``D,DISPATCH,REGIONSUM`` rows and reading row[9] — which in REGIONSUM is **TOTALDEMAND** (MW), NOT RRP. The actual RRP lives in ``D,DISPATCH,PRICE`` rows where index 9 IS the RRP in $/MWh per the AEMO ``I,DISPATCH,PRICE,5,...`` schema: [0] D [1] DISPATCH [2] PRICE [3] 5 [4] SETTLEMENTDATE [5] RUNNO [6] REGIONID [7] DISPATCHINTERVAL [8] INTERVENTION [9] RRP [10] EEP [11] ROP [12] APCFLAG ... Bug math: 5738.11 MW / 10 → 573.811 reported as "c/kWh" (~60x inflation vs the real RRP of 9.62 c/kWh). DWT provider's update() accumulated cost at the inflated rate → user's 12 kWh became $66 instead of ~$1.15. The synthetic test fixture (build_test_dispatch_zip) also wrote ``D,DISPATCH,REGIONSUM`` rows, so the parser's row-type filter matched and the tests passed. The bug was invisible to unit tests because both sides of the contract were wrong in the same way. Fix: - Parser: filter ``row[2] != "PRICE"`` (was REGIONSUM) - Fixture: emit ``D,DISPATCH,PRICE,5,...`` with the full PRICE schema - 2 new regression tests: - test_does_not_pick_up_regionsum_totaldemand_as_rrp — synthetic CSV with BOTH record types, asserts parser picks RRP not demand - test_real_nemweb_dispatch_csv_shape — mirrors actual NEMWeb file structure (REGIONSUM rows precede PRICE rows; VIC1 RRP 96.16181 → 9.616 c/kWh) Manifest bumped to 1.6.0-beta.7. Silver-checklist version assertion updated to match. Full test suite: 1140 passing. This bug has been latent since Phase 7 PR-3 (#87) shipped NEMWeb. Once beta.3's path-prefix regex fix landed and the parser started seeing real dispatch data, the wrong-row-type bug surfaced as the ~60x inflated wholesale rate that cratered DWT cost accounting.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
custom_components/pricehawk/providers/nemweb.py— anonymous public-AEMO wholesale-price source. SameWholesalePricecontract as PR-2. NO API key.aemo_api.fetch_current_rrp(no rewrite — Flow Power still depends on it).feat/openelectricity-client). When PR-2 merges to dev, GitHub auto-retargets this PR's base.Phase 7 / PR-3. Foundation for Plan 07-02b's "you don't need an OpenElectricity key" config-flow branch.
Load-bearing decision: Brisbane > Sydney for NEM dispatch timestamps
AEMO publishes NEM dispatch timestamps in AEST year-round (no DST applied per their docs).
Australia/Sydneyapplies AEDT (+11:00) during summer DST — would silently 1-hour-shift every dispatch row from October through April.Australia/Brisbane(QLD, no DST) is permanently +10:00 and matches AEMO's publishing convention.test_settlement_date_parsing_summer_no_dstis the regression guard: asserts a January 02:30 NEM-time row resolves to 16:30 UTC (offset −10:00), NOT 15:30 UTC. Documented as D-P7-9 in DECISIONS.md.Scope
custom_components/pricehawk/providers/nemweb.pytests/test_nemweb_provider.pyCHANGELOG.md[Unreleased] > AddedentryDECISIONS.mdOut of scope (deferred)
aemo_api.pyto honour AEMO's 2026-04-07 HTTP decommissioning (HTTPS-only + case-sensitive URLs) — future cleanup PRTest plan
pytest --tb=short -q→ 839 passed (was 827; +12 new)ruff check custom_components/pricehawk/providers/nemweb.py tests/test_nemweb_provider.py→ all checks passedgrep -c 'NEMWeb\|nemweb' providers/__init__.py→ 0 (same S5 boundary as PR-2 — wholesale-price sources are not Provider implementations)grep -c 'Australia/Brisbane' providers/nemweb.py→ 3 (used at runtime)grep -c 'Australia/Sydney' providers/nemweb.py→ 1 (in explanatory comment only, NOT runtime)🤖 Generated with Claude Code