v7.0.0 Scrape service with response

New services!

This major release contains 2 brand new services that should make figuring out your configuration and css selectors much easier!
It makes use of the new functionality in Home Assistant that services can now provide a response. To make this possible, significant refactoring was required.

multiscrape.get_content
This service retrieves the content of the website you want to scrape. It shows the same data for which you had to enable log_response and open the page_soup.txt file.

multiscrape.scrape
This does what it says. It scrapes based on a configuration you can provide in the service data. It is ideal for quickly trying out multiple css selectors, or to scrape data in an automation that you only need when running that automation.

A nice detail is that both services accept exactly the same configuration as you provide in your configuration yaml. Even the form_submit features are supported! However, there is a small but important caveat. Read more about it in the readme.

Changes

• Update readme and version for release with new services @danieldotnl (#343)
• Add service icons for hassfest validation @danieldotnl (#342)
• Update actions/setup-python action to v5.1.0 @renovate (#341)
• Update dependency homeassistant to v2024.3.3 @renovate (#339)
• Update dependency ruff to v0.3.4 @renovate (#338)
• Update dependency homeassistant to v2024.3.1 - autoclosed @renovate (#321)
• Update dependency ruff to v0.3.3 @renovate (#322)
• Implement service for scraping @danieldotnl (#335)
• Update dependency pip to v24 @renovate (#326)
• Update dependency colorlog to v6.8.2 @renovate (#323)
• Update release-drafter/release-drafter action to v6 @renovate (#324)
• Update dependency homeassistant to v2024.1.3 @renovate (#318)
• Update dependency ruff to v0.1.13 @renovate (#317)
• Vscode task to upgrade dependencies @danieldotnl (#316)
• Update dependency homeassistant to v2024.1.2 @renovate (#314)

Fix issue with headers when using form_submit

Changes

• Fix issue in creating renderers for dictionaries with templates @danieldotnl (#313)

Support for templates in request headers

Changes

• Support templates in header values @danieldotnl (#309)

• Update dependency ruff to v0.1.10 @renovate (#307)

• Update dependency homeassistant to v2024 @renovate (#310)

• Update dependency ruff to v0.1.11 @renovate (#308)

🌈v6.7.6: Upgrade lxml (required for HA 2024.1)

Changes

• Update manifest.json to prevent conflicts with HA (fix for 2024.1) @hmmbob (#306)
• Remove dependabot as we are using renovate instead @danieldotnl (#305)
• Update dependency homeassistant to v2023.12.4 @renovate (#300)
• Upgrade lxml and beautifulsoup versions to latest @danieldotnl (#304)
• Update dependency ruff to v0.1.9 @renovate (#299)

Parse template results into their own data type

Changes

• Configure Renovate @renovate (#295)
• Also parse value_templates on scraped data @danieldotnl (#296)
• Bump ruff from 0.1.6 to 0.1.8 @dependabot (#293)
• Bump colorlog from 6.7.0 to 6.8.0 @dependabot (#286)
• Bump actions/setup-python from 4.7.1 to 5.0.0 @dependabot (#289)
• Bump github/codeql-action from 2 to 3 @dependabot (#294)
• Bump homeassistant from 2023.11.2 to 2023.12.3 @dependabot (#292)
• Bump ruff from 0.1.4 to 0.1.6 @dependabot (#282)
• Bump homeassistant from 2023.10.5 to 2023.11.2 @dependabot (#280)
• Bump ruff from 0.1.1 to 0.1.4 @dependabot (#277)
• Update pip requirement from <23.3,>=21.0 to >=21.0,<23.4 @dependabot (#268)
• Bump homeassistant from 2023.9.0 to 2023.10.5 @dependabot (#273)
• Bump actions/checkout from 4.0.0 to 4.1.1 @dependabot (#270)
• Bump ruff from 0.0.287 to 0.1.1 @dependabot (#272)
• Bump actions/setup-python from 4.7.0 to 4.7.1 @dependabot (#262)
• Validate on master branch @danieldotnl (#247)
• Bump homeassistant from 2023.8.4 to 2023.9.0 @dependabot (#244)
• Bump actions/checkout from 3.5.2 to 4.0.0 @dependabot (#240)
• Update pip requirement from <23.2,>=21.0 to >=21.0,<23.3 @dependabot (#242)
• Bump ruff from 0.0.286 to 0.0.287 @dependabot (#243)
• Bump actions/setup-python from 4.6.1 to 4.7.0 @dependabot (#241)
• Add release drafter config @danieldotnl (#245)

Support for dictionaries and other types in attributes

I am very motivated by coffee!! If you are enjoying this multiscrape custom component, please consider to:

Changes

• Attributes set by a value_template are now parsed into their type, e.g. dictionary

📦 Dependencies

• Bump actions/checkout from 3 to 4 (#237) @dependabot
• Bump crazy-max/ghaction-github-labeler from 4.1.0 to 4.2.0 (#232) @dependabot
• Bump flake8 from 6.0.0 to 6.1.0 in /.github/workflows (#229) @dependabot
• Bump black from 23.3.0 to 23.7.0 in /.github/workflows (#221) @dependabot
• Bump pip from 23.1.2 to 23.2.1 in /.github/workflows (#228) @dependabot

Fix icon in binary_sensor

If you are enjoying this multiscrape custom component, please consider to:

Changes

• Fix issue with setting icon in binary_sensor (#227) @danieldotnl

Fix multiscrape does not start in 2023.7.1

Changes

• Integrate fix registering service schema with wrong name (#217) @danieldotnl (Fix #215)

📦 Dependencies

• Bump release-drafter/release-drafter from 5.23.0 to 5.24.0 (#212) @dependabot
• Bump reorder-python-imports from 3.9.0 to 3.10.0 in /.github/workflows (#211) @dependabot
• Bump pre-commit from 3.3.1 to 3.3.3 in /.github/workflows (#210) @dependabot

Support for scan_interval = 0 and icons after errors

Changes

• It is now possible to set scan_interval to 0. This will only update the sensors on startup or when triggered by the trigger service. If it fails on startup it will retry 3 times with 30 seconds in between. Fix #188
• Fix conversion errors when trying data could not be retrieved and HA was trying to convert the unavailable state to a float or datetime
• Add more detailed logging on request timeouts
• Icons are now also set in case of scraping errors. This is especially useful when you have defined a default value in on_error Fix #190

📦 Dependencies

• Bump pre-commit from 3.2.2 to 3.3.1 in /.github/workflows (#203) @dependabot
• Bump pip from 23.1.1 to 23.1.2 in /.github/workflows (#201) @dependabot
• Bump pip from 23.1 to 23.1.1 in /.github/workflows (#200) @dependabot
• Bump pip from 23.0.1 to 23.1 in /.github/workflows (#199) @dependabot
• Bump pre-commit from 3.1.1 to 3.2.2 in /.github/workflows (#198) @dependabot
• Bump black from 23.1.0 to 23.3.0 in /.github/workflows (#197) @dependabot
• Bump black from 22.12.0 to 23.1.0 in /.github/workflows (#184) @dependabot
• Bump pre-commit from 3.0.4 to 3.1.1 in /.github/workflows (#194) @dependabot
• Bump pip from 23.0 to 23.0.1 in /.github/workflows (#191) @dependabot
• Bump release-drafter/release-drafter from 5.22.0 to 5.23.0 (#192) @dependabot
• Bump pre-commit from 3.0.2 to 3.0.4 in /.github/workflows (#187) @dependabot
• Bump pip from 22.3.1 to 23.0 in /.github/workflows (#183) @dependabot
• Bump pre-commit from 3.0.1 to 3.0.2 in /.github/workflows (#182) @dependabot
• Bump pre-commit from 3.0.0 to 3.0.1 in /.github/workflows (#180) @dependabot
• Bump pre-commit from 2.21.0 to 3.0.0 in /.github/workflows (#179) @dependabot

Timestamp sensors

If you are enjoying this multiscrape custom component, please consider to:

Changes

• Support for scraping sensors with timestamp device_class - fix #174

📦 Dependencies

• Bump release-drafter/release-drafter from 5.21.1 to 5.22.0 (#173) @dependabot
• Bump pre-commit from 2.20.0 to 2.21.0 in /.github/workflows (#170) @dependabot
• Bump black from 22.10.0 to 22.12.0 in /.github/workflows (#169) @dependabot