Skip to content

Use cleaner logic for limiting elements to scan #3158

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
dgtlmoon merged 6 commits into from
Apr 30, 2025
+23 −11
Merged

Use cleaner logic for limiting elements to scan #3158

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
b2f0157
Use cleaner logic for limiting elements to scan
dgtlmoon Apr 29, 2025
ed25d79
Update stock-not-in-stock.js
dgtlmoon Apr 29, 2025
fac247d
Merge branch 'master' into restock-price-scan-text-fix
dgtlmoon Apr 29, 2025
36806f9
Bump test
dgtlmoon Apr 30, 2025
62141e5
Merge branch 'restock-price-scan-text-fix' of github.com:dgtlmoon/cha…
dgtlmoon Apr 30, 2025
d0cbc6b
test - also include header offset
dgtlmoon Apr 30, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
27 changes: 19 additions & 8 deletions changedetectionio/content_fetchers/res/stock-not-in-stock.js
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -125,6 +125,20 @@ async () => {
// so it's good to filter to just the 'above the fold' elements
// and it should be atleast 100px from the top to ignore items in the toolbar, sometimes menu items like "Coming soon" exist

function elementIsInEyeBallRange(element) {
// outside the 'fold' or some weird text in the heading area
// .getBoundingClientRect() was causing a crash in chrome 119, can only be run on contentVisibility != hidden
// Note: theres also an automated test that places the 'out of stock' text fairly low down
// Skip text that could be in the header area
if (element.getBoundingClientRect().bottom + window.scrollY <= 300 ) {
return false;
}
// Skip text that could be much further down (like a list of "you may like" products that have 'sold out' in there
if (element.getBoundingClientRect().bottom + window.scrollY >= 1300 ) {
return false;
}
return true;
}

// @todo - if it's SVG or IMG, go into image diff mode

Expand Down Expand Up @@ -161,9 +175,7 @@ async () => {
for (let i = elementsToScan.length - 1; i >= 0; i--) {
const element = elementsToScan[i];

// outside the 'fold' or some weird text in the heading area
// .getBoundingClientRect() was causing a crash in chrome 119, can only be run on contentVisibility != hidden
if (element.getBoundingClientRect().top + window.scrollY >= vh || element.getBoundingClientRect().top + window.scrollY <= 100) {
if (!elementIsInEyeBallRange(element)) {
continue
}

Expand All @@ -177,11 +189,11 @@ async () => {
} catch (e) {
console.warn('stock-not-in-stock.js scraper - handling element for gettext failed', e);
}

if (elementText.length) {
// try which ones could mean its in stock
if (negateOutOfStockRegex.test(elementText) && !elementText.includes('(0 products)')) {
console.log(`Negating/overriding 'Out of Stock' back to "Possibly in stock" found "${elementText}"`)
element.style.border = "2px solid green"; // highlight the element that was detected as in stock
return 'Possibly in stock';
}
}
Expand All @@ -190,10 +202,8 @@ async () => {
// OTHER STUFF THAT COULD BE THAT IT'S OUT OF STOCK
for (let i = elementsToScan.length - 1; i >= 0; i--) {
const element = elementsToScan[i];
// outside the 'fold' or some weird text in the heading area
// .getBoundingClientRect() was causing a crash in chrome 119, can only be run on contentVisibility != hidden
// Note: theres also an automated test that places the 'out of stock' text fairly low down
if (element.getBoundingClientRect().top + window.scrollY >= vh + 250 || element.getBoundingClientRect().top + window.scrollY <= 100) {

if (!elementIsInEyeBallRange(element)) {
continue
}
elementText = "";
Expand All @@ -208,6 +218,7 @@ async () => {
for (const outOfStockText of outOfStockTexts) {
if (elementText.includes(outOfStockText)) {
console.log(`Selected 'Out of Stock' - found text "${outOfStockText}" - "${elementText}" - offset top ${element.getBoundingClientRect().top}, page height is ${vh}`)
element.style.border = "2px solid red"; // highlight the element that was detected as out of stock
return outOfStockText; // item is out of stock
}
}
Expand Down
7 changes: 4 additions & 3 deletions changedetectionio/tests/restock/test_restock.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,8 @@
def set_original_response():
test_return_data = """<html>
<body>
<section id=header style="padding: 50px; height: 350px">This is the header which should be ignored always - <span>add to cart</span></section>
<!-- stock-not-in-stock.js will ignore text in the first 300px, see elementIsInEyeBallRange(), sometimes "add to cart" and other junk is here -->
Some initial text<br>
<p>Which is across multiple lines</p>
<br>
Expand Down Expand Up @@ -52,8 +54,6 @@ def test_restock_detection(client, live_server, measure_memory_usage):

set_original_response()
#assert os.getenv('PLAYWRIGHT_DRIVER_URL'), "Needs PLAYWRIGHT_DRIVER_URL set for this test"

time.sleep(1)
live_server_setup(live_server)
#####################
notification_url = url_for('test_notification_endpoint', _external=True).replace('http://localhost', 'http://changedet').replace('http', 'json')
Expand Down Expand Up @@ -84,7 +84,8 @@ def test_restock_detection(client, live_server, measure_memory_usage):
# Is it correctly show as NOT in stock?
wait_for_all_checks(client)
res = client.get(url_for("watchlist.index"))
assert b'not-in-stock' in res.data
assert b'processor-restock_diff' in res.data # Should have saved in restock mode
assert b'not-in-stock' in res.data # should be out of stock

# Is it correctly shown as in stock
set_back_in_stock_response()
Expand Down