Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix: okay-browser-daily #2585

Merged
merged 2 commits into from
Nov 2, 2024
Merged

Conversation

valekjo
Copy link
Contributor

@valekjo valekjo commented Oct 30, 2024

This PR updates okay-browser-daily Actor, so it can use product sitemaps instead of collections, and improves scraping on the detail page.

It works, but is super slow, so there is also alternative and faster solution that

  • uses product sitemaps to get product detail links
  • emulates what would happen with dom when the page was loaded
  • scrapes data from that

The emulation uses data from one additional endpoint, pre-fetched shopify config, and script copied form okay.

This is added as okay-daily-v2, needed to update yarn.lock for that.

See comments in PR for more details.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is a script that's just copied from okay webpage, without changes.

country = Country.CZ,
customTableName = null,
type = ActorType.Full,
forceApifyProxy = false,
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is new field on input.

@valekjo valekjo marked this pull request as ready for review November 1, 2024 13:27
@valekjo valekjo marked this pull request as draft November 1, 2024 13:27
@valekjo valekjo marked this pull request as ready for review November 1, 2024 13:34
@MartinaGelnerova
Copy link
Collaborator

@rarous pls merge 🙏

@rarous
Copy link
Collaborator

rarous commented Nov 1, 2024

@valekjo Can you, please, make just one actor, that is the best solution and delete all others? I really don't like growing junk :)

@valekjo
Copy link
Contributor Author

valekjo commented Nov 1, 2024

@rarous Ok, kept just one, named okay.

@rarous
Copy link
Collaborator

rarous commented Nov 2, 2024

@valekjo can you please try to rebase on trunk and try to fix yarn.lock conflicts? Otherwise, I'm unable to merge your changes.

Verified

This commit was created on GitHub.com and signed with GitHub’s verified signature. The key has expired.
@valekjo valekjo force-pushed the fix/2550-okay-daily-prices branch from ff71fd6 to c9d6c40 Compare November 2, 2024 08:11
@valekjo
Copy link
Contributor Author

valekjo commented Nov 2, 2024

@rarous Should work now.

@rarous rarous merged commit dce747d into topmonks:trunk Nov 2, 2024
@rarous
Copy link
Collaborator

rarous commented Nov 2, 2024

Thank you ❤️

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants