Skip to content

Navigation Menu

Appearance settings

View all features
- BY COMPANY SIZE
  Enterprises
  Small and medium teams
  Startups
  Nonprofits
- BY USE CASE
  App Modernization
  DevSecOps
  DevOps
  CI/CD
  View all use cases
- BY INDUSTRY
  Healthcare
  Financial services
  Manufacturing
  Government
  View all industries
View all solutions
- EXPLORE BY TOPIC
  AI
  Software Development
  DevOps
  Security
  View all topics
- EXPLORE BY TYPE
  Customer stories
  Events & webinars
  Ebooks & reports
  Business insights
  GitHub Skills
- SUPPORT & SERVICES
  Documentation
  Customer support
  Community forum
  Trust center
  Partners
- COMMUNITY
  GitHub SponsorsFund open source developers
- PROGRAMS
  Security Lab
  Maintainer Community
  Accelerator
  Archive Program
- REPOSITORIES
  Topics
  Trending
  Collections
- ENTERPRISE SOLUTIONS
  Enterprise platformAI-powered developer platform
- AVAILABLE ADD-ONS
  GitHub Advanced SecurityEnterprise-grade security features
  Copilot for BusinessEnterprise-grade AI features
  Premium SupportEnterprise-grade 24/7 support
Pricing

Search code, repositories, users, issues, pull requests...

Search

Clear

Search syntax tips

Provide feedback

We read every piece of feedback, and take your input very seriously.

Include my email address so I can be contacted

Saved searches

Use saved searches to filter your results more quickly

Name

Query

To see all available qualifiers, see our documentation.

Appearance settings

You signed in with another tab or window. Reload to refresh your session. You signed out in another tab or window. Reload to refresh your session. You switched accounts on another tab or window. Reload to refresh your session.

Dismiss alert

webrecorder / browsertrix Public

Notifications You must be signed in to change notification settings
Fork 62
Star 376

Code
Issues 238
Pull requests 16
Actions
Projects 1
Security
Insights

Additional navigation options

Code
Issues
Pull requests
Actions
Projects
Security
Insights

Rework crawl page migration#2412

Merged

ikreymer merged 70 commits intomainwebrecorder/browsertrix:mainfrom

issue-2406-crawl-migration-reworkwebrecorder/browsertrix:issue-2406-crawl-migration-reworkCopy head branch name to clipboard

Feb 20, 2025

Conversation Commits70 (70)Checks Files changed

Merged

Rework crawl page migration#2412
ikreymer merged 70 commits intomainfrom
issue-2406-crawl-migration-rework

Commits

Commits on Feb 18, 2025

migration fixes:

ikreymer
authored and
tw4l
committed
bump version to 1.14.0-beta.2

ikreymer
authored and
tw4l
committed
bump to 1.14.0-beta.3

ikreymer
authored and
tw4l
committed
typo fix

ikreymer
authored and
tw4l
committed
empty filenames migration: use existing bulk re_add method for faster migration

ikreymer
authored and
tw4l
committed
bump to 1.14.0-beta.4

ikreymer
authored and
tw4l
committed
Add version to BaseCrawl, set to 2 for new crawls/uploads
tw4l
committed
Use optimize pages background job in 0042
tw4l
committed
Hacky workaroud: Pass default org to retry_background_job
tw4l
committed
Make sure job failure emails are sent even if no oid
tw4l
committed
Fix typing error
tw4l
committed
Only include replay optimization fields if all pages are optimized
tw4l
committed
Set isMigrating at same time as pulling next crawl
tw4l
committed
Redo background_jobs typing, add superuser non-org list endpoint
tw4l
committed
Remove crawl_type from optimize_pages bg job
tw4l
committed
Handle running crawls in page optimization
tw4l
committed
Remove isMigrating filter from running crawl check
tw4l
committed
Fix bg job retry
tw4l
committed
Cast job to right type in retry
tw4l
committed
Add API endpoint to launch migrate crawls job
tw4l
committed
Use SuccessResponseId model to include job id in response
tw4l
committed
Add first draft of upgrade notes in docs
tw4l
committed
cleanup:
ikreymer
committed
add logging to optimize pages job
ikreymer
committed
logging
ikreymer
committed

Commits on Feb 19, 2025

check for empty result
ikreymer
committed
version: bump to 1.14.0-beta.5
ikreymer
committed
fix response model
ikreymer
committed
add retries for add_crawl_pages_to_db_from_wacz()
ikreymer
committed
back to single background_job.yaml, just pass scale as param
ikreymer
committed
fix parallelism setting
ikreymer
committed
retry at the wacz level, not crawl
ikreymer
committed
bg jobs: make single crawl readd a background job as well, if crawl_id is provided,
ikreymer
committed
add sort
ikreymer
committed
better call
ikreymer
committed
logging
ikreymer
committed
ensure pages are streamed from each wacz on demand
ikreymer
committed
ignore dupes on insertMany
ikreymer
committed
attempt dedup within wacz
ikreymer
committed
ignore dupe errors?
ikreymer
committed
update logging, add crawl_id + url index
ikreymer
committed
lint fix
ikreymer
committed
optimize pages: don't use $facet, don't compute actual total
ikreymer
committed
add index for default sort
ikreymer
committed
remove uniquge page counts from org metrics
ikreymer
committed
update model
ikreymer
committed
remove resources from collection list
ikreymer
committed
optimize coll /replay.json
ikreymer
committed
optimize page snapshot query:
ikreymer
committed
crawl /replay.json optimization:
ikreymer
committed
cleanup
ikreymer
committed
fix typo
ikreymer
committed
lint
ikreymer
committed
Delay rendering components doing data fetching in collection settings dialog until open (#2414)
emma-sg
authored
optimize page snapshots, add non-group alternative
ikreymer
committed

Commits on Feb 20, 2025

fix page_count update?
ikreymer
committed
tweak page snapshot sort order
ikreymer
committed
use gte
ikreymer
committed
readd unquote
ikreymer
committed
lint fix
ikreymer
committed
tweak
ikreymer
committed
more cleanup:
ikreymer
committed
test fix
ikreymer
committed
test fix
ikreymer
committed
Update backend/btrixcloud/crawlmanager.py

ikreymer
and
tw4l
authored
Pages Query: use $text search or prefix search (#2415)

ikreymer
and
tw4l
authored
Fix typo in response
tw4l
committed
Add test asserts for crawl/upload version field
tw4l
committed
remove dupe index
ikreymer
committed
set pages migration job to run at scale 1 by default
ikreymer
committed

Footer

© 2026 GitHub, Inc.

Footer navigation

Terms
Privacy
Security
Status
Community
Docs
Contact

You can’t perform that action at this time.