Skip to content

[Feature]: Deduplicate crawls #2860

New issue
New issue
Open
Feature
0 of 8 issues completed
Open
[Feature]: Deduplicate crawls#2860
Feature
0 of 8 issues completed
Assignees
ikreymer
Labels
back endRequires back end dev workfeature designThis issue tracks smaller sub issues that compose a featurefront endRequires front end dev workui/uxThis issue requires UI/UX work
Milestone
Crawl Deduplication
@SuaYoo

Description

@SuaYoo
SuaYoo
opened on Sep 29, 2025

Description

Repeated crawling of certain sites can often yield duplicate data, which adds significantly to the storage needed to store web archives. Deduplication reduces storage by not storing duplicate content.

Requirements

See subtasks

Context

Related:

Sub-issues

Metadata

Metadata

Assignees

Labels

back endRequires back end dev workfeature designThis issue tracks smaller sub issues that compose a featurefront endRequires front end dev workui/uxThis issue requires UI/UX work

Type

Feature

Projects

Webrecorder Projects

Status

Todo

Milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions