Add scheduled ingestion service - #529

Open

niedch wants to merge 22 commits into

labsai:mainfrom

niedch:scheduled-ingest-service

Commits on Jul 2, 2026

feat(IngestionService): Add scheduled ingestion service
niedch
committed
7c516e3
refactor(records): Use Records for configuration
niedch
committed
fcd57fe
refactor(service): Make ContentFetcher configurable
niedch
committed
2d3ba08
test(markdown): Testing markdown converter
niedch
committed
7e04efc
fix(weird): Cleanup later, how to fetch source configurations for rag config
niedch
committed
0c3efac
fix(some): Some fixes
niedch
committed
fd1fd52
test(unit): ContentHashTracker unit tests
niedch
committed
a9cc715
fix(merge): Fixes merge conflicts
niedch
committed
a429286
feat(remove): Remove need for toc selector
niedch
committed
1150d24
chore(config): Sanitize config
niedch
committed
f4ed894
test(ingestion): Test ingestion pipeline
niedch
committed
51eb29e
chore(fmt): Format
niedch
committed
4f7a3c0
review(pr): Review comments
niedch
committed
1a8556b
fix(postgres): Postgres ContentHashTracker
niedch
committed
0e2ebfe
fix(test): Fix integration tests
niedch
committed
ae5dfe7
review(pr): Review comments
niedch
committed
2d94f15
feat(validation): Handling
niedch
committed
acc6a5d
fix(pr): Some PR review fixes
niedch
committed
7addb02
test: Some more validations & tests
niedch
committed
1bd3927
chore: Log
niedch
committed
717f6a5
fix(pr): address PR review comments — shouldCrawl root URL rejection & cron validation
niedch
committed
07741bf
fix(webcrawler): fix matchesGlob regex corruption and add Javadoc with examples
niedch
committed
6505b71