examples: Make RAG examples a bit more generic and demoable

[antoniivanov]

For confluence-reader:

• the recursive method for find pages was crashing so repalced with more
  CQL and https://developer.atlassian.com/server/confluence/cql-field-reference/#ancestor
• added passing parent id so we can take only few but consistent set of pages for demo
  purpsoes
• Noted bugs and issues in the code and added todos

For embed-ingest-job-example:

• parameterize the table names used in the job
• add clean-up deleted rows (though afterwards I realised it's redundant for now as
  we need to drop the table first as the postgres ingestion does not
  support upserts (updates))
• as the embedding job is written in so generic way. Actually, there's no
  need to tie it to confluence at all. It would work for any dataset.
• Added multiple TODOs for missing features. The job could be even
  further generalized if our ingestion framework improves
• renamed embed-ingest-job-exmaple ot pgvector-embedder to better show its responsibilities

[duyguHsnHsn]

Looks good to me! Should we create ticket for some of the todos?

[yonitoo]

Awesome generalization. LGTM.
A bit of a side note but I recently read this SentenceTransformer issue suggesting that the stop words removal and lemmatization (part of our text cleaning) are not needed for transformer models - passing the original text is suggested. Maybe we can remove the whole cleaning part and directly embed (could be done here or as part of another story).

[antoniivanov]

Looks good to me! Should we create ticket for some of the todos?

Not yet. As we build a backlog for the next milestones, then we will create the needed tickets.

[antoniivanov]

Awesome generalization. LGTM. A bit of a side note but I recently read this SentenceTransformer issue suggesting that the stop words removal and lemmatization (part of our text cleaning) are not needed for transformer models - passing the original text is suggested. Maybe we can remove the whole cleaning part and directly embed (could be done here or as part of another story).

Ok. Maybe separately. But I also want to have some cleaning logic, because it's something you would expect ot have in a pipeline and we need to figure out how to handle it properly.