optimize srv broker and executor logic

[fkzhao]

What problem does this PR solve?

Optimize task broker and executor for reduce memory usage and deployment complexity.

Type of change

• Performance Improvement
• Refactoring

Change Log

• Enhance redis utils for message queue(use stream)
• Modify task broker logic via message queue (1.get parse event from message queue 2.use ThreadPoolExecutor async executor )
• Modify the table column name of document and task (process_duation -> process_duration maybe just a spelling mistake)
• Reformat some code style(just what i see)
• Add requirement_dev.txt for developer
• Add redis container on docker compose

[KevinHuSh]

Supper fantastic! Appreciate for your contribution!
A concern: if the name 'process_duation' is refined, how to update it to the existing users easily and smoothly, who have already deployed locally?
Do you have any opionion?

[fkzhao]

Supper fantastic! Appreciate for your contribution! A concern: if the name 'process_duation' is refined, how to update it to the existing users easily and smoothly, who have already deployed locally? Do you have any opionion?

I think the best solution is manually change column of existing table.
If the change merge to main, I will open an issue to record how to fix it.
SQL:
ALTER TABLE rag_flow.task CHANGE COLUMN process_duation process_duration FLOAT;;
ALTER TABLE rag_flow.document CHANGE COLUMN process_duation process_duration FLOAT

[KevinHuSh]

Supper fantastic! Appreciate for your contribution! A concern: if the name 'process_duation' is refined, how to update it to the existing users easily and smoothly, who have already deployed locally? Do you have any opionion?

I think the best solution is manually change column of existing table. If the change merge to main, I will open an issue to record how to fix it. SQL: ALTER TABLE rag_flow.task CHANGE COLUMN process_duation process_duration FLOAT;; ALTER TABLE rag_flow.document CHANGE COLUMN process_duation process_duration FLOAT

What about letting it be. There're bunch of users or developers who do not even know how to execute SQL.
How about letting it be.

[KevinHuSh]

As designed， broker is used for dispatching task and updating progress, and executor is for processing task including embedding since we can deploy one executor or multiple executors in different machines to enlarge the document processing through output.
I suggest that:

1. remove broker, and put the progress updating function to ragflow_server using multi-thread.
2. the executor reads tasks from redis queue and execute them.

[fkzhao]

Yeap, I'm doing this suggest.
The goal is to make the design simpler.

[yingfeng]

Hi, adopting redis as a MQ is a good alternative compared with current MySQL. However, redis does not guarantee the durability of data which might lead to task loss. A better solution is to introduce some dedicated MQ system such as redpanda, however it will introduce extra maintainence overhead. We will consider the tradeoff among maintainability, availability, as well as performance. Thanks for your contribution.

What problem does this PR solve?

Optimize task broker and executor for reduce memory usage and deployment complexity.

Type of change

* [ ]  Performance Improvement

* [ ]  Some code optimize

Change Log

* Enhance redis utils for message queue(use stream)

* Modify task broker logic via message queue (1.get parse event from message queue 2.use ThreadPoolExecutor async executor )

* Modify the table column name of document and task  (process_duation -> process_duration maybe just a spelling mistake)

* Reformat some code style(just what i see)

* Add requirement_dev.txt for developer

* Add redis container on docker compose

[fkzhao]

Supper fantastic! Appreciate for your contribution! A concern: if the name 'process_duation' is refined, how to update it to the existing users easily and smoothly, who have already deployed locally? Do you have any opionion?

I think the best solution is manually change column of existing table. If the change merge to main, I will open an issue to record how to fix it. SQL: ALTER TABLE rag_flow.task CHANGE COLUMN process_duation process_duration FLOAT;; ALTER TABLE rag_flow.document CHANGE COLUMN process_duation process_duration FLOAT

What about letting it be. There're bunch of users or developers who do not even know how to execute SQL. How about letting it be.

Got it, I'm going to roll back this change.
After thinking about it, I don’t think this is a very important change.