Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[KEPLER-MISSION] Apache RocketMQ 6.0 subtasks - native mqtt support #669

Open
2 tasks done
maixiaohai opened this issue Jan 11, 2021 · 5 comments · Fixed by #687
Open
2 tasks done

[KEPLER-MISSION] Apache RocketMQ 6.0 subtasks - native mqtt support #669

maixiaohai opened this issue Jan 11, 2021 · 5 comments · Fixed by #687

Comments

@maixiaohai
Copy link
Contributor

maixiaohai commented Jan 11, 2021 •
edited
Loading

FEATURE REQUEST

  1. Please describe the feature you are requesting.
    Add store plugin of RocketMQ.
    image
  2. Provide any additional detail on your proposed use case for this feature.
  • In our company, there are a lot of IOT devices cases. These devices use MQTT protocol and have high availability need for message service.
    To support high availability and scalability, the metadata and data now stored in memory is not enough. A storage plugin of RocketMQ seems necessary.
  • Except the consume of MQTT message for IOT devices, we cloud consume message from RocketMQ for data calculate and analyze.
  1. Indicate the importance of this issue to you (blocker, must-have, should-have, nice-to-have). Are you currently using any workarounds to address this issue?
    【blocker】
  2. If there are some sub-tasks using -[] for each subtask and create a corresponding issue to map to the sub task:
  • Support metadata storage of RocketMQ
  • Support message storage of RocketMQ
@maixiaohai maixiaohai changed the title [rocketmq-iot-bridge] Support RocketMQ as aStore of IOT Bridge [rocketmq-iot-bridge] Support RocketMQ as a store of IOT Bridge Jan 11, 2021
@maixiaohai
Copy link
Contributor Author

ping~

@vongosling
Copy link
Member

There is a similar todo as you design[1]. since we miss the core members in this sub-project. Would you like to continue this work? I would like to help you:-)

[1] https://github.com/apache/rocketmq-externals/tree/master/rocketmq-iot-bridge

@vongosling vongosling added the wip Work in process. Accept or refused to be continue. label Feb 10, 2021
@maixiaohai
Copy link
Contributor Author

There is a similar todo as you design[1]. since we miss the core members in this sub-project. Would you like to continue this work? I would like to help you:-)

[1] https://github.com/apache/rocketmq-externals/tree/master/rocketmq-iot-bridge

Thanks for your reply~
yeah, we want to make a contribution to this sub-project.

@maixiaohai
Copy link
Contributor Author

Hi, we have completed the code for storing the message to rocketmq broker, and will submit the producer part to the community next week.

Regarding the consumption part, there is a question.

  1. We use push consumers first to consume data, and found that data loss and duplication may occur during the process.
  2. So we prefer to use pull consumers. The use of the pull mode requires synchronization and management of the offset, which has achieved the purpose of mqtt broker scalability. We tend to implement this process by writing metadata to a special topic.

We hope to know what the community's opinion, or what better suggestions and tendencies there are. Thank you.

@vongosling
Copy link
Member

vongosling commented Mar 8, 2021 •
edited
Loading

The pull model is a more flexible way to implement the MQTT function, I recommend this way. You could shield some details in the MQTT proxy.

BTW, you could make PRs( many commits, but a function-complemented commmit at one time) in this branch https://github.com/apache/rocketmq-externals/tree/mqtt-mesh. It is an easy way for the community to help to collaborate with the coding.

@maixiaohai maixiaohai mentioned this issue Mar 9, 2021
Merged
6 tasks
vongosling pushed a commit that referenced this issue Jul 21, 2021
@maixiaohai
[ISSUE #669] Support RocketMQ as storage of mqtt broker (#687)
3aa8ee5 
* Config refactor, add some rocketmq config

* Add rmq publish producer

* Fix NPE start with memory mode, remove useless dependency

* Non-standard mqtt request return directly

* Add rmq subscribe consumer and related pull task, complete mqtt SUBSCRIBE and UNSUBSCRIBE protcol

* Fix subTopic not remove when disconnect

* Fix connection management, fix consume delay to make pull task to queue level

* Fix ut not pass when root topic removed from subsription

* Fix integration test not pass when not use rmq subscribe consumer

Co-authored-by: zhangxu16 <[email protected]>
@vongosling vongosling linked a pull request Jul 21, 2021 that will close this issue
[ISSUE #669] Support RocketMQ as storage of mqtt broker #687
Merged
6 tasks
@vongosling vongosling changed the title [rocketmq-iot-bridge] Support RocketMQ as a store of IOT Bridge [SOC] Apache RocketMQ 6.0 subtasks - native mqtt support Aug 5, 2021
@vongosling vongosling removed the wip Work in process. Accept or refused to be continue. label Aug 5, 2021
@vongosling vongosling pinned this issue Aug 5, 2021
@vongosling vongosling changed the title [SOC] Apache RocketMQ 6.0 subtasks - native mqtt support [SoC] Apache RocketMQ 6.0 subtasks - native mqtt support Aug 5, 2021
@vongosling vongosling added the soc Summer of Code, hosted by Google, Alibaba and so on label Aug 5, 2021
@vongosling vongosling changed the title [SoC] Apache RocketMQ 6.0 subtasks - native mqtt support [KEPLER-MISSION] Apache RocketMQ 6.0 subtasks - native mqtt support Aug 5, 2021
@vongosling vongosling removed the soc Summer of Code, hosted by Google, Alibaba and so on label Aug 5, 2021
flyingzhang pushed a commit to flyingzhang/rocketmq-externals that referenced this issue Oct 28, 2021
@maixiaohai
[ISSUE apache#669] Support RocketMQ as storage of mqtt broker (apache…
1a88e51 
…#687)

* Config refactor, add some rocketmq config

* Add rmq publish producer

* Fix NPE start with memory mode, remove useless dependency

* Non-standard mqtt request return directly

* Add rmq subscribe consumer and related pull task, complete mqtt SUBSCRIBE and UNSUBSCRIBE protcol

* Fix subTopic not remove when disconnect

* Fix connection management, fix consume delay to make pull task to queue level

* Fix ut not pass when root topic removed from subsription

* Fix integration test not pass when not use rmq subscribe consumer

Co-authored-by: zhangxu16 <[email protected]>
@maixiaohai maixiaohai mentioned this issue Nov 1, 2021
Closed
@duhenglucky duhenglucky unpinned this issue Dec 24, 2021
@vongosling vongosling pinned this issue Jan 6, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

[ISSUE #669] Support RocketMQ as storage of mqtt broker maixiaohai/rocketmq-externals
2 participants
@vongosling @maixiaohai