Dynamic packet size #282

mrdunk · 2017-04-30T10:28:20Z

Allow dynamic allocation of buffer size so rebuilding library is not required if user needs greater than MQTT_MAX_PACKET_SIZE (128) byte packet size.
See discussion at #110

knolleary · 2017-05-04T13:06:00Z

Hi, thanks for this. It is, on face value, a reasonable change. It's just needs me to have sufficient spare time to review in detail and go through the process of merging and releasing etc. Can't give you an outlook for that right now.

mrdunk · 2017-05-04T13:50:44Z

All understood. Let me know if i can be of any assistance. thanks, dunk.

…

On Thu, May 4, 2017 at 2:06 PM, knolleary ***@***.***> wrote: Hi, thanks for this. It is, on face value, a reasonable change. It's just needs me to have sufficient spare time to review in detail and go through the process of merging and releasing etc. Can't give you an outlook for that right now. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#282 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AEzU8mWigKZxukuD3_bSeP5o0OqKzYalks5r2c06gaJpZM4NMh-Y> .

hessijames79 · 2017-05-28T15:55:30Z

+1, also interested in dynamic packet size

jonnor · 2017-06-27T10:16:07Z

Any news on this? Right now when using this library have to tell everyone to download a forked copy or hack the existing define in their library folder...

knolleary · 2017-06-27T10:21:28Z

No news - will try to get to it in the next week.

Aat0304 · 2017-07-16T20:41:22Z

+1, more than interested in dynamic packet size. Thx

MarkoHoe · 2017-07-16T22:00:29Z

+1

marcostrullato · 2017-08-17T15:54:20Z

Interested as well.

Thanks

gilles906 · 2017-08-27T12:53:47Z

+1 Interested as well.

Thanks

svanscho · 2017-09-20T07:34:27Z

yep +1 here, seems like some of my messages get lost if they are bigger than 128 bytes

coogle · 2017-09-21T23:39:28Z

+1 Please merge this! It's a completely unnecessary burden to ask people to adjust a compile-time setting to build a library when it's got a dependency on this library. Looking at the code it seems completely backward compatible.

mew1033 · 2017-10-24T16:39:34Z

Definitely interested in this. Thanks for the PR!

gerhardgoosen · 2017-10-28T10:03:19Z

do i have to pull this code manually into my cloned repo.. not seeing this in the master branch

Titantompa

This was done in a hurry so there may be other comments.

Titantompa · 2017-11-07T05:40:40Z

src/PubSubClient.cpp

@@ -12,93 +12,125 @@ PubSubClient::PubSubClient() {
    this->_client = NULL;
    this->stream = NULL;
    setCallback(NULL);
+    this->buffer_size = MQTT_MAX_PACKET_SIZE;


Woudn't it be better to move the buffer allocation to a support function, rather than duplicate the same code in all constructors?

i agree in principal but
i was trying to match the existing code style. (ie, explicitly calling everything in each constructor.)

@knolleary any opinion?

Titantompa · 2017-11-07T05:41:35Z

src/PubSubClient.cpp

+}
+
+PubSubClient::~PubSubClient() {
+  free(this->buffer);


This should check if this->buffer is NULL or not since setBufferSize() can set it to NULL.

Nope. according to the spec std::free() should treat a null pointer as a no-op.
from http://www.open-std.org/JTC1/SC22/wg14/www/docs/n1124.pdf :

The free function causes the space pointed to by ptr to be deallocated, that is, made available for further allocation. If ptr is a null pointer, no action occurs.

(it makes sense because std::malloc returns null pointer on failure.)

Ah, sweet, I haven't really coded C/C++ in ages and back then it was a no-no since freeing a null pointer would cause some environments to go belly up. :-)

Titantompa · 2017-11-07T05:50:51Z

src/PubSubClient.cpp

@@ -586,3 +618,13 @@ PubSubClient& PubSubClient::setStream(Stream& stream){
 int PubSubClient::state() {
    return this->_state;
 }
+
+boolean PubSubClient::setBufferSize(uint16_t size) {
+  this->buffer = (uint8_t*)realloc(this->buffer, size);


This should use a temp variable. If realloc fails it will return a NULL pointer without freeing the original memory. Thus, if NULL is returned this method should not make any change to this->buffer since it is still valid.

yes. this makes sense.

@knolleary is this pull request likely to ever be merged? is it worth me doing this?

Titantompa · 2017-11-07T05:51:47Z

tests/src/receive_spec.cpp

@@ -242,6 +279,7 @@ int main()
    test_receive_stream();
    test_receive_max_sized_message();
    test_receive_oversized_message();
+    test_resize_buffer();


Since the merge request contains functionality to dynamically change the buffer size it would be suitable to have tests that verify that this works well. E g:

What happens if setBufferSize() fails to allocate the memory?

If the buffer size is shrunk will it fail (appropriately) to receive a message that was within the prior limits?

And the reverse, if it fails to receive a message due to size will increasing the buffer size accordingly help?

yes, all make sense. Let's see if this is likely to be merged first.

coogle · 2018-04-08T00:24:23Z

I'm curious if this is going to be merged? My project (https://github.com/ThisSmartHouse/CoogleIOT) relies on PubSubClient for it's MQTT and the lengthy topics the framework uses causes problems (specifically with my heartbeats of devices). Obviously we can side-step this issue with a compile-time flag but what can we do to actually get this dynamic allocation code into the library proper?

janvda · 2018-09-10T22:21:17Z

+1

sskorol · 2019-05-06T19:12:21Z

heavy plus for this feature (as well as for other similar configuration stuff)!..

forcing people to direct sources' modification means additional maintenance effort delegation to client code... so instead of just clicking "install" in a library manager to retrieve a new version, people have to keep an up to date overridden sources in their own libraries or frameworks... that might be quite disappointing... don't you think so?..

but anyway thanks for the great lib!..

TheCoolBrit · 2020-05-19T11:36:54Z

Any news on this?
I am not very up to date on programming so using Arduino IDE with a TTGO T-Display to display 240x135 pictures via MQTT using PubSubClient via WiFi
Getting even this far has been a long winding slog for me so Please
Can anyone show me how to change the PubSubClient to allow #define MQTT_MAX_PACKET_SIZE 64800
Thank you in anticipation.

coogle · 2020-05-19T18:03:45Z

At this point I don't understand why it hasn't been merged, other than the owner of the project just doesn't want to do it 👎

knolleary · 2020-05-19T19:10:51Z

I agree this is a long overdue feature.

The challenge has always been finding the time and energy to do the work needed to accept a change like this - especially as it's a feature I have little personal need for. I've also been fairly preoccupied with one of my other OSS projects...

That said, I have recently started dusting off my hardware so I can test some of the proposed enhancements to the library. This one is high on the list... in fact, I may well have a play with it this evening...

TheCoolBrit · 2020-05-19T21:05:23Z

What do you make to this possible alternative >
https://github.com/marvinroger/async-mqtt-client

Appreciate any comments

knolleary · 2020-05-19T23:52:07Z

Well, it took me 3 years, but we got there eventually. Thanks for your patience @mrdunk.

Aat0304 · 2020-05-20T06:43:08Z

Cool, Thanks, AatVerzonden vanaf mijn mobiele telefoon-------- Oorspronkelijk bericht --------Onderwerp: Re: [knolleary/pubsubclient] Dynamic packet size (#282)Van: knolleary Aan: knolleary/pubsubclient Cc: Aat0304 ,Comment Merged #282 into master. —You are receiving this because you commented.Reply to this email directly, view it on GitHub, or unsubscribe.

* Automatically set mqtt buffer size Current versions of pubsubclient allows the buffer size to be set dynamically: knolleary/pubsubclient#282 We can take advantage of this to avoid having to modify one of the dependencies and instead automatically up the buffer size when a powerful board is detected.

mrdunk · 2020-07-31T15:59:36Z

oh cool, just saw this made it in.
thanks knolleary!
i know it's hard to spend time maintaining stuff you don't have much use for.

mrdunk added 3 commits April 30, 2017 10:29

Dynamic buffer size.

a257083

Add test for dynamic buffer resize.

e21ac7b

Update documentation for dynamic buffer resizing.

5b23a97

mrdunk mentioned this pull request May 4, 2017

Function to change the MQTT packet size outside of the library #110

Closed

Titantompa suggested changes Nov 7, 2017

View reviewed changes

mrdunk mentioned this pull request Jan 26, 2018

Refactor to allow specifying keepAlive, socketTimeout and buffersize #389

Open

knolleary merged commit 64e9811 into knolleary:master May 19, 2020

balvig mentioned this pull request Jun 16, 2020

Automatically set mqtt buffer size 1technophile/OpenMQTTGateway#637

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Dynamic packet size #282

Dynamic packet size #282

mrdunk commented Apr 30, 2017

knolleary commented May 4, 2017

mrdunk commented May 4, 2017 via email

hessijames79 commented May 28, 2017

jonnor commented Jun 27, 2017

knolleary commented Jun 27, 2017

Aat0304 commented Jul 16, 2017

MarkoHoe commented Jul 16, 2017

marcostrullato commented Aug 17, 2017

gilles906 commented Aug 27, 2017

svanscho commented Sep 20, 2017

coogle commented Sep 21, 2017

mew1033 commented Oct 24, 2017

gerhardgoosen commented Oct 28, 2017

Titantompa left a comment

Titantompa Nov 7, 2017

mrdunk Nov 18, 2017

Titantompa Nov 7, 2017

mrdunk Nov 18, 2017

Titantompa Nov 18, 2017

Titantompa Nov 7, 2017

mrdunk Nov 18, 2017

Titantompa Nov 7, 2017

mrdunk Nov 18, 2017

coogle commented Apr 8, 2018

janvda commented Sep 10, 2018

sskorol commented May 6, 2019

TheCoolBrit commented May 19, 2020 •

edited

Loading

coogle commented May 19, 2020

knolleary commented May 19, 2020

TheCoolBrit commented May 19, 2020

knolleary commented May 19, 2020

Aat0304 commented May 20, 2020 via email

mrdunk commented Jul 31, 2020

Dynamic packet size #282

Dynamic packet size #282

Conversation

mrdunk commented Apr 30, 2017

knolleary commented May 4, 2017

mrdunk commented May 4, 2017 via email

hessijames79 commented May 28, 2017

jonnor commented Jun 27, 2017

knolleary commented Jun 27, 2017

Aat0304 commented Jul 16, 2017

MarkoHoe commented Jul 16, 2017

marcostrullato commented Aug 17, 2017

gilles906 commented Aug 27, 2017

svanscho commented Sep 20, 2017

coogle commented Sep 21, 2017

mew1033 commented Oct 24, 2017

gerhardgoosen commented Oct 28, 2017

Titantompa left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

coogle commented Apr 8, 2018

janvda commented Sep 10, 2018

sskorol commented May 6, 2019

TheCoolBrit commented May 19, 2020 • edited Loading

coogle commented May 19, 2020

knolleary commented May 19, 2020

TheCoolBrit commented May 19, 2020

knolleary commented May 19, 2020

Aat0304 commented May 20, 2020 via email

mrdunk commented Jul 31, 2020

TheCoolBrit commented May 19, 2020 •

edited

Loading