Tinycbor mynewt upstream #83

vrahane · 2017-12-01T18:57:03Z

Upstreaming changes made by mynewt and carrying over fixes

Bring tinycbor up to date with mynewt tinycbor
Changing implementation of cbor_encoder_get_extra_bytes_needed() and
cbor_encoder_get_buffer_size() as part of cbor_buf_writer APIs
Move bytes_needed field from CborEncoder to the buf writer, this is needed by the tests mainly.
Fix cbor_buf_cmp to do memcmp and return complemented result
iterate_string_chunks(): fixing NULL compare at the end of string
and moving it out of the iterate_string_chunks(). This is to avoid
buffer specific parser calls in the function
cbor_value_get_next_byte() is removed in mynewt version of tinycbor,
so, we track offsets of the buffer which can be used for comparison
in the parser tests instead of calculating the offset
Making the decoder and parser APIs backwards compatible
Adding encoder writer and parser reader as part of the encoder and
parser structure. This is to make the encoder and parser use new
function of encoder_writer and decoder_reader without breaking backwards
compatibility.
Making the old API use flat buffers by default
Adding APIs for initializing encoder and parser with custom writer and
reader
cpp test now uses tinycbor lib
Make the default writer and reader conditional based on
NO_DFLT_READER/WRITER define. This is because we want a default
reader/writer to avoid API changes.
Use it->offset instead of it->ptr to track buffer offsets
Update resolve_indicator() static api parameters to use cbor value
and access offsets instead of taking pointers as input parameters
In validate_container() do a byte by byte comparison instead of
memcmp since we no longer have access to the buffer directly
Also, use offsets instead of pointers to validate sorted maps
Added a new define for conditionally compiling in float support (NO_FLOAT_SUPPORT).
This is because we want the float support to be compiled in by
default.
Use static_assert macro instead of Static_assert. Changed to avoid
build failures.
Add api to get string chunk, this is a callback which can be used by
buffer implementations to grab a string that is divided in chunks
which spans across multiple chained buffers

thiagomacieira

Please rebase and split into multiple PR, one per logical change. I'm not too strict on the changes being atomic, but I do want to see them grouped into logical chunks that go together.

I also need an explanation for why you've done some of the changes.

carlescufi · 2017-12-01T19:50:46Z

@thiagomacieira thanks for the early feedback, is 1 PR with multiple, atomic, self-contained commits OK or do you explicitly want multiple PRs?

thiagomacieira · 2017-12-01T19:52:54Z

I would prefer multiple PRs if you can. If you need, you can have more than one commit in each PR too.

vrahane · 2017-12-07T19:20:46Z

Thank you for the quick review @thiagomacieira , I have rebased and resolved all conflicts. I would like to understand more about how you would like to split this PR into multiple PRs. This PR basically adds support for chained buffers with keeping backwards compatibility intact.

I have a suggestion, please let me know if this is what you would like :

Make a PR for encoder and parser changes combined
Make a separate PR for changes to tools
Make a separate PR for changes to tests

If you would like I can even separate out parser and encoder changes.

Thank you so much for going through this.

thiagomacieira · 2017-12-07T21:23:35Z

I'd like to see changes logically grouped. For example, the first commit in the series says "tinycbor bug fixes". That does not belong with the rest. Please submit bug fixes in separate PRs, one per bugfix.

Another one is "Fix Windows build". Well, the Windows build isn't broken right now, so you must have broken it in your series. Remove the breakage instead of adding a later patch to fix. Similarly for "Remove mynewt specific pkg.yml" -- there's no such file right now, so if there's a file to be removed, it's because an earlier commit added it. Remove the addition.

vrahane · 2017-12-07T22:26:52Z

Ah, I see what you mean now. I have created a consolidated PR for now. I will remove any bug fixes from this and put them in separate PRs. Thank you.

thiagomacieira · 2017-12-27T01:04:56Z

Please see https://github.com/thiagomacieira/tinycbor/tree/dev for an initial API. Pay close attention to 86053d7

vrahane · 2018-01-03T00:20:03Z

@thiagomacieira Are you suggesting a change to the API that the PR adds. I have rebased and resolved the conflicts that were introduced by the most recent merges.

thiagomacieira · 2018-01-04T23:03:38Z

@vrahane thanks for your attempt at rebasing, but you mustn't have done it right. Please look at your commit's change to cbor.h: it's removing a lot of code, the new API I added in 0.5. And your commit still does too much in a single go. I'd need you to split them into separate, logical chunks.

As for the API for reading and writing outside of a linear buffer, I had a need for it so I developed my own version (without looking at yours). I am asking you to review it and provide feedback. I have yet to add docs about it, but the API is unit-tested.

vrahane · 2018-01-05T00:22:25Z

Thank you for the reply. The code from cbor.h has been moved to src/cbor_buf_writer.c. The constants & enums have been moved to cbor_defs.h. Code has not been removed. There are no API changes/removals in the PR. I can try to break it more into logical chunks.

Your version looks good to me but I think some functionality is missing for it to work with chained buffers. Would you like to get on a call and discuss the comparison between your changes and the changes in the PR ? That might be the best way to get an understanding what each of us want.

thiagomacieira · 2018-01-05T01:50:28Z

You need to justify those changes: why do we need new headers?

Let's schedule a call next week (can you send me an email reminding me?). I'd like to understand why it may not work with chained buffers. I've got a PoC implementation that should work with any kind of buffer.

vrahane · 2018-01-08T20:12:50Z

There were compilation errors with cbor.h being included in src/cbor_buf_writer.c and src/cborparser.c. Hence, the new headers which separate out cbor constants and enums from inline apis. I do not have the exact compilation error log.

I have removed the extraneous header. It was needed earlier, it is not needed anymore.
I have also removed any unrelated changes from this PR.

I will send you an email shortly scheduling a call. One case where the API you are suggesting might not work is for a string that spans across multiple chained buffers. I can compare more between your suggested API and my PR and we can discuss more over the call.

thiagomacieira · 2018-01-08T23:59:11Z

What I've done for strings is basically punt the problem. Both the "dup" and "copy" string functions require linear buffers. If your buffer isn't linear, then you must use the "get_string_chunk" function, which does not try to access the string in the first place. It's up to the caller to read string from the buffers.

vrahane · 2018-01-09T00:17:26Z

There was some confusion, because the hash you had provided was from intel/tinycbor/dev and the URL pointed to your dev branch of your fork of tinycbor. Hence, the confusion.

I see what you mean. Giving it some more thought, I can try to change Mynewt's chained buffer (mbuf) API for tinycbor and test out your implementation. Would you like me to do that ?

thiagomacieira · 2018-01-09T00:25:31Z

Yes, that would be helpful. Right now, I have a strawman only and more testing is required.

ccollins476ad · 2018-01-16T23:05:24Z

(Regarding the new API in https://github.com/thiagomacieira/tinycbor/tree/dev)

Just a question - how would you expect get_string_chunk() to be used with chained buffers?

static CborError get_string_chunk(CborValue *it, const void **bufferptr, size_t *len)

Maybe my understanding of this function's semantics is incorrect. Here is my understanding: a successful call causes *bufferptr to point to a buffer containing a string chunk of size *len. What if we have a 5-byte string chunk ("aaaaa") spread across two buffers, as follows:

+-------------+     +-------+
| 65 61 61 61 | --> | 61 61 |
+-------------+     +-------+

In other words, the first three characters are in the first buffer, and the remaining two are in the second buffer.

Did you have any thoughts about how get_string_chunk() should handle a case like this?

EDIT: Also, cbor_value_advance() seems to only work with fixed length buffers (it calls _cbor_value_copy_string()). Would you agree?

thiagomacieira · 2018-01-17T03:53:41Z

That's a very good question and that's where I'd like input. It is possible with the API as I wrote it, but it's ugly.

Please note that this is more complex than what you asked. There's reading from a chained buffer and there's also reading into a chained buffer.

There are two possible implementations with the API. The internal get_string_chunk function calls the transfer_string callback with the pointer that the user passed, the number of bytes to skip and the number of bytes in the string chunk. So the callback must first skip offset bytes, then arrange for len bytes of the string chunk to be made available. The choice is here: it can store a pointer (or something else) or something else in the user's buffer then advance the buffer by len bytes, or it can do nothing.

The "do nothing" case is possible because if you're iterating chunks, then TinyCBOR will return after that, allowing the input buffer to be positioned exactly at the beginning of the string. The caller can then deal with the string directly, in zero-copy fashion. It just needs to be sure to have advanced the buffer by len before calling _cbor_value_get_string_chunk again. This is what I've done in my implementation in Qt (see https://gitlab.com/thiagomacieira/qtbase/blob/524372bc4691be05eacb845999b839a687181565/src/corelib/serialization/qcborstream.cpp#L498 and https://gitlab.com/thiagomacieira/qtbase/blob/524372bc4691be05eacb845999b839a687181565/src/corelib/serialization/qcborstream.cpp#L531).

thiagomacieira · 2018-01-17T04:19:49Z

The consequences for the API:

_cbor_value_get_string_chunk: works. Whether the value stored in bufferptr is a pointer or something different depends on what the transfer_string callback does.
_cbor_value_copy_string: works only if buffer parameter is NULL and only if it's not the "do nothing" solution.
_cbor_value_dup_string: does not work (but it calls malloc(), so you didn't want it anyway)

Let'st take your 5-character example:

   +-------------+     +-------+
   | 65 61 61 61 | --> | 61 61 |
   +-------------+     +-------+

Before get_string_chunk, the buffer state (whatever it is) points to the 0x65 byte. After the call, the two options are:

the input buffer points to the first 0x61 byte, len was set to 5. The caller must process 5 bytes, then advance the input buffer past the last 0x61.
the input buffer points to after the last 0x61, len was set to 5 and buffer was set to something non-zero that allows the caller to find the first 0x61 (for example, an offset from the beginning of the buffer).

ccollins476ad · 2018-01-17T05:24:01Z

Thanks for the explanation, Thiago. It makes sense.

When you say the transfer_string callback can "do nothing", it should still add the offset to the reader's internal pointer (or offset count), right? That seems to break cbor_value_calculate_string_length(), as it causes the parser to advance past the string descriptor. Or maybe I have misunderstood something...

thiagomacieira · 2018-01-17T16:33:17Z

No, you're correct. If it does not adjust the internal pointer, any TinyCBOR function that iterates over the string will fail.

My Qt wrapper didn't call any such function, so there was no failure. That's how it got away with "doing nothing". Or so I thought: the validate() function does need that and must be failing, it just happen not to be tested with the condition. Thanks for pointing out.

vrahane · 2018-01-25T00:52:56Z

@thiagomacieira Can you please restart the travis-ci build. It seems it has got stuck in some intermittent state.

thiagomacieira · 2018-01-29T02:24:29Z

src/cbor_decoder_reader.h

+    cbor_reader_get16 *get16;
+    cbor_reader_get32 *get32;
+    cbor_reader_get64 *get64;
+    cbor_memcmp       *cmp;


Good idea to add memcmp and memcpy. I'd been thinking how to achieve those and hadn't come up with this yet.

thiagomacieira · 2018-01-29T02:24:59Z

src/cbor_cnt_writer.h

+    /* use this count writer if you want to try out a cbor encoding to see
+     * how long it would be (before allocating memory). This replaced the
+     * code in tinycbor.h that would try to do this once the encoding failed
+     * in a buffer.  Its much easier to understand this way (for me)


I'm not going to accept this.

thiagomacieira · 2018-01-29T02:35:03Z

Ok, I like your changes, but not enough to accept them. There are minor things in the change (like structures not following my coding style), but the most important thing is that it's a large growth in size.

I need some time to study the possibilities and how they affect the code it generates. Moreover, I need to merge with my own changes I've made for dev, which already group the reading and writing. I will incorporate some of your ideas, but not all. Given my current time commitments, this should take two or three months to complete.

Not requesting changes. I'm going to use the change as ideas, but will not accept this review.

- Pull latest changes from intel/tinycbor#83

- Bring tinycbor up to date with mynewt tinycbor - Changing implementation of cbor_encoder_get_extra_bytes_needed() and cbor_encoder_get_buffer_size() as part of cbor_buf_writer APIs - Move bytes_needed field from CborEncoder to the buf writer, this is needed by the tests mainly. - Fix cbor_buf_cmp to do memcmp and return complemented result - iterate_string_chunks(): fixing NULL compare at the end of string and moving it out of the iterate_string_chunks(). This is to avoid buffer specific parser calls in the function - cbor_value_get_next_byte() is removed in mynewt version of tinycbor, so, we track offsets of the buffer which can be used for comparison in the parser tests instead of calculating the offset - Move cbor_encoder_get_extra_bytes_needed() and cbor_encoder_get_buffer_size() to be part of cbor_buf_writer APIs - Add bytes_needed field to the buf writer - Adding encoder writer and parser reader as part of the encoder and parser structure. This is to make the encoder and parser use new function of encoder_writer and decoder_reader without breaking backwards compatibility. - Making the old API use flat buffers by default - Adding APIs for initializing encoder and parser with custom writer and reader - Make the default writer and reader conditional based on NO_DFLT_READER/WRITER define. This is because we want a default reader/writer to avoid API changes. - Move enums to cbor_defs.h - Use it->offset instead of it->ptr to track buffer offsets - Update resolve_indicator() static api paramaters to use cbor value and access offsets instead of taking pointers as input parameters - In validate_container() do a byte by byte comparison instead of memcmp since we no longer have access to teh buffer directly Also, use offets instead of pointers to validate sorted maps - Added a new dfine for conditionally compiling in float support (NO_FLOAT_SUPPORT). This is because we want the float support to be compiled in by default. - Use static_assert macro instead of Static_assert. Changed to avoid build failures. - Add api to get string chunk, this is a callback which can be used by buffer implementations to grab a string that is divided in chunks which spans across multiple chained buffers

- Pulling in code from intel/tinycbor#83 as a patch to zephyr's ext/tinycbor. This is to facilitate the use of chained buffers functionality for tinycbor while it is in development on https://github.com/intel/tinycbor - Bring tinycbor up to date with mynewt tinycbor - Changing implementation of cbor_encoder_get_extra_bytes_needed() and cbor_encoder_get_buffer_size() as part of cbor_buf_writer APIs - Move bytes_needed field from CborEncoder to the buf writer, this is needed by the tests mainly. - Fix cbor_buf_cmp to do memcmp and return complemented result iterate_string_chunks(): fixing NULL compare at the end of string and moving it out of the iterate_string_chunks(). This is to avoid buffer specific parser calls in the function - cbor_value_get_next_byte() is removed in mynewt version of tinycbor, so, we track offsets of the buffer which can be used for comparison in the parser tests instead of calculating the offset - Making the decoder and parser APIs backwards compatible - Adding encoder writer and parser reader as part of the encoder and parser structure. This is to make the encoder and parser use new function of encoder_writer and decoder_reader without breaking backwards compatibility. - Making the old API use flat buffers by default - Adding APIs for initializing encoder and parser with custom writer and reader - Make the default writer and reader conditional based on NO_DFLT_READER/WRITER define. This is because we want a default reader/writer to avoid API changes. - Use it->offset instead of it->ptr to track buffer offsets - Update resolve_indicator() static api parameters to use cbor value and access offsets instead of taking pointers as input parameters - In validate_container() do a byte by byte comparison instead of memcmp since we no longer have access to the buffer directly - Also, use offsets instead of pointers to validate sorted maps - Added a new define for conditionally compiling in float support (NO_FLOAT_SUPPORT). - This is because we want the float support to be compiled in by default. - Use static_assert macro instead of Static_assert. Changed to avoid build failures. - Add api to get string chunk, this is a callback which can be used by buffer implementations to grab a string that is divided in chunks which spans across multiple chained buffers - Add KConfig and CMakeList files for configuration and build - Delete .gitignore and .gitattributes - Remove tools, tests and examples as they are not really needed for building the library Signed-off-by: Vipul Rahane <[email protected]>

- Make half float encode/decode conditional - Change defaults for CBOR_WITHOUT_OPEN_MEMSTREAM, CBOR_NO_HALF_FLOAT_TYPE and CBOR_NO_FLOATING_POINT to y so that half float, float and open memstream support along with newlib libc does not get compiled in - src/cborpretty.c, src/cbortojson.c and src/cborvalidation.c conditionally include math.h and half float type support - Conditionally include math.h in src/compilersupport_p.h to avoid newlib libc from getting compiled in - Conditionally compile src/cborparser_dup_string.c if newlib libc is compiled in Signed-off-by: Vipul Rahane <[email protected]>

thiagomacieira

Can you split the FP part into a separate PR, with just one commit?

thiagomacieira · 2018-01-29T02:31:47Z

src/cborparser_dup_string.c

@@ -33,8 +33,8 @@
 #endif

 #include "cbor.h"
-#include "compilersupport_p.h"


Our includes must always come before system's.

Note: This is commit 3ef6799f633df19a84b99ccd0f21fb8c02201690 of a PR against the tinycbor library: intel/tinycbor#83

nvlsianpu · 2018-04-06T12:51:59Z

src/compilersupport_p.h

@@ -37,7 +41,9 @@
 #  include <assert.h>
 #endif
 #include <float.h>
+#ifndef CBOR_NO_FLOATING_TYPE


CBOR_NO_FLOATING_POINT?

or CBOR_NO_HALF_FLOAT_TYPE?

thiagomacieira previously requested changes Dec 1, 2017

View reviewed changes

vrahane force-pushed the tinycbor_mynewt_upstream branch 2 times, most recently from 855498c to b60f787 Compare December 7, 2017 00:07

vrahane force-pushed the tinycbor_mynewt_upstream branch from 12230ab to 12d89ef Compare December 7, 2017 22:21

vrahane force-pushed the tinycbor_mynewt_upstream branch 2 times, most recently from f447cf8 to 53c03ec Compare December 15, 2017 20:39

vrahane mentioned this pull request Dec 15, 2017

Fix compiler warnings for tst_cpp #88

Merged

vrahane force-pushed the tinycbor_mynewt_upstream branch 2 times, most recently from 6148e89 to 423f94b Compare January 3, 2018 00:02

vrahane force-pushed the tinycbor_mynewt_upstream branch 2 times, most recently from 88c8d4c to 3ef6799 Compare January 8, 2018 23:01

vrahane force-pushed the tinycbor_mynewt_upstream branch 4 times, most recently from 209e557 to f7f4a78 Compare January 25, 2018 00:42

vrahane force-pushed the tinycbor_mynewt_upstream branch from f7f4a78 to 9a28193 Compare January 26, 2018 21:04

thiagomacieira reviewed Jan 29, 2018

View reviewed changes

vrahane added a commit to apache/mynewt-mcumgr that referenced this pull request Jan 30, 2018

Pull latest tinycbor changes from the PR#83

d046efd

- Pull latest changes from intel/tinycbor#83

vrahane mentioned this pull request Jan 30, 2018

Add tinyCBOR external library and apply mynewt upstream patch from intel/tinycbor#83 zephyrproject-rtos/zephyr#5912

Merged

vrahane force-pushed the tinycbor_mynewt_upstream branch from 9a28193 to ece8b46 Compare February 6, 2018 22:59

thiagomacieira reviewed Feb 8, 2018

View reviewed changes

ccollins476ad added a commit to apache/mynewt-mcumgr that referenced this pull request Feb 13, 2018

External tinycbor library.

42b2816

Note: This is commit 3ef6799f633df19a84b99ccd0f21fb8c02201690 of a PR against the tinycbor library: intel/tinycbor#83

ccollins476ad added a commit to apache/mynewt-mcumgr that referenced this pull request Feb 13, 2018

External tinycbor library.

dce8b05

Note: This is commit 3ef6799f633df19a84b99ccd0f21fb8c02201690 of a PR against the tinycbor library: intel/tinycbor#83

ccollins476ad added a commit to apache/mynewt-mcumgr that referenced this pull request Feb 15, 2018

External tinycbor library.

0c94fc9

Note: This is commit 3ef6799f633df19a84b99ccd0f21fb8c02201690 of a PR against the tinycbor library: intel/tinycbor#83

nvlsianpu reviewed Apr 6, 2018

View reviewed changes

nvlsianpu mentioned this pull request Apr 6, 2018

ext: lib: encoding: tinycbor: fixup newlib_libc reqirement zephyrproject-rtos/zephyr#6968

Merged

jimparis mentioned this pull request Oct 8, 2019

Fix buffer overflow in _cbor_value_copy_string zephyrproject-rtos/tinycbor#7

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Tinycbor mynewt upstream #83

Tinycbor mynewt upstream #83

vrahane commented Dec 1, 2017 •

edited

Loading

thiagomacieira left a comment

carlescufi commented Dec 1, 2017

thiagomacieira commented Dec 1, 2017

vrahane commented Dec 7, 2017

thiagomacieira commented Dec 7, 2017

vrahane commented Dec 7, 2017

thiagomacieira commented Dec 27, 2017

vrahane commented Jan 3, 2018 •

edited

Loading

thiagomacieira commented Jan 4, 2018

vrahane commented Jan 5, 2018

thiagomacieira commented Jan 5, 2018

vrahane commented Jan 8, 2018 •

edited

Loading

thiagomacieira commented Jan 8, 2018

vrahane commented Jan 9, 2018

thiagomacieira commented Jan 9, 2018

ccollins476ad commented Jan 16, 2018 •

edited

Loading

thiagomacieira commented Jan 17, 2018

thiagomacieira commented Jan 17, 2018

ccollins476ad commented Jan 17, 2018

thiagomacieira commented Jan 17, 2018

vrahane commented Jan 25, 2018

thiagomacieira Jan 29, 2018

thiagomacieira Jan 29, 2018

thiagomacieira commented Jan 29, 2018 •

edited

Loading

thiagomacieira left a comment

thiagomacieira Jan 29, 2018

nvlsianpu Apr 6, 2018

nvlsianpu Apr 6, 2018

Tinycbor mynewt upstream #83

Are you sure you want to change the base?

Tinycbor mynewt upstream #83

Conversation

vrahane commented Dec 1, 2017 • edited Loading

thiagomacieira left a comment

Choose a reason for hiding this comment

carlescufi commented Dec 1, 2017

thiagomacieira commented Dec 1, 2017

vrahane commented Dec 7, 2017

thiagomacieira commented Dec 7, 2017

vrahane commented Dec 7, 2017

thiagomacieira commented Dec 27, 2017

vrahane commented Jan 3, 2018 • edited Loading

thiagomacieira commented Jan 4, 2018

vrahane commented Jan 5, 2018

thiagomacieira commented Jan 5, 2018

vrahane commented Jan 8, 2018 • edited Loading

thiagomacieira commented Jan 8, 2018

vrahane commented Jan 9, 2018

thiagomacieira commented Jan 9, 2018

ccollins476ad commented Jan 16, 2018 • edited Loading

thiagomacieira commented Jan 17, 2018

thiagomacieira commented Jan 17, 2018

ccollins476ad commented Jan 17, 2018

thiagomacieira commented Jan 17, 2018

vrahane commented Jan 25, 2018

thiagomacieira Jan 29, 2018

Choose a reason for hiding this comment

thiagomacieira Jan 29, 2018

Choose a reason for hiding this comment

thiagomacieira commented Jan 29, 2018 • edited Loading

thiagomacieira left a comment

Choose a reason for hiding this comment

thiagomacieira Jan 29, 2018

Choose a reason for hiding this comment

nvlsianpu Apr 6, 2018

Choose a reason for hiding this comment

nvlsianpu Apr 6, 2018

Choose a reason for hiding this comment

vrahane commented Dec 1, 2017 •

edited

Loading

vrahane commented Jan 3, 2018 •

edited

Loading

vrahane commented Jan 8, 2018 •

edited

Loading

ccollins476ad commented Jan 16, 2018 •

edited

Loading

thiagomacieira commented Jan 29, 2018 •

edited

Loading