Request: binary serialization/deserialization

Wouldn’t it be useful to have the possibility to serialize the json object to a more compact form?
This would make sense for storing the container or sending it over an internet connection.

The MessagePack (http://msgpack.org/) format or some proprietary format directly derived from your internal representation would be fine.
Or could you probably just provide some generic interface for the binary serialization/deserialization?

I think this approach could be more useable than just gzipping the whole JSON data.

As an example this library did implement some alternative representations of a json container: http://greatpanic.com/progdocs/UniversalContainer.html
(But this library doesn’t work properly and doesn’t use C++ 11)

I also had a look at MessagePack, and it seems to be a reasonable approach. As it can express more than JSON, I hope that users would understand that the library would not be able to parse arbitrary MessagePack input.

starting to integrate MessagePack (#358)

I started to implement MessagePack. Commit 543745a contains a basic serializer/deserializer for JSON values as static functions. It works with a std::vector<uint8_t> and currently has no diagnosis information in case of a parse error.

Feedback would be greatly appreciated:

naming of the functions
static function vs. member function vs. free function
move implementation outside the main header?
organization of the functions (I am thinking of reorganizing the large if in the parser, maybe using re2c)
improvements of the low-level shifting/conversion stuff
byte vector vs. containers vs. ???

Wow, I couldn’t believe how fast you did implement a first version. And it seems to be working!
I think it’s a great benefit being able to exchange data with the MessagePack world. Except binary data, but binary data and JSON is a bad match anyway (base 64 encoding).

I did test the space savings in some of my use cases and got the following results.
In simple cases (small data set) with almost no numeric data the saving was 12% over the compacted JSON file. Zipping alone showed a 15% reduction. First converting to msgpack and zipping the result gave a 19% reduction. As expected, with more numeric data the compression rate did grow constantly. Nice!

The naming to_msgpack and from_msgpack makes sense and is future save if you like to add more serializers/deserializers.
I prefer the static function. It’s the right context and can be still used in a flexible way.
At the moment it doesn’t hurt to have just one header. It’s just more convenient.
I’am not sure if re2c would be necessary in the parsing part. A hand crafted parser could be faster (the same is true for the json part - rapid json). Using a switch statement (for the fixed values) or a jump table could possibly speed up parsing of the MessagePack data(needs some tests to be verified).
The low-level shifting/conversion stuff could be improved but the code will get more complicated (dealing with endianness).
A byte vector is fine. It has some nice properties. A lot of frameworks have chosen byte vectors for handling serialized data.

Thank you for your great JSON implementation!

I beg to differ. MsgPack is not standardized, whereas CBOR, for example, is. There are a myriad of binary data formats out there, but, for better or worse, CBOR is standardized in RFC 7049, whereas MsgPack is not.

+1 for CBOR!

I shall have a look at CBOR.

Ok, standards are great, I like them. CBOR is a PROPOSED standard and could still get changed. The proposal has some nice properties and it seems to be an improved MessagePack protocol. Unfortunately it’s not widely adopted and there are a lot of poor implementations. There’s no modern C++ 11 implementation and the old style C++ implementation is looking more like C code.

A big benefit of having one file with declarations and one-or-many files with definitions is that it would allow for explicit template instantiation, so that the compilation overhead related to json.hpp can be factored out into a separate translation unit.

I am currently cleaning up and bringing the test coverage to 100%. I hope to release CBOR/MessagePack by next week.

📝 clean up and added documentation for #358

Merged e906933.

@nlohmann

Hi @nlohmann, great job! I have looked at your code, some remarks:

When converting single / double, you have reimplemented memcpy. Why don't you just use it? (or std::copy, would be actually even nicer).
you could #define all those magic numbers (or put them in an enum)
msgpack / cbor are so similar that a lot of code is duplicated. probably you could template those conversions based on serialization format (so, two structs: msgpack and cbor, both of which contain all magic numbers, limits for the integers to fit in the first byte and the conversion functions from small integer to one byte)

All in all very nice job, thanks!

@marton78

@marton78 Thanks for the feedback.

Regarding memcpy: I need to have another look.
Regarding the magic numbers / reduce code duplication: There are in fact a lot of similarities between CBOR and MessagePack, but I doubt that creating a configurable super type from which both formats inherit makes the code much more readable. I may play around with it, but right now I'd rather focus on writing down the limitations of the implementations (i.e., which types are not supported).

@marton78

@marton78 Sorry, but I can't find the spot where I can use memcpy/std::copy.

Maybe it's this part?

            case value_t::number_float:
            {
                // Double-Precision Float
                v.push_back(0xfb);
                const uint8_t* helper = reinterpret_cast<const uint8_t*>(&(j.m_value.number_float));
                for (size_t i = 0; i < 8; ++i)
                {
                    v.push_back(helper[7 - i]);
                }
                break;
            }

Also, if this part of code expects the number_float_t to be double, a static_assert would be appropriate?

Yes, it's this part, std::copy_backward could be used here instead of memcpy though, since copying is done in reverse.

This looks like the endianness handling business?
If so, it can't be assumed that the host platform is little-endian. The ntoh and hton family functions are there to take the host endianness into account and reverse the bytes efficiently as appropriate, but these are not portable. It looks like boost endian library also does this and compiles to efficient byte-swapping instructions too (not saying we should add boost dependency).

Lol definetly not, look at my swap file to get some ideas:

https://github.com/user1095108/generic/blob/master/swap.hpp

FYI: I asked to advertize the latest release on the CBOR (cbor/cbor.github.io#25) and MessagePack (msgpack/msgpack#221) website.

Navigation Menu

Repository navigation

Request: binary serialization/deserialization #358

20 remaining items

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Request: binary serialization/deserialization #358

Description

Activity

nlohmann commented on Nov 9, 2016

nlohmann commented on Nov 11, 2016

CyberBrainX commented on Nov 14, 2016

user1095108 commented on Nov 17, 2016

marton78 commented on Nov 21, 2016

nlohmann commented on Nov 23, 2016

CyberBrainX commented on Nov 25, 2016

20 remaining items

TurpentineDistillery commented on Dec 2, 2016

nlohmann commented on Dec 10, 2016

nlohmann commented on Dec 11, 2016

marton78 commented on Dec 11, 2016

nlohmann commented on Dec 11, 2016

nlohmann commented on Dec 14, 2016

TurpentineDistillery commented on Dec 15, 2016

marton78 commented on Dec 15, 2016

TurpentineDistillery commented on Dec 15, 2016

user1095108 commented on Dec 15, 2016

nlohmann commented on Dec 16, 2016

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions