Repository navigation
Is there a way to forward declare nlohmann::json? #314
Description
It's preferable not to have to include large header files in other header files, but in my current project I've ended up with json.hpp included in a lot of headers because I can't find a way to forward declare the nlohmann::json type. It's made complicated by all the templating going on. I was wondering if there is actually a way that I just haven't found yet and, if there is, if it could maybe be included as part of the documentation/readme somewhere?
Activity
gregmarr commentedon Sep 21, 2016
Looks like this isn't possible for a user to do due to the default template parameters. The library would need to be updated to provide a forward declaration header with the default template parameters, and remove the defaults from the definition.
SethHamilton commentedon Sep 21, 2016
You could try adding a
#pragma onceto thejson.hppright above#ifndef NLOHMANN_JSON_HPP. That pragma is supposed to ensure the file isn't even opened on subsequent inclusions. My experience is that compilers are pretty smart about the#ifndef.agauniyal commentedon Sep 22, 2016
Unless you're building your whole project each time, it shouldn't take much time. You could also use ccache to help speed up your build process.
nlohmann commentedon Oct 6, 2016
I think it would be quite cumbersome to realize a forward declaration. All I can think of right now would be a new header with a forward declaration defining a preprocessor symbol and adjusting the current
json.hppto use that preprocessor symbol to decide whether the default template parameters should be defined.ralphtandetzky commentedon Oct 12, 2016
I added a file src/json_fwd.hpp in my forked repo. This header file is minimal and provides a forward declaration of nlohmann::basic_json and adds a typedef nlohmann::json. I've tested it with MinGW and it works. I've also made a pull request to the main repo, so I hope the new header file will be visible to the public there soon.
nlohmann commentedon Oct 12, 2016
@ralphtandetzky Thanks! Just out of curiosity: Does the forward declaration header improve your compilation times? Or maybe @rachelnertia could also have a look at this?
jaredgrubb commentedon Oct 12, 2016
Would it be possible to do something along these lines?
struct json : basic_json<...> { using basic_json::basic_json; };I know this is not what std::string does, but it would solve the forward declare problem, and it would solve another annoyance I have. Whenever I try to use a debugger, I see a lot of ugly C++ symbols around json. It would be nice that you only have to "pay for" the symbol noise if you actually do something custom.
For example:
void std::__1::vector<nlohmann::basic_json<std::__1::map, std::__1::vector, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, bool, long long, unsigned long long, double, std::__1::allocator>, std::__1::allocator<nlohmann::basic_json<std::__1::map, std::__1::vector, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, bool, long long, unsigned long long, double, std::__1::allocator> > >::__push_back_slow_path<nlohmann::basic_json<std::__1::map, std::__1::vector, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, bool, long long, unsigned long long, double, std::__1::allocator> >(nlohmann::basic_json<std::__1::map, std::__1::vector, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, bool, long long, unsigned long long, double, std::__1::allocator>&&)would become
void std::__1::vector<nlohmann::json, std::__1::allocator<nlohmann::json> >::__push_back_slow_path<nlohmann::json>(nlohmann::json&&)gregmarr commentedon Oct 12, 2016
That wouldn't solve the forward declare problem, it would make it worse. You need to have the full definition of the base class to derive from it.
You can make your own struct that derives that way if you find that it helps with debugging, but I don't think changing the type of
nlohmann::jsonat this point is a good idea.jaredgrubb commentedon Oct 12, 2016
Yes, it would solve the forward declare problem because you have a distinct type:
namespace nlohmann { struct json; }You can't do that now because you're not allowed to forward-declare a type alias (ie, typedef).
I agree there could be complications with changing the type, but it might be worth it. (It doesnt actually change convertibility or is-a relationships because 'json' still is-a 'basic_json' thing. But it is a ABI change by definition, which can have consequences).
gregmarr commentedon Oct 13, 2016
Ah, I misunderstood what you were suggesting.
25 remaining items
peku33 commentedon Jan 13, 2017
@mcourteaux Ok, sorry for my mistake. However, the heaviest (in terms of compilation time) part - streams (
iostream,sstream) is removed.Maybe it's off-topic, but using streams makes the executable really large on windows. File grows from 150kB to 600kB using mingw32.
@gregmarr How the type is changed by adding forward declaration?
gregmarr commentedon Jan 13, 2017
I was responding to the message right above mine:
stale commentedon Oct 25, 2017
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
JoroRoss commentedon Nov 1, 2017
As an experiment, I disabled ccache, and built a 146 compilation-unit project I maintain with:
#include<json.hpp>in (almost) every compilation unit - 3m17s#include<json_fwd.hpp>in (almost) every compilation unit - 2m4sjson_fwd.hpp/json.hpponly where needed - 1m40sThis was using the development branch of the
nlohmann/jsonrepo, where I extracted the forward declarations myself, along the lines of @ralphtandetzky's fork.Even if the performance gain of using forward declarations was not quite this pronounced, the forward declarations should arguably be split out anyway: modular design promotes (re-) use, and should be encouraged as a software engineering best practice. Everything Lakos wrote about organizing code in large projects in https://www.amazon.com/Large-Scale-Software-Design-John-Lakos/dp/0201633620 still holds to this day.
Please reopen this ticket.
peku33 commentedon Nov 1, 2017
@JoroRoss what other headers did you include in your files?
I'm not 100% sure, but streams are pretty heavy to compile. If you use them anyway, there is probably going to be no significant gain.
In projects with lots of small compilation units, which does not include streams, this matters.
JoroRoss commentedon Nov 3, 2017
Hey @peku33, thanks for getting back to me on this.
The code-base in question is reasonably well organized, limiting the use of large/complicated headers to compilation units that actually need them. I suppose <string> and <vector> end up in a lot of places, but we try to keep them in implementation files where possible. We (of course) use <iosfwd> where appropriate - most compilation units will pick that up some way or other, as we declare iostream operators with most of our data model classes. Only the data model compilation units (and a hand-full of collaborators) need the actual definitions.
For our own templates, we typically create two headers - one with declarations for clients to pick up, another with definitions. We often introduce dedicated compilation units with template instantiations for the production classes, and include the definition files in a unit tests compilation unit for mocking/stubbing. (We used to not do this; when we put in the effort, we got build times down from 25 minutes to around 3.)
So yeah, sort of a typical codebase, that encapsulates the use of expensive resources.
Regarding your speculations about the compilation time of <iostream>, I'm afraid to say you are wrong. That is, unless every single compilation unit in your project needs every last template definition from <streambuf>, <ostream>, <istream> and <fstream>... (if you're in that situation, you may want to consider restructuring your project!).
The overhead in build times typically comes from (redundant) parsing of headers, not from compilation (unused template declarations don't get compiled, just parsed). The nlohmann/json project luckily uses <iosfwd>, and including those forward declarations in all compilation units has an almost negligible impact on compilation times. Hardly any compilation units actually need to include the actual definitions, and even if they do, they can often limit the impact by specifically including <ostream>, <istream> or <fstream>.
For what it's worth: I just repeated my experiment from before, this time including <iostream> in (almost) every compilation unit. The build time went to 2m20s. My test is producing a CMake Debug build, so the build time is dominated by IO and parsing. The pre-compiler emits about twice as much code for the json header as it does for the iostream one (as can be readily verified by running g++ with the -E flag.)
gregmarr commentedon Nov 4, 2017
@JoroRoss Have you seen this PR? https://github.com/nlohmann/json/pull/700/files
https://github.com/theodelrieu/json/blob/61638bef9d8ff945c8ff9e6d511659b3d8223811/src/json_fwd.hpp
nlohmann commentedon Jan 28, 2018
As PR #700 is merged, forward declaration is now possible. It will be part of 3.1.0 to be released next week.