17 MB / 90 MB repo size!?

Even 17 MB download is not really that big, it is really unexpected to download that much for single-header json library. Moreover once cloned it takes 90MB on the disk, even more surprising.

And the reason for that are these benchmark test files introduced by cb873a4. Is it really good to put such large files in the repo!? Unfortunately once you have put it then it is not easy anymore to reduce repo size. You will need to rewrite.

Anyway please consider rewriting the repo, and keeping it clean & lean, then likely everyone can use it as submodule.

@nanoant

Hi @nanoant,

any idea how to reorganize without fragmenting stuff around into other repos or to other sites? I understand your concern, but at least now everything is in one place. In particular, the benchmark helped to improve at least one pull request.

Cheers,
Niels

@nlohmann

@nlohmann My concern is just these huge benchmark input files that blow the repo size, these may be just moved out from the main repo to some other repo, and then cloned on first benchmark run (in Makefile). Anything else is already pretty small.

Right now I am probably going for embedding json.hpp directly into our repo rather than using submodule for this reason.

Would a special development branch help? Then I could move most of the stuff from there and leave the master branch small. (Sorry, I have little experience with Git.)

@nlohmann

@nlohmann I think the problem is that once the object (huge file) is there in the repo then there is no difference in which branch it resides, the clone will still fetch whole repo including huge file.

I was able to reduce repo size to 3.4 MB with recipe taken from: http://manuel.manuelles.nl/blog/2011/12/22/shrinking-your-git-repository/

But still this is lots of effort, so I probably you should skip that, and leave the repo as is.

@nlohmann

@nlohmann
what do u think about breaking the repo into:

library repo: only contains the single header file
utilities repo: contain all other stuff (docs, tests, ...)

the library repo will be easy to be integrated into other 3rdparty codes (by git submodules, subtree ...)

to prevent radical changes to the current workflow, you can create another repo (a single header) as release repo and let people use that repo as dependency simply by git or cmake.

This split would make the development a bit inconvenient on my side. As I am doing this alone in my spare time, I would like to keep it as is.

Why would you include a single-header project as submodule?

That's how I use it. That way the link is explicit, and it tracks a particular version, rather than manually extracting the one header file from the repository with no indication that it came from another repo, and have to deal with the possibility that someone may modify the local copy. However, I have no problem with the size of this project as a submodule.

@nlohmann

@nlohmann you can consider using git subtree split to move out the tests (can be optional submodule of main) and other big subfolders out of main repo, then you force push small master bash to main repo. This will break obviously other ppl forks, but I guess it will bring more good than harm overall.

@nanoant

@nanoant To me, this splitting would be inconvenient, and I like having everything in one repository.

Guys. I think the best solution is to have a package manager that will download whole repo, insert json.hpp into headers dir and remove repo

To add to the discussion I will just add my argumentation here. I want to use "json" as a submodule to my project, because:

I want to be transparent is ownership, to give credits where they are due.
To facilitate updating from source.

However, the current repository is extremely large, due to the tests and benchmarks. This hinders again the distribution of my code. Therefore it would be great if a minimal repository can be maintained.

I'll also chime in here - I also like using even single-header libs as a submodule to be able to track what version I'm using and easily be able to update by just pulling the submodule. And yes, for me it is also problematic that 90mb is downloaded. It takes ages, even on my fast connection and this is gonna hit everyone using my repo as well, including autobuilders. This memory/bandwidth cost of 50+ mb just coming from a single test file is ridiculous.

@nlohmann

@nlohmann: Sure! I've created a PR and the preliminary tests seems promising.

@theodelrieu: Thanks for pointing that out! I will look into it.

Since this change can affect the workflow of the users, there is the possibility that I've overlooked some use-cases. Therefore, besides passing the CI tests, hearing others' experience would important to make sure this change will not have negative effects.

@nickaein

@nickaein @theodelrieu @gregmarr

Hi all, I finally had time to think about this issue and thing we should move forward with it.

As a first working draft, I created a branch https://github.com/nlohmann/json/tree/tests_external which loads the tests from a separate repository via

    include(FetchContent)
    FetchContent_Declare(nlohmann_json_tests
        GIT_REPOSITORY https://github.com/nlohmann/json_tests_tmp.git
        GIT_SHALLOW TRUE
    )

    FetchContent_GetProperties(nlohmann_json_tests)
    if(NOT nlohmann_json_tests_POPULATED)
        message(STATUS "Download tests")
        FetchContent_Populate(nlohmann_json_tests)
        message(STATUS "Download tests - done")
        add_subdirectory(${nlohmann_json_tests_SOURCE_DIR}/test)
    endif()

As a result, the tests folder can be removed from the repository, leaving a much smaller footprint.

However, I am not yet sure whether this is the best approach. In particular, I am afraid that tests and code get out of sync as a PR in the original repo cannot simply add tests on its own, but would need a PR in the tests repo first. Alternatively, I could only move the test files to the separate repo, but leave the JSON files in the separate repo.

Any ideas on this?

I think splitting code and tests into a separate repositories is a bad idea. Newer git versions have "sparse checkout" support, see https://www.git-scm.com/docs/git-sparse-checkout, so it would maybe possible to not checkout the tests folder.

For general users the various package managers should also help.

@nlohmann

@nlohmann
this repository already has a great but large history, IMHO this repository is just fine for contributors with all test stuff and ..., but the final users may only need a single header file, and any other thing means YAGNI.

it may be a better idea to have a separate repository for released versions of json.hpp, this approach is both easier for you and other contributors to work on and manage this repo and also it's easier for final users to fetch a light repository with a single header file.

BTW every solution has its own pros/cons

@nlohmann

@nlohmann
That's great! I'm not 100% sure about this solution yet, but seems be a good compromise.

Library and tests can get out of sync

I am afraid that tests and code get out of sync as a PR in the original repo cannot simply add tests on its own, but would need a PR in the tests repo first.

I agree. There is also the problem that after a while a user may want to checkout an older version of the library while the latest json_tests repo might not be backward-compatible with that version (e.g. a test file has been deprecated and removed in some commit).

Possible solution: We might have to (internally) tie the library version to a json_tests commit/tag. In FetchContent_Declare this can be done by using GIT_TAG. This can mitigate the out of sync problems as the PR author can explicitly point to a specific json_tests commit/tag (almost always the latest tag as the time of creating PR).

I could only move the test files to the separate repo, but leave the JSON files in the separate repo.

I believe you meant to keep the source code for tests here and only move the large JSON files into the separate repo.

I totally agree with this approach. The changes in the library and test source code are related in meaningful way and there is semantic to see them in single commit (or a single PR).

On the other hand, since the large JSON data for tests doesn't have much semantics and they rarely change, we can have them as an external repo.

Which one? test or benchmark data

There is also ~58 MBs worth of data for benchmarks. While they have smaller impact (compared to ~194 MBs of test data), we can move them if you want to be conservative.

In future though, both of them can be moved to an external repo.

Why?

Putting aside the technicalities, I believe we should consider the use-cases which make the users raise this issue. In some cases the user would be happy if along the single-header release, README, LICENSE, and CMakeList.txt (probably a stripped-down version) is also provided. For instance: #1066 and #1184.

Currently, they find the contents of releases inadequate and fall back to cloning the whole repo. This can be avoided if we add such files to the zip releases. I'm not sure if there are other considerations though.

@t-b

@t-b
Sparse checkout is an interesting solution! However, I wonder if it could be effective in the scenarios that people are complaining about repo size. The users can already use this solution, yet we still see complains. In addition, their hand might be tied in using this feature (e.g. in a automated environment like CI where they can only specify a URL to git repo).
And lastly, we still increase the size of .git directory in future whenever we add/modify test data.

Nevertheless, this solution can probably help some of the users. We can recommend this (as a temporary solution at least), until further notice.

@azadkuh

but the final users may only need a single header file, and any other thing means YAGNI.

There is already single header files available for such users at Releases page. Does providing them in a separate repo have any benefits?

Thanks for the comments!

Right now, I think it would be best to only move the test and benchmark data to a different repository and leave the actual code in the main repository. They are updated rather infrequently, and maybe one can even mark a subset of tests that can be run without external data - these tests can then still be executed without any download.
I am not sure whether the contents of the releases need to be adjusted. I think most complaints about the size came from people who just added the library as submodule and never cared about releases in the first place.

Hi all, please have a look at #2081 where I implemented the approach of having a separate test data repository.

Review round 2 (nlohmann#92)

Navigation Menu

Repository navigation

17 MB / 90 MB repo size!? #96

9 remaining items

Library and tests can get out of sync

Which one? test or benchmark data

Why?

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

17 MB / 90 MB repo size!? #96

Description

Activity

nlohmann commented on Jun 22, 2015

nanoant commented on Jun 22, 2015

nlohmann commented on Jun 22, 2015

nanoant commented on Jun 22, 2015

azadkuh commented on Sep 20, 2016

nlohmann commented on Oct 15, 2016

nlohmann commented on Feb 23, 2017

gregmarr commented on Feb 23, 2017

nanoant commented on Feb 23, 2017

nlohmann commented on Feb 23, 2017

fnc12 commented on Feb 23, 2017

tdegeus commented on Apr 15, 2017

therocode commented on Jul 5, 2017

9 remaining items

nickaein commented on Apr 28, 2019

nlohmann commented on Apr 29, 2020

t-b commented on Apr 29, 2020

azadkuh commented on Apr 29, 2020

nickaein commented on May 1, 2020

Library and tests can get out of sync

Which one? test or benchmark data

Why?

nickaein commented on May 1, 2020

nlohmann commented on May 1, 2020

nlohmann commented on May 3, 2020

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions