nix-git

26 jul 2013

motivation

the checkout of nixpkgs usually takes very long but we can’t do anything about it since the command to compress the repository needs to be ran on the remote repository. as we were discussing this already, the checkout size could be cut by half, which is very good.

discussion

[1] metiones that:

The remote repo should be configured to run gc as needed after a commit is made. See the documentation of gc.auto in git-gc and git-config man pages.

However, a remote repo shouldn’t need all that much garbage collection, since it will rarely have dangling (unreachable) commits. These usually result from things like branch deletion and rebasing, which typically happen only in local repos.

So gc is needed more for repacking, which is for saving storage space rather than removing actual garbage. The gc.auto variable is sufficient for taking care of this.

[2] says: > From the git-gc man page: > > Users are encouraged to run this task on a regular basis within each repository to maintain good disk space utilization and good operating performance. > > Emphasis mine. Bare repositories are repositories too! > > Further explanation: one of the housekeeping tasks that git-gc performs is packing and repacking of loose objects. Even if you never have any dangling objects in your bare repository, you will – over time – accumulate lots of loose objects. These loose objects should periodically get packed, for efficiency. Similarly, if a large number of packs accumulate, they should periodically get repacked into larger (fewer) packs.

statistics

nixpkgs

joachim@lenovo-t530 /tmp/ % git clone git@github.com:NixOS/nixpkgs.git

joachim@lenovo-t530 /tmp/nixpkgs (git)-[master] % du -sh .
250M    .

joachim@lenovo-t530 /tmp/nixpkgs (git)-[master] % git gc --aggressive
Counting objects: 253187, done.
Delta compression using up to 8 threads.
Compressing objects: 100% (230102/230102), done.
Writing objects: 100% (253187/253187), done.
Total 253187 (delta 175779), reused 67451 (delta 0)
Checking connectivity: 253187, done.
git gc --aggressive  242.33s user 4.53s system 517% cpu 47.712 total

joachim@lenovo-t530 /tmp/nixpkgs (git)-[master] % du -sh .
139M    .

nixos

joachim@lenovo-t530 /tmp/ % git clone git@github.com:NixOS/nixos.git

joachim@lenovo-t530 /tmp/nixos (git)-[master] % du -sh .
15M     .

joachim@lenovo-t530 /tmp/nixos (git)-[master] % git gc --aggressive
Counting objects: 31750, done.
Delta compression using up to 8 threads.
Compressing objects: 100% (31432/31432), done.
Writing objects: 100% (31750/31750), done.
Total 31750 (delta 23940), reused 7545 (delta 0)
git gc --aggressive  14.27s user 0.54s system 536% cpu 2.762 total

joachim@lenovo-t530 /tmp/nixos (git)-[master] % du -sh
7.7M    .

nix

joachim@lenovo-t530 /tmp/ % git@github.com:NixOS/nix.git

joachim@lenovo-t530 /tmp/nix (git)-[master] % du -sh .
17M     .

joachim@lenovo-t530 /tmp/nix (git)-[master] % git gc --aggressive
Counting objects: 24178, done.
Delta compression using up to 8 threads.
Compressing objects: 100% (23901/23901), done.
Writing objects: 100% (24178/24178), done.
Total 24178 (delta 18644), reused 5345 (delta 0)
git gc --aggressive  17.91s user 0.45s system 524% cpu 3.497 total

joachim@lenovo-t530 /tmp/nix (git)-[master] % du -sh .
7.8M    .

what to do now?

with some luck someone at support@github.com will run: git gc –aggressive for nixpkgs.git from time to time just to keep compression at a good level.

article source