Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Copy closure progress: Size tracking and a bug fix #6

Closed
wants to merge 4 commits into from
Closed

Copy closure progress: Size tracking and a bug fix #6

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
65cf851
build-remote: CopyClosure::copyTo takes a new argument
shlevy Apr 13, 2012
6af2be4
nix-copy-closure: Have pv be size-aware when used
shlevy Apr 13, 2012
aba00bb
nix-copy-closure: Put the progressViewer after decompression always t…
shlevy Apr 13, 2012
a142501
nix-copy-closure: Don't hard-code the pv path when invoking remotely
shlevy Apr 17, 2012
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 3 additions & 2 deletions perl/lib/Nix/CopyClosure.pm
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,6 @@ sub copyTo {

$compressor = "$compressor |" if $compressor ne "";
$decompressor = "$decompressor |" if $decompressor ne "";
$progressViewer = "$progressViewer |" if $progressViewer ne "";

# Get the closure of this path.
my @closure = reverse(topoSortPaths(computeFSClosure(0, $includeOutputs,
Expand All @@ -35,7 +34,9 @@ sub copyTo {
if (scalar @missing > 0) {
print STDERR "copying ", scalar @missing, " missing paths to ‘$sshHost’...\n";
unless ($dryRun) {
open SSH, "| $compressor $progressViewer ssh $sshHost @{$sshOpts} '$decompressor nix-store --import' > /dev/null" or die;
my $size = `nix-store -q --size @missing | awk '{n+=\$1} END {printf n}'`;
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This introduces a dependency on awk and is also a bit inefficient. The summation can also easily be done in Perl, e.g. (not tested)

use List::Util;
my $size = sum (split ' ', `nix-store -q --size @missing`);

It's also possible to prevent a call to nix-store by using the queryPathInfo() function in Perl.

$progressViewer = "$progressViewer -s $size |" if $progressViewer ne "";
open SSH, "| $compressor ssh $sshHost @{$sshOpts} '$decompressor $progressViewer nix-store --import' > /dev/null" or die;
exportPaths(fileno(SSH), $sign, @missing);
close SSH or die "copying store paths to remote machine `$sshHost' failed: $?";
}
Expand Down
2 changes: 1 addition & 1 deletion scripts/build-remote.pl.in
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -248,7 +248,7 @@ if ($@) {
print STDERR "somebody is hogging $uploadLock, continuing...\n";
unlink $uploadLock;
}
Nix::CopyClosure::copyTo($hostName, [ @sshOpts ], [ $drvPath, @inputs ], "", "", 0, 0, $maybeSign ne "");
Nix::CopyClosure::copyTo($hostName, [ @sshOpts ], [ $drvPath, @inputs ], "", "", 0, 0, $maybeSign ne "", "");
close UPLOADLOCK;


Expand Down
9 changes: 6 additions & 3 deletions scripts/nix-copy-closure.in
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -63,7 +63,7 @@ while (@ARGV) {
$includeOutputs = 1;
}
elsif ($arg eq "--show-progress") {
$progressViewer = "@pv@";
$progressViewer = "-apebt";
}
elsif ($arg eq "--dry-run") {
$dryRun = 1;
Expand All @@ -81,10 +81,12 @@ openSSHConnection $sshHost or die "$0: unable to start SSH\n";


if ($toMode) { # Copy TO the remote machine.
$progressViewer = "pv $progressViewer" if $progressViewer ne "";
Nix::CopyClosure::copyTo($sshHost, [ @sshOpts ], [ @storePaths ], $compressor, $decompressor, $includeOutputs, $dryRun, $sign, $progressViewer);
}

else { # Copy FROM the remote machine.
$progressViewer = "@pv@ $progressViewer" if $progressViewer ne "";

# Query the closure of the given store paths on the remote
# machine. Paths are assumed to be store paths; there is no
Expand All @@ -106,10 +108,11 @@ else { # Copy FROM the remote machine.
print STDERR "copying ", scalar @missing, " missing paths from ‘$sshHost’...\n";
$compressor = "| $compressor" if $compressor ne "";
$decompressor = "$decompressor |" if $decompressor ne "";
$progressViewer = "$progressViewer |" if $progressViewer ne "";
my $size = `ssh $sshHost @sshOpts "nix-store -q --size $extraOpts @missing | awk '{n+=\\\$1} END {printf n}'"`;
$progressViewer = "$progressViewer -s $size |" if $progressViewer ne "";
unless ($dryRun) {
my $extraOpts = $sign ? "--sign" : "";
system("set -f; ssh $sshHost @sshOpts 'nix-store --export $extraOpts @missing $compressor' | $progressViewer $decompressor $Nix::Config::binDir/nix-store --import > /dev/null") == 0
system("set -f; ssh $sshHost @sshOpts 'nix-store --export $extraOpts @missing $compressor' | $decompressor $progressViewer $Nix::Config::binDir/nix-store --import > /dev/null") == 0
or die "copying store paths from remote machine `$sshHost' failed: $?";
}
}
Expand Down