This error cannot be resolved when I measure boot time：

[lebron8dong]

https://github.com/firecracker-microvm/nsdi2020-data

This is the firecracker open source test script.

Excuting an order
cd ./prep
make
Report an error：

cd firecracker && tools/devtool -y build --release
[Firecracker devtool] Starting build (release, musl) ...
Updating crates.io index
warning: spurious network error (2 tries remaining): [7] Couldn't connect to server; class=Net (12)
warning: spurious network error (1 tries remaining): [7] Couldn't connect to server; class=Net (12)
error: failed to fetch https://github.com/rust-lang/crates.io-index

Caused by:
[7] Couldn't connect to server; class=Net (12)
make[1]: *** [Makefile:21: firecracker/build/cargo_target/x86_64-unknown-linux-musl/release/firecracker] Error 101
make[1]: Leaving directory '/home/aiot/nsdi2020-data/prep/firecracker'
make: *** [Makefile:4: all] Error 2

[lebron8dong]

I changed the mirror source but got the same error.

[lebron8dong]

How to measure the boot time of firecracker？

[JonathanWoollett-Light]

How to measure the boot time of firecracker？

To measure boot time with recent versions of Firecracker it may be useful to look at https://github.com/firecracker-microvm/firecracker/blob/main/tests/integration_tests/performance/test_boottime.py. If you can use these tests it is highly preferred to https://github.com/firecracker-microvm/nsdi2020-data.

[roypat]

Hi @lebron8dong,
As Jonathan mentioned, to measure the boottime of an up-to-date Firecracker binary, we recommend using https://github.com/firecracker-microvm/firecracker/blob/main/tests/integration_tests/performance/test_boottime.py. This can be run through our test orchestration via

AWS_EMF_ENVIRONMENT=local AWS_EMF_NAMESPACE=local tools/devtool -y test -- integration_tests/performance/test_boottime.py::test_boottime

which will print 10 boottime samples per test case (and this number can be easily adjusted in test_boottime.py itself).

However, if you do want to run the (almost) exact setup used in the paper from 2020, you can try applying the following diff to the nsdata-2020 repository, and then following the instruction in the README.md. With the below changes, I was able to re-run the old benchmarks. Please do keep in mind that those will be using Firecracker 0.20.0, which is 4 years old and will not contain any of the improvement done since then.

diff --git a/README.md b/README.md
index 735f363..017b9cc 100644
--- a/README.md
+++ b/README.md
@@ -35,8 +35,8 @@ sudo apt update
 sudo apt install -y \
     iperf3 \
     jq \
-    hwloc-nox
-    numactl \
+    hwloc-nox \
+    numactl
\```
 
 ## Running the tests
diff --git a/prep/img/Makefile b/prep/img/Makefile
index 7e2b639..7ce36ef 100644
--- a/prep/img/Makefile
+++ b/prep/img/Makefile
@@ -17,7 +17,7 @@ TARGETS += ../../img/bench-ssh-vmlinuz
 TARGETS += ../../img/bench-ssh-disk.img
 
 LK := ../../bin/linuxkit
-LOOP ?= /dev/loop0
+LOOP ?= /dev/loop21
 
 build: $(TARGETS)
 
diff --git a/prep/linuxkit/Makefile b/prep/linuxkit/Makefile
index da18cf7..327ab90 100644
--- a/prep/linuxkit/Makefile
+++ b/prep/linuxkit/Makefile
@@ -1,6 +1,6 @@
 # Quick hacky makefile to build linuxkit with docker
 
-URL := https://github.com/linuxkit/linuxkit/releases/download/v0.7/linuxkit-linux-amd64
+URL := https://github.com/linuxkit/linuxkit/releases/download/v1.0.0/linuxkit-linux-amd64
 
 ../../bin/linuxkit:
        curl -fsSL -o $@ $(URL)
diff --git a/prep/qemu/Dockerfile b/prep/qemu/Dockerfile
index 9f7f6f4..51f2c4e 100644
--- a/prep/qemu/Dockerfile
+++ b/prep/qemu/Dockerfile
@@ -11,7 +11,7 @@ RUN apt update && \
         python \
         libc6-dev-i386
 
-RUN git clone git://git.qemu.org/qemu.git && \
+RUN git clone https://github.com/qemu/qemu && \
     cd qemu && \
     git checkout v4.2.0

I'll go ahead and close this issue, but please feel free to reopen it if you have further questions.
Thanks,
Patrick