Use Shallow Submodule Init to Reduce Clone Times

[TShapinsky]

Reopening with new PR due to being unable to reopen #1603 (force push/rebase caused history mismatch).

Added --depth 1 to the git submodule update command in Makefile.in.

To test I setup a docker image which built the repository with ./configure --prefix=/opt/riscv --with-arch=rv32gc --with-abi=ilp32d. The image with this change was 14.77 GB where as the version without this change was 16.93 GB. That is a savings of 2.16 GB. This won't be a big deal for people developing in a stable environment where they only init the submodules once, but for cases where the build is happening in a container this is a big gain.

A different approach would be to populate the shallow property in .gitmodules. However, this approach only saves about 100MB, perhaps because the shallow property is not applied recursively.

[TShapinsky]

This can be removed. But right now it is hard to debug actions as all jobs exit when one fails.

[TShapinsky]

`-c "http.version=HTTP/1.1" forces http 1.1 to avoid error which was seen in previous PR.

[cmuellner]

Looks like the git server of uclibc-ng does not like shallow clones.

I've seen that shallow clones can be configured in .gitmodules.
Maybe something like this works (untested):

diff --git a/.gitmodules b/.gitmodules
index 68c2932..058d88a 100644
--- a/.gitmodules
+++ b/.gitmodules
@@ -2,44 +2,56 @@
        path = binutils
        url = https://sourceware.org/git/binutils-gdb.git
        branch = binutils-2_43-branch
+       shallow = true
 [submodule "gcc"]
        path = gcc
        url = https://gcc.gnu.org/git/gcc.git
        branch = releases/gcc-14
+       shallow = true
 [submodule "glibc"]
        path = glibc
        url = https://sourceware.org/git/glibc.git
+       shallow = true
 [submodule "dejagnu"]
        path = dejagnu
        url = https://git.savannah.gnu.org/git/dejagnu.git
        branch = master
+       shallow = true
 [submodule "newlib"]
        path = newlib
        url = https://sourceware.org/git/newlib-cygwin.git
        branch = master
+       shallow = true
 [submodule "gdb"]
        path = gdb
        url = https://sourceware.org/git/binutils-gdb.git
        branch = gdb-15-branch
+       shallow = true
 [submodule "qemu"]
        path = qemu
        url = https://gitlab.com/qemu-project/qemu.git
+       shallow = true
 [submodule "musl"]
        path = musl
        url = https://git.musl-libc.org/git/musl
        branch = master
+       shallow = true
 [submodule "spike"]
        path = spike
        url = https://github.com/riscv-software-src/riscv-isa-sim.git
        branch = master
+       shallow = true
 [submodule "pk"]
        path = pk
        url = https://github.com/riscv-software-src/riscv-pk.git
        branch = master
+       shallow = true
 [submodule "llvm"]
        path = llvm
        url = https://github.com/llvm/llvm-project.git
        branch = release/17.x
+       shallow = true
 [submodule "uclibc-ng"]
        path = uclibc-ng
        url = https://git.uclibc-ng.org/git/uclibc-ng.git
+       shallow = false

[TShapinsky]

Looks like the git server of uclibc-ng does not like shallow clones.

I've seen that shallow clones can be configured in .gitmodules. Maybe something like this works (untested):

diff --git a/.gitmodules b/.gitmodules
index 68c2932..058d88a 100644
--- a/.gitmodules
+++ b/.gitmodules
@@ -2,44 +2,56 @@
        path = binutils
        url = https://sourceware.org/git/binutils-gdb.git
        branch = binutils-2_43-branch
+       shallow = true
 [submodule "gcc"]
        path = gcc
        url = https://gcc.gnu.org/git/gcc.git
        branch = releases/gcc-14
+       shallow = true
 [submodule "glibc"]
        path = glibc
        url = https://sourceware.org/git/glibc.git
+       shallow = true
 [submodule "dejagnu"]
        path = dejagnu
        url = https://git.savannah.gnu.org/git/dejagnu.git
        branch = master
+       shallow = true
 [submodule "newlib"]
        path = newlib
        url = https://sourceware.org/git/newlib-cygwin.git
        branch = master
+       shallow = true
 [submodule "gdb"]
        path = gdb
        url = https://sourceware.org/git/binutils-gdb.git
        branch = gdb-15-branch
+       shallow = true
 [submodule "qemu"]
        path = qemu
        url = https://gitlab.com/qemu-project/qemu.git
+       shallow = true
 [submodule "musl"]
        path = musl
        url = https://git.musl-libc.org/git/musl
        branch = master
+       shallow = true
 [submodule "spike"]
        path = spike
        url = https://github.com/riscv-software-src/riscv-isa-sim.git
        branch = master
+       shallow = true
 [submodule "pk"]
        path = pk
        url = https://github.com/riscv-software-src/riscv-pk.git
        branch = master
+       shallow = true
 [submodule "llvm"]
        path = llvm
        url = https://github.com/llvm/llvm-project.git
        branch = release/17.x
+       shallow = true
 [submodule "uclibc-ng"]
        path = uclibc-ng
        url = https://git.uclibc-ng.org/git/uclibc-ng.git
+       shallow = false

Yeah, I'm testing this out locally. I can't get it to take... but hopefully I'll figure it out soon.

It would definitely be preferable to do it this way instead of forcing it.

[karaketir16]

I did not notice this pull request.

I created #1616, I hope it helps.

[TommyMurphyTM1234]

I think that this merged PR:

• reduce submodule clone time and size using depth option #1616

renders this one moot and, as such, it can probably be closed?

Edit: oh, sorry, that one wasn't merged. But this one was and, among other things, implements the shallow clone for all submodules:

• Update build workflow #1608
  • efc7cc9