please add BSDTAR to be able to install ARCH distributions

[beta-tester]

hello,
i would like to be able to install an ARCH distribution via NOOBS to a RPi2.
currently a very outdated version of ARCH is available via NOOBS and only for an old RPi1.
for RPi2 there is no supported ARCH distribution there.
i started a discussion with a question, why it is not possible to use NOOBS to install a more recent ARCH distribution.
(https://www.raspberrypi.org/forums/viewtopic.php?p=822213#p822213)
it sounds, it is because the recent ARCH distributions makes use of extendend access permission flags that the normal TAR archive is not able to carry and restore. for that reason the ARCH distributions uses BSDTAR to backup and restore these kind of flags.

so, to be able to install an recent ARCH distribution via NOOBS it is necessary, that NOOBS uses BSDTAR for install an ARCH distribution.

please add BSDTAR to the installation pipeline or exchange the normal TAR with BSDTAR.

i and other users would be very happy, if it could be included into NOOBS... thanks

[NicoHood]

I dont like those +1 posts, but I agree on this. Arch for Pi2 would be great. Ubuntu Mate as well.

[procount]

OK, I'll bite and see if I can add it to my version of NOOBS (PINN).
I've added it as a package to buildroot and built it.
Can BSDTAR replace TAR for all other distros, or should it ONLY be used for Arch?

[lurch]

Can BSDTAR replace TAR for all other distros, or should it ONLY be used for Arch?

When I was corresponding with one of the ArchLinuxArm guys over email about this ages ago, he assured me that bsdtar was effectively a superset of gnutar, but I didn't have time to do the testing required.

I don't want to sound dismissive, but for the audience it's aimed at, stability of NOOBS is much more important than the ability to install less-popular OSes such as ArchLinux.

[procount]

Well, I've now built a version of NOOBS v1.5 and replaced tar with bsdtar.
If anyone wants to test it, I'll post it on my github, but I guess the latest Arch distro will have to be repackaged for NOOBS in order to test it and hosted somewhere.
Oh, and I guess I'll have to rename this test version NOOBS to avoid conflicts too.

[maxnet]

When I was corresponding with one of the ArchLinuxArm guys over email about this ages ago,
he assured me that bsdtar was effectively a superset of gnutar, but I didn't have time to do the testing required.

Think we are currently using Busybox's tar.

But note that standard GNU tar does have options to store extra attributes as well (e.g. --selinux --acls --xattrs).
If they need any file system flag so exotic that it would not be covered even if we had gnu tar, wouldn't it make more sense to let them use partition disk images?
(which are already supported by NOOBS)

[procount]

It's not as simple as just replacing tar with bsdtar. there are some extra considerations that need adding to the code so it might take me a bit longer, but I;m getting there....

[procount]

Using BSDTAR results in the following error after the tar file has been extracted (on ext4):
"./usr/bin/ping6: Cannot restore extended attributes on this file system,
./usr/bin/ping: (null)
./usr/bin/rlogin: (null)
./usr/bin/rsh: (null)
./usr/bin/rcp: (null)
bsdtar< Error exit delayed from previous errors"

[maxnet]

"./usr/bin/ping6: Cannot restore extended attributes on this file system,

In addition to enabling the libarchive, attr and acl buildroot packages, you may also need to modify the Linux kernel configuration to enable things like posix ACLs and security labels, if they are using those.

[procount]

I'm so glad there are linux gurus here to keep pointing me down the correct rabbit hole! :) I had no idea about those.

[procount]

@maxnet - you were right (of course). I've now got Arch untar'd, but it needs some more work to get the rfs loaded.

[procount]

And... it now loads properly. Just got to work out why i;ve got to adjust it manually

[NicoHood]

Maybe you can share the current state of your fixes? I cannot find the branch on your fork, so I guess you havent uploaded it yet?

[procount]

Look at my branch new_bsdtar_150. I think I pushed this version after I fixed everything so it should install ARCH direct from it's existing bsdtar file, but it needs some help from it's partition_setup.sh file which I've not pushed yet. I was going to do that when I got something else working......

[maxnet]

Look at my branch new_bsdtar_150

Any specific reason you do are using bsdtar for the ext4 partition, but not for FAT there?

[NicoHood]

Is this page up to date?
https://github.com/raspberrypi/noobs/wiki/NOOBS-partitioning-explained

It says stuff like "This is what the riscos-boot.bin file is for, and why the RISC OS 'root' partition is still stored as a raw partition and not as a tarball." But before it says that it creates an extended partition for each OS.

I tried to mount the first partition, but i could not find the right type. Am I doing anything wrong?

sudo mount /dev/sdc1 /mnt -t vfat

[procount]

@naxnet - No not really, except I wanted to introduce the minimum of changes. If you think bsdtar is compatible enough to completely replace 'tar' then I can easily change it. Although I think it requrers more changes as IIRC, I couldn't pipe gzip to bsdtar, but bsdtar can apparantly handle the decompression itself.

[procount]

@NicoHood - No, it is not up to date w.r.t. v1.5 where the partitioning scheme has changed slightly. I think the RISCOS stuff is unchanged, but the settings partition is now the first logical parition (5) in the extended partition(2), leaving primary partitions (3) and (4) available.

[procount]

@maxnet -Oh yes - I also forgot that if you use BSDTAR on FAT partition that it complains that it CAN'T set the extended attributes (cos it's FAT of course), so you then have to add extra cmdline flags to ignore those errors.

[lurch]

I've added a small note to https://github.com/raspberrypi/noobs/wiki/NOOBS-partitioning-explained to say that the partitioning scheme has changed slightly for NOOBS v1.5.x - I'll update it in more detail when I have more time.

But the first partition on a NOOBS SD card is always still FAT32, so I dunno why your mount command didn't work. If you're trying to do that on a Raspberry Pi itself, note that you'll need to use /dev/mmcblk0p1 instead of e.g. /dev/sdc1 like you would if using a USB card-reader on a PC.

[NicoHood]

I tried to install Arch for Pi2 now. I used your bsdtar branch for that.
I first downloaded the .tar.gz file from here:
http://archlinuxarm.org/platforms/armv7/broadcom/raspberry-pi-2

Since we need two separate files for boot and root in noobs I used a simple trick:
I opened the archive in the archive manager and moved the boot folder outside and removed it.
Since I am not expert with those archives (espacially with those special attributes) this was the simplest way to to it.
I then packed the boot folder similar to the ubuntu tutorial I made.

I also had to extract the gz and then pack it again with xz (otherwise it was not found by noobs).

gzip -d ArchLinuxARM-rpi-2-latest.tar.gz
ls ArchLinuxARM-rpi-2-latest.tar -l --block-size=1MB
xz -9 -e ArchLinuxARM-rpi-2-latest.tar

partition_setup.sh

The partition_setup.sh file should work like this. However the fstab entry does not have a root entry?

# 
# /etc/fstab: static file system information
#
# <file system> <dir>   <type>  <options>   <dump>  <pass>
/dev/mmcblk0p1  /boot   vfat    defaults        0       0

I then modified it like this:

#!/bin/sh

set -ex

if [ -z "$part1" ] || [ -z "$part2" ]; then
  printf "Error: missing environment variable part1 or part2\n" 1>&2
  exit 1
fi

mkdir -p /tmp/1 /tmp/2

mount "$part1" /tmp/1
mount "$part2" /tmp/2

sed /tmp/1/cmdline.txt -i -e "s|root=/dev/[^ ]*|root=${part2}|"
#sed /tmp/2/etc/fstab -i -e "s|^.* / |${part2}  / |"
sed /tmp/2/etc/fstab -i -e "s|^.* /boot |${part1}  /boot |"

umount /tmp/1
umount /tmp/2

os.json

Question: Is there a way to specify two logins? Because we also have the user alarm with password alarm.
What version and date should we add?

{
  "name": "Arch_Pi2",
  "version": "TODO",
  "release_date": "TODO",
  "kernel": "TODO",
  "description": "An Arch Linux port for ARM devices",
  "url": "http://archlinuxarm.org/platforms/armv7/broadcom/raspberry-pi-2",
  "username": "root",
  "password": "root",
  "supported_hex_revisions": "1040,1041",
  "feature_level": 0
}

partitions.json

Question: The old arch has "mkfs_options": "-c", option. Do we need this instead?

{
  "partitions": [
    {
      "label": "boot",
      "filesystem_type": "FAT",
      "partition_size_nominal": 100,
      "want_maximised": false,
      "uncompressed_tarball_size": 18
    },
    {
      "label": "ArchLinuxARM-rpi-2-latest",
      "filesystem_type": "ext4",
      "partition_size_nominal": 1500,
      "want_maximised": true,
      "mkfs_options": "-O ^huge_file",
      "uncompressed_tarball_size": 578
    }
  ]
}

I copied the logo and slides from the old 1.3.9 ARCH for Pi1.

I am currently running the installation to test. But I am also excited for you optimized script to just install the single .tar.gz file.

[NicoHood]

Okay my tutorial might not be optimal, but it works.
arch has kernel 4.1 and the slides from arch1 are very tiny. We should remove or update them. Nothing really important.

I suggest to add a PR and let @maxnet and @lurch have a look at your fix. Great work!

[procount]

I hope to put up my simplified script soon (will nake it easier) but I'm busy painting bedrooms atm!

[lurch]

However the fstab entry does not have a root entry?

Linux works fine without it, since the root partition is specified in the cmdline.txt. AFAIK having the root entry in fstab is more of a 'convenience' for the humans looking at the file?

Question: Is there a way to specify two logins? Because we also have the user alarm with password alarm.

Not at the moment. In fact those fields aren't actually used for anything - again they're more for human reference (and were originally included on https://www.raspberrypi.org/downloads/ back when it was auto-generated). Feel free to add new entries for root_username and root_password.
(At one time there was a plan for a NOOBS-based automated testing system, which would have used the username and password fields, but that plan kinda got shelved)

What version and date should we add?

The version can be whatever you want (another field that used to be on the auto-generated downloads page), and the date should be the date that the 'upstream' OS-files got built. If NOOBS is booted with an internet connection, the date field is used by NOOBS to determine whether the version of the OS available online is newer than the one present on the SD card (and if it is, it then network-installs the OS, ignoring the files on the SD-card).

Question: The old arch has "mkfs_options": "-c", option. Do we need this instead?

I'm no mkfs expert - that's a question for the ArchLinux developers.

IMHO "tiny" installation slides are better than no slides at all.

[procount]

I have put draft installation files for Arch (on Pi2) for use with my bsdtar branch on sourceforge at:
https://sourceforge.net/projects/pinn/files/OS-images/ArchPi2/.
Copy these files into the /os/Arch folder on your SD card along with the ArchLinuxArm-rpi-2-latest.tar.gz file from the Arch website. os.json and partitions.json need to be updated to reflect the latest version details. These could also be used as the basis for the Pi1 version too.

[maxnet]

No not really, except I wanted to introduce the minimum of changes. If you think bsdtar is compatible
enough to completely replace 'tar' then I can easily change it.

Have not tested compatibility.
However do think creating different behavior per file system is a more invasive change, then switching all to another single program (be it bsdtar or something else).

Linux works fine without it, since the root partition is specified in the cmdline.txt. AFAIK having the root
entry in fstab is more of a 'convenience' for the humans looking at the file?

With traditional Linux distributions /etc/fstab is more than just for humans, and is also used by fsck, and to specify the final mount options for the root file system
E.g. one could specify noatime there if you do not want that reads from storage cause writes as well to update file access time.

However mounting the root file system read-write straight away from cmdline.txt (instead of initially ro), and only recovering by journal, is indeed also possible.

[lurch]

With traditional Linux distributions /etc/fstab is more than just for humans, and is also used by fsck, and to specify the final mount options for the root file system

Aha, I suspected there might be more to it, thanks!

[procount]

@maxnet

However do think creating different behavior per file system is a more invasive change, then switching all to another single program (be it bsdtar or something else).

IIRC in my test I used bsdtar to decompress the existing Raspbian boot and root tar files. The ext4 root file was fine, but extracting the FAT boot file resulted in errors of not being able to set the extended attributes. So this means it still needs different behaviour for the filesystems even if you used bsdtar (or whatever) for all distros (to set the --no-owner etc flags on FAT to avoid the error), whereas these extended attributes are required for Arch.
I did not test whether recompressing Raspbian using bsdtar would avoid the problem, but even if it did, you would still need the workaround for compatibility when installing old distros.
Suggestions?

[maxnet]

IIRC in my test I used bsdtar to decompress the existing Raspbian boot and root tar files. The ext4 root file was fine, but extracting the FAT boot file resulted in errors of not being able to set the extended attributes.

There are plenty of other file systems commonly used on Linux that do not support extended attributes at all (e.g. nfs), so doubt it should complain about that.
However it could be about normal unix permissions (uid/gid) not being supported on FAT either, which could indeed be a problem as well.

[XECDesign]

bsdtar has been added