Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

zpool: pool I/O is currently suspended #104

Closed
kenorb opened this issue Dec 11, 2013 · 38 comments
Closed

zpool: pool I/O is currently suspended #104

kenorb opened this issue Dec 11, 2013 · 38 comments

Comments

@kenorb
Copy link

kenorb commented Dec 11, 2013

I've the following errors when running the following commands:

$ sudo zfs mount WD_1TB
cannot open 'WD_1TB': pool I/O is currently suspended
cannot open 'WD_1TB': pool I/O is currently suspended

$ sudo zfs unmount WD_1TB
cannot open 'WD_1TB': pool I/O is currently suspended
cannot open 'WD_1TB': pool I/O is currently suspended

$ sudo zpool destroy -f WD_1TB
cannot open 'WD_1TB': pool I/O is currently suspended

$ zpool export WD_1TB
(hanging)

$ sudo zpool clear WD_1TB
cannot clear errors for WD_1TB: I/O error

My zpool WD_1TB is active and ONLINE
I'm using WD HDD via USB.

More description about the problem here:
http://superuser.com/questions/685882/zpool-pool-i-o-is-currently-suspended

Restart helps, but only temporary, as it is happening quite often.

I think this is happening and the same HDD is phisically re-connected and the device name is changed (from disk1 to disk3).
What are the methods of re-initializing that zpool again with the correct references?
I think I already tried everything.

@ilovezfs
Copy link
Contributor

This is the EAGAIN error:
https://github.com/zfs-osx/zfs/blob/master/lib/libzfs/libzfs_util.c#L389
https://github.com/zfs-osx/zfs/blob/master/lib/libzfs/libzfs_util.c#L468

To debug this, you need to add printf statements everywhere in the code where EAGAIN is returned so we can find out where this comes from.

@ilovezfs
Copy link
Contributor

As for devfs device numbers' changing being related, that doesn't seem likely because I can import and export pools until the cows come home while changing the device numbers every time. What evidence do you have that it has anything to do with the numbering?

@ilovezfs
Copy link
Contributor

I see in IRC that the disk was actually disconnected and reconnected while the pool was imported. Given that this is a single partition pool, not a raidz or mirror vdev, there is no reason to expect the pool to continue to function after the device has been disconnected and reconnected without exporting it first. At that point, your only choice is to reboot. Your underlying issue here apparently has nothing to do with ZFS. Why is the device being disconnected in the first place? Are you disconnecting the cable yourself? Or perhaps your USB cable or controller is defective? Is there a third party driver installed for the drive? Or perhaps this is just another example of why ZFS should not be run over USB: http://code.google.com/p/maczfs/wiki/USB

@lundman
Copy link
Contributor

lundman commented Dec 11, 2013

I'd be curious to what happens if you zpool set failmode=continue on the pool. it wont fix your problem at all, but just curious

@ilovezfs
Copy link
Contributor

If the output of zpool status shows a different device number from the current number assigned to the drive after it has been physically reconnected, and yet the pool still says it's "online," I wonder whether the pool will continue to say it's "online" if you physically disconnect the disk again.

In IRC you said zpool status said "online" and reported the vdev to be at disk1s2, and yet the disk was reconnected and assigned to disk3. At that point, please physically disconnect the drive again, and recheck the output of zpool status. Does it continue to say "online" at that point?

@kenorb
Copy link
Author

kenorb commented Dec 11, 2013

The problem why it was reconnected, because I was closing my laptop (into sleep mode) and I had to disconnect the external HDD. Before that I've ejected the HDD as usual (after few minutes of waiting preventing my shutdown, I had to do umount process with force). Then when recovered from the sleep mode, it started to do these things.
This zpool was online for some time, but when I was forcing all destruction commands to get rid of that zpool, now actually it has status of UNAVAIL:

NAME        STATE     READ WRITE CKSUM
WD_1TB      UNAVAIL      0     0     0  insufficient replicas
  disk1s2   UNAVAIL      0     0     0  cannot open

And again, I can't do anything. Neither import/export or destroy.

I'd be curious to what happens if you zpool set failmode=continue on the pool. it wont fix your problem at all, but just curious

$ sudo zpool set failmode=continue WD_1TB
cannot set property for 'WD_1TB': pool I/O is currently suspended

I wonder whether the pool will continue to say it's "online" if you physically disconnect the disk again.

Usually when I've disconnected the HDD (after already having the problems), it was still saying online, even the HDD wasn't connected at all.

P.S. The HDD is brand new, so there are no any hardware issues and my OSX is quite fresh.

@ilovezfs
Copy link
Contributor

@lundman meant to set the failmode=continue in advance.

"This zpool was online for some time." How long? Without your doing anything, does it eventually fault the pool, as it should?

"Before that I've ejected the HDD as usual (after few minutes of waiting preventing my shutdown, I had to do umount process with force)." If you disconnect the drive without having exported the pool, you are going to have to reboot. The pool will still be imported, though should be faulted, if not immediately, then after some timeout period or after some I/O error threshold count has been reached. If you attempt to reimport it, that is supposed to fail because you would be trying to import a pool with the same GUID as the pool that is already imported and faulted.

Merely reconnecting the drive is not going to un-suspend the pool that had the vdev rug pulled out from under it.

@kenorb
Copy link
Author

kenorb commented Dec 11, 2013

Usually I don't reboot, as I've too many things open and it takes me half an hour to recover. Eventually over the weekend, when the OSX is sluggish or I hit the kernel crash.
Unfortunatelly I've a lot of kernel crashes everyday once I started playing with zfs.

I had one few minutes ago:

Wed Dec 11 12:32:07 2013
panic(cpu 6 caller 0xffffff7f831b9b8c): "VERIFY0(" "sm->sm_space" ") failed\n"@space_map.c:96
Backtrace (CPU 6), Frame : Return Address
0xffffff81ea2fb880 : 0xffffff8002822f69 
...
0xffffff81ea2fbfb0 : 0xffffff80028f35e6 
  Kernel Extensions in backtrace:
     net.lundman.zfs(1.0)[14F42D8F-A41D-3F76-8432-23C4B3F98AFE]@0xffffff7f8313a000->0xffffff7f83332fff
        dependency: com.apple.iokit.IOStorageFamily(1.9)[9B09B065-7F11-3241-B194-B72E5C23548B]@0xffffff7f82df4000
        dependency: net.lundman.spl(1.0.0)[8B9174E1-DBF8-30A9-8D06-F8541ECDDE5D]@0xffffff7f8312d000

when I've tried to unload zfs extension:

sudo kextunload -b net.lundman.zfs

So currently my zpools are working after crash and I'll be monitoring the problem for the next days.

Is there anything that I've to do extra apart of ejecting the HDD using Finder for the next time?

@ilovezfs
Copy link
Contributor

You will always panic if you try to unload the kext while a pool is imported.

@ilovezfs
Copy link
Contributor

No idea what panic you're hitting there, but it smells like pool corruption. Also, please run sudo nvram boot-args="-v keepsyms=y" and reboot so that you will have panic reports that are actually useful.

You are simply going to have to reboot whenever you disconnect a drive from an imported pool. If you are having trouble exporting the pool, the correct thing to do is to shutdown, wait for the machine to finally "hang" and then force power off, and power back on. I cannot help with the 30 minutes to resume thing other than suggesting that you start using the app resume feature in OS X if you've disabled it.

@ilovezfs
Copy link
Contributor

@kenorb OK I found a workaround for you. If you symlink the new devfs entries to where zfs thinks the vdev is, you can regain access to the pool without a reboot.

So if zpool status says the vdev is /dev/disk2s1, but the reattached drive is at disk4, then do the following:

cd /dev
sudo rm -f disk2s1
sudo ln -s disk4s1 disk2s1
sudo zpool clear -F WD_1TB
sudo zpool export WD_1TB
sudo rm disk2s1
sudo zpool import WD_1TB

@ilovezfs
Copy link
Contributor

@ilovezfs
Copy link
Contributor

I verified that ZFS on Linux handles things the same way we do. However, they typically don't face the issue because most of the time users are NOT using the /dev/sda, /dev/sdb, etc. entries. If you detach and reattach a device /dev/sde on ZOL it will become /dev/sdf and ZOL will not automatically find it, and will fault the pool. It appears that the default timeout is 2 minutes before it will decide to fault it. Symlinking /dev/sdf to /dev/sde does allow ZFS to recover just as it does for us.

So @ryao suggested we could use a UUID on the label.

I think we're essentially trying to find our own answer to this question http://zfsonlinux.org/faq.html#WhatDevNamesShouldIUseWhenCreatingMyPool

@lundman We could use the DAMediaUUID, but that's just the GPT partition UUID, except in the case of Core Storage in which case it is the Core Storage UUID for the Logical Volume. But since ZFS does not have to be on a partitioned disk, we could also consider the DAMediaPath. However, that will change if the port slot used changes or if a disk image is detached and reattached. Or we could make a UUID up ourselves. Maybe some combination of the pool guid and the location in the vdev tree. But then what happens in the case of a copied disk image, where both the image and its copy are attached. How do we handle collisions? I guess we probably want /var/run/zfs/by-media-uuid, /var/run/zfs/by-media-path, and /var/run/zfsdev-uuid? Would we want to allow users to create pools using the DAMediaPath as ZOL does?

Or maybe we want to put a temporary per-device-per-session cookie/uuid on the label of each zfs device every time the pool is imported?

@kenorb
Copy link
Author

kenorb commented Dec 12, 2013

Thank you.
Before I'll continue reading workarounds, I'd like to note that it happened again yesterday, so maybe it could be helpful.
I've finished my work and I've tried to unmount the disk by ejecting it via Finder. After 5 minutes of awaiting (freeze), I've tried 'Force Eject...' few times. The disk was unmounted from file-system (mountpoint disappeared), but the popup with 'Cancel' and 'Force Eject...' grayed out was still there (even till now).

There was one error in dmesg like:

ZFS: Reclaim thread appears dead (20001) -- good luck

Today, restored from sleep mode, I've my hdd online (whatever it's connected or not).
When I'm trying to mount it, it says:

$ sudo zfs mount WD_1TB
cannot open 'WD_1TB': pool I/O is currently suspended
cannot open 'WD_1TB': pool I/O is currently suspended

(the message is repeated twice), so it's exactly the same situation as stated in the 1st place.

I'll follow now your suggestions.

@ilovezfs
Copy link
Contributor

As I said, the timeout period is 2 minutes so whether "it's connected or not" is normal.

You should do your mounting and unmounting from Terminal, using zfs mount / zfs unmount or zpool import / zpool export.

As I explained above, none of this is a bug but is entirely normal behavior given that we are not using a UUID for the device that would allow ZFS to find it after you've disconnected and reconnected the drive, changing its BSD name.

Given that this is an enhancement request for functionality not present in upstream ZFS, you should not expect that anything will change any time soon with respect to this behavior.

@kenorb
Copy link
Author

kenorb commented Dec 12, 2013

I don't think zpool status change from online after 2 minutes, it actually doesn't change at all.

$ zpool list -v # it's disk3 in diskutil list
NAME   SIZE  ALLOC   FREE    CAP  DEDUP  HEALTH  ALTROOT
WD_1TB   931G   433G   498G    46%  1.00x  ONLINE  -
  disk1s2   931G   433G   498G     16.0E
$ cd /dev
$ sudo mv disk1s2 disk1s2.bak # backup old dev
$ sudo ln -s disk3s2 disk1s2 # link existing one to old one
$ sudo zpool clear WD_1TB
$ sudo zfs mount WD_1TB
cannot mount '/WD_1TB': directory is not empty
cannot open 'WD_1TB': operation not applicable to datasets of this type

Despite of errors and annoying popup, it actually worked and I've my file system mounted again, great.
Thank you.

@kenorb
Copy link
Author

kenorb commented Dec 12, 2013

I'll try to work more on umounting process to see if that can be more efficient and see if that will happen again. But it seems zfs is struggling with umounting the file system.
And I think it's a good idea to mount the disk via uuid, not by the device name which could change.

@ilovezfs
Copy link
Contributor

I was able to observe it change status. It's a 2 minute timeout, so I think something actually has to try to access it.

Please create your issues here and not on stackexchange. If you notice, that issue is already here: #102. I can explain the trick to suppress it that I showed @alanruttenberg if you'd like on IRC. He'd said he'd write it up but I guess hasn't gotten around to it. https://github.com/zfs-osx/zfs/wiki/OSX_complains_ZFS_disk_unreadable_at_startup.

As for "cannot mount '/WD_1TB': directory is not empty," that means exactly what it says. That directory already existed and had files in it at the time you tried to mount. You would need to trash it or rm -rf it first before mounting.

@alanruttenberg
Copy link

It's half written up, with the transcript still there for the rest. Page
name was wrong on the front page (case, "_" versus "-") sigh.

https://github.com/zfs-osx/zfs/wiki/Osx-complains-zfs-disk-unreadable-at-startup

I fixed the link on the front page of the wiki.

-Alan

On Thu, Dec 12, 2013 at 8:01 AM, ilovezfs [email protected] wrote:

I was able to observe it change status. It's a 2 minute timeout, so I
think something actually has to try to access it.

Please create your issues here and not on stackexchange. If you notice,
that issue is already here: #102#102.
I can explain the trick to suppress it that I showed @alanruttenberghttps://github.com/alanruttenbergif you'd like on IRC. He'd said he'd write it up but I guess hasn't gotten
around to it.
https://github.com/zfs-osx/zfs/wiki/OSX_complains_ZFS_disk_unreadable_at_startup
.

As for "cannot mount '/WD_1TB': directory is not empty," that means
exactly what it says. That directory already existed and had files in it at
the time you tried to mount. You would need to trash it or rm -rf it first
before mounting.


Reply to this email directly or view it on GitHubhttps://github.com//issues/104#issuecomment-30418499
.

@ilovezfs
Copy link
Contributor

@alanruttenberg Thank you. Please post this in #102 also.

@alanruttenberg
Copy link

Ok, finished the write up and added the note in #102. I didn't close the
issue in case you wanted to add a comment.

Thanks a bunch!

-Alan

On Thu, Dec 12, 2013 at 8:35 AM, ilovezfs [email protected] wrote:

@alanruttenberg https://github.com/alanruttenberg Thank you. Please
post this in #102 #102 also,


Reply to this email directly or view it on GitHubhttps://github.com//issues/104#issuecomment-30420634
.

@ilovezfs
Copy link
Contributor

@alanruttenberg great, thank you. @kenorb please take a look at @alanruttenberg's work.

@lundman
Copy link
Contributor

lundman commented Dec 16, 2013

Just a quick comment on "ZFS: Reclaim thread appears dead (20001) -- good luck".

This can be ignored. There was a case (our condvar were/are broken) where the reclaim thread would actually die, so I made up some arbitrary high counter to indicate it has (20000), but it really is amazing how many reclaim nodes we can have in 500ms.

@kenorb
Copy link
Author

kenorb commented Dec 19, 2013

If the I/O is suspended, is there any way of unsuspend it?
Is there any way of ignoring 'pool I/O is currently suspended' by destroying the zpool by force? The current force option doesn't work as it should. Where is the real blocker? Why other file systems doesn't have similar problems, just only zfs? It's wrong by design by anyway?

$ zpool destroy -f WD_1TB
cannot open 'WD_1TB': pool I/O is currently suspended

@ilovezfs
Copy link
Contributor

The pool's I/O is suspended because ZFS is not seeing your disk there at all. If you have used the symlink trick to point ZFS to the new location of the disk you disconnected and reconnected, then you can issue zpool clear -F WD_1TB. If it still does not see the disk it will continue to tell you the I/O is suspended.

@kenorb
Copy link
Author

kenorb commented Dec 19, 2013

If the device doesn't exist any more, can the force option with destroy/detach/remove/clear command ignore the wrong access to the device node and remove the zpool, instead of returning the I/O error? Is there any code fix that can be applied to that logic?
I'm rather after proper long-term fixes, than the constant workarounds.

Some dtrace trace:

 read_nocancel(0x3, "Index Refs Address            Size       Wired      Name (Version) <Linked Against>\n    1   82 0xffffff7f807b8000 0x8d80     0x8d80     com.apple.kpi.bsd (13.0.2)\n    2    6 0xffffff7f80d36000 0x28c0     0x28c0     com.apple.kpi.dsep (13.0.2)\n    3  101 0x", 0x4000)       = 12565 0
 close_nocancel(0x3)         = 0 0
 wait4_nocancel(0x1282, 0x7FFF5D70C944, 0x0)         = 4738 0
 open("/dev/zfs\0", 0x2, 0x7FA8AA800001)         = 3 0
 open_nocancel("/etc/mtab\0", 0x0, 0x1B6)        = 4 0
 open("/dev/zfs\0", 0x2, 0x0)        = 5 0
 fstat64(0x4, 0x7FFF5D70C7C8, 0x7FFF5D70C88C)        = 0 0
 lseek(0x4, 0x0, 0x1)        = 0 0
 lseek(0x4, 0x0, 0x0)        = 0 0
 lseek(0x4, 0x0, 0x1)        = 0 0
 getfsstat64(0x0, 0x0, 0x1)      = 5 0
 getfsstat64(0x7FA8AA801A00, 0x54B0, 0x1)        = 5 0
 lseek(0x4, 0x1, 0x1)        = 1 0
 lseek(0x4, 0x0, 0x1)        = 1 0
 lseek(0x4, 0x1, 0x1)        = 2 0
 lseek(0x4, 0x0, 0x1)        = 2 0
 lseek(0x4, 0x1, 0x1)        = 3 0
 lseek(0x4, 0x0, 0x1)        = 3 0
 lseek(0x4, 0x1, 0x1)        = 4 0
 lseek(0x4, 0x0, 0x1)        = 4 0
 lseek(0x4, 0x1, 0x1)        = 5 0
 lseek(0x4, 0x0, 0x1)        = 5 0
 ioctl(0x3, 0xFFFFFFFFD1B05A12, 0x7FFF5D70B2F0)      = 0 0
 write_nocancel(0x2, "cannot open 'WD_1TB': pool I/O is currently suspended\n\0", 0x36)      = 54 0

@ilovezfs
Copy link
Contributor

No it cannot, and that would not make sense in general because most of the time ZFS is not used on a single disk, so the pool would remain imported and addressable, with the status of the unavailable disk reported alongside the status of the rest of the disks.

@kenorb
Copy link
Author

kenorb commented Dec 19, 2013

How about removing the entry in zpool?

$ zpool remove WD_1TB /dev/disk2s2
cannot remove /dev/disk2s2: only inactive hot spares, cache, top-level, or log devices can be removed

Or set/change/rename the device node of that zpool?

@ilovezfs
Copy link
Contributor

Nope. zpool remove is for something else altogether.

The solution to this issue, if we decide to pursue the enhancement, will be along these lines, as I said above:
#104 (comment)

@dxlr8r
Copy link

dxlr8r commented Feb 12, 2014

What about an external RAID connected to the Mac through USB? Same problem?

@kenorb
Copy link
Author

kenorb commented Feb 12, 2014

You can't use any of USB storage devices if you'll be re-connecting them or suspending your machine. You can consider to use ZFS-OSX storage only if you're using your OSX as a server, so you never disconnect your HDDs, otherwise you'll have to Power Off your machine each time in the hard way and eventually it'll die. I lost 2 HDDs in this way and my OSX was crashing every day, so it's up to you to experiment and comment your feedback on a new "enhancement", so it may go further. Personally I wouldn't recommend it.

@ilovezfs
Copy link
Contributor

@dxlr8r what problem are you referring to?

@dxlr8r
Copy link

dxlr8r commented Feb 12, 2014

I won't take the chance, I use the Mac Mini as a server, but something could happen and it I won't risk the data. What about FW2 (800Mbit ain't to bad, the NIC is only 1000Mbit anyway)? Or maybe even eSATA (there are ways to get thunderbolt -> eSATA).

@ilovezfs , http://code.google.com/p/maczfs/wiki/USB the troubles concerning using USB drives and ZFS.

@ilovezfs
Copy link
Contributor

@dxlr8r Those are problems that affect ZFS+USB on every platform. Yes direct attached SATA, eSATA, Firewire, Thunderbolt, and Thunderbolt2 are better choices.

@dxlr8r
Copy link

dxlr8r commented Feb 12, 2014

ok, thanks :)

@ilovezfs
Copy link
Contributor

Perhaps we can consider some of the enhancements I proposed above now that we have zed.

@ilovezfs
Copy link
Contributor

Closing in favor of #167

lyz-code added a commit to lyz-code/blue-book that referenced this issue Aug 15, 2023
- [Fast castle boom](https://www.youtube.com/watch?v=JsTNM7j6fs4&t=119)
- How to play Arena: [Hera's guide](https://piped.video/watch?v=8gXI4XGMPzQ&t=0), [Tatoh game in arena](https://www.youtube.com/watch?v=3qg4Xwm8CAo&t=1211)
- [How to play Hideout](https://www.youtube.com/watch?v=DdK8QveBegw&t=652)
- [How to play Blackforest](https://www.youtube.com/watch?v=1V_jsU9PF8Y)
- Inside the mind of a pro player: [Episode 1](https://www.youtube.com/watch?v=54hRmrdzO-I), [Episode 2](https://www.youtube.com/watch?v=sZCs6dwH5qk&t=1727)

feat(age_of_empires# Strategies against civilisations): Strategies against civilisations

I'm using only the mongols, and so far I've seen/heard from the pros the next strategies:

- Aztecs:
    - Steppe lancers good against eagle warriors
    - Heavy scorpions against eagle warriors and skirms
- Cumans:
    - [Scout, if it drops two TCs in feudal, tower rush into archers](https://www.youtube.com/watch?v=H9QUNtFII1g&t=0)
    - [Put initial pressure](https://www.youtube.com/watch?v=R9qaFZzZgBY&t=1925): Nice initial pressure
- Incas:
    - Steppe lancers good against eagle warriors
    - Heavy scorpions against eagle warriors and skirms
- Khmer: boom, map control, monks and albadiers
- Mayans:
    - Steppe lancers good against eagle warriors
    - Heavy scorpions against eagle warriors and skirms
- Romans:
    - [Hera guide on how to beat them](https://www.youtube.com/watch?v=SA44-Y3XUy0&t=842)
- Tartars: heavy scorpions
- Turks:
    - [How to defend against them in Arena](https://www.youtube.com/watch?v=AI_JRA_nCpw&t=3710)

feat(age_of_empires#Nice Games): Nice games

Tournaments:

- 2023 Masters of Arena 7 Final Tatoh vs Vinchester:
    - [Casted by T90](https://www.youtube.com/watch?v=3qg4Xwm8CAo&t=1211s)
    - [Pov by Tatoh](https://www.youtube.com/watch?v=AI_JRA_nCpw&t=8854)

Showmatches:

- [Hera vs TheViper | Battlegrounds 3 | BO5](https://www.youtube.com/watch?v=AlKMRQNMVzo&t=4306)
- [The Viper VS Tatoh PA7](https://www.youtube.com/watch?v=5_p3TXasBHY&t=5319)

1vs1 games:

- [Hindustanis vs Portuguese | Arabia | Hera vs Yo](https://www.youtube.com/watch?v=iZ7eWLLbh34)
- [Dravidians vs Turks | African Clearing | Hera vs Yo](https://www.youtube.com/watch?v=tZyVLDwBfd4)

feat(ansible_snippets#Run command on a working directory): Run command on a working directory

```yaml
- name: Change the working directory to somedir/ and run the command as db_owner
  ansible.builtin.command: /usr/bin/make_database.sh db_user db_name
  become: yes
  become_user: db_owner
  args:
    chdir: somedir/
    creates: /path/to/database
```

feat(ansible_snippets#Run handlers in the middle of the tasks file): Run handlers in the middle of the tasks file

If you need handlers to run before the end of the play, add a task to flush them using the [meta module](https://docs.ansible.com/ansible/latest/collections/ansible/builtin/meta_module.html#meta-module), which executes Ansible actions:

```yaml
tasks:
  - name: Some tasks go here
    ansible.builtin.shell: ...

  - name: Flush handlers
    meta: flush_handlers

  - name: Some other tasks
    ansible.builtin.shell: ...
```

The `meta: flush_handlers` task triggers any handlers that have been notified at that point in the play.

Once handlers are executed, either automatically after each mentioned section or manually by the `flush_handlers meta` task, they can be notified and run again in later sections of the play.

feat(ansible_snippets#Run command idempotently): Run command idempotently

```yaml
- name: Register the runner in gitea
  become: true
  command: act_runner register --config config.yaml --no-interactive --instance {{ gitea_url }} --token {{ gitea_docker_runner_token }}
  args:
    creates: /var/lib/gitea_docker_runner/.runner
```

feat(ansible_snippets#Get the correct architecture string): Get the correct architecture string

If you have an `amd64` host you'll get `x86_64`, but sometimes you need the `amd64` string. On those cases you can use the next snippet:

```yaml
---
deb_architecture:
  aarch64: arm64
  x86_64: amd64

---
- name: Download the act runner binary
  become: True
  ansible.builtin.get_url:
    url: https://dl.gitea.com/act_runner/act_runner-linux-{{ deb_architecture[ansible_architecture] }}
    dest: /usr/bin/act_runner
    mode: '0755'
```

feat(ansible_snippets#Check the instances that are going to be affected by playbook run): Check the instances that are going to be affected by playbook run

Useful to list the instances of a dynamic inventory

```bash
ansible-inventory -i aws_ec2.yaml --list
```

feat(ansible_snippets#Check if variable is defined or empty): Check if variable is defined or empty

In Ansible playbooks, it is often a good practice to test if a variable exists and what is its value.

Particularity this helps to avoid different “VARIABLE IS NOT DEFINED” errors in Ansible playbooks.

In this context there are several useful tests that you can apply using [Jinja2 filters](https://docs.ansible.com/ansible/latest/user_guide/playbooks_filters.html) in Ansible.

feat(ansible_snippets): Check if Ansible variable is defined (exists)

```yaml
tasks:

- shell: echo "The variable 'foo' is defined: '{{ foo }}'"
  when: foo is defined

- fail: msg="The variable 'bar' is not defined"
  when: bar is undefined
```

feat(ansible_snippets#Check if Ansible variable is empty): Check if Ansible variable is empty

```yaml
tasks:

- fail: msg="The variable 'bar' is empty"
  when: bar|length == 0

- shell: echo "The variable 'foo' is not empty: '{{ foo }}'"
  when: foo|length > 0
```

feat(ansible_snippets#Check if Ansible variable is defined and not empty): Check if Ansible variable is defined and not empty

```yaml
tasks:

- shell: echo "The variable 'foo' is defined and not empty"
  when: (foo is defined) and (foo|length > 0)

- fail: msg="The variable 'bar' is not defined or empty"
  when: (bar is not defined) or (bar|length == 0)
```

feat(ansible_snippets#Download a file): Download a file

```yaml
- name: Download foo.conf
  ansible.builtin.get_url:
    url: http://example.com/path/file.conf
    dest: /etc/foo.conf
    mode: '0440'
```

feat(authentik#Monitorization): Disregard monitorization

I've skimmed through the prometheus metrics exposed at `:9300/metrics` in the core and they aren't that useful :(

feat(bash_snippets#Get the root path of a git repository): Get the root path of a git repository

```bash
git rev-parse --show-toplevel
```

feat(bash_snippets#Get epoch gmt time): Get epoch gmt time

```bash
date -u '+%s'
```

feat(bash_snippets#Check the length of an array with jq): Check the length of an array with jq

```
echo '[{"username":"user1"},{"username":"user2"}]' | jq '. | length'
```

feat(bash_snippets#Exit the script if there is an error): Exit the script if there is an error

```bash
set -eu
```

feat(bash_snippets#Prompt the user for data): Prompt the user for data

```bash
read -p "Ask whatever" choice
```

feat(beets): Guide on how to start using it

You’ll want to set a few basic options before you start using beets. The [configuration](https://beets.readthedocs.io/en/stable/reference/config.html) is stored in a text file. You can show its location by running `beet config -p`, though it may not exist yet. Run `beet config -e` to edit the configuration in your favorite text editor. The file will start out empty, but here’s good place to start:

```yaml
directory: ~/music

library: ~/data/musiclibrary.db
```

The default configuration assumes you want to start a new organized music folder (that directory above) and that you’ll copy cleaned-up music into that empty folder using beets’ `import` command. But you can configure beets to behave many other ways:

- Start with a new empty directory, but move new music in instead of copying it (saving disk space). Put this in your config file:

    ```yaml
    import:
        move: yes
    ```

- Keep your current directory structure; importing should never move or copy files but instead just correct the tags on music. Put the line `copy: no` under the `import:` heading in your config file to disable any copying or renaming. Make sure to point `directory` at the place where your music is currently stored.

- Keep your current directory structure and do not correct files’ tags: leave files completely unmodified on your disk. (Corrected tags will still be stored in beets’ database, and you can use them to do renaming or tag changes later.) Put this in your config file:

    ```yaml
    import:
        copy: no
        write: no
    ```

    to disable renaming and tag-writing.

feat(beets#Importing your library): Importing your library

The next step is to import your music files into the beets library database. Because this can involve modifying files and moving them around, data loss is always a possibility, so now would be a good time to make sure you have a recent backup of all your music. We’ll wait.

There are two good ways to bring your existing library into beets. You can either: (a) quickly bring all your files with all their current metadata into beets’ database, or (b) use beets’ highly-refined autotagger to find canonical metadata for every album you import. Option (a) is really fast, but option (b) makes sure all your songs’ tags are exactly right from the get-go. The point about speed bears repeating: using the autotagger on a large library can take a very long time, and it’s an interactive process. So set aside a good chunk of time if you’re going to go that route.

If you’ve got time and want to tag all your music right once and for all, do this:

```bash
beet import /path/to/my/music
```

(Note that by default, this command will copy music into the directory you specified above. If you want to use your current directory structure, set the import.copy config option.) To take the fast, un-autotagged path, just say:

```bash
beet import -A /my/huge/mp3/library
```

Note that you just need to add `-A` for “don’t autotag”.

feat(pydantic#To investigate): Investigate libraries

[Integration of pydantic with pandas](https://pandera.readthedocs.io/en/stable/pydantic_integration.html)

feat(copier): Introduce copier

[Copier](https://github.com/copier-org/copier) is a library and CLI app for rendering project templates.

- Works with local paths and Git URLs.
- Your project can include any file and Copier can dynamically replace values in any kind of text file.
- It generates a beautiful output and takes care of not overwriting existing files unless instructed to do so.

This long article covers:

- [Installation](copier.md#installation)
- [Basic concepts](copier.md#basic-concepts)
- [Usage](copier.md#usage)

feat(kubectl_commands#Run a pod in a defined node): Run a pod in a defined node

Get the node hostnames with `kubectl get nodes`, then override the node with:

```bash
kubectl run mypod --image ubuntu:18.04 --overrides='{"apiVersion": "v1", "spec": {"nodeSelector": { "kubernetes.io/hostname": "my-node.internal" }}}' --command -- sleep 100000000000000
```

feat(diffview): Introduce DiffView

[Diffview](https://github.com/sindrets/diffview.nvim) is a single tabpage interface for easily cycling through diffs for all modified files for any git rev.

Installation:

If you're using it with NeoGit and Packer use:

```lua
  use {
    'NeogitOrg/neogit',
    requires = {
      'nvim-lua/plenary.nvim',
      'sindrets/diffview.nvim',
      'nvim-tree/nvim-web-devicons'
    }
  }
```

Usage:

Calling `:DiffviewOpen` with no args opens a new `Diffview` that compares against the current index. You can also provide any valid git rev to view only changes for that rev.

Examples:

- `:DiffviewOpen`
- `:DiffviewOpen HEAD~2`
- `:DiffviewOpen HEAD~4..HEAD~2`
- `:DiffviewOpen d4a7b0d`
- `:DiffviewOpen d4a7b0d^!`
- `:DiffviewOpen d4a7b0d..519b30e`
- `:DiffviewOpen origin/main...HEAD`

You can also provide additional paths to narrow down what files are shown `:DiffviewOpen HEAD~2 -- lua/diffview plugin`.

Additional commands for convenience:

- `:DiffviewClose`: Close the current diffview. You can also use `:tabclose`.
- `:DiffviewToggleFiles`: Toggle the file panel.
- `:DiffviewFocusFiles`: Bring focus to the file panel.
- `:DiffviewRefresh`: Update stats and entries in the file list of the current Diffview.

With a Diffview open and the default key bindings, you can:

- Cycle through changed files with `<tab>` and `<s-tab>`
- You can stage changes with `-`
- Restore a file with `X`
- Refresh the diffs with `R`
- Go to the file panel with `<leader>e`

feat(docker#Add healthcheck to your dockers): Add healthcheck to your dockers

Health checks allow a container to expose its workload’s availability. This stands apart from whether the container is running. If your database goes down, your API server won’t be able to handle requests, even though its Docker container is still running.

This makes for unhelpful experiences during troubleshooting. A simple `docker ps` would report the container as available. Adding a health check extends the `docker ps` output to include the container’s true state.

You configure container health checks in your Dockerfile. This accepts a command which the Docker daemon will execute every 30 seconds. Docker uses the command’s exit code to determine your container’s healthiness:

- `0`: The container is healthy and working normally.
- `1`: The container is unhealthy; the workload may not be functioning.

Healthiness isn’t checked straightaway when containers are created. The status will show as starting before the first check runs. This gives the container time to execute any startup tasks. A container with a passing health check will show as healthy; an unhealthy container displays unhealthy.

In docker-compose you can write the healthchecks like the next snippet:

```yaml
---
version: '3.4'

services:
  jellyfin:
    image: linuxserver/jellyfin:latest
    container_name: jellyfin
    restart: unless-stopped
    healthcheck:
      test: curl http://localhost:8096/health || exit 1
      interval: 10s
      retries: 5
      start_period: 5s
      timeout: 10s
```

feat(docker#List the dockers of a registry): List the dockers of a registry

List all repositories (effectively images):

```bash
$: curl -X GET https://myregistry:5000/v2/_catalog
> {"repositories":["redis","ubuntu"]}
```

List all tags for a repository:

```bash
$: curl -X GET https://myregistry:5000/v2/ubuntu/tags/list
> {"name":"ubuntu","tags":["14.04"]}
```

If the registry needs authentication you have to specify username and password in the curl command

```bash
curl -X GET -u <user>:<pass> https://myregistry:5000/v2/_catalog
curl -X GET -u <user>:<pass> https://myregistry:5000/v2/ubuntu/tags/list
```

feat(git#Remove tags): Remove tags

To delete a tag you can run:

```bash
git tag -d {{tag_name}}
```

To remove them remotely do

```bash
git push --delete origin {{ tag_name }}
```

fix(gitea): Configure the gitea actions

So far there is [only one possible runner](https://gitea.com/gitea/act_runner) which is based on docker and [`act`](https://github.com/nektos/act). Currently, the only way to install act runner is by compiling it yourself, or by using one of the [pre-built binaries](https://dl.gitea.com/act_runner). There is no Docker image or other type of package management yet. At the moment, act runner should be run from the command line. Of course, you can also wrap this binary in something like a system service, supervisord, or Docker container.

You can create the default configuration of the runner with:

```bash
./act_runner generate-config > config.yaml
```

You can tweak there for example the `capacity` so you are able to run more than one workflow in parallel.

Before running a runner, you should first register it to your Gitea instance using the following command:

```bash
./act_runner register --config config.yaml --no-interactive --instance <instance> --token <token>
```

Finally, it’s time to start the runner.

```bash
./act_runner --config config.yaml daemon
```

If you want to create your own act docker, you can start with this dockerfile:

```dockerfile
FROM node:16-bullseye

LABEL prune=false

RUN mkdir /root/.aws
COPY files/config /root/.aws/config
COPY files/credentials /root/.aws/credentials

RUN apt-get update && apt-get install -y \
  python3 \
  python3-pip \
  python3-venv \
  screen \
  vim \
  && python3 -m pip install --upgrade pip \
  && rm -rf /var/lib/apt/lists/*

RUN pip install \
  molecule==5.0.1 \
  ansible==8.0.0 \
  ansible-lint \
  yamllint \
  molecule-plugins[ec2,docker,vagrant] \
  boto3 \
  botocore \
  testinfra \
  pytest

RUN wget https://download.docker.com/linux/static/stable/x86_64/docker-24.0.2.tgz \
  && tar xvzf docker-24.0.2.tgz \
  && cp docker/* /usr/bin \
  && rm -r docker docker-*
```

It's prepared for:

- Working within an AWS environment
- Run Ansible and molecule
- Build dockers

feat(gitea#Build a docker within a gitea action): Build a docker within a gitea action

Assuming you're using the custom gitea_runner docker proposed above you can build and upload a docker to a registry with this action:

```yaml
---
name: Publish Docker image

"on": [push]

jobs:
  build-and-push:
    runs-on: ubuntu-latest
    steps:
      - name: Checkout code
        uses: https://github.com/actions/checkout@v3

      - name: Login to Docker Registry
        uses: https://github.com/docker/login-action@v2
        with:
          registry: my_registry.org
          username: ${{ secrets.REGISTRY_USERNAME }}
          password: ${{ secrets.REGISTRY_PASSWORD }}

      - name: Set up QEMU
        uses: https://github.com/docker/setup-qemu-action@v2

      - name: Set up Docker Buildx
        uses: https://github.com/docker/setup-buildx-action@v2

      - name: Extract metadata (tags, labels) for Docker
        id: meta
        uses: https://github.com/docker/metadata-action@v4
        with:
          images: my_registry.org/the_name_of_the_docker_to_build

      - name: Build and push
        uses: docker/build-push-action@v2
        with:
          context: .
          platforms: linux/amd64,linux/arm64
          push: true
          cache-from: type=registry,ref=my_registry.org/the_name_of_the_docker_to_build:buildcache
          cache-to: type=registry,ref=my_registry.org/the_name_of_the_docker_to_build:buildcache,mode=max
          tags: ${{ steps.meta.outputs.tags }}
          labels: ${{ steps.meta.outputs.labels }}
```

It uses a pair of nice features:

- Multi-arch builds
- [Cache](https://docs.docker.com/build/ci/github-actions/cache/) to speed up the builds

As it reacts to all events it will build and push:

- A tag with the branch name on each push to that branch
- A tag with the tag on tag push

feat(gitea#Bump the version of a repository on commits on master): Bump the version of a repository on commits on master

- Create a SSH key for the CI to send commits to protected branches.
- Upload the private key to a repo or organization secret called `DEPLOY_SSH_KEY`.
- Upload the public key to the repo configuration deploy keys
- Create the `bump.yaml` file with the next contents:

    ```yaml
    ---
    name: Bump version

    "on":
      push:
        branches:
          - main

    jobs:
      bump_version:
        if: "!startsWith(github.event.head_commit.message, 'bump:')"
        runs-on: ubuntu-latest
        name: "Bump version and create changelog"
        steps:
          - name: Check out
            uses: actions/checkout@v3
            with:
              fetch-depth: 0  # Fetch all history

          - name: Configure SSH
            run: |
                echo "${{ secrets.DEPLOY_SSH_KEY }}" > ~/.ssh/deploy_key
                chmod 600 ~/.ssh/deploy_key
                dos2unix ~/.ssh/deploy_key
                ssh-agent -a $SSH_AUTH_SOCK > /dev/null
                ssh-add ~/.ssh/deploy_key

          - name: Bump the version
            run: cz bump --changelog --no-verify

          - name: Push changes
            run: |
              git remote add ssh [email protected]:templates/ansible-role.git
              git pull ssh main
              git push ssh main
              git push ssh --tags
    ```

    It assumes that you have `cz` (commitizen) and `dos2unix` installed in your runner.

feat(gitea#Skip gitea actions job on changes of some files): Skip gitea actions job on changes of some files

There are some expensive CI pipelines that don't need to be run for example if you changed a line in the `README.md`, to skip a pipeline on changes of certain files you can use the `paths-ignore` directive:

```yaml
---
name: Ansible Testing

"on":
  push:
    paths-ignore:
      - 'meta/**'
      - Makefile
      - README.md
      - renovate.json
      - CHANGELOG.md
      - .cz.toml
      - '.gitea/workflows/**'

jobs:
  test:
    name: Test
    runs-on: ubuntu-latest
    steps:
        ...
```

The only downside is that if you set this pipeline as required in the branch protection, the merge button will look yellow instead of green when the pipeline is skipped.

feat(gotify): Introduce gotify

[Gotify](https://github.com/gotify/server) is a simple server for sending and receiving messages in real-time per WebSocket.

Not there yet:

- [Reactions on the notifications](gotify/server#494)

feat(grafana): Introduce grafana

[Grafana](https://grafana.com/grafana) is a web application to create dashboards.

[Installation](https://grafana.com/docs/grafana/latest/setup-grafana/installation/docker/#run-grafana-via-docker-compose): We're going to install it with docker-compose and connect it to [Authentik](authentik.md).

[Create the Authentik connection](https://goauthentik.io/integrations/services/grafana/):

Assuming that you have [the terraform authentik provider configured](authentik.md), use the next terraform code:

```hcl

variable "grafana_name" {
  type        = string
  description = "The name shown in the Grafana application."
  default     = "Grafana"
}

variable "grafana_redirect_uri" {
  type        = string
  description = "The redirect url configured on Grafana."
}

variable "grafana_icon" {
  type        = string
  description = "The icon shown in the Grafana application"
  default     = "/application-icons/grafana.svg"
}

resource "authentik_application" "grafana" {
  name              = var.grafana_name
  slug              = "grafana"
  protocol_provider = authentik_provider_oauth2.grafana.id
  meta_icon         = var.grafana_icon
  lifecycle {
    ignore_changes = [
      # The terraform provider is continuously changing the attribute even though it's set
      meta_icon,
    ]
  }
}

resource "authentik_provider_oauth2" "grafana" {
  name               = var.grafana_name
  client_id          = "grafana"
  authorization_flow = data.authentik_flow.default-authorization-flow.id
  property_mappings = [
    data.authentik_scope_mapping.email.id,
    data.authentik_scope_mapping.openid.id,
    data.authentik_scope_mapping.profile.id,
  ]
  redirect_uris = [
    var.grafana_redirect_uri,
  ]
  signing_key = data.authentik_certificate_key_pair.default.id
  access_token_validity = "minutes=120"
}

data "authentik_certificate_key_pair" "default" {
  name = "authentik Self-signed Certificate"
}

data "authentik_flow" "default-authorization-flow" {
  slug = "default-provider-authorization-implicit-consent"
}

output "grafana_oauth_id" {
  value = authentik_provider_oauth2.grafana.client_id
}

output "grafana_oauth_secret" {
  value = authentik_provider_oauth2.grafana.client_secret
}
```

feat(jellyfin#Jellyfin Desktop): Introduce Jellyfin Desktop

- Download the latest deb package from the [releases page](https://github.com/jellyfin/jellyfin-media-player/releases)
- Install the dependencies
- Run `dpkg -i`

If you're on a TV you may want to [enable the TV mode](jellyfin/jellyfin-media-player#11) so that the remote keys work as expected. The play/pause/next/prev won't work until [this issue is solved](jellyfin/jellyfin-media-player#3), but it's not that bad to use the "Ok" and then navigate with the arrow keys.

feat(jellyfin#Jellycon): Introduce Jellycon

JellyCon is a lightweight Kodi add-on that lets you browse and play media files directly from your Jellyfin server within the Kodi interface. It can be thought of as a thin frontend for a Jellyfin server.

It's not very pleasant to use though.

feat(kodi): Introduce Kodi

[Kodi](https://kodi.tv/) is a entertainment center software. It basically converts your device into a smart tv

feat(koel): Introduce Koel

[koel](https://koel.dev/) is a personal music streaming server.

Note: Use [`mopidy`](mopidy.md) instead

There are [docker-compose files](https://github.com/koel/docker) to host the service. Although they behave a little bit weird

For example, you need to [specify the DB_PORT](koel/docker#168). It has had several PR to fix it but weren't merged [1](https://github.com/koel/docker/pull/165/files), [2](https://github.com/koel/docker/pull/162/files).

The API is [not very well documented](koel/koel#535):

- [Here you can see how to authenticate](https://github.com/X-Ryl669/kutr/wiki/Communication-API#authentication)
- [Here are the api docs](https://github.com/koel/koel/blob/master/api-docs/api.yaml#L763)

feat(zfs#Rename or move a dataset): Rename or move a dataset

NOTE: if you want to rename the topmost dataset look at [rename the topmost dataset](#rename-the-topmost-dataset) instead.
File systems can be renamed by using the `zfs rename` command. You can perform the following operations:

- Change the name of a file system.
- Relocate the file system within the ZFS hierarchy.
- Change the name of a file system and relocate it within the ZFS hierarchy.

The following example uses the `rename` subcommand to rename of a file system from `kustarz` to `kustarz_old`:

```bash
zfs rename tank/home/kustarz tank/home/kustarz_old
```

The following example shows how to use zfs `rename` to relocate a file system:

```bash
zfs rename tank/home/maybee tank/ws/maybee
```

In this example, the `maybee` file system is relocated from `tank/home` to `tank/ws`. When you relocate a file system through rename, the new location must be within the same pool and it must have enough disk space to hold this new file system. If the new location does not have enough disk space, possibly because it has reached its quota, rename operation fails.

The rename operation attempts an unmount/remount sequence for the file system and any descendent file systems. The rename command fails if the operation is unable to unmount an active file system. If this problem occurs, you must forcibly unmount the file system.

You'll loose the snapshots though, as explained below.

feat(zfs#Rename the topmost dataset): Rename the topmost dataset

If you want to rename the topmost dataset you [need to rename the pool too](openzfs/zfs#4681) as these two are tied.

```bash
$: zpool status -v

  pool: tets
 state: ONLINE
 scrub: none requested
config:

        NAME        STATE     READ WRITE CKSUM
        tets        ONLINE       0     0     0
          c0d1      ONLINE       0     0     0
          c1d0      ONLINE       0     0     0
          c1d1      ONLINE       0     0     0

errors: No known data errors
```

To fix this, first export the pool:

```bash
$ zpool export tets
```

And then imported it with the correct name:

```bash
$ zpool import tets test
```

After the import completed, the pool contains the correct name:

```bash
$ zpool status -v

  pool: test
 state: ONLINE
 scrub: none requested
config:

        NAME        STATE     READ WRITE CKSUM
        test        ONLINE       0     0     0
          c0d1      ONLINE       0     0     0
          c1d0      ONLINE       0     0     0
          c1d1      ONLINE       0     0     0

errors: No known data errors
```

Now you may need to fix the ZFS mountpoints for each dataset

```bash
zfs set mountpoint="/opt/zones/[Newmountpoint]" [ZFSPOOL/[ROOTor other filesystem]
```

feat(zfs#Rename or move snapshots): Rename or move snapshots

If the dataset has snapshots you need to rename them too. They must be renamed within the same pool and dataset from which they were created though. For example:

```bash
zfs rename tank/home/cindys@083006 tank/home/cindys@today
```

In addition, the following shortcut syntax is equivalent to the preceding syntax:

```bash
zfs rename tank/home/cindys@083006 today
```

The following snapshot rename operation is not supported because the target pool and file system name are different from the pool and file system where the snapshot was created:

```bash
$: zfs rename tank/home/cindys@today pool/home/cindys@saturday
cannot rename to 'pool/home/cindys@today': snapshots must be part of same
dataset
```

You can recursively rename snapshots by using the `zfs rename -r` command. For example:

```bash
$: zfs list
NAME                         USED  AVAIL  REFER  MOUNTPOINT
users                        270K  16.5G    22K  /users
users/home                    76K  16.5G    22K  /users/home
users/home@yesterday            0      -    22K  -
users/home/markm              18K  16.5G    18K  /users/home/markm
users/home/markm@yesterday      0      -    18K  -
users/home/marks              18K  16.5G    18K  /users/home/marks
users/home/marks@yesterday      0      -    18K  -
users/home/neil               18K  16.5G    18K  /users/home/neil
users/home/neil@yesterday       0      -    18K  -
$: zfs rename -r users/home@yesterday @2daysago
$: zfs list -r users/home
NAME                        USED  AVAIL  REFER  MOUNTPOINT
users/home                   76K  16.5G    22K  /users/home
users/home@2daysago            0      -    22K  -
users/home/markm             18K  16.5G    18K  /users/home/markm
users/home/markm@2daysago      0      -    18K  -
users/home/marks             18K  16.5G    18K  /users/home/marks
users/home/marks@2daysago      0      -    18K  -
users/home/neil              18K  16.5G    18K  /users/home/neil
users/home/neil@2daysago       0      -    18K  -
```

feat(zfs#See the differences between two backups): See the differences between two backups

To identify the differences between two snapshots, use syntax similar to the following:

```bash
$ zfs diff tank/home/tim@snap1 tank/home/tim@snap2
M       /tank/home/tim/
+       /tank/home/tim/fileB
```

The following table summarizes the file or directory changes that are identified by the `zfs diff` command.

| File or Directory Change | Identifier |
| --- | --- |
| File or directory has been modified or file or directory link has changed | M |
| File or directory is present in the older snapshot but not in the more recent snapshot | — |
| File or directory is present in the more recent snapshot but not in the older snapshot | + |
| File or directory has been renamed | R |

feat(zfs#Create a cold backup of a series of datasets): Create a cold backup of a series of datasets

If you've used the `-o keyformat=raw -o keylocation=file:///etc/zfs/keys/home.key` arguments to encrypt your datasets you can't use a `keyformat=passphase` encryption on the cold storage device. You need to copy those keys on the disk. One way of doing it is to:

- Create a 100M LUKS partition protected with a passphrase where you store the keys.
- The rest of the space is left for a partition for the zpool.

feat(zfs#Clear a permanent ZFS error in a healthy pool): Clear a permanent ZFS error in a healthy pool

Sometimes when you do a `zpool status` you may see that the pool is healthy but that there are "Permanent errors" that may point to files themselves or directly to memory locations.

You can read [this long discussion](openzfs/zfs#9705) on what does these permanent errors mean, but what solved the issue for me was to run a new scrub

`zpool scrub my_pool`

It takes a long time to run, so be patient.

feat(zfs#ZFS pool is in suspended mode): ZFS pool is in suspended mode

Probably because you've unplugged a device without unmounting it.

If you want to remount the device [you can follow these steps](openzfsonosx/zfs#104 (comment)) to symlink the new devfs entries to where zfs thinks the vdev is. That way you can regain access to the pool without a reboot.

So if zpool status says the vdev is /dev/disk2s1, but the reattached drive is at disk4, then do the following:

```bash
cd /dev
sudo rm -f disk2s1
sudo ln -s disk4s1 disk2s1
sudo zpool clear -F WD_1TB
sudo zpool export WD_1TB
sudo rm disk2s1
sudo zpool import WD_1TB
```

If you don't care about the zpool anymore, sadly your only solution is to [reboot the server](openzfs/zfs#5242). Real ugly, so be careful when you umount zpools.

feat(linux_snippets#Get the current git branch): Get the current git branch

```bash
git branch --show-current
```

feat(linux_snippets#Install latest version of package from backports): Install latest version of package from backports

Add the backports repository:

```bash
vi /etc/apt/sources.list.d/bullseye-backports.list
```

```
deb http://deb.debian.org/debian bullseye-backports main contrib
deb-src http://deb.debian.org/debian bullseye-backports main contrib
```

Configure the package to be pulled from backports

```bash
vi /etc/apt/preferences.d/90_zfs
```

```
Package: src:zfs-linux
Pin: release n=bullseye-backports
Pin-Priority: 990
```

feat(linux_snippets#Rename multiple files matching a pattern): Rename multiple files matching a pattern

There is `rename` that looks nice, but you need to install it. Using only `find` you can do:

```bash
find . -name '*yml' -exec bash -c 'echo mv $0 ${0/yml/yaml}' {} \;
```

If it shows what you expect, remove the `echo`.

feat(linux_snippets#Force ssh to use password authentication): Force ssh to use password authentication

```bash
ssh -o PreferredAuthentications=password -o PubkeyAuthentication=no [email protected]
```
feat(linux_snippets#Do a tail -f with grep): Do a tail -f with grep

```bash
tail -f file | grep --line-buffered my_pattern
```

feat(linux_snippets#Check if a program exists in the user's PATH): Check if a program exists in the user's PATH

```bash
command -v <the_command>
```

Example use:

```bash
if ! command -v <the_command> &> /dev/null
then
    echo "<the_command> could not be found"
    exit
fi
```

feat(mediatracker): Introduce MediaTracker

[MediaTracker](https://github.com/bonukai/MediaTracker) is a self hosted media tracker for movies, tv shows, video games, books and audiobooks

[Installation](https://github.com/bonukai/MediaTracker#installation):

With docker compose:

```yaml
version: "3"
services:
  mediatracker:
    container_name: mediatracker
    ports:
      - 7481:7481
    volumes:
      - /home/YOUR_HOME_DIRECTORY/.config/mediatracker/data:/storage
      - assetsVolume:/assets
    environment:
      SERVER_LANG: en
      TMDB_LANG: en
      AUDIBLE_LANG: us
      TZ: Europe/London
    image: bonukai/mediatracker:latest

volumes:
  assetsVolume: null
```

If you attach more than one docker network the container becomes unreachable :S.

Install the jellyfin plugin:

They created a [Jellyfin plugin](https://github.com/bonukai/jellyfin-plugin-mediatracker) so that all scrobs are sent automatically to the mediatracker

- Add new Repository in Jellyfin (Dashboard -> Plugins -> Repositories -> +) from url `https://raw.githubusercontent.com/bonukai/jellyfin-plugin-mediatracker/main/manifest.json`
- Install MediaTracker plugin from Catalogue (Dashboard -> Plugins -> Catalogue)

Some tips on usage:

- Add the shows you want to watch to the watchlist so that it's easier to find them
- When you're ending an episode, click on the episode number on the watchlist element and then rate the episode itself.

- You can create public lists to share with the rest of the users, the way to share it though [is a bit archaic so far](bonukai/MediaTracker#527), it's only through the list link, in the interface they won't be able to see it.

feat(molecule#Molecule doesn't find the `molecule.yaml` file): Molecule doesn't find the `molecule.yaml` file

This is expected default behavior since Molecule searches for scenarios using the `molecule/*/molecule.yml` glob. But if you would like to change the suffix to yaml, you can do that if you set the `MOLECULE_GLOB` environment variable like this:

```bash
export MOLECULE_GLOB='molecule/*/molecule.yaml'
```

feat(python_jinja2#Escape jinja expansion on a jinja template): Escape jinja expansion on a jinja template

```jinja
{% raw %}

Anything in this block is treated as raw text,
including {{ curly braces }} and
{% other block-like syntax %}

{% endraw %}
```

feat(qbittorrent#Tools): Add interesting tools to explore

- [qbittools](https://github.com/buroa/qbittools): a feature rich CLI for the management of torrents in qBittorrent.
- [qbit_manage](https://github.com/StuffAnThings/qbit_manage): tool will help manage tedious tasks in qBittorrent and automate them.

feat(sanoid#Prune snapshots): Prune snapshots

If you want to manually prune the snapshots after you tweaked `sanoid.conf` you can run:

```bash
sanoid --prune-snapshots
```

feat(sanoid#Send encrypted backups to a encrypted dataset): Send encrypted backups to a encrypted dataset

`syncoid`'s default behaviour is to create the destination dataset without encryption so the snapshots are transferred and can be read without encryption. You can check this with the `zfs get encryption,keylocation,keyformat` command both on source and destination.

To prevent this from happening you have to [pass the `--sendoptions='w'](jimsalterjrs/sanoid#548) to `syncoid` so that it tells zfs to send a raw stream. If you do so, you also need to [transfer the key file](jimsalterjrs/sanoid#648) to the destination server so that it can do a `zfs loadkey` and then mount the dataset. For example:

```bash
server-host:$ sudo zfs list -t filesystem
NAME                    USED  AVAIL     REFER  MOUNTPOINT
server_data             232M  38.1G      230M  /var/server_data
server_data/log         111K  38.1G      111K  /var/server_data/log
server_data/mail        111K  38.1G      111K  /var/server_data/mail
server_data/nextcloud   111K  38.1G      111K  /var/server_data/nextcloud
server_data/postgres    111K  38.1G      111K  /var/server_data/postgres

server-host:$ sudo zfs get keylocation server_data/nextcloud
NAME                   PROPERTY     VALUE                                    SOURCE
server_data/nextcloud  keylocation  file:///root/zfs_dataset_nextcloud_pass  local

server-host:$ sudo syncoid --recursive --skip-parent --sendoptions=w server_data [email protected]:backup_pool
INFO: Sending oldest full snapshot server_data/log@autosnap_2021-06-18_18:33:42_yearly (~ 49 KB) to new target filesystem:
17.0KiB 0:00:00 [1.79MiB/s] [=================================================>                                                                                                  ] 34%
INFO: Updating new target filesystem with incremental server_data/log@autosnap_2021-06-18_18:33:42_yearly ... syncoid_caedrium.com_2021-06-22:10:12:55 (~ 15 KB):
41.2KiB 0:00:00 [78.4KiB/s] [===================================================================================================================================================] 270%
INFO: Sending oldest full snapshot server_data/mail@autosnap_2021-06-18_18:33:42_yearly (~ 49 KB) to new target filesystem:
17.0KiB 0:00:00 [ 921KiB/s] [=================================================>                                                                                                  ] 34%
INFO: Updating new target filesystem with incremental server_data/mail@autosnap_2021-06-18_18:33:42_yearly ... syncoid_caedrium.com_2021-06-22:10:13:14 (~ 15 KB):
41.2KiB 0:00:00 [49.4KiB/s] [===================================================================================================================================================] 270%
INFO: Sending oldest full snapshot server_data/nextcloud@autosnap_2021-06-18_18:33:42_yearly (~ 49 KB) to new target filesystem:
17.0KiB 0:00:00 [ 870KiB/s] [=================================================>                                                                                                  ] 34%
INFO: Updating new target filesystem with incremental server_data/nextcloud@autosnap_2021-06-18_18:33:42_yearly ... syncoid_caedrium.com_2021-06-22:10:13:42 (~ 15 KB):
41.2KiB 0:00:00 [50.4KiB/s] [===================================================================================================================================================] 270%
INFO: Sending oldest full snapshot server_data/postgres@autosnap_2021-06-18_18:33:42_yearly (~ 50 KB) to new target filesystem:
17.0KiB 0:00:00 [1.36MiB/s] [===============================================>                                                                                                    ] 33%
INFO: Updating new target filesystem with incremental server_data/postgres@autosnap_2021-06-18_18:33:42_yearly ... syncoid_caedrium.com_2021-06-22:10:14:11 (~ 15 KB):
41.2KiB 0:00:00 [48.9KiB/s] [===================================================================================================================================================] 270%

server-host:$ sudo scp /root/zfs_dataset_nextcloud_pass 192.168.122.94:
```

```bash
backup-host:$ sudo zfs set keylocation=file:///root/zfs_dataset_nextcloud_pass  backup_pool/nextcloud
backup-host:$ sudo zfs load-key backup_pool/nextcloud
backup-host:$ sudo zfs mount backup_pool/nextcloud
```

If you also want to keep the `encryptionroot` you need to [let zfs take care of the recursion instead of syncoid](jimsalterjrs/sanoid#614). In this case you can't use syncoid's stuff like `--exclude` from the manpage of zfs:

```
-R, --replicate
   Generate a replication stream package, which will replicate the specified file system, and all descendent file systems, up to the named snapshot.  When received, all properties, snap‐
   shots, descendent file systems, and clones are preserved.

   If the -i or -I flags are used in conjunction with the -R flag, an incremental replication stream is generated.  The current values of properties, and current snapshot and file system
   names are set when the stream is received.  If the -F flag is specified when this stream is received, snapshots and file systems that do not exist on the sending side are destroyed.
   If the -R flag is used to send encrypted datasets, then -w must also be specified.
```

In this case this should work:

```bash
/sbin/syncoid --recursive --force-delete --sendoptions="Rw" zpool/backups [email protected]:zpool/backups
```

feat(terraform#Create a list of resources based on a list of strings): Create a list of resources based on a list of strings

```hcl
variable "subnet_ids" {
  type = list(string)
}

resource "aws_instance" "server" {
  # Create one instance for each subnet
  count = length(var.subnet_ids)

  ami           = "ami-a1b2c3d4"
  instance_type = "t2.micro"
  subnet_id     = var.subnet_ids[count.index]

  tags = {
    Name = "Server ${count.index}"
  }
}
```

If you want to use this generated list on another resource extracting for example the id you can use

```hcl
aws_instance.server.*.id
```

feat(vim#Run a command when opening vim): Run a command when opening vim

```bash
nvim -c ':DiffViewOpen'
```

fix(zfs_exporter): Update the alerts to the more curated version

feat(zfs_exporter#Useful inhibits): Useful inhibits

Some you may want to inhibit some of these rules for some of your datasets. These subsections should be added to the `alertmanager.yml` file under the `inhibit_rules` field.

Ignore snapshots on some datasets: Sometimes you don't want to do snapshots on a dataset

```yaml
- target_matchers:
    - alertname = ZfsDatasetWithNoSnapshotsError
    - hostname = my_server_1
    - filesystem = tmp
```

Ignore snapshots growth: Sometimes you don't mind if the size of the data saved in the filesystems doesn't change too much between snapshots doesn't change much specially in the most frequent backups because you prefer to keep the backup cadence. It's interesting to have the alert though so that you can get notified of the datasets that don't change that much so you can tweak your backup policy (even if zfs snapshots are almost free).

```yaml
  - target_matchers:
    - alertname =~ "ZfsSnapshotType(Frequently|Hourly)SizeError"
    - filesystem =~ "(media/(docs|music))"
```
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

5 participants