Added hosts support, as well as support for plain domains.

[athyfr]

This is a commit to add hosts file support, as an option to compare to @miallo's implementation.

Solves #2

[athyfr]

This is relatively new for me, since I've not often done sophisticated Bash scripting before.

Thanks for correcting my mistakes!

[miallo]

Nice! Looking good! Could you maybe try it out? Then we would know, if we missed anything of the weird nix quoting within the sometimes rather unintuitive bash syntax ^^

[athyfr]

I got an error:

Mar 16 13:10:10 athyfr-nixos-laptop blocklist-start[1931128]: /nix/store/f7pa05gazc2dhkw4j55qg22l2daxhxcw-blocklist_update.sh: line 40: domain_regex^([A-Za-z0-9-]+(?<!-)\.)+[A-Za-z0-9-]+$: command not found

[athyfr]

Alright, I'll test again.

[athyfr]

Argh.. I can't figure out how to properly update my system flake input..

[athyfr]

(Without updating other system flake inputs, because that would have too much build time)

[athyfr]

Argh.. I can't figure out how to properly update my system flake input..

Seeing NixOS/nix#8817, the solution is cd /etc/nixos && nix flake update blocklist-updater

[athyfr]

It's returning this strange error, and I don't know why. (when evaluating Blocklist Project's Adobe blocklist):

ipset v7.23: Error in line 1: Syntax error: cannot parse 3dns: resolving to IPv4 address failed

This causes a full bail, so it needs to be solved. For now though, maybe I'll just try a different blocklist for testing.

[miallo]

Something that is apparently clear is that bash in general and in this "in nix script" specifically does not seem to be a good fit. Maybe we could consider rewriting it in a more expressive language? 🤔
If you like the challenge, then we could stick with bash though 😂

[athyfr]

Something that is apparently clear is that bash in general and in this "in nix script" specifically does not seem to be a good fit. Maybe we could consider rewriting it in a more expressive language? 🤔 If you like the challenge, then we could stick with bash though 😂

I don't have any arguments against switching.

In fact, I have a few potential options:

• Using another shell scripting language (e.g. Fish) might make some syntax a little easier. It would still be much of the same, though.
• Modularizing the shell scripts a little more, and importing them from .sh files instead of inlining them in Nix would make syntax highlighting, and debugging more convenient.
• Many online sources recommend Python as an alternative, for better syntax, though it does have additional performance overhead.

If you have a more niche language suggestion, I'm all ears!
I know that often they can have big advantages.

[athyfr]

We could also figure out bash-in-nix syntax highlighting. This Reddit article has some suggestions for Vim-based code-editors.

What code editor do you use @miallo? (I'm using Zed+nix-zed, but that doesn't have support yet.)

[miallo]

If you have a more niche language suggestion, I'm all ears! I know that often they can have big advantages.

For better overview moved to separate issue #11

[miallo]

What code editor do you use @miallo?

I am indeed using neovim (though so far I have not looked at treesitter - will change that once I have a bit more time. Thanks for the reference!)

[athyfr]

What code editor do you use @miallo?

I am indeed using neovim (though so far I have not looked at treesitter - will change that once I have a bit more time. Thanks for the reference!)

Your welcome! I should consider migrating to Neovim too, but I hear it has a steep learning curve..

[athyfr]

We should use this blocklist for testing, since StevenBlack's main blocklist takes minutes to load..

This one takes awhile too (bad sign), but at least it's smaller for testing.

Maybe I'll put this off now until language migration. It might ease the performance, and would prevent merge conflicts (and wasted effort).

Thanks! ^^

[miallo]

We should use this blocklist for testing, since StevenBlack's main blocklist takes minutes to load..

This one takes awhile too (bad sign), but at least it's smaller for testing.

FYI: if you want to test things without actually specifying a complete list you could just put a snippet of the list you want to test against into

blocklistedIPs=''
# Facebook
0.0.0.0 0-act.channel.facebook.com
0.0.0.0 0-edge-chat.facebook.com
0.0.0.0 0.beta.facebook.com
0.0.0.0 1-act.channel.facebook.com
'';
blocklists=lib.mkForce []; # do not download the default list every time

Since it is just appended

nix-blocklist-updater/update_blocklist.nix

Line 35 in c25634f

echo "${blocklistedIPs}">> $BLFILE

it behaves the same but without downloading something

There you can easily adapt the number of entries and you can also edit the entries more flexibly ;)

[miallo]

I should consider migrating to Neovim too, but I hear it has a steep learning curve..

It certainly isn't easy to master. For the first years in university I only used the most limited subset of features and even with that it provided certain big improvements for my workflows. When I became a programmer then I obviously started using it more and more and since then, ever so often I run into something where I tell myself "There must be a better way" and when looking it up, I try to remember the solution for the next time I need it (sometimes it works, some solutions I have searched like 50 times before ^^). Especially the beginning is indeed hard, but at least I justify it in to myself by saying "It kind of is a big part of my job to type, so it is fine spending some time on sharpening this axe". In general I recommend to others though to go with whatever fits for them best ;)

[miallo]

What is the state of this MR? Is it ready for testing, or do you prefer to do this in the rust implementation?

[athyfr]

What is the state of this MR? Is it ready for testing, or do you prefer to do this in the rust implementation?

I'm thinking I'll wait until the Rust implementation. It'll save effort (not implementing it twice), and it might be easier in Rust anyways.

[athyfr]

I'll close it for now.

[athyfr]

Just a note for the future:

Some hosts blocklists use 127.0.0.1 instead of 0.0.0.0, so it is worth including in the regex. We'll need to prevent blocking localhost, though, because a lot of hosts blocklists include the localhost DNS connection..

Hosts files can redirect IPs as well as domains.. as seen here for example (not that that is a good blocklist). These need to be parsed as well, but might suggest a more modular way for this to work.