Blacklist scrapers

Blacklist

RocketMap provides a Blacklist to lock out some Apps / Pages who are collecting your data from your page

iptables

Note: do not use this if you have your page run over Cloudflare, some Cloudflare IPs are in this blacklist!

Run the following commands as root (sudo -i would do it aswell)

cat <<EOF > /usr/local/bin/update-rm-blacklist
#!/bin/bash
set -e
ipset create blacklist hash:net || true
curl https://blist.devkat.org/blacklist.json \
    | jq -r '.[] | "\(.[0])-\(.[1])"' \
    | xargs -n1 ipset add blacklist 2>/dev/null || true
iptables-save \
    | grep -q "-A INPUT -p tcp -m multiport --dports 80,443,5000 -m set --match-set blacklist src -j DROP" \
    | iptables -A INPUT -p tcp -m multiport --dports 80,443,5000 -m set --match-set blacklist src -j DROP
EOF
chmod +x /usr/local/bin/update-rm-blacklist
update-rm-blacklist

As the list may be updated from time to time you may want to rerun the second command

nginx

Run the following commands as root (sudo -i would do it aswell)

cat <<EOF > /usr/local/bin/update-rm-blacklist
#!/bin/bash
set -e
mkdir -p /etc/nginx/snippets/
ipset create blacklist hash:net || true
curl https://blist.devkat.org/blacklist.json \
    | jq -r '.[] | "\(.[0])-\(.[1])"' \
    | xargs -n1 ipset add blacklist 2>/dev/null || true
echo "allow all;" > /etc/nginx/snippets/blacklist.conf
ipset list blacklist | grep -oE "^[1-9][0-9]*(\.0|\.[1-9][0-9]*){3}/[1-9][0-9]*$" \
    | while read RANGE; do
    echo "deny \$RANGE;" >> /etc/nginx/snippets/blacklist.conf
done
EOF
chmod +x /usr/local/bin/update-rm-blacklist
update-rm-blacklist

and don't forget to configure nginx

Auto updateing the blacklist

If you have cron installed, you can run

# as root
echo "21 2 * * * /usr/local/bin/update-rm-blacklist" > /etc/cron.d/update-rm-blacklist
# not root
echo "21 2 * * * /usr/local/bin/update-rm-blacklist" | sudo tee /etc/cron.d/update-rm-blacklist

It will update the blacklist every night at 02:21