Skip to content

Reinstalling mon with existing app server triggers missing iptables-persistent dependency #7119

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
nathandyer opened this issue Feb 21, 2024 · 8 comments · Fixed by #7417
Closed

Reinstalling mon with existing app server triggers missing iptables-persistent dependency #7119

nathandyer opened this issue Feb 21, 2024 · 8 comments · Fixed by #7417
Assignees
@legoktm
Milestone
SecureDrop 2.12.0

Comments

@nathandyer
Copy link
Contributor

Description

During a recent SD install to a freshly provisioned server running Ubuntu Server 20.04.6, the ./securedrop-admin install process ran into an error during the run, citing that the /etc/iptables directory was missing and that it couldn't continue.

To workround this, I had to do an apt install iptables-persistent on the server, at which point ./securedrop-admin install was able to complete the installation without issue.

We should determine if this is a required package, and if so, add it as a dependency.

Steps to Reproduce

  1. Install a fresh copy of Ubuntu Server 20.04.6 and configure it according to the SD docs
  2. Copy SSH keys from the Admin Workstation to it
  3. Run ./securedrop-admin install (along with setup and sdconfig beforehand, if necessary)

Expected Behavior

The install happens without issue.

Actual Behavior

An error regarding a missing /etc/iptables directory that stops the installation.

Comments

Not sure if this was a one-off, and also not sure of any potential security concerns about the iptables-persistent package.
@nathandyer
Copy link
Contributor Author

I just did a clean install of one of my SecureDrop servers, and ran into this again. Out of the box, it's not possible to install SecureDrop on a freshly provisioned Ubuntu Server (ubuntu server 20.04.6) without manually installing this additional package. With the noble transition (and moving away from iptables) I'm not sure if this is worth fixing directly, but we probably ought to at least document it in the meantime for orgs who are doing a clean install.

@legoktm
Copy link
Member

legoktm commented Jan 17, 2025

Can you provide the full ansible log/output? At what step did it fail?

@nathandyer
Copy link
Contributor Author

Full output from the run that failed:


amnesia@amnesia:~/Persistent/securedrop$ ./securedrop-admin --force install
/home/amnesia/Persistent/securedrop/admin/.venv3/bin/securedrop-admin:4: DeprecationWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html
  __import__('pkg_resources').require('securedrop-admin==0.1.0')
INFO: Skipping update check because --force argument was provided.
INFO: Now installing SecureDrop on remote servers.
INFO: You will be prompted for the sudo password on the servers.
INFO: The sudo password is only necessary during initial installation.
SUDO password: 

PLAY [Ensure validation is run before prod install] ****************************

TASK [Gathering Facts] *********************************************************
ok: [localhost]

TASK [validate : include_tasks] ************************************************
included: /home/amnesia/Persistent/securedrop/install_files/ansible-base/roles/validate/tasks/validate_tails_environment.yml for localhost

TASK [validate : Check /etc/os-release for Tails string] ***********************
ok: [localhost]

TASK [validate : Confirm host OS is Tails.] ************************************
ok: [localhost] => {
    "changed": false,
    "msg": "All assertions passed"
}

TASK [validate : Check for persistence volume.] ********************************
ok: [localhost] => (item=/live/persistence/TailsData_unlocked/persistence.conf)
ok: [localhost] => (item=/live/persistence/TailsData_unlocked/openssh-client)
ok: [localhost] => (item=/home/amnesia/Persistent/securedrop)

TASK [validate : Confirm persistence volume is configured.] ********************
ok: [localhost] => (item={'changed': False, 'stat': {'exists': True, 'path': '/live/persistence/TailsData_unlocked/persistence.conf', 'mode': '0600', 'isdir': False, 'ischr': False, 'isblk': False, 'isreg': True, 'isfifo': False, 'islnk': False, 'issock': False, 'uid': 115, 'gid': 122, 'size': 589, 'inode': 13, 'dev': 65024, 'nlink': 1, 'atime': 1737145626.8560019, 'mtime': 1737145626.520002, 'ctime': 1737145626.8480017, 'wusr': True, 'rusr': True, 'xusr': False, 'wgrp': False, 'rgrp': False, 'xgrp': False, 'woth': False, 'roth': False, 'xoth': False, 'isuid': False, 'isgid': False, 'blocks': 8, 'block_size': 4096, 'device_type': 0, 'readable': False, 'writeable': False, 'executable': False, 'pw_name': 'tails-persistent-storage', 'gr_name': 'tails-persistent-storage', 'mimetype': 'unknown', 'charset': 'unknown', 'version': None, 'attributes': [], 'attr_flags': ''}, 'invocation': {'module_args': {'path': '/live/persistence/TailsData_unlocked/persistence.conf', 'follow': False, 'get_md5': False, 'get_checksum': True, 'get_mime': True, 'get_attributes': True, 'checksum_algorithm': 'sha1'}}, 'failed': False, 'item': '/live/persistence/TailsData_unlocked/persistence.conf', 'ansible_loop_var': 'item'}) => {
    "ansible_loop_var": "item",
    "changed": false,
    "item": {
        "ansible_loop_var": "item",
        "changed": false,
        "failed": false,
        "invocation": {
            "module_args": {
                "checksum_algorithm": "sha1",
                "follow": false,
                "get_attributes": true,
                "get_checksum": true,
                "get_md5": false,
                "get_mime": true,
                "path": "/live/persistence/TailsData_unlocked/persistence.conf"
            }
        },
        "item": "/live/persistence/TailsData_unlocked/persistence.conf",
        "stat": {
            "atime": 1737145626.8560019,
            "attr_flags": "",
            "attributes": [],
            "block_size": 4096,
            "blocks": 8,
            "charset": "unknown",
            "ctime": 1737145626.8480017,
            "dev": 65024,
            "device_type": 0,
            "executable": false,
            "exists": true,
            "gid": 122,
            "gr_name": "tails-persistent-storage",
            "inode": 13,
            "isblk": false,
            "ischr": false,
            "isdir": false,
            "isfifo": false,
            "isgid": false,
            "islnk": false,
            "isreg": true,
            "issock": false,
            "isuid": false,
            "mimetype": "unknown",
            "mode": "0600",
            "mtime": 1737145626.520002,
            "nlink": 1,
            "path": "/live/persistence/TailsData_unlocked/persistence.conf",
            "pw_name": "tails-persistent-storage",
            "readable": false,
            "rgrp": false,
            "roth": false,
            "rusr": true,
            "size": 589,
            "uid": 115,
            "version": null,
            "wgrp": false,
            "woth": false,
            "writeable": false,
            "wusr": true,
            "xgrp": false,
            "xoth": false,
            "xusr": false
        }
    },
    "msg": "All assertions passed"
}
ok: [localhost] => (item={'changed': False, 'stat': {'exists': True, 'path': '/live/persistence/TailsData_unlocked/openssh-client', 'mode': '0700', 'isdir': True, 'ischr': False, 'isblk': False, 'isreg': False, 'isfifo': False, 'islnk': False, 'issock': False, 'uid': 1000, 'gid': 1000, 'size': 4096, 'inode': 393217, 'dev': 65024, 'nlink': 2, 'atime': 1737146092.8320084, 'mtime': 1737146093.2400084, 'ctime': 1737146093.2400084, 'wusr': True, 'rusr': True, 'xusr': True, 'wgrp': False, 'rgrp': False, 'xgrp': False, 'woth': False, 'roth': False, 'xoth': False, 'isuid': False, 'isgid': False, 'blocks': 8, 'block_size': 4096, 'device_type': 0, 'readable': True, 'writeable': True, 'executable': True, 'pw_name': 'amnesia', 'gr_name': 'amnesia', 'mimetype': 'inode/directory', 'charset': 'binary', 'version': '3083046043', 'attributes': ['extents'], 'attr_flags': 'e'}, 'invocation': {'module_args': {'path': '/live/persistence/TailsData_unlocked/openssh-client', 'follow': False, 'get_md5': False, 'get_checksum': True, 'get_mime': True, 'get_attributes': True, 'checksum_algorithm': 'sha1'}}, 'failed': False, 'item': '/live/persistence/TailsData_unlocked/openssh-client', 'ansible_loop_var': 'item'}) => {
    "ansible_loop_var": "item",
    "changed": false,
    "item": {
        "ansible_loop_var": "item",
        "changed": false,
        "failed": false,
        "invocation": {
            "module_args": {
                "checksum_algorithm": "sha1",
                "follow": false,
                "get_attributes": true,
                "get_checksum": true,
                "get_md5": false,
                "get_mime": true,
                "path": "/live/persistence/TailsData_unlocked/openssh-client"
            }
        },
        "item": "/live/persistence/TailsData_unlocked/openssh-client",
        "stat": {
            "atime": 1737146092.8320084,
            "attr_flags": "e",
            "attributes": [
                "extents"
            ],
            "block_size": 4096,
            "blocks": 8,
            "charset": "binary",
            "ctime": 1737146093.2400084,
            "dev": 65024,
            "device_type": 0,
            "executable": true,
            "exists": true,
            "gid": 1000,
            "gr_name": "amnesia",
            "inode": 393217,
            "isblk": false,
            "ischr": false,
            "isdir": true,
            "isfifo": false,
            "isgid": false,
            "islnk": false,
            "isreg": false,
            "issock": false,
            "isuid": false,
            "mimetype": "inode/directory",
            "mode": "0700",
            "mtime": 1737146093.2400084,
            "nlink": 2,
            "path": "/live/persistence/TailsData_unlocked/openssh-client",
            "pw_name": "amnesia",
            "readable": true,
            "rgrp": false,
            "roth": false,
            "rusr": true,
            "size": 4096,
            "uid": 1000,
            "version": "3083046043",
            "wgrp": false,
            "woth": false,
            "writeable": true,
            "wusr": true,
            "xgrp": false,
            "xoth": false,
            "xusr": true
        }
    },
    "msg": "All assertions passed"
}
ok: [localhost] => (item={'changed': False, 'stat': {'exists': True, 'path': '/home/amnesia/Persistent/securedrop', 'mode': '0755', 'isdir': True, 'ischr': False, 'isblk': False, 'isreg': False, 'isfifo': False, 'islnk': False, 'issock': False, 'uid': 1000, 'gid': 1000, 'size': 4096, 'inode': 1068759, 'dev': 65024, 'nlink': 21, 'atime': 1737145951.356003, 'mtime': 1733943511.6200135, 'ctime': 1733943511.6200135, 'wusr': True, 'rusr': True, 'xusr': True, 'wgrp': False, 'rgrp': True, 'xgrp': True, 'woth': False, 'roth': True, 'xoth': True, 'isuid': False, 'isgid': False, 'blocks': 8, 'block_size': 4096, 'device_type': 0, 'readable': True, 'writeable': True, 'executable': True, 'pw_name': 'amnesia', 'gr_name': 'amnesia', 'mimetype': 'inode/directory', 'charset': 'binary', 'version': '233108564', 'attributes': ['extents'], 'attr_flags': 'e'}, 'invocation': {'module_args': {'path': '/home/amnesia/Persistent/securedrop', 'follow': False, 'get_md5': False, 'get_checksum': True, 'get_mime': True, 'get_attributes': True, 'checksum_algorithm': 'sha1'}}, 'failed': False, 'item': '/home/amnesia/Persistent/securedrop', 'ansible_loop_var': 'item'}) => {
    "ansible_loop_var": "item",
    "changed": false,
    "item": {
        "ansible_loop_var": "item",
        "changed": false,
        "failed": false,
        "invocation": {
            "module_args": {
                "checksum_algorithm": "sha1",
                "follow": false,
                "get_attributes": true,
                "get_checksum": true,
                "get_md5": false,
                "get_mime": true,
                "path": "/home/amnesia/Persistent/securedrop"
            }
        },
        "item": "/home/amnesia/Persistent/securedrop",
        "stat": {
            "atime": 1737145951.356003,
            "attr_flags": "e",
            "attributes": [
                "extents"
            ],
            "block_size": 4096,
            "blocks": 8,
            "charset": "binary",
            "ctime": 1733943511.6200135,
            "dev": 65024,
            "device_type": 0,
            "executable": true,
            "exists": true,
            "gid": 1000,
            "gr_name": "amnesia",
            "inode": 1068759,
            "isblk": false,
            "ischr": false,
            "isdir": true,
            "isfifo": false,
            "isgid": false,
            "islnk": false,
            "isreg": false,
            "issock": false,
            "isuid": false,
            "mimetype": "inode/directory",
            "mode": "0755",
            "mtime": 1733943511.6200135,
            "nlink": 21,
            "path": "/home/amnesia/Persistent/securedrop",
            "pw_name": "amnesia",
            "readable": true,
            "rgrp": true,
            "roth": true,
            "rusr": true,
            "size": 4096,
            "uid": 1000,
            "version": "233108564",
            "wgrp": false,
            "woth": false,
            "writeable": true,
            "wusr": true,
            "xgrp": true,
            "xoth": true,
            "xusr": true
        }
    },
    "msg": "All assertions passed"
}

TASK [validate : Check for v3 SSH auth files] **********************************
ok: [localhost] => (item=app-ssh.auth_private)
ok: [localhost] => (item=mon-ssh.auth_private)

TASK [validate : Count the number of v3 SSH auth files] ************************
ok: [localhost]

TASK [validate : Check for Journalist client auth file] ************************
ok: [localhost]

TASK [validate : Check for Source THS file] ************************************
ok: [localhost]

TASK [validate : Check for Tor v3 key file] ************************************
ok: [localhost]

TASK [validate : Confirm that a valid set of SSH auth files is present] ********
ok: [localhost] => {
    "changed": false,
    "msg": "All assertions passed"
}

TASK [validate : Confirm that the Journalist auth file is present] *************
ok: [localhost] => {
    "changed": false,
    "msg": "All assertions passed"
}

PLAY [Prepare servers for installation] ****************************************
[WARNING]: raw module does not support the environment keyword
[WARNING]: raw module does not support the environment keyword

TASK [prepare-servers : Install python and packages required by installer] *****
ok: [app]
changed: [mon]

TASK [prepare-servers : Check SecureBoot status] *******************************
ok: [mon]
ok: [app]

TASK [prepare-servers : Verify that SecureBoot is not enabled] *****************
ok: [app] => {
    "changed": false,
    "msg": "All assertions passed"
}
ok: [mon] => {
    "changed": false,
    "msg": "All assertions passed"
}

TASK [prepare-servers : Remove cloud-init and ufw] *****************************
ok: [app]
changed: [mon]

TASK [prepare-servers : Ensure dist-upgrade before SecureDrop install] *********
ok: [app]
ok: [mon]

PLAY [Add FPF apt repository and install base packages.] ***********************

TASK [Gathering Facts] *********************************************************
ok: [mon]
ok: [app]

TASK [Check if install has been done before] ***********************************
ok: [app]
ok: [mon -> app(10.20.2.2)]

TASK [Include restrict role early when using ssh over localnet] ****************

TASK [restrict-direct-access : include_tasks] **********************************
included: /home/amnesia/Persistent/securedrop/install_files/ansible-base/roles/restrict-direct-access/tasks/dh_moduli.yml for app, mon

TASK [restrict-direct-access : Check whether Diffie-Hellman groups have been updated] ***
ok: [app]
ok: [mon]

TASK [restrict-direct-access : include_tasks] **********************************
included: /home/amnesia/Persistent/securedrop/install_files/ansible-base/roles/restrict-direct-access/tasks/ssh.yml for app, mon

TASK [restrict-direct-access : Copy SSH client config file.] *******************
ok: [app]
changed: [mon]

TASK [restrict-direct-access : Copy SSH server config file.] *******************
ok: [app]
changed: [mon]

TASK [restrict-direct-access : Remove cloud-init's sshd_config.d] **************
ok: [app]
changed: [mon]

TASK [restrict-direct-access : Copy pam common-auth config file.] **************
changed: [app]
changed: [mon]

TASK [restrict-direct-access : Ensure sshd is running.] ************************
ok: [app]
ok: [mon]

TASK [restrict-direct-access : include_tasks] **********************************
included: /home/amnesia/Persistent/securedrop/install_files/ansible-base/roles/restrict-direct-access/tasks/iptables.yml for app, mon

TASK [restrict-direct-access : Gather localhost facts first] *******************
ok: [app -> localhost]
ok: [mon -> localhost]

TASK [restrict-direct-access : Determine local platform specific routing info] ***
ok: [app]
ok: [mon]

TASK [restrict-direct-access : Record admin network interface] *****************
ok: [app]
ok: [mon]

TASK [restrict-direct-access : Hacky work-around for Mac/Linux interface structure divergence] ***
ok: [app]
ok: [mon]
[DEPRECATION WARNING]: Use 'ansible.utils.ipaddr' module instead. This feature 
will be removed from ansible.netcommon in a release after 2024-01-01. 
Deprecation warnings can be disabled by setting deprecation_warnings=False in 
ansible.cfg.
[DEPRECATION WARNING]: Use 'ansible.utils.ipaddr' module instead. This feature 
will be removed from ansible.netcommon in a release after 2024-01-01. 
Deprecation warnings can be disabled by setting deprecation_warnings=False in 
ansible.cfg.

TASK [restrict-direct-access : Compute admin network CIDR] *********************
ok: [mon -> localhost]
ok: [app -> localhost]
[DEPRECATION WARNING]: Use 'ansible.utils.ipaddr' module instead. This feature 
will be removed from ansible.netcommon in a release after 2024-01-01. 
Deprecation warnings can be disabled by setting deprecation_warnings=False in 
ansible.cfg.
[DEPRECATION WARNING]: Use 'ansible.utils.ipaddr' module instead. This feature 
will be removed from ansible.netcommon in a release after 2024-01-01. 
Deprecation warnings can be disabled by setting deprecation_warnings=False in 
ansible.cfg.

TASK [restrict-direct-access : Copy IPv4 iptables rules.] **********************
ok: [app]
fatal: [mon]: FAILED! => {"changed": false, "checksum": "3f7cbe1abe35f31dfe8e8aaccbca7f68afe4c0a2", "msg": "Destination directory /etc/iptables does not exist"}

NO MORE HOSTS LEFT *************************************************************

NO MORE HOSTS LEFT *************************************************************

PLAY RECAP *********************************************************************
app                        : ok=22   changed=1    unreachable=0    failed=0    skipped=3    rescued=0    ignored=0   
localhost                  : ok=13   changed=0    unreachable=0    failed=0    skipped=1    rescued=0    ignored=0   
mon                        : ok=21   changed=6    unreachable=0    failed=1    skipped=3    rescued=0    ignored=0   

ERROR (run with -v for more): Command '['ansible-playbook', "--scp-extra-args='-O'", '/home/amnesia/Persistent/securedrop/install_files/ansible-base/securedrop-prod.yml', '--ask-become-pass']' returned non-zero exit status 2.

@legoktm
Copy link
Member

legoktm commented Jan 17, 2025

Thanks, I see the bug. iptables-persistent is installed in the "Install base apt dependencies" step, in the common role, but if you're using ssh-over-local network, then restrict-direct-access goes first, before iptables is installed. Easy fix is to explicitly install iptables-persistent right before we drop stuff into /etc/iptables.

I did multiple 2.11.1 installs today, so let me try to figure out why I'm not hitting this!

@zenmonkeykstop
Copy link
Contributor

Do you think this merits a hotfix? Fresh installs being busted is bad, though less so if it's ssh-over-local, which is not the preferred config.

@zenmonkeykstop
Copy link
Contributor

Also fwiw, testing for the past 2 releases for me was over ssh-over-local and I didn't hit this.

@legoktm
Copy link
Member

legoktm commented Jan 17, 2025 •
edited
Loading

Just to clarify Nathan, in this case you already had a provisioned app server, and you were just reinstalling mon, right?

The logic is:

    - name: Check if install has been done before
      stat:
        path: /var/www/securedrop
      delegate_to: "{{ groups['securedrop_application_server'][0] }}"
      register: sd_dir_check

    - name: Include restrict role early when using ssh over localnet
      include_role:
        name: restrict-direct-access
      vars:
        # Don't wait for tor client auth, might not exist yet
        fetch_tor_client_auth_configs: false
      when:
        - not enable_ssh_over_tor
        - sd_dir_check.stat.exists

So restrict-direct-access is run early only when 1) ssh over local network AND 2) /var/www/securedrop already exists on the app server. If you're doing a clean install on both app+mon, there should be no issue.

I did multiple 2.11.1 installs today, so let me try to figure out why I'm not hitting this!

Ironically, I've had my mon server setup, and solely been reinstalling my app server, with no issues (i.e. the opposite of what I think you were doing).

@nathandyer
Copy link
Contributor Author

@legoktm Wow, what are the chances! :) Yes that's correct, I was only reinstalling mon, with my long-running app server remaining in-place.

@legoktm legoktm changed the title Investigate a potentially missing depdendency Reinstalling mon with existing app server triggers missing iptables-persistent dependency Jan 17, 2025
legoktm added a commit that referenced this issue Jan 17, 2025
@legoktm
Ensure /etc/iptables exists before writing to it
96a0f03 
In the specific case of installing a fresh mon server when the app
server is already configured AND you're using ssh over the local
network, we'll try to write to /etc/iptables before the
iptables-persistent package is installed.

This is because we end up running the restrict-direct-access role before
the common role, which installs the base packages.

The easy fix is to install iptables-persistent ahead of time if we see
that it's necessary.

Fixes #7119.
@legoktm legoktm mentioned this issue Jan 17, 2025
Merged
5 tasks
@legoktm legoktm self-assigned this Jan 17, 2025
@legoktm legoktm moved this to In Progress in SecureDrop dev cycle Jan 17, 2025
@legoktm legoktm added this to the SecureDrop 2.12.0 milestone Jan 17, 2025
legoktm added a commit that referenced this issue Jan 17, 2025
@legoktm
Ensure /etc/iptables exists before writing to it
8efcabd 
In the specific case of installing a fresh mon server when the app
server is already configured AND you're using ssh over the local
network, we'll try to write to /etc/iptables before the
iptables-persistent package is installed.

This is because we end up running the restrict-direct-access role before
the common role, which installs the base packages.

The easy fix is to install iptables-persistent ahead of time if we see
that it's necessary.

Fixes #7119.
@cfm cfm closed this as completed in #7417 Jan 24, 2025
@github-project-automation github-project-automation bot moved this from In Progress to Done in SecureDrop dev cycle Jan 24, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@legoktm legoktm

Labels
None yet
Projects
SecureDrop dev cycle
Archived in project
Milestone
SecureDrop 2.12.0
Development

Successfully merging a pull request may close this issue.

Ensure /etc/iptables exists before writing to it freedomofpress/securedrop
3 participants
@legoktm @zenmonkeykstop @nathandyer