Trouble with network.system #6922

smithjm · 2013-08-27T21:12:04Z

I'm trying to use the network module on a Fedora 17 host with a very simple config, using salt 0.16.3 on both the master and minion. The interface applies correctly, but the network.system fails.

/srv/base/net/init.sls:

system:
  network.system:
    - enabled: true
    - hostname: myhost.someplace.com
    - gateway: 10.198.19.254
    - gatewaydev: eth0
    - nozeroconf: true
    - requires_reboot: true

eth0:
  network.managed:
    - enabled: true
    - type: eth
    - proto: none
    - ipaddr: 10.198.16.45
    - netmask:  255.255.252.0
    - dns:
      - 10.192.0.1
      - 10.209.0.233

This seems to match the online documentation pretty precisely, however, when I try to apply the state to the host, I get the following error:

myhost.someplace.com:

State: - network
Name:      eth0
Function:  managed
    Result:    True
    Comment:   Interface eth0 is up to date.
    Changes:   

State: - network
Name:      system
Function:  system
    Result:    False
    Comment:   An exception occurred in this state: Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/salt/state.py", line 1237, in call
    *cdata['args'], **cdata['kwargs'])
  File "/usr/lib/python2.7/site-packages/salt/states/network.py", line 350, in system
    new = __salt__['ip.build_network_settings'](**kwargs)
  File "/usr/lib/python2.7/site-packages/salt/modules/rh_ip.py", line 1017, in build_network_settings
    opts = _parse_network_settings(settings, current_network_settings)
  File "/usr/lib/python2.7/site-packages/salt/modules/rh_ip.py", line 682, in _parse_network_settings
    _raise_error_network('networking', valid)
  File "/usr/lib/python2.7/site-packages/salt/modules/rh_ip.py", line 732, in _raise_error_network
    msg = _error_msg_network(option, expected)
  File "/usr/lib/python2.7/site-packages/salt/modules/rh_ip.py", line 98, in _error_msg_network
    return msg.format(option, '|'.join(expected))
TypeError: sequence item 4: expected string, bool found

    Changes:

The text was updated successfully, but these errors were encountered:

basepi · 2013-08-27T21:28:15Z

So even though we return an error, the settings were actually applied properly? This is probably a sorting issue in the expected-vs-found settings comparison, plus a small type mismatch (the bool error). Shouldn't be too difficult to fix.

Thanks for the report!

smithjm · 2013-08-27T21:33:57Z

Actually, the error is on "network.system" which I don't think actually gets applied. However, the "network.managed" portion applies correctly (defining eth0). Thanks!

smithjm · 2013-08-28T12:44:56Z

Interestingly, when I comment out the three lines that have boolean values like so:

system:
  network.system:
    {# - enabled: true #}
    - hostname: myhost.someplace.com
    - gateway: 10.198.19.254
    - gatewaydev: eth0
    {# - nozeroconf: true #}
    {# - requires_reboot: true #}

I still get:

State: - network
Name:      system
Function:  system
    Result:    False
        Comment:   An exception occurred in this state: Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/salt/state.py", line 1237, in call
    *cdata['args'], **cdata['kwargs'])
  File "/usr/lib/python2.7/site-packages/salt/states/network.py", line 350, in system
    new = __salt__['ip.build_network_settings'](**kwargs)
  File "/usr/lib/python2.7/site-packages/salt/modules/rh_ip.py", line 1017, in build_network_settings
    opts = _parse_network_settings(settings, current_network_settings)
  File "/usr/lib/python2.7/site-packages/salt/modules/rh_ip.py", line 682, in _parse_network_settings
    _raise_error_network('networking', valid)
  File "/usr/lib/python2.7/site-packages/salt/modules/rh_ip.py", line 732, in _raise_error_network
    msg = _error_msg_network(option, expected)
  File "/usr/lib/python2.7/site-packages/salt/modules/rh_ip.py", line 98, in _error_msg_network
    return msg.format(option, '|'.join(expected))
TypeError: sequence item 4: expected string, bool found

        Changes:

smithjm · 2013-08-28T13:24:04Z

I have now successfully configured a much more complex host, using bonded interfaces, vlan bonding, and a bridge, without errors (by commenting out the entire network.system section and managing /etc/sysconfig/network by hand). This confirms two things:

the network.managed interface, bond, and bridge code is fine
the network.system code is where the problem is (which we already knew)

smithjm · 2013-12-04T21:01:01Z

Just wanted to confirm I'm still seeing this in 0.17.2, against both Fedora 17 and Fedora 19:

system:
network.system:
- enabled: True
- hostname: {{ grains['fqdn'] }}
- gateway: {{ gwinfo['gateway'] }}
- gatewaydev: {{ gwdev }}
- nozeroconf: True
- requires_reboot: True

renders on one of my minions to:

system:
network.system:
- enabled: true
- hostname: somename.our.domain.com
- gateway: 10.0.0.1
- gatewaydev: eth0
- nozeroconf: True
- requires_reboot: True

which in turn generates:

State: - network
Name:      system
Function:  system
    Result:    False
    Comment:   An exception occurred in this state: Traceback (most recent call last):

File "/usr/lib/python2.7/site-packages/salt/state.py", line 1278, in call
_cdata['args'], *_cdata['kwargs'])
File "/usr/lib/python2.7/site-packages/salt/states/network.py", line 348, in system
new = salt'ip.build_network_settings'
File "/usr/lib/python2.7/site-packages/salt/modules/rh_ip.py", line 1041, in build_network_settings
opts = _parse_network_settings(settings, current_network_settings)
File "/usr/lib/python2.7/site-packages/salt/modules/rh_ip.py", line 684, in _parse_network_settings
_raise_error_network('networking', valid)
File "/usr/lib/python2.7/site-packages/salt/modules/rh_ip.py", line 734, in _raise_error_network
msg = _error_msg_network(option, expected)
File "/usr/lib/python2.7/site-packages/salt/modules/rh_ip.py", line 100, in _error_msg_network
return msg.format(option, '|'.join(expected))
TypeError: sequence item 4: expected string, bool found

    Changes:

The rest of the network state still applies correctly as before, but the 'network.system' state does not apply.

sroegner · 2014-01-14T16:22:11Z

I had the same problem on CentOS 6 and was able to work around it - replace
- enabled: True
with
- networking: True
and the state will be applied as expected

cachedout · 2014-05-16T22:10:54Z

I can no longer replicate this bug in 2014.1.4. I would like to close this unless there are more recent reports of this failing.

The-Loeki · 2015-05-28T12:09:28Z

Replicatable on Ubuntu 14.04 w/ salt 2015.5 and no /etc/default/networking file;

Function: network.system
Result: False
Comment: An exception occurred in this state: Traceback (most recent call last):
           File "/usr/lib/python2.7/dist-packages/salt/state.py", line 1563, in call
             **cdata['kwargs'])
           File "/usr/lib/python2.7/dist-packages/salt/states/network.py", line 425, in system
             new = __salt__['ip.build_network_settings'](**kwargs)
           File "/usr/lib/python2.7/dist-packages/salt/modules/debian_ip.py", line 1805, in build_network_settings
             opts = _parse_network_settings(settings, current_network_settings)
           File "/usr/lib/python2.7/dist-packages/salt/modules/debian_ip.py", line 1288, in _parse_network_settings
             _raise_error_network('networking', valid)
           File "/usr/lib/python2.7/dist-packages/salt/modules/debian_ip.py", line 200, in _raise_error_network
             msg = _error_msg_network(option, expected)
           File "/usr/lib/python2.7/dist-packages/salt/modules/debian_ip.py", line 179, in _error_msg_network
             return msg.format(option, '|'.join(expected))
         TypeError: sequence item 4: expected string, bool found

Cause of the trace

The trace is caused because CONFIG_TRUE and/or CONFIG_FALSE are passed on to the expected var; These contain actual bools True / False, tripping over the '|'.join.

@rallytime / @cachedout: Putting those two bools behind quotes fixes the trace, but I'm very far from sure those aren't needed for value testing.
The alternative would of course be something like '|'.join([str(expect) for expect in expected]).
Also, I haven't checked the other ip modules

Cause of the cause of the trace

_parse_current_network_settings initializes opts['networking'] = '', then proceeds to look for CONFIGURE_INTERFACES in /etc/default/networking.
Hence if that file or that var in that file can't be read, the value will be wrong and the message will be thrown.
Properly of course it'd contain False or True depending on your world view of things, which leads to my aforementioned note of value testing (in which case it probably would need to be 'True' or 'False')

As a side note, setting - networking: True doesn't make a difference for me; I have to fill the CONFIGURE_INTERFACES with something valid

The-Loeki · 2015-07-03T15:46:50Z

@rallytime bump?

rallytime · 2015-07-03T18:36:46Z

@The-Loeki Since the code has changed so much since 2014.1.4 and 2015.5 (and since @cachedout wasn't able to reproduce it) I suspect this might be something different/new. Would you mind opening a new request with all of the details you posted above?

The-Loeki · 2015-07-06T09:48:24Z

@rallytime: The cause of the trace lies in the fact that the CONFIG_TRUE and CONFIG_FALSE variables contain bool True/False; they trip over '|'.join. This hasn't changed and is the direct cause of all these traces whenever CONFIG_TRUE or CONFIG_FALSE are tested.
AFAIK this behaviour is the same on at least 2014+, and the same piece of code exists in both rhip and debian_ip.

Now as to why that test is triggered is a different discussion and an apparent semi-rarity; the one I documented is again reproducable with little to no reason to assume this has changed between 2014.1 and 2015.5
Also, although it's a different platform, notice the problem here is exactly the same; check network.system trips up violently if not provided with a bool somewhere in the config files; the only difference is that sroegners workaround (a formula-configured override) works for RHEL's, and not for Debians.

All in all there's two different components to this problem which need addressing; the '|'.join should deal with bool and/or the bool vals need to become a string (I would've patched, but I can't tell which one is the most reasonable here),

and, since /etc/default/networking can not be assumed to be there this shouldn't be cause for a trace but rather for a sensible (formula-supplied?) default or a decent error message.

The-Loeki · 2015-08-04T10:44:42Z

I've now got a CentOS 7 machine clean install exhibiting a variant of this behaviour (the first part of this bug is very similar).

  File "/usr/lib/python2.7/site-packages/salt/state.py", line 1561, in call
    **cdata['kwargs'])
  File "/usr/lib/python2.7/site-packages/salt/states/network.py", line 425, in system
    new = __salt__['ip.build_network_settings'](**kwargs)
  File "/usr/lib/python2.7/site-packages/salt/modules/rh_ip.py", line 1097, in build_network_settings
    opts = _parse_network_settings(settings, current_network_settings)
  File "/usr/lib/python2.7/site-packages/salt/modules/rh_ip.py", line 715, in _parse_network_settings
    opts['networking'] = current['networking']
KeyError: 'networking'

The workaround in this case is the exact opposite of @sroegner; enabled: True won't check for current['networking'] therefore won't throw the KeyError

Problem is again that /etc/sysconfig/network remains empty, therefore the current dict as well.

dsnopek · 2017-03-29T18:43:31Z

I'm seeing this issue with a very simple state that actually comes straight from the examples in the documentation:

system:
  network.system:
    - hostname: server.example.com
    - apply_hostname: True
    - retain_settings: True

This is on Debian 8 with Salt 2016.11.3

dsnopek · 2017-03-29T18:52:55Z

Er, went through the stack trace which is this:

Traceback (most recent call last):
                File "/usr/lib/python2.7/dist-packages/salt/state.py", line 1746, in call
                  **cdata['kwargs'])
                File "/usr/lib/python2.7/dist-packages/salt/loader.py", line 1703, in wrapper
                  return f(*args, **kwargs)
                File "/usr/lib/python2.7/dist-packages/salt/states/network.py", line 528, in system
                  new = __salt__['ip.build_network_settings'](**kwargs)
                File "/usr/lib/python2.7/dist-packages/salt/modules/debian_ip.py", line 1998, in build_network_settings
                  opts = _parse_network_settings(settings, current_network_settings)
                File "/usr/lib/python2.7/dist-packages/salt/modules/debian_ip.py", line 1445, in _parse_network_settings
                  _raise_error_network('networking', valid)
                File "/usr/lib/python2.7/dist-packages/salt/modules/debian_ip.py", line 203, in _raise_error_network
                  msg = _error_msg_network(option, expected)
                File "/usr/lib/python2.7/dist-packages/salt/modules/debian_ip.py", line 182, in _error_msg_network
                  return msg.format(option, '|'.join(expected))
              TypeError: sequence item 4: expected string, bool found

And found that this was a failed attempt to build an error message. The actual error appears to be no value for the 'networking' option, which I guess comes from 'enabled' in the state. If I change my state to:

system:
  network.system:
    - enabled: True
    - hostname: server.example.com
    - apply_hostname: True
    - retain_settings: True

... then it doesn't error out! (Still haven't tested if it does what I want yet, though.) This could probably use a documentation fix!

nixjdm · 2017-08-11T20:12:02Z

I'm not sure what all enabled is doing for network.system, but it does seem necessary. Adding that line fixes the issue for me. The docs definitely need updated if enabled is truly necessary, with an explanation. Or else the code needs fixed if it shouldn't really be required.

lastmikoi · 2018-02-05T17:33:08Z

Just stumbled upon that issue under 2016.11.9, I had an empty /etc/sysconfig/network out of a cloud-init instance and it kept failing. @dsnopek's workaround worked perfectly but not before I lost a lot of time trying to troubleshoot that issue on my own :). Having either the docs or the code updated to reflect that behaviour would definitively be helpful.

eliasp · 2018-09-26T15:18:45Z

Still applies to 2018.3.2 on Ubuntu 18.04, where it can be easily reproduced using:

salt-call -l info state.single network.system name=set-hostname hostname=localhost

Resulting in:

[INFO    ] Loading fresh modules for state activity
[INFO    ] Running state [set-hostname] at time 17:10:01.541094
[INFO    ] Executing state network.system for [set-hostname]
[WARNING ] /var/cache/salt/minion/extmods/modules/__pycache__/icinga2.cpython-36.py:27: DeprecationWarning: Use of 'salt.utils.is_windows' detected. This function has been moved to 'salt.utils.platform.is_windows' as of Salt 2018.3.0. This warning will be removed in Salt Neon.[WARNING ] /var/cache/salt/minion/extmods/modules/__pycache__/icinga2.cpython-36.py:30: DeprecationWarning: Use of 'salt.utils.which' detected. This function has been moved to 'salt.utils.path.which' as of Salt 2018.3.0. This warning will be removed in Salt Neon.
[INFO    ] Executing command ['systemctl', 'status', 'networking.service', '-n', '0'] in directory '/root'
[INFO    ] Using existing setting -- Setting: networking Value:
[ERROR   ] An exception occurred in this state: Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/salt/state.py", line 1905, in call
    **cdata['kwargs'])
  File "/usr/lib/python3/dist-packages/salt/loader.py", line 1830, in wrapper
    return f(*args, **kwargs)
  File "/usr/lib/python3/dist-packages/salt/states/network.py", line 564, in system
    new = __salt__['ip.build_network_settings'](**kwargs)
  File "/usr/lib/python3/dist-packages/salt/modules/debian_ip.py", line 2009, in build_network_settings
    opts = _parse_network_settings(settings, current_network_settings)
  File "/usr/lib/python3/dist-packages/salt/modules/debian_ip.py", line 1448, in _parse_network_settings
    _raise_error_network('networking', valid)
  File "/usr/lib/python3/dist-packages/salt/modules/debian_ip.py", line 204, in _raise_error_network
    msg = _error_msg_network(option, expected)
  File "/usr/lib/python3/dist-packages/salt/modules/debian_ip.py", line 183, in _error_msg_network
    return msg.format(option, '|'.join(expected))
TypeError: sequence item 4: expected str instance, bool found

[INFO    ] Completed state [set-hostname] at time 17:10:02.839376 (duration_in_ms=1298.282)
local:
----------
          ID: set-hostname    Function: network.system
      Result: False
     Comment: An exception occurred in this state: Traceback (most recent call last):
                File "/usr/lib/python3/dist-packages/salt/state.py", line 1905, in call
                  **cdata['kwargs'])
                File "/usr/lib/python3/dist-packages/salt/loader.py", line 1830, in wrapper
                  return f(*args, **kwargs)
                File "/usr/lib/python3/dist-packages/salt/states/network.py", line 564, in system
                  new = __salt__['ip.build_network_settings'](**kwargs)
                File "/usr/lib/python3/dist-packages/salt/modules/debian_ip.py", line 2009, in build_network_settings
                  opts = _parse_network_settings(settings, current_network_settings)
                File "/usr/lib/python3/dist-packages/salt/modules/debian_ip.py", line 1448, in _parse_network_settings
                  _raise_error_network('networking', valid)
                File "/usr/lib/python3/dist-packages/salt/modules/debian_ip.py", line 204, in _raise_error_network
                  msg = _error_msg_network(option, expected)
                File "/usr/lib/python3/dist-packages/salt/modules/debian_ip.py", line 183, in _error_msg_network
                  return msg.format(option, '|'.join(expected))
              TypeError: sequence item 4: expected str instance, bool found
     Started: 17:10:01.541094
    Duration: 1298.282 ms
     Changes:

Summary for local
------------
Succeeded: 0
Failed:    1
------------
Total states run:     1
Total run time:   1.298 s

waynew · 2019-01-28T17:53:23Z

Just pushed a fix for the error message producing a stack trace.

cr1st1p · 2019-10-23T11:48:13Z

Salt version 2019.2.0, ubuntu 18.04
Same problem as @dsnopek .
I have a simple state, as documented, to set the hostname. It crashes with the same stacktrace.
If I add the 'enabled: True', it says:

          ID: network_hostname
    Function: network.system
      Result: True
     Comment: Global network settings are up to date.
     Started: 11:41:57.782510
    Duration: 4409.324 ms
     Changes:   
              ----------                                                              
              network_settings:
                  --- 
                  +++ 
                  @@ -1,2 +1,2 @@
                  -NETWORKING=no
                  
                  +NETWORKING=yes
                  
                   HOSTNAME=salt-master-dev

The problems:

I am not very sure it did change the hostname - it was already changed. I have to check on a different setup. UPDATE: it changed it, on a different system.
and, more importantly, I don't get what is about that 'NETWORKING=yes' change. The server HAD networking enabled, by my other states, via 'netplan'.

waynew · 2020-01-24T14:55:12Z

@cr1st1p can you check the neon RC?

waynew · 2020-02-28T21:36:42Z

@cr1st1p I'm going to go ahead and close this for now - if this error is still happening with Salt v3000 please let me know and I'll re-open.

kbaikov · 2020-11-15T03:11:38Z

@rallytime @waynew this error is still happening with Salt 3002.1 on ubuntu 20.04
Fails with the following message:
ID: system
Function: network.system
Result: False
Comment: Invalid network setting -- Setting: networking, Expected: [yes|on|true|1|True|no|off|false|0|False]

Unless enabled: True is added as per comment: #6922 (comment)

rallytime · 2020-11-16T14:53:45Z

Hi @kbaikov - Thanks for the ping! I am no longer involved with the Salt project, so unfortunately I can't reopen this.

@waynew Can you take a look here?

omgitsedwin · 2021-03-25T14:20:36Z

Can confirm error still exists on Ubuntu 18.04 on Salt 3002.6.

basepi modified the milestones: Approved, Outstanding Bugs Apr 21, 2014

cachedout added the Cannot Reproduce label May 16, 2014

thenewwazoo mentioned this issue Aug 19, 2014

State fails when network.system's hostname arg has no dots #15085

Closed

rallytime closed this as completed Nov 6, 2014

rallytime reopened this Jul 6, 2015

waynew mentioned this issue Jan 25, 2019

6922 fix network error errors #51343

Merged

waynew added the fixed-pls-verify fix is linked, bug author to confirm fix label Jan 28, 2019

sagetherage assigned waynew Jan 24, 2020

waynew closed this as completed Feb 28, 2020

OrangeDog mentioned this issue Jun 9, 2020

[Tech-Debt] Avoid use of deprecated net-tools #57541

Closed

Christoph-TKE mentioned this issue May 2, 2022

[BUG] network.system state messes up DNS search domain #62018

Open

6 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Trouble with network.system #6922

Trouble with network.system #6922

smithjm commented Aug 27, 2013

basepi commented Aug 27, 2013

smithjm commented Aug 27, 2013

smithjm commented Aug 28, 2013

smithjm commented Aug 28, 2013

smithjm commented Dec 4, 2013

sroegner commented Jan 14, 2014

cachedout commented May 16, 2014

The-Loeki commented May 28, 2015

The-Loeki commented Jul 3, 2015

rallytime commented Jul 3, 2015

The-Loeki commented Jul 6, 2015

The-Loeki commented Aug 4, 2015

dsnopek commented Mar 29, 2017

dsnopek commented Mar 29, 2017

nixjdm commented Aug 11, 2017

lastmikoi commented Feb 5, 2018

eliasp commented Sep 26, 2018

waynew commented Jan 28, 2019

cr1st1p commented Oct 23, 2019 •

edited

Loading

waynew commented Jan 24, 2020

waynew commented Feb 28, 2020

kbaikov commented Nov 15, 2020

rallytime commented Nov 16, 2020

omgitsedwin commented Mar 25, 2021

Trouble with network.system #6922

Trouble with network.system #6922

Comments

smithjm commented Aug 27, 2013

basepi commented Aug 27, 2013

smithjm commented Aug 27, 2013

smithjm commented Aug 28, 2013

smithjm commented Aug 28, 2013

smithjm commented Dec 4, 2013

which in turn generates:

sroegner commented Jan 14, 2014

cachedout commented May 16, 2014

The-Loeki commented May 28, 2015

Cause of the trace

Cause of the cause of the trace

The-Loeki commented Jul 3, 2015

rallytime commented Jul 3, 2015

The-Loeki commented Jul 6, 2015

The-Loeki commented Aug 4, 2015

dsnopek commented Mar 29, 2017

dsnopek commented Mar 29, 2017

nixjdm commented Aug 11, 2017

lastmikoi commented Feb 5, 2018

eliasp commented Sep 26, 2018

waynew commented Jan 28, 2019

cr1st1p commented Oct 23, 2019 • edited Loading

waynew commented Jan 24, 2020

waynew commented Feb 28, 2020

kbaikov commented Nov 15, 2020

rallytime commented Nov 16, 2020

omgitsedwin commented Mar 25, 2021

cr1st1p commented Oct 23, 2019 •

edited

Loading