Splunk 9.1.3 Startup Errors

[dpericaxon]

Hello,

We recently upgraded from 9.0.7 to 9.1.3 for the Universal Forwarder using the image provided here

The errors we see are:

TASK [splunk_universal_forwarder : Debug hec_token] ****************************
ok: [localhost] => {
    "hec_token": {
        "changed": true,
        "excep_str": "No Exception",
        "failed": false,
        "json": {
            "messages": [
                {
                    "text": "Could not find object id=http://splunk_hec_token/",
                    "type": "ERROR"
                }
            ]
        },
        "status": 404
    }
}

Which looks like it gets resolved with:

TASK [splunk_universal_forwarder : Debug create_hec_token] *********************
ok: [localhost] => {
    "create_hec_token": {
        "changed": false,
        "skip_reason": "Conditional result was False",
        "skipped": true
    }
}

We're not using HEC in this case so I think this error can be ignored.

We then see this error happen twice:

included: /opt/ansible/roles/splunk_universal_forwarder/tasks/../../../roles/splunk_common/tasks/check_for_required_restarts.yml for localhost
Tuesday 12 March 2024  22:25:58 +0000 (0:00:00.042)       0:00:32.858 ********* 

TASK [splunk_universal_forwarder : Check for required restarts] ****************
fatal: [localhost]: FAILED! => {}

MSG:

The conditional check 'restart_required.content' failed. The error was: error while evaluating conditional (restart_required.content): 'dict object' has no attribute 'content'
...ignoring
Tuesday 12 March 2024  22:25:58 +0000 (0:00:00.300)       0:00:33.159 ********* 

TASK [splunk_universal_forwarder : debug] **************************************
ok: [localhost] => (item=failed) => {}

MSG:

Hello World - 
ok: [localhost] => (item=msg) => {}

MSG:

Hello World - 
Tuesday 12 March 2024  22:25:58 +0000 (0:00:00.071)       0:00:33.230 ********* 
Tuesday 12 March 2024  22:25:58 +0000 (0:00:00.039)       0:00:33.269 ********* 

TASK [Check all instances for required restarts] *******************************
included: /opt/ansible/roles/splunk_common/tasks/check_for_required_restarts.yml for localhost
Tuesday 12 March 2024  22:25:58 +0000 (0:00:00.050)       0:00:33.320 ********* 

TASK [Check for required restarts] *********************************************
fatal: [localhost]: FAILED! => {}

MSG:

The conditional check 'restart_required.content' failed. The error was: error while evaluating conditional (restart_required.content): 'dict object' has no attribute 'content'
...ignoring
Tuesday 12 March 2024  22:25:58 +0000 (0:00:00.305)       0:00:33.625 ********* 

TASK [debug] *******************************************************************
ok: [localhost] => (item=failed) => {}

MSG:

Hello World - 
ok: [localhost] => (item=msg) => {}

MSG:

Hello World - 
Tuesday 12 March 2024  22:25:58 +0000 (0:00:00.058)       0:00:33.683 ********* 

PLAY RECAP *********************************************************************
localhost                  : ok=108  changed=22   unreachable=0    failed=0    skipped=80   rescued=0    ignored=2   

We still see events being forwarded and things appear to look okay. Is there anything else I should check or be concerned about?

[adityapinglesf]

@dpericaxon looking into the issue here, thanks for reporting