[hostcfg]Sometimes has_asic_per_scope is not rendered and stays as jinja template in running config

[dgsudharsan]

Description

The below PRs introduced the concept of has_asic_per_scope field present as a template and when hostcfgd reads, it renders the template and write value back to redis CONFIG_DB.

https://github.com/sonic-net/sonic-host-services/blob/7ab097843ead1376e9eed28f07f36670ebbb4066/scripts/hostcfgd#L326

sonic-net/sonic-host-services#14
#11796

However, in our testing we rarely see an issue where the jina template is not rendered.

21:28:07 __init__.load_config_facts               L0228 INFO   | Getting config basic facts 

21:28:10 __init__.load_config_facts               L0234 DEBUG  | raw config basic facts: 

r-boxer-sw01 | SUCCESS => { 


            "lldp": { 

                "auto_restart": "enabled", 

                "has_global_scope": "True", 

                "has_per_asic_scope": "{% if not DEVICE_RUNTIME_METADATA['ETHERNET_PORTS_PRESENT'] or ('CHASSIS_METADATA' in DEVICE_RUNTIME_METADATA and DEVICE_RUNTIME_METADATA['CHASSIS_METADATA']['module_type'] in ['supervisor']) %}False{% else %}True{% endif %}", 

                "has_timer": "True", 

                "high_mem_alert": "disabled", 

                "state": "enabled", 

                "support_syslog_rate_limit": "true" 

            }, 

These are the sequence of events inferred from the logs
May 15 21:16:43.647038 r-boxer-sw01 NOTICE sudo: admin : TTY=pts/0 ; PWD=/home/admin ; USER=root ; COMMAND=/usr/local/bin/config ztp disable -y
May 15 21:16:46.644863 r-boxer-sw01 NOTICE sudo: admin : TTY=pts/0 ; PWD=/home/admin ; USER=root ; COMMAND=/usr/local/bin/config save -y
May 15 21:17:51.619898 r-boxer-sw01 NOTICE sudo: admin : TTY=pts/0 ; PWD=/home/admin ; USER=root ; COMMAND=/usr/local/bin/config qos reload
May 15 21:17:53.671974 r-boxer-sw01 INFO hostcfgd: Feature lldp is enabled and started
May 15 21:18:06.864260 r-boxer-sw01 NOTICE sudo: admin : TTY=pts/0 ; PWD=/home/admin ; USER=root ; COMMAND=/usr/local/bin/config save -y

Steps to reproduce the issue:

Describe the results you received:

has_per_asic_scope is not rendered

Describe the results you expected:

has_per_asic_scope should be rendered in running configuration.

Output of show version:

SONiC Software Version: SONiC.202211_RC14.1-7a20284d5_Internal
SONiC OS Version: 11
Distribution: Debian 11.7
Kernel: 5.10.0-18-2-amd64
Build commit: 7a20284d5
Build date: Mon May 15 08:25:27 UTC 2023
Built by: sw-r2d2-bot@r-build-sonic-ci03-241

Platform: x86_64-mlnx_msn2010-r0
HwSKU: ACS-MSN2010
ASIC: mellanox
ASIC Count: 1
Serial Number: MT1749X10061
Model Number: MSN2010-CB2F
Hardware Revision: A1
Uptime: 10:45:32 up  5:03,  1 user,  load average: 1.73, 1.29, 0.83
Date: Tue 16 May 2023 10:45:32

Docker images:
REPOSITORY                    TAG                                IMAGE ID       SIZE
docker-orchagent              202211_RC14.1-7a20284d5_Internal   0fc64fc322d8   474MB
docker-orchagent              latest                             0fc64fc322d8   474MB
docker-fpm-frr                202211_RC14.1-7a20284d5_Internal   d67edfed605b   485MB
docker-fpm-frr                latest                             d67edfed605b   485MB
docker-teamd                  202211_RC14.1-7a20284d5_Internal   c7ce112666a0   455MB
docker-teamd                  latest                             c7ce112666a0   455MB
docker-macsec                 latest                             2bb3b235239d   457MB
docker-syncd-mlnx             202211_RC14.1-7a20284d5_Internal   32a8efca57d4   875MB
docker-syncd-mlnx             latest                             32a8efca57d4   875MB
docker-platform-monitor       202211_RC14.1-7a20284d5_Internal   e25ac509c6a1   876MB
docker-platform-monitor       latest                             e25ac509c6a1   876MB
docker-snmp                   202211_RC14.1-7a20284d5_Internal   7eaa7cdd0480   484MB
docker-snmp                   latest                             7eaa7cdd0480   484MB
docker-sonic-telemetry        202211_RC14.1-7a20284d5_Internal   ae709d1c90e1   737MB
docker-sonic-telemetry        latest                             ae709d1c90e1   737MB
docker-dhcp-relay             latest                             b2a6609fdebc   448MB
docker-lldp                   202211_RC14.1-7a20284d5_Internal   b08776442c76   481MB
docker-lldp                   latest                             b08776442c76   481MB
docker-mux                    202211_RC14.1-7a20284d5_Internal   587126b4cf8a   487MB
docker-mux                    latest                             587126b4cf8a   487MB
docker-sonic-p4rt             202211_RC14.1-7a20284d5_Internal   95e9f399dad7   521MB
docker-sonic-p4rt             latest                             95e9f399dad7   521MB
docker-router-advertiser      202211_RC14.1-7a20284d5_Internal   c5c079884edd   439MB
docker-router-advertiser      latest                             c5c079884edd   439MB
docker-database               202211_RC14.1-7a20284d5_Internal   e6c14ef7c403   439MB
docker-database               latest                             e6c14ef7c403   439MB
docker-eventd                 202211_RC14.1-7a20284d5_Internal   d6459ab971ab   438MB
docker-eventd                 latest                             d6459ab971ab   438MB
docker-sonic-mgmt-framework   202211_RC14.1-7a20284d5_Internal   54ed482e06dc   551MB
docker-sonic-mgmt-framework   latest                             54ed482e06dc   551MB
docker-nat                    202211_RC14.1-7a20284d5_Internal   442727751172   423MB
docker-nat                    latest                             442727751172   423MB
docker-sflow                  202211_RC14.1-7a20284d5_Internal   912fbf77e6ef   421MB
docker-sflow                  latest                             912fbf77e6ef   421MB

Output of show techsupport:

(paste your output here or download and attach the file here )

Additional information you deem important (e.g. issue happens only occasionally):

log.tar.gz

[arlakshm]

@abdosi can you take a look

[liat-grozovik]

@abdosi could you please share an eta for such a fix?

[dgsudharsan]

@abdosi Any progress on this issue?

[stepanblyschak]

@abdosi Recently MSFT reported that COFIG_DB shouldn't be updated at runtime - #20462 and this issue is related to that, do you have an ETA for a fix ?