System Ready (#10479)

Why I did it
At present, there is no mechanism in an event driven model to know that the system is up with all the essential sonic services and also, all the docker apps are ready along with port ready status to start the network traffic. With the asynchronous architecture of SONiC, we will not be able to verify if the config has been applied all the way down to the HW. But we can get the closest up status of each app and arrive at the system readiness.

How I did it
A new python based system monitor tool is introduced under system-health framework to monitor all the essential system host services including docker wrapper services on an event based model and declare the system is ready. This framework gives provision for docker apps to notify its closest up status. CLIs are provided to fetch the current system status and also service running status and its app ready status along with failure reason if any.

How to verify it
"show system-health sysready-status" click CLI
Syslogs for system ready

files/build_templates/init_cfg.json.j2

@@ -57,6 +57,12 @@
             "has_global_scope": {% if feature + '.service' in installer_services.split(' ') %}true{% else %}false{% endif %},
             "has_per_asic_scope": {% if feature + '@.service' in installer_services.split(' ') %}true{% else %}false{% endif %},
             "auto_restart": "{{autorestart}}",
+{# Set check_up_status to true here when app readiness will be marked in state db  #}
+{# For now, to support the infrastrucure, setting the check_up_status to false for bgp,swss,pmon  #}
+{# Once apps like bgp,synd supports app readiness, then bgp,syncd can set check_up_status to true #}
+{%- if feature in ["bgp", "swss", "pmon"] %}
+            "check_up_status" : "false",
+{%- endif %}
 {%- if include_kubernetes == "y" %}
 {%- if feature in ["lldp", "pmon", "radv", "snmp", "telemetry"] %}
             "set_owner": "kube", {% else %}

files/build_templates/sonic_debian_extension.j2

@@ -890,3 +890,6 @@ sudo cp $BUILD_SCRIPTS_DIR/mask_disabled_services.py $FILESYSTEM_ROOT/tmp/
 sudo chmod a+x $FILESYSTEM_ROOT/tmp/mask_disabled_services.py
 sudo LANG=C chroot $FILESYSTEM_ROOT /tmp/mask_disabled_services.py
 sudo rm -rf $FILESYSTEM_ROOT/tmp/mask_disabled_services.py
+
+
+sudo LANG=C DEBIAN_FRONTEND=noninteractive chroot $FILESYSTEM_ROOT apt-get -y install python3-dbus

src/sonic-host-services-data/debian/sonic-host-services-data.determine-reboot-cause.service

@@ -5,6 +5,7 @@ After=rc-local.service database.service
 
 [Service]
 Type=simple
+RemainAfterExit=yes
 ExecStart=/usr/local/bin/determine-reboot-cause
 
 [Install]

src/sonic-yang-models/yang-models/sonic-feature.yang

@@ -86,6 +86,13 @@ module sonic-feature{
                     type feature-owner;
                     default "local";
                 }
+
+                leaf check_up_status {
+                    description "This configuration controls the system ready tool to check
+                        the app ready/up status";
+                    type boolean;
+                    default false;
+                }
             }
         }
     }