Support building CHIP with GN (project-chip#1456)

* Add nlunit-test to GN build

* Add nlassert to GN build

* Add nlfaultinjection to GN build

* Add nlio to GN build

* Add GN files to gitignore

* Add GN build support using pw_toolchain & pw_build

This uses build infrastructure from pigweed to enable building CHIP with
GN.

* Add lwIP to GN build

* Add Mbed TLS to GN build

* Add SystemConfig to GN build

* Add CHIPConfig to GN build

* Add src/lib/support to GN build

* Add src/system to GN build

* Add src/inet to GN build

* Add BLE to GN build

* Add src/lib/core to GN build

* Add src/controller to GN build

* Add libCHIP.a to GN build

* Add src/system/tests to GN build

* Add CHIPVersion.h generation to GN build

* Add src/lib/support/tests to GN build

* Add src/inet/tests to GN build

* Add src/ble/tests to GN build

* Add src/lib/core/tests to GN build

* Add src/crypto to GN build

* Add src/app to GN build

* Add src/crypto/tests to GN build

* Add support for building nRF5 lock app

* Run unit tests with some help from pigweed

This enables running tests during the build. Just run

  gn gen out/debug
  ninja -C out/debug check

* Add qrcodetool to GN build

* Add transport to GN build

* Add src/setup_payout/tests to GN build

* Add src/setup_payload to GN build

* Add ARM toolchain support

* Make the build relocatable

* Add nRF5 lock example superproject build

* Add support for combined build via target_os="all"

This configures host_clang, host_gcc, and nRF5 lock app.

To run a maximal build:

  gn gen out/debug --args='target_os="all"'
  ninja -C out/debug check

* Make bootstrap display cleaner

* Add GN build wrapper script

This script does an end to end bootstrap & build with GN from a fresh
clone.

Usage: ./gn_build.sh

* Add examples/chip-tool to GN build

* Remove shared imports

Importing everything from chip.gni will require some dependencies to be
present even if they are not used. Import overrides separately.

Also add args to disable various builds at the top level.

* Add transport tests to GN build

* Add examples/shell to GN build

* Add tips to gn_build.sh

To run a GN build of chip:
  ./gn_build.sh

* Enable building with homebrew openssl on Mac via pkg-config

* Add mbedtls build to top level build

* Add nRF5 lock app back to example build

But only if NRF5_SDK_ROOT is set in the environment.

* Harmonize library outputs with automake

This changes most libraries to match the names used by automake.

* Restyled by shfmt

* Move the GN build files to //gn

And use the secondary_source feature to still allow loading them via
//build. This breaks using ${chip_root} in superproject builds, but that
was unnecessary anyway.

We should try to unwind this later as it causes some confusion about
what path to specify.

* Fix lint warning "Except block directly handles BaseException"

* Move activate above bootstrap

The usual case is to source activate.sh, not bootstrap.sh, so move that
first. bootstrap.sh is only needed after an update.

* Add github actions for GN

* Move project specific build logic to gn/chip

* Add comments to GN build GitHub workflow helpers

* Fix the nRF5 lock app CI build

Remove a broken link that was left behind after moving build files to //gn.

* Updating Build Version

* Re-merge repos.conf with .gitmodules

This was unmerged by a recent commit.

* Move bootstrap.sh & activate.sh to scripts

* Fixup activate.sh & bootstrap.sh after moving

Co-authored-by: Keir Mierle <[email protected]>
Co-authored-by: Restyled.io <[email protected]>
Co-authored-by: Justin Wood <[email protected]>

.github/workflows/gn_build.yaml

@@ -0,0 +1,48 @@
+name: GN Builds
+
+on:
+    push:
+    pull_request:
+
+jobs:
+    build:
+        name: GN Build
+
+        strategy:
+            matrix:
+                type: [main, clang, mbedtls]
+        env:
+            BUILD_TYPE: ${{ matrix.type }}
+            BUILD_VERSION: 0.2.18
+            BUILD_IMAGE: chip-build-openssl
+            BUILD_ORG: connectedhomeip
+
+        runs-on: ubuntu-latest
+
+        container:
+            image: "connectedhomeip/chip-build-openssl:0.2.18"
+            volumes:
+                - "/tmp/log_output:/tmp/test_logs"
+
+        steps:
+            - name: Checkout
+              uses: actions/checkout@v2
+            - name: Checkout Submodules
+              run: make -f Makefile-bootstrap repos
+            - name: Bootstrap
+              run: scripts/build/gn_bootstrap.sh
+            - name: Setup Build
+              run: |
+                  case $BUILD_TYPE in
+                     "main") GN_ARGS='';;
+                     "clang") GN_ARGS='is_clang=true';;
+                     "linux-embedded") GN_ARGS='device_platform="linux"';;
+                     "mbedtls") GN_ARGS='chip_crypto="mbedtls"';;
+                     *) ;;
+                  esac
+
+                  scripts/build/gn_gen.sh --args="$GN_ARGS"
+            - name: Run Build
+              run: scripts/build/gn_build.sh
+            - name: Run Tests
+              run: scripts/tests/gn_tests.sh

.gitignore

@@ -20,6 +20,9 @@ configure
 src/include/BuildConfig.h.in
 src/include/BuildConfig.h.in~
 
+# GN build system
+out/
+
 # Repos stuff
 .repos-warning-stamp
 

.gitmodules

@@ -52,7 +52,7 @@
 	branch = master
 	ignore = dirty
 	commit = 5f0e36eeb236a507631e9c822e43f742b7b6738a
-[submodule "third_party/ot-br-posix/repo"]
+[submodule "ot-br-posix"]
 	path = third_party/ot-br-posix/repo
 	url = https://github.com/openthread/ot-br-posix.git
 	branch = master

.gn

@@ -0,0 +1,28 @@
+# Copyright (c) 2020 Project CHIP Authors
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# The location of the build configuration file.
+buildconfig = "//build/config/BUILDCONFIG.gn"
+
+# CHIP uses angle bracket includes.
+check_system_includes = true
+
+# Allow loading paths relative to //gn.
+secondary_source = "//gn/"
+
+import("//gn/build_overrides/pigweed.gni")
+
+default_args = {
+  pw_unit_test_AUTOMATIC_RUNNER = "$dir_pigweed/targets/host/run_test"
+}

BUILD.gn

@@ -0,0 +1,163 @@
+# Copyright (c) 2020 Project CHIP Authors
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+import("//build_overrides/chip.gni")
+import("//build_overrides/mbedtls.gni")
+import("//build_overrides/nlassert.gni")
+import("//build_overrides/nlio.gni")
+import("//build_overrides/nlunit_test.gni")
+import("//build_overrides/pigweed.gni")
+
+# This build file should not be used in superproject builds.
+assert(chip_root == "//")
+
+import("${chip_root}/gn/chip/tests.gni")
+import("${chip_root}/gn/chip/tools.gni")
+
+if (current_toolchain != "${dir_pw_toolchain}/dummy:dummy") {
+  # This is a real toolchain. Build CHIP.
+  group("default") {
+    deps = [ ":all" ]
+  }
+
+  group("all") {
+    deps = [
+      "${chip_root}/src/app",
+      "${chip_root}/src/ble",
+      "${chip_root}/src/controller",
+      "${chip_root}/src/crypto",
+      "${chip_root}/src/inet",
+      "${chip_root}/src/lib",
+      "${chip_root}/src/lib/core",
+      "${chip_root}/src/lib/support",
+      "${chip_root}/src/lwip:all",
+      "${chip_root}/src/system",
+      "${chip_root}/src/transport",
+      "${mbedtls_root}:mbedtls",
+      "${nlassert_root}:nlassert",
+      "${nlio_root}:nlio",
+      "${nlunit_test_root}:nlunit-test",
+    ]
+
+    if (chip_build_tests) {
+      deps += [ ":tests" ]
+    }
+
+    if (chip_build_tools) {
+      deps += [
+        "${chip_root}/examples/chip-tool",
+        "${chip_root}/examples/shell",
+        "${chip_root}/src/lib/shell",
+        "${chip_root}/src/qrcodetool",
+        "${chip_root}/src/setup_payload",
+      ]
+    }
+  }
+
+  if (chip_build_tests) {
+    import("${chip_root}/gn/chip/chip_test_group.gni")
+
+    chip_test_group("tests") {
+      deps = [
+        "${chip_root}/src/app/plugin/tests",
+        "${chip_root}/src/ble/tests",
+        "${chip_root}/src/crypto/tests",
+        "${chip_root}/src/inet/tests",
+        "${chip_root}/src/lib/core/tests",
+        "${chip_root}/src/lib/support/tests",
+        "${chip_root}/src/lwip/tests",
+        "${chip_root}/src/setup_payload/tests",
+        "${chip_root}/src/system/tests",
+        "${chip_root}/src/transport/tests",
+      ]
+    }
+
+    group("check") {
+      deps = [ ":tests_run" ]
+    }
+  }
+} else {
+  # This is the dummy toolchain. Configure various real toolchains.
+  import("${chip_root}/gn/chip/chip_build.gni")
+  declare_args() {
+    # Set this to false to disable all builds by default.
+    enable_default_builds = true
+  }
+
+  declare_args() {
+    # Enable building chip with clang.
+    enable_host_clang_build = enable_default_builds
+
+    # Enable building chip with gcc.
+    enable_host_gcc_build = enable_default_builds
+
+    # Build building chip with gcc & mbedtls.
+    enable_host_gcc_mbdtls_build = enable_default_builds
+
+    # Build the nRF5 lock app example.
+    enable_nrf5_lock_app_build = false
+  }
+
+  if (enable_nrf5_lock_app_build) {
+    import("//build_overrides/nrf5_lock_app.gni")
+  }
+
+  import("//build/toolchain/host_clang/toolchains.gni")
+  import("//build/toolchain/host_gcc/toolchains.gni")
+  chip_build("host_clang") {
+    toolchain = "//build/toolchain/host_clang:${host_os}_${host_cpu}_clang"
+  }
+
+  chip_build("host_gcc") {
+    toolchain = "//build/toolchain/host_gcc:${host_os}_${host_cpu}_gcc"
+  }
+
+  chip_build("host_gcc_mbedtls") {
+    toolchain = "//build/toolchain/host_gcc:${host_os}_${host_cpu}_gcc_mbedtls"
+  }
+
+  group("default") {
+    deps = []
+    if (enable_host_clang_build) {
+      deps += [ ":all_host_clang" ]
+    }
+    if (enable_host_gcc_build) {
+      deps += [ ":all_host_gcc" ]
+    }
+    if (enable_host_gcc_mbdtls_build) {
+      deps += [ ":all_host_gcc_mbedtls" ]
+    }
+    if (enable_nrf5_lock_app_build) {
+      deps += [ "${nrf5_lock_app_root}:nrf5(${nrf5_lock_app_root}/toolchain:nrf5_lock_app)" ]
+    }
+  }
+
+  if (chip_build_tests) {
+    group("check") {
+      deps = []
+      if (enable_host_clang_build) {
+        deps += [ ":check_host_clang" ]
+      }
+      if (enable_host_gcc_build) {
+        deps += [ ":check_host_gcc" ]
+      }
+      if (enable_host_gcc_mbdtls_build) {
+        deps += [ ":check_host_gcc_mbedtls" ]
+      }
+      if (enable_nrf5_lock_app_build) {
+        deps += [ ":default" ]
+      }
+    }
+  }
+}

examples/chip-tool/BUILD.gn

@@ -0,0 +1,31 @@
+# Copyright (c) 2020 Project CHIP Authors
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+import("//build_overrides/chip.gni")
+
+import("${chip_root}/gn/chip/tools.gni")
+
+assert(chip_build_tools)
+
+executable("chip-standalone-demo") {
+  sources = [ "main.cpp" ]
+
+  public_deps = [ "${chip_root}/src/lib" ]
+
+  output_dir = "${root_out_dir}/bin"
+}
+
+group("chip-tool") {
+  deps = [ ":chip-standalone-demo" ]
+}

examples/lock-app/nrf5/.gn

@@ -0,0 +1,26 @@
+# Copyright (c) 2020 Project CHIP Authors
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# The location of the build configuration file.
+buildconfig = "//build/config/BUILDCONFIG.gn"
+
+# CHIP uses angle bracket includes.
+check_system_includes = true
+
+# Allow loading paths relative to //gn.
+secondary_source = "//third_party/connectedhomeip/gn/"
+
+default_args = {
+  import("//args.gni")
+}