Merge branch 'main' into 20059-fix-last_enrolled_at

.github/ISSUE_TEMPLATE/feature-request.md

@@ -9,17 +9,26 @@ assignees: ''
 
 <!--
 Thanks for filing an issue!  Please use the prompts below to provide as much context as you can about your use case and motivations.
-- How might this have a positive effect on your organization?
-- What is the current situation? Why does the current situation hurt?
-- What are you doing right now to work around this issue? What's non-ideal about it?
 -->
 
 ## Problem
-TODO
-<!-- Describe the problem you're trying to solve. -->
+
+<!-- Describe the problem you're trying to solve. What are you trying to accomplish? 
+
+Example: I want to order a pair of shoes from my food delivery app, which does not show options for stores that don't carry food. -->
+
+## What have you tried?
+
+<!-- Described what actions you have taken in the product today to try and solve this problem. Why didn't that workflow or result work for you? What is missing? 
+
+Example: I searched for shoe stores in my food delivery app, but there were no results available. -->
 
 ## Potential solutions
-<!-- You can leave this blank, or propose a solution. You can also attach any screenshots or other visuals that might help convey your meaning. -->
-1. 
-2. 
-3. 
+
+<!-- Propose a solution. What would your ideal workflow look like? You can also attach any screenshots or other visuals that might help convey your meaning. 
+
+Example: My food delivery app should have a new search mapping for other categories of goods that can be delivered like 'clothing' or 'home goods' in the suggested searches menu. -->
+
+## What is the expected workflow as a result of your proposal?
+
+<!-- Example: I search for the shoe store > I click on the pair of shoes in the size and color I want > I am given an estimated delivery time and price > I pay for my order  > A driver picks up the order and delivers it to me > I am able to track the delivery in the same way I would track a food order. --> 

.github/ISSUE_TEMPLATE/story.md

@@ -37,10 +37,11 @@ What else should contributors [keep in mind](https://fleetdm.com/handbook/compan
 - [ ] REST API changes: TODO <!-- Specify changes as a draft PR to the REST API doc page and request the API DRI for review (codeowner reviews are not automatically requested for drafts). Remove this checkbox if there are no changes necessary. Move this item to the engineering list below if engineering will design the API changes. -->
 - [ ] Fleet's agent (fleetd) changes: TODO <!-- Specify changes to fleetd. If the change requires a new Fleet (server) version, consider specifying to only enable this change in new Fleet versions. Remove this checkbox if there are no changes necessary. -->
 - [ ] Permissions changes: TODO <!-- Specify changes as a draft PR to the Manage access doc page. If doc changes aren't necessary, explicitly mention no changes to the doc page. Remove this checkbox if there are no permissions changes. -->
-- [ ] Outdated documentation changes: TODO <!-- Specify required documentation changes (public-facing fleetdm.com/docs or contributors) & redirects to add to /website/config/routes.js. -->
 - [ ] Changes to paid features or tiers: TODO  <!-- Specify "Fleet Free" or "Fleet Premium".  If only certain parts of the user story involve paid features, specify which parts.  Implementation of paid features should live in the `ee/` directory. -->
 
 ### Engineering
+- [ ] Reference documentation changes: TODO <!-- Specify references documentation changes at fleetdm.com/docs -->
+- [ ] Usage documentation changes: TODO <!-- Specify usage documentation changes at fleetdm.com/docs/using-fleet -->
 - [ ] Database schema migrations: TODO <!-- Specify what changes to the database schema are required. (This will be used to change migration scripts accordingly.) Remove this checkbox if there are no changes necessary. -->
 - [ ] Load testing: TODO  <!-- List any required scalability testing to be conducted.  Remove this checkbox if there is no scalability testing required. -->
 

.github/workflows/fleet-and-orbit.yml

@@ -389,7 +389,7 @@ jobs:
 
       - name: Uninstall pkg
         run: |
-          ./orbit/tools/cleanup/cleanup_macos.sh
+          sudo ./orbit/tools/cleanup/cleanup_macos.sh
 
   orbit-ubuntu:
     timeout-minutes: 60

.github/workflows/integration.yml

@@ -238,7 +238,7 @@ jobs:
 
     - name: Uninstall Orbit
       run: |
-        ./orbit/tools/cleanup/cleanup_macos.sh
+        sudo ./orbit/tools/cleanup/cleanup_macos.sh
 
   orbit-ubuntu:
     timeout-minutes: 10

.github/workflows/verify-fleetd-base.yml

@@ -15,8 +15,10 @@ on:
         required: false
         default: 'https://download.fleetdm.com'
         type: string
+  schedule:
+    - cron: '0 5 * * *' # Nightly 5AM UTC, not at the same time as release-fleetd-base workflow
 
-# This workflow is intended to be called by release-fleetd-base workflow, so it does not have a concurrency group.
+# This workflow is called by release-fleetd-base workflow, so it does not have its own concurrency group.
 
 defaults:
   run:
@@ -30,7 +32,7 @@ jobs:
   verify-checksums:
     runs-on: ubuntu-latest
     env:
-      BASE_URL: ${{ github.event.inputs.base-url || 'https://download-testing.fleetdm.com' }}
+      BASE_URL: ${{ github.event.inputs.base-url || 'https://download.fleetdm.com' }}
     steps:
       - name: Harden Runner
         uses: step-security/harden-runner@63c24ba6bd7ba022e95695ff85de572c04a18142 # v2.7.0
@@ -65,7 +67,7 @@ jobs:
   verify-fleetd-base-msi:
     runs-on: windows-latest
     env:
-      BASE_URL: ${{ github.event.inputs.base-url || 'https://download-testing.fleetdm.com' }}
+      BASE_URL: ${{ github.event.inputs.base-url || 'https://download.fleetdm.com' }}
     steps:
       - name: Harden Runner
         uses: step-security/harden-runner@63c24ba6bd7ba022e95695ff85de572c04a18142 # v2.7.0
@@ -92,7 +94,7 @@ jobs:
   verify-fleetd-base-pkg:
     runs-on: macos-latest
     env:
-      BASE_URL: ${{ github.event.inputs.base-url || 'https://download-testing.fleetdm.com' }}
+      BASE_URL: ${{ github.event.inputs.base-url || 'https://download.fleetdm.com' }}
     steps:
       - name: Harden Runner
         uses: step-security/harden-runner@63c24ba6bd7ba022e95695ff85de572c04a18142 # v2.7.0

articles/certificates-in-fleetd.md

@@ -0,0 +1,62 @@
+# Certificates in fleetd
+
+There are three components in fleetd connecting to the Fleet server using TLS: `orbit`, `Fleet Desktop` and `osqueryd`.
+This article aims to describe how TLS CA root certificates are configured in fleetd to connect to a Fleet server securely.
+
+## Default
+
+The default behavior is using the `fleetctl package` command without the `--fleet-certificate` flag.
+
+- By default, `orbit` and `Fleet Desktop` will use the system's CA root store to connect to Fleet.
+- `osqueryd` doesn't support using the system's CA root store, it requires passing in a certificate file with the root CA store (via the `--tls_server_certs` flag). The `fleetctl` executable contains an embedded `certs.pem` file generated from https://curl.se/docs/caextract.html [0]. When generating a fleetd package with `fleetctl package` such embedded `certs.pem` file is added to the package [1]. Fleetd configures `osqueryd` to use the `certs.pem` file as CA root store by setting the `--tls_server_certs` argument to such path.
+
+## Using `--fleet-certificate` in `fleetctl package`
+
+When using `--fleet-certificate` in `fleetctl package`, such certificate file is used as a CA root store by `orbit`, `Fleet Desktop` and `osqueryd` (the system's CA store is not used when generating the fleetd package this way).
+
+## Issues with internal and/or intermediates certificates
+
+TLS clients require the CA root and all intermediate certificates that signed the leaf server certificate to be verified.
+This means that if the bundled certificate in fleetd [1] doesn't have intermediate certificates that signed the leaf certificate, then the Fleet server will have to be configured to serve the "fullchain".
+Here's a list of some scenarios assuming your Fleet server certificate has an intermediate signing certificate:
+- ✅ Using fullchain in the Fleet server and root CA only client side.
+- ✅ Using fullchain in the Fleet server and root+intermediate bundle client side.
+- ✅ Using the leaf certificate in the Fleet server and root+intermediate bundle client side.
+- ✅ Using the leaf certificate + intermediate bundle in the Fleet server and root CA only client side.
+- ❌ Using the leaf certificate in the Fleet server and root CA only client side. In this scenario the client side (fleetd) doesn't know of the intermediate certificate and thus cannot verify it.
+
+We've seen TLS certificate issues in the following configurations: (for more information see https://github.com/fleetdm/fleet/issues/6085):
+- Certificates signed by internal CA/intermediates.
+- Certificates issued by Let's Encrypt (that do not serve the fullchain certificate).
+
+When there are certificate issues you will see the following kind of errors in server logs:
+```
+2024/07/05 15:03:52 http: TLS handshake error from <remote_ip>:<remote_port>: remote error: tls: bad certificate
+2024/07/05 15:03:53 http: TLS handshake error from <remote_ip>:<remote_port>: local error: tls: bad record MAC
+```
+and the following kind of errors on the client side (fleetd):
+```
+2024-07-05T15:04:52-03:00 DBG get config error="POST /api/fleet/orbit/config: Post \"https://fleet.example.com/api/fleet/orbit/config\": tls: failed to verify certificate: x509: certificate signed by unknown authority"
+```
+```
+W0705 15:16:44.739495 1251102656 init.cpp:760] Error reading config: Request error: certificate verify failed
+```
+
+To troubleshoot issues with certificates you can use `fleetctl debug connection` command, e.g.:
+```sh
+fleetctl debug connection \
+  --fleet-certificate ./your-ca-root.pem \
+  https://fleet.example.com
+```
+
+[0]: We have a Github CI action that runs daily that updates the [certs.pem on the repository](https://github.com/fleetdm/fleet/blob/main/orbit/pkg/packaging/certs.pem) whenever there's a new version of `cacert.pem` in https://curl.se/docs/caextract.html. Such file is embedded into the `fleetctl` executable and used when generating fleetd packages.
+[1]: The bundled certificate in fleetd is installed in `/opt/orbit` in macOS/Linux and `C:\Program Files\Orbit` on Windows. By default its name is `certs.pem`, but it will have a different name if the `--fleet-certificate` flag was used when generating the package (`fleetctl package`).
+
+
+<meta name="articleTitle" value="Certificates in fleetd">
+<meta name="authorFullName" value="Lucas Manuel Rodriguez">
+<meta name="authorGitHubUsername" value="lucasmrod">
+<meta name="category" value="guides">
+<meta name="publishedOn" value="2024-08-09">
+<meta name="articleImageUrl" value="../website/assets/images/articles/apple-developer-certificates-on-linux-for-configuration-profile-signing-1600x900@2x.png">
+<meta name="description" value="TLS certificates in fleetd">

articles/sysadmin-diaries-lost-device.md

@@ -0,0 +1,155 @@
+# Sysadmin diaries: lost device
+
+![Sysadmin diaries: lost device](../website/assets/images/articles/[email protected])
+
+Picture this: an employee calls you in a panic from an airport halfway across the country. They have just realized they left their company-issued laptop on the plane. Cue the sinking feeling. The device contains sensitive company data, and the thought of it falling into the wrong hands is enough to induce a cold sweat. But fear not! With Fleet's Mobile Device Management (MDM) capabilities, you can handle this situation swiftly and securely. Let us walk through how to lock or wipe a lost device using Fleet remotely.
+
+
+## The scenario: a lost device
+
+Imagine you receive a call from Jamie, a sales executive who has just landed in Chicago for a crucial client meeting. In their rush to deplane, they accidentally leave their laptop in the seatback pocket. Realizing the mistake after reaching the terminal, Jamie calls you, anxious and stressed about the potential data breach.
+
+
+## Keep calm and use Fleet
+
+First, take a deep breath. Fleet has got you covered using MDM. You can remotely lock and wipe the lost device to ensure your company’s data remains secure.
+
+
+### Step 1: identify the device
+
+Start by identifying the device in Fleet. Navigate to the **Hosts** page in the Fleet web UI. Use the search functionality to quickly find Jamie’s laptop by entering the hostname or any other relevant identifier.
+
+
+### Step 2: remote lock
+
+
+#### Using the Fleet web UI
+
+1. Once you have located the device, click on it to open the **Host Overview** page.
+
+2. In the **Actions** menu, select **Lock**.
+
+3. A confirmation dialog will appear. Confirm that you want to lock the device.
+
+
+#### Using the Fleet API
+
+Alternatively, you can use the Fleet REST API to lock the device. Here is the API call you need to make:
+
+``` bash
+
+POST /api/v1/fleet/hosts/:id/lock
+
+```
+
+Replace `:id` with Jamie’s laptop's actual ID. This command sends a signal to lock the device as soon as it comes online. For macOS, this requires MDM to be enabled. For Windows and Linux, scripts need to be enabled.
+
+If you wanted to call this from the command line, you could use `curl` with a command like this:
+
+```bash
+
+curl -X GET  https://fleet.company.com/api/v1/fleet/hosts/123/lock  -H "Authorization: Bearer <your_API_key>"
+
+```
+
+
+#### Optional steps for macOS
+
+You can customize the locking message for macOS devices and set a PIN using an XML payload. Here is how:
+
+1. Create a file named `command-lock-macos-host.xml` with the following content:
+
+    ```xml
+
+    <?xml version="1.0" encoding="UTF-8"?>
+    <!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
+    <plist version="1.0">
+    <dict>
+        <key>Command</key>
+        <dict>
+            <key>Message</key>
+            <string>This device has been locked. Contact IT on (123) 456-7890.</string>
+            <key>PIN</key>
+            <string>123456</string>
+            <key>RequestType</key>
+            <string>DeviceLock</string>
+        </dict>
+    </dict>
+    </plist>
+
+    ```
+
+2. Customize the message and PIN as needed.
+
+3. Safely store the recovery PIN using a secure method like 1Password.
+
+4. Run the following command using the Fleet CLI tool, replacing `hostname` with the actual hostname in Fleet and the payload path with the file’s location:
+
+    ```bash
+
+    fleetctl mdm run-command --hosts=hostname --payload=command-lock-macos-host.xml
+
+    ```
+
+
+### Step 3: remote wipe (if necessary)
+
+If you determine the device is at a high risk of being compromised, you may decide to wipe it. This is a more drastic step, but sometimes, it is necessary to protect sensitive information.
+
+
+#### Using the Fleet web UI
+
+1. On the same **Host Overview** page, go to the **Actions** menu and select **Wipe**.
+
+2. Confirm the wipe action that appears in the dialog.
+
+
+#### Using the Fleet API
+
+To wipe the device via the API, use the following call:
+
+```bash
+
+POST /api/v1/fleet/hosts/:id/wipe
+
+```
+
+Again, replace `:id` with the device’s ID. The wipe command will be executed once the device is online. MDM must be enabled for macOS and Windows, and scripts must be enabled for Linux.
+
+
+### Step 4: confirm and reassure
+
+After you have locked and potentially wiped the device, inform Jamie of the steps actioned. Reassure them that the company’s data is now secure and provide any further instructions they may need, such as getting a replacement device.
+
+
+### Unlocking macOS
+
+If the device is found and needs to be unlocked:
+
+
+
+1. Enter the security PIN (stored in Fleet, returned from the API call, or the XML file) in the device's input field.
+2. The device will open to the regular login screen and ask for a password.
+3. If the password is unavailable, select the option to enter the recovery key/disk encryption key (this option might be behind a ? icon).
+4. Retrieve the disk encryption key from Fleet’s web UI.
+5. Enter the disk encryption key on the laptop, which should prompt you to create a new password.
+6. You will then be logged into the default device profile, which allows you to complete any needed actions (e.g., wiping or recovering data).
+
+
+## Conclusion
+
+Losing a device is stressful, but Fleet’s MDM capabilities can help you manage it effectively. You can protect sensitive data and prevent unauthorized access by remotely locking or wiping the lost device. Remember, stay calm, and rely on Fleet to secure your endpoints.
+
+Fleet’s MDM features ensure that your data remains protected even if a device is lost. So, the next time you get that dreaded call, you will know exactly what to do.
+
+
+
+
+
+<meta name="articleTitle" value="Sysadmin diaries: lost device">
+<meta name="authorFullName" value="JD Strong">
+<meta name="authorGitHubUsername" value="spokanemac">
+<meta name="category" value="guides">
+<meta name="publishedOn" value="2024-07-09">
+<meta name="articleImageUrl" value="../website/assets/images/articles/[email protected]">
+<meta name="description" value="In this sysadmin diary, we explore what actions can be taken with Fleet when a device is lost.">

articles/what-are-fleet-policies.md

@@ -6,7 +6,7 @@ Read more about [getting and staying compliant across your fleet.](https://fleet
 
 ## Add policies
 
-To reduce the barrier of entry with Fleet, we’ve introduced our [standard templates](fleetdm.com/queries) that allow users to choose from a library of pre-made policies. Those handy with [osquery](https://osquery.io/) and SQL can still create custom policies to their heart’s content: 
+To reduce the barrier of entry with Fleet, we’ve introduced our [standard templates](https://fleetdm.com/queries) that allow users to choose from a library of pre-made policies. Those handy with [osquery](https://osquery.io/) and SQL can still create custom policies to their heart’s content: 
 
 1. In the top navigation, select **Policies**.
 

changes/18849-config-profiles-exclude-labels

@@ -0,0 +1,5 @@
+* Added the database migrations to create the new `exclude` column for labels associated with MDM profiles (and declarations).
+* Added the API changes to support the `labels_include_all` and `labels_exclude_any` fields (and accept the deprecated `labels` field as an alias for `labels_include_all`).
+* Added `fleetctl gitops` and `fleetctl apply` support for `labels_include_all` and `labels_exclude_any` to configure a custom setting.
+* Updated the profile reconciliation logic to handle the new "exclude any" labels.
+* Fix bug where macOS declarations were stuck in "to be removed" state indefinitely.

changes/19127-update-logic-and-copy-around-host-identifiers

@@ -0,0 +1,2 @@
+- Update `fleetctl query --hosts` to work with hostnames, host UUIDs, and/or hardware serial numbers.
+- Clarify various help and error texts around host identifiers.

changes/19143-mdm-cmd-filters

@@ -0,0 +1 @@
+- `fleetctl get mdm_commands` now returns 20 rows and supports `--host` `--type` filters to improve response time

changes/19144-pkg-matching

@@ -0,0 +1 @@
+* Improved the matching of `pkg` installer files to existing software

changes/19281-add-host-name-to-event-descriptions

@@ -0,0 +1 @@
+* Add host's display name to calendar event descriptions

changes/19352-calendar-real-time

@@ -0,0 +1 @@
+- In maintenance windows using Google Calendar, calendar event is now recreated within 30 seconds if deleted or moved to the past.

changes/19557-empty-hover-styles

@@ -0,0 +1 @@
+* Update empty state styles in 4 places, clean up

changes/19694-vul-page-bugs

@@ -0,0 +1 @@
+* Fix 3 UI bugs on the Software page

changes/19789-serial-number--1

@@ -0,0 +1 @@
+When osquery returns a serial number of -1 (default value), we keep the existing serial number in the database.

changes/19800-renew-scep-migration

@@ -0,0 +1 @@
+* Added support for renewing SCEP certificates with custom enrollment profiles.

changes/19963-ios-ipados-as-platforms

@@ -0,0 +1,3 @@
+- Added iOS/iPadOS builtin manual labels. IMPORTANT: Before migrating to this version, make sure to delete any labels with name "iOS" or "iPadOS".
+- Added aggregation of iOS/iPadOS OS versions.
+- Added change to custom profiles for iOS/iPadOS to go from 'pending' straight to 'verified' (skip 'verifying').

changes/20050-vuln-software

@@ -0,0 +1 @@
+- Adds back the "Vulnerable" filter for the host details software table

changes/20057-connected-tweaks

@@ -0,0 +1 @@
+* Improved the accuracy of the heuristic used to deterimine if a host is connected to Fleet via MDM by using osquery data for hosts that didn't send a Checkout message.

changes/20075-fleetctl-apply-validation

@@ -0,0 +1 @@
+- Add .yml and .yaml file type validation and error message to fleetctl apply

changes/20077-align-view-all-hosts-link-sw-page

@@ -0,0 +1 @@
+- Align the "View all hosts" links in the Software titles and versions tables.

changes/20080-lock-disable-credential-caching

@@ -0,0 +1 @@
+- Disable credential caching and reboot on Windows lock