Secrets Sync Client Count Updates (#24752)

* Client Count Routing Updates (#24733)

* updates client count routing for sync and future additions

* adds copyright header to clients sync template

* adds missing copyright headers

* UI: Adds secret_syncs to mirage /activity endpoint (#24846)

* add secret_syncs to mirage endpoint

* import clients handler

* UI: Set up client charts for incoming sync data (#24852)

* sum stacked bar values for tooltip total

* make tooltip dynamic based on chartLegend

* remove redundant helper

* add secret_syncs to client count utils

* move sum function to helper

* update horizontal bar chart to include sync_clients

* calculate sum of bars in tooltip

* rename color palette const, define chart legends in each parent component instead of token.js

* update tooltips

* update mirage handler to add sys/ namespace

* update mirage handler to add sys/ namespace

* use pushObject

* update test

* UI: Secret sync bar chart (#24926)

* install lineal

* add ember-style-modifier dep

* Add client count types for serialized data

* Add sync bar chart component with tests

* Chart is responsive

* address comments

* Clients Counts Parent Route (#24899)

* adds interfaces for clients models

* moves date formatting logic from clients activity adapter to utils file

* adds clients counts route

* updates links to clients route to point to top level and updates redirect to counts overview route

* removes clients base route and moves overview and sync routes under counts

* adds clients counts page component

* converts clients route to ts

* adds billing start timestamp to clients config mirage response and updates counts route to always attempt to fetch activity

* fixes issue with updating namespace and auth mount query params always triggering client counts route model hook

* adds tests for clients counts page component

* adds missing copyright header to client-counts type file

* Update ui/app/components/clients/page/counts.hbs

Co-authored-by: claire bontempo <[email protected]>

* fixes bad import in sync-bar-chart

* updates clients counts route to bypass query if there is not start_time

* pins d3-shape to 1.3.7 for now -- makes lineal play nice with old charts

* fixes sync bar chart tooltip assertion

---------

Co-authored-by: claire bontempo <[email protected]>

* UI: convert line-chart to lineal (#24961)

* lineal chart alongside svg

* Add version-history to sync handler for testing

* line chart is TS, test updated

* remove d3-shape resolution

* fix clients/token-test

* use chartHeight in running-total template

* use M/yy key instead of timestamp, chart is responsive

* Add test for swapping datasets

* add more edge case tests

* more test

* remove untrue assertion

* fix weird decimal when between 1.1k and 2k

* address feedback

* Update line-chart to use timestamp instead of month key

* Add timestamp to all places where month is on the clients activity response

* Client Counts Overview (#24969)

* adds counts base component for use in client counts child routes

* adds clients counts overview page component

* splits out monthly new chart from clients running total component

* adds missing copyright headers

* moves running total related assertions from token to overview acceptance test

* removes new client assertions from running-total test and adds tests for monthly-new component

* updates copy in running-total component

* fixes clients overview tests

* fixes timestamp stub not being restored in monthly-new test

* fixes mfa-login test

* renames counts component to activity

* removes unused selectedAuthMethod arg from running-total component

* adds timestamp back to running-total component

* Secrets sync UI: add sync page component (#24982)

* adds counts base component for use in client counts child routes

* adds clients counts overview page component

* splits out monthly new chart from clients running total component

* adds missing copyright headers

* move sync-bar-chart to charts/ folder

* update types and rename chart

* rename template file

* moves running total related assertions from token to overview acceptance test

* removes new client assertions from running-total test and adds tests for monthly-new component

* updates copy in running-total component

* fixes clients overview tests

* fixes timestamp stub not being restored in monthly-new test

* fixes mfa-login test

* fix 0 values erroring charts

* separate timestamp again

* address merge conflicts

* finish building sync chart component WIP css

* renames counts component to activity

* update import

* revert name to dataKey

* update styling for charts without legends

* use monthly stat chart component for layout

* use monthly chart stats in monthly new

* implement stat wrapper;

* remove extra grid div

* rename component

* fix legend css;

* update test[

* remove arbitrarily setting max

* add single month view

* use stat text

* update line chart tests

* rename line chart

* update tests

---------

Co-authored-by: Jordan Reimer <[email protected]>

* update selectors

* add sync page tests

* Secrets Sync UI: Add secrets syncs to csv export (#25056)

* update mirage and add sync clients to export csv

* fix sync legend label

* remove word

* update copy in modal

* update mirage

* fix attribution tooltip text

* Clients Counts Token Route (#25019)

* renames token route and page component back to dashboard

* adds client counts token route and page component

* updates charts in token page to use ChartContainer component

* adds tests for clients token page component

* restore clients dashboard test

* use var for chart title sync page

* updates clients token page to show usage stats when querying single month

* updates token page clients averages to only include entity and non-entity clients in calculation

* fixes monthly total counts lower than new clients in mirage handler

* fixes token test

---------

Co-authored-by: [email protected] <[email protected]>

* Clients Usage Stats/Running Total Updates (#25094)

* updates clients usage counts and running totals

* updates usage stats total copy

* fixes client counts overview tests

* Secrets sync UI: cleanup and consolidation of components (#25090)

* rename authMethod to mountPath

* generalize count template copy

* add todo to delete monthly new component

* rename to tokenTab

* wrap filters in conditional checking for start timestamp

* some users may not have access to /config endpoint

* fix querying when user has no billing date permissions and clicks current billing period

* extend activity component from counts page

* Revert "extend activity component from counts page"

This reverts commit 1d0e85c.

* rename to startTimestampISO

* remove timestamp from route and just use activity model responseTimestamp

* fix chart y domain max

* fix typos in usage stat and running totals component

* delete backing class for display only template;

* updates tests

* adds comment for fetching license to get start date for billing

* cleans up unused client counts files (#25157)

* adds changelog

* fix assertion copy

* adds changelog description

* updates enterprise sidebar nav test

---------

Co-authored-by: [email protected] <[email protected]>
Co-authored-by: claire bontempo <[email protected]>
Co-authored-by: Chelsea Shaw <[email protected]>

changelog/24752.txt

@@ -0,0 +1,3 @@
+```release-note:improvement
+ui: Separates out client counts dashboard to overview and entity/non-entity tabs
+```

ui/app/adapters/clients/activity.js

@@ -4,18 +4,16 @@
  */
 
 import ApplicationAdapter from '../application';
-import { getUnixTime } from 'date-fns';
+import { formatDateObject } from 'core/utils/client-count-utils';
 
 export default class ActivityAdapter extends ApplicationAdapter {
   // javascript localizes new Date() objects but all activity log data is stored in UTC
   // create date object from user's input using Date.UTC() then send to backend as unix
   // time params from the backend are formatted as a zulu timestamp
   formatQueryParams(queryParams) {
     let { start_time, end_time } = queryParams;
-    start_time = start_time.timestamp || getUnixTime(Date.UTC(start_time.year, start_time.monthIdx, 1));
-    // day=0 for Date.UTC() returns the last day of the month before
-    // increase monthIdx by one to get last day of queried month
-    end_time = end_time.timestamp || getUnixTime(Date.UTC(end_time.year, end_time.monthIdx + 1, 0));
+    start_time = start_time.timestamp || formatDateObject(start_time);
+    end_time = end_time.timestamp || formatDateObject(end_time, true);
     return { start_time, end_time };
   }
 

ui/app/app.js

@@ -117,7 +117,7 @@ export default class App extends Application {
         services: ['flash-messages', 'router', 'store', 'version'],
         externalRoutes: {
           kvSecretDetails: 'vault.cluster.secrets.backend.kv.secret.details',
-          clientCountDashboard: 'vault.cluster.clients.dashboard',
+          clientCountOverview: 'vault.cluster.clients',
         },
       },
     },

ui/app/components/clients/activity.ts

@@ -0,0 +1,198 @@
+/**
+ * Copyright (c) HashiCorp, Inc.
+ * SPDX-License-Identifier: BUSL-1.1
+ */
+
+// base component for counts child routes that can be extended as needed
+// contains getters that filter and extract data from activity model for use in charts
+
+import Component from '@glimmer/component';
+import { isAfter, isBefore, isSameMonth, fromUnixTime } from 'date-fns';
+import { parseAPITimestamp } from 'core/utils/date-formatters';
+import { calculateAverage } from 'vault/utils/chart-helpers';
+
+import type ClientsActivityModel from 'vault/models/clients/activity';
+import type {
+  ClientActivityNewClients,
+  ClientActivityMonthly,
+  ClientActivityResourceByKey,
+} from 'vault/models/clients/activity';
+import type ClientsVersionHistoryModel from 'vault/models/clients/version-history';
+
+interface Args {
+  activity: ClientsActivityModel;
+  versionHistory: ClientsVersionHistoryModel[];
+  startTimestamp: number;
+  endTimestamp: number;
+  namespace: string;
+  mountPath: string;
+}
+
+export default class ClientsActivityComponent extends Component<Args> {
+  average = (
+    data:
+      | ClientActivityMonthly[]
+      | (ClientActivityResourceByKey | undefined)[]
+      | (ClientActivityNewClients | undefined)[]
+      | undefined,
+    key: string
+  ) => {
+    return calculateAverage(data, key);
+  };
+
+  get startTimeISO() {
+    return fromUnixTime(this.args.startTimestamp).toISOString();
+  }
+
+  get endTimeISO() {
+    return fromUnixTime(this.args.endTimestamp).toISOString();
+  }
+
+  get byMonthActivityData() {
+    const { activity, namespace } = this.args;
+    return namespace ? this.filteredActivityByMonth : activity.byMonth;
+  }
+
+  get byMonthNewClients() {
+    return this.byMonthActivityData ? this.byMonthActivityData?.map((m) => m?.new_clients) : [];
+  }
+
+  get filteredActivityByMonth() {
+    const { namespace, mountPath, activity } = this.args;
+    if (!namespace && !mountPath) {
+      return activity.byMonth;
+    }
+    const namespaceData = activity.byMonth
+      .map((m) => m.namespaces_by_key[namespace as keyof typeof m.namespaces_by_key])
+      .filter((d) => d !== undefined);
+
+    if (!mountPath) {
+      return namespaceData.length === 0 ? undefined : namespaceData;
+    }
+
+    const mountData = mountPath
+      ? namespaceData.map((namespace) => namespace?.mounts_by_key[mountPath]).filter((d) => d !== undefined)
+      : namespaceData;
+
+    return mountData.length === 0 ? undefined : mountData;
+  }
+
+  get filteredActivityByNamespace() {
+    const { namespace, activity } = this.args;
+    return activity.byNamespace.find((ns) => ns.label === namespace);
+  }
+
+  get filteredActivityByAuthMount() {
+    return this.filteredActivityByNamespace?.mounts?.find((mount) => mount.label === this.args.mountPath);
+  }
+
+  get filteredActivity() {
+    return this.args.mountPath ? this.filteredActivityByAuthMount : this.filteredActivityByNamespace;
+  }
+
+  get isCurrentMonth() {
+    const { activity } = this.args;
+    const current = parseAPITimestamp(activity.responseTimestamp) as Date;
+    const start = parseAPITimestamp(activity.startTime) as Date;
+    const end = parseAPITimestamp(activity.endTime) as Date;
+    return isSameMonth(start, current) && isSameMonth(end, current);
+  }
+
+  get isDateRange() {
+    const { activity } = this.args;
+    return !isSameMonth(
+      parseAPITimestamp(activity.startTime) as Date,
+      parseAPITimestamp(activity.endTime) as Date
+    );
+  }
+
+  // (object) top level TOTAL client counts for given date range
+  get totalUsageCounts() {
+    const { namespace, activity } = this.args;
+    return namespace ? this.filteredActivity : activity.total;
+  }
+
+  get upgradeDuringActivity() {
+    const { versionHistory, activity } = this.args;
+    if (versionHistory) {
+      // filter for upgrade data of noteworthy upgrades (1.9 and/or 1.10)
+      const upgradeVersionHistory = versionHistory.filter(
+        ({ version }) => version.match('1.9') || version.match('1.10')
+      );
+      if (upgradeVersionHistory.length) {
+        const activityStart = parseAPITimestamp(activity.startTime) as Date;
+        const activityEnd = parseAPITimestamp(activity.endTime) as Date;
+        // filter and return all upgrades that happened within date range of queried activity
+        const upgradesWithinData = upgradeVersionHistory.filter(({ timestampInstalled }) => {
+          const upgradeDate = parseAPITimestamp(timestampInstalled) as Date;
+          return isAfter(upgradeDate, activityStart) && isBefore(upgradeDate, activityEnd);
+        });
+        return upgradesWithinData.length === 0 ? null : upgradesWithinData;
+      }
+    }
+    return null;
+  }
+
+  // (object) single month new client data with total counts + array of namespace breakdown
+  get newClientCounts() {
+    if (this.isDateRange || !this.byMonthActivityData) {
+      return null;
+    }
+    return this.byMonthActivityData[0]?.new_clients;
+  }
+
+  // total client data for horizontal bar chart in attribution component
+  get totalClientAttribution() {
+    const { namespace, activity } = this.args;
+    if (namespace) {
+      return this.filteredActivityByNamespace?.mounts || null;
+    } else {
+      return activity.byNamespace || null;
+    }
+  }
+
+  // new client data for horizontal bar chart
+  get newClientAttribution() {
+    // new client attribution only available in a single, historical month (not a date range or current month)
+    if (this.isDateRange || this.isCurrentMonth) return null;
+
+    if (this.args.namespace) {
+      return this.newClientCounts?.mounts || null;
+    } else {
+      return this.newClientCounts?.namespaces || null;
+    }
+  }
+
+  get hasAttributionData() {
+    const { mountPath, namespace } = this.args;
+    if (!mountPath) {
+      if (namespace) {
+        const mounts = this.filteredActivityByNamespace?.mounts?.map((mount) => ({
+          id: mount.label,
+          name: mount.label,
+        }));
+        return mounts && mounts.length > 0;
+      }
+      return !!this.totalClientAttribution && this.totalUsageCounts && this.totalUsageCounts.clients !== 0;
+    }
+
+    return false;
+  }
+
+  get upgradeExplanation() {
+    if (this.upgradeDuringActivity) {
+      if (this.upgradeDuringActivity.length === 1) {
+        const version = this.upgradeDuringActivity[0]?.version || '';
+        if (version.match('1.9')) {
+          return ' How we count clients changed in 1.9, so keep that in mind when looking at the data.';
+        }
+        if (version.match('1.10')) {
+          return ' We added monthly breakdowns and mount level attribution starting in 1.10, so keep that in mind when looking at the data.';
+        }
+      }
+      // return combined explanation if spans multiple upgrades
+      return ' How we count clients changed in 1.9 and we added monthly breakdowns and mount level attribution starting in 1.10. Keep this in mind when looking at the data.';
+    }
+    return null;
+  }
+}

ui/app/templates/components/clients/attribution.hbs → ui/app/components/clients/attribution.hbs

@@ -31,7 +31,7 @@
         <p class="chart-description">{{this.chartText.newCopy}}</p>
         <Clients::HorizontalBarChart
           @dataset={{this.barChartNewClients}}
-          @chartLegend={{@chartLegend}}
+          @chartLegend={{this.attributionLegend}}
           @totalCounts={{@newUsageCounts}}
           @noDataMessage="There are no new clients for this namespace during this time period."
         />
@@ -40,22 +40,14 @@
       <div class="chart-container-right" data-test-chart-container="total-clients">
         <h2 class="chart-title">Total clients</h2>
         <p class="chart-description">{{this.chartText.totalCopy}}</p>
-        <Clients::HorizontalBarChart
-          @dataset={{this.barChartTotalClients}}
-          @chartLegend={{@chartLegend}}
-          @totalCounts={{@totalUsageCounts}}
-        />
+        <Clients::HorizontalBarChart @dataset={{this.barChartTotalClients}} @chartLegend={{this.attributionLegend}} />
       </div>
     {{else}}
       <div
         class={{concat (unless this.barChartTotalClients "chart-empty-state ") "chart-container-wide"}}
         data-test-chart-container="single-chart"
       >
-        <Clients::HorizontalBarChart
-          @dataset={{this.barChartTotalClients}}
-          @chartLegend={{@chartLegend}}
-          @totalCounts={{@totalUsageCounts}}
-        />
+        <Clients::HorizontalBarChart @dataset={{this.barChartTotalClients}} @chartLegend={{this.attributionLegend}} />
       </div>
       <div class="chart-subTitle">
         <p class="chart-subtext" data-test-attribution-subtext>{{this.chartText.totalCopy}}</p>
@@ -71,9 +63,10 @@
         <p class="data-details">{{format-number this.topClientCounts.clients}}</p>
       </div>
     {{/if}}
-    <div class="legend-center">
-      <span class="light-dot"></span><span class="legend-label">{{capitalize (get @chartLegend "0.label")}}</span>
-      <span class="dark-dot"></span><span class="legend-label">{{capitalize (get @chartLegend "1.label")}}</span>
+    <div class="legend">
+      {{#each this.attributionLegend as |legend idx|}}
+        <span class="legend-colors dot-{{idx}}"></span><span class="legend-label">{{capitalize legend.label}}</span>
+      {{/each}}
     </div>
   {{else}}
     <div class="chart-empty-state">
@@ -96,9 +89,16 @@
     </M.Header>
     <M.Body>
       <p class="has-bottom-margin-s">
-        This export will include the namespace path, authentication method path, and the associated total, entity, and
-        non-entity clients for the below
-        {{if this.formattedEndDate "date range" "month"}}.
+        This export will include the namespace path, mount path and associated total, entity, non-entity and secrets sync
+        clients for the
+        {{if this.formattedEndDate "date range" "month"}}
+        below.
+      </p>
+      <p class="has-bottom-margin-s">
+        The
+        <code>mount_path</code>
+        for secrets sync clients is the KV v2 engine path and for entity/non-entity clients is the corresponding
+        authentication method path.
       </p>
       <p class="has-bottom-margin-s is-subtitle-gray">SELECTED DATE {{if this.formattedEndDate " RANGE"}}</p>
       <p class="has-bottom-margin-s" data-test-export-date-range>

ui/app/components/clients/attribution.js

@@ -18,7 +18,6 @@ import { format, isSameMonth } from 'date-fns';
  * @example
  * ```js
  *  <Clients::Attribution
- *    @chartLegend={{this.chartLegend}}
  *    @totalUsageCounts={{this.totalUsageCounts}}
  *    @newUsageCounts={{this.newUsageCounts}}
  *    @totalClientAttribution={{this.totalClientAttribution}}
@@ -31,7 +30,6 @@ import { format, isSameMonth } from 'date-fns';
  *    @upgradeExplanation="We added monthly breakdowns and mount level attribution starting in 1.10, so keep that in mind when looking at the data."
  *  />
  * ```
- * @param {array} chartLegend - (passed to child) array of objects with key names 'key' and 'label' so data can be stacked
  * @param {object} totalUsageCounts - object with total client counts for chart tooltip text
  * @param {object} newUsageCounts - object with new client counts for chart tooltip text
  * @param {array} totalClientAttribution - array of objects containing a label and breakdown of client counts for total clients
@@ -47,6 +45,11 @@ import { format, isSameMonth } from 'date-fns';
 export default class Attribution extends Component {
   @tracked showCSVDownloadModal = false;
   @service download;
+  attributionLegend = [
+    { key: 'entity_clients', label: 'entity clients' },
+    { key: 'non_entity_clients', label: 'non-entity clients' },
+    { key: 'secret_syncs', label: 'secrets sync clients' },
+  ];
 
   get formattedStartDate() {
     if (!this.args.startTimestamp) return null;
@@ -123,10 +126,10 @@ export default class Attribution extends Component {
   }
 
   destructureCountsToArray(object) {
-    // destructure the namespace object  {label: 'some-namespace', entity_clients: 171, non_entity_clients: 20, clients: 191}
+    // destructure the namespace object  {label: 'some-namespace', entity_clients: 171, non_entity_clients: 20, secret_syncs: 10, clients: 201}
     // to get integers for CSV file
-    const { clients, entity_clients, non_entity_clients } = object;
-    return [clients, entity_clients, non_entity_clients];
+    const { clients, entity_clients, non_entity_clients, secret_syncs } = object;
+    return [clients, entity_clients, non_entity_clients, secret_syncs];
   }
 
   constructCsvRow(namespaceColumn, mountColumn = null, totalColumns, newColumns = null) {
@@ -146,19 +149,25 @@ export default class Attribution extends Component {
     const csvData = [];
     // added to clarify that the row of namespace totals without an auth method (blank) are not additional clients
     // but indicate the total clients for that ns, including its auth methods
+    const upgrade = this.args.upgradeExplanation
+      ? `\n **data contains an upgrade, mount summation may not equal namespace totals`
+      : '';
     const descriptionOfBlanks = this.isSingleNamespace
       ? ''
-      : `\n  *namespace totals, inclusive of auth method clients`;
+      : `\n  *namespace totals, inclusive of mount clients ${upgrade}`;
     const csvHeader = [
       'Namespace path',
-      `"Authentication method ${descriptionOfBlanks}"`,
+      `"Mount path ${descriptionOfBlanks}"`,
       'Total clients',
       'Entity clients',
       'Non-entity clients',
+      'Secrets sync clients',
     ];
 
     if (newAttribution) {
-      csvHeader.push('Total new clients, New entity clients, New non-entity clients');
+      csvHeader.push(
+        'Total new clients, New entity clients, New non-entity clients, New secrets sync clients'
+      );
     }
 
     totalAttribution.forEach((totalClientsObject) => {
@@ -199,7 +208,7 @@ export default class Attribution extends Component {
     const endRange = this.formattedEndDate ? `-${this.formattedEndDate}` : '';
     const csvDateRange = this.formattedStartDate + endRange;
     return this.isSingleNamespace
-      ? `clients_by_auth_method_${csvDateRange}`
+      ? `clients_by_mount_path_${csvDateRange}`
       : `clients_by_namespace_${csvDateRange}`;
   }