update README files

Author: Bamieh

src/legacy/core_plugins/telemetry/README.md

@@ -0,0 +1,9 @@
+# Kibana Telemetry Service
+
+Telemetry allows Kibana features to have usage tracked in the wild. The general term "telemetry" refers to multiple things:
+
+1. Integrating with the telemetry service to express how to collect usage data (Collecting).
+2. Sending a payload of usage data up to Elastic's telemetry cluster.
+3. Viewing usage data in the Kibana instance of the telemetry cluster (Viewing).
+
+This plugin is responsible for sending usage data to the telemetry cluster. For collecting usage data, use 

src/plugins/usage_collection/README.md

@@ -1,62 +1,105 @@
-# Kibana Telemetry Service
+# Kibana Usage Collection Service
 
-Telemetry allows Kibana features to have usage tracked in the wild. The general term "telemetry" refers to multiple things:
+Usage Collection allows collecting usage data for other services to consume (telemetry and monitoring).
+To integrate with the telemetry services for usage collection of your feature, there are 2 steps:
 
-1. Integrating with the telemetry service to express how to collect usage data (Collecting).
-2. Sending a payload of usage data up to Elastic's telemetry cluster, once per browser per day (Sending).
-3. Viewing usage data in the Kibana instance of the telemetry cluster (Viewing).
+1. Create a usage collector.
+2. Register the usage collector.
 
-You, the feature or plugin developer, mainly need to worry about the first meaning: collecting. To integrate with the telemetry services for usage collection of your feature, there are 2 steps:
-
-1. Create a usage collector using a factory function
-2. Register the usage collector with the Telemetry service
+## Creating and Registering Usage Collector
 
-NOTE: To a lesser extent, there's also a need to update the telemetry payload of Kibana stats and telemetry cluster field mappings to include your fields. This part is typically handled not by you, the developer, but different maintainers of the telemetry cluster. Usually, this step just means talk to the Platform team and have them approve your data model or added fields.
+All you need to provide is a `type` for organizing your fields, and a `fetch` method for returning your usage data. Then you need to make the Telemetry service aware of the collector by registering it.
 
-## Creating and Registering Usage Collector
+### New Platform:
 
-A usage collector object is an instance of a class called `UsageCollector`. A factory function on `server.usage.collectorSet` object allows you to create an instance of this class. All you need to provide is a `type` for organizing your fields, and a `fetch` method for returning your usage data. Then you need to make the Telemetry service aware of the collector by registering it.
+Make sure `usageCollection` is in your optional Plugins.
 
-Example:
+```json
+// plugin/kibana.json
+{
+  "id": "...",
+  "optionalPlugins": ["usageCollection"]
+}
+```
 
-```js
-// create usage collector
-const myCollector = server.usage.collectorSet.makeUsageCollector({
-  type: MY_USAGE_TYPE,
-  fetch: async callCluster => {
+```ts
+// server/plugin.ts
+class Plugin {
+  setup(core, plugins) {
+    registerMyPluginUsageCollector(plugins.usageCollection);
+  }
+}
+import { PluginSetupContract as UsageCollection } from 'src/plugins/usage_collection/server';
+import { CallCluster } from 'src/legacy/core_plugins/elasticsearch';
+
+// server/collectors/register.ts
+export function registerMyPluginUsageCollector(usageCollection: UsageCollection): void {
+  // create usage collector
+  const myCollector = usageCollection.makeUsageCollector({
+    type: MY_USAGE_TYPE,
+    fetch: async (callCluster: CallCluster) => {
 
     // query ES and get some data
     // summarize the data into a model
     // return the modeled object that includes whatever you want to track
 
-    return {
-      my_objects: {
-        total: SOME_NUMBER
-      }
-    };
-  },
-});
-
-// register usage collector
-server.usage.collectorSet.register(myCollector);
+      return {
+        my_objects: {
+          total: SOME_NUMBER
+        }
+      };
+    },
+  });
+
+  // register usage collector
+  usageCollection.registerCollector(myCollector);
+}
 ```
 
 Some background: The `callCluster` that gets passed to the `fetch` method is created in a way that's a bit tricky, to support multiple contexts the `fetch` method could be called. Your `fetch` method could get called as a result of an HTTP API request: in this case, the `callCluster` function wraps `callWithRequest`, and the request headers are expected to have read privilege on the entire `.kibana` index. The use case for this is stats pulled from a Kibana Metricbeat module, where the Beat calls Kibana's stats API in Kibana to invoke collection.
 
-The fetch method also might be called through an internal background task on the Kibana server, which currently lives in the `kibana_monitoring` module of the X-Pack Monitoring plugin, that polls for data and uploads it to Elasticsearch through a bulk API exposed by the Monitoring plugin for Elasticsearch. In this case, the `callCluster` method will be the internal system user and will have read privilege over the entire `.kibana` index.
-
 Note: there will be many cases where you won't need to use the `callCluster` function that gets passed in to your `fetch` method at all. Your feature might have an accumulating value in server memory, or read something from the OS.
 
+### Migrating to NP from Legacy Plugins:
+
+Pass `usageCollection` to the setup NP plugin setup function under plugins. Inside the `setup` function call the `registerCollector` like what you'd do in the NP example above.
+
+```js
+// index.js
+export const myPlugin = (kibana: any) => {
+  return new kibana.Plugin({
+    init: async function (server) {
+      const usageCollection = server.newPlatform.setup.plugins.usageCollection;
+      const plugins = {
+        usageCollection,
+      };
+      plugin(initializerContext).setup(core, plugins);
+    }
+  });
+}
+```
+
+### Legacy Plugins:
 
 Typically, a plugin will create the collector object and register it with the Telemetry service from the `init` method of the plugin definition, or a helper module called from `init`.
 
+```js
+// index.js
+export const myPlugin = (kibana: any) => {
+  return new kibana.Plugin({
+    init: async function (server) {
+      const usageCollecion = server.newPlatform.setup.plugins.usageCollection;
+      registerMyPluginUsageCollector(usageCollecion);
+    }
+  });
+}
+```
+
 ## Update the telemetry payload and telemetry cluster field mappings
 
 There is a module in the telemetry service that creates the payload of data that gets sent up to the telemetry cluster. 
 
-As of the time of this writing (pre-6.5.0) there are a few unpleasant realities with this module. Today, this module has to be aware of all the features that have integrated with it, which it does from hard-coding. It does this because at the time of creation, the payload implemented a designed model where X-Pack plugin info went together regardless if it was ES-specific or Kibana-specific. In hindsight, all the Kibana data could just be put together, X-Pack or not, which it could do in a generic way. This is a known problem and a solution will be implemented in an upcoming refactoring phase, as this would break the contract for model of data sent in the payload.
-
-The second reality is that new fields added to the telemetry payload currently mean that telemetry cluster field mappings have to be updated, so they can be searched and aggregated in Kibana visualizations. This is also a short-term obligation. In the next refactoring phase, collectors will need to use a proscribed data model that eliminates maintenance of mappings in the telemetry cluster.
+New fields added to the telemetry payload currently mean that telemetry cluster field mappings have to be updated, so they can be searched and aggregated in Kibana visualizations. This is also a short-term obligation. In the next refactoring phase, collectors will need to use a proscribed data model that eliminates maintenance of mappings in the telemetry cluster.
 
 ## Testing
 
@@ -65,7 +108,7 @@ There are a few ways you can test that your usage collector is working properly.
 1. The `/api/stats?extended=true` HTTP API in Kibana (added in 6.4.0) will call the fetch methods of all the registered collectors, and add them to a stats object you can see in a browser or in curl. To test that your usage collector has been registered correctly and that it has the model of data you expected it to have, call that HTTP API manually and you should see a key in the `usage` object of the response named after your usage collector's `type` field. This method tests the Metricbeat scenario described above where `callCluster` wraps `callWithRequest`.
 2. There is a dev script in x-pack that will give a sample of a payload of data that gets sent up to the telemetry cluster for the sending phase of telemetry. Collected data comes from:
     - The `.monitoring-*` indices, when Monitoring is enabled. Monitoring enhances the sent payload of telemetry by producing usage data potentially of multiple clusters that exist in the monitoring data. Monitoring data is time-based, and the time frame of collection is the last 15 minutes.
-    - Live-pulled from ES API endpoints. This will get just real-time stats without context of historical data. ✳
+    - Live-pulled from ES API endpoints. This will get just real-time stats without context of historical data.
     - The dev script in x-pack can be run on the command-line with:
       ```
       cd x-pack
@@ -76,17 +119,9 @@ There are a few ways you can test that your usage collector is working properly.
 3. In Dev mode, Kibana will send telemetry data to a staging telemetry cluster. Assuming you have access to the staging cluster, you can log in and check the latest documents for your new fields.
 4. If you catch the network traffic coming from your browser when a telemetry payload is sent, you can examine the request payload body to see the data. This can be tricky as telemetry payloads are sent only once per day per browser. Use incognito mode or clear your localStorage data to force a telemetry payload.
 
-✳ At the time of this writing, there is an open issue that in the sending phase, Kibana usage collectors are not "live-pulled" from Kibana API endpoints if Monitoring is disabled. The implementation on this depends on a new secure way to live-pull the data from the end-user's browser, as it would not be appropriate to supply only partial data if the logged-in user only has partial access to `.kibana`.
-
 ## FAQ
 
-1. **Can telemetry track UI interactions, such as button click?**  
-   Brief answer: no. Telemetry collection happens on the server-side so the usage data will only include information that the server-side is aware of. There is no generic way to do this today, but UI-interaction KPIs can be tracked with a custom server endpoint that gets called for tracking when the UI event happens.
-2. **Does the telemetry service have a hook that I can call whenever some event happens in my feature?**  
-   Brief answer: no. Telemetry collection is a fetch model, not a push model. Telemetry fetches info from your collector.
-3. **How should I design my data model?**  
+1. **How should I design my data model?**  
    Keep it simple, and keep it to a model that Kibana will be able to understand. In short, that means don't rely on nested fields (arrays with objects). Flat arrays, such as arrays of strings are fine.
-4. **Can the telemetry payload include dynamic fields?**  
-   Yes. When you talk to the Platform team about new fields being added, point out specifically which properties will have dynamic inner fields.
-5. **If I accumulate an event counter in server memory, which my fetch method returns, won't it reset when the Kibana server restarts?**  
+2. **If I accumulate an event counter in server memory, which my fetch method returns, won't it reset when the Kibana server restarts?**  
    Yes, but that is not a major concern. A visualization on such info might be a date histogram that gets events-per-second or something, which would be impacted by server restarts, so we'll have to offset the beginning of the time range when we detect that the latest metric is smaller than the earliest metric. That would be a pretty custom visualization, but perhaps future Kibana enhancements will be able to support that.

src/plugins/usage_collection/server/plugin.ts

@@ -46,10 +46,10 @@ export class Plugin {
   }
 
   public start() {
-    this.logger.info('Starting plugin');
+    this.logger.debug('Starting plugin');
   }
 
   public stop() {
-    this.logger.info('Stopping plugin');
+    this.logger.debug('Stopping plugin');
   }
 }