[salesforce] Add salesforce package foundation along with login_rest data stream

.github/CODEOWNERS

@@ -152,6 +152,7 @@
 /packages/radware @elastic/security-external-integrations
 /packages/redis @elastic/obs-service-integrations
 /packages/redisenterprise @elastic/obs-service-integrations
+/packages/salesforce @elastic/obs-service-integrations
 /packages/santa @elastic/security-external-integrations
 /packages/security_detection_engine @elastic/protections
 /packages/sentinel_one @elastic/security-external-integrations

packages/salesforce/_dev/build/build.yml

@@ -0,0 +1,3 @@
+dependencies:
+  ecs:
+    reference: git@v8.4.0

packages/salesforce/_dev/build/docs/README.md

@@ -0,0 +1,169 @@
+# Salesforce Integration
+
+## Overview
+
+The Salesforce integration allows you to monitor a [Salesforce](https://www.salesforce.com/) instance. Salesforce is a customer relationship management (CRM) platform. It provides an ecosystem for businesses to manage marketing, sales, commerce, service, and IT teams from anywhere with one integrated CRM platform.
+
+Use the Salesforce integration to:
+- Gain insights into login and other operational activities by the users of your organization.
+- Create visualizations to monitor, measure and analyze the usage trend and key data, and derive business insights.
+- Create alerts to reduce the MTTD and also the MTTR by referencing relevant logs when troubleshooting an issue.
+
+As an example, you can use the data from this integration to understand the activity patterns of users based on region or the distribution of users by license type. 
+
+## Data streams
+
+The Salesforce integration collects log events using the REST API of Salesforce.
+
+**Logs** help you keep a record of events happening in Salesforce.
+Log data streams collected by the Salesforce integration include [Login](https://developer.salesforce.com/docs/atlas.en-us.object_reference.meta/object_reference/sforce_api_objects_eventlogfile_login.htm).
+
+Data streams:
+- `login_rest`: Tracks login activity of users who log in to Salesforce.
+
+## Compatibility
+
+This integration has been tested against Salesforce `Spring '22 (v54.0) release`.
+
+In order to find out the Salesforce version of your Instance, see below:
+
+1. On the Home tab in Salesforce Classic, in the top right corner of the screen is a link to releases like `Summer '22`. This indicates your release.
+
+2. An alternative way to find out the version of Salesforce is by hitting the following URL:
+	- Format: (Salesforce Instance URL)/services/data
+	- Example: https://elastic1234-dev-ed.my.salesforce.com/services/data
+
+Example response:
+```xml
+<Versions>
+	<Version>
+		<label>Winter '22</label>
+		<url>/services/data/v53.0</url>
+		<version>53.0</version>
+	</Version>
+	<Version>
+		<label>Spring '22</label>
+		<url>/services/data/v54.0</url>
+		<version>54.0</version>
+	</Version>
+	<Version>
+		<label>Summer '22</label>
+		<url>/services/data/v55.0</url>
+		<version>55.0</version>
+	</Version>
+</Versions>
+```
+The last one on the list is the release of your instance. In the example above, the version is `Summer '22` i.e. `v55.0`.
+
+## Prerequisites
+
+You need Elasticsearch for storing and searching your data and Kibana for visualizing and managing it.
+You can use our hosted Elasticsearch Service on Elastic Cloud, which is recommended or self-manage the Elastic Stack on your own hardware.
+
+In your Salesforce instance, ensure that `API Enabled permission` is selected for the user profile. Follow the below steps to enable the same:
+
+1. Go to `Setup` > `Quick Find` > `Users`, and Click on `Users`.
+2. Click on the profile link associated with the `User Account` used for data collection.
+3. Search for `API Enabled` permission on the same page. In case it’s not present, search it under `System Permissions` and check if `API Enabled` privilege is selected. If not, enable it for data collection.
+
+## Set Up
+
+For step-by-step instructions on how to set up an integration, see the [Getting started](https://www.elastic.co/guide/en/welcome-to-elastic/current/getting-started-observability.html) guide.
+
+## Configuration
+
+You need the following information from your Salesforce instance to configure this integration in Elastic:
+
+### Salesforce Instance URL
+
+The instance your Salesforce Organization uses is indicated in the URL of your browser's address bar in Salesforce Classic. The value before 'salesforce.com' is your Salesforce Instance.
+
+Example URL: https://na9.salesforce.com/home/home.jsp
+
+In the above example, the value before 'salesforce.com' is your Salesforce Instance. In this example, the Salesforce Organization is located on NA9. 
+
+The Salesforce Instance URL is: https://na9.salesforce.com
+
+In Salesforce Lightning, it is available under the user name in the “View Profile” tab.
+
+### Client Key and Client Secret for Authentication
+
+In order to use this integration, you need to create a new Salesforce Application using OAuth. Follow the steps below to create a connected application in Salesforce:
+
+1. Login to [Salesforce](https://login.salesforce.com/) with the same user credentials that you want to collect data with.
+2. Click on Setup on the top right menu bar. On the Setup page search `App Manager` in the `Search Setup` search box at the top of the page, then select `App Manager`.
+3. Click *New Connected App*.
+4. Provide a name for the connected application. This will be displayed in the App Manager and on its App Launcher tile. 
+5. Enter the API name. The default is a version of the name without spaces. Only letters, numbers, and underscores are allowed. If the original app name contains any other characters, edit the default name.
+6. Enter the contact email for Salesforce.
+7. Under the API (Enable OAuth Settings) section of the page, select *Enable OAuth Settings*.
+8. In the Callback URL enter the Instance URL (Please refer to `Salesforce Instance URL`)
+9. Select the following OAuth scopes to apply to the connected app:
+	- Manage user data via APIs (api). 
+	- Perform requests at any time (refresh_token, offline_access).
+	- (Optional) In case of data collection, if any permission issues arise, add the Full access (full) scope.
+10. Select *Require Secret for the Web Server Flow* to require the app's client secret in exchange for an access token.
+11. Select *Require Secret for Refresh Token Flow* to require the app's client secret in the authorization request of a refresh token and hybrid refresh token flow.
+12. Click Save. It may take approximately 10 minutes for the changes to take effect.
+13. Click Continue and then under API details click Manage Consumer Details, Verify the user account using Verification Code.
+14. Copy `Consumer Key` and `Consumer Secret` from the Consumer Details section, which should be populated as value to Client ID and Client Secret respectively in the configuration.
+
+For more details on how to Create a Connected App refer to the salesforce documentation [here](https://help.salesforce.com/apex/HTViewHelpDoc?id=connected_app_create.htm).
+
+### Username
+
+User Id of the registered user in Salesforce.
+
+### Password
+
+Password used for authenticating the above user.
+
+## Additional Information
+
+Follow the steps below, in case you need to find the API version:
+
+1. Go to `Setup` > `Quick Find` > `Apex Classes`.
+2. Click the `New` button.
+3. Click the `Version Settings` tab.
+4. Refer to the `Version` dropdown for the API Version number.
+
+## Validation
+
+After the integration is successfully configured, clicking on the Assets tab of the Salesforce Integration should display a list of available dashboards. Click on the dashboard available for your configured datastream. It should be populated with the required data.
+
+## Troubleshooting
+
+- In case of data ingestion if the user finds the following type of error logs:
+```
+{
+    "log.level": "error",
+    "@timestamp": "2022-11-24T12:59:36.835+0530",
+    "log.logger": "input.httpjson-cursor",
+    "log.origin": {
+        "[file.name](http://file.name/)": "compat/compat.go",
+        "file.line": 124
+    },
+    "message": "Input 'httpjson-cursor' failed with: input.go:130: input 8A049E17A5CA661D failed (id=8A049E17A5CA661D)\n\toauth2 client: error loading credentials using user and password: oauth2: cannot fetch token: 400 Bad Request\n\tResponse: {\"error\":\"invalid_grant\",\"error_description\":\"authentication failure\"}",
+    "[service.name](http://service.name/)": "filebeat",
+    "id": "8A049E17A5CA661D",
+    "ecs.version": "1.6.0"
+}
+```
+Please check if the `API Enabled permission` is provided to the `profile` associated with the `username` used as part of the integration.
+Please refer to the Prerequisites section above for more information.
+
+If the error continues follow these steps:
+
+1. Go to `Setup` > `Quick Find` > `Manage Connected Apps`.
+2. Click on the Connected App name created by you to generate the client id and client secret (Refer to Client Key and Client Secret for Authentication) under the Master Label.
+3. Click on Edit Policies, and select `Relax IP restrictions` from the dropdown for IP Relaxation.
+
+## Logs reference
+
+### Login Rest
+
+This is the `login_rest` data stream. It represents events containing details about your organization's user login history.
+
+{{event "login_rest"}}
+
+{{fields "login_rest"}}

packages/salesforce/_dev/deploy/docker/docker-compose.yml

@@ -0,0 +1,15 @@
+version: '2.3'
+services:
+  salesforce:
+    image: docker.elastic.co/observability/stream:v0.8.0
+    hostname: salesforce
+    ports:
+      - 8010
+    volumes:
+      - ./files:/files:ro
+    environment:
+      PORT: 8010
+    command:
+      - http-server
+      - --addr=:8010
+      - --config=/files/config.yml

packages/salesforce/_dev/deploy/docker/files/config.yml

@@ -0,0 +1,71 @@
+rules:
+  - path: /services/oauth2/token
+    methods: ["POST"]
+    responses:
+      - status_code: 200
+        headers:
+          content-type: ['application/json']
+        body: '{"access_token":"access_token","instance_url":"https://temporary-intance-url","id":"https://login.salesforce.com/id/temp_id/temp_token","token_type":"Bearer","issued_at":"1633689089545","signature":"signature"}'
+  - path: /services/data/v54.0/query
+    methods: ["GET"]
+    query_params:
+      q: ["SELECT Id,CreatedDate,LogDate,LogFile FROM EventLogFile WHERE Interval = 'Hourly' AND EventType = 'Login' ORDER BY LogDate ASC NULLS FIRST"]
+    responses:
+      - status_code: 200
+        body: '{"totalSize":1,"done":true,"records":[{"attributes":{"type":"EventLogFile","url":"/services/data/v54.0/sobjects/EventLogFile/0AT5j00002GVrfnGAD"},"Id":"0AT5j00002GVrfnGAD","CreatedDate":"2022-09-14T21:43:41.000+0000","LogDate":"2022-09-13T00:00:00.000+0000","LogFile":"/services/data/v54.0/sobjects/EventLogFile/0AT5j00002GVrfnGAD/LogFile"}]}'
+        headers:
+          content-type: ["text/json"]
+  - path: /services/data/v54.0/query
+    methods: ["GET"]
+    query_params:
+      q: ["SELECT Action,CreatedByContext,CreatedById,CreatedByIssuer,CreatedDate,DelegateUser,Display,Id,ResponsibleNamespacePrefix,Section FROM SetupAuditTrail ORDER BY CreatedDate ASC NULLS FIRST"]
+    responses:
+      - status_code: 200
+        body: '{"done":true,"records":[{"Action":"insertConnectedApplication","CreatedByContext":"Einstein","CreatedById":"0055j000000utlPAAQ","CreatedByIssuer":null,"CreatedDate":"2022-08-16T09:26:38.000+0000","DelegateUser":"user1","Display":"For user user@elastic.co, the User Verified Email status changed to verified","Id":"0Ym5j000019nwonCAA","ResponsibleNamespacePrefix":"namespaceprefix","Section":"Connected Apps","attributes":{"type":"SetupAuditTrail","url":"/services/data/v54.0/sobjects/SetupAuditTrail/0Ym5j000019nwonCAA"}}],"totalSize":160}'
+        headers:
+          content-type: ["text/json"]
+  - path: /services/data/v54.0/sobjects/EventLogFile/0AT5j00002GVrfnGAD/LogFile
+    methods: ["GET"]
+    responses:
+      - status_code: 200
+        headers:
+          content-type: ["text/csv"]
+        body: |-
+          "EVENT_TYPE","TIMESTAMP","REQUEST_ID","ORGANIZATION_ID","USER_ID","RUN_TIME","CPU_TIME","URI","SESSION_KEY","LOGIN_KEY","USER_TYPE","REQUEST_STATUS","DB_TOTAL_TIME","BROWSER_TYPE","API_TYPE","API_VERSION","USER_NAME","TLS_PROTOCOL","CIPHER_SUITE","AUTHENTICATION_METHOD_REFERENCE","TIMESTAMP_DERIVED","USER_ID_DERIVED","CLIENT_IP","URI_ID_DERIVED","LOGIN_STATUS","SOURCE_IP"
+          "Login","20221122044615.591","4ehU_U-nbQyAPFl1cJILm-","00D5j000000VI3n","0055j000000utlP","83","30","/index.jsp","","QfNecrLXSII6fsBq","Standard","Success","52435102","Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/94.0.4606.71 Safari/537.36","f","9998.0","user@elastic.co","TLSv1.2","ECDHE-RSA-AES256-GCM-SHA384","","2022-11-22T04:46:15.591Z","0055j000000utlPAAQ","81.2.69.142","s4heK3WbH-lcJIL3-n","LOGIN_NO_ERROR","81.2.69.142"
+  - path: /services/data/v54.0/query
+    methods: ["GET"]
+    query_params:
+      q: ["SELECT Id,CreatedDate,LogDate,LogFile FROM EventLogFile WHERE Interval = 'Hourly' AND EventType = 'Logout' ORDER BY LogDate ASC NULLS FIRST"]
+    responses:
+      - status_code: 200
+        body: '{"totalSize":1,"done":true,"records":[{"attributes":{"type":"EventLogFile","url":"/services/data/v54.0/sobjects/EventLogFile/0AT5j00002GWEsRGAX"},"Id":"0AT5j00002GWEsRGAX","CreatedDate":"2022-09-19T21:03:41.000+0000","LogDate":"2022-09-18T00:00:00.000+0000","LogFile":"/services/data/v54.0/sobjects/EventLogFile/0AT5j00002GWEsRGAX/LogFile"}]}'
+        headers:
+          content-type: ["text/json"]
+  - path: /services/data/v54.0/sobjects/EventLogFile/0AT5j00002GWEsRGAX/LogFile
+    methods: ["GET"]
+    responses:
+      - status_code: 200
+        headers:
+          content-type: ["text/csv"]
+        body: |-
+          "EVENT_TYPE","TIMESTAMP","REQUEST_ID","ORGANIZATION_ID","USER_ID","USER_TYPE","SESSION_TYPE","SESSION_LEVEL","BROWSER_TYPE","PLATFORM_TYPE","RESOLUTION_TYPE","APP_TYPE","CLIENT_VERSION","API_TYPE","API_VERSION","USER_INITIATED_LOGOUT","SESSION_KEY","LOGIN_KEY","TIMESTAMP_DERIVED","USER_ID_DERIVED","CLIENT_IP"
+          "Logout","20221122073725.779","4exLFFQZ1234xFl1cJNwOV","00D5j001234VI3n","0055j000000utlP","S","O","1","Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/94.0.4606.81 Safari/537.36","1015","9999","1000","9998","f","54.0","0","WvtsJ1235oW24EbH","Obv9123BzbaxqCo1","2022-11-22T07:37:25.779Z","0055j000000utlPAAQ","81.2.69.142"
+  - path: /services/data/v54.0/query
+    methods: ["GET"]
+    query_params:
+      q: ["SELECT Id,CreatedDate,LogDate,LogFile FROM EventLogFile WHERE Interval = 'Hourly' AND (EventType = 'ApexCallout' OR EventType = 'ApexExecution' OR EventType = 'ApexRestApi' OR EventType = 'ApexSoap' OR EventType = 'ApexTrigger' OR EventType = 'ExternalCustomApexCallout') ORDER BY LogDate ASC NULLS FIRST"]
+    responses:
+      - status_code: 200
+        body: '{"done":true,"records":[{"CreatedDate":"2022-10-01T23:22:27.000+0000","Id":"0AT5j00002GWEsRGAY","LogDate":"2022-09-30T00:00:00.000+0000","LogFile":"/services/data/v54.0/sobjects/EventLogFile/0AT5j00002GWEsRGAY/LogFile","attributes":{"type":"EventLogFile","url":"/services/data/v54.0/sobjects/EventLogFile/0AT5j00002GWEsRGAY"}}],"totalSize":1}'
+        headers:
+          content-type: ["text/json"]
+  - path: /services/data/v54.0/sobjects/EventLogFile/0AT5j00002GWEsRGAY/LogFile
+    methods: ["GET"]
+    responses:
+      - status_code: 200
+        headers:
+          content-type: ["text/csv"]
+        body: |-
+          "EVENT_TYPE","TIMESTAMP","REQUEST_ID","ORGANIZATION_ID","USER_ID","RUN_TIME","CPU_TIME","URI","SESSION_KEY","LOGIN_KEY","TYPE","METHOD","SUCCESS","TIME","REQUEST_SIZE","RESPONSE_SIZE","URL","TIMESTAMP_DERIVED","USER_ID_DERIVED","CLIENT_IP","URI_ID_DERIVED"
+          "ApexCallout","20221122044615.591","ABCDE","00D5j000000VABC","0055j000000ABCD","1305","10","CALLOUT-LOG","ABCDEF","ABCDEFGH","OData","GET","1","1293","10","256","https://temp.sh/odata/Accounts","2022-11-22T04:46:15.591Z","0055j012345utlPAAQ","127.0.0.1","0055j000000utlPABCD"

packages/salesforce/changelog.yml

@@ -0,0 +1,7 @@
+# newer versions go on top
+
+- version: 0.1.0
+  changes:
+    - description: Salesforce integration package with "login_rest" data stream.
+      link: https://github.com/elastic/integrations/pull/4261
+      type: enhancement

packages/salesforce/data_stream/login_rest/_dev/test/pipeline/test-common-config.yml

@@ -0,0 +1,5 @@
+dynamic_fields:
+  event.ingested: ".*"
+fields:
+  tags:
+    - preserve_original_event

packages/salesforce/data_stream/login_rest/_dev/test/pipeline/test-login-rest.log

@@ -0,0 +1 @@
+{"API_TYPE":"f","API_VERSION":"9998.0","AUTHENTICATION_METHOD_REFERENCE":"","BROWSER_TYPE":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/IP_ADDRESS_REMOVED Safari/537.36","CIPHER_SUITE":"ECDHE-RSA-AES256-GCM-SHA384","CLIENT_IP":"175.16.199.0","CPU_TIME":"63","DB_TOTAL_TIME":"93531912","EVENT_TYPE":"Login","LOGIN_KEY":"NK2oPJrJze8tH+vC","LOGIN_STATUS":"LOGIN_NO_ERROR","ORGANIZATION_ID":"00D5j000000VI3n","REQUEST_ID":"4lGJ1R0ZSWVXwFl1cJIRH-","REQUEST_STATUS":"Success","RUN_TIME":"156","SESSION_KEY":"","SOURCE_IP":"175.16.199.0","TIMESTAMP":"20220913052243.429","TIMESTAMP_DERIVED":"2022-09-13T05:22:43.429Z","TLS_PROTOCOL":"TLSv1.2","URI":"/index.jsp","URI_ID_DERIVED":"s4heK3WbH-lcJIL3-n","USER_ID":"0055j000000utlP","USER_ID_DERIVED":"0055j000000utlPAAQ","USER_NAME":"user@elastic.co","USER_TYPE":"Standard"}