Squashed commit of the following:

commit 3e60a91
Author: jaymode <[email protected]>
Date:   Mon Feb 4 12:34:23 2019 -0700

    add licensing for authorization engine

commit 1c9a8e1
Author: jaymode <[email protected]>
Date:   Mon Feb 4 12:17:54 2019 -0700

    fix inconsistency in parameter name/type

commit 34aa55a
Author: Jay Modi <[email protected]>
Date:   Mon Feb 4 11:45:01 2019 -0700

    Authorization engines evaluate privileges for APIs (elastic#38219)

    This commit moves the evaluation of privileges from a few transport
    actions into the authorization engine. The APIs are used by other
    applications for making decisions and if a different authorization
    engine is used that is not role based, we should still allow these APIs
    to work. By moving this evaluation out of the transport action, the
    transport actions no longer have a dependency on roles.

commit 54d7b4c
Merge: e5615d2 715e581
Author: jaymode <[email protected]>
Date:   Mon Feb 4 08:06:14 2019 -0700

    Merge branch 'master' into security_authz_engine

commit e5615d2
Author: Jay Modi <[email protected]>
Date:   Mon Feb 4 07:53:37 2019 -0700

    Move request interceptors to AuthorizationService (elastic#38137)

    This change moves the RequestInterceptor iteration from the action
    filter to the AuthorizationService. This is done to remove the need
    for the use of a role within the request interceptors and replace it
    with the AuthorizationEngine. The AuthorizationEngine interface was
    also enhanced with a new method that is used to determine if a users
    permission on one index is a subset of their permissions on a list
    of indices or aliases.

    Additionally, this change addresses some leftover cleanups.

commit 0e1c191
Merge: 3280607 b7de8e1
Author: jaymode <[email protected]>
Date:   Thu Jan 31 08:56:45 2019 -0700

    Merge branch 'master' into security_authz_engine

commit 3280607
Author: Jay Modi <[email protected]>
Date:   Tue Jan 29 14:17:37 2019 -0700

    Allow authorization engines as an extension (elastic#37785)

    Authorization engines can now be registered by implementing a plugin,
    which also has a service implementation of a security extension. Only
    one extension may register an authorization engine and this engine will
    be used for all users except reserved realm users and internal users.

commit d628008
Author: jaymode <[email protected]>
Date:   Tue Jan 29 10:06:09 2019 -0700

    fix RBACEngine after restricted indices changes

commit 5074683
Merge: 74f2e99 3c9f703
Author: jaymode <[email protected]>
Date:   Tue Jan 29 08:09:39 2019 -0700

    Merge branch 'master' into security_authz_engine

commit 74f2e99
Merge: 7846ee8 899dfc3
Author: jaymode <[email protected]>
Date:   Fri Jan 25 15:02:07 2019 -0700

    Merge branch 'master' into security_authz_engine

commit 7846ee8
Merge: b9a2c81 a81931b
Author: jaymode <[email protected]>
Date:   Thu Jan 24 07:52:08 2019 -0700

    Merge branch 'master' into security_authz_engine

commit b9a2c81
Author: jaymode <[email protected]>
Date:   Tue Jan 22 09:48:11 2019 -0700

    Fix resolving restricted indices after merging

commit d98a77a
Merge: 83cde40 5c1a1f7
Author: jaymode <[email protected]>
Date:   Tue Jan 22 09:09:23 2019 -0700

    Merge branch 'master' into security_authz_engine

commit 83cde40
Author: Jay Modi <[email protected]>
Date:   Tue Jan 22 08:03:19 2019 -0700

    Add javadoc to the AuthorizationEngine interface (elastic#37620)

    This commit adds javadocs to the AuthorizationEngine interface aimed at
    developers of an authorization engine. Additionally, some classes were
    also moved to the core project so that they are ready to be exposed
    once we allow authorization engines to be plugged in.

commit 9a240c6
Author: Jay Modi <[email protected]>
Date:   Thu Jan 17 19:33:35 2019 -0700

    Encapsulate request, auth, and action name (elastic#37495)

    This change introduces a new class called RequestInfo that encapsulates
    the common objects that are passed to the authorization engine methods.
    By doing so, we give ourselves a way of adding additional data without
    breaking the interface. Additionally, this also reduces the need to
    ensure we pass these three parameters in the same order everywhere for
    consistency.

commit 6278eab
Merge: c555a44 4351a5e
Author: jaymode <[email protected]>
Date:   Thu Jan 17 07:51:32 2019 -0700

    Merge branch 'master' into security_authz_engine

commit c555a44
Merge: 1362ab6 ecf0de3
Author: jaymode <[email protected]>
Date:   Wed Jan 16 10:24:33 2019 -0700

    Merge branch 'master' into security_authz_engine

commit 1362ab6
Author: Jay Modi <[email protected]>
Date:   Wed Jan 16 10:23:45 2019 -0700

    Replace AuthorizedIndices class with a List (elastic#37328)

    This change replaces the AuthorizedIndices class with a simple list.
    The change to a simple list does remove the lazy loading of the
    authorized indices in favor of simpler code as the loading of this
    list is now an asynchronous operation that is delegated to the
    authorization engine.

commit 0246442
Merge: 8ccdc19 a2a40c5
Author: jaymode <[email protected]>
Date:   Tue Jan 15 10:49:12 2019 -0700

    Merge branch 'master' into security_authz_engine

commit 8ccdc19
Author: Jay Modi <[email protected]>
Date:   Mon Jan 7 13:43:22 2019 -0700

    Introduce asynchronous RBACEngine (elastic#36245)

    In order to support the concept of different authorization engines, this
    change begins the refactoring of the AuthorizationService to support
    this. Previously, the asynchronous work for authorization was performed
    by the AsyncAuthorizer class, but this tied the authorization service
    to a role based implementation. In this change, the authorize method
    become asynchronous and delegates much of the actual permission checking
    to an AuthorizationEngine. The pre-existing RBAC permission checking
    has been abstracted into the RBACEngine. The majority of calls to
    AuthorizationEngine instances are asynchronous as the underlying
    implementation may need to make network calls that should not block
    the current thread, which are often network threads.

    This change is meant to be built upon. The basic concepts are introduced
    without proper documentation, plumbing to enable other
    AuthorizationEngine types, and some items we may want to refactor.
    For example, the AuthorizedIndices class is lazily loaded but this might
    actually be something we want to make asynchronous. We pass a lot of the
    same arguments to the various methods and it would be prudent to wrap
    these in a class; this class would provide a way for us to pass
    additional items needed by future enhancements without breaking the
    interface and requiring updates to all implementations.

    See elastic#32435

Author: jaymode

build.gradle

@@ -233,6 +233,9 @@ allprojects {
     "org.elasticsearch.plugin:aggs-matrix-stats-client:${version}": ':modules:aggs-matrix-stats',
     "org.elasticsearch.plugin:percolator-client:${version}": ':modules:percolator',
     "org.elasticsearch.plugin:rank-eval-client:${version}": ':modules:rank-eval',
+    // for security example plugins
+    "org.elasticsearch.plugin:x-pack-core:${version}": ':x-pack:plugin:core',
+    "org.elasticsearch.client.x-pack-transport:${version}": ':x-pack:transport-client'
   ]
 
   /*

plugins/examples/security-authorization-engine/build.gradle

@@ -0,0 +1,46 @@
+apply plugin: 'elasticsearch.esplugin'
+
+esplugin {
+  name 'security-authorization-engine'
+  description 'An example spi extension plugin for security that implements an Authorization Engine'
+  classname 'org.elasticsearch.example.AuthorizationEnginePlugin'
+  extendedPlugins = ['x-pack-security']
+}
+
+dependencies {
+  compileOnly "org.elasticsearch.plugin:x-pack-core:${version}"
+  testCompile "org.elasticsearch.client.x-pack-transport:${version}"
+}
+
+
+integTestRunner {
+    systemProperty 'tests.security.manager', 'false'
+}
+
+integTestCluster {
+  dependsOn buildZip
+  setting 'xpack.security.enabled', 'true'
+  setting 'xpack.ilm.enabled', 'false'
+  setting 'xpack.ml.enabled', 'false'
+  setting 'xpack.monitoring.enabled', 'false'
+  setting 'xpack.license.self_generated.type', 'trial'
+
+  // This is important, so that all the modules are available too.
+  // There are index templates that use token filters that are in analysis-module and
+  // processors are being used that are in ingest-common module.
+  distribution = 'default'
+
+  setupCommand 'setupDummyUser',
+               'bin/elasticsearch-users', 'useradd', 'test_user', '-p', 'x-pack-test-password', '-r', 'custom_superuser'
+  waitCondition = { node, ant ->
+    File tmpFile = new File(node.cwd, 'wait.success')
+    ant.get(src: "http://${node.httpUri()}/_cluster/health?wait_for_nodes=>=${numNodes}&wait_for_status=yellow",
+            dest: tmpFile.toString(),
+            username: 'test_user',
+            password: 'x-pack-test-password',
+            ignoreerrors: true,
+            retries: 10)
+    return tmpFile.exists()
+  }
+}
+check.dependsOn integTest

plugins/examples/security-authorization-engine/src/main/java/org/elasticsearch/example/AuthorizationEnginePlugin.java

@@ -0,0 +1,30 @@
+/*
+ * Licensed to Elasticsearch under one or more contributor
+ * license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright
+ * ownership. Elasticsearch licenses this file to you under
+ * the Apache License, Version 2.0 (the "License"); you may
+ * not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.elasticsearch.example;
+
+import org.elasticsearch.plugins.ActionPlugin;
+import org.elasticsearch.plugins.Plugin;
+
+/**
+ * Plugin class that is required so that the code contained here may be loaded as a plugin.
+ * Additional items such as settings and actions can be registered using this plugin class.
+ */
+public class AuthorizationEnginePlugin extends Plugin implements ActionPlugin {
+}

plugins/examples/security-authorization-engine/src/main/java/org/elasticsearch/example/CustomAuthorizationEngine.java

@@ -0,0 +1,238 @@
+/*
+ * Licensed to Elasticsearch under one or more contributor
+ * license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright
+ * ownership. Elasticsearch licenses this file to you under
+ * the Apache License, Version 2.0 (the "License"); you may
+ * not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.elasticsearch.example;
+
+import org.elasticsearch.action.ActionListener;
+import org.elasticsearch.cluster.metadata.AliasOrIndex;
+import org.elasticsearch.xpack.core.security.action.user.GetUserPrivilegesRequest;
+import org.elasticsearch.xpack.core.security.action.user.GetUserPrivilegesResponse;
+import org.elasticsearch.xpack.core.security.action.user.GetUserPrivilegesResponse.Indices;
+import org.elasticsearch.xpack.core.security.action.user.HasPrivilegesRequest;
+import org.elasticsearch.xpack.core.security.action.user.HasPrivilegesResponse;
+import org.elasticsearch.xpack.core.security.action.user.HasPrivilegesResponse.ResourcePrivileges;
+import org.elasticsearch.xpack.core.security.authc.Authentication;
+import org.elasticsearch.xpack.core.security.authz.AuthorizationEngine;
+import org.elasticsearch.xpack.core.security.authz.ResolvedIndices;
+import org.elasticsearch.xpack.core.security.authz.RoleDescriptor;
+import org.elasticsearch.xpack.core.security.authz.RoleDescriptor.IndicesPrivileges;
+import org.elasticsearch.xpack.core.security.authz.accesscontrol.IndicesAccessControl;
+import org.elasticsearch.xpack.core.security.authz.accesscontrol.IndicesAccessControl.IndexAccessControl;
+import org.elasticsearch.xpack.core.security.authz.permission.FieldPermissions;
+import org.elasticsearch.xpack.core.security.authz.privilege.ApplicationPrivilegeDescriptor;
+import org.elasticsearch.xpack.core.security.authz.privilege.ConditionalClusterPrivilege;
+import org.elasticsearch.xpack.core.security.user.User;
+
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.LinkedHashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.stream.Collectors;
+
+/**
+ * A custom implementation of an authorization engine. This engine is extremely basic in that it
+ * authorizes based upon the name of a single role. If users have this role they are granted access.
+ */
+public class CustomAuthorizationEngine implements AuthorizationEngine {
+
+    @Override
+    public void resolveAuthorizationInfo(RequestInfo requestInfo, ActionListener<AuthorizationInfo> listener) {
+        final Authentication authentication = requestInfo.getAuthentication();
+        if (authentication.getUser().isRunAs()) {
+            final CustomAuthorizationInfo authenticatedUserAuthzInfo =
+                new CustomAuthorizationInfo(authentication.getUser().authenticatedUser().roles(), null);
+            listener.onResponse(new CustomAuthorizationInfo(authentication.getUser().roles(), authenticatedUserAuthzInfo));
+        } else {
+            listener.onResponse(new CustomAuthorizationInfo(authentication.getUser().roles(), null));
+        }
+    }
+
+    @Override
+    public void authorizeRunAs(RequestInfo requestInfo, AuthorizationInfo authorizationInfo, ActionListener<AuthorizationResult> listener) {
+        if (isSuperuser(requestInfo.getAuthentication().getUser().authenticatedUser())) {
+            listener.onResponse(AuthorizationResult.granted());
+        } else {
+            listener.onResponse(AuthorizationResult.deny());
+        }
+    }
+
+    @Override
+    public void authorizeClusterAction(RequestInfo requestInfo, AuthorizationInfo authorizationInfo,
+                                       ActionListener<AuthorizationResult> listener) {
+        if (isSuperuser(requestInfo.getAuthentication().getUser())) {
+            listener.onResponse(AuthorizationResult.granted());
+        } else {
+            listener.onResponse(AuthorizationResult.deny());
+        }
+    }
+
+    @Override
+    public void authorizeIndexAction(RequestInfo requestInfo, AuthorizationInfo authorizationInfo,
+                                     AsyncSupplier<ResolvedIndices> indicesAsyncSupplier,
+                                     Map<String, AliasOrIndex> aliasOrIndexLookup,
+                                     ActionListener<IndexAuthorizationResult> listener) {
+        if (isSuperuser(requestInfo.getAuthentication().getUser())) {
+            indicesAsyncSupplier.getAsync(ActionListener.wrap(resolvedIndices -> {
+                Map<String, IndexAccessControl> indexAccessControlMap = new HashMap<>();
+                for (String name : resolvedIndices.getLocal()) {
+                    indexAccessControlMap.put(name, new IndexAccessControl(true, FieldPermissions.DEFAULT, null));
+                }
+                IndicesAccessControl indicesAccessControl =
+                    new IndicesAccessControl(true, Collections.unmodifiableMap(indexAccessControlMap));
+                listener.onResponse(new IndexAuthorizationResult(true, indicesAccessControl));
+            }, listener::onFailure));
+        } else {
+            listener.onResponse(new IndexAuthorizationResult(true, IndicesAccessControl.DENIED));
+        }
+    }
+
+    @Override
+    public void loadAuthorizedIndices(RequestInfo requestInfo, AuthorizationInfo authorizationInfo,
+                                      Map<String, AliasOrIndex> aliasOrIndexLookup, ActionListener<List<String>> listener) {
+        if (isSuperuser(requestInfo.getAuthentication().getUser())) {
+            listener.onResponse(new ArrayList<>(aliasOrIndexLookup.keySet()));
+        } else {
+            listener.onResponse(Collections.emptyList());
+        }
+    }
+
+    @Override
+    public void validateIndexPermissionsAreSubset(RequestInfo requestInfo, AuthorizationInfo authorizationInfo,
+                                                  Map<String, List<String>> indexNameToNewNames,
+                                                  ActionListener<AuthorizationResult> listener) {
+        if (isSuperuser(requestInfo.getAuthentication().getUser())) {
+            listener.onResponse(AuthorizationResult.granted());
+        } else {
+            listener.onResponse(AuthorizationResult.deny());
+        }
+    }
+
+    @Override
+    public void checkPrivileges(Authentication authentication, AuthorizationInfo authorizationInfo,
+                                HasPrivilegesRequest hasPrivilegesRequest,
+                                Collection<ApplicationPrivilegeDescriptor> applicationPrivilegeDescriptors,
+                                ActionListener<HasPrivilegesResponse> listener) {
+        if (isSuperuser(authentication.getUser())) {
+            listener.onResponse(getHasPrivilegesResponse(authentication, hasPrivilegesRequest, true));
+        } else {
+            listener.onResponse(getHasPrivilegesResponse(authentication, hasPrivilegesRequest, false));
+        }
+    }
+
+    @Override
+    public void getUserPrivileges(Authentication authentication, AuthorizationInfo authorizationInfo, GetUserPrivilegesRequest request,
+                                  ActionListener<GetUserPrivilegesResponse> listener) {
+        if (isSuperuser(authentication.getUser())) {
+            listener.onResponse(getUserPrivilegesResponse(true));
+        } else {
+            listener.onResponse(getUserPrivilegesResponse(false));
+        }
+    }
+
+    private HasPrivilegesResponse getHasPrivilegesResponse(Authentication authentication, HasPrivilegesRequest hasPrivilegesRequest,
+                                                           boolean authorized) {
+        Map<String, Boolean> clusterPrivMap = new HashMap<>();
+        for (String clusterPriv : hasPrivilegesRequest.clusterPrivileges()) {
+            clusterPrivMap.put(clusterPriv, authorized);
+        }
+        final Map<String, ResourcePrivileges> indices = new LinkedHashMap<>();
+        for (IndicesPrivileges check : hasPrivilegesRequest.indexPrivileges()) {
+            for (String index : check.getIndices()) {
+                final Map<String, Boolean> privileges = new HashMap<>();
+                final HasPrivilegesResponse.ResourcePrivileges existing = indices.get(index);
+                if (existing != null) {
+                    privileges.putAll(existing.getPrivileges());
+                }
+                for (String privilege : check.getPrivileges()) {
+                    privileges.put(privilege, authorized);
+                }
+                indices.put(index, new ResourcePrivileges(index, privileges));
+            }
+        }
+        final Map<String, Collection<ResourcePrivileges>> privilegesByApplication = new HashMap<>();
+        Set<String> applicationNames = Arrays.stream(hasPrivilegesRequest.applicationPrivileges())
+            .map(RoleDescriptor.ApplicationResourcePrivileges::getApplication)
+            .collect(Collectors.toSet());
+        for (String applicationName : applicationNames) {
+            final Map<String, HasPrivilegesResponse.ResourcePrivileges> appPrivilegesByResource = new LinkedHashMap<>();
+            for (RoleDescriptor.ApplicationResourcePrivileges p : hasPrivilegesRequest.applicationPrivileges()) {
+                if (applicationName.equals(p.getApplication())) {
+                    for (String resource : p.getResources()) {
+                        final Map<String, Boolean> privileges = new HashMap<>();
+                        final HasPrivilegesResponse.ResourcePrivileges existing = appPrivilegesByResource.get(resource);
+                        if (existing != null) {
+                            privileges.putAll(existing.getPrivileges());
+                        }
+                        for (String privilege : p.getPrivileges()) {
+                            privileges.put(privilege, authorized);
+                        }
+                        appPrivilegesByResource.put(resource, new HasPrivilegesResponse.ResourcePrivileges(resource, privileges));
+                    }
+                }
+            }
+            privilegesByApplication.put(applicationName, appPrivilegesByResource.values());
+        }
+        return new HasPrivilegesResponse(authentication.getUser().principal(), authorized, clusterPrivMap, indices.values(),
+            privilegesByApplication);
+    }
+
+    private GetUserPrivilegesResponse getUserPrivilegesResponse(boolean isSuperuser) {
+        final Set<String> cluster = isSuperuser ? Collections.singleton("ALL") : Collections.emptySet();
+        final Set<ConditionalClusterPrivilege> conditionalCluster = Collections.emptySet();
+        final Set<GetUserPrivilegesResponse.Indices> indices = isSuperuser ? Collections.singleton(new Indices(Collections.singleton("*"),
+            Collections.singleton("*"), Collections.emptySet(), Collections.emptySet(), true)) : Collections.emptySet();
+
+        final Set<RoleDescriptor.ApplicationResourcePrivileges> application = isSuperuser ?
+            Collections.singleton(
+                RoleDescriptor.ApplicationResourcePrivileges.builder().application("*").privileges("*").resources("*").build()) :
+            Collections.emptySet();
+        final Set<String> runAs = isSuperuser ? Collections.singleton("*") : Collections.emptySet();
+        return new GetUserPrivilegesResponse(cluster, conditionalCluster, indices, application, runAs);
+    }
+
+    public static class CustomAuthorizationInfo implements AuthorizationInfo {
+
+        private final String[] roles;
+        private final CustomAuthorizationInfo authenticatedAuthzInfo;
+
+        CustomAuthorizationInfo(String[] roles, CustomAuthorizationInfo authenticatedAuthzInfo) {
+            this.roles = roles;
+            this.authenticatedAuthzInfo = authenticatedAuthzInfo;
+        }
+
+        @Override
+        public Map<String, Object> asMap() {
+            return Collections.singletonMap("roles", roles);
+        }
+
+        @Override
+        public CustomAuthorizationInfo getAuthenticatedUserAuthorizationInfo() {
+            return authenticatedAuthzInfo;
+        }
+    }
+
+    private boolean isSuperuser(User user) {
+        return Arrays.binarySearch(user.roles(), "custom_superuser") > -1;
+    }
+}

plugins/examples/security-authorization-engine/src/main/java/org/elasticsearch/example/ExampleAuthorizationEngineExtension.java

@@ -0,0 +1,35 @@
+/*
+ * Licensed to Elasticsearch under one or more contributor
+ * license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright
+ * ownership. Elasticsearch licenses this file to you under
+ * the Apache License, Version 2.0 (the "License"); you may
+ * not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.elasticsearch.example;
+
+import org.elasticsearch.common.settings.Settings;
+import org.elasticsearch.xpack.core.security.SecurityExtension;
+import org.elasticsearch.xpack.core.security.authz.AuthorizationEngine;
+
+/**
+ * Security extension class that registers the custom authorization engine to be used
+ */
+public class ExampleAuthorizationEngineExtension implements SecurityExtension {
+
+    @Override
+    public AuthorizationEngine getAuthorizationEngine(Settings settings) {
+        return new CustomAuthorizationEngine();
+    }
+}

plugins/examples/security-authorization-engine/src/main/resources/META-INF/services/org.elasticsearch.xpack.core.security.SecurityExtension

@@ -0,0 +1 @@
+org.elasticsearch.example.ExampleAuthorizationEngineExtension