Deprecate and replace old Index privileges related to indexing documents

[bizybot]

Existing Index privilege names fail to convey the meaning properly and a few like create have their own unexpected semantics (it allows indexing document only via Index API). Since we are adding a new create_doc privilege, we want to take this opportunity to clean up the existing privileges and either provide replica privilege if the semantics are correct or provide an alternative that is semantically correct. Also, the naming can be explicit about the operation that is performed.

Following are the existing Index privileges related to documents indexing lifecycle:
create: The privilege allows for indexing documents but only via Index/Bulk API but no update via Update API. It also allows update mapping actions. This one does not prevent document updates when using Index APIs.
index: The privilege allows indexing documents via Index/Bulk API, also allows for update action and mapping actions.
delete: The privilege allows deletion of documents
write: This privilege allows all write operations (index, deletes, updates) along with update mapping actions.
read: Read-only access to actions like count, explain, get, mget, get indexed scripts, etc.

Following are the proposed new Index privileges:

• create_doc: This one is with changed semantics, allowing only indexing new documents but no updates. Plus this one removes the unexpected semantics of restricting this operation via Index API only. This will allow indexing new documents via Index/Bulk API and allows update mapping action. Add 'create_doc' index privilege #45806
• index_doc: This will be similar to index privilege.
• delete_doc: This will be similar to delete privilege.
• write_doc: This will be similar to write privilege.
• read_doc: This will be similar to read privilege.

Privilege ▼ Actions ►	Index new document	Update document	Delete document	Update mapping	Read actions (search, count etc.)
create_doc	Allow	Deny	Deny	Allow	Deny
index_doc	Allow	Allow	Deny	Allow	Deny
delete_doc	Deny	Deny	Allow	Deny	Deny
write_doc	Allow	Allow	Allow	Allow	Deny
read_doc	Deny	Deny	Deny	Deny	Allow

Additional considerations:

• During the discussion of naming of create_doc we had a proposal to split the index privilege into:
  index_doc: Privilege to index documents. It also includes create_doc privilege.
  I think this would be a breaking change but adding it to discuss this.
  update_doc: Privilege to update documents. Not sure how this would work, for example, an upsert case allows update of an existing document or index new document if it does not exist.
  If we want to support this splitting of index privilege into two with above semantics then it would be a breaking change and might not be possible other than to modify the index engine to handle it.

• read_doc privilege: We could consider some more granular approach. I see an enhancement request: https://github.com/elastic/enhancements/issues/5470 which talks about providing a way to restrict users to certain search types?

Proposed path forward:

• When do we introduce these new Index privilege changes and deprecate old?
  Introduce these new index privileges in v8.0.0 and start logging deprecated warnings.
• When do we remove the usage of old Index privileges? v9.0.0
• Can we provide some migration tool/upgrade assistance?
  We have an issue opened to discuss this possibility to enhance the deprecation info API Allow Deprecation Info API to check for deprecated security configuration #47714
  We could consider providing a tool for migration but it might not be possible to migrate all the role descriptors using the deprecated privilege names.
  The problem exists for the privilege which does not have the exact alternative or would change the behavior if we use the alternative, eg. create with possible alternative create_doc.
  • There are different places where the index privilege is used during role definition:
    • file (roles.yml).(If an alternative exists for the privilege we can update the role definitions in the file. This option is less likely as we would want to avoid file mutations)
    • built-in roles (If an alternative exists for the privilege we can update the role definitions)
    • role definitions indexed in security index via API (If an alternative exists for the privilege we can update the role definitions)
    • dynamic role definitions - custom roles provider extension (This is a custom security extension and will require users to update the extension).
• Do we want to consider the additional changes like splitting the index privilege or more granular privileges for read?

This issue exists to gather feedback on the path forward and comments around introducing/removing/updating index privileges in relation to indexing documents.

[elasticmachine]

Pinging @elastic/es-security

[elasticmachine]

Pinging @elastic/es-distributed

[elasticmachine]

Pinging @elastic/es-core-features

[bizybot]

Hi @ywelsch I have added your suggestions to split the index into index_doc and update_doc privilege but wanted to confirm if it is what you were proposing. I feel that it would be breaking change and do you think strongly about it? Thanks for your inputs.

[bizybot]

UI currently uses GET _security/privilege/_builtin to list the built-in privileges. We could enhance the response to include the deprecated index privilege names and alternatives/suggestions as:

{
  "cluster": [
    "..."
  ],
  "index": [
    "all",
    "create",
    "create_doc"
    "...",
    "write"
  ],
  "deprecations": {
    "cluster": [],
    "index": [
      {
        "name": "create",
        "alternative/suggestion": "create_doc",
        "is_exact_replacement": false
      },
      {
        "name": "index",
        "alternative/suggestion": "index_doc",
        "is_exact_replacement": true
      }
    ]
  }
}

[tvernum]

@bizybot The whole point of creating new names is so that we can make breaking changes. We explicily don't want the new names to be exact replicas of the old names.

• index_doc should not allow the _update API. It should behave like the old create privilege. The _update API is not a standard indexing API and has different security implications. It should not be part of the standard "indexing" privilege.

• update_doc should allow the use of the _update API (which is also op_type=UPDATE). I don't think it should allow any other sort of indexing, mostly because the name implies update-only and the principle of least surprise implies that it should therefore be update-only. If admins want index+update (but not delete) then they can grant two privileges.