[RFC] Refined index authorization (replacement for do_not_fail_on_forbidden)

[nibix]

Background

The do_not_fail_on_forbidden setting switches globally between two different modes how OpenSearch reacts to cases when users have no privileges for indices.

By default, OpenSearch will resolve any patterns specified in action requests against all indices on the cluster. If then the current user does not have access privileges for any of these indices, OpenSearch will yield a 403 error.

If do_not_fail_on_forbidden is enabled, the behavior is changed: The resolution is limited to indices to which the user has access. Thus, the operation will succeed.

A very common example for the difference is a _search request to all indices (GET /_search, or GET /*/_search). For users which have not complete read-access to all indices on the cluster, this request will always fail with a 403 error if do_not_fail_on_forbidden disabled. If do_not_fail_on_forbidden is enabled, these users will get the search result limited to the indices they have access to.

Historically, the do_not_fail_on_forbidden mode was implemented to make working in Dashboards more smooth. However, this is a very minimal implementation. This leads to confusion regarding the behavior of operations which are not supported (cf #1815, #2257, #2113).

Additionally, the OpenSearch documentation is quite sparse on the option. All documentation regarding the option is located in sections related to multi-tenancy (cf https://opensearch.org/docs/latest/security/multi-tenancy/multi-tenancy-config/). It just states:

If true, the security plugin removes any content that a user is not allowed to see from search results. If false, the plugin returns a security exception. Default is false.

This is not really helpful because:

• The documentation does not describe use-cases for this option and does not explain scenarios in which it should be enabled and in which it should be disabled.
• Even though it is only mentioned in the multi-tenancy chapters of the documentation, the setting might be also useful for users without multi-tenancy enabled, or even for users which do not utilize Dashboards.
• There are nuances in the behavior which are left unclear. This pertains for example to behavior when using aggregations or when referring to concrete indices (like GET /_search/index_i_have_no_privileges_for).

To make matters more complicated, there is also the do_not_fail_on_forbidden_empty setting which is undocumented. If it is false, _search operations will fail with a 403 error if the given index expression matches no indices the user has access to. If it is true, _search operations will yield an empty result set if the given index expression matches no indices the user has access to.

One reason why no recommendations are given how these settings shall be used, might be that it is actually quite difficult to make good recommendation. The reason is that there are competing pro/con arguments for these settings:

• Pro do_not_fail_on_forbidden:
  • Enables the use of GET /_search, GET /_cat/indices and similar operations for all users.
  • A user does not have to think about indices they do not have access to.
  • The presence of the security plugin gets transparent to applications like Dashboards which expect to be able to issue requests to * indices.
• Con do_not_fail_on_forbidden:
  • Having a fail-fast mechanism like do_not_fail_on_forbidden: false can be helpful to verify that a user actually has all permissions to indices they are supposed to. The operation of do_not_fail_on_forbidden: true might conceal issues in the privileges configuration.
  • If do_not_fail_on_forbidden and do_not_fail_on_forbidden_empty is true, a search request on any concrete index (like GET /_search/index_i_have_no_privileges_for) will result in an empty result set, if the user does not have privileges for that index. That might be counter-intuitive.
  • If do_not_fail_on_forbidden_empty is false, a similar issue still exists if a user performs a search containing an existing index and a non-existing index like GET /existing_index,idexx_with_typo/_search. The index idexx_with_typo will be silently dropped without giving the user a notice of the typo.

Despite these pros and cons, a quick research over the OpenSearch forums provides the impression that it is usually recommended to activate the setting:

• https://forum.opensearch.org/t/cant-give-access-to-specific-indices-to-a-user/7932
• https://forum.opensearch.org/t/index-level-permission-of-security-plugin-is-not-working-with-list-of-indices-in-role/8640/8
• https://forum.opensearch.org/t/no-permissions-on-indices-data-read-search-template-when-the-permission-is-attached-in-the-role/9991
• https://forum.opensearch.org/t/how-to-get-a-list-of-indices-for-which-i-have-been-granted-permission/16692

Path to a improved solution

To improve the situation, one should seek for a solution which fulfills the following requirements:

• The solution has well-defined and well-documented semantics
• The behavior matches user expectations
• Applications like Dashboards work well with out-of-the-box default settings
• User errors like typos shall not be concealed

Additional desirable properties are:

• Reduced number of configuration options. Any configuration option has the following disadvantage:
  • It increases the need for documentation.
  • It increases the need for consideration by the user and thus the chance that a user needs support.
  • It increases the amount of code.
  • It increases the amount of tests and thus development round trip time.
• A generic implementation which does not need significant special cases

Looking at the pro/con list above, a possible synthesis for these could be moving the choice for the desired mode from a global option to the request level. Thus, a user could choose for any request which behavior seems to be the most fitting.

Existing OpenSearch options

Coincidentally, OpenSearch already provides request-level index options which are supported for most operations: https://github.com/opensearch-project/OpenSearch/blob/main/server/src/main/java/org/opensearch/action/support/IndicesOptions.java

One of these options is ignore_unavailable which is documented as follows:

Specifies whether to include missing or closed indexes in the response. Default is false.

The behavior which is controlled by this option seems to be at least similar to the behavior of do_not_fail_on_forbidden.

Another one of these options is allow_no_indices:

Whether to ignore wildcards that don’t match any indexes. Default is true.

The behavior which is controlled by this option seems to be at least similar to the behavior of do_not_fail_on_forbidden_empty.

One might consider whether it is possible to re-use these options for a replacement for do_not_fail_on_forbidden.

Proposed solution

I propose that the security plugin re-uses the existing index options ignore_unavailable and allow_no_indices to provide a request-level replacement for controlling the behavior in case a user requests indices they do not have privileges for. (Thus, the behavior control which is currently supplied by the global configuration options do_not_fail_on_forbidden and do_not_fail_on_forbidden_empty).

The options do_not_fail_on_forbidden and do_not_fail_on_forbidden_empty shall be put on a deprecation path and eventually removed.

The security plugin should follow the current semantics of the OpenSearch index resolution algorithm which implements ignore_unavailable and allow_no_indices (cf https://github.com/opensearch-project/OpenSearch/blob/main/server/src/main/java/org/opensearch/cluster/metadata/IndexNameExpressionResolver.java )

Specification of behavior

This specification follows the behavior currently implemented in the class IndexNameExpressionResolver.java.

The behavior depends on two factors:

• The index expression in the considered action request specified concrete index names or uses patterns
• The index options of the considered action request

The specified behavior applies for all actions which operate on index name expressions. These actions are:

• search
• any action wrapping search such as msearch, search templates, etc.
• delete by query
• update by query
• count
• diverse cat action
• index level actions except create index

The specified behavior explicitly not applies for:

• get
• mget
• delete document by id
• update document by id

Concrete index name, ignore_unavailable=false, allow_no_indices=true

An index request references one or more indices by their concrete names. The index option ignore_unavailable has its default value false. The index option allow_no_indices has its default value true.

Examples: GET /index1/_search, GET /index1,index2/_search

• If the user has privileges for the specified indices, the operation shall succeed
• If the user does not have privileges for any of the specified indices, the operation shall fail with an error 403 or 404 (see open questions below).

Concrete index name, ignore_unavailable=true, allow_no_indices=true

An index request references one or more indices by their concrete names. The index option ignore_unavailable has the value true. The index option allow_no_indices has its default value true.

Examples: GET /index1/_search?ignore_unavailable=true, GET /index1,index2/_search?ignore_unavailable=true

• If the user has privileges for the specified indices, the operation shall succeed
• If the user has privileges only for a subset of the specified indices, the operation shall proceed limited to the indices with privileges
• If the user does not have privileges for any of the specified indices, the operation shall yield an empty result set.

Concrete index name, ignore_unavailable=true, allow_no_indices=false

An index request references one or more indices by their concrete names. The index option ignore_unavailable has the value true. The index option allow_no_indices has the value false.

Examples: GET /index1/_search?ignore_unavailable=true&allow_no_indices=false, GET /index1,index2/_search?ignore_unavailable=true&allow_no_indices=false

• If the user has privileges for the specified indices, the operation shall succeed
• If the user has privileges only for a subset of the specified indices, the operation shall proceed limited to the indices with privileges
• If the user does not have privileges for any of the specified indices, the operation shall fail with an error 403 or 404 (see open questions below).

Index pattern, allow_no_indices=true

An index request references indices by a pattern. The index option allow_no_indices has its default value true. The index option ignore_unavailable is disregarded when index patterns are used.

Examples: GET /index*/_search

• If the user has privileges for all indices matched by the specified patterns, the operation shall succeed
• If the user has privileges only for a subset of the indices matched by the specified patterns, the operation shall proceed limited to the indices with privileges.
• If the user does not have privileges for any of indices matched by the specified patterns, the operation shall yield an empty result set.

Index pattern, allow_no_indices=false

An index request references indices by a pattern. The index option allow_no_indices has the value false. The index option ignore_unavailable is disregarded when index patterns are used.

Examples: GET /index*/_search?allow_no_indices=false

• If the user has privileges for all indices matched by the specified patterns, the operation shall succeed
• If the user has privileges only for a subset of the indices matched by the specified patterns, the operation shall proceed limited to the indices with privileges.
• If the user does not have privileges for any of the specified indices, the operation shall fail with an error 403 or 404 (see open questions below).

Mixture of concrete index names and patterns

An index request contains both concrete index names and patterns.

Examples: GET /one_index,other_indices*/_search

In case of mixed concrete index names and patterns, the aforementioned rules are combined. Failures caused by ignore_unavailable=false on concrete index names take precedence. This means:

• If the user has privileges for all indices matched by the specified expression, the operation shall succeed
• If ignore_unavailable is false, and there are concrete index names without privileges, the request shall fail with 403 or 404.
• Otherwise, if the user has privileges for a subset of the matched indices, the operation shall proceed limited to the indices with privileges.
• If allow_no_indices is false and no indices with privileges are present, the request shall fail with 403 or 404.
• Otherwise, an empty result set shall be returned

Transition path

This RFC specifies a breaking change. The specified behavior is different to either of the currently configurable behaviors of the security plugin. Is such a breaking change feasible? How could a transition path look like?

Open questions

This sections lists questions which might require further discussion.

• Shall operations which fail due to missing index privileges fail with an error 403 or 404? More precisely, shall the security plugin expose the fact to the user that privileges are missing - or shall it pretend to the user that the index does not exist?
• Are there any other factors to be considered?

Complications

This section lists cases where special consideration might be necessary when exploring the details of the implementation

• Some actions implement custom index resolution algorithms. Examples for such actions are actions operating on data streams. The security plugin might also require special considerations for these.

[peternied]

Thanks for writing this up; however, I'm not sure what the user problem is that DNFOF and these new options address, could you go into a little detail for how and why these options would be used?

The use case that comes to mind for me is "As a user, I have a rough idea the data in the OpenSearch cluster that I want to search, I want to search over as much of the data as possible". While this case has some utility to discover what access is available on an ad-hoc basis, it seems like repeated searches would be against (more?) specific indices.

For this 'discovery' based scenario, it seems like having ignore_unavailable=true and allow_no_indices=true would be the primary usage - are there other scenarios that would be missed with this simpler approach?

[nibix]

@peternied

Thanks for writing this up; however, I'm not sure what the user problem is that DNFOF and these new options address, could you go into a little detail for how and why these options would be used?
The use case that comes to mind for me is "As a user, I have a rough idea the data in the OpenSearch cluster that I want to search, I want to search over as much of the data as possible".

Roughly, that's correct. But I would also include "I want to be able to use all available functionality to discover data".

As an illustration: I have set up a demo cluster with indices a1, a2 and b1. A test user has a role which only gives access to the indices a*:

role_index_a:
  index_permissions:
    - index_patterns:
        - 'a*'
      allowed_actions:
        - '*'

With DNFOF disabled, this happens for GET /_cat/indices and GET /_search :

$ curl -k -u user_a:user_a https://localhost:9200/_cat/indices?pretty
{
  "error" : {
    "root_cause" : [
      {
        "type" : "security_exception",
        "reason" : "no permissions for [indices:monitor/settings/get] and User [name=user_a, backend_roles=[], requestedTenant=null]"
      }
    ],
    "type" : "security_exception",
    "reason" : "no permissions for [indices:monitor/settings/get] and User [name=user_a, backend_roles=[], requestedTenant=null]"
  },
  "status" : 403
}

$ curl -k -u user_a:user_a https://localhost:9200/_search?pretty
{
  "error" : {
    "root_cause" : [
      {
        "type" : "security_exception",
        "reason" : "no permissions for [indices:data/read/search] and User [name=user_a, backend_roles=[], requestedTenant=null]"
      }
    ],
    "type" : "security_exception",
    "reason" : "no permissions for [indices:data/read/search] and User [name=user_a, backend_roles=[], requestedTenant=null]"
  },
  "status" : 403
}

With DNFOF enabled, we get these results:

$ curl -k -u user_a:user_a https://localhost:9200/_cat/indices?pretty
green open a1 E8SOIMogRpq9G6q6UwvZPw 1 1 1 0   8kb   4kb
green open a2 hPNXvCzFSX6lJI5TjNbj5w 1 1 1 0 7.9kb 3.9kb

$ curl -k -u user_a:user_a https://localhost:9200/_search?pretty
{
  "took" : 22,
  "timed_out" : false,
  "_shards" : {
    "total" : 2,
    "successful" : 2,
    "skipped" : 0,
    "failed" : 0
  },
  "hits" : {
    "total" : {
      "value" : 2,
      "relation" : "eq"
    },
    "max_score" : 1.0,
    "hits" : [
      {
        "_index" : "a1",
        "_id" : "a1_1",
        "_score" : 1.0,
        "_source" : {
          "agent" : {
            "name" : "test.example.com"
          }
        }
      },
      {
        "_index" : "a2",
        "_id" : "a2_1",
        "_score" : 1.0,
        "_source" : {
          "agent" : {
            "name" : "test2.example.com"
          }
        }
      }
    ]
  }
}

Looking at Dashboards: If I log in Dashboards as the user user_a with DNFOF disabled, I cannot create index patterns if I do not know patterns which match the indices I have access to. So, just writing * does not work:

I need to write a* to be able to create an index pattern:

With DNFOF enabled, this is not necessary:

Another scenario that illustrates a further problem with DNFOF disabled:

• Assume that the cluster initially only has the indices a1 and a2.
• Now, the user user_a can use GET /_search and GET /_cat/indices without issues even though DNFOF is disabled
• Afterwards, an admin creates an index b1
• Suddenly, the actions GET /_search and GET /_cat/indices no longer work for the user user_a if DNFOF is disabled.

[nibix]

@peternied

For this 'discovery' based scenario, it seems like having ignore_unavailable=true and allow_no_indices=true would be the primary usage - are there other scenarios that would be missed with this simpler approach?

I am not sure if I understand you correctly here. Do you mean with "simpler approach" to always follow a behavior that would correspond to the behavior I specified above when ignore_unavailable=true and allow_no_indices=true would be given?

That's possibly a matter of taste, but in the past few years the only relevant argument I was hearing against using "DNFOF" was "I do not want to have indices in my requests silently dropped". This would happen with ignore_unavailable=true and allow_no_indices=true, though. It is certainly possible to argue against and in favor of that argument, but I guess it is an argument that should be taken into account. Thus, it seems it would make sense to give users the choice.

[peternied]

Looking at Dashboards: If I log in Dashboards as the user user_a with DNFOF disabled, I cannot create index patterns if I do not know patterns which match the indices I have access to. So, just writing * does not work:
I need to write a* to be able to create an index pattern:

Discovering data from the FE is important, but its conflating API design on the backend service with UX in Dashboards FE. While they are linked - index patterns in dashboards are persisted and reused. This is an important distinction to manage predictability and cost of query calculation.

Improved Discovery View

Maybe a better alternative would be to alter the discover view so one could dynamically edit the index pattern used in those queries or select a saved index pattern, defaulting to using * with DNFOF enabled. This is a good case to use a query with the DNFOF flag. However, when you are construction a persistent index pattern for use with view/dashboard not using DNFOF is important for data consistency.

If we alter this UX I think it could be very useful to introduce users to the cluster data more quickly

FYI @kavilla @joshuarrrr @shanilpa

[peternied]

For this 'discovery' based scenario, it seems like having ignore_unavailable=true and allow_no_indices=true would be the primary usage - are there other scenarios that would be missed with this simpler approach?

I am not sure if I understand you correctly here. Do you mean with "simpler approach" to always follow a behavior that would correspond to the behavior I specified above when ignore_unavailable=true and allow_no_indices=true would be given?

@nibix Yes - my consideration with new query option(s) is how obvious they are to the user. The current DNFOF abstraction is leaky with allow_no_indices, it requires knowledge of the underlying data to know how to interpret the response, if we were to redesign how discovery type flows work in OpenSearch I think we could reduce the options to better serve customer use cases.

I think OpenSearch query authors have great use cases for DNFOF and I think non-authors scenarios ¹ should not have minimal unpredictability of DNFOF.

Footnotes

1. Scenarios such as a websites search box's text input being sent to OpenSearch ↩

[nibix]

Discovering data from the FE is important, but its conflating API design on the backend service with UX in Dashboards FE.

I have provided the screenshots from Dashboards in the context of explaining use cases for DNFOF resp. any replacement. And the original, historic use case for DNFOF is exactly this. With DNFOF disabled, there is no API or interface that Dashboards could use to determine the indices which are available for the privileges of a particular user. Of course, API design and UX design are different things. But, at least in the current state, the way Dashboards is using the APIs matches more the DNFOF enabled case.

[nibix]

my consideration with new query option(s) is how obvious they are to the user.

One important point of this proposal: The index options are not new, but they already exist since quite a while. While the options are certainly not the most widely known options, client software already uses them when a certain behavior is needed. The closest example is Dashboards:

https://github.com/opensearch-project/OpenSearch-Dashboards/blob/6710ec3eb1417a5cdf65e01074fc658ea27d7e2a/src/plugins/index_pattern_management/public/components/create_index_pattern_wizard/lib/get_indices.ts#L288

The idea of the proposal is to use the fact that the existing semantics of these options match quite well the index authorization conditions introduced by the security plugin. Thus, client software which uses these options would benefit from the expanded semantics.

[peternied]

One important point of this proposal: The index options are not new, but they already exist since quite a while.

These existing options are tied to how all queries on a cluster behave, not individual queries. While they might exist in the open-source community, in managed services like Amazon OpenSearch these options are not accessible because the security configuration is not accessible.

With this proposal we have an opportunity to define new behavior that better fits our user scenario. I am not convinced we should reuse those existing options. I would prefer we add functionality iteratively. From a technical perspective I'm not sure it changes much in the implementation, what do you think about breaking out customizability of the behavior separately?

[peternied]

@nibix I think it might be useful to rename this area as a new feature instead of a modification to index authorization (sorry to once again show my bias around DNFOF's usage).

What would you think about calling this 'search discovery mode' or something along those lines? How well does this align with what you are thinking about this proposal?

[nibix]

With this proposal we have an opportunity to define new behavior that better fits our user scenario. I am not convinced we should reuse those existing options. I would prefer we add functionality iteratively. From a technical perspective I'm not sure it changes much in the implementation, what do you think about breaking out customizability of the behavior separately?

I would also agree that there won't be major differences in the implementation. Yet, I believe we should focus on the POV of the user/client in this stage of the RFC. Implementation details should be secondary.

Regarding the question whether it would be better to re-use the existing index options or adding new options:

There are pros and cons for both ways. I believe one main point of this RFC is to find out the ways where the pros outweigh the cons.

One argument in favor of using the ignore_unavailable semantics:

• This is already a generalized option, which covers several conditions: We are covering both non-existing indices and closed indices. Thus, it seems to be straight forward to add another condition under that umbrella.

One argument in favor of adding a separate way, which is independent of the ignore_unavailable and allow_no_indices option:

• Greater control over the behavior

One argument against it:

• More options the user has to think about and use. Thus, more documentation to be written, maintained and read and greater cognitive load for the user.

Coming back to the pro-argument "Greater control over the behavior". IMHO, we need to ask the question whether there are cases where this greater control is actually required. Is it necessary for a user to distinguish between "an index does not exist" and "an index does exist but I do not have privileges" when doing a search? From a security perspective, one could even ask the question whether a user should be able to distinguish between these cases. So far, I have no really good ideas for supporting the "Greater control over the behavior" argument. If it should be really not necessary to have this control, I would argue that the software also should not provide such a control.

[nibix]

What would you think about calling this 'search discovery mode' or something along those lines? How well does this align with what you are thinking about this proposal?

Just for clarification for me: Would you argue to rename the whole RFC or just a specific question? At least, my idea behind this RFC is not only about search discovery, but generally about making the behavior of the software more straight forward, more intuitive and simpler to understand for the user.

[stephen-crawford]

[Triage] Hi @nibix and @peternied, I am going to mark this issue triaged since as an RFC it seems like the path forward is to get more engagement and then go from there.