Support Hive metastore impersonation

[ebyhr]

Fix #43

[electrum]

Would it be sufficient to call set_ugi() after connecting to the metastore? The delegation token support says it is there for map/reduce tasks, where the tasks would run on a different machine (and thus wouldn't have access to the original credentials).

[ebyhr]

Thank you for the advise. I think it's sufficient too and replaced the delegation token logic with set_ugi().

[AnupGS]

Hi, impersonation support is missed out for get table operation. is there any plan to add support for that as well?

[electrum]

@AnupGS What is the use case for impersonation for read calls? We could certainly add those, but I'm not aware of any reason to do so. When using SQL standard authorization, Presto performs all the security checks using the security information provided by the metastore.

[findepi]

@electrum this is required because HMS may impose additional checks on get_table call.
It may be, for example, that Presto user is allowed to list tables only, but it's not allowed to read from any of them.
With storage-based auth enabled in HMS, Presto user won't be able to do get_table on any table.

@ebyhr @electrum I will share with you exact impersonation rules we have implemented.

@ebyhr thanks for working on this! And apologies for not opening our impersonation impl earlier, as I promised.
It was dependent on several other changes we made.

[electrum]

A few minor comments, otherwise looks good. Apologies for the delay in reviewing. I'd like to merge this for the next release.

@findepi or @kokosing do you want to look at the product tests? They seem fine to me, but I'm not an expert on them.

[electrum]

@findepi Thanks for explaining. @ebyhr It sounds like we need impersonation for the read calls. If you have time to update this PR to include those, great, otherwise we can merge this as-is and follow up with those later.

[ebyhr]

@findepi @electrum Thanks for your reviews and comments. I'm just adding impersonation for get_table and the dependent methods. I'll push it later.

[AnupGS]

Hi @electrum, @findepi has explained it perfectly. presto user is unable to get table if storage based auth is enabled.

[findepi]

A couple of comments. Well done!

[findepi]

According to https://github.com/prestosql/presto/blob/c808deea818325b300cb3d185ef1af0f7a056cd8/presto-hive/src/main/java/io/prestosql/plugin/hive/metastore/thrift/ThriftHiveMetastore.java#L144

this.impersonationEnabled = thriftConfig.isImpersonationEnabled() && hiveMetastoreAuthentication instanceof KerberosHiveMetastoreAuthentication;

the impersonation is supported only when kerberos is enabled
i think with setUGI, we should support impersonation with and without kerberos

-- but this also signals we don't have a direct test for an impersonation, something that would fail when impersonation does not work

[findepi]

I think this should be visible if you add authorization here. (It should be included here anyway, it contains TestRoles, this is important to run here)

[findepi]

When applying changes, please separate them from rebase (or delay rebase), so that the changes themselves can be viewed on GH ui.

[electrum]

Wow, adding impersonation for get calls was more complicated than I thought. Thanks for doing that.

[findepi]

I think this should be visible if you add authorization here. (It should be included here anyway, it contains TestRoles, this is important to run here)

[findepi]

Any particular reason to include cli and hive_file_header tests?
(@kokosing do you know why do we run cli tests so often?)

[ebyhr]

No particular reason to include those tests. I followed existing hdfs-impersonation tests.

[ebyhr]

Let me push updated code once. Below comments are not yet applied.
#1441 (comment) & #1441 (comment)

[ebyhr]

Pushed so that we can share the current status.

[electrum]

We can remove this field if we add isImpersonationEnabled() to HiveMetastore. Then updateIdentity() can simply call delegate.isImpersonationEnabled().

[ebyhr]

Should we always return false in FileHiveMetastore and throw an exception in GlueHiveMetastore? At least, we can't inject ThriftHiveMetastoreConfig into FileHiveMetastore. We can do it in GlueHiveMetastore.

[findepi]

Yes, always false. we don't support impersonation there

[electrum]

I had a few comments, mostly around the usage of ThriftHiveMetastoreConfig, which is actually the cause of the Travis test failures. This should be easy to fix. After that, this PR should be ready to go. Thanks for all your hard work on this. It's much more involved than I imagined.

[findepi]

Minor comments.
Thanks @ebyhr

[findepi]

Simply:

Suggested change

	this(new ConnectorIdentity("dummy_identity", Optional.empty(), Optional.empty()));
	this.username = "dummy_identity";

[findepi]

Still, I'd prefer the username be Optional<String> in this class.
There is only one place where getUsername() is called, so it's not a big deal and improves clarify.

[findepi]

HiveIdentity.none() is going to be used as a cache key many times at the same time, so using a singleton would be nice.

[electrum]

One more thing: we can remove AVRO_SCHEMA_URL from TestGroups in product-tests

[BlueStalker]

Actually I wonder how this is working, the set_ugi implementation in hive metastore server-side is like adding the user information in the session, and there is no additional impersonation or authentication implemented. In fact, the earlier version of implementation (delegation token based) looks pretty promising and follow what hive did internally. Have you guys tested this in the real secure hive metastore environment, for example, if you create a table, is it owned by the authenticated user (mostly presto) or the real impersonation user (end-user)? @ebyhr @findepi