ZEPPELIN-1472 - Create new LdapRealm based on Apache Knox LdapRealm #1513

gss2002 · 2016-10-14T00:53:02Z

What is this PR for?

ZEPPELIN-1472 - Create new LdapRealm based on Apache Knox LdapRealm: Provides LdapRealm Functionality similar to what Apache Knox provides. This is critical as in large enterprise environments Active Directory Global Catalogs are used for lookup with samAccountName and using a DN Template is not an option as their are multiple OUs. Also searching on "userPrincipalName" is risky in an AD environment since the explicit UPN vs Implicit UPN can be different this is definitely the case with environments using Office 365. And the LDAP userPrincipalName attribute is the explicit UPN which can be defined by the directory administrator to any value and it can be duplicated. SamAccountName is unique per domain and Microsoft states best practice is to not allow duplicate samAccountName's across the forest.

What type of PR is it?

[Improvement]

Todos

None

What is the Jira issue?

https://issues.apache.org/jira/browse/ZEPPELIN-1472

How should this be tested?

Setup shiro.ini to use the following configuration:
ldapADGCRealm = org.apache.zeppelin.realm.LdapRealm
ldapADGCRealm.contextFactory.systemUsername = CN=hdpbind,OU=Svc,DC=exadc,DC=w2k,DC=example,DC=com
ldapADGCRealm.contextFactory.systemPassword = ldapPassword
ldapADGCRealm.searchBase = dc=w2k,dc=example,dc=com
ldapADGCRealm.userSearchBase = dc=w2k,dc=example,dc=com
ldapADGCRealm.groupSearchBase = dc=w2k,dc=example,dc=com
ldapADGCRealm.contextFactory.url = ldap://exampledc1.exadc.w2k.example.com:3268
ldapADGCRealm.userSearchAttributeName = sAMAccountName
ldapADGCRealm.contextFactory.authenticationMechanism = simple
ldapADGCRealm.userObjectClass = user
ldapADGCRealm.groupObjectClass = group
ldapADGCRealm.memberAttribute = member

Questions:

Does the licenses files need update? n
Is there breaking changes for older versions? n
Does this needs documentation? n

Class

gss2002 · 2016-10-14T00:53:45Z

@khalidhuseynov made the requested changes and updated documentation. Please let me know what you think. Also I will be willing to create a jira and move the other Realms.

gss2002 · 2016-10-14T14:35:33Z

Rerun build as error is not related to this patch.

gss2002 · 2016-10-14T18:22:27Z

@khalidhuseynov and @zjffdu can we look at committing this since tests have passed. If not let me know what else is needed.

Thanks

jongyoul · 2016-10-31T05:48:39Z

@gss2002 Hi, Thanks for the contribution. This PR actually has no problem itself or no breaking current behavior. So LGTM. But I'm just curious about the use case. Do you know the case to use this? May be in your company?

jongyoul · 2016-10-31T05:50:27Z

@nazgul33 I've heard your team used LDAP. Can you adopt and test this PR in your team?

nazgul33 · 2016-11-04T11:03:16Z

@jongyoul I could make it work with openLDAP server. I had no time to play around with group settings, but at least I could authenticate user by

ldapADGCRealm.userSearchAttributeName = cn
ldapADGCRealm.userSearchAttributeName = uid

and the followings are settings I had to modify when I'm using phpldapadmin to manage users.
ldapADGCRealm.contextFactory.authenticationMechanism = simple
ldapADGCRealm.userObjectClass = posixAccount
ldapADGCRealm.groupObjectClass = posixGroup
ldapADGCRealm.memberAttribute = memberUid

for me, it works.

gss2002 · 2016-11-05T13:19:42Z

@nazgul33 and @jongyoul this has been tested from a group perspective:

Here is the example:
[gsenia@hdp25sandbox ~]$ cat /etc/zeppelin/conf/shiro.ini

Sample LDAP configuration, for user Authentication, currently tested for single Realm

[main]
ldapRealm = org.apache.zeppelin.realm.LdapRealm
ldapRealm.contextFactory.systemUsername = CN=hdplookup,OU=hadoop,DC=hdpusr,DC=senia,DC=org
ldapRealm.contextFactory.systemPassword = ldapBind12
ldapRealm.searchBase = dc=hdpusr,dc=senia,dc=org
ldapRealm.userSearchBase = dc=hdpusr,dc=senia,dc=org
ldapRealm.groupSearchBase = dc=hdpusr,dc=senia,dc=org
ldapRealm.authorizationEnabled = true
ldapRealm.contextFactory.url = ldap://seniadc1.hdpusr.senia.org:3268
ldapRealm.userSearchAttributeName = sAMAccountName
ldapRealm.contextFactory.authenticationMechanism = simple
ldapRealm.groupObjectClass = group
ldapRealm.memberAttribute = member
ldapRealm.rolesByGroup = hdpeng: admin

securityManager.realms = $ldapRealm

sessionManager = org.apache.shiro.web.session.mgt.DefaultWebSessionManager

If caching of user is required then uncomment below lines

#cacheManager = org.apache.shiro.cache.MemoryConstrainedCacheManager
#securityManager.cacheManager = $cacheManager

securityManager.sessionManager = $sessionManager

86,400,000 milliseconds = 24 hour

securityManager.sessionManager.globalSessionTimeout = 86400000
shiro.loginUrl = /api/login

[roles]

'admin' role has all permissions, indicated by the wildcard '*'

admin = *

[urls]

anon means the access is anonymous.

authcBasic means Basic Auth Security

authc means Form based Auth Security

To enfore security, comment the line below and uncomment the next one

#/api/version = anon
#/* = anon
/api/interpreter/* = authc, roles[admin]
/api/configurations/* = authc, roles[admin]
/api/credential/* = authc, roles[admin]

nazgul33 · 2016-11-05T13:25:52Z

@gss2002 Thanks for the sample configuration.
I'm going to test this PR further with group mappings on Monday.

nazgul33 · 2016-11-07T08:02:42Z

@gss2002 can you explain how to map ldap groups to shiro roles?
What i'm trying to do is to configure user rights to notebooks.

shiro.ini
----
ldapADGCRealm.userObjectClass = posixAccount
ldapADGCRealm.groupObjectClass = posixGroup
ldapADGCRealm.memberAttribute = memberUid
ldapADGCRealm.rolesByGroup = \
        dataservice:admin, \
        woowabros:woowabros

securityManager.realms = $ldapADGCRealm

[roles]
admin = *
woowabros = *

Added "testuser" in group "woowabros" by setting memberUid attribute of woowabros.
if I access a notebook with security setup
owner = nazgul33
read = woowabros

an error pops out like this

Insufficient privileges to read notebook.

Allowed users or roles: [woowabros]

But the user testuser belongs to: [testuser]

gss2002 · 2016-11-07T17:53:26Z

@nazgul33 refactoring some code fix coming.. To utilize groups and users

gss2002 · 2016-11-08T18:44:46Z

This is replaced by #1614

…Provides LdapRealm Functionality similar to Apache Knox ### What is this PR for? ZEPPELIN-1472 - Create new LdapRealm based on Apache Knox LdapRealm: Provides LdapRealm Functionality similar to what Apache Knox provides. This is critical as in large enterprise environments Active Directory Global Catalogs are used for lookup with samAccountName and using a DN Template is not an option as their are multiple OUs. Also searching on "userPrincipalName" is risky in an AD environment since the explicit UPN vs Implicit UPN can be different this is definitely the case with environments using Office 365. And the LDAP userPrincipalName attribute is the explicit UPN which can be defined by the directory administrator to any value and it can be duplicated. SamAccountName is unique per domain and Microsoft states best practice is to not allow duplicate samAccountName's across the forest. In addition to the above changes I have adjusted and moved the LdapGroupRealm and ActiveDirectoryGroupRealm into the org.apache.zeppelin.realm package structure to make all Realm's consistent. The LdapRealm class also works with role to group mapping for usage within Zeppelin for notebook authorization. I have adjusted SecurityUtils to use ClassName vs realmName in determining what to use as you may have companies that decide to use their own custom realmname in shiro.ini and may not realize you cannot so using className is much safer. Example - SecurityUtils String name = realm.getClass().getName(); if (name.equals("org.apache.shiro.realm.text.IniRealm")) { allRoles = ((IniRealm) realm).getIni().get("roles"); break; } else if (name.equals("org.apache.zeppelin.realm.LdapRealm")) { allRoles = ((LdapRealm) realm).getListRoles(); break; } Example - SecurityRestApi: String name = realm.getClass().getName(); if (LOG.isDebugEnabled()) { LOG.debug("RealmClass.getName: " + name); } if (name.equals("org.apache.shiro.realm.text.IniRealm")) { usersList.addAll(getUserListObj.getUserList((IniRealm) realm)); rolesList.addAll(getUserListObj.getRolesList((IniRealm) realm)); } else if (name.equals("org.apache.zeppelin.realm.LdapGroupRealm")) { usersList.addAll(getUserListObj.getUserList((JndiLdapRealm) realm, searchText)); } else if (name.equals("org.apache.zeppelin.realm.LdapRealm")) { usersList.addAll(getUserListObj.getUserList((LdapRealm) realm, searchText)); rolesList.addAll(getUserListObj.getRolesList((LdapRealm) realm)); } else if (name.equals("org.apache.zeppelin.realm.ActiveDirectoryGroupRealm")) { usersList.addAll(getUserListObj.getUserList((ActiveDirectoryGroupRealm) realm, searchText)); } else if (name.equals("org.apache.shiro.realm.jdbc.JdbcRealm")) { usersList.addAll(getUserListObj.getUserList((JdbcRealm) realm)); } Please see feedback from previous PRs related to this JIRA: #1513 ### What type of PR is it? [Improvement] ### Todos * [ ] - Task ### What is the Jira issue? https://issues.apache.org/jira/browse/ZEPPELIN-1472 ### How should this be tested? Update shiro.ini to use configuration similar to below: # Sample LDAP configuration, for user Authentication, currently tested for single Realm [main] ldapADGCRealm = org.apache.zeppelin.realm.LdapRealm ldapADGCRealm.contextFactory.systemUsername = CN=hdplookup,OU=hadoop,DC=hdpusr,DC=senia,DC=org ldapADGCRealm.contextFactory.systemPassword = ldapBindPassword ldapADGCRealm.searchBase = dc=hdpusr,dc=senia,dc=org ldapADGCRealm.userSearchBase = dc=hdpusr,dc=senia,dc=org ldapADGCRealm.groupSearchBase = dc=hdpusr,dc=senia,dc=org ldapADGCRealm.authorizationEnabled = true ldapADGCRealm.contextFactory.url = ldap://seniadc1.hdpusr.senia.org:3268 ldapADGCRealm.userSearchAttributeName = sAMAccountName ldapADGCRealm.contextFactory.authenticationMechanism = simple ldapADGCRealm.groupObjectClass = group ldapADGCRealm.memberAttribute = member ldapADGCRealm.rolesByGroup = hdpeng: admin, \ hadoopusers: user securityManager.realms = $ldapADGCRealm sessionManager = org.apache.shiro.web.session.mgt.DefaultWebSessionManager ### If caching of user is required then uncomment below lines #cacheManager = org.apache.shiro.cache.MemoryConstrainedCacheManager #securityManager.cacheManager = $cacheManager securityManager.sessionManager = $sessionManager # 86,400,000 milliseconds = 24 hour securityManager.sessionManager.globalSessionTimeout = 86400000 shiro.loginUrl = /api/login [roles] # 'admin' role has all permissions, indicated by the wildcard '*' admin = * user = * [urls] # anon means the access is anonymous. # authcBasic means Basic Auth Security # authc means Form based Auth Security # To enfore security, comment the line below and uncomment the next one #/api/version = anon #/** = anon /api/interpreter/** = authc, roles[admin] /api/configurations/** = authc, roles[admin] /api/credential/** = authc, roles[admin] /api/login = authc /api/login/logout = authc /api/security/ticket = authc /** = authc, roles[admin, user] ### Screenshots (if appropriate) ### Questions: * Does the licenses files need update? n * Is there breaking changes for older versions? n * Does this needs documentation? y merge latest commits Author: gss2002 <[email protected]> Author: gss2002 <[email protected]> Closes #1614 from gss2002/ZEPPELIN-1472 and squashes the following commits: d6a7cea [gss2002] ZEPPELIN-1472 - LdapRealm Additions based on Knox LdapRealm and support of using roles with LdapRealms. Also adjusted to use className and not actual name of the realm in shiro.ini. As using realmName in code could cause problems for people who want to use alternate names. Also migrated the LdapGroupRealm and ActiveDirectoryRealm to org.apache.zeppelin.realm packages per a recommendation. 1702cc5 [gss2002] Merge pull request #1 from apache/master

…Provides LdapRealm Functionality similar to Apache Knox ### What is this PR for? ZEPPELIN-1472 - Create new LdapRealm based on Apache Knox LdapRealm: Provides LdapRealm Functionality similar to what Apache Knox provides. This is critical as in large enterprise environments Active Directory Global Catalogs are used for lookup with samAccountName and using a DN Template is not an option as their are multiple OUs. Also searching on "userPrincipalName" is risky in an AD environment since the explicit UPN vs Implicit UPN can be different this is definitely the case with environments using Office 365. And the LDAP userPrincipalName attribute is the explicit UPN which can be defined by the directory administrator to any value and it can be duplicated. SamAccountName is unique per domain and Microsoft states best practice is to not allow duplicate samAccountName's across the forest. In addition to the above changes I have adjusted and moved the LdapGroupRealm and ActiveDirectoryGroupRealm into the org.apache.zeppelin.realm package structure to make all Realm's consistent. The LdapRealm class also works with role to group mapping for usage within Zeppelin for notebook authorization. I have adjusted SecurityUtils to use ClassName vs realmName in determining what to use as you may have companies that decide to use their own custom realmname in shiro.ini and may not realize you cannot so using className is much safer. Example - SecurityUtils String name = realm.getClass().getName(); if (name.equals("org.apache.shiro.realm.text.IniRealm")) { allRoles = ((IniRealm) realm).getIni().get("roles"); break; } else if (name.equals("org.apache.zeppelin.realm.LdapRealm")) { allRoles = ((LdapRealm) realm).getListRoles(); break; } Example - SecurityRestApi: String name = realm.getClass().getName(); if (LOG.isDebugEnabled()) { LOG.debug("RealmClass.getName: " + name); } if (name.equals("org.apache.shiro.realm.text.IniRealm")) { usersList.addAll(getUserListObj.getUserList((IniRealm) realm)); rolesList.addAll(getUserListObj.getRolesList((IniRealm) realm)); } else if (name.equals("org.apache.zeppelin.realm.LdapGroupRealm")) { usersList.addAll(getUserListObj.getUserList((JndiLdapRealm) realm, searchText)); } else if (name.equals("org.apache.zeppelin.realm.LdapRealm")) { usersList.addAll(getUserListObj.getUserList((LdapRealm) realm, searchText)); rolesList.addAll(getUserListObj.getRolesList((LdapRealm) realm)); } else if (name.equals("org.apache.zeppelin.realm.ActiveDirectoryGroupRealm")) { usersList.addAll(getUserListObj.getUserList((ActiveDirectoryGroupRealm) realm, searchText)); } else if (name.equals("org.apache.shiro.realm.jdbc.JdbcRealm")) { usersList.addAll(getUserListObj.getUserList((JdbcRealm) realm)); } Please see feedback from previous PRs related to this JIRA: apache#1513 ### What type of PR is it? [Improvement] ### Todos * [ ] - Task ### What is the Jira issue? https://issues.apache.org/jira/browse/ZEPPELIN-1472 ### How should this be tested? Update shiro.ini to use configuration similar to below: # Sample LDAP configuration, for user Authentication, currently tested for single Realm [main] ldapADGCRealm = org.apache.zeppelin.realm.LdapRealm ldapADGCRealm.contextFactory.systemUsername = CN=hdplookup,OU=hadoop,DC=hdpusr,DC=senia,DC=org ldapADGCRealm.contextFactory.systemPassword = ldapBindPassword ldapADGCRealm.searchBase = dc=hdpusr,dc=senia,dc=org ldapADGCRealm.userSearchBase = dc=hdpusr,dc=senia,dc=org ldapADGCRealm.groupSearchBase = dc=hdpusr,dc=senia,dc=org ldapADGCRealm.authorizationEnabled = true ldapADGCRealm.contextFactory.url = ldap://seniadc1.hdpusr.senia.org:3268 ldapADGCRealm.userSearchAttributeName = sAMAccountName ldapADGCRealm.contextFactory.authenticationMechanism = simple ldapADGCRealm.groupObjectClass = group ldapADGCRealm.memberAttribute = member ldapADGCRealm.rolesByGroup = hdpeng: admin, \ hadoopusers: user securityManager.realms = $ldapADGCRealm sessionManager = org.apache.shiro.web.session.mgt.DefaultWebSessionManager ### If caching of user is required then uncomment below lines #cacheManager = org.apache.shiro.cache.MemoryConstrainedCacheManager #securityManager.cacheManager = $cacheManager securityManager.sessionManager = $sessionManager # 86,400,000 milliseconds = 24 hour securityManager.sessionManager.globalSessionTimeout = 86400000 shiro.loginUrl = /api/login [roles] # 'admin' role has all permissions, indicated by the wildcard '*' admin = * user = * [urls] # anon means the access is anonymous. # authcBasic means Basic Auth Security # authc means Form based Auth Security # To enfore security, comment the line below and uncomment the next one #/api/version = anon #/** = anon /api/interpreter/** = authc, roles[admin] /api/configurations/** = authc, roles[admin] /api/credential/** = authc, roles[admin] /api/login = authc /api/login/logout = authc /api/security/ticket = authc /** = authc, roles[admin, user] ### Screenshots (if appropriate) ### Questions: * Does the licenses files need update? n * Is there breaking changes for older versions? n * Does this needs documentation? y merge latest commits Author: gss2002 <[email protected]> Author: gss2002 <[email protected]> Closes apache#1614 from gss2002/ZEPPELIN-1472 and squashes the following commits: d6a7cea [gss2002] ZEPPELIN-1472 - LdapRealm Additions based on Knox LdapRealm and support of using roles with LdapRealms. Also adjusted to use className and not actual name of the realm in shiro.ini. As using realmName in code could cause problems for people who want to use alternate names. Also migrated the LdapGroupRealm and ActiveDirectoryRealm to org.apache.zeppelin.realm packages per a recommendation. 1702cc5 [gss2002] Merge pull request #1 from apache/master

gss2002 added 2 commits October 13, 2016 20:36

ZEPPELIN-1472 - Create new LdapRealm based on Apache Knox LdapRealm

3493875

Class

Merge remote-tracking branch 'upstream/master' into ZEPPELIN-1472

8991d64

gss2002 closed this Oct 14, 2016

gss2002 reopened this Oct 14, 2016

ZEPPELIN-1472 - REFACTOR

28bab74

gss2002 closed this Nov 8, 2016

gss2002 mentioned this pull request Nov 8, 2016

ZEPPELIN-1472 - Create new LdapRealm based on Apache Knox LdapRealm: Provides LdapRealm Functionality similar to Apache Knox #1614

Closed

1 task

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

ZEPPELIN-1472 - Create new LdapRealm based on Apache Knox LdapRealm #1513

ZEPPELIN-1472 - Create new LdapRealm based on Apache Knox LdapRealm #1513

Uh oh!

gss2002 commented Oct 14, 2016

Uh oh!

gss2002 commented Oct 14, 2016

Uh oh!

gss2002 commented Oct 14, 2016

Uh oh!

gss2002 commented Oct 14, 2016

Uh oh!

jongyoul commented Oct 31, 2016

Uh oh!

jongyoul commented Oct 31, 2016

Uh oh!

nazgul33 commented Nov 4, 2016

Uh oh!

gss2002 commented Nov 5, 2016

Uh oh!

nazgul33 commented Nov 5, 2016

Uh oh!

nazgul33 commented Nov 7, 2016

Uh oh!

gss2002 commented Nov 7, 2016

Uh oh!

gss2002 commented Nov 8, 2016

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

ZEPPELIN-1472 - Create new LdapRealm based on Apache Knox LdapRealm #1513

ZEPPELIN-1472 - Create new LdapRealm based on Apache Knox LdapRealm #1513

Uh oh!

Conversation

gss2002 commented Oct 14, 2016

What is this PR for?

What type of PR is it?

Todos

What is the Jira issue?

How should this be tested?

Questions:

Uh oh!

gss2002 commented Oct 14, 2016

Uh oh!

gss2002 commented Oct 14, 2016

Uh oh!

gss2002 commented Oct 14, 2016

Uh oh!

jongyoul commented Oct 31, 2016

Uh oh!

jongyoul commented Oct 31, 2016

Uh oh!

nazgul33 commented Nov 4, 2016

Uh oh!

gss2002 commented Nov 5, 2016

Sample LDAP configuration, for user Authentication, currently tested for single Realm

If caching of user is required then uncomment below lines

86,400,000 milliseconds = 24 hour

'admin' role has all permissions, indicated by the wildcard '*'

anon means the access is anonymous.

authcBasic means Basic Auth Security

authc means Form based Auth Security

To enfore security, comment the line below and uncomment the next one

Uh oh!

nazgul33 commented Nov 5, 2016

Uh oh!

nazgul33 commented Nov 7, 2016

Uh oh!

gss2002 commented Nov 7, 2016

Uh oh!

gss2002 commented Nov 8, 2016

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants