AMBARI-24646 'ambari-server setup-ldap' fails with AttributeError when master_key is not persisted

[ghost]

What changes were proposed in this pull request?

STR
Installed ambari-server and configured password encryption, but chose not to persist master key

[root@ctr ~]# ambari-server setup-security
Using python  /usr/bin/python
Security setup options...
===========================================================================
Choose one of the following options:
[1] Enable HTTPS for Ambari server.
[2] Encrypt passwords stored in ambari.properties file.
[3] Setup Ambari kerberos JAAS configuration.
[4] Setup truststore.
[5] Import certificate to truststore.
===========================================================================
Enter choice, (1-5): 2
Password encryption is enabled.
Do you want to reset Master Key? [y/n] (n): y
Master Key not persisted.
Enter current Master Key:
Enter new Master Key:
Re-enter master key:
Do you want to persist master key. If you choose not to persist, you need to provide the Master Key while starting the ambari server as an env variable named AMBARI_SECURITY_MASTER_KEY or the start will prompt for the master key. Persist [y/n] (y)? n
Adjusting ambari-server permissions and ownership...
Ambari Server 'setup-security' completed successfully.

Then export environment variable
export AMBARI_SECURITY_MASTER_KEY=hadoop

Thereafter ran the following:
Issue #1 - Gave AttributeError after accepting the 'Save settings' prompt, instead of asking for master key

[root@ctr ~]# ambari-server setup-ldap -v
Using python  /usr/bin/python
INFO: Loading properties from /etc/ambari-server/conf/ambari.properties
INFO: Loading properties from /etc/ambari-server/conf/ambari.properties
INFO: about to run command: ps -p 5596
INFO:
process_pid=12677
Please select the type of LDAP you want to use (AD, IPA, Generic LDAP):Generic LDAP
Primary LDAP Host (ldap.ambari.apache.org): ctr
Primary LDAP Port (389):
Secondary LDAP Host <Optional>:
Secondary LDAP Port <Optional>:
Use SSL [true/false] (false):
User object class (posixUser):
User ID attribute (uid):
Group object class (posixGroup):
Group name attribute (cn):
Group member attribute (memberUid):
Distinguished name attribute (dn):
Search Base (dc=ambari,dc=apache,dc=org): dc=apache,dc=org
Referral method [follow/ignore] (follow):
Bind anonymously [true/false] (false):
Bind DN (uid=ldapbind,cn=users,dc=ambari,dc=apache,dc=org): uid=hdfs,ou=people,ou=dev,dc=apache,dc=org
Enter Bind DN Password:
Confirm Bind DN Password:
Handling behavior for username collisions [convert/skip] for LDAP sync (skip):
Force lower-case user names [true/false]:
Results from LDAP are paginated when requested [true/false]:
====================
Review Settings
====================
Primary LDAP Host (ldap.ambari.apache.org):  ctr
Primary LDAP Port (389):  389
Use SSL [true/false] (false):  false
User object class (posixUser):  posixUser
User ID attribute (uid):  uid
Group object class (posixGroup):  posixGroup
Group name attribute (cn):  cn
Group member attribute (memberUid):  memberUid
Distinguished name attribute (dn):  dn
Search Base (dc=ambari,dc=apache,dc=org):  dc=apache,dc=org
Referral method [follow/ignore] (follow):  follow
Bind anonymously [true/false] (false):  false
Handling behavior for username collisions [convert/skip] for LDAP sync (skip):  skip
ambari.ldap.connectivity.bind_dn: uid=hdfs,ou=people,ou=dev,dc=apache,dc=org
ambari.ldap.connectivity.bind_password: *****
Save settings [y/n] (y)? y
INFO: Loading properties from /etc/ambari-server/conf/ambari.properties
Traceback (most recent call last):
File "/usr/sbin/ambari-server.py", line 1060, in <module>
mainBody()
File "/usr/sbin/ambari-server.py", line 1030, in mainBody
main(options, args, parser)
File "/usr/sbin/ambari-server.py", line 980, in main
action_obj.execute()
File "/usr/sbin/ambari-server.py", line 79, in execute
self.fn(*self.args, **self.kwargs)
File "/usr/lib/ambari-server/lib/ambari_server/setupSecurity.py", line 860, in setup_ldap
encrypted_passwd = encrypt_password(LDAP_MGR_PASSWORD_ALIAS, mgr_password, options)
File "/usr/lib/ambari-server/lib/ambari_server/serverConfiguration.py", line 858, in encrypt_password
return get_encrypted_password(alias, password, properties, options)
File "/usr/lib/ambari-server/lib/ambari_server/serverConfiguration.py", line 867, in get_encrypted_password
masterKey = get_original_master_key(properties, options)
File "/usr/lib/ambari-server/lib/ambari_server/serverConfiguration.py", line 1022, in get_original_master_key
if options is not None and options.master_key is not None and options.master_key:
AttributeError: Values instance has no attribute 'master_key'
[root@ctr ~]#

Issue #2 - Kept asking for Master key on the prompt, despite giving correct values

[root@ctr ~]# ambari-server setup
Using python  /usr/bin/python
Setup ambari-server
Checking SELinux...
WARNING: Could not run /usr/sbin/sestatus: OK
Customize user account for ambari-server daemon [y/n] (n)?
Adjusting ambari-server permissions and ownership...
Checking firewall status...
Checking JDK...
Do you want to change Oracle JDK [y/n] (n)?
Check JDK version for Ambari Server...
JDK version found: 8
Minimum JDK version is 8 for Ambari. Skipping to setup different JDK for Ambari Server.
Checking GPL software agreement...
Completing setup...
Configuring database...
Enter advanced database configuration [y/n] (n)?
Configuring database...
Enter current Master Key:
Default properties detected. Using built-in database.
Enter current Master Key:
Configuring ambari database...
Checking PostgreSQL...
Configuring local database...
Configuring PostgreSQL...
Backup for pg_hba found, reconfiguration not required
Creating schema and user...
done.
Creating tables...
done.
Enter current Master Key:
Enter current Master Key:
Enter current Master Key:

Issue #3 - Gave an incorrect master key this time and the shell kept on printing "ERROR: ERROR: Master key does not match." and kept scrolling the page

[root@ctr ~]# ambari-server setup
Using python  /usr/bin/python
Setup ambari-server
Checking SELinux...
WARNING: Could not run /usr/sbin/sestatus: OK
Customize user account for ambari-server daemon [y/n] (n)?
Adjusting ambari-server permissions and ownership...
Checking firewall status...
Checking JDK...
Do you want to change Oracle JDK [y/n] (n)?
Check JDK version for Ambari Server...
JDK version found: 8
Minimum JDK version is 8 for Ambari. Skipping to setup different JDK for Ambari Server.
Checking GPL software agreement...
Completing setup...
Configuring database...
Enter advanced database configuration [y/n] (n)?
Configuring database...
Enter current Master Key:
ERROR: ERROR: Master key does not match.
ERROR: ERROR: Master key does not match.
ERROR: ERROR: Master key does not match.
ERROR: ERROR: Master key does not match.
ERROR: ERROR: Master key does not match.
ERROR: ERROR: Master key does not match.
ERROR: ERROR: Master key does not match.
ERROR: ERROR: Master key does not match.
ERROR: ERROR: Master key does not match.
ERROR: ERROR: Master key does not match.
ERROR: ERROR: Master key does not match.
^C
Aborting ... Keyboard Interrupt.

Note/Workaround: The issues are seen when master key is not persisted as part of the initial password encryption step

How was this patch tested?

live cluster check and unit tests run
NOTE: code at trunk is significantly different, will see whether trunk requires any changes as well

[asfgit]

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/Ambari-Github-PullRequest-Builder/3973/
Test FAILed.
Test FAILured.

[asfgit]

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/Ambari-Github-PullRequest-Builder/3974/
Test FAILed.
Test FAILured.

[ghost]

retest this please

[asfgit]

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/Ambari-Github-PullRequest-Builder/3980/
Test FAILed.
Test FAILured.

[asfgit]

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/Ambari-Github-PullRequest-Builder/3981/
Test PASSed.