3.2.0.4

Release notes - hops - Version 3.2.0.4

Bug

HOPS-1638 project daily cost is not updated

HOPS-1636 Token revocation can set app in unrecoverable state

HOPS-1634 Fix uri checks for multi-namenode deployments using consul

Improvement

HOPS-1637 NPE in getBlockLocation if all storages are dead

HOPS-1635 Use rondb clusterj lib and make using clusterJ caching configurable

HOPS-1633 Fix missing locks for under construction replica

3.2.0.3

Release notes - hops - Version 3.2.0.3

Bug

HOPS-1638 project daily cost is not updated

HOPS-1636 Token revocation can set app in unrecoverable state

HOPS-1634 Fix uri checks for multi-namenode deployments using consul

HOPS-1628 Use TLS for Yarn web UI

HOPS-1624 TestLeaseManager fail on master since February 2nd

Improvement

HOPS-1637 NPE in getBlockLocation if all storages are dead

HOPS-1633 Fix missing locks for under construction replica

HOPS-1632 reduce quota manager batch size

HOPS-1630 Reduce subtree batch lock size

HOPS-1629 Fix copyFromLocal to create multipel clients when using multiple threads

HOPS-1627 Distcp to add no-rename copy option

3.2.0.6-RC0

Release notes - hops - Version 3.2.0.6

Bug

HOPS-1654 Reconnect ClusterJ when NDB restarts

Improvement

HOPS-1655 Fix incorrect calculation of quota updates

HOPS-1653 Limit the number of blocks in a file and the number of children in a directory

HOPS-1652 Provenance adding too many log entries in a single transaction

HOPS-1650 Adding lots of log entries in the rename transaction may kill NDB

HOPS-1648 Add a small delay in DN before retrying to connect to NN

Task

HOPS-1649 Add Livy option to CertificateLocalizationService

3.2.0.2

Release notes - hops - Version 3.2.0.2

Bug

HOPS-1638 project daily cost is not updated

HOPS-1636 Token revocation can set app in unrecoverable state

HOPS-1628 Use TLS for Yarn web UI

HOPS-1624 TestLeaseManager fail on master since February 2nd

HOPS-1620 [HDFS-6434] Default permission for creating file should be 644 for WebHdfs/HttpFS

HOPS-1619 [HDFS-10650] DFSClient#mkdirs and DFSClient#primitiveMkdir should use default directory permission

Improvement

HOPS-1631 Fix retying incrementail block reports

HOPS-1629 Fix copyFromLocal to create multipel clients when using multiple threads

HOPS-1623 unload schema when format --force called

HOPS-1622 Increase application certificate key size

3.2.0.4-RC0

Release notes - hops - Version 3.2.0.4

Bug

HOPS-1638 project daily cost is not updated

HOPS-1636 Token revocation can set app in unrecoverable state

HOPS-1634 Fix uri checks for multi-namenode deployments using consul

Improvement

HOPS-1637 NPE in getBlockLocation if all storages are dead

HOPS-1635 Use rondb clusterj lib and make using clusterJ caching configurable

HOPS-1633 Fix missing locks for under construction replica

3.2.0.1

Release notes - hops - Version 3.2.0.1

Bug

HOPS-1616 TestGetFileChecksum keep failing

HOPS-1614 Fix quota calculation

HOPS-1595 Transactions should not use object comming from other transactions

HOPS-1592 fix block count in the block report

HOPS-1590 Fix lease manager for rename operation

HOPS-1588 hdfs_lease_creation_locks rows not created during an upgrade

HOPS-1587 refreshNodes should be called on the leader

Hops sub-tasks

HOPS-1613 NamenodeFsck.getStorages uses objects from other transaction

HOPS-1612 NamenodeFsck.getCorruptReplicas uses objects from other transaction

HOPS-1611 NamenodeFsck.getNumReplicas uses objects from other transaction

HOPS-1610 BlockManager.addStoredBlockImmediateTx uses objects from other transaction

HOPS-1609 BlockManager.addStoredBlockUnderConstructionImmediateTx uses objects from other transaction

HOPS-1608 BlockManager.markBlockAsCorruptTx uses objects from other transaction

HOPS-1607 BlockManager.addStoredBlockUnderConstructionTx uses objects from other transaction

HOPS-1606 BlockManager.addStoredBlockTx uses object from other transaction

HOPS-1605 BlockManager.computeReplicationWorkForBlock uses object from other transaction

HOPS-1603 BlockManager.processOverReplicatedBlocksOnReCommission uses objects from other transaction

HOPS-1602 BlockManager.processIncrementalBlockReport uses objects from other transaction

HOPS-1601 BlockManager.addBlocks uses objects from other transaction

HOPS-1600 LeaseManager.getNumUnderConstructionBlocks uses object from other transaction

HOPS-1599 FSNamesystem.listCorruptFileBlocks uses objects from other transaction

HOPS-1598 FSNamesystem.commitBlockSynchronization uses object from other transaction

HOPS-1597 DecommissionManager.handleInsufficientlyReplicated uses object from other transaction

HOPS-1596 DecommissionManager.pruneSufficientlyReplicated uses object from other transaction

Improvement

HOPS-1581 NDB 8.X support

3.2.0.0

Release notes - hops - Version 3.2.0.0

Bug

HOPS-1592 fix block count in the block report
HOPS-1590 Fix lease manager for rename operation
HOPS-1589 Do not reuse id in proto definition.
HOPS-1588 hdfs_lease_creation_locks rows not created during an upgrade
HOPS-1586 Fix variable IDs in the 3.2.0.0 SQL update file
HOPS-1583 Access storage exception on ACLs when calling delete
HOPS-1580 Leader election use bind address for http address
HOPS-1579 SuperuserKeystoresLoader should use the login user to identify crypto material
HOPS-1578 Access storage exception on ACLs when calling setStoragePolicy
HOPS-1573 Fix garbage collection in Lease Paths
HOPS-1572 Fix DB Access Exception in Lease Manager
HOPS-1571 Fix hopsfs retry cache explosions
HOPS-1561 HopsX509Authenticator should not do a reverse DNS lookup since service discovery
HOPS-1559 Updated security material are not propagated Container
HOPS-1558 accept android handler from consul
HOPS-1546 unchecked cast to INodeDirectory
HOPS-1523 TestEncryptionZonesWithKMS brocken on branch 3.2.0.0
HOPS-1520 TestEncryptionZones broken on branch 3
HOPS-1512 Apply YARN-9205 to fix GPU support on hops-3.2.0.0
HOPS-1480 DevRemoteCRLFetcher missing in 3.2.0.0
HOPS-1432 ApplicationImpl.AppLogInitDoneTransition should save jwt to the state store
HOPS-1390 HDFS concat should keep srcs order Export

Improvement

HOPS-1585 replica under construction fix
HOPS-1568 Add support for large XAttrs (>13500)
HOPS-1551 User specific X.509 certificates for Hops TLS
HOPS-1377 Update to move code base to hadoop3

Task

HOPS-1560 Use system's configured nameserver

Hops sub-tasks

HOPS-1405 Fix hdfs scripts and var table
HOPS-1401 update systemd scripts
HOPS-1400 Put back hadoop-client-minicluster
HOPS-1399 Fix dist profile
HOPS-1396 Update documentation
HOPS-1391 Fix TestGetBlocks
HOPS-1388 Cleanup Dal and Dal-impl
HOPS-1387 Fix TestX509SecurityHandler.testApplicationSubmission
HOPS-1386 Fix TestDistCpSystem
HOPS-1385 Support for YARN Quota
HOPS-1384 Put back check invariants in Hadoop-client-modulres
HOPS-1382 Check ACLs in TestDistCpWithAcls
HOPS-1379 Support for YARN TimelineService HBase tests
HOPS-1378 Move most of the codebase to hadoop 3.2.0

3.2.0.2-RC0

Release notes - hops - Version 3.2.0.2

Improvement

HOPS-1623 unload schema when format --force called
HOPS-1622 Increase application certificate key size

Bug

HOPS-1620 [HDFS-6434] Default permission for creating file should be 644 for WebHdfs/HttpFS
HOPS-1619 [HDFS-10650] DFSClient#mkdirs and DFSClient#primitiveMkdir should use default directory permission

3.2.0.1-RC0

    Release notes - hops - Version 3.2.0.1

Improvement

• [HOPS-1581] - NDB 8.X support

Bug

• [HOPS-1587] - refreshNodes should be called on the leader
• [HOPS-1588] - hdfs_lease_creation_locks rows not created during an upgrade
• [HOPS-1590] - Fix lease manager for rename operation
• [HOPS-1592] - fix block count in the block report
• [HOPS-1595] - Transactions should not use object comming from other transactions
• [HOPS-1614] - Fix quota calculation
• [HOPS-1616] - TestGetFileChecksum keep failing

Hops sub-tasks

• [HOPS-1596] - DecommissionManager.pruneSufficientlyReplicated uses object from other transaction
• [HOPS-1597] - DecommissionManager.handleInsufficientlyReplicated uses object from other transaction
• [HOPS-1598] - FSNamesystem.commitBlockSynchronization uses object from other transaction
• [HOPS-1599] - FSNamesystem.listCorruptFileBlocks uses objects from other transaction
• [HOPS-1600] - LeaseManager.getNumUnderConstructionBlocks uses object from other transaction
• [HOPS-1601] - BlockManager.addBlocks uses objects from other transaction
• [HOPS-1602] - BlockManager.processIncrementalBlockReport uses objects from other transaction
• [HOPS-1603] - BlockManager.processOverReplicatedBlocksOnReCommission uses objects from other transaction
• [HOPS-1605] - BlockManager.computeReplicationWorkForBlock uses object from other transaction
• [HOPS-1606] - BlockManager.addStoredBlockTx uses object from other transaction
• [HOPS-1607] - BlockManager.addStoredBlockUnderConstructionTx uses objects from other transaction
• [HOPS-1608] - BlockManager.markBlockAsCorruptTx uses objects from other transaction
• [HOPS-1609] - BlockManager.addStoredBlockUnderConstructionImmediateTx uses objects from other transaction
• [HOPS-1610] - BlockManager.addStoredBlockImmediateTx uses objects from other transaction
• [HOPS-1611] - NamenodeFsck.getNumReplicas uses objects from other transaction
• [HOPS-1612] - NamenodeFsck.getCorruptReplicas uses objects from other transaction
• [HOPS-1613] - NamenodeFsck.getStorages uses objects from other transaction

2.8.2.10

    Release notes - hops - Version 2.8.2.10

Improvement

• [HOPS-1552] - DataNodes discover NameNodes using Service Discovery
• [HOPS-1565] - Fixes in Mover tools to properly handle single datanode clusters
• [HOPS-1566] - Missing partition by for hdfs_file_provenance_xattrs_buffer

Bug

• [HOPS-1545] - apply HDFS-9625
• [HOPS-1547] - WebHDFS connection hungs when RM is colocated with DN
• [HOPS-1550] - IndividualBlockLock.announceBlockDoesNotExist should not use global blockId variable
• [HOPS-1553] - Fix NNThroughputBenchmark
• [HOPS-1554] - CountersQueue throw EmptyCountersQueueException under highload
• [HOPS-1561] - HopsX509Authenticator should not do a reverse DNS lookup since service discovery
• [HOPS-1562] - ACL mask modification failed due to ClassCastException
• [HOPS-1563] - Missing ACE lock in deprecated rename function
• [HOPS-1564] - Fix Inode File locking
• [HOPS-1572] - Fix DB Access Exception in Lease Manager

Task

• [HOPS-1560] - Use system's configured nameserver

Hops sub-tasks

• [HOPS-1549] - Use Project's x.509 in WebHDFS for application requests