v0.5.7-rc2

Issues Addressed

• Existing computations may starve initialization of new computations. See #1637
• Duchy mill writing output blob error should be transient. See #1644
• Avoid failing Measurements at Duchy due to stale ComputationParticipant state. See #1663
• Fix batch get metrics limit in reports service. See #1529

Highlights

• Mill horizontal autoscaling. See #1697
  • This significantly reduces cost during low traffic periods.
• More performance improvements for Report creation. See #1641
• Improved Duchy fault tolerance.
• The HMSS EDP client library is published as a Maven artifact See #1710

Potentially Requiring Action

• As part of the Mill horizontal autoscaling feature, the K8s Deployment objects for Mills are replaced with PodTemplate objects.
  • The PodTemplates are used by new MillJobScheduler Deployment to schedule Mill Jobs.
  • The MillJobScheduler Deployment needs some permissions on K8s resources in the same cluster using K8s RBAC. For the dev configuration, this consists of a new Role and RoleBinding.
• Kubernetes object configuration changes for HMSS. Some of these are required even if HMSS is not enabled. See the updated deployment guides for GKE and EKS.
  • There is a new HMSS Mill.
  • The Kingdom internal API and public API servers have a new required --hmss-protocol-config-config option. In the dev configuration, this comes from the hmss_protocol_config_config.textproto file in the certs-and-configs Secret.
  • The Herald now needs access to blob storage. This means it needs to run under a K8s ServiceAccount that has access to the appropriate bucket, as well as the appropriate command line options.
  • The Herald and the HMSS Mill have a new --key-encryption-key-file option for worker (non-aggregator) Duchies. In the dev configuration, this comes from the certs-and-configs Secret.
  • The Duchy Requisition fulfillment server may need more heap space to handle HMSS Requisitions.
  • The EDP simulator may need more heap space to fulfill HMSS Requisitions.

What's Changed

• Fix batch get metrics limit in reports service by @tristanvuong2021 in #1529
• Optimize privacy budget charge for reach and frequency measurement. by @ple13 in #1650
• Use index in existing Measurements check when creating Measurements by @tristanvuong2021 in #1641
• Add executable binary classes and docker images for HMSS. by @renjiezh in #1632
• Create panel match internal versions of protobuf messages by @robinsons in #1661
• Add metrics for HMSS mill. by @renjiezh in #1633
• Update common-jvm to 0.85.0 by @renjiezh in #1664
• Make duchy storage writing error transient. by @renjiezh in #1656
• Add etag support to system ComputationParticipant. by @SanjayVas in #1654
• Add ComputationParticipants.GetComputationParticipant to system API. by @SanjayVas in #1657
• Add FulfillRequisitionRequestBuilder by @kungfucraig in #1638
• Add support for reach only HMSS. by @ple13 in #1655
• chore: update rules_proto dep to 6.0.2 by @SanjayVas in #1667
• Fix api-linter error in system GetComputationParticipant. by @SanjayVas in #1666
• Fix requisition_params being set when set method wasn't called. by @SanjayVas in #1669
• chore: update common-cpp dep to 0.13.0 by @SanjayVas in #1672
• Allow Herald to skip stage to catch up with Measurement state. by @renjiezh in #1671
• Retry updating ComputationParticipant to avoid Measurement failure. by @SanjayVas in #1665
• Create Population Requisition Fulfiller by @jojijac0b in #1527
• Update panel match client to use internal proto definitions. by @robinsons in #1670
• Add Amazon EDP root certificate by @YuhongWang-Amazon in #1676
• Create ExchangeCheckpoint protobuf message for Kingdom-less panel match by @robinsons in #1675
• Add target for exporting shareshuffle library as a Maven artifact. by @SanjayVas in #1674
• Publish Maven artifacts on release. by @SanjayVas in #1680
• Implement CertificateManager for Kingdom-less panel match protocol. by @robinsons in #1681
• fix: use version for common-jvm dep by @SanjayVas in #1687
• Add library to compute the variances for HMSS protocol. by @ple13 in #1677
• Add prioritizedStages for duchy claimTask by @renjiezh in #1673
• HSMM reach-only integration test by @renjiezh in #1679
• Add protocol config to the request header by @kungfucraig in #1690
• Set participant_child_reference_id for claimWork log entries. by @SanjayVas in #1683
• Implement ApiClient for Kingdom-less panel match protocol. by @robinsons in #1685
• fix!: shorten overly long name for HMSS Mill Deployment by @SanjayVas in #1692
• Add enabledHmssMeasurementConsumers into kingdom config by @renjiezh in #1688
• Refactor HMSS to remove sketch params by @renjiezh in #1689
• fix: publish all Maven artifacts, not just one by @SanjayVas in #1694
• Add HMSS in process tests. by @ple13 in #1691
• Make max parallel claimed exchange step logic reusable. by @robinsons in #1699
• feat!: enable horizontal autoscaling of Liquid Legions v2 Mill by @SanjayVas in #1697
• Added method for batching search the findAcdpBalanceEntries by @RibeiroAna in #1684

New Contributors

• @RibeiroAna made their first contribution in #1684

Full Changelog: v0.5.6...v0.5.7-rc2