v0.5.7-rc3

Issues Addressed

• Existing computations may starve initialization of new computations. See #1637
• Duchy mill writing output blob error should be transient. See #1644
• Avoid failing Measurements at Duchy due to stale ComputationParticipant state. See #1663
• Fix batch get metrics limit in reports service. See #1529
• CreateEventGroup on v0.5.x fails with v0.4.x EventGroup message containing encrypted metadata. See #1705
• Multiple Computations are claimed at the same time. See #1722
• Retryable storage errors may be treated as non-retryable. See #1731

Highlights

• Mill horizontal autoscaling. See #1697
  • This significantly reduces cost during low traffic periods.
• More performance improvements for Report creation. See #1641
• Improved Duchy fault tolerance.
• The HMSS EDP client library is published as a Maven artifact See README.
• Kingdom-less panel exchange is supported on Google Cloud. See the deployment guide.

Potentially Requiring Action

• As part of the Mill horizontal autoscaling feature, the K8s Deployment objects for Mills are replaced with PodTemplate objects.
  • The PodTemplates are used by new MillJobScheduler Deployment to schedule Mill Jobs.
  • The MillJobScheduler Deployment needs some permissions on K8s resources in the same cluster using K8s RBAC. For the dev configuration, this consists of a new Role and RoleBinding.
  • Maximum concurrency for LLv2 and HMSS can be configured using the --llv2-maximum-concurrency and --hmss-maximum-concurrency options to the scheduler, respectively.
    • You will need to ensure that there are enough machine resources available at maximum concurrency. For example, ensuring that the cluster autoscaler has a sufficiently high value for the maximum number of nodes.
  • The value of the --work-lock-duration option for the Mill PodTemplate must match the corresponding option for the scheduler Deployment.
    • --llv2-work-lock-duration or --hmss-work-lock-duration, depending on the Mill type.
• Kubernetes object configuration changes for HMSS. Some of these are required even if HMSS is not enabled. See the updated deployment guides for GKE and EKS.
  • There is a new HMSS Mill.
  • The Kingdom internal API and public API servers have a new required --hmss-protocol-config-config option. In the dev configuration, this comes from the hmss_protocol_config_config.textproto file in the certs-and-configs Secret.
  • The Herald now needs access to blob storage. This means it needs to run under a K8s ServiceAccount that has access to the appropriate bucket, as well as the appropriate command line options.
  • The Herald and the HMSS Mill have a new --key-encryption-key-file option for worker (non-aggregator) Duchies. In the dev configuration, this comes from the certs-and-configs Secret.
  • The Duchy Requisition fulfillment server may need more heap space to handle HMSS Requisitions.
  • The EDP simulator may need more heap space to fulfill HMSS Requisitions.

What's Changed

• Fix batch get metrics limit in reports service by @tristanvuong2021 in #1529
• Optimize privacy budget charge for reach and frequency measurement. by @ple13 in #1650
• Use index in existing Measurements check when creating Measurements by @tristanvuong2021 in #1641
• Add executable binary classes and docker images for HMSS. by @renjiezh in #1632
• Create panel match internal versions of protobuf messages by @robinsons in #1661
• Add metrics for HMSS mill. by @renjiezh in #1633
• Update common-jvm to 0.85.0 by @renjiezh in #1664
• Make duchy storage writing error transient. by @renjiezh in #1656
• Add etag support to system ComputationParticipant. by @SanjayVas in #1654
• Add ComputationParticipants.GetComputationParticipant to system API. by @SanjayVas in #1657
• Add FulfillRequisitionRequestBuilder by @kungfucraig in #1638
• Add support for reach only HMSS. by @ple13 in #1655
• chore: update rules_proto dep to 6.0.2 by @SanjayVas in #1667
• Fix api-linter error in system GetComputationParticipant. by @SanjayVas in #1666
• Fix requisition_params being set when set method wasn't called. by @SanjayVas in #1669
• chore: update common-cpp dep to 0.13.0 by @SanjayVas in #1672
• Allow Herald to skip stage to catch up with Measurement state. by @renjiezh in #1671
• Retry updating ComputationParticipant to avoid Measurement failure. by @SanjayVas in #1665
• Create Population Requisition Fulfiller by @jojijac0b in #1527
• Update panel match client to use internal proto definitions. by @robinsons in #1670
• Add Amazon EDP root certificate by @YuhongWang-Amazon in #1676
• Create ExchangeCheckpoint protobuf message for Kingdom-less panel match by @robinsons in #1675
• Add target for exporting shareshuffle library as a Maven artifact. by @SanjayVas in #1674
• Publish Maven artifacts on release. by @SanjayVas in #1680
• Implement CertificateManager for Kingdom-less panel match protocol. by @robinsons in #1681
• fix: use version for common-jvm dep by @SanjayVas in #1687
• Add library to compute the variances for HMSS protocol. by @ple13 in #1677
• Add prioritizedStages for duchy claimTask by @renjiezh in #1673
• HSMM reach-only integration test by @renjiezh in #1679
• Add protocol config to the request header by @kungfucraig in #1690
• Set participant_child_reference_id for claimWork log entries. by @SanjayVas in #1683
• Implement ApiClient for Kingdom-less panel match protocol. by @robinsons in #1685
• fix!: shorten overly long name for HMSS Mill Deployment by @SanjayVas in #1692
• Add enabledHmssMeasurementConsumers into kingdom config by @renjiezh in #1688
• Refactor HMSS to remove sketch params by @renjiezh in #1689
• fix: publish all Maven artifacts, not just one by @SanjayVas in #1694
• Add HMSS in process tests. by @ple13 in #1691
• Make max parallel claimed exchange step logic reusable. by @robinsons in #1699
• feat!: enable horizontal autoscaling of Liquid Legions v2 Mill by @SanjayVas in #1697
• Added method for batching search the findAcdpBalanceEntries by @RibeiroAna in #1684
• Prepare example Kingdom-less exchange daemon for GKE. by @robinsons in #1698
• fix: ensure CreateEventGroup succeeds with legacy MC public key by @SanjayVas in #1706
• Add covariance calculation for hmss. by @ple13 in #1702
• fix: switch Duchy cluster autoscaling profile back to BALANCED by @SanjayVas in #1715
• fix: update Bazel lockfile by @SanjayVas in #1712
• fix: don't export OpenTelemetry traces using prometheus exporter by @SanjayVas in #1713
• Make Kingdom-less panel exchange a cron job. by @robinsons in #1711
• feat!: enable horizontal autoscaling of HMSS Mill by @SanjayVas in #1707
• fix: ensure Mill job should process claimed work item by @SanjayVas in #1718
• refactor: use suspendCoroutine instead of Deferred for K8s API calls by @SanjayVas in #1714
• Remove unused code from duchy tests by @tristanvuong2021 in #1725
• Remove unnecessary null assertions by @tristanvuong2021 in #1723
• docs: add information about the HMSS EDP library Maven artifact by @SanjayVas in #1710
• fix: Resolve race condition of Duchy claimTask in spanner implementation. by @renjiezh in #1726
• docs: Add command line of key encryption key generation by @renjiezh in #1727
• Add In Process Reach Measurement Accuracy Test for HMSS. by @ple13 in #1717
• docs: Update Dev Standards doc for Conventional Commits by @SanjayVas in #1730
• fix: Throw StorageException as TransientError from the mill by @renjiezh in #1731
• fix: Add duchy id into computation blob storage prefix. by @renjiezh in #1719
• fix: Wait for EDP simulator to initialize before considering it ready by @SanjayVas in #1732

New Contributors

• @RibeiroAna made their first contribution in #1684

Full Changelog: v0.5.6...v0.5.7-rc3