Skip to content

v0.5.7
Compare
Choose a tag to compare
@SanjayVas SanjayVas released this 09 Aug 16:54
· 102 commits to main since this release
v0.5.7
aabb17a
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Learn about vigilant mode.

Issues Addressed

  • Existing computations may starve initialization of new computations. See #1637
  • Duchy mill writing output blob error should be transient. See #1644
  • Avoid failing Measurements at Duchy due to stale ComputationParticipant state. See #1663
  • Fix batch get metrics limit in reports service. See #1529
  • CreateEventGroup on v0.5.x fails with v0.4.x EventGroup message containing encrypted metadata. See #1705
  • Multiple Computations are claimed at the same time. See #1722
  • Retryable storage errors may be treated as non-retryable. See #1731

Highlights

  • Mill horizontal autoscaling. See #1697
    • This significantly reduces cost during low traffic periods.
  • More performance improvements for Report creation. See #1641
  • Improved Duchy fault tolerance.
  • The HMSS EDP client library is published as a Maven artifact See README.
  • Kingdom-less panel exchange is supported on Google Cloud. See the deployment guide.

Potentially Requiring Action

  • As part of the Mill horizontal autoscaling feature, the K8s Deployment objects for Mills are replaced with PodTemplate objects.
    • The PodTemplates are used by new MillJobScheduler Deployment to schedule Mill Jobs.
    • The MillJobScheduler Deployment needs some permissions on K8s resources in the same cluster using K8s RBAC. For the dev configuration, this consists of a new Role and RoleBinding.
    • Maximum concurrency for LLv2 and HMSS can be configured using the --llv2-maximum-concurrency and --hmss-maximum-concurrency options to the scheduler, respectively.
      • You will need to ensure that there are enough machine resources available at maximum concurrency. For example, ensuring that the cluster autoscaler has a sufficiently high value for the maximum number of nodes.
    • The value of the --work-lock-duration option for the Mill PodTemplate must match the corresponding option for the scheduler Deployment.
      • --llv2-work-lock-duration or --hmss-work-lock-duration, depending on the Mill type.
  • Kubernetes object configuration changes for HMSS. Some of these are required even if HMSS is not enabled. See the updated deployment guides for GKE and EKS.
    • There is a new HMSS Mill.
    • The Kingdom internal API and public API servers have a new required --hmss-protocol-config-config option. In the dev configuration, this comes from the hmss_protocol_config_config.textproto file in the certs-and-configs Secret.
    • The Herald now needs access to blob storage. This means it needs to run under a K8s ServiceAccount that has access to the appropriate bucket, as well as the appropriate command line options.
    • The Herald and the HMSS Mill have a new --key-encryption-key-file option for worker (non-aggregator) Duchies. In the dev configuration, this comes from the certs-and-configs Secret.
    • The Duchy Requisition fulfillment server may need more heap space to handle HMSS Requisitions.
    • The EDP simulator may need more heap space to fulfill HMSS Requisitions.

What's Changed

  • Fix batch get metrics limit in reports service by @tristanvuong2021 in #1529
  • Optimize privacy budget charge for reach and frequency measurement. by @ple13 in #1650
  • Use index in existing Measurements check when creating Measurements by @tristanvuong2021 in #1641
  • Add executable binary classes and docker images for HMSS. by @renjiezh in #1632
  • Create panel match internal versions of protobuf messages by @robinsons in #1661
  • Add metrics for HMSS mill. by @renjiezh in #1633
  • Update common-jvm to 0.85.0 by @renjiezh in #1664
  • Make duchy storage writing error transient. by @renjiezh in #1656
  • Add etag support to system ComputationParticipant. by @SanjayVas in #1654
  • Add ComputationParticipants.GetComputationParticipant to system API. by @SanjayVas in #1657
  • Add FulfillRequisitionRequestBuilder by @kungfucraig in #1638
  • Add support for reach only HMSS. by @ple13 in #1655
  • chore: update rules_proto dep to 6.0.2 by @SanjayVas in #1667
  • Fix api-linter error in system GetComputationParticipant. by @SanjayVas in #1666
  • Fix requisition_params being set when set method wasn't called. by @SanjayVas in #1669
  • chore: update common-cpp dep to 0.13.0 by @SanjayVas in #1672
  • Allow Herald to skip stage to catch up with Measurement state. by @renjiezh in #1671
  • Retry updating ComputationParticipant to avoid Measurement failure. by @SanjayVas in #1665
  • Create Population Requisition Fulfiller by @jojijac0b in #1527
  • Update panel match client to use internal proto definitions. by @robinsons in #1670
  • Add Amazon EDP root certificate by @YuhongWang-Amazon in #1676
  • Create ExchangeCheckpoint protobuf message for Kingdom-less panel match by @robinsons in #1675
  • Add target for exporting shareshuffle library as a Maven artifact. by @SanjayVas in #1674
  • Publish Maven artifacts on release. by @SanjayVas in #1680
  • Implement CertificateManager for Kingdom-less panel match protocol. by @robinsons in #1681
  • fix: use version for common-jvm dep by @SanjayVas in #1687
  • Add library to compute the variances for HMSS protocol. by @ple13 in #1677
  • Add prioritizedStages for duchy claimTask by @renjiezh in #1673
  • HSMM reach-only integration test by @renjiezh in #1679
  • Add protocol config to the request header by @kungfucraig in #1690
  • Set participant_child_reference_id for claimWork log entries. by @SanjayVas in #1683
  • Implement ApiClient for Kingdom-less panel match protocol. by @robinsons in #1685
  • fix!: shorten overly long name for HMSS Mill Deployment by @SanjayVas in #1692
  • Add enabledHmssMeasurementConsumers into kingdom config by @renjiezh in #1688
  • Refactor HMSS to remove sketch params by @renjiezh in #1689
  • fix: publish all Maven artifacts, not just one by @SanjayVas in #1694
  • Add HMSS in process tests. by @ple13 in #1691
  • Make max parallel claimed exchange step logic reusable. by @robinsons in #1699
  • feat!: enable horizontal autoscaling of Liquid Legions v2 Mill by @SanjayVas in #1697
  • Added method for batching search the findAcdpBalanceEntries by @RibeiroAna in #1684
  • Prepare example Kingdom-less exchange daemon for GKE. by @robinsons in #1698
  • fix: ensure CreateEventGroup succeeds with legacy MC public key by @SanjayVas in #1706
  • Add covariance calculation for hmss. by @ple13 in #1702
  • fix: switch Duchy cluster autoscaling profile back to BALANCED by @SanjayVas in #1715
  • fix: update Bazel lockfile by @SanjayVas in #1712
  • fix: don't export OpenTelemetry traces using prometheus exporter by @SanjayVas in #1713
  • Make Kingdom-less panel exchange a cron job. by @robinsons in #1711
  • feat!: enable horizontal autoscaling of HMSS Mill by @SanjayVas in #1707
  • fix: ensure Mill job should process claimed work item by @SanjayVas in #1718
  • refactor: use suspendCoroutine instead of Deferred for K8s API calls by @SanjayVas in #1714
  • Remove unused code from duchy tests by @tristanvuong2021 in #1725
  • Remove unnecessary null assertions by @tristanvuong2021 in #1723
  • docs: add information about the HMSS EDP library Maven artifact by @SanjayVas in #1710
  • fix: Resolve race condition of Duchy claimTask in spanner implementation. by @renjiezh in #1726
  • docs: Add command line of key encryption key generation by @renjiezh in #1727
  • Add In Process Reach Measurement Accuracy Test for HMSS. by @ple13 in #1717
  • docs: Update Dev Standards doc for Conventional Commits by @SanjayVas in #1730
  • fix: Throw StorageException as TransientError from the mill by @renjiezh in #1731
  • fix: Add duchy id into computation blob storage prefix. by @renjiezh in #1719
  • fix: Wait for EDP simulator to initialize before considering it ready by @SanjayVas in #1732
  • docs: Update Duchy deployment guides for Mill autoscaling by @SanjayVas in #1734
  • fix: Specify version for panel exchange Maven artifacts by @SanjayVas in #1738
  • Resolve claim lock contention issue by separating transaction by @renjiezh in #1741

New Contributors

Full Changelog: v0.5.6...v0.5.7

Contributors

kungfucraig, SanjayVas, and 7 other contributors
Assets 2
Loading