Pulsar tiered storage is introduced by PIP-17 to offload cold data from BookKeeper to external storage. Ledger is the basic offload unit, and one ledger will trigger offload only when the ledger rollover. Pulsar topic offload can be triggered by the following ways:
- Manually trigger offload by using the
bin/pulsar-admincommand. - Automatically trigger offload by the offload policy.
For triggering offload, the offload policy is the most common way. The offload policy can be defined in cluster level, namespace level and topic level, and the offload policy is triggered by the following ways:
- One ledger is closed or rollover
- Check the offload policy
- Trigger offload if the offload policy is satisfied
If one topic has multiple ledgers and the latest ledgers rollover triggered offload, all the previous ledgers will be added into pending offload queue and trigger offload one by one. However, if the topic is unloaded and loaded again, the offload process will be interrupted and needs to waiting for the next ledger rollover to trigger offload. This will cause the offload process is not efficient and the offload process is not triggered in time.
Trigger offload on topic load stage to improve the offload process efficiency and make sure the offload process is triggered in time.
When the topic is loaded, we can check the offload policy to see if the offload policy is satisfied. If the offload policy is satisfied, we can trigger offload immediately. This will improve the offload process efficiency and make sure the offload process is triggered in time.
In order to reduce the impact on topic load when Pulsar is upgraded from the old versions, I introduce a flag named triggerOffloadOnTopicLoad to control whether enable this feature or not.
Fully compatible.
- Mailing List discussion thread: https://lists.apache.org/thread/2ndomp8v4wkcykzthhlyjqfmswor88kv
- Mailing List voting thread: https://lists.apache.org/thread/q4mfn8x69hbgv19nmqx4dmknl3vsn9y8