BucketLifecycleConfiguration Resource doesn't gets refreshed rules.

[luvk1412]

Describe the bug
BucketLifecycleConfiguration.rules doesn't fetches refreshed rules after performing a put operation.
I am talking about this resource : https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/s3.html#bucketlifecycleconfiguration

Steps to reproduce
Just try to insert several rules in a loop one by one. Basic steps.
Loop :

• BucketLifecycleConfiguration.load()
• BucketLifecycleConfiguration.rules // to get latest rules
• Add new rule to list
• BucketLifecycleConfiguration.put(new_rules_list)

Performing above in loop even for 20+ rules would result in not adding all of them to bucket lifecycle configuration.

Expected behavior
All the rules i am trying to add should be added.

[swetashre]

@luvk1412 - Thank you for your post. If you are using load() operation after putting a new rule to your bucket then you should get all the rules if put operation is successful. I am not able to reproduce the behavior by calling load() after putting a new rule.
Can you provide me the code sample you are using so that i can try to reproduce the behavior ?

[luvk1412]

Hi @swetashre , Below is the code. If i run this code on my staging bucket, all 40 rules are not added in the bucket.

import boto3
import logging

s3 = boto3.resource('s3')

logging.basicConfig(filename='client1.log', format='%(asctime)s - %(levelname)s - %(message)s', level=logging.INFO,
                    datefmt='%Y-%m-%d %I:%M:%S %p')
logger = logging.getLogger()
bucket_lifecycle = s3.BucketLifecycleConfiguration("staging.common.logfilter") # Value is riva-hbase-backups


def add_s3_lifecycle(event):
    bucket_lifecycle.load()
    rules = bucket_lifecycle.rules
    new_rule = get_lifecycle_rule(event)
    rules.append(new_rule)
    logger.info('Adding rule : %s,  final rules: %s', new_rule, rules)
    response = bucket_lifecycle.put(LifecycleConfiguration={'Rules': rules})
    logger.info('S3 config put response : %s', response)


def get_lifecycle_rule(event):
    return {
        'Expiration': {
            'Days': 120
        },
        'ID': get_lifecycle_id(event),
        'Filter': {
            'Prefix': get_event_prefix(event)
        },
        'Status': 'Enabled',
        'Transitions': [
            {
                'Days': 60,
                'StorageClass': 'GLACIER'
            }
        ],
        'AbortIncompleteMultipartUpload': {
            'DaysAfterInitiation': 7
        }
    }


def remove_s3_lifecycle(event):
    try:
        logger.info('Going to delete lifecycle for event : %s', event)
        bucket_lifecycle.load()
        rules = bucket_lifecycle.rules
        logger.info('Received rules from s3 bucket : %s', rules)
        event_id = get_lifecycle_id(event)
        final_rules = [rule for rule in rules if rule['ID'] != event_id]
        if len(rules) != len(final_rules):
            logger.info('Removing rule %s , final rules: %s', event_id, final_rules)
            response = bucket_lifecycle.put(LifecycleConfiguration={'Rules': final_rules})
            bucket_lifecycle.load()
            logger.info('S3 config put response : %s', response)
    except Exception as e:
        logger.error('Error deleting lifecycle rule for event : %s', event)
        logger.exception(e)


def get_lifecycle_id(event):
    return 'temp:{}'.format(get_event_prefix(event))


def get_event_prefix(event):
    return "test"+'/'+event+'/' 


if __name__ == '__main__':
    bucket_lifecycle.load()
    r_init = bucket_lifecycle.rules
    logger.info("Initial number of rules in bucket : %s", len(r_init))
    events_lst = (('demo', 'rule0'),('demo', 'rule1'),('demo', 'rule2'),('demo', 'rule3'),('demo', 'rule4'),('demo', 'rule5'),('demo', 'rule6'),('demo', 'rule7'),('demo', 'rule8'),('demo', 'rule9'),('demo', 'rule10'),('demo', 'rule11'),('demo', 'rule12'),('demo', 'rule13'),('demo', 'rule14'),('demo', 'rule15'),('demo', 'rule16'),('demo', 'rule17'),('demo', 'rule18'),('demo', 'rule19'),('demo', 'rule20'),('demo', 'rule21'),('demo', 'rule22'),('demo', 'rule23'),('demo', 'rule24'),('demo', 'rule25'),('demo', 'rule26'),('demo', 'rule27'),('demo', 'rule28'),('demo', 'rule29'),('demo', 'rule30'),('demo', 'rule31'),('demo', 'rule32'),('demo', 'rule33'),('demo', 'rule34'),('demo', 'rule35'),('demo', 'rule36'),('demo', 'rule37'),('demo', 'rule38'),('demo', 'rule39'))
    logger.info("Going to add %s rules", len(events_lst))
    for ad, eve in events_lst:
        add_s3_lifecycle(ad+':'+eve)
    bucket_lifecycle.load()
    r_fin = bucket_lifecycle.rules
    logger.info("Number of rule added in bucket : %s", len(r_fin) - r_init)

[swetashre]

@luvk1412 - Thank you for providing me the sample code. There can be only one configuration per bucket but the configuration can have multiple rules. So can you please try the below code and see if that works for you ? I just concatenated all the rules before sending so that each rule will fall under same configuration.

import boto3

s3 = boto3.resource('s3')

bucket_lifecycle = s3.BucketLifecycleConfiguration('bucketname')

rules = bucket_lifecycle.rules

def add_s3_rule(event):
    new_rule = get_lifecycle_rule(event)
    rules.append(new_rule)

for ad, eve in events_lst:
    add_s3_rule(ad+':'+eve)

# so in this case each rule falls under same configuration
response = bucket_lifecycle.put(LifecycleConfiguration={'Rules': rules}) 

[luvk1412]

@swetashre I know this will work but in my application each rule addition is dependent on some task A, and task A and the rule addition for it are a part of a transaction. So i cannot combine this, the way you told. But shouldn't the load call update the configuration and then configuration.rules provide latest rules after each put ? I mean i am thinking this way that each time i fetch the configuration and put a new configuration and fetch should get the latest configuration. I am aware that each bucket has single configuration but put is changing whole configuration in my code right doing put of each role and updating the whole configuration. Am i missing something here, isn't load supposed to work this way according to documentation ?

[swetashre]

@luvk1412 - When we use load method it uses get_bucket_lifecycle_configuration api to get the response and in this case the api is sometimes giving less number of rule than it is present in the bucket at that time. It might be because of eventual consistency. The load is working as expected but this is the behavior of service which is causing the issue. You can use sleep but i don't know how convenient it will be for your use case.

[nehagarwaal]

I am also stuck with the same issue.