Add lifecycle rules to all non-production buckets

[erikamov]

User story / feature request

As a Developer,
When I create test data on GC Buckets designated for testing or development,
I want the test data deleted after 30 days.

As part of the Cal-ITP Cost Reduction Plan, we need to clear test buckets to reduce Goggle Cloud monthly costs.
Cal-ITP Test Buckets Cleanup.pdf

Acceptance Criteria

When I list GC Buckets designated for testing or development the should not contain data older than 30 days.

Notes

Make sure Autoclass disabled on those Buckets.

[erikamov]

Post message about the cleanup and attached PDF on slack channel calitp-data-infra. Should I post in any other channel?

[tiffanychu90]

• One things I'm noticing for my set of BQ tables for dbt (sheet Pivot Table 1), I always follow the instructions here to run poetry run dbt run --full-refresh if you haven't been creating new tables for a couple weeks.
  • I only really work in one section of the warehouse though (mart_gtfs), so I probably could have a lot of tables in tiffany_* portions of cal-itp-data-infra-staging deleted. What would be the command I should run to just bring in mart_gtfs?
  • If the analysts can always just bring in the specific portion of the tables they need for adding new tables in dbt, which seems fairly infrequent nowadays except for me + Vivek, we can delete christian, eric, mine (and I'll repopulate with the portion I need).
• Let's clean up other users who aren't working on adding warehouse tables in the foreseeable future: soren, anyone else?

[erikamov]

• One things I'm noticing for my set of BQ tables for dbt (sheet Pivot Table 1), I always follow the instructions here to run poetry run dbt run --full-refresh if you haven't been creating new tables for a couple weeks.

  • I only really work in one section of the warehouse though (mart_gtfs), so I probably could have a lot of tables in tiffany_* portions of cal-itp-data-infra-staging deleted. What would be the command I should run to just bring in mart_gtfs?
  • If the analysts can always just bring in the specific portion of the tables they need for adding new tables in dbt, which seems fairly infrequent nowadays except for me + Vivek, we can delete christian, eric, mine (and I'll repopulate with the portion I need).

• Let's clean up other users who aren't working on adding warehouse tables in the foreseeable future: soren, anyone else?

I usually run only the table that I need, and you can use "+" before the path to create any dependent models, for example: poetry run dbt run -s +models/mart/gtfs/dim_agency.sql. Or you if you use poetry run dbt run -s +models/mart/gtfs/ will run all models under models/mart/gtfs/ folder and any other models needed to generate them. Can also do the same for tests: poetry run dbt test -s +models/mart/gtfs/

See item ii: