[BACKWARDS INCOMPATIBLE] S3 env var

[jhsinger-klotho]

• Does any part of it require special attention?
• Does it relate to or fix any issue? closes #227 and closes https://github.com/klothoplatform/klotho-pro/issues/51

Changes our logic so that we create a bucket per ID with an FS annotation. We also create a new InternalResource concept so that we can track the resource we use for proxy/dispatcher (payloads bucket)

We use env vars to inject the bucket names into the fs runtime and we have a file per bucket at the moment. Opened #249 as an enhancement so we can limit the number of runtime files in the future.

We also can fully do sanitization at the IAC layer and now our payloads bucket will have the wording InternalKlothoPayloads.

opened klotho #247 because we cant fully remove the payloads bucket from template config until we take out the hardcoding from pubsub

Standard checks

Further testing https://github.com/klothoplatform/klotho/actions/runs/4198937619

• Unit tests: Any special considerations? changed some of the transform ones where possible
• Docs: Do we need to update any docs, internal or public? Im not sure because we never documented that we dont create multiple buckets
• Backwards compatibility: Will this break existing apps? If so, what would be the extra work required to keep them working? No. This is none and stated in the task, we will need to send out an email

[ghost]

Does this need to be sanitized? I forget what (if any) restrictions we enforce on ids.

[jhsinger-klotho]

so this is just for the import itself in the file. I cut a card so that we actually just pass the bucket name to the open call, but that requires a lot of extra work, so for now we are just importing an fs runtime per bucket and thats why we need different import modules

[ghost]

Ditto on sanitization here.

[jhsinger-klotho]

the name is just the env var name, so it doesnt go through sanitization. The actual bucket name will be the value and that is dynamically generated based on the sanitized bucket name

[ghost]

Do annotation ids have any restriction, do you know offhand? For example, can they contain spaces? (afaik, an env var can't.)

[jhsinger-klotho]

yeah the annotation ids are the same regex as the app name, so i believe that it should be valid env vars.

[ghost]

Sorry, I forgot to change the radio box to "Approve" when I submitted the previous review. Neither of my comments is blocking.

[github-actions]

Package	Line Rate	Health
github.com/klothoplatform/klotho/pkg/analytics	3%	❌
github.com/klothoplatform/klotho/pkg/annotation	23%	❌
github.com/klothoplatform/klotho/pkg/cli	4%	❌
github.com/klothoplatform/klotho/pkg/core	21%	❌
github.com/klothoplatform/klotho/pkg/env_var	82%	✔
github.com/klothoplatform/klotho/pkg/exec_unit	54%	➖
github.com/klothoplatform/klotho/pkg/infra/kubernetes	59%	➖
github.com/klothoplatform/klotho/pkg/infra/kubernetes/helm	39%	❌
github.com/klothoplatform/klotho/pkg/input	72%	➖
github.com/klothoplatform/klotho/pkg/lang	38%	❌
github.com/klothoplatform/klotho/pkg/lang/dockerfile	0%	❌
github.com/klothoplatform/klotho/pkg/lang/golang	36%	❌
github.com/klothoplatform/klotho/pkg/lang/javascript	48%	❌
github.com/klothoplatform/klotho/pkg/lang/python	63%	➖
github.com/klothoplatform/klotho/pkg/lang/yaml	0%	❌
github.com/klothoplatform/klotho/pkg/logging	6%	❌
github.com/klothoplatform/klotho/pkg/multierr	95%	✔
github.com/klothoplatform/klotho/pkg/provider/aws	50%	➖
github.com/klothoplatform/klotho/pkg/provider/aws/resources	70%	➖
github.com/klothoplatform/klotho/pkg/runtime	78%	✔
github.com/klothoplatform/klotho/pkg/static_unit	33%	❌
github.com/klothoplatform/klotho/pkg/updater	30%	❌
github.com/klothoplatform/klotho/pkg/validation	74%	➖
github.com/klothoplatform/klotho/pkg/yaml_util	79%	✔
Summary	42% (4166 / 9912)	❌