fix: synthesis fails if tree.json exceeds 512MB

[rix0rrr]

In large applications with a lot of constructs, tree.json can grow to exceed 512MB, which means it can't be serialized anymore.

In this PR, introduce the concept of a "subtree reference". Once a construct tree grows to exceed a fixed number of nodes we write subtrees to individual other files, and put a reference to those files into the original tree.

The number of nodes can be configured with the context value @aws-cdk/core.TreeMetadata:maxNodes, and defaults to 500,000 (that assumes an average size of 1kB per node, which is an overestimate for safety. If we find that this number is too high in practice we may still lower it in the future).

Fixes #27261.

For unupdated consumers, there is graceful degradation here: the parts of the tree they will be able to see are cut off at certain tree depths, but only very large applications would be affected by this. Tree data consumers can be updated gradually to deal with these references.

---

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license

[aws-cdk-automation]

(This review is outdated)

✅ Updated pull request passes all PRLinter validations. Dismissing previous PRLinter review.

[QuantumNeuralCoder]

Few improvements

1. Length tracking

private _length = 0;
public get length() {
  return this._length;
}

2. isEmpty check

public isEmpty(): boolean {
  return this.head === undefined;
}

3. clear method

public clear() {
  this.head = undefined;
  this.last = undefined;
  this._length = 0;
}

4. Iterator support

public *[Symbol.iterator](): IterableIterator<A> {
  let current = this.head;
  while (current) {
    yield current.value;
    current = current.next;
  }
}

for use as

const q = new LinkedQueue<number>([1, 2, 3]);
for (const item of q) {
  console.log(item); // 1, 2, 3
}

[rix0rrr]

I'm not quite comfortable adding these methods that we wouldn't be using.

• length/isEmpty: sure, I guess. Though if possible I prefer coding style of the form "do, then check" rather than "will next call succeed / do next call". This is a style that will avoid TOCTOU's without having to constantly mentally think about whether you are in a situation where it does or does not apply.
• clear: not using it, so why add it?
• iterator: I'm not comfortable writing a for loop over a data structure that gets mutated. I understand we could code the list to make that work, but I would get uneasy seeing the client side of that code, since it relies on too many unknowns.

[QuantumNeuralCoder]

Comment was to make it a more general purpose ds. This is great for the current ask.

[QuantumNeuralCoder]

since balancing the tree is not a consideration here, does replacing bfs with dfs make sense for readability of templates and keeping related referenced nodes together?

[rix0rrr]

My goal here is to show clients that a aren't aware of subtree references "as much as possible". That means they get to see "a bit of Stack1-Stack100" rather than "everything from Stack1 and nothing from Stack2-Stack99"

[QuantumNeuralCoder]

Wondering if there is a metadata (not same as CDK metadata or maybe same) section/file that should be included to summarize stats like number of nodes in the template, how many refs were generated etc. might be good for traceability and troubleshooting if things go wrong.

[QuantumNeuralCoder]

there is an assumptions stated above where number of nodes as opposed to size of the template determines when we trigger the splitting algo. since each node varies in size, would it make sense to keep track of the size as well. For example a node with many many props may cause the template to become fuller faster without hitting the number of nodes limitation.

[rix0rrr]

Estimating the size of a node is an expensive operation. I used to have a solution that sampled node sizes by stringifying the entire JSON string, but it is slower and you need to think about sampling frequencies and size bias: leaf nodes (resources) are typically larger than intermediate nodes (grouping constructs).

I did a conservative, adjustable estimate here which will be cheap to compute. I think this will suffice for now, we can always come back it and do something more sophisticated if we find this to be causing problems.

[mergify]

Thank you for contributing! Your pull request will be updated from main and then merged automatically (do not update manually, and be sure to allow changes to be pushed to your fork).

[mergify]

Thank you for contributing! Your pull request will be updated from main and then merged automatically (do not update manually, and be sure to allow changes to be pushed to your fork).

[aws-cdk-automation]

AWS CodeBuild CI Report

• CodeBuild project: AutoBuildv2Project1C6BFA3F-wQm2hXv2jqQv
• Commit ID: 6f37d38
• Result: SUCCEEDED
• Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

[mergify]

Thank you for contributing! Your pull request will be updated from main and then merged automatically (do not update manually, and be sure to allow changes to be pushed to your fork).

[github-actions]

Comments on closed issues and PRs are hard for our team to see.
If you need help, please open a new issue that references this one.