Composable turbo.json #2706

mehulkar · 2022-11-15T03:13:19Z

This feature allows monorepos to place a turbo.json file in workspaces and add a "extends" key to extend from the root configuration to add/remove to the pipeline. After this PR, individual workspaces should be able to:

add keys to task defintiions (e.g. changing dependsOn or outputs configs)
reset keys in task definitions to defaults (e.g. dependsOn: []`)
add tasks (e.g add a lint command only in the workspace that implements it)

Limitations

Some of these limitations are edge cases, but still listing them out. You cannot:

extend config other than the pipeline tasks. For example, remoteCache, globalEnv, etc cannot be changed for individual workspaces.
extend from other workspaces.
configure the root turbo.json to extend from anything.
add or subtract to existing array configs (e.g. if your root config has dependsOn: ['something'], you can't append to this array, you can only overwrite it.)
declare tasks in the package#task syntax in workspaces. The package is inferred.

These limitations are based on a lot of research to keep the design space open, gather use cases, and avoid making the wrong decisions before putting the feature into the hands of real users. In short: to make it possible to ship this sooner!

Example

./turbo.json (root)

 {
 "pipeline": {
 		"build": {
 			"outputs": ["dist/**"],
 			"dependsOn": ["lint"]
 		}
 	}
 }

./packages/ui/turbo.json (ui workspace)

 {
 	"extends": ["//"],
 	"pipeline": {
 		"build": {
 			"dependsOn": ["lint", "compile"],
 			"outputs": ["lib/**"]
 		}
 	}
 }

With these configs, when you run turbo build, all workspaces except the ui workspace will cache the dist/ directory and run their lint commands first. The ui workspace will run lint and compile first, and cache the lib/ directory instead.

Review Notes

Implementation is not super robust, it just looks for a workspace config and root config and merges them. I'd like to get this out as feature-complete work, before adding any crawling behavior for future work.
Read the test cases (*.t files) to see what use cases this supports
Happy to pair-review anything.

Tests

When workspace adds a new task that is not defined in the root config
When workspace adds new keys into task config that are not defined in the root
When workspace has a task definition, but omits some keys that root config has
When workspace overrides individual keys of a task config
When workspace doesn't have a turbo.json

Pre-req work

Deferred Work

All limitations noted above
Docs
Perf optimization: avoid reading workspace configs if we know they don't exist (e.g. do a glob find up front)
fieldsMeta + Merge is won't scale as new keys are added. Need a more elegant solution

vercel · 2022-11-15T03:13:22Z

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name	Status	Preview	Comments	Updated
examples-svelte-web	🔄 Building (Inspect)	Visit Preview		Feb 13, 2023 at 9:07PM (UTC)
turbo-site	🔄 Building (Inspect)	Visit Preview		Feb 13, 2023 at 9:07PM (UTC)

9 Ignored Deployments

Name	Status	Preview	Updated
examples-basic-web	⬜️ Ignored (Inspect)	Visit Preview	Feb 13, 2023 at 9:07PM (UTC)
examples-cra-web	⬜️ Ignored (Inspect)	Visit Preview	Feb 13, 2023 at 9:07PM (UTC)
examples-designsystem-docs	⬜️ Ignored (Inspect)	Visit Preview	Feb 13, 2023 at 9:07PM (UTC)
examples-kitchensink-blog	⬜️ Ignored (Inspect)	Visit Preview	Feb 13, 2023 at 9:07PM (UTC)
examples-native-web	⬜️ Ignored (Inspect)	Visit Preview	Feb 13, 2023 at 9:07PM (UTC)
examples-nonmonorepo	⬜️ Ignored (Inspect)	Visit Preview	Feb 13, 2023 at 9:07PM (UTC)
examples-tailwind-web	⬜️ Ignored (Inspect)	Visit Preview	Feb 13, 2023 at 9:07PM (UTC)
examples-vite-web	⬜️ Ignored (Inspect)	Visit Preview	Feb 13, 2023 at 9:07PM (UTC)
turbo-vite-web	⬜️ Ignored (Inspect)		Feb 13, 2023 at 9:07PM (UTC)

github-actions · 2022-11-15T03:14:14Z

🟢 CI successful 🟢

Thanks

nathanhammond

Integration tests could, in many cases, remove possible future failures by using a bit of jq.

(I know it's a different code path under the hood, so I don't know if that was intentional, thus a comment.)

nathanhammond · 2023-02-10T10:10:46Z

cli/integration_tests/composable_config/composing-add-keys.t

+# 3. Change input file and assert cache miss
+  $ echo "more text" >> $TARGET_DIR/apps/add-keys/src/foo.txt
+  $ ${TURBO} run add-keys-task --filter=add-keys
+  \xe2\x80\xa2 Packages in scope: add-keys (esc)
+  \xe2\x80\xa2 Running add-keys-task in 1 packages (esc)
+  \xe2\x80\xa2 Remote caching disabled (esc)
+  add-keys:add-keys-underlying-task: cache miss, executing b7390d43fae5a180
+  add-keys:add-keys-underlying-task: 
+  add-keys:add-keys-underlying-task: > add-keys-underlying-task
+  add-keys:add-keys-underlying-task: > echo "running add-keys-underlying-task"
+  add-keys:add-keys-underlying-task: 
+  add-keys:add-keys-underlying-task: running add-keys-underlying-task
+  add-keys:add-keys-task: cache miss, executing b256f31e3957fee3
+  add-keys:add-keys-task: 
+  add-keys:add-keys-task: > add-keys-task
+  add-keys:add-keys-task: > echo "running add-keys-task" > out/foo.min.txt
+  add-keys:add-keys-task: 
+
+   Tasks:    2 successful, 2 total
+  Cached:    0 cached, 2 total
+    Time:\s*[\.0-9]+m?s  (re)
+
+# 4. Set env var and assert cache miss
+  $ SOME_VAR=somevalue ${TURBO} run add-keys-task --filter=add-keys
+  \xe2\x80\xa2 Packages in scope: add-keys (esc)
+  \xe2\x80\xa2 Running add-keys-task in 1 packages (esc)
+  \xe2\x80\xa2 Remote caching disabled (esc)
+  add-keys:add-keys-underlying-task: cache hit, replaying output b7390d43fae5a180
+  add-keys:add-keys-underlying-task: 
+  add-keys:add-keys-underlying-task: > add-keys-underlying-task
+  add-keys:add-keys-underlying-task: > echo "running add-keys-underlying-task"
+  add-keys:add-keys-underlying-task: 
+  add-keys:add-keys-underlying-task: running add-keys-underlying-task
+  add-keys:add-keys-task: cache miss, executing 8e76bee49319e00e
+  add-keys:add-keys-task: 
+  add-keys:add-keys-task: > add-keys-task
+  add-keys:add-keys-task: > echo "running add-keys-task" > out/foo.min.txt
+  add-keys:add-keys-task: 
+
+   Tasks:    2 successful, 2 total
+  Cached:    1 cached, 2 total
+    Time:\s*[\.0-9]+m?s  (re)
+


What if we used dry-run to reduce visual complexity for non-output-related tests? We're testing the hashes here, not the log messages.

Suggested change

# 3. Change input file and assert cache miss

$ echo "more text" >> $TARGET_DIR/apps/add-keys/src/foo.txt

$ ${TURBO} run add-keys-task --filter=add-keys

\xe2\x80\xa2 Packages in scope: add-keys (esc)

\xe2\x80\xa2 Running add-keys-task in 1 packages (esc)

\xe2\x80\xa2 Remote caching disabled (esc)

add-keys:add-keys-underlying-task: cache miss, executing b7390d43fae5a180

add-keys:add-keys-underlying-task:

add-keys:add-keys-underlying-task: > add-keys-underlying-task

add-keys:add-keys-underlying-task: > echo "running add-keys-underlying-task"

add-keys:add-keys-underlying-task:

add-keys:add-keys-underlying-task: running add-keys-underlying-task

add-keys:add-keys-task: cache miss, executing b256f31e3957fee3

add-keys:add-keys-task:

add-keys:add-keys-task: > add-keys-task

add-keys:add-keys-task: > echo "running add-keys-task" > out/foo.min.txt

add-keys:add-keys-task:

Tasks: 2 successful, 2 total

Cached: 0 cached, 2 total

Time:\s*[\.0-9]+m?s (re)

# 4. Set env var and assert cache miss

$ SOME_VAR=somevalue ${TURBO} run add-keys-task --filter=add-keys

\xe2\x80\xa2 Packages in scope: add-keys (esc)

\xe2\x80\xa2 Running add-keys-task in 1 packages (esc)

\xe2\x80\xa2 Remote caching disabled (esc)

add-keys:add-keys-underlying-task: cache hit, replaying output b7390d43fae5a180

add-keys:add-keys-underlying-task:

add-keys:add-keys-underlying-task: > add-keys-underlying-task

add-keys:add-keys-underlying-task: > echo "running add-keys-underlying-task"

add-keys:add-keys-underlying-task:

add-keys:add-keys-underlying-task: running add-keys-underlying-task

add-keys:add-keys-task: cache miss, executing 8e76bee49319e00e

add-keys:add-keys-task:

add-keys:add-keys-task: > add-keys-task

add-keys:add-keys-task: > echo "running add-keys-task" > out/foo.min.txt

add-keys:add-keys-task:

Tasks: 2 successful, 2 total

Cached: 1 cached, 2 total

Time:\s*[\.0-9]+m?s (re)

# 3. Change input file and assert cache miss

$ echo "more text" >> $TARGET_DIR/apps/add-keys/src/foo.txt

$ ${TURBO} run add-keys-task --filter=add-keys --dry=json | jq -r ".tasks[].hash"

b7390d43fae5a180

b256f31e3957fee3

# 4. Set env var and assert cache miss

$ SOME_VAR=somevalue ${TURBO} run add-keys-task --filter=add-keys --dry=json | jq -r ".tasks[].hash"

b7390d43fae5a180

8e76bee49319e00e

nathanhammond · 2023-02-10T10:17:03Z

cli/integration_tests/composable_config/composing-add-tasks.t

+  $ ${TURBO} run added-task --filter=add-tasks
+  \xe2\x80\xa2 Packages in scope: add-tasks (esc)
+  \xe2\x80\xa2 Running added-task in 1 packages (esc)
+  \xe2\x80\xa2 Remote caching disabled (esc)
+  add-tasks:added-task: cache miss, executing 49e39f2b39207531
+  add-tasks:added-task: 
+  add-tasks:added-task: > added-task
+  add-tasks:added-task: > echo "running added-task" > out/foo.min.txt
+  add-tasks:added-task: 
+
+   Tasks:    1 successful, 1 total
+  Cached:    0 cached, 1 total
+    Time:\s+[.0-9]+m?s  (re)
+


One bonus example:

Suggested change

$ ${TURBO} run added-task --filter=add-tasks

\xe2\x80\xa2 Packages in scope: add-tasks (esc)

\xe2\x80\xa2 Running added-task in 1 packages (esc)

\xe2\x80\xa2 Remote caching disabled (esc)

add-tasks:added-task: cache miss, executing 49e39f2b39207531

add-tasks:added-task:

add-tasks:added-task: > added-task

add-tasks:added-task: > echo "running added-task" > out/foo.min.txt

add-tasks:added-task:

Tasks: 1 successful, 1 total

Cached: 0 cached, 1 total

Time:\s+[.0-9]+m?s (re)

$ ${TURBO} run added-task --filter=add-tasks --dry=json | jq -r ".tasks[].hash"

49e39f2b39207531

nathanhammond · 2023-02-10T10:25:16Z

cli/integration_tests/composable_config/composing-cache.t

+  $ HASH=$(cat tmp.log | grep -E "cached:cached-task-1.* executing .*" | awk '{print $5}')
+  $ echo $HASH
+  [a-z0-9]{16} (re)
+  $ tar -tf $TARGET_DIR/node_modules/.cache/turbo/$HASH.tar.zst;


Can we use --cache-dir for these tests since many tests implicitly rely on that value?

That's a good idea, (correct me if I'm wrong), we'd still have to do this dance to get the task hash. Is there any benefit to replacing$TARGET_DIR/node_modules/.cache/turbo with a custom value?

The only reason is that it would make it possible to write a simple "check the result" helper function.

The "grab hash" thing is actually the thing I want to fix more than anything else, but we don't make that easy.

I'll keep things the way they are, since they're already written, but I'll write new tests the way you suggest if applicable. E.g. #3778.

mehulkar · 2023-02-10T19:30:01Z

Integration tests could, in many cases, remove possible future failures by using a bit of jq.

(I know it's a different code path under the hood, so I don't know if that was intentional, thus a comment.)

I'm open to this, but did it this way for a few reasons:

the code path is different as you mention
IMO we're a bit low on testing run and especially given the port, I think it would be good to have full "e2e" runs under many different scenarios.
I imagine that as we roll out this feature multiple configs will be a common case, so it makes sense to improve coverage in this purview, rather than, say integration_tests/basic_monorepo. In fact, composable_config/missing-workspace-config is essentially testing the current common scenario (no overrides).

I do agree that reading log output for the purpose of reading tests is hard though. Two things on that:

I could use --output-logs=hash-only when possible to reduce noise
Once the initial tests are written, it's not to bad to see diffs (especially in Github UI). And the cram VSCode extension makes things a lot easier to read also.

In any case, let me know what you think, I could see some of these test cases being reduced

gsoltis

I left a bunch of notes, but I think it nearly all boils down to one suggestion: at some level, either in fs or in graph, I think we should distinguish between a turbo file not being found vs some unexpected error occurring that needs to be reported to the user.

cli/internal/graph/graph.go

cli/internal/core/engine.go

Adds a generic method to validate TurboJSON structs and validate when getting the tasks.

Co-authored-by: Greg Soltis <[email protected]>

Only allow the error where workspace config is missing, since this config is not required. Throw other errors, including when the contents of the workspace config are malformed. This allows us to provide better error messaging to the end user

When `inputs` is configured for a task, we want to ensure that changes to the `turbo.json` in that workspace cause a cache miss. This is similar to how we treat the `package.json` in the workspace, and also how we treat the root `turbo.json` for the global hash.

github-actions bot added area: docs Improvements or additions to documentation area: site labels Nov 15, 2022

vercel bot deployed to Preview – turbo-site November 15, 2022 03:14 View deployment

nathanhammond mentioned this pull request Nov 15, 2022

Early exit when no files are changed. actions/labeler#456

Merged

mehulkar force-pushed the mehulkar/turbo-538-composable-turbojson branch from 7607a0f to 533d0f0 Compare November 17, 2022 00:57

mehulkar changed the base branch from main to mk/topo-graphs November 17, 2022 00:57

github-actions bot added the area: turborepo label Nov 17, 2022

vercel bot deployed to Preview – turbo-site November 17, 2022 00:59 View deployment

mehulkar changed the title ~~lets get this started~~ Composable turbo.json Nov 17, 2022

mehulkar force-pushed the mk/topo-graphs branch from ca5c1e3 to 9a71af5 Compare November 17, 2022 19:43

mehulkar force-pushed the mehulkar/turbo-538-composable-turbojson branch from c10c116 to 5cab00b Compare November 17, 2022 23:13

vercel bot deployed to Preview – turbo-site November 17, 2022 23:14 View deployment

Base automatically changed from mk/topo-graphs to main November 17, 2022 23:46

mehulkar force-pushed the mehulkar/turbo-538-composable-turbojson branch from 5cab00b to 1525aee Compare November 17, 2022 23:50

vercel bot deployed to Preview – turbo-site November 17, 2022 23:51 View deployment

mehulkar removed area: docs Improvements or additions to documentation area: site labels Jan 11, 2023

mehulkar force-pushed the mehulkar/turbo-538-composable-turbojson branch from 0333956 to 3fcffe1 Compare January 12, 2023 07:56

vercel bot deployed to Preview – turbo-site January 12, 2023 07:57 View deployment

mehulkar force-pushed the mehulkar/turbo-538-composable-turbojson branch 2 times, most recently from 47dbfaa to 79a63da Compare January 18, 2023 03:37

vercel bot deployed to Preview – turbo-site January 18, 2023 03:38 View deployment

mehulkar force-pushed the mehulkar/turbo-538-composable-turbojson branch 3 times, most recently from f0ac23c to e9fd2ad Compare January 18, 2023 07:56

vercel bot deployed to Preview – turbo-site January 18, 2023 07:57 View deployment

vercel bot deployed to Preview – examples-tailwind-web January 18, 2023 19:02 View deployment

vercel bot deployed to Preview – examples-kitchensink-blog January 18, 2023 19:02 View deployment

nathanhammond reviewed Feb 10, 2023

View reviewed changes

gsoltis suggested changes Feb 10, 2023

View reviewed changes

mehulkar and others added 6 commits February 10, 2023 16:52

Detect invalid workspace turbo.json configurations (#3749)

f44f24f

Adds a generic method to validate TurboJSON structs and validate when getting the tasks.

Update cli/internal/core/engine.go

b710774

Co-authored-by: Greg Soltis <[email protected]>

Update cli/internal/core/engine.go

ebc7653

Co-authored-by: Greg Soltis <[email protected]>

Rename test now that it covers more things

806f75f

Improve error bubbling (#3760)

ef32cf3

Only allow the error where workspace config is missing, since this config is not required. Throw other errors, including when the contents of the workspace config are malformed. This allows us to provide better error messaging to the end user

Merge branch 'main' into mehulkar/turbo-538-composable-turbojson

c9e1079

vercel bot deployed to Preview – examples-vite-web February 11, 2023 22:28 View deployment

vercel bot deployed to Preview – examples-basic-web February 11, 2023 22:28 View deployment

vercel bot deployed to Preview – examples-tailwind-web February 11, 2023 22:28 View deployment

vercel bot deployed to Preview – examples-designsystem-docs February 11, 2023 22:28 View deployment

vercel bot deployed to Preview – examples-cra-web February 11, 2023 22:28 View deployment

vercel bot deployed to Preview – examples-svelte-web February 11, 2023 22:28 View deployment

vercel bot deployed to Preview – examples-native-web February 11, 2023 22:28 View deployment

vercel bot deployed to Preview – examples-kitchensink-blog February 11, 2023 22:28 View deployment

vercel bot deployed to Preview – turbo-site February 11, 2023 22:28 View deployment

vercel bot deployed to Preview – examples-nonmonorepo February 11, 2023 22:29 View deployment

mehulkar added 3 commits February 13, 2023 09:30

Merge branch 'main' into mehulkar/turbo-538-composable-turbojson

7a1602c

Merge branch 'main' into mehulkar/turbo-538-composable-turbojson

aa1ad2a

mehulkar mentioned this pull request Feb 13, 2023

Include workspace turbo.json in inputs #3778

Merged

gsoltis approved these changes Feb 13, 2023

View reviewed changes

Merge branch 'main' into mehulkar/turbo-538-composable-turbojson

fdf37fe

mehulkar merged commit 5b44c92 into main Feb 13, 2023

mehulkar deleted the mehulkar/turbo-538-composable-turbojson branch February 13, 2023 21:34

This was referenced Feb 13, 2023

fix: support prune without turbo.json #3785

Merged

Update $schema for turbo.json. #3789

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Composable turbo.json #2706

Composable turbo.json #2706

mehulkar commented Nov 15, 2022 •

edited

Loading

vercel bot commented Nov 15, 2022 •

edited

Loading

github-actions bot commented Nov 15, 2022 •

edited

Loading

nathanhammond left a comment

nathanhammond Feb 10, 2023

nathanhammond Feb 10, 2023

nathanhammond Feb 10, 2023

mehulkar Feb 10, 2023

nathanhammond Feb 13, 2023

mehulkar Feb 13, 2023

mehulkar commented Feb 10, 2023

gsoltis left a comment

Composable turbo.json #2706

Composable turbo.json #2706

Conversation

mehulkar commented Nov 15, 2022 • edited Loading

vercel bot commented Nov 15, 2022 • edited Loading

github-actions bot commented Nov 15, 2022 • edited Loading

🟢 CI successful 🟢

nathanhammond left a comment

Choose a reason for hiding this comment

nathanhammond Feb 10, 2023

Choose a reason for hiding this comment

nathanhammond Feb 10, 2023

Choose a reason for hiding this comment

nathanhammond Feb 10, 2023

Choose a reason for hiding this comment

mehulkar Feb 10, 2023

Choose a reason for hiding this comment

nathanhammond Feb 13, 2023

Choose a reason for hiding this comment

mehulkar Feb 13, 2023

Choose a reason for hiding this comment

mehulkar commented Feb 10, 2023

gsoltis left a comment

Choose a reason for hiding this comment

mehulkar commented Nov 15, 2022 •

edited

Loading

vercel bot commented Nov 15, 2022 •

edited

Loading

github-actions bot commented Nov 15, 2022 •

edited

Loading