Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Phased Builds DIFF (do not merge) #1

Open
wants to merge 24 commits into
base: main
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
24 commits
Select commit Hold shift + click to select a range
c86fcaa
phased only: edits to command-line.json
elliot-nelson Sep 26, 2021
25aa052
phased only: edits to rush-project.json
elliot-nelson Sep 26, 2021
c78b0df
merge main
elliot-nelson Sep 28, 2021
c3465e1
Merge branch 'main' into phased
elliot-nelson Sep 29, 2021
d24d5df
update names
elliot-nelson Sep 29, 2021
e676596
add proposed syntax for skipping phases
elliot-nelson Sep 29, 2021
bbfbaa7
fix rush-project.json format
elliot-nelson Sep 29, 2021
50787d0
Some phased command-related tweaks
iclanton Oct 3, 2021
86dc71e
Merge pull request #2 from iclanton/ianc/phased-updates
elliot-nelson Oct 3, 2021
eb5f694
Updates to reflect schema changes.
iclanton Oct 14, 2021
94c73de
Merge pull request #3 from iclanton/ianc/update-schema
elliot-nelson Oct 14, 2021
019912b
Move incremental back to the commands.
iclanton Oct 14, 2021
5ce910c
Merge pull request #4 from iclanton/ianc/update-schema
elliot-nelson Oct 14, 2021
73ac083
Add a missint .eslintrc.js
iclanton Dec 25, 2021
035e877
Update the build cache entry name pattern.
iclanton Dec 25, 2021
328c3dd
Update the experiment name.
iclanton Dec 25, 2021
b621ad5
Update experiment name.
iclanton Dec 30, 2021
c6f3e30
Merge pull request #5 from iclanton/phased-fixes
elliot-nelson Jan 3, 2022
6fcf66e
Remove the explicit "rebuild" command.
iclanton Jan 3, 2022
875b76b
Merge pull request #6 from iclanton/remove-rebuild
elliot-nelson Jan 4, 2022
0bc45e1
Update rush-project.json schema.
iclanton Jan 15, 2022
b593536
Merge pull request #7 from iclanton/update-rush-project.json
elliot-nelson Jan 16, 2022
fa281fb
Comment out parameter with addPhasesToCommand and skipPhasesForComman…
iclanton Jan 18, 2022
d5c9afb
Merge pull request #8 from iclanton/remove-phase-control-options
elliot-nelson Jan 19, 2022
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 7 additions & 0 deletions apps/acme-dashboard/.eslintrc.js
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
// This is a workaround for https://github.com/eslint/eslint/issues/3458
require('@rushstack/eslint-config/patch/modern-module-resolution');

module.exports = {
extends: [ '@rushstack/eslint-config/profile/web-app' ],
parserOptions: { tsconfigRootDir: __dirname }
};
2 changes: 1 addition & 1 deletion common/config/rush/build-cache.json
Original file line number Diff line number Diff line change
Expand Up @@ -23,7 +23,7 @@
* Setting this property overrides the cache entry ID. If this property is set, it must contain
* a [hash] token. It may also contain a [projectName] or a [projectName:normalize] token.
*/
// "cacheEntryNamePattern": "[projectName:normalize]-[hash]"
"cacheEntryNamePattern": "[projectName:normalize]-[phaseName:normalize]-[hash]",

/**
* Use this configuration with "cacheProvider"="azure-blob-storage"
Expand Down
108 changes: 97 additions & 11 deletions common/config/rush/command-line.json
Original file line number Diff line number Diff line change
Expand Up @@ -12,15 +12,38 @@
*/
"commands": [
{
"commandKind": "bulk",
"commandKind": "phased",
"name": "build",
"summary": "Phased build",
"description": "Ditto",
"safeForSimultaneousRushProcesses": false,

"enableParallelism": true,
"incremental": true,
"phases": [
"_phase:compile",
"_phase:lint",
"_phase:test",
"_phase:update-readme",
"_phase:push-notes"
]
},
{
// Interesting question here... for a command like "rush update-readme", which already did have
// a broken out option before, should we just call "rushx update-readme" (standard bulk), or
// should we call to an individual phase (_phase:update-readme) instead?
//
// Ultimately both would be allowed, so in this case I've shown what switching it to phased
// would look like.
"commandKind": "phased",
"name": "update-readme",
"summary": "Update readmes for all projects",
"description": "Ditto",
"safeForSimultaneousRushProcesses": false,

"enableParallelism": true,
"ignoreDependencyOrder": true,
"ignoreMissingScript": true,
"incremental": true
"incremental": true,
"phases": ["_phase:update-readme"]
}
// {
// /**
Expand Down Expand Up @@ -179,24 +202,87 @@
// }
],

/* PHASES */
"phases": [

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How are these phases mapped to Rush actions? I think we're missing the definition in "commands".

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If this is a realistic example, we should show the intended Rush shell commands for realistic everyday tasks:

  • How do I build everything including all phases?
  • How do I update the README and do nothing else?
  • How do I do a --lite build where we skip unit tests and linting?
  • For these things, what's the equivalent of rush rebuild?

Maybe the answer is that initially each of these problems requires a completely new command action name. (rush build-lite?) But if so we can still enumerate them to get a sense of how painful that feature cut will be in practice.

Copy link
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These are great questions.

Since presumably the monorepo had this --lite need before phases, I implemented a solution in main (a custom build script that lets it properly handle --lite from the rushx command line).

That --lite flag is rough for the phased approach though. As an example, I imagined that the user wanted build.js to pass --lite both to Heft and to the push_notes.js custom script. If I invent a new phased command build-lite, then it can have phases ['_phase:compile', '_phase:readme_updater'] etc., and you don't need to pass --lite to Heft because you've baked that into the phase. But that's not generically possible if you were doing something with the --lite flag.

I guess one possible approach is to implement lite phases, push-notes and push-notes-lite, and only include the appropriate one. But they'll never share build cache, and would have to be leaf nodes (no dependents), otherwise how would a dependent phase know which one to rely on.

{
"name": "_phase:compile",
"dependencies": {
"upstream": ["_phase:compile"]
},
"ignoreMissingScript": true,
"allowWarningsOnSuccess": false
},
{
"name": "_phase:lint",
"dependencies": {
"self": ["_phase:compile"]
},
"ignoreMissingScript": true,
"allowWarningsOnSuccess": false
},
{
"name": "_phase:test",
"dependencies": {
"self": ["_phase:compile"]
},
"ignoreMissingScript": true,
"allowWarningsOnSuccess": false
},
{
"name": "_phase:update-readme",
"dependencies": {},
"ignoreMissingScript": true,
"allowWarningsOnSuccess": false
},
{
"name": "_phase:push-notes",
"dependencies": {
"upstream": ["_phase:update-readme"],
"self": ["_phase:compile", "_phase:update-readme"]
},
"ignoreMissingScript": true,
"allowWarningsOnSuccess": false
}
],

/**
* Custom "parameters" introduce new parameters for specified Rush command-line commands.
* For example, you might define a "--production" parameter for the "rush build" command.
*/
"parameters": [
{
"parameterKind": "flag",
"longName": "--lite",
"description": "Build without lint and unit tests",
// {
// "parameterKind": "flag",
// "longName": "--lite",
// "description": "Build without lint and unit tests",

"associatedCommands": ["build", "rebuild"]
},
// "associatedCommands": ["build", "rebuild"],

// // Note we don't pass --lite to any phases (including the :compile phase), because
// // the compile phase has a baked in --lite argument already.
// "associatedPhases": [],

// // // Here's a proposed, but optional, addition to parameter behavior: in addition to the
// // // "associatedPhases" property (which controls whether a parameter will be PASSED to
// // // a phase command), we could support "addPhasesToCommand" and "skipPhasesForCommand".
// // //
// // // The phase list for a phased command is already an unordered list, so adding and
// // // removing from that list is pretty trivial. This addition would allow us to retain
// // // the original meaning of "--lite" from the main branch.
// // "addPhasesToCommand": [],
// // "skipPhasesForCommand": [
// // "_phase:lint",
// // "_phase:test",
// // "_phase:push-notes"
// // ]
// },
{
"parameterKind": "flag",
"longName": "--production",
"description": "A custom flag parameter that is passed to the scripts that are invoked when building projects",

"associatedCommands": ["build"]
"associatedCommands": ["build"],

"associatedPhases": ["_phase:compile", "_phase:push-notes"]
}
// {
// /**
Expand Down
2 changes: 2 additions & 0 deletions common/config/rush/experiments.json
Original file line number Diff line number Diff line change
Expand Up @@ -35,4 +35,6 @@
* This will not replay warnings from the cached build.
*/
// "buildCacheWithAllowWarningsInSuccessfulBuild": true

"phasedCommands": true
}
29 changes: 28 additions & 1 deletion rigs/acme-classic-rig/profiles/library/config/rush-project.json
Original file line number Diff line number Diff line change
@@ -1,3 +1,30 @@
{
"projectOutputFolderNames": ["lib", "lib-commonjs", "dist", "temp"]
// Structure proposed by @octogonz for phase options
//
// Folders like "temp/coverage" are (probably) already folders written to by
// your unit tests, while folders like "temp/lint" are possible places we could
// have phases write their log outputs to (since the phase is likely not to have
// any other tangible output files to cache).
"operationSettings": [
{
"operationName": "_phase:compile",
"outputFolderNames": ["lib", "lib-commonjs", "dist", "temp/compile"]
},
{
"operationName": "_phase:lint",
"outputFolderNames": ["temp/lint"]
},
{
"operationName": "_phase:test",
"outputFolderNames": ["temp/test", "temp/jest-reports", "temp/coverage"]
},
{
"operationName": "_phase:update-readme",
"outputFolderNames": ["temp/update-readme"]
},
{
"operationName": "_phase:push-notes",
"outputFolderNames": ["temp/push-notes"]
}
]
}
29 changes: 28 additions & 1 deletion rigs/acme-classic-rig/profiles/web-app/config/rush-project.json
Original file line number Diff line number Diff line change
@@ -1,3 +1,30 @@
{
"projectOutputFolderNames": ["lib", "lib-commonjs", "dist", "temp"]
// Structure proposed by @octogonz for phase options
//
// Folders like "temp/coverage" are (probably) already folders written to by
// your unit tests, while folders like "temp/lint" are possible places we could
// have phases write their log outputs to (since the phase is likely not to have
// any other tangible output files to cache).
"operationSettings": [
{
"operationName": "_phase:compile",
"outputFolderNames": ["lib", "lib-commonjs", "dist", "temp/compile"]
},
{
"operationName": "_phase:lint",
"outputFolderNames": ["temp/lint"]
},
{
"operationName": "_phase:test",
"outputFolderNames": ["temp/test", "temp/jest-reports", "temp/coverage"]
},
{
"operationName": "_phase:update-readme",
"outputFolderNames": ["temp/update-readme"]
},
{
"operationName": "_phase:push-notes",
"outputFolderNames": ["temp/push-notes"]
}
]
}