Skip to content

fix: sdk release documentation is not correct since agent-js-monorepo split #1036

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
mergify merged 9 commits into from
Sep 18, 2020
Merged

fix: sdk release documentation is not correct since agent-js-monorepo split #1036

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
9 commits
Select commit Hold shift + click to select a range
f0340b0
rm src/{agent,bootstrap}
gobengo Sep 17, 2020
fa4a9f3
fix: nix -A agent-js would not have dotfiles at root
gobengo Sep 17, 2020
ac0d828
docs: update docs/process/release.adoc to account for agent-js-monore…
gobengo Sep 17, 2020
5cdf960
rm release.adoc comment lines
gobengo Sep 17, 2020
e614ed3
get rid of my shell prompt character
gobengo Sep 17, 2020
5d5ab3a
Merge branch 'master' into issues/1035
Sep 18, 2020
ed7b857
Update docs/process/release.adoc
Sep 18, 2020
0026874
Update docs/process/release.adoc
Sep 18, 2020
9c7dca8
Update docs/process/release.adoc
Sep 18, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
212 changes: 102 additions & 110 deletions docs/process/release.adoc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -73,86 +73,82 @@ git checkout master && git pull

Verify that you can build DFX and the JavaScript agent from the `+master+` branch without errors.

. Build the `dfx` binary by running the following command:
. Store a path to your latest system `dfx` as a variable. This step requires you to have the latest `dfx` binary installed.
+
[source, bash]
----
nix-build ./dfx.nix -A build --option extra-binary-caches https://nix.dfinity.systems
export dfx_system=$(which dfx);

! test -z "$dfx_system" \
&& echo "Please Proceed" \
|| echo 'ERROR: `dfx` not found'
----
+
This command prints logs to the console.
The last line printed is the build output directory.
+
For example, you should see a path similar to this:
+
`/nix/store/7qsrb3pikzam301h6khjzwpg35ry7miz-dfinity-sdk-rust-unknown`
. Copy the `/nix/store/...-dfinity-sdk-rust-unknown` path displayed in the console.
// (_for the rest of these instructions, assume that `dfx` is a reference to
// `result/bin/dfx or wherever the binary was built with this command`_)
. Create a temporary `alias` for the binary to use in successive steps by appending `/bin/dfx` to the `/nix/store/...-dfinity-sdk-rust-unknown` path.
+
For example, add `/bin/dfx` to the `/nix/store/...-dfinity-sdk-rust-unknown` path to create an alias by running a command similar to the following:
You should see "Please Proceed"
+
[source, bash]
----
alias dfx=/nix/store/7qsrb3pikzam301h6khjzwpg35ry7miz-dfinity-sdk-rust-unknown/bin/dfx
----
. Verify the binary version by running the following command:
. Change directory to the sdk repository
+
[source, bash]
----
dfx --version
export dfx_repository="$HOME/dfinity/sdk" # may be different for you
cd $dfx_repository
----
. Delete the existing `dfx` cache to ensure you're not using a stale binary.
+
[source, bash]
----
dfx cache delete
----
. Build the JavaScript agent by running the following command:
. Build the `dfx` binary by running the following command:
Copy link
Contributor

@lsgunnlsgunn lsgunnlsgunn Sep 18, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How can step at line 76 happen before this step? Just as text that seems confusing to me, but as I said I would have to do a real run through to see if it makes sense in context.

Copy link
Contributor Author

@gobengo gobengo Sep 18, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Great question.

Line 76 is saving a reference to the 'production release' of the sdk that I assume you already have installed on your system. That's why I call the variable $dfx_system. But later, we're going to build the binary for a 'release candidate' ('rc') for dfx, so I call that variable $dfx_rc (later).

$dfx_system is really only needed so you can compare the outputs of $dfx_system --version and $dfx_rc --version to make sure they are different in the right way in a later step. This step was already in the release, but was ambiguous to me, because it basically just said 'do $dfx_rc --version and validate the output', but not describe the expected output.

+
[source, bash]
----
nix-build . -A agent-js --option extra-binary-caches https://nix.dfinity.systems
export sdk_rc="$(nix-build ./dfx.nix -A build --option extra-binary-caches https://nix.dfinity.systems)"
export dfx_rc="$sdk_rc/bin/dfx"
----
+
This command prints logs to the console.
The last line printed is the build output directory.
This stores some shell variables. Make sure `$dfx_rc` points to a real file.
+
For example, you should see a path similar to the following:
```
test -x $dfx_rc \
&& echo 'Please proceed.' \
|| echo 'Cant find executable $dfx_rc'="$dfx_rc"
# You should see 'Please proceed'
```
+
`/nix/store/s9al333cb07lgzjl90hy3c5n9p59380p-dfinity-sdk-agent-js`
. Copy the `/nix/store/...-dfinity-sdk-agent-js` path displayed in the console.
. List the files in the `/nix/store/...-dfinity-sdk-agent-js` path by running a command similar to the following:
. Verify the binary version. The version of `$dfx_rc` should be greater than the version from `$dfx_system`.
+
[source, bash]
----
ls -l /nix/store/s9al333cb07lgzjl90hy3c5n9p59380p-dfinity-sdk-agent-js
$dfx_system --version
# example stdout: dfx 0.6.7

$dfx_rc --version
# example stdout: dfx 0.6.8
----
+
You should see output similar to the following:

. Delete the existing `dfx` cache to ensure you're not using a stale binary.
+
....
total 64
-r--r--r-- 1 username admin 30439 Dec 31 1969 dfinity-agent-0.6.0.tgz
dr-xr-xr-x 4 username admin 128 Dec 31 1969 nix-support
....
. Append the `/nix/store/` path with the `dfinity-agent-n.n.n.tgz` file name and
store the full path to the archive (`.tgz`) file in a temporary environment variable by running a command similar to the following:
[source, bash]
----
$dfx_rc cache delete
----
. Build the JavaScript agent by running the following command:
+
[source, bash]
----
export JS_AGENT_PATH=/nix/store/s9al333cb07lgzjl90hy3c5n9p59380p-dfinity-sdk-agent-js/dfinity-agent-0.6.0.tgz
export agent_js_rc="$(nix-build . -A agent-js --option extra-binary-caches https://nix.dfinity.systems)"
export agent_js_rc_npm_packed="$(sh -c 'echo "$1"' sh $agent_js_rc/dfinity-agent-*.tgz)"

test -f $agent_js_rc_npm_packed \
&& echo 'Please proceed' \
|| echo 'Cannot find npm-packed dfinity-agent-*.tgz'
# You should see 'Please proceed'
----
+
. Ensure `dfx` and `replica` are not running in the background by running the following commands:
+
[source, bash]
----
ps -ef | grep replica
ps -ef | grep dfx
ps -ef | grep -E 'replica|dfx' | grep -v grep
----
+
These commands should only list `+grep+` as a running process.
If there are any `replica` or `dfx` processes running, use the `kill` command to terminate them.

=== Validate the default project
Expand All @@ -163,39 +159,55 @@ Verify that you can build, deploy, and call the default `hello_world` project wi
+
[source, bash]
----
dfx new hello_world
$dfx_rc new hello_world
cd hello_world
----
. Install the locally-built JavaScript agent for the new project.
+
[source, bash]
----
npm install ${JS_AGENT_PATH}
npm install "$agent_js_rc_npm_packed"
----
. Start the local `replica` as a background process.
+
[source, bash]
----
dfx start --background
$dfx_rc start --background
----
. Create, build, and install canisters by running the following commands:
. Create, build, and install canisters by running:
+
[source, bash]
----
dfx canister create --all
dfx build
dfx canister install --all
$dfx_rc deploy
----
. Call the canister and verify the result.
+
[source, bash]
----
dfx canister call hello_world greet everyone
$dfx_rc canister call hello_world greet everyone
----
. Save the canister URLs
+
[source, bash]
----
export hello_world_candid_url="http://localhost:8000/candid?canisterId=$($dfx_rc canister id hello_worlds)"
export hello_world_assets_url="http://localhost:8000/?canisterId=$($dfx_rc canister id hello_world_assets)"
----
. Verify the default front-end in a browser.
.. In a new incognito or private window, navigate to the default localhost address and port `8000` as specified in the `dfx.json` configuration file.
.. Append `?canisterId=ic:<YOUR-CANISTER-IDENTIFIER>` with the canister identifier for the
`hello_world_assets` canister to the URL (for example, `http://localhost:8000/?canisterId=cxeji-wacaa-aaaaa-aaaaa-aaaaa-aaaaa-aaaaa-q`).
+
. Verify hello_world_assets in a web browser. If possible, use "Private Browsing" or "Incognito Mode".
+
To open it, run:
+
[source, bash]
----
if test -x $(which open); then
open "$hello_world_assets_url"
else
echo "Please open the following in your web browser: $hello_world_assets_url"
fi
# Either your browser will open to the hello_world_assets canister, or you will see the URL printed on stdout.
----
+
.. Verify that you are prompted to type a greeting in a prompt window.
.. Type a greeting, then click *OK* to return the greeting in an alert window.
.. Verify there are no errors in the console by opening the Developer Tools.
Expand All @@ -204,8 +216,19 @@ For example, in the browser, right-click>, then click Inspect and select Console
Warnings can be ignored.

. Verify the Candid UI.
.. In a new incognito or private window, navigate to the default localhost address and port `8000` as specified in the `dfx.json` configuration file.
.. Append the `candid` endpoint and `canisterId` for the `hello_world` canister to the URL (for example, `http://localhost:8000/candid?canisterId=75hes-oqbaa-aaaaa-aaaaa-aaaaa-aaaaa-aaaaa-q`).
+
To open it, run:
+
[source, bash]
----
if test -x $(which open); then
open "$hello_world_candid_url"
else
echo "Please open the following in your web browser: $hello_world_candid_url"
fi
# Either your browser will open to the hello_world Candid UI, or you will see the URL printed on stdout.
----
+
.. Verify UI loads, then test the greet function by entering text and clicking *Call* or clicking *Lucky*,
.. Verify there are no errors in the console by opening the Developer Tools.
+
Expand All @@ -215,7 +238,7 @@ Warnings can be ignored.
+
[source, bash]
----
dfx stop
$dfx_rc stop
----
. Delete the test project by running the following commands:
+
Expand All @@ -227,13 +250,6 @@ rm -rf hello_world

==== Update the version

. Navigate back to the top-level of the `sdk` repo.
. Enter the sdk `nix` development environment by running the following command:
+
[source, bash]
----
nix-shell --option extra-binary-caches https://nix.dfinity.systems
----
. Set the new version in a temporary environment variable.
+
For example, replace `<n.n.n>` with a specific version number:
Expand All @@ -242,48 +258,28 @@ For example, replace `<n.n.n>` with a specific version number:
----
export NEW_DFX_VERSION=<n.n.n>
----
. If you're not already there, navigate back to the top-level of the `sdk` repo.
. Enter the sdk `nix` development environment by running the following command:
+
[source, bash]
----
nix-shell --option extra-binary-caches https://nix.dfinity.systems
----
. Create a new branch for your changes by running the following commands:
+
[source, nix-shell]
----
git switch -c $USER/release-$NEW_DFX_VERSION
----
. Update the `version` field in this file:
+
....
`src/dfx/Cargo.toml`
....
. Update the first `version` field in `src/dfx/Cargo.toml` to be equal to `$NEW_DFX_VERSION`
. Apply these changes to `Cargo.lock` by running the following command:
+
[source, nix-shell]
----
cargo build
----
. Update the `@dfinity/agent` JavaScript package by running the following command:
+
[source, nix-shell]
----
cd src/agent/javascript/ && npm version $NEW_DFX_VERSION
----
. Navigate back to the root of sdk repo by running the following command:
+
[source, nix-shell]
----
cd ../../..
----
. Update the `@dfinity/bootstrap` JavaScript package by running the following command:
+
[source, nix-shell]
----
cd src/bootstrap/ && npm version $NEW_DFX_VERSION
----
. Navigate back to the root of sdk repo by running the following command:
+
[source, nix-shell]
----
cd ../..
----
. Append the new version to `public/manifest.json` by adding it to the `versions` list.

. Append the new version to `public/manifest.json` by appending it to the `versions` list.
+
For example:
+
Expand All @@ -301,7 +297,7 @@ For example:
}
----
+
*Ensure* `latest` remains the same.
*Ensure* `tags.latest` remains the same.


==== Create a pull request and tag
Expand Down Expand Up @@ -376,25 +372,21 @@ Hydra will spur into action when `stable` advances, so if the branch is already

==== Publish Javascript agent to NPM

. Change to the `javascript` agent directory and clean it by running the following commands:
. Create and switch to a new directory to release from by running the following commands:
+
[source, nix-shell]
----
cd src/agent/javascript
git clean -dfx .
pushd $(mktemp -d)
cp -R $agent_js_rc/. ./
----
. Make sure you have latest version of node modules and don't have stale files by running the following command:
+
[source, nix-shell]
----
npm install
----
. Build files by running the following command:

. Change the package.json's `version` property:
+
[source, nix-shell]
----
npm run build
npm version $NEW_DFX_VERSION
Copy link
Contributor

@p-shahi p-shahi Sep 18, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A part of me thinks, we'd want to do this as a part of the JS agent release process.

Copy link
Contributor Author

@gobengo gobengo Sep 18, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is the JS agent release process

Copy link
Contributor

@p-shahi p-shahi Sep 18, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What I'm saying is I'm not so sure we want to couple the release process of the SDK and the release process of the JS and Rust Agents. This is really an artifact of those different components being part of the same repository. Decoupling the components gives us a chance to develop each repos own release process, which will be more important once they are open sourced. For now this is fine and a non blocker but I don't imagine we want to cd agent-js && npm version $NEW_DFX_VERSION for the lifecycle of the sdk

Copy link
Contributor Author

@gobengo gobengo Sep 18, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@p-shahi You're right, and @hansl and I talked yesterday about

  • agent-js-monorepo CI will publish to npm, e.g. for @dfinity/agent and eventually other packages
  • that will unblock sdk pulling from the npm registry instead of git
  • that will unblock us from needing to release sdk-then-agent, and we will then be able to release tagged version of agent to npm, and have sdk depend on new versions as it sees fit.

So yeah, I think we're saying the same thing. This PR just makes the release docs work at all. But soon I will change them again to be more better.

Copy link
Contributor Author

@gobengo gobengo Sep 18, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

dfinity/icp-js-core#32

----

. Run the following command to check that every `.js` file has a `.d.ts` assigned and that every `.js` and `.d.ts` file has a source file that is not a test:
+
[source, nix-shell]
Expand Down
4 changes: 2 additions & 2 deletions nix/agent-js/agent-js-monorepo.nix
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,7 @@ let
installPhase = ''
# $out: Everything!
mkdir -p $out
cp -R ./* $out/
cp -R ./. $out/

# $lib/node_modules: fetched npm dependencies
mkdir -p $lib
Expand All @@ -33,7 +33,7 @@ let
# $agent: npm subpackage @dfinity/agent
mkdir -p $agent
cp -R node_modules $agent/
cp -R ./packages/agent/* $agent/
cp -R ./packages/agent/. $agent/

# $bootstrap: npm subpackage @dfinity/bootstrap
mkdir -p $bootstrap
Expand Down
2 changes: 1 addition & 1 deletion nix/agent-js/agent-js.nix
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,7 @@ pkgs.stdenv.mkDerivation {
installPhase = ''
# $out: everything
mkdir -p $out
cp -R ${pkgs.agent-js-monorepo.agent}/* $out/
cp -R ${pkgs.agent-js-monorepo.agent}/. $out/
cp -R packages/agent/dfinity-agent*.tgz $out/

# $lib/node_modules: node_modules dir that must be resolvable by npm
Expand Down
Loading