[RLlib] Use observations (input_dict) for exploration #27979

Stefan-1313 · 2022-08-18T09:26:32Z

Why are these changes needed?

Use observations (input_dict) for exploration.
For details, please see issue: #26437 .

Related issue number

If this pull request is correct(ed), it closes enhancement issue #26437.

Checks

[ I did, some afterward, I don't know if this is how you want it... ] I've signed off every commit(by using the -s flag, i.e., git commit -s) in this PR.
[ I'm on Windows, does not work ] I've run scripts/format.sh to lint the changes in this PR.
[ I think this will be auto generated, I updated the doc string in the code for the developer API. ] I've included any doc changes needed for https://docs.ray.io/en/master/.
[ Yes, except for tests that already failed before my changes. Among them tests when using framework "tf", perhaps this is due to my environment, when using frameworks "torch" and "tf2" tests pass ] I've made sure the tests are passing. Note that there might be a few flaky tests, see the recent failures at https://flakey-tests.ray.io/
Testing Strategy
- [ X ] Unit tests
- Release tests
- This PR is not tested :(
- [ X ] A test program I normally use, runs fine for both "torch" and "tf2" frameworks.

I did my best, this is my first Pull Request ever so please go easy on me :).
I'm not an very experienced Git user, so I had trouble fixing all the sign off stuff.

EDIT:

I tried to fix the sign off issue with the instructions in the DCO check (git rebase HEAD~7 --signoff and git push --force-with-lease origin master), and now I have seen to signed off everyone's commit :O. That cannot be how it should be... But the DCO check is not complaining any more! :)

Signed-off-by: Stefan van der Kleij <[email protected]>

…s` (ray-project#27284) Add optional last_activity_at field to /api/component_activities to record end time of most recently finished activity Signed-off-by: Nikita Vemuri <[email protected]> Signed-off-by: Stefan van der Kleij <[email protected]>

) Signed-off-by: Stefan van der Kleij <[email protected]>

Signed-off-by: Eric Liang <[email protected]> Signed-off-by: Stefan van der Kleij <[email protected]>

…-project#27370) We decided to allow escaping the parent pg via `PlacementGroupSchedulingStrategy(placement_group=None)` instead of using "DEFAULT". Our doc is updated with that but in the code it's still not allowed. Signed-off-by: Stefan van der Kleij <[email protected]>

…c_code dir (ray-project#27350) Edit pass. Move code into doc_code dir. Code in doc_code is verified by CI Signed-off-by: Stefan van der Kleij <[email protected]>

…_code dir (ray-project#27341) Signed-off-by: Stefan van der Kleij <[email protected]>

Co-authored-by: Kai Fricke <[email protected]> Signed-off-by: Stefan van der Kleij <[email protected]>

Signed-off-by: simon-mo <[email protected]> Signed-off-by: Stefan van der Kleij <[email protected]>

…ay-project#26969) Moves FastAPI into its own section instead of appearing in a duplicated note. Co-authored-by: simon-mo <[email protected]> Signed-off-by: Stefan van der Kleij <[email protected]>

…opy-editing Signed-off-by: Eric Liang <[email protected]> Signed-off-by: Stefan van der Kleij <[email protected]>

…t#27377) Why are these changes needed? python/tests/test_state_api.py runs for 5min in normal run Signed-off-by: Stefan van der Kleij <[email protected]>

Signed-off-by: Stefan van der Kleij <[email protected]>

…an a NumPy conversion (ray-project#26891) Converting a Pandas DataFrame column to an ndarray (e.g. via df[col].values) can often result in a full copy of the column in order to construct the ndarray due to Pandas' 2D block management. This PR ports tensor extension type checking to checking the dtype, which is always an O(1) check. Signed-off-by: Clark Zinzow <[email protected]> Signed-off-by: Stefan van der Kleij <[email protected]>

Support a GPU column for the new dashboard Have first node be default expanded Signed-off-by: Alan Guo [email protected] fixes ray-project#13889 Addresses comment from ray-project#26996 Signed-off-by: Stefan van der Kleij <[email protected]>

…ray-project#27232) Signed-off-by: Stefan van der Kleij <[email protected]>

Signed-off-by: Dmitri Gekhtman <[email protected]> This PR adds a page of guidance on GPU deployment with Ray/K8s. This page is a modified and slightly expanded version of the existing page https://docs.ray.io/en/latest/cluster/kubernetes-gpu.html moves managed K8s service intro links to their own page Signed-off-by: Stefan van der Kleij <[email protected]>

Update API references to beta. Needed as we are going to beta in 2.0. I left out RL/Scikit-Learn/HuggingFace. Signed-off-by: Stefan van der Kleij <[email protected]>

Signed-off-by: Stefan van der Kleij <[email protected]>

…icas from target group (ray-project#26685)" (ray-project#27283)" (ray-project#27348) Signed-off-by: Stefan van der Kleij <[email protected]>

- Currently not all code under ray-core/doc_code is covered by CI. - tf_example.py and torch_example.py are not used anywhere. Signed-off-by: Jiajun Yao <[email protected]> Signed-off-by: Stefan van der Kleij <[email protected]>

ray-project#27928) Signed-off-by: Stefan van der Kleij <[email protected]> Signed-off-by: Stefan van der Kleij <[email protected]>

Signed-off-by: Cheng Su <[email protected]> This is to run ray 2.0.0rc0 on https://docs.ray.io/en/master/data/examples/nyc_taxi_basic_processing.html and fix the notebook end-to-end, make sure the output and wording is matched. The page after this PR - https://ray--27927.org.readthedocs.build/en/27927/data/examples/nyc_taxi_basic_processing.html . Signed-off-by: Stefan van der Kleij <[email protected]> Signed-off-by: Stefan van der Kleij <[email protected]>

…y-project#27901) Fix 2.0.0 release blocker bug where Ray State API and Jobs not accessible if the override URL doesn't support adding additional subpaths. This PR keeps the localhost dashboard URL in the internal KV store and only overrides in values printed or returned to the user. images.githubusercontent.com/6900234/184809934-8d150874-90fe-4b45-a13d-bce1807047de.png"> Signed-off-by: Stefan van der Kleij <[email protected]> Signed-off-by: Stefan van der Kleij <[email protected]>

Signed-off-by: Stefan van der Kleij <[email protected]> Signed-off-by: Stefan van der Kleij <[email protected]>

…both PyTorch and TensorFlow. Signed-off-by: Stefan van der Kleij Signed-off-by: Stefan van der Kleij <[email protected]> Signed-off-by: Stefan van der Kleij <[email protected]>

…ring. Signed-off-by: Stefan van der Kleij Signed-off-by: Stefan van der Kleij <[email protected]> Signed-off-by: Stefan van der Kleij <[email protected]>

Signed-off-by: Stefan van der Kleij Signed-off-by: Stefan van der Kleij <[email protected]> Signed-off-by: Stefan van der Kleij <[email protected]>

stale · 2022-09-20T16:21:14Z

This pull request has been automatically marked as stale because it has not had recent activity. It will be closed in 14 days if no further activity occurs. Thank you for your contributions.

If you'd like to keep this open, just leave any comment, and the stale label will be removed.

Stefan-1313 · 2022-09-21T17:36:42Z

Commenting because of stale[bot].

This pull request is still usefull, and should still be able to merge without (much) problems.
Only a few files have changed and the sign off command I did somehow made it such that all commits that where done since I created my fork are seen as commits for this pull request.

Only look at the latest commits, only these are mine and are the relevant ones.

xwjiang2010 · 2022-09-21T22:44:02Z

@Stefan-1313
Thanks for the PR!
Could you clean up the commit history? With its current state, it is hard to get it reviewed and merged.

To fix it, i would just start a new brach, cherry-pick your PR onto that branch. Please also do git commit --amend --signoff to make sure that DCO check will pass. Then the same procedure with git push stuff (that will give you a new PR but I think it's ok in this case).

@kouroshHakha assigning this to you as you have the most context about the problem the PR is addressing.

Stefan-1313 · 2022-09-22T20:37:46Z

@xwjiang2010 I will try to clean up my commit history.
Currently I'm a bit low on time, but I will do it when I find some time.
Should I then create a new pull request? If yes, then I close this one after I created the new pull request.

xwjiang2010 · 2022-09-22T21:56:57Z

Yes, that would work. Thank you!

kouroshHakha · 2022-09-23T16:12:34Z

@Stefan-1313 Please assign the clean PR to me. Thanks.

stale · 2022-10-28T20:01:57Z

This pull request has been automatically marked as stale because it has not had recent activity. It will be closed in 14 days if no further activity occurs. Thank you for your contributions.

If you'd like to keep this open, just leave any comment, and the stale label will be removed.

stale · 2022-11-12T05:32:12Z

Hi again! The issue will be closed because there has been no more activity in the 14 days since the last message.

Please feel free to reopen or open a new issue if you'd still like it to be addressed.

Again, you can always ask for help on our discussion forum or Ray's public slack channel.

Thanks again for opening the issue!

Stefan-1313 requested review from sven1977, gjoliver, avnishn, ArturNiederfahrenhorst, smorad, maxpumperla, kouroshHakha and krfricke as code owners August 18, 2022 09:26

kouroshHakha and others added 22 commits August 18, 2022 11:35

[RLlib] Fix default view_requirement in policy.py (ray-project#27255)

b73c07c

Signed-off-by: Stefan van der Kleij <[email protected]>

[RLlib] Fix A2C release tests (ray-project#27314)

2214c87

Signed-off-by: Stefan van der Kleij <[email protected]>

[RLlib] Fix the last cartpole-crashing premerge test. (ray-project#27315

9b49b02

) Signed-off-by: Stefan van der Kleij <[email protected]>

Raise the (runtime_env max size) gRPC max message size to 500MiB

02beea1

Signed-off-by: Eric Liang <[email protected]> Signed-off-by: Stefan van der Kleij <[email protected]>

[docs] ray core cross-lang docs: edit pass & move python code into do…

6d3c63f

…c_code dir (ray-project#27350) Edit pass. Move code into doc_code dir. Code in doc_code is verified by CI Signed-off-by: Stefan van der Kleij <[email protected]>

[docs] ray core namespace docs: edit pass & move python code into doc…

faeae05

…_code dir (ray-project#27341) Signed-off-by: Stefan van der Kleij <[email protected]>

[air/train/docs] gbdt trainer user guide (ray-project#27362)

bc46872

Co-authored-by: Kai Fricke <[email protected]> Signed-off-by: Stefan van der Kleij <[email protected]>

[Serve] Remove the warning for async handles in 2.0 (ray-project#27346)

dc34dca

Signed-off-by: simon-mo <[email protected]> Signed-off-by: Stefan van der Kleij <[email protected]>

[Doc] [Serve] Edit grammar/usage/organization for HTTP adapters page (r…

7a99774

…ay-project#26969) Moves FastAPI into its own section instead of appearing in a duplicated note. Co-authored-by: simon-mo <[email protected]> Signed-off-by: Stefan van der Kleij <[email protected]>

[docs] Adjust the set of global doc owners to those responsible for c…

abeb66f

…opy-editing Signed-off-by: Eric Liang <[email protected]> Signed-off-by: Stefan van der Kleij <[email protected]>

[Core] Move test_state_api test back to large test groups (ray-projec…

4795720

…t#27377) Why are these changes needed? python/tests/test_state_api.py runs for 5min in normal run Signed-off-by: Stefan van der Kleij <[email protected]>

[RLlib] Training step docs. (ray-project#27344)

399628e

Signed-off-by: Stefan van der Kleij <[email protected]>

[air] Fix BatchPredictor.predict_pipelined not working with GPU stage (…

a5c7f74

…ray-project#27232) Signed-off-by: Stefan van der Kleij <[email protected]>

[air] Update to beta (ray-project#27393)

4cb9ff1

Update API references to beta. Needed as we are going to beta in 2.0. I left out RL/Scikit-Learn/HuggingFace. Signed-off-by: Stefan van der Kleij <[email protected]>

[RLlib] Add OPE Learning Tests (ray-project#27154)

7d5c7dd

Signed-off-by: Stefan van der Kleij <[email protected]>

Revert "Revert "[Serve] ServeHandle detects ActorError and drop repl…

352b9e0

…icas from target group (ray-project#26685)" (ray-project#27283)" (ray-project#27348) Signed-off-by: Stefan van der Kleij <[email protected]>

[Doc] Test ray core doc code (ray-project#27334)

e1ac191

- Currently not all code under ray-core/doc_code is covered by CI. - tf_example.py and torch_example.py are not used anywhere. Signed-off-by: Jiajun Yao <[email protected]> Signed-off-by: Stefan van der Kleij <[email protected]>

Stefan-1313 requested review from AmeerHajAli, ckw017, kfstorm, jovany-wang, WangTaoTheTonic, alanwguo and robertnishihara as code owners August 18, 2022 09:36

Stefan-1313 force-pushed the master branch 2 times, most recently from 2e2e5ca to 986bb47 Compare August 18, 2022 09:40

christy and others added 7 commits August 18, 2022 12:56

Replace robot image with emoji and replace word Trainer with Algorithm (

1a055ce

ray-project#27928) Signed-off-by: Stefan van der Kleij <[email protected]> Signed-off-by: Stefan van der Kleij <[email protected]>

[RLlib] Add DTTorchPolicy (ray-project#27889)

68a849a

Signed-off-by: Stefan van der Kleij <[email protected]> Signed-off-by: Stefan van der Kleij <[email protected]>

Added possibility to use observation information while exploring for …

4270d00

…both PyTorch and TensorFlow. Signed-off-by: Stefan van der Kleij Signed-off-by: Stefan van der Kleij <[email protected]> Signed-off-by: Stefan van der Kleij <[email protected]>

Also updated the policy files for using state information while explo…

deafc6d

…ring. Signed-off-by: Stefan van der Kleij Signed-off-by: Stefan van der Kleij <[email protected]> Signed-off-by: Stefan van der Kleij <[email protected]>

Included documentation for changes to API.

6e5f365

Signed-off-by: Stefan van der Kleij Signed-off-by: Stefan van der Kleij <[email protected]> Signed-off-by: Stefan van der Kleij <[email protected]>

Stefan-1313 force-pushed the master branch from 986bb47 to 6e5f365 Compare August 18, 2022 10:56

stale bot added the stale The issue is stale. It will be closed within 7 days unless there are further conversation label Sep 20, 2022

stale bot removed the stale The issue is stale. It will be closed within 7 days unless there are further conversation label Sep 21, 2022

xwjiang2010 assigned kouroshHakha Sep 21, 2022

stale bot added the stale The issue is stale. It will be closed within 7 days unless there are further conversation label Oct 28, 2022

stale bot closed this Nov 12, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[RLlib] Use observations (input_dict) for exploration #27979

[RLlib] Use observations (input_dict) for exploration #27979

Stefan-1313 commented Aug 18, 2022 •

edited

Loading

stale bot commented Sep 20, 2022

Stefan-1313 commented Sep 21, 2022

xwjiang2010 commented Sep 21, 2022

Stefan-1313 commented Sep 22, 2022

xwjiang2010 commented Sep 22, 2022

kouroshHakha commented Sep 23, 2022

stale bot commented Oct 28, 2022

stale bot commented Nov 12, 2022

[RLlib] Use observations (input_dict) for exploration #27979

[RLlib] Use observations (input_dict) for exploration #27979

Conversation

Stefan-1313 commented Aug 18, 2022 • edited Loading

Why are these changes needed?

Related issue number

Checks

EDIT:

stale bot commented Sep 20, 2022

Stefan-1313 commented Sep 21, 2022

xwjiang2010 commented Sep 21, 2022

Stefan-1313 commented Sep 22, 2022

xwjiang2010 commented Sep 22, 2022

kouroshHakha commented Sep 23, 2022

stale bot commented Oct 28, 2022

stale bot commented Nov 12, 2022

Stefan-1313 commented Aug 18, 2022 •

edited

Loading