Orca: fix np 1.24 type deprecations #7591
Conversation
| @@ -45,7 +45,8 @@ python -m pytest -v test/bigdl/orca/learn/ray \ | |||
| --ignore=test/bigdl/orca/learn/ray/pytorch/test_estimator_horovod_backend.py \ | |||
| --ignore=test/bigdl/orca/learn/ray/pytorch/test_estimator_ray_runtime.py \ | |||
| --ignore=test/bigdl/orca/learn/ray/pytorch/test_estimator_ray_dataset.py \ | |||
| --ignore=test/bigdl/orca/learn/ray/tf/ | |||
| --ignore=test/bigdl/orca/learn/ray/tf/ \ | |||
| --ignore=test/bigdl/orca/learn/ray/mxnet/ | |||
There was a problem hiding this comment.
WIP: need to add mxnet elsewhere
There was a problem hiding this comment.
Refer to other jobs to add if condition for py37; we can only run mxnet for py37
| @@ -94,7 +94,7 @@ def group_index(iter): | |||
| def transform_predict(predictions): | |||
| # list of np array | |||
| if isinstance(predictions[0], list): | |||
| predictions = np.array(predictions).T.tolist() | |||
| predictions = np.array(predictions, dtype=object).T.tolist() | |||
There was a problem hiding this comment.
np 1.24 requires array to have homogenous shape
There was a problem hiding this comment.
not correct to be here; we can't treat everything to be object.
There was a problem hiding this comment.
python 3.7 test already gives the warning:
/home/yinchen/BigDL/python/orca/src/bigdl/orca/learn/utils.py:97: VisibleDeprecationWarning: Creating an ndarray from ragged nested sequences (which is a list-or-tuple of lists-or-tuples-or ndarrays with different lengths or shapes) is deprecated. If you meant to do this, you must specify 'dtype=object' when creating the ndarray.
predictions = np.array(predictions).T.tolist()Command
cd python/orca
python -m pytest --show-capture=all --full-trace -r P -v test/bigdl/orca/learn/test_utils.py::TestUtil::test_convert_predict_rdd_to_xshard_multi_outputThere was a problem hiding this comment.
@sgwhat We can fix this issue by avoiding directly setting all predictions to np?
predictions = np.array(predictions).T.tolist()
hamham223
changed the title
| @@ -94,7 +94,7 @@ def group_index(iter): | |||
| def transform_predict(predictions): | |||
| # list of np array | |||
| if isinstance(predictions[0], list): | |||
| predictions = np.array(predictions).T.tolist() | |||
| predictions = [[item[i] for item in predictions] for i in range(len(predictions[0]))] | |||
There was a problem hiding this comment.
will predictions return as shape [[1, 2, 3], [4, 5, 6], [7, 8]] ?
There was a problem hiding this comment.
Before running Line 97:
predictions = [[[1, 1], [1, 1, 1]], [[2, 2], [2, 2, 2]]]
After Line 97:
predictions = [[[1, 1], [2, 2]], [[1, 1, 1], [2, 2, 2]]]
Test scripts: https://github.com/intel-analytics/BigDL/blob/main/python/orca/test/bigdl/orca/learn/test_utils.py#L97
|
Merge this first, one ray ctx job is irrelevant. |
Description
1. Why the change?
See #7548
3. Summary of the change
fix numpy deprecations:
np.int->np.int32np.float->np.float32np.str->str4. How to test?
The following Orca unit test: