Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[RLlib] "TypeError: 'int' object is not iterable when using from_jsonable with nested Discrete in Dict" #50131

Open
junyeop opened this issue Jan 29, 2025 · 0 comments
Labels
bug Something that is supposed to be working; but isn't P1 Issue that should be fixed within a few weeks rllib RLlib related issues rllib-newstack rllib-offline-rl Offline RL problems

Comments

@junyeop
Copy link

junyeop commented Jan 29, 2025

What happened + What you expected to happen

  1. When calling the _map_to_episodes function in offline_prelearner.py, an issue arises during the creation of SingleAgentEpisode. The problem occurs in the from_jsonable_if_needed function when providing observation_space and action_space. It checks with is_composite_space and then uses the from_jsonable method from gymnasium.spaces, which expects sample_n to be of type dict[str, list[Any]]. However, when combining a Dict (composite space) with a Discrete (primitive space), the resulting structure does not conform to the expected list[Any] format, leading to an error.

  2. Offline learning should be possible when providing observation_space and action_space. The implementation must work correctly even when combining composite spaces and primitive spaces (Dict + Discrete).

# ray/rllib/offline/offline_prelearner.py
def _map_to_episodes(
    is_multi_agent: bool,
    batch: Dict[str, Union[list, np.ndarray]],
    schema: Dict[str, str] = SCHEMA,
    to_numpy: bool = False,
    input_compress_columns: Optional[List[str]] = None,
    observation_space: gym.Space = None,
    action_space: gym.Space = None,
    **kwargs: Dict[str, Any],
) -> Dict[str, List[EpisodeType]]:
    input_compress_columns = input_compress_columns or []

    if observation_space and action_space:
        convert = from_jsonable_if_needed  # 'int' object is not iterable error (Dict + Discrete)
    else:
        # ... existing code ...

    episodes = []
    for i, obs in enumerate(batch[schema[Columns.OBS]]):
        # ... existing code ...
        
        episode = SingleAgentEpisode(
            id_=str(batch[schema[Columns.EPS_ID]][i]),
            # ... existing code ...
            actions=[
                convert(
                    unpack_if_needed(batch[schema[Columns.ACTIONS]][i]),
                    action_space,
                )
                if Columns.ACTIONS in input_compress_columns
                else convert(batch[schema[Columns.ACTIONS]][i], action_space)
            ],
            # ... existing code ...
        )

        # ... existing code ...

    return {"episodes": episodes}
# ray/rllib/utils/spaces/space_utils.py
def is_composite_space(space: gym.Space) -> bool:
    if type(space) in [
        gym.spaces.Dict,
        gym.spaces.Graph,
        gym.spaces.Sequence,
        gym.spaces.Tuple,
    ]:
        return True
    else:
        return False

def from_jsonable_if_needed(
    sample: Union[ActType, ObsType], space: gym.Space
) -> Union[ActType, ObsType, List]:
    if is_composite_space(space):
        return space.from_jsonable(sample)[0]  # 'int' object is not iterable error (Dict + Discrete)
    else:
        return sample
# gymnasium/spaces/dict.py
def from_jsonable(self, sample_n: dict[str, list[Any]]) -> list[dict[str, Any]]:
    dict_of_list: dict[str, list[Any]] = {
        key: space.from_jsonable(sample_n[key])
        for key, space in self.spaces.items()
    }

    n_elements = len(next(iter(dict_of_list.values())))
    result = [
        {key: value[n] for key, value in dict_of_list.items()}
        for n in range(n_elements)
    ]
    return result
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
Cell In[7], line 3
      1 from ray.rllib.offline.offline_data import OfflinePreLearner
----> 3 episodes = OfflinePreLearner._map_to_episodes(
      4     is_multi_agent=False, batch=batch, observation_space=observation_space, action_space=action_space
      5 )

File /opt/conda/lib/python3.11/site-packages/ray/rllib/offline/offline_prelearner.py:450, in OfflinePreLearner._map_to_episodes(is_multi_agent, batch, schema, to_numpy, input_compress_columns, observation_space, action_space, **kwargs)
    413     NotImplementedError
    414 else:
    415     # Build a single-agent episode with a single row of the batch.
    416     episode = SingleAgentEpisode(
    417         id_=str(batch[schema[Columns.EPS_ID]][i]),
    418         agent_id=agent_id,
    419         # Observations might be (a) serialized and/or (b) converted
    420         # to a JSONable (when a composite space was used). We unserialize
    421         # and then reconvert from JSONable to space sample.
    422         observations=[
    423             convert(unpack_if_needed(obs), observation_space)
    424             if Columns.OBS in input_compress_columns
    425             else convert(obs, observation_space),
    426             convert(
    427                 unpack_if_needed(batch[schema[Columns.NEXT_OBS]][i]),
    428                 observation_space,
    429             )
    430             if Columns.OBS in input_compress_columns
    431             else convert(
    432                 batch[schema[Columns.NEXT_OBS]][i], observation_space
    433             ),
    434         ],
    435         infos=[
    436             {},
    437             batch[schema[Columns.INFOS]][i]
    438             if schema[Columns.INFOS] in batch
    439             else {},
    440         ],
    441         # Actions might be (a) serialized and/or (b) converted to a JSONable
    442         # (when a composite space was used). We unserializer and then
    443         # reconvert from JSONable to space sample.
    444         actions=[
    445             convert(
    446                 unpack_if_needed(batch[schema[Columns.ACTIONS]][i]),
    447                 action_space,
    448             )
    449             if Columns.ACTIONS in input_compress_columns
--> 450             else convert(batch[schema[Columns.ACTIONS]][i], action_space)
    451         ],
    452         rewards=[batch[schema[Columns.REWARDS]][i]],
    453         terminated=batch[
    454             schema[Columns.TERMINATEDS]
    455             if schema[Columns.TERMINATEDS] in batch
    456             else "dones"
    457         ][i],
    458         truncated=batch[schema[Columns.TRUNCATEDS]][i]
    459         if schema[Columns.TRUNCATEDS] in batch
    460         else False,
    461         # TODO (simon): Results in zero-length episodes in connector.
    462         # t_started=batch[Columns.T if Columns.T in batch else
    463         # "unroll_id"][i][0],
    464         # TODO (simon): Single-dimensional columns are not supported.
    465         # Extra model outputs might be serialized. We unserialize them here
    466         # if needed.
    467         # TODO (simon): Check, if we need here also reconversion from
    468         # JSONable in case of composite spaces.
    469         extra_model_outputs={
    470             k: [
    471                 unpack_if_needed(v[i])
    472                 if k in input_compress_columns
    473                 else v[i]
    474             ]
    475             for k, v in batch.items()
    476             if (
    477                 k not in schema
    478                 and k not in schema.values()
    479                 and k not in ["dones", "agent_index", "type"]
    480             )
    481         },
    482         len_lookback_buffer=0,
    483     )
    485     if to_numpy:
    486         episode.to_numpy()

File /opt/conda/lib/python3.11/site-packages/ray/rllib/utils/spaces/space_utils.py:115, in from_jsonable_if_needed(sample, space)
     99 """Returns a jsonabled space sample, if the space is composite.
    100 
    101 Checks, if the space is composite and converts the sample to a JSONable
   (...)
    111     composite sample jsonable to an actual `space` sample..
    112 """
    114 if is_composite_space(space):
--> 115     return space.from_jsonable(sample)[0]
    116 else:
    117     return sample

File /opt/conda/lib/python3.11/site-packages/gymnasium/spaces/dict.py:225, in Dict.from_jsonable(self, sample_n)
    223 def from_jsonable(self, sample_n: dict[str, list[Any]]) -> list[dict[str, Any]]:
    224     """Convert a JSONable data type to a batch of samples from this space."""
--> 225     dict_of_list: dict[str, list[Any]] = {
    226         key: space.from_jsonable(sample_n[key])
    227         for key, space in self.spaces.items()
    228     }
    230     n_elements = len(next(iter(dict_of_list.values())))
    231     result = [
    232         {key: value[n] for key, value in dict_of_list.items()}
    233         for n in range(n_elements)
    234     ]

File /opt/conda/lib/python3.11/site-packages/gymnasium/spaces/dict.py:226, in <dictcomp>(.0)
    223 def from_jsonable(self, sample_n: dict[str, list[Any]]) -> list[dict[str, Any]]:
    224     """Convert a JSONable data type to a batch of samples from this space."""
    225     dict_of_list: dict[str, list[Any]] = {
--> 226         key: space.from_jsonable(sample_n[key])
    227         for key, space in self.spaces.items()
    228     }
    230     n_elements = len(next(iter(dict_of_list.values())))
    231     result = [
    232         {key: value[n] for key, value in dict_of_list.items()}
    233         for n in range(n_elements)
    234     ]

File /opt/conda/lib/python3.11/site-packages/gymnasium/spaces/discrete.py:146, in Discrete.from_jsonable(self, sample_n)
    144 def from_jsonable(self, sample_n: list[int]) -> list[np.int64]:
    145     """Converts a list of json samples to a list of np.int64."""
--> 146     return [np.int64(x) for x in sample_n]

TypeError: 'int' object is not iterable

Versions / Dependencies

ray[rllib]==2.41.0

Reproduction script

episode_example.jsonl
{
  "eps_id": "test01",
  "obs": {
    "vector_obs": [...],
    "vision_obs": [...] 
  },
  "actions": {
    "action": 0,
    "offset_x": 4,
    "offset_y": 7
  },
  "rewards": 0,
  "terminateds": false,
  "truncateds": false,
  "new_obs": {
    "vector_obs": [...], 
    "vision_obs": [...] 
  }
}
import numpy as np
from pathlib import Path
from gymnasium.spaces import Dict, Box, Discrete
from ray.rllib.algorithms.marwil import MARWILConfig
from ray.rllib.offline.offline_data import OfflineData, OfflinePreLearner

base_dir = Path.cwd()
dataset_path = base_dir / {episode_example_path} 

observation_space = Dict({
    "vision_obs": Box(low=-1, high=1, shape=({vision_obs_size},), dtype=np.float32),
    "vector_obs": Box(low=-1, high=1, shape=({vector_obs_size},), dtype=np.float32)
})

# 'int' object is not iterable error (Dict + Discrete)
action_space = Dict({
    "action": Discrete({action_size}),
    "offset_x": Discrete({offset_x_size}),
    "offset_y": Discrete({offset_y_size})
})

config = MARWILConfig()
config.environment(
    observation_space=observation_space,
    action_space=action_space
)

config.offline_data(
    input_=[dataset_path.as_posix()], 
    input_read_method="read_json",
    dataset_num_iters_per_learner=1
)

offline_data = OfflineData(config)
batch = offline_data.data.take_batch(10)

# 'int' object is not iterable error (Dict + Discrete)
episodes = OfflinePreLearner._map_to_episodes(
    is_multi_agent=False, batch=batch, observation_space=observation_space, action_space=action_space
)

Issue Severity

None

@junyeop junyeop added bug Something that is supposed to be working; but isn't triage Needs triage (eg: priority, bug/not-bug, and owning component) labels Jan 29, 2025
@junyeop junyeop changed the title [<Ray component: Core|RLlib|etc...>] "TypeError: 'int' object is not iterable when using is_composite_space with nested Discrete in Dict" [RLlib] "TypeError: 'int' object is not iterable when using is_composite_space with nested Discrete in Dict" Jan 29, 2025
@junyeop junyeop changed the title [RLlib] "TypeError: 'int' object is not iterable when using is_composite_space with nested Discrete in Dict" [RLlib] "TypeError: 'int' object is not iterable when using from_jsonable with nested Discrete in Dict" Jan 29, 2025
@jcotant1 jcotant1 added the rllib RLlib related issues label Jan 30, 2025
@simonsays1980 simonsays1980 added P1 Issue that should be fixed within a few weeks rllib-offline-rl Offline RL problems rllib-newstack and removed triage Needs triage (eg: priority, bug/not-bug, and owning component) labels Feb 21, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something that is supposed to be working; but isn't P1 Issue that should be fixed within a few weeks rllib RLlib related issues rllib-newstack rllib-offline-rl Offline RL problems
Projects
None yet
Development

No branches or pull requests

3 participants