d2c.envs

BaseEnv

class BaseEnv[source]

Bases: Generic[gym.core.ObsType, gym.core.ActType]

The main base environment class derived from OpenAI Gym class.

It encapsulates an environment with arbitrary behind-the-scenes dynamics. The dynamics can be a model learned from the batch data. It can also be a ready-made external model. Please inherit this class to build these two class environments as needed.

The main API methods that users of this class need to know are:

  • step()

  • reset()

And set the following attributes:

  • action_space: The Space object corresponding to valid actions

  • observation_space: The Space object corresponding to valid observations

  • reward_range: A tuple corresponding to the min and max possible rewards

Note: a default reward range set to [-inf,+inf] already exists. Set it if you want a narrower range.

The methods are accessed publicly as “step”, “reset”, etc…

action_space: gym.spaces.space.Space[ActType]
observation_space: gym.spaces.space.Space[ObsType]
render(mode='human')[source]

Compute the render frames as specified by render_mode attribute during initialization of the environment.

The set of supported modes varies per environment. (And some third-party environments may not support rendering at all.) By convention, if render_mode is:

  • None (default): no render is computed.

  • human: render return None. The environment is continuously rendered in the current display or terminal. Usually for human consumption.

  • rgb_array: return a single frame representing the current state of the environment. A frame is a numpy.ndarray with shape (x, y, 3) representing RGB values for an x-by-y pixel image.

  • rgb_array_list: return a list of frames representing the states of the environment since the last reset. Each frame is a numpy.ndarray with shape (x, y, 3), as with rgb_array.

  • ansi: Return a strings (str) or StringIO.StringIO containing a terminal-style text representation for each time step. The text can include newlines and ANSI escape sequences (e.g. for colors).

Note

Make sure that your class’s metadata ‘render_modes’ key includes the list of supported modes. It’s recommended to call super() in implementations to use the functionality of this method.

abstract reset(*, seed: Optional[int] = None, return_info: bool = False, options: Optional[dict] = None) → Union[ObsType, Tuple[ObsType, dict]][source]

Resets the environment to an initial state and returns an initial observation.

This method should also reset the environment’s random number generator(s) if seed is an integer or if the environment has not yet initialized a random number generator. If the environment already has a random number generator and reset is called with seed=None, the RNG should not be reset. Moreover, reset should (in the typical use case) be called with an integer seed right after initialization and then never again.

Returns

the initial observation. info (optional dictionary): a dictionary containing extra information, this is only returned if return_info is set to true

Return type

observation (object)

step(action: ActType) → Tuple[ObsType, float, bool, dict][source]

Run one timestep of the environment’s dynamics. When end of episode is reached, you are responsible for calling reset() to reset this environment’s state.

Accepts an action and returns a tuple (observation, reward, done, info).

Param

object action: an action provided by the agent

Returns

observation, reward, done, info

External Env

D4rlEnv

class D4rlEnv(env_name: str, obs_shift: Optional[numpy.ndarray] = None, obs_scale: Optional[numpy.ndarray] = None)[source]

Bases: Generic[gym.core.ObsType, gym.core.ActType]

The Env for D4RL benchmark.

Parameters

env_name (str) – the name of env.

action_space: gym.spaces.space.Space[ActType]
observation_space: gym.spaces.space.Space[ObsType]
reset(**kwargs: Any) → Union[numpy.ndarray, Tuple[numpy.ndarray, dict]][source]

Resets the environment to an initial state and returns an initial observation.

This method should also reset the environment’s random number generator(s) if seed is an integer or if the environment has not yet initialized a random number generator. If the environment already has a random number generator and reset is called with seed=None, the RNG should not be reset. Moreover, reset should (in the typical use case) be called with an integer seed right after initialization and then never again.

Returns

the initial observation. info (optional dictionary): a dictionary containing extra information, this is only returned if return_info is set to true

Return type

observation (object)

step(a: numpy.ndarray) → Tuple[numpy.ndarray, float, bool, dict][source]

Run one timestep of the environment’s dynamics. When end of episode is reached, you are responsible for calling reset() to reset this environment’s state.

Accepts an action and returns a tuple (observation, reward, done, info).

Param

object action: an action provided by the agent

Returns

observation, reward, done, info

benchmark_env

benchmark_env(config: Optional[Any] = None, benchmark_name: Optional[str] = None, **kwargs: Any) → Union[d2c.envs.base.BaseEnv, Callable[[…], d2c.envs.base.BaseEnv]][source]

Get the Environment according to the benchmark.

Parameters

  • config – the configuration object. When it is not None, an instance object of the env class will be returned.

  • benchmark_name (str) – the name of the benchmark. When config is None, and benchmark_name is not None, the env class will be returned.

  • kwargs – some parameters like obs_shift, obs_scale

Learned Env

LeaEnv

class LeaEnv(config: Any)[source]

Bases: Generic[gym.core.ObsType, gym.core.ActType]

An environment instance that contain the trained dynamics model.

When training the model-based RL and evaluating the trained RL policy, this environment will be used.

The usage usually is as below:

  1. Train the dynamics model(e.g. a neural network model) with the batch data;

  2. Load the trained dynamics model and use the environment:

env = Env(config)
env.load_model()
env.reset()
env.step(a)

See also

Please refer to BaseEnv for other APIs’ usage.

Parameters

config – the configuration that contains the config information of environment.

action_space: gym.spaces.space.Space[ActType]
property d_num

The number of the dynamics models.

property dynamics_module

The dynamics module of the Env.

property dynamics_type

The type of the dynamics model.

property dynamics_with_reward

If the dynamics model predict the reward or not.

get_dynamics()[source]

Get the dynamics model.

load()[source]

The API for loading the trained dynamics model.

observation_space: gym.spaces.space.Space[ObsType]
property r_fn

The reward function.

reset(*, seed: Optional[int] = None, return_info: bool = False, options: Optional[dict] = None)[source]

Resets the environment to an initial state and returns an initial observation. There is difference for RNN dynamics and other dynamics. For RNN dynamics model, there is warm-up in this method. Make sure the input warm_input is contained in parameter options and has shape like that (batch, timesteps, feature_dim), the feature_dim is the sum of state-dimension and action-dimension.

Parameters

  • seed (int) – seed for random number generator(s)

  • return_info (bool) –

  • options (dict) – a dict contain init_s and warm_input. init_s``(np.ndarray or Tensor) is the initial state. For RNN dynamics, ``init_s is the state that is just following the warm_input. warm_input: the warm-up input for LSTM dynamics model.

Returns

the initial observation.

step(a: Union[numpy.ndarray, torch.Tensor]) → Tuple[numpy.ndarray, numpy.ndarray, numpy.ndarray, dict][source]

Run one timestep of the environment’s dynamics. When end of episode is reached, you are responsible for calling reset() to reset this environment’s state.

Accepts an action and returns a tuple (observation, reward, done, info).

Param

object action: an action provided by the agent

Returns

observation, reward, done, info

step_raw(s: Union[numpy.ndarray, torch.Tensor], a: Union[numpy.ndarray, torch.Tensor], with_dist: bool = False) → Union[Tuple[List, List, List], Tuple[List, List, List, List]][source]

Run one timestep of the environment’s dynamics.

This method is usually used in RL training process. Accepts a state and an action, returns a tuple (observation, reward, done,).

There will be ensemble dynamics models. So every dynamics model will compute the results and the returned results will be a list contain all the results.

Parameters

  • s – a batch of state

  • a – a batch of action

  • with_dist (bool) – if return the distribution of the predict next states.

Returns

A tuple including three items:

  • s_p: a list, the agent’s observation of current environments

  • r: a list, the amount of rewards returned after previous actions

  • d: a list, whether these episodes have ended

Dynamics

BaseDyna

class BaseDyna(state_dim: int, action_dim: int, model_params: Union[Dict, easydict.EasyDict, Any], optimizers: Union[Dict, easydict.EasyDict, Any], train_data: d2c.utils.replaybuffer.ReplayBuffer, batch_size: int = 64, weight_decays: float = 0.0, test_data_ratio: float = 0.1, with_reward: bool = False, device: Optional[Union[str, int, torch.device]] = None)[source]

Bases: abc.ABC

The base class for learning dynamics.

It comes into different classes of dynamics with different network structure. All the dynamics model must inherit BaseDyna.

A dynamic class typically has the following parts:

Parameters

  • state_dim (int) – the dimension of the state.

  • action_dim (int) – the dimension of the action.

  • model_params – the parameters for construct the models.

  • optimizers – the parameters for create the optimizers.

  • train_data (ReplayBuffer) – the dataset of the batch data.

  • batch_size (int) – the size of data batch for training.

  • weight_decays (float) – L2 regularization coefficient of the networks.

  • test_data_ratio (float) – the ratio of the test dataset.

  • with_reward (bool) – if the output of the dynamics contains the reward or not.

  • device – which device to create this model on. Default to None.

TYPE: ClassVar[str] = 'none'
abstract dynamics_fns(s: Union[numpy.ndarray, torch.Tensor], a: Union[numpy.ndarray, torch.Tensor]) → Any[source]

Predict the next state.

Parameters

  • s – the input state.

  • a – the input action.

property global_step

The global training step.

print_train_info() → None[source]

Print the training information in training process.

restore(ckpt_name: str) → None[source]

Restore the dynamics model.

Parameters

ckpt_name (str) – the file path of the model saved.

save(ckpt_name) → None[source]

Save the dynamics model.

Parameters

ckpt_name (str) – the file path for model saving.

test_step() → None[source]

Test the model with test dataset.

train_step() → None[source]

Train the dynamics model for one step.

write_train_summary(summary_writer) → None[source]

Record the training information.

Parameters

summary_writer – a tf file writer.

ProbDyna

class ProbDyna(local_mode: bool = True, **kwargs: Any)[source]

Bases: d2c.envs.learned.dynamics.base.BaseDyna

Implementation of dynamics with probabilistic neural network.

Use the deep fully-connected network as the dynamics model. The inputs are the current state and action and the outputs is (mean, std) of the distribution of the predict next state.

Parameters

local_mode (bool) – local_mode means that this dynamics model predicts the difference to the current state.

TYPE: ClassVar[str] = 'prob'
dynamics_fns(s: Union[numpy.ndarray, torch.Tensor], a: Union[numpy.ndarray, torch.Tensor]) → Tuple[List, Dict][source]

Predict the next state.

Parameters

  • s – the input state.

  • a – the input action.

register_dyna

register_dyna(cls: Type[d2c.envs.learned.dynamics.base.BaseDyna]) → None[source]

Registering the dynamics class.

Parameters

cls – Dynamics class inheriting BaseDyna.

make_dynamics

make_dynamics(config: Union[d2c.utils.utils.Flags, Any], data: Optional[d2c.utils.replaybuffer.ReplayBuffer] = None, restore: bool = False) → d2c.envs.learned.dynamics.base.BaseDyna[source]

Construct the Dynamics Agent.

Parameters

  • config – the configuration.

  • data – the data buffer.

  • restore (bool) – If restore the dynamics models from the saved model file.

Returns

Dynamics needed.