d2c.models

Base

BaseAgent

class BaseAgent(env: d2c.envs.learned.env.LeaEnv, model_params: Union[Dict, easydict.EasyDict, Any], optimizers: Union[Dict, easydict.EasyDict, Any], train_data: d2c.utils.replaybuffer.ReplayBuffer, batch_size: int = 64, weight_decays: float = 0.0, update_freq: int = 1, update_rate: float = 0.005, discount: float = 0.99, empty_dataset: Optional[d2c.utils.replaybuffer.ReplayBuffer] = None, device: Optional[Union[str, int, torch.device]] = None)[source]

Bases: abc.ABC

The base class for learning policy and interacting with environment.

We aim to modularizing RL algorithms. It comes into 4 classes of offline RL algorithms in D2C. All the RL algorithms must inherit BaseAgent.

An agent class typically has the following parts:

  • train_step(): train the policy for one step;

  • save(): save the trained models;

  • restore(): restore the trained models;

  • _get_modules(): create the network factories needed for building the models that construct an agent;

  • test_policies(): return the trained policy of this agent

Parameters

  • env (BaseEnv) – the environment learned that contains the dynamics model. It provides the information of the environment, like observation information and action information. It can also provide the trained dynamics models for the model-based RL algorithms.

  • model_params (Dict) – the parameters for constructing all the models of the algorithm. It can be a dict like {q: [[256, 256], 2], p: [[256, 256],]} that contains the parameters of the Q net and the actor(policy) net. [256, 256] means a two-layer FC network with 256 units in each layer and the number 2 means the number of the Q nets.

  • optimizers (Dict) – the parameters for create the optimizers. It can be a dict like {q: ['adam', 3e-4], p: ['adam', 3e-4]}. It contains the type of the optimizer and the learning rate for every network model.

  • train_data (ReplayBuffer) – the dataset of the batch data.

  • batch_size (int) – the size of data batch for training.

  • weight_decays (float) – L2 regularization coefficient of the networks.

  • update_freq (int) – the frequency of update the parameters of the target network.

  • update_rate (float) – the rate of update the parameters of the target network.

  • discount (float) – the discount factor for computing the cumulative reward.

  • empty_dataset (ReplayBuffer) – a replay buffer for storing the generated virtual data by the simulator. It should be empty and the training beginning.

  • device – which device to create this model on. Default to None.

property global_step

The global training step.

print_train_info() → None[source]

Print the training information in training process.

abstract restore(ckpt_name: str) → None[source]

Restore the agent from the saved model file.

Parameters

ckpt_name (str) – the file path of the model saved.

abstract save(ckpt_name: str) → None[source]

Save the whole agent.

Parameters

ckpt_name (str) – the file path for model saving.

property test_policies

The trained policy.

train_step() → None[source]

Train the agent for one step.

write_train_summary(summary_writer: torch.utils.tensorboard.writer.SummaryWriter) → None[source]

Record the training information.

Parameters

summary_writer (SummaryWriter) – a file writer.

BaseAgentModule

class BaseAgentModule(modules: Union[d2c.utils.utils.Flags, Any])[source]

Bases: torch.nn.modules.module.Module, abc.ABC

The base class for AgentModule of any agent.

Build the models for the Agent according to the input network factories.

The following method should be implementation:

  • _build_modules(): build the models needed using the input network factories.

Parameters

modules – the network factories that generated by an Agent method _get_modules().

training: bool

Model-free

DOGEAgent

class DOGEAgent(policy_noise: float = 0.2, update_actor_freq: int = 2, noise_clip: float = 0.5, alpha: float = 2.5, N: int = 20, initial_lambda: float = 5, lambda_lr: float = 0.0003, train_d_steps: int = 100000, **kwargs: Any)[source]

Bases: d2c.models.base.BaseAgent

Implementation of DOGE

Parameters

  • policy_noise (float) – the noise used in updating policy network.

  • update_actor_freq (int) – the update frequency of actor network.

  • noise_clip (float) – the clipping range used in updating policy network.

  • alpha (float) – the value of alpha, which controls the weight for TD3 learning relative to behavior cloning.

  • N (int) – the number of noise samples to train distance function

  • initial_lambda (float) – the vale of initial Lagrangian multiplier

  • lambda_lr (float) – the update step size of Lagrangian multiplier

  • train_d_steps (float) – the total training steps to train distance function

See also

Please refer to BaseAgent for more detailed explanation.

restore(ckpt_name: str) → None[source]

Restore the agent from the saved model file.

Parameters

ckpt_name (str) – the file path of the model saved.

save(ckpt_name: str) → None[source]

Save the whole agent.

Parameters

ckpt_name (str) – the file path for model saving.

H2OAgent

class H2OAgent(update_actor_freq: int = 1, rollout_sim_freq: int = 1000, rollout_sim_num: int = 1000, automatic_entropy_tuning: bool = True, log_alpha_init_value: float = 0.0, log_alpha_prime_init_value: float = 1.0, target_entropy: float = 0.0, backup_entropy: bool = False, alpha_multiplier: float = 1.0, sampling_n_next_states: int = 10, s_prime_std_ratio: float = 1.0, noise_std_discriminator: float = 0.1, cql_lagrange: bool = False, cql_target_action_gap: float = 1.0, cql_temp: float = 1.0, cql_clip_diff_min: int = - 1000, cql_clip_diff_max: int = 1000, min_q_weight: float = 0.01, use_td_target_ratio: bool = True, use_value_regularization: bool = True, use_adaptive_weighting: bool = True, use_variant: bool = False, clip_dynamics_ratio_min: float = 1e-05, clip_dynamics_ratio_max: float = 1.0, adaptive_weighting_min: float = 1e-45, adaptive_weighting_max: float = 10, joint_noise_std: float = 0.0, max_traj_length: int = 1000, **kwargs: Any)[source]

Bases: d2c.models.base.BaseAgent

Implementation of H2O

Parameters

  • update_actor_freq (int) – the update frequency of actor network.

  • rollout_sim_freq (int) – the rollout frequency of simulation samples.

  • rollout_sim_num (int) – number of simulation samples per rollout.

  • automatic_entropy_tuning (bool) – whether to adopt automatic tuning of entropy coefficient (alpha) in entropy-regularized RL algorithms.

  • log_alpha_init_value (float) – initialization value for log alpha.

  • log_alpha_prime_init_value (float) – initialization value for log alpha prime.

  • target_entropy (float) – target entropy value (from CQL).

  • backup_entropy (bool) – whether to apply entropy backup (from CQL).

  • alpha_multiplier (float) – alpha multiplier (from CQL).

  • sampling_n_next_states (int) – the number of s’ resampled from certain s,a pair when performing dynamics gap quantification.

  • s_prime_std_ratio (float) – the multiplier on the standard deviation of s’ when performing dynamics gap quantification.

  • noise_std_discriminator (float) – the standard deviation of noise applied on discriminator training.

  • cql_lagrange (bool) – whether to apply alpha prime (from CQL).

  • cql_target_action_gap (float) – lagrange threshold (from CQL).

  • cql_temp (float) – temperature coefficient of regularization term in solving the inner-loop maximization problem.

  • cql_clip_diff_min (int) – min value of value regularizaion term (q_diff).

  • cql_clip_diff_max (int) – max value of value regularizaion term (q_diff).

  • min_q_weight (float) – multiplier on value regularization term (beta).

  • use_td_target_ratio (bool) – whether to use dynamics ratio to fix bellman error.

  • use_value_regularization (bool) – whether to use value regularization.

  • use_adaptive_weighting (bool) – whether to use adaptive weight (omega).

  • use_variant (bool) – whether to use H2O-variant.

  • clip_dynamics_ratio_min (float) – min value of dynamics ratio.

  • clip_dynamics_ratio_max (float) – max value of dynamics ratio.

  • adaptive_weighting_min (float) – min value of adaptive weight (omega).

  • adaptive_weighting_max (float) – max value of adaptive weight (omega).

  • joint_noise_std (float) – the standard deviation of joint noise (to introduce dynamics gap).

  • max_traj_length (int) – the maximum length of sampled trajectories.

See also

Please refer to BaseAgent for more detailed explanation.

kl_sim_divergence(states: torch.Tensor, actions: torch.Tensor, next_states: torch.Tensor) → torch.Tensor[source]
log_sim_real_dynacmis_ratio(states: torch.Tensor, actions: torch.Tensor, next_states: torch.Tensor) → torch.Tensor[source]
real_sim_dynacmis_ratio(states: torch.Tensor, actions: torch.Tensor, next_states: torch.Tensor) → torch.Tensor[source]
restore(ckpt_name: str) → None[source]

Restore the agent from the saved model file.

Parameters

ckpt_name (str) – the file path of the model saved.

save(ckpt_name: str) → None[source]

Save the whole agent.

Parameters

ckpt_name (str) – the file path for model saving.

IQLAgent

class IQLAgent(temperature: float = 2.0, expectile: float = 0.8, **kwargs: Any)[source]

Bases: d2c.models.base.BaseAgent

Implementation of IQL

Parameters

  • temperature (float) – the value of temperature, which controls the weight for maximum of the Q-function to behavior cloning.

  • expectile (float) – the hyperparameter of expectile regression.

See also

Please refer to BaseAgent for more detailed explanation.

restore(ckpt_name: str) → None[source]

Restore the agent from the saved model file.

Parameters

ckpt_name (str) – the file path of the model saved.

save(ckpt_name: str) → None[source]

Save the whole agent.

Parameters

ckpt_name (str) – the file path for model saving.

TD3BCAgent

class TD3BCAgent(policy_noise: float = 0.2, update_actor_freq: int = 2, noise_clip: float = 0.5, alpha: float = 2.5, **kwargs: Any)[source]

Bases: d2c.models.base.BaseAgent

Implementation of TD3+BC

Parameters

  • policy_noise (float) – the noise used in updating policy network.

  • update_actor_freq (int) – the update frequency of actor network.

  • noise_clip (float) – the clipping range used in updating policy network.

  • alpha (float) – the value of alpha, which controls the weight for TD3 learning relative to behavior cloning.

See also

Please refer to BaseAgent for more detailed explanation.

restore(ckpt_name: str) → None[source]

Restore the agent from the saved model file.

Parameters

ckpt_name (str) – the file path of the model saved.

save(ckpt_name: str) → None[source]

Save the whole agent.

Parameters

ckpt_name (str) – the file path for model saving.

Model-based

Imitation

DMILAgent

class DMILAgent(alpha1: float = 10, alpha2: float = 10, train_f_steps: int = 1000, rollout_freq: int = 1000, rollout_size: Optional[int] = None, **kwargs: Any)[source]

Bases: d2c.models.base.BaseAgent

Implementation of DMIL.

Parameters

  • alpha1 (float) – The hyperparameter alpha for policy.

  • alpha2 (float) – The hyperparameter alpha for dynamics model.

  • train_f_steps (float) – The total training steps to train the dynamics model.

  • rollout_freq (int) – The frequency value for the dynamics model rollout.

  • rollout_size (int) – The size of the rollout data.

See also

Please refer to BaseAgent for more detailed explanation.

generate_rollout(batch: Dict) → None[source]
restore(ckpt_name: str) → None[source]

Restore the agent from the saved model file.

Parameters

ckpt_name (str) – the file path of the model saved.

save(ckpt_name: str) → None[source]

Save the whole agent.

Parameters

ckpt_name (str) – the file path for model saving.

BCAgent

class BCAgent(test_data_ratio: float = 0.0, test_freq: Optional[int] = None, **kwargs: Any)[source]

Bases: d2c.models.base.BaseAgent

Implementation of Behavior cloning via maximum likelihood.

Parameters

  • test_data_ratio (float) – The ratio of the test data in the training data.

  • test_freq (float) – The frequency of validation.

restore(ckpt_name: str) → None[source]

Restore the agent from the saved model file.

Parameters

ckpt_name (str) – the file path of the model saved.

save(ckpt_name: str) → None[source]

Save the whole agent.

Parameters

ckpt_name (str) – the file path for model saving.

Planning

Utils

get_agent

get_agent(model_name: str) → Callable[[…], d2c.models.base.BaseAgent][source]

Get the RL Agent.

Parameters

model_name (str) – the RL algorithm name.

Returns

an Agent corresponding to input name.

Note

The input name should be in the keys of dict AGENT_MODULES_DICT:

Imitation

‘bc’, ‘dmil’

Planning

‘mopp’

Model-free RL

‘td3_bc’, ‘doge’, ‘h2o’, ‘iql’

Model-based RL

make_agent

make_agent(config: Union[d2c.utils.utils.Flags, Any], env: Optional[d2c.envs.base.BaseEnv] = None, data: Optional[d2c.utils.replaybuffer.ReplayBuffer] = None, restore_agent: bool = False) → d2c.models.base.BaseAgent[source]

Construct the Agent

Construct an RL Agent with the config and other objects needed.

Parameters

  • config – the configuration.

  • env (Env) – an Env object.

  • data (ReplayBuffer) – the dataset of the batch data.

  • restore_agent (bool) – if restore the Agent from the saved model file.

Returns

an Agent constructed with the inputs.

Note

When training an agent, the parameter “restore_agent” should be False. When evaluating a trained policy, the parameter “restore_agent” should be True in “reward eval” mode and False in “FQE eval” mode.