d2c.evaluators

BaseEval

class BaseEval[source]

Bases: abc.ABC

The base class of the evaluator.

This is used for policy evaluation. There are two main classes of evaluation methods:

  1. Use the simulator to evaluate the policy;

  2. Use offline policy evaluation methods to evaluate the policy.

TYPE: ClassVar[str] = 'none'
abstract eval(*args)[source]

The API for evaluating.

Simulator Eval

BMEval

class BMEval(result_dir: str, agent: d2c.models.base.BaseAgent, env: d2c.envs.base.BaseEnv, n_eval_episodes: int = 20, score_normalize: bool = False, score_norm_min: Optional[float] = None, score_norm_max: Optional[float] = None, seed: Optional[int] = None)[source]

Bases: d2c.evaluators.base.BaseEval

The evaluator for the benchmark experiments.

The main API methods for users are:

Parameters

  • result_dir (str) – the path of the folder for saving the evaluating results.

  • agent (BaseAgent) – The agent to be evaluated.

  • env (BaseEnv) – An env for the benchmark dataset.

  • n_eval_episodes (int) – the number of evaluating episodes.

  • score_normalize (bool) – if normalizing the evaluating score.

  • score_norm_min (float) – the minimum value for normalizing the score.

  • score_norm_max (float) – the maximum value for normalizing the score.

  • seed (int) – the random seed for env.

eval(step: int) → Dict[source]

The evaluation API methods for policies evaluation.

Parameters

step (int) – The step number of the agent training process.

save_eval_results() → None[source]

Save the whole evaluation results across the agent training process.

Call it when agent training finished.

bm_eval

bm_eval(agent: d2c.models.base.BaseAgent, env: d2c.envs.base.BaseEnv, config: Union[Any, d2c.utils.utils.Flags]) → d2c.evaluators.sim.benchmark.BMEval[source]

The API of building the evaluators with simulator.

Parameters

  • agent (BaseAgent) – The agent to be evaluated.

  • env (BaseEnv) – An env to evaluate the policy.

  • config – The configuration object.

Off-policy Eval

Fitted-Q Eval

FQE

class FQE(policy: torch.nn.modules.module.Module, state_dim: int, action_dim: int, train_data: d2c.utils.replaybuffer.ReplayBuffer, model_params: Union[List, Tuple] = ([1024, 1024, 1024, 1024], 1), optimizers: Union[List, Tuple] = ('adam', 0.0001), batch_size: int = 256, weight_decays: float = 0.0, update_freq: int = 100, update_rate: float = 1, discount: float = 0.99, device: Optional[Union[str, int, torch.device]] = None)[source]

Bases: object

Fitted Q Evaluation.

FQE is an off-policy evaluation method that approximates a Q function \(Q_ heta (s, a)\) with the trained policy \(\pi_\phi(s)\).

References

Parameters

  • policy (nn.Module) – the policy to be evaluated.

  • state_dim (int) – dimension of the state.

  • action_dim (int) – the dimension of the action.

  • train_data (ReplayBuffer) – the dataset of the batch data.

  • model_params (List) – the parameters for constructing the critic network. It can be a list like [[256, 256], 2]`. [256, 256] means a two-layer FC network with 256 units in each layer and the number 2 means the number of the Q nets for ensemble.

  • optimizers (List) – the parameters for create the optimizers. It can be a dict like ['adam', 1e-4]. It contains the type of the optimizer and the learning rate for the critic network model.

  • batch_size (int) – the size of data batch for training.

  • weight_decays (float) – L2 regularization coefficient of the networks.

  • update_freq (int) – the frequency of update the parameters of the target network.

  • update_rate (float) – the rate of update the parameters of the target network.

  • discount (float) – the discount factor for computing the cumulative reward.

  • device – which device to create this model on. Default to None.

get_q(s: torch.Tensor, a: torch.Tensor) → torch.Tensor[source]
property global_step

The global training step.

print_train_info() → None[source]

Print the training information in training process.

restore(ckpt_name: str) → None[source]
save(ckpt_name: str) → None[source]
train_step() → None[source]

Train the agent for one step.

write_train_summary(summary_writer: torch.utils.tensorboard.writer.SummaryWriter) → None[source]

Record the training information.

Parameters

summary_writer (SummaryWriter) – a file writer.

FQEval

class FQEval(agent: d2c.models.base.BaseAgent, data: d2c.utils.replaybuffer.ReplayBuffer, state_dim: int, action_dim: int, save_dir: str, train_steps: int = 250000, print_freq: int = 1000, summary_freq: int = 100, model_params: Union[List, Tuple] = ([1024, 1024, 1024, 1024], 1), optimizers: Union[List, Tuple] = ('adam', 0.0001), batch_size: int = 256, weight_decays: float = 0.0, update_freq: int = 100, update_rate: float = 1, discount: float = 0.99, device: Optional[Union[str, int, torch.device]] = None, wandb_project: Optional[str] = None, wandb_name: Optional[str] = None, wandb_mode: Optional[str] = 'online', start: int = 0, steps: int = 100)[source]

Bases: d2c.evaluators.base.BaseEval

Evaluator with fitted-Q evaluation.

The main implementation of this method:

  1. Load the policy model that is to be evaluated;

  2. Train the Q-net using FQE with respect to the loaded policy;

  3. Compare the Q-value computed by the trained Q-nets.

Parameters

  • agent (BaseAgent) – The agent object that contains the trained policy to be evaluated.

  • data (ReplayBuffer) – The dataset used to train the Q function in FQE.

  • state_dim (int) – Dimension of the state.

  • action_dim (int) – The dimension of the action.

  • save_dir (str) – The absolute path of the folder to save the Q function model and the evaluating results.

  • train_steps (int) – The number of steps of training the Q function.

  • print_freq (int) – The frequency of printing the training metric information.

  • summary_freq (int) – The frequency of recording the training metric information.

  • model_params (List) – The parameters for constructing the critic network. It can be a list like [[256, 256], 2]`. [256, 256] means a two-layer FC network with 256 units in each layer and the number 2 means the number of the Q nets for ensemble.

  • optimizers (List) – The parameters for create the optimizers. It can be a dict like ['adam', 1e-4]. It contains the type of the optimizer and the learning rate for the critic network model.

  • batch_size (int) – The size of data batch for training.

  • weight_decays (float) – L2 regularization coefficient of the networks.

  • update_freq (int) – The frequency of update the parameters of the target network.

  • update_rate (float) – The rate of update the parameters of the target network.

  • discount (float) – The discount factor for computing the cumulative reward.

  • device – Which device to create this model on. Default to None.

  • wandb_project (str) – The project of the W&B logger for recoding the training and evaluating information.

  • wandb_name (str) – W&B run name.

  • wandb_mode (str) – Can be “online”, “offline” or “disabled”. Defaults to online.

  • start (int) – The index of the start point of the evaluating data in the whole dataset.

  • steps (int) – The number of the evaluating times beginning with the start point.

TYPE: ClassVar[str] = 'fqe'
eval() → None[source]

The API for evaluating.

classmethod from_config(agent: d2c.models.base.BaseAgent, data: d2c.utils.replaybuffer.ReplayBuffer, config: Union[d2c.utils.utils.Flags, Any])[source]

MBOPE

class MBOPE(agent: d2c.models.base.BaseAgent, data: d2c.utils.replaybuffer.ReplayBuffer, env: d2c.envs.learned.env.LeaEnv, save_dir: str, discount: float = 0.99, episode_steps: int = 20, eval_size: int = 256, wandb_project: Optional[str] = None, wandb_name: Optional[str] = None, wandb_mode: Optional[str] = 'online', start: int = 0, steps: int = 100)[source]

Bases: d2c.evaluators.base.BaseEval

Model-based off-policy evaluation. Using the trained dynamics model to evaluate the policy through predicting the cumulative reward.

Parameters

  • agent (BaseAgent) – The agent with the trained policy.

  • data (ReplayBuffer) – The dataset of the batch data.

  • env (LeaEnv) – The env object that contains the trained dynamics.

  • save_dir (str) – The directory for saving the evaluating results.

  • discount (float) – The discount coefficient for computing the cumulative reward.

  • episode_steps (int) – The number of steps for dynamics rollout.

  • eval_size (int) – The batch size of the evaluating data.

  • wandb_project (Optional[str]) – The wandb project name.

  • wandb_name (Optional[str]) – The name of the wandb logger object.

  • wandb_mode (Optional[str]) – The mode of the wandb logger. Default to be ‘online’.

  • start (int) – The start point of the evaluating data.

  • steps (int) – The evaluating data size.

TYPE: ClassVar[str] = 'mb_ope'
eval() → None[source]

The API for evaluating.

classmethod from_config(agent: d2c.models.base.BaseAgent, data: d2c.utils.replaybuffer.ReplayBuffer, env: d2c.envs.learned.env.LeaEnv, config: Union[d2c.utils.utils.Flags, Any])[source]

Utils

register_ope

register_ope(cls: Type[d2c.evaluators.base.BaseEval]) → None[source]

Registering the OPE methods.

Parameters

cls – OPE class inheriting BaseEval.

make_ope

make_ope(name: str, from_config: bool, **kwargs) → d2c.evaluators.base.BaseEval[source]

Creating the OPE.

Parameters

  • name (str) – The name of the registered OPE type. The available types are: ‘fqe’.

  • from_config (bool) – If using the config to create the OPE.

  • kwargs – The OPE arguments.

Returns

An OPE object.