d2c.trainers

BaseTrainer

class BaseTrainer(agent: Union[d2c.models.base.BaseAgent, Any], env: d2c.envs.learned.env.LeaEnv, train_data: d2c.utils.replaybuffer.ReplayBuffer, config: Union[Any, d2c.utils.utils.Flags])

Bases: abc.ABC

The base class for RL trainer.

We aim to modularizing the RL training. There are all training methods for the models used in the RL algorithms, like Q-value function model, behavior policy model, policy model, dynamics model.

We recommend that descendants of BaseTrainer implement the following methods:

• __init__(): initialize the Trainer, including getting several configurations from the config and creating the folders for saving models;

• train(): the main interface for training. Train the models needed in turn according to the configuration;

• _train_behavior(): train the behavior policy model in advance using the batch data;

• _train_dynamics(): train the dynamics model in advance using the batch data. The dynamics trained here are used in the model-based RL and the env learned;

• _train_q(): train the Q-value function model with respect to behavior policy in advance using Fitted-Q Evaluation with the batch data;

• _train_vae_s(): train the VAE model that is about the state;

• _train_agent(): train the RL agent. The main training process of the most RL algorithms is here.

Parameters

• agent (BaseAgent) – the agent of the RL algorithm.

• env (LeaEnv) – the env with the dynamics which will be trained.

• train_data (FileReplayBuffer) – the replay buffer that contains the batch data.

• config – the configuration.

abstract train()

Training the models needed.

Trainer

class Trainer(agent: Union[d2c.models.base.BaseAgent, Any], train_data: d2c.utils.replaybuffer.ReplayBuffer, config: Union[Any, d2c.utils.utils.Flags], env: Optional[d2c.envs.learned.env.LeaEnv] = None, evaluator: Optional[Union[Any, d2c.evaluators.base.BaseEval]] = None)

Bases: d2c.trainers.base.BaseTrainer

Implementation of the trainer.

Parameters

evaluator – the evaluation for testing the training polices. It should contain an external perfect env such as BMEval. You can input an evaluator when training in the benchmark experiments.

See also

Please refer to BaseTrainer for more detailed explanation.

static check_ckpt(_model_ckpt_dir: str) → Tuple[Optional[torch.utils.tensorboard.writer.SummaryWriter], str]

Determine if the model files exist.

When calling the train() method, it will check if the models have been trained and decide if to create a file writer.

Parameters

_model_ckpt_dir (str) – the file path of the model that will be trained.

Returns

a file_writer for recording the model training information. If the model has already been trained, it will return None.

train() → None

Training the models needed.