Fast Papers

# Implementation Matters in Deep RL: A Case Study on PPO and TRPO

<p class="fastpapers__source">
<span class="fastpapers__source__authors">Engstrom et al., 2019</span>
|
<a class="fastpapers__source__link" href="https://openreview.net/forum?id=r1etN1rtPB">https://openreview.net/forum?id=r1etN1rtPB</a>
</p>

<div class="fastpapers__tags">
<a class="fastpapers__tags__tag fastpapers__tags__tag--venue" href="#venue-iclr2020">ICLR2020</a>
<a class="fastpapers__tags__tag fastpapers__tags__tag--topic" href="#topic-reinforcement-learning">Reinforcement Learning</a>
</div>

- Proximal Policy Optimization (PPO) has a lot of "code-level optimizations" such as value function clipping, reward scaling, orthogonal initialization, layer scaling, and learning rate annealing.
- An ablation study shows that the clipping objective of PPO is not as important as its code-level optimizations. 
  - PPO without clipped objective (PPO-NoClip) achieves similar performance to PPO.
  - TRPO with the "code-level optimizations" (TRPO+) performs better than PPO in 3 of 4 tasks.
- The clipped objective fails to maintain the KL-based trust region of PPO, but it is maintained by its code-level optimizations.

# How much Position Information Do Convolutional Neural Networks Encode?

<p class="fastpapers__source">
<span class="fastpapers__source__authors">Islam, Jia, and Bruce, 2019</span>
|
<a class="fastpapers__source__link" href="https://openreview.net/forum?id=rJeB36NKvB">https://openreview.net/forum?id=rJeB36NKvB</a>
</p>

- CNNs implicitly encode absolute positional information, and this comes mainly from the zero-padding.
- Freeze the encoder network (VGG or ResNet) pre-trained on ImageNet, and use its features as the input to the position encoding module (PosENet).
- Train the PosENet with a ground truth image independent of the encoder's image input.
- If the CNN cannot encode absolute positional information, the output of the module will be random. It the CNN can, the output will resemble the ground truth image.
- Experiments show that the module can successfully predict the position map, and that more zero-padding results in higher correlation and lower mean absolute error.

# Intrinsic Motivation for Encouraging Synergistic Behavior

<p class="fastpapers__source">
<span class="fastpapers__source__authors">Chitnis et al., 2020</span>
|
<a class="fastpapers__source__link" href="https://arxiv.org/abs/2002.05189">https://arxiv.org/abs/2002.05189</a>
</p>

- Consider a sparse-reward two-agent environment, where two agents must cooperate to perform a task.
- If the task requires both agents to cooperatively act, then a single agent would be unable to complete it.
- Train a forward model that predicts the next state given the current state and a single agent's action, where the other agent does nothing. This model does not understand cooperation, since it is only trained on single-agent transitions.
- This forward model will predict the outcome well if the two agents act separately, but if the two actions are synergistic, the forward model will have large prediction error. This prediction error can be used as the intrinsic reward.
- Instead of using real transitions, a joint forward model can be used to calcualte the prediction error. Although this model generates less accurate transitions, it allows for the computation of analytical gradients.
- Experiments show that agents trained with this intrinsic reward perform better in 6 synergistic tasks than using just the extrinsic rewards or using the single-agent surprised-based intrinsic reward. Using joint forward model with analytical gradients increases the agents' performance further.