Accelerating Training in Pommerman with Imitation and Reinforcement Learning

Introduction

Reinforcement learning has achieved success in solving several complex problems, ranging from game playing to robotics and autonomous driving . Many algorithms originally developed for gameplay have been subsequently adapted for real-world applications, highlighting the importance of the former from both theoretical and practical perspectives.

However, many of the current algorithms in RL have been designed for single-agent domains, where the environment is either stationary , or else is subject to a fixed set of rules or policies .

In addition, RL algorithms are prone to sample inefficiency, due to which it takes vast amount of training to reach to desirabele performance .

Research Question

In this paper, we aim to strike a balance between the purity of from-scratch RL policy search, with the limitations of imitating a noisy expert policy. The objective of the game is to survive and to kill the opponents by placing bombs.

As the probabilities of the actions drawn from deterministic policies trained using imitation are very skewed, the entropy coefficient in the PPO surrogate objective has been kept to zero.

Methodology

The key contributions of this paper are, (i) a stable learning paradigm for imitation followed by RL-driven improvements without allowing policy forgetting, (ii) a significant reduction in training duration compared to prior literature, and (iii) extensive evaluation of the proposed method in terms of behaviour as well as performance against baseline agents. Trust Region Policy Optimization (TRPO) maximizes an objective function similar to vanilla policy gradient method, subject to a.

Study Design

Although the method is noisy, we observed stabilisation over the course of training.

Results & Findings

Restricted communication is not peculiar to Pommerman, but can be found in several practical situations such as drone swarms as well. Third, constraints such as partial observability and sparse rewards further increase the complexity of the problem, leading to the possibility of policy degeneration.

• Restricted communication is not peculiar to Pommerman, but can be found in several practical situations such as drone swarms as well.
• Third, constraints such as partial observability and sparse rewards further increase the complexity of the problem, leading to the possibility of policy degeneration.
• Two approaches from prior literature that address these issues are to either roll out the environment through tree search or to undertake extensive training , both.
• In addition, there are power-ups which can increase the blast radius of the bomb, increase ammo capacity to place more than one bomb simultaneously, and the.
• Wooden walls can be destroyed by the bombs and might reveal power-ups, whereas stone walls are unaffected.

Practical Applications

At the end of episode, we get only single reward for the team and it may not be clear how to assign credit to individual agents. Under this scenario, both team members get a positive reward from the environment, but this could reinforce the suicidal behaviour of the first agent.

Similarly, one agent could eliminate both opponents whereas its teammate just camps; both agents would get positive rewards, reinforcing a lazy agent .

• Reward is 0.5, if at the end of episode there is only enemy agent alive (could be a loss or a tie, but at least one enemy was killed).

Related Work

Research on Pommerman has led to various approaches categorized into model-free and tree-search-based RL. Studies highlight the effectiveness of different search techniques and the challenges of coordination and learning in multi-agent environments.

Proposed approach

The proposed methodology models the problem as a Markov decision process, focusing on a model-free approach using Proximal Policy Optimization (PPO). The paper discusses the challenges of partial observability and sparse rewards in training agents.