Name
Title | ||
|---|---|---|
Believe What You See: Implicit Constraint Approach for Offline Multi-Agent Reinforcement Learning. |
Abstract | ||
|---|---|---|
Learning from datasets without interaction with environments (Offline Learning) is an essential step to apply Reinforcement Learning (RL) algorithms in real-world scenarios.However, compared with the single-agent counterpart, offline multi-agent RL introduces more agents with the larger state and action space, which is more challenging but attracts little attention. We demonstrate current offline RL algorithms are ineffective in multi-agent systems due to the accumulated extrapolation error. In this paper, we propose a novel offline RL algorithm, named Implicit Constraint Q-learning (ICQ), which effectively alleviates the extrapolation error by only trusting the state-action pairs given in the dataset for value estimation. Moreover, we extend ICQ to multi-agent tasks by decomposing the joint-policy under the implicit constraint. Experimental results demonstrate that the extrapolation error is successfully controlled within a reasonable range and insensitive to the number of agents. We further show that ICQ achieves the state-of-the-art performance in the challenging multi-agent offline tasks (StarCraft II). Our code is public online at https://github.com/YiqinYang/ICQ. |
Year | Venue | DocType |
|---|---|---|
2021 | Annual Conference on Neural Information Processing Systems | Conference |
Citations | PageRank | References |
0 | 0.34 | 0 |
Authors | ||
8 | ||
Name | Order | Citations | PageRank |
|---|---|---|---|
| Yiqin Yang | 1 | 0 | 2.03 |
| Xiaoteng Ma | 2 | 2 | 4.08 |
| Chenghao Li | 3 | 0 | 0.68 |
| Zewu Zheng | 4 | 0 | 0.34 |
| Qiyuan Zhang | 5 | 0 | 1.01 |
| Gao Huang | 6 | 875 | 53.36 |
| Jun Yang | 7 | 6 | 3.59 |
| Qianchuan Zhao | 8 | 524 | 70.53 |