Paper Info
Title
Gnu-RL - A Precocial Reinforcement Learning Solution for Building HVAC Control Using a Differentiable MPC Policy.
Abstract
Reinforcement learning (RL) was first demonstrated to be a feasible approach to controlling heating, ventilation, and air conditioning (HVAC) systems more than a decade ago. However, there has been limited progress towards a practical and scalable RL solution for HVAC control. While one can train an RL agent in simulation, it is not cost-effective to create a model for each thermal zone or building. Likewise, existing RL agents generally take a long time to learn and are opaque to expert interrogation, making them unattractive for real-world deployment. To tackle these challenges, we propose Gnu-RL: a novel approach that enables practical deployment of RL for HVAC control and requires no prior information other than historical data from existing HVAC controllers. To achieve this, Gnu-RL adopts a recently-developed Differentiable Model Predictive Control (MPC) policy, which encodes domain knowledge on planning and system dynamics, making it both sample-efficient and interpretable. Prior to any interaction with the environment, a Gnu-RL agent is pre-trained on historical data using imitation learning, which enables it to match the behavior of the existing controller. Once it is put in charge of controlling the environment, the agent continues to improve its policy end-to-end, using a policy gradient algorithm. We evaluate Gnu-RL on both an EnergyPlus model and a real-world testbed. In both experiments, our agents were directly deployed in the environment after offline pre-training on expert demonstration. In the simulation experiment, our approach saved 6.6% energy compared to the best published RL result for the same environment, while maintaining a higher level of occupant comfort. Next, Gnu-RL was deployed to control the HVAC of a real-world conference room for a three-week period. Our results show that Gnu-RL saved 16.7% of cooling demand compared to the existing controller and tracked temperature set-point better.
Year
DOI
Venue
2019
10.1145/3360322.3360849
BuildSys '19: The 6th ACM International Conference on Systems for Energy-Efficient Buildings, Cities, and Transportation New York NY USA November, 2019
Keywords
Field
DocType
Deep Reinforcement Learning,HVAC Control
Control theory,Domain knowledge,Computer science,Model predictive control,HVAC,Testbed,Control engineering,System dynamics,Reinforcement learning,Scalability
Conference
ISBN
Citations 
PageRank 
978-1-4503-7005-9
6
0.43
References 
Authors
0
3
Name
Order
Citations
PageRank
Bingqing Chen171.80
Zicheng Cai260.43
Mario Berges323829.42