City Research Online

Abstract

The DeeP-Mod framework builds an environment model using features from a Deep Dynamic Programming Network (DDPN), trained via a Deep Q-Network (DQN). While Deep Q-Learning is effective in decision-making, state information is lost in deeper DQN layers due to mixed state-action representations. We address this by using Dynamic Programming (DP) to train a DDPN, where value iteration ensures the output represents state values, not state-action pairs. Extracting features from the DDPN preserves state information, enabling task and action set independence. We show that a reduced DDPN can be trained using features extracted from the original DDPN trained on an identical problem. This reduced DDPN achieves faster convergence under noise and outperforms the original DDPN. Finally, we introduce the DeeP-Mod framework, which creates an environment model using the evolution of features extracted from a DDPN in response to actions. A second DDPN, which learns directly from this feature model rather than raw states, can learn an effective feature-value representation and thus optimal policy. A key advantage of DeeP-Mod is that an externally defined environment model is not needed at any stage, making DDPN applicable to a wide range of environments.

Publication Type:	Conference or Workshop Item (Paper)
Additional Information:	This version of the contribution has been accepted for publication, after peer review but is not the Version of Record and does not reflect post-acceptance improvements, or any corrections. The Version of Record is available online at: https://doi.org/10.1007/978-981-95-1233-1_3. Use of this Accepted Version is subject to the publisher’s Accepted Manuscript terms of use https://www.springernature.com/gp/open-research/policies/accepted-manuscript-terms
Publisher Keywords:	Deep Dynamic Programming Network, Neural Network, Dynamic Programming, Feature Extraction, Feature-based Model, Environmental Modelling, Deep Q-Learning
Subjects:	Q Science > QA Mathematics > QA75 Electronic computers. Computer science
Departments:	School of Science & Technology School of Science & Technology > Department of Computer Science
SWORD Depositor:	Symplectic Administrator

Export

Downloads