Computer Science > Machine Learning
[Submitted on 15 Apr 2021 (v1), last revised 29 May 2022 (this version, v2)]
Title:Scale Invariant Monte Carlo under Linear Function Approximation with Curvature based step-size
View PDFAbstract:We study the feature-scaled version of the Monte Carlo algorithm with linear function approximation. This algorithm converges to a scale-invariant solution, which is not unduly affected by states having feature vectors with large norms. The usual versions of the MCMC algorithm, obtained by minimizing the least-squares criterion, do not produce solutions that give equal importance to all states irrespective of feature-vector norm -- a requirement that may be critical in many reinforcement learning contexts. To speed up convergence in our algorithm, we introduce an adaptive step-size based on the curvature of the iterate convergence path -- a novelty that may be useful in more general optimization contexts as well. A key contribution of this paper is to prove convergence, in the presence of adaptive curvature based step-size and heavy-ball momentum. We provide rigorous theoretical guarantees and use simulations to demonstrate the efficacy of our ideas.
Submission history
From: Rahul Madhavan [view email][v1] Thu, 15 Apr 2021 10:37:38 UTC (1,122 KB)
[v2] Sun, 29 May 2022 06:56:45 UTC (614 KB)
References & Citations
Bibliographic and Citation Tools
Bibliographic Explorer (What is the Explorer?)
Litmaps (What is Litmaps?)
scite Smart Citations (What are Smart Citations?)
Code, Data and Media Associated with this Article
CatalyzeX Code Finder for Papers (What is CatalyzeX?)
DagsHub (What is DagsHub?)
Gotit.pub (What is GotitPub?)
Papers with Code (What is Papers with Code?)
ScienceCast (What is ScienceCast?)
Demos
Recommenders and Search Tools
Influence Flower (What are Influence Flowers?)
Connected Papers (What is Connected Papers?)
CORE Recommender (What is CORE?)
IArxiv Recommender
(What is IArxiv?)
arXivLabs: experimental projects with community collaborators
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.