Title:Finding Options that Minimize Planning Time

Abstract:While adding temporally abstract actions, or options, to an agent's action repertoire can often accelerate learning and planning, existing approaches for determining which specific options to add are largely heuristic. We aim to formalize the problem of selecting the optimal set of options for planning. Specifically we consider a problem of computing the smallest set of options so that planning converges in less than a given maximum of $\ell$ value-iteration passes. We first show that the problem is not only NP-hard, but also $2^{\log^{1 - \epsilon} n}$-hard to approximate in general unless $NP \subseteq DTIME(n^{poly \log n})$, and $O(\log n)$-hard to approximate even if the task is constrained to be deterministic. We present an polynomial time approximation algorithm for computing the optimal options for tasks with bounded return and goal states. The algorithm has bounded suboptimalty of $O(n)$ in general and $O(\log n)$ for deterministic task. We also analyze the complementary problem of finding the set of $k$ options that minimize the number of value-iteration passes until convergence and showed it is NP-hard and show a polynomial time approximation algorithm. Finally, we empirically evaluate its performance against both the optimal options and a representative collection of heuristic approaches in simple grid-based domains including the classic four rooms problem.