BAQ-TD keeps TD learning rates over rate-limited channels
Original title: Adaptive Quantization Preserves Fast Rates for TD Learning over Rate-Limited Channels
Sending O(d log T) bits over horizon T is enough for the order-optimal 1/T rate, up to logarithmic factors.
The paper studies policy evaluation with linear function approximation when temporal-difference updates must be quantized and sent over a finite-capacity channel. Batched Adaptively Quantized TD batches updates to cut variance and encodes innovations with adaptive quantization. A high-probability finite-time bound shows that above a finite capacity, BAQ-TD matches unquantized TD rates up to logarithmic factors, with an exponential communication improvement over prior work.
- The agent sends quantized TD update directions to a server.
- Above a finite capacity, rates match unquantized TD up to log factors.
- Total communication is O(d log T) bits for a d-dimensional parameter.
- The claimed rate is order-optimal tilde-O(1/T).
From the original article
1 min read at arxiv.orgMotivated by reinforcement learning subject to communication constraints, we study a policy evaluation problem with linear function approximation over a rate-limited channel. In our setup, an agent interacting with an unknown environment (modeled as a Markov Decision Process) transmits quantized estimates of a temporal-difference (TD) learning update direction to a server over a finite-capacity channel.
By Sreejeet Maity · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.