Model-free reinforcement learning with noisy actions for automated experimental control in optics

verfasst von: Lea Richtmann, Viktoria S. Schmiesing, Dennis Wilken, Jan Heine, Aaron Tranter, Avishek Anand, Tobias J. Osborne, Michèle Heurs
Abstract: Setting up and controlling optical systems is often a challenging and tedious task. The high number of degrees of freedom to control mirrors, lenses, or phases of light makes automatic control challenging, especially when the complexity of the system cannot be adequately modeled due to noise or non-linearities. Here, we show that reinforcement learning (RL) can overcome these challenges when coupling laser light into an optical fiber, using a model-free RL approach that trains directly on the experiment without pre-training on simulations. By utilizing the sample-efficient algorithms Soft Actor-Critic (SAC), Truncated Quantile Critics (TQC), or CrossQ, our agents learn to couple with 90% efficiency. A human expert reaches this efficiency, but the RL agents are quicker. In particular, the CrossQ agent outperforms the other agents in coupling speed while requiring only half the training time. We demonstrate that direct training on an experiment can replace extensive system modeling. Our result exemplifies RL’s potential to tackle problems in optics, paving the way for more complex applications where full noise modeling is not feasible.
Organisationseinheit(en): Institut für Gravitationsphysik
Institut für Theoretische Physik
Institut für Photonik
Externe Organisation(en): Australian National University
Delft University of Technology (TU Delft)
Typ: Artikel
Journal: Transactions on Machine Learning Research
Band: 2025
Anzahl der Seiten: 32
ISSN: 2835-8856
Publikationsdatum: 07.2025
Publikationsstatus: Veröffentlicht
Peer-reviewed: Ja
ASJC Scopus Sachgebiete: Artificial intelligence, Maschinelles Sehen und Mustererkennung
Elektronische Version(en): https://doi.org/10.48550/arXiv.2405.15421 (Zugang: Offen)

BibTeX

@article{207ee5e1a140440eb217992a266fdb77,
title = "Model-free reinforcement learning with noisy actions for automated experimental control in optics",
abstract = "Setting up and controlling optical systems is often a challenging and tedious task. The high number of degrees of freedom to control mirrors, lenses, or phases of light makes automatic control challenging, especially when the complexity of the system cannot be adequately modeled due to noise or non-linearities. Here, we show that reinforcement learning (RL) can overcome these challenges when coupling laser light into an optical fiber, using a model-free RL approach that trains directly on the experiment without pre-training on simulations. By utilizing the sample-efficient algorithms Soft Actor-Critic (SAC), Truncated Quantile Critics (TQC), or CrossQ, our agents learn to couple with 90% efficiency. A human expert reaches this efficiency, but the RL agents are quicker. In particular, the CrossQ agent outperforms the other agents in coupling speed while requiring only half the training time. We demonstrate that direct training on an experiment can replace extensive system modeling. Our result exemplifies RL{\textquoteright}s potential to tackle problems in optics, paving the way for more complex applications where full noise modeling is not feasible.",
keywords = "cs.LG, physics.optics, J.2; I.2.1",
author = "Lea Richtmann and Schmiesing, {Viktoria S.} and Dennis Wilken and Jan Heine and Aaron Tranter and Avishek Anand and Osborne, {Tobias J.} and Mich{\`e}le Heurs",
note = "Publisher Copyright: {\textcopyright} 2025, Transactions on Machine Learning Research. All rights reserved.",
year = "2025",
month = jul,
doi = "10.48550/arXiv.2405.15421",
language = "English",
volume = "2025",
journal = "Transactions on Machine Learning Research",
issn = "2835-8856",
number = "7",
}