Abstract: Deep reinforcement learning has led to many recent-and groundbreaking-advancements. However, these advances have often come at the cost of both the scale and complexity of the underlying RL algorithms. Increases in complexity have in turn made it more difficult for researchers to reproduce published RL algorithms or rapidly prototype ideas. To address this, we introduce Acme, a tool to simplify the development of novel RL algorithms that is specifically designed to enable simple agent implementations that can be run at various scales of execution. Our aim is also to make the results of various RL algorithms developed in academia and industrial labs easier to reproduce and extend. To this end we are releasing baseline implementations of various algorithms, created using our framework. In this work we introduce the major design decisions behind Acme and show how these are used to construct these baselines. We also experiment with these agents at different scales of both complexity and computation-including distributed versions. Ultimately, we show that the design decisions behind Acme lead to agents that can be scaled both up and down and that, for the most part, greater levels of parallelization result in agents with equivalent performance, just faster.

ChainerRL: A Deep Reinforcement Learning Library

Abstract: In this paper, we introduce ChainerRL, an open-source Deep Reinforcement Learning (DRL) library built using Python and the Chainer deep learning framework. ChainerRL implements a comprehensive set of DRL algorithms and techniques drawn from the state-of-the-art research in the field. To foster reproducible research, and for instructional purposes, ChainerRL provides scripts that closely replicate the original papers' experimental settings and reproduce published benchmark results for several algorithms. Lastly, ChainerRL offers a visualization tool that enables the qualitative inspection of trained agents. The ChainerRL source code can be found on GitHub: this https URL .

MushroomRL: Simplifying Reinforcement Learning Research

Abstract: MushroomRL is an open-source Python library developed to simplify the process of implementing and running Reinforcement Learning (RL) experiments. Compared to other available libraries, MushroomRL has been created with the purpose of providing a comprehensive and flexible framework to minimize the effort in implementing and testing novel RL methodologies. Indeed, the architecture of MushroomRL is built in such a way that every component of an RL problem is already provided, and most of the time users can only focus on the implementation of their own algorithms and experiments. The result is a library from which RL researchers can significantly benefit in the critical phase of the empirical analysis of their works. MushroomRL stable code, tutorials and documentation can be found at this https URL.

Tianshou: a Highly Modularized Deep Reinforcement Learning Library

Abstract: In this paper, we present Tianshou, a highly modularized Python library for deep reinforcement learning (DRL) that uses PyTorch as its backend. Tianshou intends to be research-friendly by providing a flexible and reliable infrastructure of DRL algorithms. It supports online and offline training with more than 20 classic algorithms through a unified interface. To facilitate related research and prove Tianshou's reliability, we have released Tianshou's benchmark of MuJoCo environments, covering eight classic algorithms with state-of-the-art performance. We open-sourced Tianshou at https://github.com/thu-ml/tianshou/.

In this paper, we present primarycolorreadableRLtools, a dependency-free, header-only pure C++ library for deep and reinforcement learning. We show the advantages of having modular components that can be tightly integrated at compile time. We explain and justify the novel design decisions and show how they lead to state of the art results and can facilitate research and development as well as hyperparameter search and deployment on extremely limited, microcontroller-based platforms.
To achieve high performance we purposefully trade-off some degree of ease of use that comes with other, Python-based rl libraries. In our view, the steeper learning curve (for the user) is the biggest limitation of primarycolorreadableRLtools, but we believe that there is a place for a high-performance rl library that aims beyond entry-level usage.
We release primarycolorreadableRLtools to the community and we commit to continue to improve and optimize existing components, integrating support for additional hardware and (rl) algorithms, therefore contributing to democratizing the access to state of the art rl methods for continuous control.
Future work will focus on the simulation-to-reality transfer of the capabilities demonstrated in simulation.
We are particularly interested in training adaptive continuous control policies using rl in the setting of multirotor vehicles, and future work in these areas will be built upon and integrated into primarycolorreadableRLtools.
Jonas Degrave, Federico Felici, Jonas Buchli, Michael Neunert, Brendan Tracey,
Francesco Carpanese, Timo Ewalds, Roland Hafner, Abbas Abdolmaleki, Diego
De Las Casas, Craig Donner, Leslie Fritz, Cristian Galperti, Andrea Huber,
James Keeling, Maria Tsimpoukelli, Jackie Kay, Antoine Merle, Jean-Marc
Moret, Seb Noury, Federico Pesamosca, David Pfau, Olivier Sauter, Cristian
Sommariva, Stefano Coda, Basil Duval, Ambrogio Fasoli, Pushmeet Kohli, Koray
Kavukcuoglu, Demis Hassabis, and Martin Riedmiller.
Magnetic control of tokamak plasmas through deep reinforcement
learning.
Nature, 602(7897):414–419, February 2022.
Nikita Rudin, David Hoeller, Philipp Reist, and Marco Hutter.
Learning to walk in minutes using massively parallel deep
reinforcement learning.
In Aleksandra Faust, David Hsu, and Gerhard Neumann, editors, Proceedings of the 5th Conference on Robot Learning, volume 164 of Proceedings of Machine Learning Research, pages 91–100. PMLR, 08–11 Nov
2022.
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E. Hinton.
ImageNet classification with deep convolutional neural networks.
Communications of the ACM, 60(6):84–90, 2012.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.
Deep residual learning for image recognition.
In 2016 IEEE Conference on Computer Vision and Pattern
Recognition (CVPR), pages 770–778, 2016.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A. Rusu, Joel Veness,
Marc G. Bellemare, Alex Graves, Martin Riedmiller, Andreas K. Fidjeland,
Georg Ostrovski, Stig Petersen, Charles Beattie, Amir Sadik, Ioannis
Antonoglou, Helen King, Dharshan Kumaran, Daan Wierstra, Shane Legg, and
Demis Hassabis.
Human-level control through deep reinforcement learning.
Nature, 518(7540):529–533, February 2015.
John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp
Moritz.
Trust region policy optimization.
In Francis Bach and David Blei, editors, Proceedings of the 32nd
International Conference on Machine Learning, volume 37 of Proceedings
of Machine Learning Research, pages 1889–1897, Lille, France, 07–09 Jul
2015. PMLR.
John Schulman, Philipp Moritz, Sergey Levine, Michael Jordan, and Pieter
Abbeel.
High-dimensional continuous control using generalized advantage
estimation.
In Proceedings of the International Conference on Learning
Representations (ICLR), 2016.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov.
Proximal Policy Optimization Algorithms, August 2017.
arXiv:1707.06347 [cs].
OpenAI, Ilge Akkaya, Marcin Andrychowicz, Maciek Chociej, Mateusz Litwin, Bob
McGrew, Arthur Petron, Alex Paino, Matthias Plappert, Glenn Powell, Raphael
Ribas, Jonas Schneider, Nikolas Tezak, Jerry Tworek, Peter Welinder, Lilian
Weng, Qiming Yuan, Wojciech Zaremba, and Lei Zhang.
Solving Rubik's Cube with a Robot Hand, October 2019.
arXiv:1910.07113 [cs, stat].
Matthew W. Hoffman, Bobak Shahriari, John Aslanides, Gabriel Barth-Maron,
Nikola Momchev, Danila Sinopalnikov, Piotr Stańczyk, Sabela Ramos, Anton
Raichuk, Damien Vincent, Léonard Hussenot, Robert Dadashi, Gabriel
Dulac-Arnold, Manu Orsini, Alexis Jacq, Johan Ferret, Nino Vieillard, Seyed
Kamyar Seyed Ghasemipour, Sertan Girgin, Olivier Pietquin, Feryal Behbahani,
Tamara Norman, Abbas Abdolmaleki, Albin Cassirer, Fan Yang, Kate Baumli,
Sarah Henderson, Abe Friesen, Ruba Haroun, Alex Novikov, Sergio Gómez
Colmenarejo, Serkan Cabi, Caglar Gulcehre, Tom Le Paine, Srivatsan
Srinivasan, Andrew Cowie, Ziyu Wang, Bilal Piot, and Nando de Freitas.
Acme: A Research Framework for Distributed Reinforcement
Learning, September 2022.
arXiv:2006.00979 [cs].
Shengyi Huang, Rousslan Fernand Julien Dossa, Chang Ye, Jeff Braga, Dipam
Chakraborty, Kinal Mehta, and Joao G.M. Araujo.
Cleanrl: High-quality single-file implementations of deep
reinforcement learning algorithms.
Journal of Machine Learning Research, 23(274):1–18, 2022.
Eric Liang, Richard Liaw, Robert Nishihara, Philipp Moritz, Roy Fox, Ken
Goldberg, Joseph Gonzalez, Michael Jordan, and Ion Stoica.
RLlib: Abstractions for distributed reinforcement learning.
In Jennifer Dy and Andreas Krause, editors, Proceedings of the
35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pages 3053–3062. PMLR, 10–15 Jul
2018.
Jun Tian.
Reinforcementlearning.jl: A reinforcement learning package for the
julia programming language, 2020.
Carlo D'Eramo, Davide Tateo, Andrea Bonarini, Marcello Restelli, and Jan
Peters.
Mushroomrl: Simplifying reinforcement learning research.
Journal of Machine Learning Research, 22(131):1–5, 2021.
Antonin Raffin, Ashley Hill, Adam Gleave, Anssi Kanervisto, Maximilian
Ernestus, and Noah Dormann.
Stable-baselines3: Reliable reinforcement learning implementations.
Journal of Machine Learning Research, 22(268):1–8, 2021.
Jiayi Weng, Huayu Chen, Dong Yan, Kaichao You, Alexis Duburcq, Minghao Zhang,
Yi Su, Hang Su, and Jun Zhu.
Tianshou: A highly modularized deep reinforcement learning library.
Journal of Machine Learning Research, 23(267):1–6, 2022.
Yasuhiro Fujita, Prabhat Nagarajan, Toshiki Kataoka, and Takahiro Ishikawa.
Chainerrl: A deep reinforcement learning library.
Journal of Machine Learning Research, 22(77):1–14, 2021.
Jonas Eschmann.
Reward function design in reinforcement learning.
Reinforcement Learning Algorithms: Analysis and Applications,
pages 25–33, 2021.
Nicolas Heess, Dhruva TB, Srinivasan Sriram, Jay Lemmon, Josh Merel, Greg
Wayne, Yuval Tassa, Tom Erez, Ziyu Wang, S. M. Ali Eslami, Martin Riedmiller,
and David Silver.
Emergence of Locomotion Behaviours in Rich Environments, July
2017.
arXiv:1707.02286 [cs].
Artem Molchanov, Tao Chen, Wolfgang Hönig, James A Preiss, Nora Ayanian,
and Gaurav S Sukhatme.
Sim-to-(multi)-real: Transfer of low-level robust control policies to
multiple quadrotors.
In 2019 IEEE/RSJ International Conference on Intelligent Robots
and Systems (IROS), pages 59–66. IEEE, 2019.
Takahiro Miki, Joonho Lee, Jemin Hwangbo, Lorenz Wellhausen, Vladlen Koltun,
and Marco Hutter.
Learning robust perceptive locomotion for quadrupedal robots in the
wild.
Science Robotics, 7(62):eabk2822, January 2022.
Bharathan Balaji, Sunil Mallya, Sahika Genc, Saurabh Gupta, Leo Dirac, Vineet
Khare, Gourav Roy, Tao Sun, Yunzhe Tao, Brian Townsend, Eddie Calleja, Sunil
Muralidhara, and Dhanasekar Karuppasamy.
DeepRacer: Autonomous Racing Platform for Experimentation
with Sim2Real Reinforcement Learning.
In 2020 IEEE International Conference on Robotics and
Automation (ICRA), pages 2746–2754, Paris, France, May 2020. IEEE.
Kevin Zakka, Laura Smith, Nimrod Gileadi, Taylor Howell, Xue Bin Peng, Sumeet
Singh, Yuval Tassa, Pete Florence, Andy Zeng, and Pieter Abbeel.
ROBOPIANIST: A Benchmark for High-Dimensional Robot
Control.
Scott Fujimoto, Herke van Hoof, and David Meger.
Addressing function approximation error in actor-critic methods.
In Jennifer Dy and Andreas Krause, editors, Proceedings of the
35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pages 1587–1596. PMLR, 10–15 Jul
2018.
Tuomas Haarnoja, Aurick Zhou, Kristian Hartikainen, George Tucker, Sehoon Ha,
Jie Tan, Vikash Kumar, Henry Zhu, Abhishek Gupta, Pieter Abbeel, and Sergey
Levine.
Soft Actor-Critic Algorithms and Applications, January 2019.
arXiv:1812.05905 [cs, stat].
Jiayi Weng, Min Lin, Shengyi Huang, Bo Liu, Denys Makoviichuk, Viktor
Makoviychuk, Zichen Liu, Yufan Song, Ting Luo, Yukun Jiang, Zhongwen Xu, and
Shuicheng Yan.
Envpool: A highly parallel reinforcement learning environment
execution engine.
In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh,
editors, Advances in Neural Information Processing Systems, volume 35,
pages 22409–22421. Curran Associates, Inc., 2022.
Aleksei Petrenko, Zhehui Huang, Tushar Kumar, Gaurav Sukhatme, and Vladlen
Koltun.
Sample factory: Egocentric 3d control from pixels at 100000 fps with
asynchronous reinforcement learning.
In International Conference on Machine Learning, pages
7652–7662. PMLR, 2020.
Emanuel Todorov, Tom Erez, and Yuval Tassa.
MuJoCo: A physics engine for model-based control.
In 2012 IEEE/RSJ International Conference on
Intelligent Robots and Systems, pages 5026–5033, Vilamoura-Algarve,
Portugal, October 2012. IEEE.
Jeff Bezanson, Stefan Karpinski, Viral B. Shah, and Alan Edelman.
Julia: A Fast Dynamic Language for Technical Computing,
September 2012.
arXiv:1209.5145 [cs].
Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman,
Jie Tang, and Wojciech Zaremba.
OpenAI Gym, June 2016.
arXiv:1606.01540 [cs].
Wenda Zhao, Jacopo Panerati, and Angela P. Schoellig.
Learning-based bias correction for time difference of arrival
ultra-wideband localization of resource-constrained mobile robots.
IEEE Robotics and Automation Letters, 6(2):3639–3646, 2021.
Bardienus P. Duisterhof, Srivatsan Krishnan, Jonathan J. Cruz, Colby R.
Banbury, William Fu, Aleksandra Faust, Guido C. H. E. de Croon, and
Vijay Janapa Reddi.
Learning to Seek: Autonomous Source Seeking with Deep
Reinforcement Learning Onboard a Nano Drone Microcontroller,
January 2021.
arXiv:1909.11236 [cs, eess].
Timothy P. Lillicrap, Jonathan J. Hunt, Alexander Pritzel, Nicolas Heess, Tom
Erez, Yuval Tassa, David Silver, and Daan Wierstra.
Continuous control with deep reinforcement learning, 2016.
arXiv:1509.02971 [cs, stat].
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine.
Soft actor-critic: Off-policy maximum entropy deep reinforcement
learning with a stochastic actor.
In Jennifer Dy and Andreas Krause, editors, Proceedings of the
35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pages 1861–1870. PMLR, 10–15 Jul
2018.
Arsenii Kuznetsov, Pavel Shvechikov, Alexander Grishin, and Dmitry Vetrov.
Controlling overestimation bias with truncated mixture of continuous
distributional quantile critics.
In Hal Daumé III and Aarti Singh, editors, Proceedings of the
37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pages 5556–5566. PMLR, 13–18 Jul
2020.
Nicolas Bach, Andrew Melnik, Malte Schilling, Timo Korthals, and Helge Ritter.
Learn to Move Through a Combination of Policy Gradient
Algorithms: DDPG, D4PG, and TD3.
In Giuseppe Nicosia, Varun Ojha, Emanuele La Malfa, Giorgio Jansen,
Vincenzo Sciacca, Panos Pardalos, Giovanni Giuffrida, and Renato Umeton,
editors, Machine Learning, Optimization, and Data Science,
volume 12566, pages 631–644. Springer International Publishing, Cham, 2020.
Series Title: Lecture Notes in Computer Science.
Ashish Kumar, Zipeng Fu, Deepak Pathak, and Jitendra Malik.
RMA: Rapid Motor Adaptation for Legged Robots.
In Proceedings of Robotics: Science and Systems, Virtual, July
2021.
Volodymyr Mnih, Adria Puigdomenech Badia, Mehdi Mirza, Alex Graves, Timothy
Lillicrap, Tim Harley, David Silver, and Koray Kavukcuoglu.
Asynchronous methods for deep reinforcement learning.
In Maria Florina Balcan and Kilian Q. Weinberger, editors, Proceedings of The 33rd International Conference on Machine Learning,
volume 48 of Proceedings of Machine Learning Research, pages
1928–1937, New York, New York, USA, 20–22 Jun 2016. PMLR.
Mike Innes.
Flux: Elegant machine learning with Julia.
Journal of Open Source Software, 3(25):602, May 2018.
Lorenz Meier, Dominik Honegger, and Marc Pollefeys.
Px4: A node-based multithreaded open source robotics framework for
deeply embedded platforms.
In 2015 IEEE International Conference on Robotics and Automation
(ICRA), pages 6235–6240, 2015.
Mark Towers, Jordan K Terry, Ariel Kwiatkowski, John U. Balis, Gianluca
de Cola, Tristan Deleu, Manuel Goulão, Andreas Kallinteris, Arjun KG, Markus
Krimmel, Rodrigo Perez-Vicente, Andrea Pierré, Sander Schulhoff, Jun Jet
Tai, Andrew Jin Shen Tan, and Omar G. Younis.
Gymnasium.

Checklist

For all authors...

Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Did you describe the limitations of your work?

Did you discuss any potential negative societal impacts of your work?

Have you read the ethics review guidelines and ensured that your paper conforms to them?

If you are including theoretical results...

Did you state the full set of assumptions of all theoretical results?

Did you include complete proofs of all theoretical results?

If you ran experiments (e.g. for benchmarks)...

Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?

Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?

Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?

Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?

If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...

If your work uses existing assets, did you cite the creators?

Did you mention the license of the assets?

Did you include any new assets either in the supplemental material or as a URL?

Did you discuss whether and how consent was obtained from people whose data you're using/curating?

Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content?

If you used crowdsourcing or conducted research with human subjects...

Did you include the full text of instructions given to participants and screenshots, if applicable?

Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?

Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?

Reproducibility statement
To establish easy reproducibility we publish the source code through the project page and provide instructions to recreate the results presented in this paper.
The dependency-free nature of primarycolorreadableRLtools makes it easy to reproduce the results. The user only needs CMake as a build tool and a compiler compatible with C++17.
To take advantage of device-dependent hardware and to attain maximum performance, platform-dependent libraries like Apple Accelerate, Intel oneMKL or Nvidia CUDA might be needed.
We provide instructions to install these library dependencies and more in the accompanying documentation.
The easiest way to reproduce the results reported in this paper is using the binary releases that we provide and should run out of the box on a wide variety of systems (macOS, Windows and Linux). The easiest way to get started using and modifying the code we provide self-contained Docker containers with basic examples (e.g. MNIST digit classification) leading up to the full experiments reported in this paper.
Additionally, we would like to strike that the best results were all accomplished by running primarycolorreadableRLtools on a MacBook Pro with the M1 processor, greatly lowering the barriers to entry in comparison to other setups requiring workstations or even hpc.

Analysis of the deep RL landscape
In this section, we analyze the function approximator models used in the major deep rl for continuous control publications as well as the compute resources employed to solve particular continuous control problems.
From Table {{table:0129da23-c361-43e8-a0b2-921547e76f88}} , it can be observed that while the network structure has been fixed, the hyperparameters have been varying to a certain degree. We can observe an upward trend in the number of hidden units and hence in the number of parameters. While ddpg was an early outlier in its usage of the relu activation function for rl on continuous control problems as well as using neural networks with uncommonly many parameters for that era, we can observe that more recently wider and deeper network architectures have become more commonplace. Additionally, the community seems to have converged on using relu activations.
Selection of works that introduced impactful algorithms and the respective neural network dimensions used for their value function approximations. For the calculation of the number of parameters, an input size of 20 and an output size of 1 is assumed
Year
Name
Hidden dim
#Params
Non-linearity
gray!102015
TRPO {{cite:6b5b42d}}
[50, 50]
1.0x
tanh
2015
GAE {{cite:2ae5ad7}}
[100, 50, 25]
2.3x
tanh
gray!102016
DDPG {{cite:d6162a9}}
[400, 300]
35.3x
ReLU
2017
PPO {{cite:40a9727}}
[64, 64]
1.5x
tanh
gray!102018
TD3 {{cite:219ce23}}
[400, 300]
35.3x
ReLU
2018
SAC {{cite:502d39c}}
[256, 256]
19.6x
ReLU
gray!102019
SACv2 {{cite:8b9083d}}
[256, 256]
19.6x
ReLU
2020
TQC {{cite:bd7bf04}}
[512, 512, 512]
147.0x
ReLU
gray!102020
D4PG&TD3{{cite:13e8f7f}}
[256, 256]
19.6x
ReLU
2021
PPO&RMA {{cite:b99f2e7}}
[128, 128, 128]
9.8x
ReLU
{{table:0129da23-c361-43e8-a0b2-921547e76f88}}It is important to keep in mind that the number of parameters roughly grows quadratically with the dimensionality of the hidden layers which leads to the hidden dimensionalities often obfuscating the (relative) scale of the model used. In Table {{table:0129da23-c361-43e8-a0b2-921547e76f88}}  this can e.g. be observed in the case of tqc where it appears that in comparison to trpo the dimensions have been roughly multiplied by ten and the network has been made {{formula:5faca80a-2fc5-47d4-aadf-2f61624e9fe9}}  deeper hence the naively expected increase in model size is {{formula:d751f089-8a29-4c96-a34d-0f9e1dd44491}}  fold, but in reality it is closer to 150 fold. It is important to make sure that algorithmic improvements are not confused with gains from just scaling up the models. Fortunately, in the case of tqc these effects are properly evaluated in {{cite:bd7bf04}} in an ablation study but in general, this factor is largely neglected.
Selected wall-clock training times for different algorithms and tasks
Year
Name
Task
CPUs/GPUs
Time
gray!102016
A2C {{cite:292e73a}}
MuJoCo
16 / 0
{{formula:ffa3f3c1-1e35-47d6-be2e-caddaf85ae3a}}
2015
GAE {{cite:2ae5ad7}}
3D biped locomotion
16 / 0
{{formula:041eaf0c-5930-4c04-9cc7-56f35a04926a}}
gray!102015
GAE {{cite:2ae5ad7}}
Quadruped locomotion
32 / 0
{{formula:61ff9d16-44e5-4e40-88b2-e098e815ee55}}
2019
PPO {{cite:8adac56}}
Rubik's cube
29440 / 64
{{formula:531a863e-9c6f-4436-acac-f705957419ca}}
gray!102021
PPO {{cite:b99f2e7}}
Quadruped locomotion
8 / 1
{{formula:b77e4953-f1b9-4606-913a-a9f197dab0f3}}
2022
PPO {{cite:46b0c62}}
Quadruped locomotion
8 / 1
{{formula:16feecd7-e85b-436c-a186-c078410267fa}}
{{table:fb76c0d6-ff6d-439a-8f6e-e48b91661d4e}}In Table {{table:fb76c0d6-ff6d-439a-8f6e-e48b91661d4e}} , we display the reported training times required to solve a number of continuous control tasks. We can see that the time is wildly varying, reaching from 20 minutes to 14 days, while the required resources vary from a single workstation to a full hpc cluster. While the enormous resources put into learning to manipulate a Rubik's cube is an outlier in time and hardware the remaining examples still show that training time is a major barrier for applying rl to continuous control problems. While the optimizations presented in {{cite:46b0c62}} are a step in the right direction for quadruped locomotion we push the boundary further in case of the baseline pendulum swing-up and the popular Ant-v4 MuJoCo environment.

Programming paradigm
bptstyle
identifierstyle=,
commentstyle=,
keywordstyle=,
numberstyle=,
stringstyle=,
basicstyle=,
breakatwhitespace=false,
breaklines=true,
captionpos=b,
keepspaces=true,
numbersep=5pt,
showspaces=false,
showstringspaces=false,
showtabs=false,
tabsize=2,

Toy example for tag dispatch towards different implementations of elementary matrix operations
-0.05-0.05
[roundcorner=10pt, linewidth=0, backgroundcolor=listingbackgroundcolor, userdefinedwidth=1.1align=center]bptstylelanguage=C++]code/multipledispatch.cpp

{{figure:2415a08c-0601-43eb-a051-3082920351c3}}To enable maximum performance, we are avoiding C++ vmt lookups by not using an object-oriented paradigm but a rather functional paradigm heavily based on templating and method overloading resembling a static, compile-time defined interpretation of the multiple dispatch paradigm. Multiple dispatch has been popularized by the Julia programming language {{cite:56efbcd}} and is based on advanced function overloading.
Leveraging multiple dispatch, higher-level functions like the forward or backward pass of a fully connected neural network just specify the actions that should be taken on the different sub-components/layers and the actual implementation used is dependent on the type of the arguments. In this way, it is simple to share code between GPU and CPU implementations by just implementing the lower-level primitives for the respective device types and then signaling the implementations through the argument type (e.g., using the tag dispatch technique). A toy example for this is displayed in Figure {{figure:2415a08c-0601-43eb-a051-3082920351c3}} . In this case, some algorithm is using a matrix multiplication operation on two objects. During the implementation of the algorithm, we do not need to care about the type of the operands and just let them be specified by wildcard template parameters. When this function is called by the user, the compiler infers the template parameters and dispatches the call to the appropriate implementation. If the user does not have a GPU available he simply does not include the implementation_gpu.h and hence has no dependency on further dependencies that the GPU implementation would entail (e.g., the CUDA toolkit). In the case where there is no specialized implementation for a particular hardware, the compiler will fall back to the generic implementation which in this example could simply consist of a nested loop. The generic implementations are pure C++ and are guaranteed to have no dependencies. We can also see that the compiler will check the dimensions of the operands automatically at compile time such that the algorithm can not be called with incompatible shapes. To create more complex dispatch behaviors and operand type checking C++ features like static_assert and enable_if can be leveraged through the sfinae mechanism. In this way, we can maintain composability while still providing all the structure to the compiler at compile-time.
In the case of Julia, this leads to unparalleled composability which manifests in a small number of people being able to implement and maintain e.g. a deep learning library (Flux {{cite:8d60225}}) that is competitive with PyTorch and TensorFlow which are backed by much more resources.
In contrast to Julia, which reaches almost native performance while performing the multiple dispatch resolution at runtime, we make sure that all the function calls can be resolved at compile time. Additionally, Julia is not suited for our purposes because it does not fit to run on microcontrollers due to its runtime size and stochastic, non-realtime behavior due to the garbage collection-based memory management. Nevertheless, in our benchmark presented later in this manuscript, we found that Julia is our closest competitor when it comes to training performance.
Furthermore, we find it important to emphasize that we focus on building a library not a framework.Write Libraries, Not Frameworks [link] The main feature of frameworks is that they restrict the freedoms of the user to make a small set of tasks easier to accomplish. In certain, repetitive problem settings this might be justified, but in many cases, the overhead coming with the steep learning curves and finding workarounds after bumping into the tight restrictions of frameworks is not worth it. The major conceptual difference is that frameworks provide a context from which they invoke the user's code while in the case of libraries, the user is entirely in control and invokes the components he needs. If not specifically made interoperable, the contexts provided by frameworks are usually incompatible while with libraries this is not generally the case.
In our implementation, this for example concretely manifests in the way function approximators are used in the rl algorithms. By using templating, any function approximator type can be specified by the user at compile time. As long as he also provides the required forward and backward functions.
As demonstrated in Figure {{figure:2415a08c-0601-43eb-a051-3082920351c3}}  we establish the convention of making a device-dependent context available in each function via tag dispatch to simplify the usage of different compute devices like accelerators or microcontrollers.

Deep reinforcement learning frameworks and libraries
Overview over different rl libraries/frameworks, the deep learning platform they build upon, and their popularity in terms of Github stars and publication citations (data as of 2023-02-22)
Name
Platform
Stars / Citations
gray!10Acme {{cite:077dbd2}}
Python / JAX
3019 / 152
CleanRL {{cite:d553e54}}
Python / PyTorch
2154 / 12
gray!10ChainerRL {{cite:b71fcf2}}
Python / Chainer
1093 / 89
Stable Baselines3 {{cite:947e77c}}
Python / PyTorch
5031 / 407
gray!10MushroomRL {{cite:4040b15}}
Python / TF & PyTorch
657 / 36
PFRL {{cite:b71fcf2}}
Python / PyTorch
991 / 89
gray!10ReinforcementLearning.jl {{cite:be31402}}
Julia / Flux
474 / n/a
RLlib {{cite:4a37a66}}
Python / PyTorch
25242 / 575
gray!10Tianshou {{cite:d21dd6e}}
Python / PyTorch
5779 / 49
{{table:85a05944-522a-4bbd-bd08-0f678e82def5}}
Details for the (accelerated) training in web browsers, smartphones and smartwatches
To be able to run the entire training of the Pendulum-v1 using td3 in web browsers we compile primarycolorreadableRLtools to wasm using the Emscripten compiler which has C++17 support. For benchmarking, we just need to expose a single function that starts a full training run and is callable from JavaScript. We also take advantage of web browsers being the most widespread ui framework and implement additional functions to visualize the training progress. A live demo of the training process can be tested with any recent browser at our project page (https://rl.tools).
The results in Figure {{figure:0c7f822e-ca0c-4223-848d-c75a522ae53d}}  show a severe slowdown of the wasm-based training in the browser vs. the native training, which is to be expected given that the browsers have additional safety checks to be able to safely execute untrusted code. Additionally, because of the lack of available gemm libraries, we run the generic implementation of our forward and backward passes which also reduces the training speed. We believe that with the upcoming WebGPU standard it might be possible to drastically speed up the training in the browser and possibly make it competitive with training on native hardware, carrying the additional benefit of greatly simplified distribution and taking advantage of the ubiquitous availability of web browsers.
We also benchmark primarycolorreadableRLtools on a smartphone (iPhone 11) and a smartwatch (Apple Watch Series 4) taking advantage of the same backend used in case of the MacBook M1 where we use the dispatch technique described in Section  to dispatch matrix multiplications to the Apple Accelerate framework. The Accelerate framework is available on all the mentioned devices and takes maximum advantage of the particular hardware available on each chip, leading to very competitive performance and efficiency. We speculate that e.g. the fast performance on the M1 CPU is partially due to the usage of the specialized amx through the Accelerate framework.

PPO modification: adaptive learning rate
In the following, we are documenting the theory behind the adaptive learning rate mechanism that showed high efficacy in improving the sample efficiency of ppo in our (ablation) study. We found this adaptive learning rate mechanism when studying the rl_games ppo implementation and would like to give credit to its authors (a subset of the authors of {{cite:30acafb}}) for devising it. We are documenting it here because of its high efficacy in our ablation study and because it is neither mentioned in {{cite:30acafb}} nor used or documented anywhere else to the best of our knowledge.
The idea behind the adaptive learning rate mechanism is to regulate the kl divergence of the action distribution before and after the (gradient-based) policy updates. To accomplish this, a new hyperparameter {{formula:30c9f401-7a79-42dc-9f3c-1c619a9534e4}}  for the target kl divergence is introduced (e.g. {{formula:d61e891a-683c-48e9-9c1f-7da41ea9912f}}  in the Ant-v4 experiment). The kl divergence between the current policy's distribution and the distribution of the policy used to collect the data is calculated for each training batch of the policy.
To stabilize the kl divergence, a sort of feedback control law is introduced where the control input is the learning rate and the output of the plant is the aforementioned kl divergence. From this point of view, the adaptive learning rate mechanism resembles a bang-bang controller that can either reduce or increase the learning rate by a fixed, multiplicative factor {{formula:57fa3843-6a78-4004-8044-a2a6796d59e3}}  or leave it unchanged. In the Ant-v4 experiment e.g. {{formula:a1ba2c07-10f3-4d02-b1a1-84fe883de3b7}} . Now, having defined the three possible control outputs the control law is based on simple thresholding:

temp =  {ll
t CLR  if  DKL(old|) < 0.5 TKL
tCLR  if  DKL(old|) > 2 TKL
t  else
.
t+1 =  (min, (temp, max))

Where {{formula:00b7817c-186f-4eee-8407-a76747aeb791}}  is the learning rate at batch step {{formula:abcfd6f2-baa5-46fa-937e-9ced64769039}} , {{formula:6ef19d11-a813-4869-a6e2-61d09df36e38}}  is the stochastic policy used to sample the data for the current epoch and {{formula:f397ca84-011c-47f3-b4d8-ae30354b0118}}  is the current policy updated by minibatch gradient descent. {{formula:136b8a9c-6da5-448c-b376-8a4401af9c33}}  and {{formula:cb935d86-0702-4c05-a524-52be3b83cfaf}}  are additional hyperparameters limiting the range of the learning rate ({{formula:f9ecc656-2288-4925-b041-8a3a0f867d07}}  and {{formula:e8ac8432-ede2-4f75-aafd-2fede24e9a8b}}  in the case of the Ant-v4 experiments).
Ablation study.
Learning curves of our ppo implementation with and without an adaptive learning rate and observation normalization on the Ant-v4 environment. We show the mean and standard deviation across 100 runs with different seeds.
{{figure:225eca2c-4c61-4c65-ba3e-561d4c39d82f}}To analyze the mentioned impact of the observation normalization and adaptive learning rate we conduct a small ablation study where we compare the training behavior when turning on/off these modifications in our ppo implementation. Figure {{figure:225eca2c-4c61-4c65-ba3e-561d4c39d82f}}  shows the learning curves for these two scenarios where the scene with the adaptive learning rate and observation normalization is exactly the same as in the experiment depicted in Figure {{figure:be13c363-a90f-40e0-b30c-67d722546c42}} . We can observe that these modifications drastically improve the sample complexity and slightly improve the final performance. Especially the beginning of the learning process appears much more healthy in the case where the modifications are active compared to the standard setup where the agent takes around 1000000 steps to start earning increased rewards. In our experience, a return of 3000-4000 corresponds to a very good walking policy where the agent is using all four legs of the ant for walking while above this threshold the gait becomes a hopping motion using two legs to go forward and two to stabilize. Hence at the point where the standard implementation is just starting to pick up rewards ({{formula:da4fbd97-b313-42f3-94de-2793629a2d9e}}  interaction steps), the modified algorithm already learned a very decent walking gait.

Embedded platforms

Crazyflie: A small, open-source quadrotor which only weights 27 g including the battery. The Crazyflie's main processor is a STM32F405 microcontroller using the ARM Cortex-M4 architecture, featuring 192 KB of ram and running at 168 MHz.

Pixhawk 6C: We use a Pixracer Pro An fcu that belongs to the family of Pixhawk fcu and implements the Pixhawk 6C standard. Hence, the PixRacer Pro supports the common PX4 firmware {{cite:085011f}} and can be used in many different vehicle types (aerial, ground, marine) but is predominantly used in multirotor vehicles of varying sizes. The main processor used in the Pixhawk 6C standard is a STM32H743 using the ARM Cortex-M7 architecture. The PixRacer Pro runs at 460 MHz and comes with 1024 KB of ram.

teensy 4.1: A general-purpose embedded device powered by an i.MX RT1060 ARM Cortex-M7 microcontroller with 1024 KB on-chip and 16 MB off-chip ram that is running at 600 MHz.

ESP32: One of the most common microcontrollers for iot and edge devices due to its built-in Wi-Fi and Bluetooth. Close to 1 billion devices built around this chip and its predecessor have been sold worldwide. Hence it is widely available and relatively cheap (around {{formula:3afcc982-fd1c-49bf-90ce-c0149416003d}} 5 for a development kit). For our purposes, the ESP32 is interesting because it deviates from the previous platforms in that its processor is based on the Xtensa LX7 architecture. In addition to the original version of the ESP32 based on the Xtensa architecture, we also evaluate the ESP32-C3 version based on the RISC-V architecture.

Deployment on embedded systems: qualitative evaluation
First, we qualitatively evaluate the ease of integration of primarycolorreadableRLtools on different platforms. The Crazyflie uses an open-source purely C-based firmware but since its toolchain is based on the ARM port of the gcc hence we can integrate primarycolorreadableRLtools in less than 25 lines of code by using an adapter function that turns off C++ name mangling and hence is linkable with the C code. To access the weights of a neural network that has been trained a priori we export the network weights as C++ code through code generation which can then simply be imported in the firmware using an #include directive. This also allows us to export the shapes of the matrices to adhere to our design premise of having all dimensions being defined at compile-time.
In the case of the Pixhawk 6C the integration is even simpler as its firmware, PX4 is already C++ based, and we are able to implement the inference on an exported neural network in less than 10 lines of code.
The teensy 4.1 uses an Arduino-based toolchain which is naturally using C++14 but can be switched to C++17 by adding the appropriate flag. For inference, we can simply include the code as described for the previous devices. The teensy 4.1 is the most powerful microcontroller-based platform in this selection hence as shown in Figure {{figure:0c7f822e-ca0c-4223-848d-c75a522ae53d}} , we also demonstrate that training of deep rl algorithms is possible on this kind of limited device. The only modification to the training code that is needed is to adjust the memory allocation to account for the different memory spaces (on- and off-chip) in the teensy 4.1. In primarycolorreadableRLtools this is a simple change because all data structures and algorithms are agnostic to whether the matrices/containers are allocated dynamically or statically. We chose the teensy 4.1 over the other microcontrollers because of the memory required to store the replay buffer of td3. Anecdotally, we also observe that training the Pendulum-v1 swing-up is possible with replay buffer sizes much smaller than 10000 (equal to the number of steps) making our proposed solution feasible also to trained on substantially memory-limited platforms.
Even though ESP32 comes in two versions with entirely different microcontroller architectures it proves to be straightforward to run inference on it using primarycolorreadableRLtools for the same reasons as described with the previous platforms. Additionally, the CMake-based build system provided by the manufacturer made the process even more straightforward.
This comparatively low effort of integrating primarycolorreadableRLtools on a microcontroller platform shows the virtue of our library being header-only and free of dependencies.

Extended Benchmarks
Soft Actor-Critic (sac)
In addition to td3 {{cite:219ce23}} we also implement sac {{cite:8b9083d}} and test it under the same circumstances as described in Section REF . We implement the full sac algorithm including the adaptive learning of the entropy regularization parameter. From Figure {{figure:18f17ea5-df79-4d4b-9db9-906ead0ad5ab}} , we can see the same trend as in Figure {{figure:0c7f822e-ca0c-4223-848d-c75a522ae53d}}  with primarycolorreadableRLtools being about 6 to 19 times faster than the popular RLlib and Stable Baselines3 deep rl libraries. Note that due to time constraints, we focus the benchmark comparison on the most relevant related libraries. Since the performance multiples roughly validate the differences in Figure {{figure:0c7f822e-ca0c-4223-848d-c75a522ae53d}} , we expect that the remaining libraries would attain similar relative performance multiples.
Benchmark of sac in the Pendulum-v1 environment.
{{figure:18f17ea5-df79-4d4b-9db9-906ead0ad5ab}}
Acrobot-v1
Furthermore, to show that the wall-clock training-time performance of primarycolorreadableRLtools is not just limited to the Pendulum-v1 environment, we also implement the popular Acrobot-v1 environment. The Acrobot is a common mechanical system that is dynamically more complex than the pendulum. The original Acrobot-v1 in the OpenAI gym{{cite:a84a596}}/Gymnasium{{cite:312461e}} only has a discrete action space containing the torques {{formula:66c88223-3309-4119-b92d-6e58678ffb51}} . Because primarycolorreadableRLtools is focused on continuous control, we modify the action space to be continuous and contain torques in {{formula:108c2c31-bfad-44ed-8bc1-7ea0460f9d24}} . We observe that for some random seeds td3 and sac can solve the continuous Acrobot-v1 in about {{formula:0ef4b4eb-d750-41b4-9963-f2bb597233bd}}  environment interaction steps. Hence, we benchmark the training time for 50000 steps using the same hyperparameters as in REF . Figures {{figure:82ad3a9f-f5a9-4d23-b54c-cc43b07c38bb}}  and {{figure:e03db6b9-91ec-47bb-aaa9-c4001976de1f}}  show the wall clock training time performance using td3 and sac respectively.
Benchmark of td3 in the Acrobot-v1 environment (changed from discrete to continuous actions).
{{figure:82ad3a9f-f5a9-4d23-b54c-cc43b07c38bb}}Benchmark of sac in the Acrobot-v1 environment (changed from discrete to continuous actions).
{{figure:e03db6b9-91ec-47bb-aaa9-c4001976de1f}}We can observe that even when switching to a more dynamically complex environment, the relative benefit in the wall-clock training time of primarycolorreadableRLtools remains roughly the same. In our experience, this pattern holds for all simulators that can be expressed as ode. Anecdotally in addition to the Acrobot and Pendulum, we also observe this magnitude of a wall-clock training time improvement when using primarycolorreadableRLtools for multirotor vehicles and (racing) cars. We do not include these environments in the benchmark because unlike the Pendulum-v1 (and the Acrobot-v1 to some degree) they are not supported by a broad variety of deep rl libraries.

Ablation Study
We conduct an ablation study to investigate the contribution of different components and optimizations to the fast wall-clock training time achieved by primarycolorreadableRLtools. Figure {{figure:7cf620b5-d510-4140-94c8-e2e8e74c4f63}}  shows the resulting training times after removing different components and optimizations from the setup. The “Baseline” is exactly the same setup used in the Pendulum-v1 (td3) training in the other experiments in Section REF  and Appendix . We simulate the slowness of the Python environment by slowing down the C++ implementation by the average time required for a step in the Python implementation. We can observe that the C++ implementation of the Pendulum-v1 dynamics has a measurable, but not dominating impact on the training time. Additionally, we ablate the different optimization levels -O0, -O1, -O2 and -O3 (used in the Baseline) of the C++ compiler. We can observe that the compiler optimizations have a sizeable impact on the training time. When removing all optimizations (-O0) primarycolorreadableRLtools is roughly on par with Stable Baselines3 (cf. Figure {{figure:0c7f822e-ca0c-4223-848d-c75a522ae53d}} ).
Furthermore, while exploring the compiler optimizations we noticed that the -ffast-math option yields a big improvement in training time (almost halving it) compared to the earlier experiments with just -O3. Importantly, we did not observe a degradation of the performance in terms of rewards when using -ffast-math. Hence, primarycolorreadableRLtools is actually about 11 to 25 times faster than RLlib and Stable Baselines3 (based on the measurements in Figure {{figure:7cf620b5-d510-4140-94c8-e2e8e74c4f63}}  and {{figure:0c7f822e-ca0c-4223-848d-c75a522ae53d}} ).
Comparison of the impact of different components on the training time. The “Baseline” contains all optimizations (except using fast math).
{{figure:7cf620b5-d510-4140-94c8-e2e8e74c4f63}}


Multistep Laplace Adomian Decomposition Method(MLADM)

Numerical Treatment of the Model for HIV Infection of CD4

Abstract: A new method for approximate analytic series solution called multistep Laplace Adomian Decomposition Method (MLADM) has been proposed for solving the model for HIV infection of CD4

Computational Analysis of the Model Describing HIV Infection of CD4+T Cells

Abstract: An analysis of the model underpinning the description of the spread of HIV infection of CD4+T cells is examined in detail in this work. Investigations of the disease free and endemic equilibrium are done using the method of Jacobian matrix. An iteration technique, namely, the homotopy decomposition method (HDM), is implemented to give an approximate solution of nonlinear ordinary differential equation systems. The technique is described and illustrated with numerical examples. The approximated solution obtained via HDM is compared with those obtained via other methods to prove the trustworthiness of HDM. Moreover, the lessening and simplicity in calculations furnish HDM with a broader applicability.

Adomian Decomposition Method Combined with Pade Approximation and Laplace Transform for Solving a Model of HIV Infection of CD4+T Cells

Abstract: Theclassical Adomian decomposition method (ADM) is implemented to solve a model of HIV infection of CD4(+)Tcells. The results indicate that the approximate solution by using the ADM is the same as that by using the Laplace ADM, but it can be obtained in a more efficient way. We also use Pade approximation and Laplace transform as a posttreatment technique to obtain the result of the ADM. The advantage of the posttreatment is illustrated by numerical experiments.

Used technique for solving HIV model

It is noteworthy that there is no exact solution for HIV model. Ergo, the numerical methods are used to solve it. Table {{table:889e738d-6896-427d-bbcf-b43da29dd30f}}  shows some approaches applied to this model.


YOLOv3: An Incremental Improvement

Abstract: We present some updates to YOLO! We made a bunch of little design changes to make it better. We also trained this new network that's pretty swell. It's a little bigger than last time but more accurate. It's still fast though, don't worry. At 320x320 YOLOv3 runs in 22 ms at 28.2 mAP, as accurate as SSD but three times faster. When we look at the old .5 IOU mAP detection metric YOLOv3 is quite good. It achieves 57.9 mAP@50 in 51 ms on a Titan X, compared to 57.5 mAP@50 in 198 ms by RetinaNet, similar performance but 3.8x faster. As always, all the code is online at https://pjreddie.com/yolo/

Image Captioning

Abstract: This paper discusses and demonstrates the outcomes from our experimentation on Image Captioning. Image captioning is a much more involved task than image recognition or classification, because of the additional challenge of recognizing the interdependence between the objects/concepts in the image and the creation of a succinct sentential narration. Experiments on several labeled datasets show the accuracy of the model and the fluency of the language it learns solely from image descriptions. As a toy application, we apply image captioning to create video captions, and we advance a few hypotheses on the challenges we encountered.

Summary of ML models used on Hollywood2 dataset.

Multi-modal dataset and ML models.
We used the Hollywood2 video dataset {{cite:4186d7e}}.
To obtain aligned inputs for multi-modal models, we selected the 30th frame and extracted audio data from each video.
We deployed seven pre-trained ML models that cover five classes of learning tasks: single-label and multi-label classification, object localization, regression, and sequence generation.
And they have different model architectures, input modalities and output formats.
To evaluate the performance of model links, we used task-specific metrics, including accuracy, mean average precision (mAP), intersection over union (IoU) of the bounding box, mean absolute error (MAE), and word error rate (WER).
Tab. {{table:0c365ff7-eab3-4b38-ad76-534f1f41742c}}  summarizes information of these ML models.


The Ryerson Audio-Visual Database of Emotional Speech and Song (RAVDESS): A dynamic, multimodal set of facial and vocal expressions in North American English

Abstract: The RAVDESS is a validated multimodal database of emotional speech and song. The database is gender balanced consisting of 24 professional actors, vocalizing lexically-matched statements in a neutral North American accent. Speech includes calm, happy, sad, angry, fearful, surprise, and disgust expressions, and song contains calm, happy, sad, angry, and fearful emotions. Each expression is produced at two levels of emotional intensity, with an additional neutral expression. All conditions are available in face-and-voice, face-only, and voice-only formats. The set of 7356 recordings were each rated 10 times on emotional validity, intensity, and genuineness. Ratings were provided by 247 individuals who were characteristic of untrained research participants from North America. A further set of 72 participants provided test-retest data. High levels of emotional validity and test-retest intrarater reliability were reported. Corrected accuracy and composite "goodness" measures are presented to assist researchers in the selection of stimuli. All recordings are made freely available under a Creative Commons license and can be downloaded at https://doi.org/10.5281/zenodo.1188976.

VoxCeleb: A Large-Scale Speaker Identification Dataset

Abstract: Most existing datasets for speaker identification contain samples obtained under quite constrained conditions, and are usually hand-annotated, hence limited in size. The goal of this paper is to generate a large scale text-independent speaker identification dataset collected 'in the wild'. We make two contributions. First, we propose a fully automated pipeline based on computer vision techniques to create the dataset from open-source media. Our pipeline involves obtaining videos from YouTube; performing active speaker verification using a two-stream synchronization Convolutional Neural Network (CNN), and confirming the identity of the speaker using CNN based facial recognition. We use this pipeline to curate VoxCeleb which contains hundreds of thousands of 'real world' utterances for over 1,000 celebrities. Our second contribution is to apply and compare various state of the art speaker identification techniques on our dataset to establish baseline performance. We show that a CNN based architecture obtains the best performance for both identification and verification.

VoxCeleb2: Deep Speaker Recognition

Abstract:
The objective of this paper is speaker recognition under noisy and unconstrained conditions.
We make two key contributions. First, we introduce a very large-scale audio-visual speaker recognition dataset collected from open-source media. Using a fully automated pipeline, we curate VoxCeleb2 which contains over a million utterances from over 6,000 speakers. This is several times larger than any publicly available speaker recognition dataset.
Second, we develop and compare Convolutional Neural Network (CNN) models and training strategies that can effectively recognise identities from voice under various conditions. The models trained on the VoxCeleb2 dataset surpass the performance of previous works on a benchmark dataset by a significant margin.

Deep Audio-Visual Speech Recognition

Abstract: The goal of this work is to recognise phrases and sentences being spoken by a talking face, with or without the audio. Unlike previous works that have focussed on recognising a limited number of words or phrases, we tackle lip reading as an open-world problem – unconstrained natural language sentences, and in the wild videos. Our key contributions are: (1) we compare two models for lip reading, one using a CTC loss, and the other using a sequence-to-sequence loss. Both models are built on top of the transformer self-attention architecture; (2) we investigate to what extent lip reading is complementary to audio speech recognition, especially when the audio signal is noisy; (3) we introduce and publicly release a new dataset for audio-visual speech recognition, LRS2-BBC, consisting of thousands of natural sentences from British television. The models that we train surpass the performance of all previous work on a lip reading benchmark dataset by a significant margin.

LRS3-TED: a large-scale dataset for visual speech recognition

Abstract: This paper introduces a new multi-modal dataset for visual and audio-visual speech recognition. It includes face tracks from over 400 hours of TED and TEDx videos, along with the corresponding subtitles and word alignment boundaries. The new dataset is substantially larger in scale compared to other public datasets that are available for general research.

MELD: A Multimodal Multi-Party Dataset for Emotion Recognition in Conversations

Abstract: Emotion recognition in conversations is a challenging task that has recently gained popularity due to its potential applications. Until now, however, a large-scale multimodal multi-party emotional conversational database containing more than two speakers per dialogue was missing. Thus, we propose the Multimodal EmotionLines Dataset (MELD), an extension and enhancement of EmotionLines. MELD contains about 13,000 utterances from 1,433 dialogues from the TV-series Friends. Each utterance is annotated with emotion and sentiment labels, and encompasses audio, visual and textual modalities. We propose several strong multimodal baselines and show the importance of contextual and multimodal information for emotion recognition in conversations. The full dataset is available for use at http://affective-meld.github.io.

The statistics of some existing datasets.

The recent large-scale audio-visual datasets played a crucial role in the success of talking-head video generation. On the one hand, the increasingly enriched datasets capture the diversity of visual contents in lighting conditions, identities, poses, video qualities, and phrases, enabling the training of robust talking-head models for realistic scenes. On the other hand, the audio, as well as other annotated attributes of the datasets provides meaningful ways for examing and comparing the performance of different algorithms.
In this section, we summarize the attributes of the recently released speech-related audio-visual datasets, ranging from lab-controlled to in-the-wild environment data (Tab. {{table:331ad9a2-fc34-4691-8d8a-29ed4aeaa7a6}}  and Fig. {{figure:636cf421-2caa-4a5d-976a-26f59b0044f6}} ) and select some representative ones (Tab. {{table:65ddde89-2295-4547-acc4-48ea299c32d5}} ) as the benchmark datasets for Sect. .

Tab. {{table:331ad9a2-fc34-4691-8d8a-29ed4aeaa7a6}}  lists many existing datasets and we select six datasets as our benchmark datasets, which can be categorize into 4 categories (Tab. {{table:65ddde89-2295-4547-acc4-48ea299c32d5}} ).
However, those datasets are not calibrated (e.g., scaling, cropping, face tracker, and head position).
LRW, VoxCeleb2, and LRS3-TED are videos that is pre-processed with certain protocols by using the face detector and tracker proposed in Nagrani et al. {{cite:21868b1}}.
This pre-processing method is not publicly available, leading to a problem that the generative models trained on these datasets may not be able to generate videos with samples outside of these datasets due to the lack of the pre-processing.
In this section, we introduce a uniformed pre-possessing protocol so that the generative models trained on one dataset can be easily transferred to test videos outside of the dataset.
The key stages of the pipeline are:


Unsupervised Visual Representation Learning by Context Prediction

Abstract: This work explores the use of spatial context as a source of free and plentiful supervisory signal for training a rich visual representation. Given only a large, unlabeled image collection, we extract random pairs of patches from each image and train a convolutional neural net to predict the position of the second patch relative to the first. We argue that doing well on this task requires the model to learn to recognize objects and their parts. We demonstrate that the feature representation learned using this within-image context indeed captures visual similarity across images. For example, this representation allows us to perform unsupervised visual discovery of objects like cats, people, and even birds from the Pascal VOC 2011 detection dataset. Furthermore, we show that the learned ConvNet can be used in the R-CNN framework and provides a significant boost over a randomly-initialized ConvNet, resulting in state-of-the-art performance among algorithms which use only Pascal-provided training set annotations.

Unsupervised Learning of Visual Representations by Solving Jigsaw Puzzles

Abstract: In this paper we study the problem of image representation learning without human annotation. By following the principles of self-supervision, we build a convolutional neural network (CNN) that can be trained to solve Jigsaw puzzles as a pretext task, which requires no manual labeling, and then later repurposed to solve object classification and detection. To maintain the compatibility across tasks we introduce the context-free network (CFN), a siamese-ennead CNN. The CFN takes image tiles as input and explicitly limits the receptive field (or context) of its early processing units to one tile at a time. We show that the CFN includes fewer parameters than AlexNet while preserving the same semantic learning capabilities. By training the CFN to solve Jigsaw puzzles, we learn both a feature mapping of object parts as well as their correct spatial arrangement. Our experimental evaluations show that the learned features capture semantically relevant content. Our proposed method for learning visual representations outperforms state of the art methods in several transfer learning benchmarks.

Unsupervised Representation Learning by Predicting Image Rotations

Abstract: Over the last years, deep convolutional neural networks (ConvNets) have transformed the field of computer vision thanks to their unparalleled capacity to learn high level semantic image features. However, in order to successfully learn those features, they usually require massive amounts of manually labeled data, which is both expensive and impractical to scale. Therefore, unsupervised semantic feature learning, i.e., learning without requiring manual annotation effort, is of crucial importance in order to successfully harvest the vast amount of visual data that are available today. In our work we propose to learn image features by training ConvNets to recognize the 2d rotation that is applied to the image that it gets as input. We demonstrate both qualitatively and quantitatively that this apparently simple task actually provides a very powerful supervisory signal for semantic feature learning. We exhaustively evaluate our method in various unsupervised feature learning benchmarks and we exhibit in all of them state-of-the-art performance. Specifically, our results on those benchmarks demonstrate dramatic improvements w.r.t. prior state-of-the-art approaches in unsupervised representation learning and thus significantly close the gap with supervised feature learning. For instance, in PASCAL VOC 2007 detection task our unsupervised pre-trained AlexNet model achieves the state-of-the-art (among unsupervised methods) mAP of 54.4% that is only 2.4 points lower from the supervised case. We get similarly striking results when we transfer our unsupervised learned features on various other tasks, such as ImageNet classification, PASCAL classification, PASCAL segmentation, and CIFAR-10 classification. The code and models of our paper will be published on: https://github.com/gidariss/FeatureLearningRotNet .

Context Encoders: Feature Learning by Inpainting

Abstract: We present an unsupervised visual feature learning algorithm driven by context-based pixel prediction. By analogy with auto-encoders, we propose Context Encoders -- a convolutional neural network trained to generate the contents of an arbitrary image region conditioned on its surroundings. In order to succeed at this task, context encoders need to both understand the content of the entire image, as well as produce a plausible hypothesis for the missing part(s). When training context encoders, we have experimented with both a standard pixel-wise reconstruction loss, as well as a reconstruction plus an adversarial loss. The latter produces much sharper results because it can better handle multiple modes in the output. We found that a context encoder learns a representation that captures not just appearance but also the semantics of visual structures. We quantitatively demonstrate the effectiveness of our learned features for CNN pre-training on classification, detection, and segmentation tasks. Furthermore, context encoders can be used for semantic inpainting tasks, either stand-alone or as initialization for non-parametric methods.

Split-Brain Autoencoders: Unsupervised Learning by Cross-Channel Prediction

Abstract: We propose split-brain autoencoders, a straightforward modification of the traditional autoencoder architecture, for unsupervised representation learning. The method adds a split to the network, resulting in two disjoint sub-networks. Each sub-network is trained to perform a difficult task -- predicting one subset of the data channels from another. Together, the sub-networks extract features from the entire input signal. By forcing the network to solve cross-channel prediction tasks, we induce a representation within the network which transfers well to other, unseen tasks. This method achieves state-of-the-art performance on several large-scale transfer learning benchmarks.

Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks

Abstract: In recent years, supervised learning with convolutional networks (CNNs) has seen huge adoption in computer vision applications. Comparatively, unsupervised learning with CNNs has received less attention. In this work we hope to help bridge the gap between the success of CNNs for supervised learning and unsupervised learning. We introduce a class of CNNs called deep convolutional generative adversarial networks (DCGANs), that have certain architectural constraints, and demonstrate that they are a strong candidate for unsupervised learning. Training on various image datasets, we show convincing evidence that our deep convolutional adversarial pair learns a hierarchy of representations from object parts to scenes in both the generator and discriminator. Additionally, we use the learned features for novel tasks - demonstrating their applicability as general image representations.

Adversarial Feature Learning

Abstract: The ability of the Generative Adversarial Networks (GANs) framework to learn generative models mapping from simple latent distributions to arbitrarily complex data distributions has been demonstrated empirically, with compelling results showing that the latent space of such generators captures semantic variation in the data distribution. Intuitively, models trained to predict these semantic latent representations given data may serve as useful feature representations for auxiliary problems where semantics are relevant. However, in their existing form, GANs have no means of learning the inverse mapping -- projecting data back into the latent space. We propose Bidirectional Generative Adversarial Networks (BiGANs) as a means of learning this inverse mapping, and demonstrate that the resulting learned feature representation is useful for auxiliary supervised discrimination tasks, competitive with contemporary approaches to unsupervised and self-supervised feature learning.

Representation Learning with Contrastive Predictive Coding

Abstract: While supervised learning has enabled great progress in many applications, unsupervised learning has not seen such widespread adoption, and remains an important and challenging endeavor for artificial intelligence. In this work, we propose a universal unsupervised learning approach to extract useful representations from high-dimensional data, which we call Contrastive Predictive Coding. The key insight of our model is to learn such representations by predicting the future in latent space by using powerful autoregressive models. We use a probabilistic contrastive loss which induces the latent space to capture information that is maximally useful to predict future samples. It also makes the model tractable by using negative sampling. While most prior work has focused on evaluating representations for a particular modality, we demonstrate that our approach is able to learn useful representations achieving strong performance on four distinct domains: speech, images, text and reinforcement learning in 3D environments.

Momentum Contrast for Unsupervised Visual Representation Learning

Abstract: We present Momentum Contrast (MoCo) for unsupervised visual representation learning. From a perspective on contrastive learning as dictionary look-up, we build a dynamic dictionary with a queue and a moving-averaged encoder. This enables building a large and consistent dictionary on-the-fly that facilitates contrastive unsupervised learning. MoCo provides competitive results under the common linear protocol on ImageNet classification. More importantly, the representations learned by MoCo transfer well to downstream tasks. MoCo can outperform its supervised pre-training counterpart in 7 detection/segmentation tasks on PASCAL VOC, COCO, and other datasets, sometimes surpassing it by large margins. This suggests that the gap between unsupervised and supervised representation learning has been largely closed in many vision tasks.

A Simple Framework for Contrastive Learning of Visual Representations

Abstract: This paper presents SimCLR: a simple framework for contrastive learning of visual representations. We simplify recently proposed contrastive self-supervised learning algorithms without requiring specialized architectures or a memory bank. In order to understand what enables the contrastive prediction tasks to learn useful representations, we systematically study the major components of our framework. We show that (1) composition of data augmentations plays a critical role in defining effective predictive tasks, (2) introducing a learnable nonlinear transformation between the representation and the contrastive loss substantially improves the quality of the learned representations, and (3) contrastive learning benefits from larger batch sizes and more training steps compared to supervised learning. By combining these findings, we are able to considerably outperform previous methods for self-supervised and semi-supervised learning on ImageNet. A linear classifier trained on self-supervised representations learned by SimCLR achieves 76.5% top-1 accuracy, which is a 7% relative improvement over previous state-of-the-art, matching the performance of a supervised ResNet-50. When fine-tuned on only 1% of the labels, we achieve 85.8% top-5 accuracy, outperforming AlexNet with 100X fewer labels.

Bootstrap Your Own Latent: A New Approach to Self-Supervised Learning

Abstract: We introduce Bootstrap Your Own Latent (BYOL), a new approach to self-supervised image representation learning. BYOL relies on two neural networks, referred to as online and target networks, that interact and learn from each other. From an augmented view of an image, we train the online network to predict the target network representation of the same image under a different augmented view. At the same time, we update the target network with a slow-moving average of the online network. While state-of-the art methods rely on negative pairs, BYOL achieves a new state of the art without them. BYOL reaches $74.3\%$ top-1 classification accuracy on ImageNet using a linear evaluation with a ResNet-50 architecture and $79.6\%$ with a larger ResNet. We show that BYOL performs on par or better than the current state of the art on both transfer and semi-supervised benchmarks. Our implementation and pretrained models are given on GitHub.

Unsupervised Learning of Visual Features by Contrasting Cluster Assignments

Abstract: Unsupervised image representations have significantly reduced the gap with supervised pretraining, notably with the recent achievements of contrastive learning methods. These contrastive methods typically work online and rely on a large number of explicit pairwise feature comparisons, which is computationally challenging. In this paper, we propose an online algorithm, SwAV, that takes advantage of contrastive methods without requiring to compute pairwise comparisons. Specifically, our method simultaneously clusters the data while enforcing consistency between cluster assignments produced for different augmentations (or views) of the same image, instead of comparing features directly as in contrastive learning. Simply put, we use a swapped prediction mechanism where we predict the cluster assignment of a view from the representation of another view. Our method can be trained with large and small batches and can scale to unlimited amounts of data. Compared to previous contrastive methods, our method is more memory efficient since it does not require a large memory bank or a special momentum network. In addition, we also propose a new data augmentation strategy, multi-crop, that uses a mix of views with different resolutions in place of two full-resolution views, without increasing the memory or compute requirements much. We validate our findings by achieving 75.3% top-1 accuracy on ImageNet with ResNet-50, as well as surpassing supervised pretraining on all the considered transfer tasks.

Summary of self-supervised learning pretext tasks.

Swapping assignments between multiple views

To sum up, we opted to provide a high-level overview of each of the previously discussed methods as this article is intended for self-supervised applications in medical imaging which renders it prone to nonspecialist readers from the medical field. One more point to mention is that despite the fact that these methods are developed on natural images, they can be transferred to the medical imaging field as we will see in the next section. Such property encouraged us to briefly discuss them before proceeding with the application of self-supervised learning in medical imaging. Table {{table:ceb128be-ea93-4686-9572-74c92f7bf4d7}}  summarizes the discussed pretext tasks according to their categories, while table {{table:79859376-94ed-4027-8921-dfe824d4473e}}  provides an access to the code repository of these works which is provided in appendix A.


Field study. Mixed-methods investigation of 3,634 Microsoft developers. Two surveys collected qualitative and quantitative insights about working from home conditions during the COVID-19 lockdown.

Sample study. Longitudinal study involving 192 software engineers living in countries with comparable COVID-19 lockdown measures, surveying 51 variables. Data were analyzed using correlations, multiple linear regressions, and covariance-based structural equation modeling to assess predictive-causal relations.

Quality of family life and time improved, although it might have led to a lack of focus, poor work-life boundaries, communications, and sync issues, developers adapt over time.

Well-being and productivity are related, professionals adapt to the condition over time, improving their well-being and productivity, introverts are disproportionally affected by the lockdown, no predictor variable was significantly able to causally explain the variance in well-being and productivity.

A Tale of Two Cities: Software Developers Working from Home during the COVID-19 Pandemic

Abstract: The COVID-19 pandemic has shaken the world to its core and has provoked an overnight exodus of developers who normally worked in an office setting to working from home. The magnitude of this shift and the factors that have accompanied this new unplanned work setting go beyond what the software engineering community has previously understood to be remote work. To find out how developers and their productivity were affected, we distributed two surveys (with a combined total of 3,634 responses that answered all required questions) weeks apart to understand the presence and prevalence of the benefits, challenges, and opportunities to improve this special circumstance of remote work. From our thematic qualitative analysis and statistical quantitative analysis, we find that there is a dichotomy of developer experiences influenced by many different factors (that for some are a benefit, while for others a challenge). For example, a benefit for some was being close to family members but for others having family members share their working space and interrupting their focus, was a challenge. Our surveys led to powerful narratives from respondents and revealed the scale at which these experiences exist to provide insights as to how the future of (pandemic) remote work can evolve.

Predictors of Well-being and Productivity among Software Professionals during the COVID-19 Pandemic -- A Longitudinal Study

Abstract: The COVID-19 pandemic has forced governments worldwide to impose movement restrictions on their citizens. Although critical to reducing the virus' reproduction rate, these restrictions come with far-reaching social and economic consequences. In this paper, we investigate the impact of these restrictions on an individual level among software engineers who were working from home. Although software professionals are accustomed to working with digital tools, but not all of them remotely, in their day-to-day work, the abrupt and enforced work-from-home context has resulted in an unprecedented scenario for the software engineering community. In a two-wave longitudinal study (N=192), we covered over 50 psychological, social, situational, and physiological factors that have previously been associated with well-being or productivity. Examples include anxiety, distractions, coping strategies, psychological and physical needs, office set-up, stress, and work motivation. This design allowed us to identify the variables that explained unique variance in well-being and productivity. Results include (1) the quality of social contacts predicted positively, and stress predicted an individual's well-being negatively when controlling for other variables consistently across both waves; (2) boredom and distractions predicted productivity negatively; (3) productivity was less strongly associated with all predictor variables at time two compared to time one, suggesting that software engineers adapted to the lockdown situation over time; and (4) longitudinal analyses did not provide evidence that any predictor variable causal explained variance in well-being and productivity. Overall, we conclude that working from home was per se not a significant challenge for software engineers.

Overview of prior studies about software engineers working from home

Several large software companies, such as Stack Overflow or Red Hat, have embraced working from home by designing ad hoc schemes {{cite:0eacb02}}, {{cite:01d62df}}.
Organizations do so to increase their employees' job satisfaction and productivity while simultaneously reducing their operating expenses, such as office rent {{cite:8dbd5aa}}, {{cite:65993ea}}.
However, thus far, the software engineering literature did not primarily investigate working from home challenges, with a few exceptions.
To find previous work, we looked into peer-reviewed publications in Scopus.
We identified six relevant papers.
Considering the vast but recent impact of COVID-19, we also selected non-peer-reviewed pre-prints on arXiv (three in total).
Table {{table:09c0a1e7-d070-4e65-9b1c-0572bb988656}}  summarizes prior studies of remote working issues related to software engineers.


TIE: A Framework for Embedding-based Incremental Temporal Knowledge Graph Completion

Abstract: Reasoning in a temporal knowledge graph (TKG) is a critical task for information retrieval and semantic search. It is particularly challenging when the TKG is updated frequently. The model has to adapt to changes in the TKG for efficient training and inference while preserving its performance on historical knowledge. Recent work approaches TKG completion (TKGC) by augmenting the encoder-decoder framework with a time-aware encoding function. However, naively fine-tuning the model at every time step using these methods does not address the problems of 1) catastrophic forgetting, 2) the model's inability to identify the change of facts (e.g., the change of the political affiliation and end of a marriage), and 3) the lack of training efficiency. To address these challenges, we present the Time-aware Incremental Embedding (TIE) framework, which combines TKG representation learning, experience replay, and temporal regularization. We introduce a set of metrics that characterizes the intransigence of the model and propose a constraint that associates the deleted facts with negative labels. Experimental results on Wikidata12k and YAGO11k datasets demonstrate that the proposed TIE framework reduces training time by about ten times and improves on the proposed metrics compared to vanilla full-batch training. It comes without a significant loss in performance for any traditional measures. Extensive ablation studies reveal performance trade-offs among different evaluation metrics, which is essential for decision-making around real-world TKG applications.

Learning Sequence Encoders for Temporal Knowledge Graph Completion

Abstract: Research on link prediction in knowledge graphs has mainly focused on static multi-relational data. In this work we consider temporal knowledge graphs where relations between entities may only hold for a time interval or a specific point in time. In line with previous work on static knowledge graphs, we propose to address this problem by learning latent entity and relation type representations. To incorporate temporal information, we utilize recurrent neural networks to learn time-aware representations of relation types which can be used in conjunction with existing latent factorization methods. The proposed approach is shown to be robust to common challenges in real-world KGs: the sparsity and heterogeneity of temporal expressions. Experiments show the benefits of our approach on four temporal KGs. The data sets are available under a permissive BSD-3 license.

TempoQR: Temporal Question Reasoning over Knowledge Graphs

Abstract:
Knowledge Graph Question Answering (KGQA) involves retrieving facts from a Knowledge Graph (KG) using natural language queries. A KG is a curated set of facts consisting of entities linked by relations. Certain facts include also temporal information forming a Temporal KG (TKG). Although many natural questions involve explicit or implicit time constraints, question answering (QA) over TKGs has been a relatively unexplored area. Existing solutions are mainly designed for simple temporal questions that can be answered directly by a single TKG fact.
 This paper puts forth a comprehensive embedding-based framework for answering complex questions over TKGs. Our method termed temporal question reasoning (TempoQR) exploits TKG embeddings to ground the question to the specific entities and time scope it refers to. It does so by augmenting the question embeddings with context, entity and time-aware information by employing three specialized modules. The first computes a textual representation of a given question, the second combines it with the entity embeddings for entities involved in the question, and the third generates question-specific time embeddings. Finally, a transformer-based encoder learns to fuse the generated temporal information with the question representation, which is used for answer predictions. Extensive experiments show that TempoQR improves accuracy by 25--45 percentage points on complex temporal questions over state-of-the-art approaches and it generalizes better to unseen question types.

Typical benchmark datasets for temporal knowledge graph reasoning.

This section summarized the 18 benchmark datasets for temporal KGR collected from the published papers. The statistic of them are presented in Tab. {{table:4b39a405-7f78-4f96-804f-141cd956888a}} , and the descriptions are listed below.


Interpretation of Neural Networks is Fragile

Abstract: In order for machine learning to be trusted in many applications, it is critical to be able to reliably explain why the machine learning algorithm makes certain predictions. For this reason, a variety of methods have been developed recently to interpret neural network predictions by providing, for example, feature importance maps. For both scientific robustness and security reasons, it is important to know to what extent can the interpretations be altered by small systematic perturbations to the input data, which might be generated by adversaries or by measurement biases. In this paper, we demonstrate how to generate adversarial perturbations that produce perceptively indistinguishable inputs that are assigned the same predicted label, yet have very different interpretations. We systematically characterize the robustness of interpretations generated by several widely-used feature importance interpretation methods (feature importance maps, integrated gradients, and DeepLIFT) on ImageNet and CIFAR-10. In all cases, our experiments show that systematic perturbations can lead to dramatically different interpretations without changing the label. We extend these results to show that interpretations based on exemplars (e.g. influence functions) are similarly susceptible to adversarial attack. Our analysis of the geometry of the Hessian matrix gives insight on why robustness is a general challenge to current interpretation approaches.

How to Manipulate CNNs to Make Them Lie: the GradCAM Case

Abstract: Recently many methods have been introduced to explain CNN decisions. However, it has been shown that some methods can be sensitive to manipulation of the input. We continue this line of work and investigate the explanation method GradCAM. Instead of manipulating the input, we consider an adversary that manipulates the model itself to attack the explanation. By changing weights and architecture, we demonstrate that it is possible to generate any desired explanation, while leaving the model's accuracy essentially unchanged. This illustrates that GradCAM cannot explain the decision of every CNN and provides a proof of concept showing that it is possible to obfuscate the inner workings of a CNN. Finally, we combine input and model manipulation. To this end we put a backdoor in the network: the explanation is correct unless there is a specific pattern present in the input, which triggers a malicious explanation. Our work raises new security concerns, especially in settings where explanations of models may be used to make decisions, such as in the medical domain.

Fooling Neural Network Interpretations via Adversarial Model Manipulation

Abstract: We ask whether the neural network interpretation methods can be fooled via adversarial model manipulation, which is defined as a model fine-tuning step that aims to radically alter the explanations without hurting the accuracy of the original models, e.g., VGG19, ResNet50, and DenseNet121. By incorporating the interpretation results directly in the penalty term of the objective function for fine-tuning, we show that the state-of-the-art saliency map based interpreters, e.g., LRP, Grad-CAM, and SimpleGrad, can be easily fooled with our model manipulation. We propose two types of fooling, Passive and Active, and demonstrate such foolings generalize well to the entire validation set as well as transfer to other interpretation methods. Our results are validated by both visually showing the fooled explanations and reporting quantitative metrics that measure the deviations from the original explanations. We claim that the stability of neural network interpretation method with respect to our adversarial model manipulation is an important criterion to check for developing robust and reliable neural network interpretation method.

Explanations can be manipulated and geometry is to blame

Abstract: Explanation methods aim to make neural networks more trustworthy and interpretable. In this paper, we demonstrate a property of explanation methods which is disconcerting for both of these purposes. Namely, we show that explanations can be manipulated arbitrarily by applying visually hardly perceptible perturbations to the input that keep the network's output approximately constant. We establish theoretically that this phenomenon can be related to certain geometrical properties of neural networks. This allows us to derive an upper bound on the susceptibility of explanations to manipulations. Based on this result, we propose effective mechanisms to enhance the robustness of explanations.

"How do I fool you?": Manipulating User Trust via Misleading Black Box Explanations

Abstract: As machine learning black boxes are increasingly being deployed in critical domains such as healthcare and criminal justice, there has been a growing emphasis on developing techniques for explaining these black boxes in a human interpretable manner. There has been recent concern that a high-fidelity explanation of a black box ML model may not accurately reflect the biases in the black box. As a consequence, explanations have the potential to mislead human users into trusting a problematic black box. In this work, we rigorously explore the notion of misleading explanations and how they influence user trust in black box models. Specifically, we propose a novel theoretical framework for understanding and generating misleading explanations, and carry out a user study with domain experts to demonstrate how these explanations can be used to mislead users. Our work is the first to empirically establish how user trust in black box models can be manipulated via misleading explanations.

Fairwashing Explanations with Off-Manifold Detergent

Abstract: Explanation methods promise to make black-box classifiers more transparent. As a result, it is hoped that they can act as proof for a sensible, fair and trustworthy decision-making process of the algorithm and thereby increase its acceptance by the end-users. In this paper, we show both theoretically and experimentally that these hopes are presently unfounded. Specifically, we show that, for any classifier $g$, one can always construct another classifier $\tilde{g}$ which has the same behavior on the data (same train, validation, and test error) but has arbitrarily manipulated explanation maps. We derive this statement theoretically using differential geometry and demonstrate it experimentally for various explanation methods, architectures, and datasets. Motivated by our theoretical insights, we then propose a modification of existing explanation methods which makes them significantly more robust.

Interpretable Deep Learning under Fire

Abstract:
Providing explanations for deep neural network (DNN) models is crucial for their use in security-sensitive domains. A plethora of interpretation models have been proposed to help users understand the inner workings of DNNs: how does a DNN arrive at a specific decision for a given input? The improved interpretability is believed to offer a sense of security by involving human in the decision-making process. Yet, due to its data-driven nature, the interpretability itself is potentially susceptible to malicious manipulations, about which little is known thus far.
Here we bridge this gap by conducting the first systematic study on the security of interpretable deep learning systems (IDLSes). We show that existing \imlses are highly vulnerable to adversarial manipulations. Specifically, we present ADV^2, a new class of attacks that generate adversarial inputs not only misleading target DNNs but also deceiving their coupled interpretation models. Through empirical evaluation against four major types of IDLSes on benchmark datasets and in security-critical applications (e.g., skin cancer diagnosis), we demonstrate that with ADV^2 the adversary is able to arbitrarily designate an input's prediction and interpretation. Further, with both analytical and empirical evidence, we identify the prediction-interpretation gap as one root cause of this vulnerability -- a DNN and its interpretation model are often misaligned, resulting in the possibility of exploiting both models simultaneously. Finally, we explore potential countermeasures against ADV^2, including leveraging its low transferability and incorporating it in an adversarial training framework. Our findings shed light on designing and operating IDLSes in a more secure and informative fashion, leading to several promising research directions.

Perturbing Inputs for Fragile Interpretations in Deep Natural Language Processing

Abstract: Interpretability methods like Integrated Gradient and LIME are popular choices for explaining natural language model predictions with relative word importance scores. These interpretations need to be robust for trustworthy NLP applications in high-stake areas like medicine or finance. Our paper demonstrates how interpretations can be manipulated by making simple word perturbations on an input text. Via a small portion of word-level swaps, these adversarial perturbations aim to make the resulting text semantically and spatially similar to its seed input (therefore sharing similar interpretations). Simultaneously, the generated examples achieve the same prediction label as the seed yet are given a substantially different explanation by the interpretation methods. Our experiments generate fragile interpretations to attack two SOTA interpretation methods, across three popular Transformer models and on three different NLP datasets. We observe that the rank order correlation and top-K intersection score drops by over 20% when less than 10% of words are perturbed on average. Further, rank-order correlation keeps decreasing as more words get perturbed. Furthermore, we demonstrate that candidates generated from our method have good quality metrics.

Counterfactual Explanations Can Be Manipulated

Abstract: Counterfactual explanations are emerging as an attractive option for providing recourse to individuals adversely impacted by algorithmic decisions. As they are deployed in critical applications (e.g. law enforcement, financial lending), it becomes important to ensure that we clearly understand the vulnerabilities of these methods and find ways to address them. However, there is little understanding of the vulnerabilities and shortcomings of counterfactual explanations. In this work, we introduce the first framework that describes the vulnerabilities of counterfactual explanations and shows how they can be manipulated. More specifically, we show counterfactual explanations may converge to drastically different counterfactuals under a small perturbation indicating they are not robust. Leveraging this insight, we introduce a novel objective to train seemingly fair models where counterfactual explanations find much lower cost recourse under a slight perturbation. We describe how these models can unfairly provide low-cost recourse for specific subgroups in the data while appearing fair to auditors. We perform experiments on loan and violent crime prediction data sets where certain subgroups achieve up to 20x lower cost recourse under the perturbation. These results raise concerns regarding the dependability of current counterfactual explanation techniques, which we hope will inspire investigations in robust counterfactual explanations.

Making Corgis Important for Honeycomb Classification: Adversarial Attacks on Concept-based Explainability Tools

Abstract: Methods for model explainability have become increasingly critical for testing the fairness and soundness of deep learning. Concept-based interpretability techniques, which use a small set of human-interpretable concept exemplars in order to measure the influence of a concept on a model’s internal representation of input, are an important thread in this line of research. In this work we show that these explainability methods can suffer the same vulnerability to adversarial attacks as the models they are meant to analyze. We demonstrate this phenomenon on two well-known concept-based interpretability methods: TCAV and faceted feature visualization. We show that by leveraging the geometry of the problem and carefully perturbing the examples of the concept that is being investigated, we can radically change the output of the interpretability method. The attacks that we propose can either induce positive interpretations (polka dots are an important concept for a model when classifying zebras) or negative interpretations (stripes are not an important factor in identifying images of a zebra). Our work highlights the fact that in safety-critical applications, there is need for security around not only the machine learning pipeline but also the model interpretation process.

On the Privacy Risks of Algorithmic Recourse

Abstract: As predictive models are increasingly being employed to make consequential decisions, there is a growing emphasis on developing techniques that can provide algorithmic recourse to affected individuals. While such recourses can be immensely beneficial to affected individuals, potential adversaries could also exploit these recourses to compromise privacy. In this work, we make the first attempt at investigating if and how an adversary can leverage recourses to infer private information about the underlying model's training data. To this end, we propose a series of novel membership inference attacks which leverage algorithmic recourse. More specifically, we extend the prior literature on membership inference attacks to the recourse setting by leveraging the distances between data instances and their corresponding counterfactuals output by state-of-the-art recourse methods. Extensive experimentation with real world and synthetic datasets demonstrates significant privacy leakage through recourses. Our work establishes unintended privacy leakage as an important risk in the widespread adoption of recourse methods.

Summary of adversarial attacks on explanations of machine learning models. We abbreviate the following: data (D), model (M), image (I), tabular (T), language (Lg), neural network (N), black-box (B), local (L), global (G). Appendix  lists other abbreviations.

Adversarial attacks on model explanations

Table {{table:ba667c20-3a33-4d95-98e4-954410f171ad}}  lists attacks on explanation methods, with the corresponding strategy of changing data, e.g. an adversarial example manipulates the explanation without impacting the prediction {{cite:8391906}}, {{cite:cde21fb}}, changing the model, e.g. fine-tuning or regularizing weights manipulates explanations without impacting the predictive performance {{cite:0cc3d8d}}, {{cite:8ed64d7}}, or changing both data and the model, e.g. in the case when an attacker poisons the training dataset {{cite:e23d526}}. {{cite:c14815b}} manipulate Grad-CAM explanations of a convolutional neural network by changing its weights, but also proposes to leave a backdoor in the network (triggered by specific input patterns), which allows retrieving original explanations. {{cite:e3e2a42}} extend fooling explanations through fine-tuning and backdooring to consider: (i) a red-herring attack that manipulates the explanation to cover an adversarial change in the model's prediction, e.g. a misclassification, and (ii) a fully disguising attack that aims to show the original explanation for a changed prediction.

For each of the attacks in Table {{table:ba667c20-3a33-4d95-98e4-954410f171ad}} , we record the mentioned data modalities with the corresponding datasets used in experiments, as well as model algorithms. Observation 1. The majority of proposed attacks assume prior knowledge that the explained model is a neural network, e.g. to utilize gradient descent in constructing adversarial examples or changing model parameters, as opposed to black-box approaches that could work with various model algorithms.

Comparing Tables {{table:ba667c20-3a33-4d95-98e4-954410f171ad}}  & {{table:112c6e07-9b09-44f9-a002-b11847d06dcd}}  highlights existing insecurities in XAI methods; namely, not clearly addressed are backdooring attacks {{cite:c14815b}}, {{cite:e3e2a42}}, data poisoning attacks {{cite:e23d526}}, {{cite:6283e99}}, attacks specific to language {{cite:6fa6944}} and concept-based explainability {{cite:46cc821}}.