Paavo Parmas' Research Website

Description: Introduces OrderGrad, a family of policy gradient (RL) and reparameterization gradient estimators for order-statistic objectives. While standard RL targets the mean reward, OrderGrad targets objectives such as CVaR, medians, trimmed means, top-m, and best-of-K by changing rank weights rather than rewriting the whole training algorithm. Using our code, the implementation is simple and requires only one line of code for a reward transformation. Everything else about your code, whether you use GRPO, PPO, etc., can remain the same. The approach is also orthogonal to these algorithms and can be combined with whichever one you want. From a technical perspective, the method is a generalization of existing max@K optimization methods—max@K is the special case of the top rank, while OrderGrad generalizes to arbitrary weightings of different ranks.

My contribution: Conceptualized the method, did the theoretical derivations, created the OrderGrad code and did the toy experiments, proposed the TopM-BottomM joint experiment, led the project, and led the writing. See the paper for the other contributions.

Description: Studies advantage estimation for pass@K and max@K policy gradients, especially the role of baselines and advantage centering. The paper introduces a Leave-Two-Out baseline and MaxPO for group-based reinforcement learning in LLM post-training. A key finding is that the L2O baseline can cancel out any comparator-based baseline that depends on the other K-1 samples in the set. In essence, this means that our method may be the most natural baseline for set objectives, such as max@K. An earlier version of this preprint was submitted to ICML2026 but not accepted (OpenReview).

My contribution: Equal contribution; conceptualization, all key theoretical derivations, project oversight, example code, and significant comments and editing of the paper. Other key contributors did the experiments and writing (see the author contributions section in the paper for details).

Description: Extends retry-based objectives such as max@K/ReMax to continuous action spaces using pathwise derivative estimators, and introduces ReMAC, an actor-critic implementation.

My contribution: Mentored Soichiro Nishimori, advised on the theoretical analysis, algorithm design, and experimental design, co-conceptualized the idea, and revised the manuscript.

Description: Studies a stochastic bandit algorithm motivated by retry-aware objectives such as pass@K and max@K, and proves a sublinear regret bound for a Gaussian two-retry setting. This work is interesting because it provides a theoretical backing for using these popular objectives for exploration.

My contribution: Proposed the algorithm, created the initial implementation and preliminary Gaussian bandit experiments, derived the initial expected-improvement balance condition, initiated the theoretical collaboration, and oversaw the project. Most of the hard theoretical derivations were done by Bingkui and Junpei, and the final experiments were done by Soichiro.

Description: Introduces ReMax, a retry-based reinforcement learning objective that evaluates a policy by the expected maximum return over multiple action samples, and promotes exploration. The motivation for the approach is that agents only benefit from exploration if they encounter the same or similar situation multiple times—if each state is unique, then the greedy (maximum expectation) action would be optimal each time. But due to uncertainty about the rewards, and the ability to retry at later encounters, it becomes beneficial to explore to increase future rewards. This motivation naturally led to the ReMax objective. This objective leads to the emergence of a stochastic exploratory policy despite the objective being greedy maximization of the reward. The policy becomes exploratory without explicit exploration bonuses, such as entropy bonuses. The paper derives an unbiased policy-gradient estimator for optimizing objectives defined over sets of sampled actions (now also called SetRL) through our expected-improvement formulation. An earlier version of this idea appeared in our 2022 preprint, where resetting leads to objectives over trajectory-trees (OpenReview), which also considered uncertain reward functions and showed that, under the ReMax objective, the optimal policy generally becomes stochastic. This contrasts with standard reinforcement learning, where the optimal policy is deterministic. The 2026 paper extends these ideas into a practical policy-gradient algorithm (RePPO). I believe the linked 2022 preprint was the first work to derive the basic set policy gradient, and later works such as Vector Policy Optimization and other SetRL variants optimize objectives that are mathematically special cases (often with some specific focus and impressive scaled up experiments, or with improved gradient estimation techniques).

My contribution: The project started in 2021 when I proposed the idea to Sotetsu Koyamada who initially worked on it based on my guidance. I moved to UTokyo and the project restarted in August 2024 when Soichiro Nishimori started working on it as his internship project under my guidance. My contributions in the final manuscript are as follows: I conceptualized the idea; did all of the key mathematical derivations (derivation of the gradient estimator, the expected improvement formulation, proposal of the final version of continuous m, came up with the bandit algorithm, proposed RePPO, hypothesized the adaptive properties of ReMax, e.g., the ones in Figure 1); example implementations (first REINFORCE implementation for ReMax, first ReMax bandit implementations, some debugging); significant comments and editing on the paper. The first author Soichiro, who I mentored, did most of the final implementations, experimentation and was the lead writer (see author contributions in the paper for full details).

Description: Re-examines policy-gradient learning with differentiable simulators under discontinuities, introducing DDCG for estimator switching in nonsmooth regions and IVW-H for variance control in differentiable robotics control tasks. One of the key findings is that variance, not bias, may be the major culprit in gradient-based learning with differentiable simulators—at least on the standard continuous control robot benchmarks considered in this paper. The IVW-H method is similar to my previous total propagation in the sense that it also has separate time-step-wise weights for combining estimators, but the implementation is simpler on differentiable simulators. I expect that total propagation would perform better though. This work showed that composite gradient estimators can yield large improvements in performance on standard benchmark tasks.

My contribution: Conceptualized the idea; derived most equations including the DDCG method; made the first IVW implementation for the DFlex simulator tasks; proposed IVW-H implementation details; oversaw the project; and contributed interpretation, comments, and edits.

Description: Separates the horizon used to generate model-based rollout distributions from the horizon used for policy-gradient training, balancing distribution shift, model bias, and gradient instability. The main strength of this paper is the empirical performance. Both the computational and data efficiency are several times better than the standard MBPO algorithm. Moreover, the hyperparameters are fixed and shared across all of the tasks, unlike the standard MBPO. Please use this as a baseline instead of MBPO or more generally, as a simple ensemble model-based RL baseline.

My contribution: Advised Akihiro Kubo through the project, making many suggestions about technical points, experiments and about the writing. Helped debug the code.

Description: Scales composite policy-gradient estimators into modern model-based reinforcement learning, combining likelihood ratio and reparameterization-style gradients in a practical algorithm. The new TotalPropagation X algorithm is considerably simpler to use and implement compared to the original TotalPropagation.

My contribution: Invented TPX, wrote and debugged most of the code, ran the experiments, performed the analysis, and wrote the paper.

Description: Introduces automatic propagation, a message-passing framework that generalizes automatic differentiation and allows custom learning algorithms to be packaged as reusable propagators. The main motivation for this work was to create a framework that allows automatically using the total propagation algorithm and other methods that I had created. These algorithms are not easily implemented using standard automatic differentiation software, and Proppo is a wrapper on top of AD that alters and automates specialized computations. Additionally, the chaotic recurrent neural network toy experiments in the paper showed that total propagation can improve the gradient estimation accuracy by ~100 times compared to naïve gradient combination schemes with no time-step-wise weighting.

My contribution: I invented automatic propagation software, designed and implemented most of the code including the key parts, ran the experiments, and wrote the paper. Takuma worked part-time to help write the code and give advice on the software engineering aspects in the early stages of the project. He also gave comments on the paper.

Description: Gives a first-principles view of likelihood ratio (LR) and reparameterization (RP) gradients as two ways of tracking the movement of probability mass, connecting the estimators via the divergence theorem. In particular, the LR estimator becomes an integral over the surface, while the RP estimator becomes an integral over the volume. Additionally, it characterizes the space of all gradient estimators that are constructed as linear combinations of the function value and its derivative. This includes LR and RP as special cases.

My contribution: I conceputalized the idea, did all of the mathematical analysis, the discussion, the experiments, and wrote the paper.

Description: I worked on this project as an intern at DeepMind. The work studies multiagent learning using policy-gradient dynamics inspired by hedging and replicator dynamics. Proposes the Neural Replicator Dynamics method for multiagent learning.

My contribution: My contributions were mostly in mathematically and computationally analyzing the algorithm. This led to a better understanding and influenced the writing of some sections, as well as influenced which experimental results were included.

Description: Develops a collection of stochastic-gradient algorithms for improved Monte Carlo gradient estimation, motivated by chaos-like gradient variance in model-based reinforcement learning. The thesis includes PIPPS, total propagation, probabilistic computation graphs, policy-gradient estimators, and a unified view of likelihood ratio and reparameterization gradients.

My contribution: Single-author PhD dissertation.

Description: This is my most impactful work to date but I feel it could have been even more impactful. The starting point of the research was to develop a trajectory-sampling-based method for policy optimization in model-based RL; however, the key findings are more general. The core contributions are: (1) demonstrated that chaos causes gradient variance explosion in long chains of non-linear computations (e.g., model-based RL, deep learning, time-series models, etc., while the focus was on MBRL); (2) showed that 0th-order likelihood ratio gradients can be drastically better than 1st-order backpropagation/reparameterization gradients in practical problems (at the time, it was widely believed that you should always use 1st-order gradients when available); (3) developed a mixed gradient estimator that takes the average of 0th-order and 1st-order gradient estimators k g₀ + (1 − k) g₁, and created a step-wise combination scheme total propagation as a replacement for backpropagation.

My contribution: This work grew out of my lab visits to Carl Edward Rasmussen's lab at the University of Cambridge, and Jan Peters' lab at TU Darmstadt for ~3 months each. The original particle sampling idea was also tried in Andrew McHutchon's PhD thesis, but they couldn't get it to work. My initial idea was to use probabilistic line searches to try to obtain good performance with this method. However, while running the experiments, I ran into problems with unstable learning and tried to figure out what the issue was. At the time, I was at Carl's lab, and he contributed to figuring out the issues, e.g., by suggesting plotting the landscapes and the gradients, and frequently discussing the issue with me. After leaving his lab, I continued analyzing the problem, and figured it out. To try to obtain better gradient estimates, I figured that likelihood ratio gradients may be better. My initial performance with these methods was not great, and I thought that baseline techniques may be necessary to improve the performance. I showed the results to Jan Peters and discussed baseline techniques, and he urged me to add a variance reduction baseline. Finally, after returning to my home institute, I combined both likelihood ratio and reparameterization gradients with a new total propagation algorithm and achieved good performance. Kenji Doya was my PhD advisor and helped improve the clarity of the paper. In summary, I completed the experimental analysis, decided on the experimental details (such as using fixed seeds, errorbars, etc.), ran the experiments, created the particle-sampling-based implementation, proposed and implemented the idea of using inverse-variance weighting for combining gradients as well as the total propagation algorithm, did the mathematical derivations and wrote the paper.

Description: Uses the total derivative rule as a visual and algorithmic framework for constructing gradient estimators on stochastic computation graphs, including reinforcement-learning applications. For example, it gives a unified derivation for the stochastic and deterministic policy gradient theorems, and allows deriving them by inspection visually from the graph.

My contribution: Single-author paper.