In this work we continue to build upon recent advances in reinforcement learning for finite Markov processes. A common approach among previous existing algorithms, both single-actor and distributed, is to either clip rewards or to apply a transformation method on Q-functions to handle a large variety of magnitudes in real discounted returns. We theoretically show that one of the most successful methods may not yield an optimal policy if we have a non-deterministic process. As a solution, we argue that distributional reinforcement learning lends itself to remedy this situation completely. By the introduction of a conjugated distributional operator we may handle a large class of transformations for real returns with guaranteed theoretical convergence. We propose an approximating single-actor algorithm based on this operator that trains agents directly on unaltered rewards using a proper distributional metric given by the Cramér distance. To evaluate its performance in a stochastic setting we train agents on a suite of 55 Atari 2600 games using sticky-actions and obtain state-of-the-art performance compared to other well-known algorithms in the Dopamine framework.

It is well known that ensemble methods often provide enhanced performance in reinforcement learning. In this paper, we explore this concept further by using group-aided training within the distributional reinforcement learning paradigm. Specifically, we propose an extension to categorical reinforcement learning, where distributional learning targets are implicitly based on the total information gathered by an ensemble. We empirically show that this may lead to much more robust initial learning, a stronger individual performance level, and good efficiency on a per-sample basis.

In this paper we study the geometric location of periodic points of power series defined over fields of prime characteristic p. More specifically, we find a lower bound for the absolute value of all periodic points in the open unit disk of minimal period pⁿ of 2-ramified power series. We prove that this bound is optimal for a large class of power series. Our main technical result is a computation of the first significant terms of the pnth iterate of 2-ramified power series. As a by-product we obtain a self-contained proof of the characterization of 2-ramified power series.

We study germs in one variable having a parabolic fixed point at the origin, over an ultrametric ground field of positive characteristic. It is conjectured that for such a germ the origin is isolated as a periodic point. Our main result is an affirmative solution of this conjecture in the case of a generic germ with a prescribed multiplier. The genericity condition is explicit: That the power series is minimally ramified, i.e., that the degree of the first nonlinear term of each of its iterates is as small as possible. Our main technical result is a computation of the first significant terms of a minimally ramified power series. From this we obtain a lower bound for the norm of nonzero periodic points, from which we deduce our main result. As a by-product we give a new and self-contained proof of a characterization of minimally ramified power series in terms of the iterative residue.

We study ultrametric germs in one variable having an irrationally indifferent fixed point at the origin with a prescribed multiplier. We show that for many values of the multiplier, the cycles in the unit disk of the corresponding monic quadratic polynomial are "optimal" in the following sense: They minimize the distance to the origin among cycles of the same minimal period of normalized germs having an irrationally indifferent fixed point at the origin with the same multiplier. We also give examples of multipliers for which the corresponding quadratic polynomial does not have optimal cycles. In those cases we exhibit a higher degree polynomial such that all of its cycles are optimal. The proof of these results reveals a connection between the geometric location of periodic points of ultrametric power series and the lower ramification numbers of wildly ramified field automorphisms. We also give an extension of Sen's theorem on wildly ramified field automorphisms that is of independent interest.

We find the exact radius of linearization disks at indifferentfixed points ofquadratic maps in C_p. We also show thatthe radius is invariant under power series perturbations.Localizing all periodic orbits of these quadratic-like maps wethen show that periodic points are not the only obstruction for linearization. In so doing, we provide the first known examples in the dynamics ofpolynomials over C_p where the boundary of the linearization disk does not contain any periodic point.

We continue the study of the linearizability near an indifferent fixed point of a power series f, defined over a field of prime characteristic p. It is known since the work of Herman and Yoccoz in 1981 that Siegel’s linearization theorem is true also for non-Archimedean fields. However, they also showed that the condition in Siegel’s theorem is ‘usually’ not satisfied over fields of prime characteristic. Indeed, as proven by the author in a former paper, there exist power series f such that the associated conjugacy function diverges. We prove that if the degrees of the monomials of a power series f are divisible by p, then f is analytically linearizable. We find a lower (sometimes the best) bound of the size of the corresponding linearization disc. In the cases where we find the exact size of the linearization disc, we show, using the Weierstrass degree of the conjugacy, that f has an indifferent periodic point on the boundary. We also give a class of polynomials containing a monomial of degree prime to p, such that the conjugacy diverges.

Let K be a complete ultrametric field. We give lower and upper bounds for the size of linearization discs for power series over K near hyperbolic fixed points. These estimates are maximal in the sense that there exist examples where these estimates give the exact size of the corresponding linearization disc. In particular, at repelling fixed points, the linearization disc is equal to the maximal disc on which the power series is injective.

Let K be a complete ultrametric field of characteristic zero whose corresponding residue field k is also of characteristic zero. We give lower and upper bounds for the size of linearization disks for power series over K near an indifferent fixed point. These estimates are maximal in the sense that there exist examples where these estimates give the exact size of the corresponding linearization disc. Similar estimates in the remaining cases, i.e. the cases in which K is either a p-adic field or a field of prime characteristic, were obtained in various papers on the p-adic case [5, 18, 35, 42] later generalized in [28, 30], and in [29, 31] concerning the prime characteristic case.