One parameter is always enough

We construct an elementary equation with a single real valued parameter that is capable of fitting any “scatter plot” on any number of points to within a fixed precision. Specifically, given given a fixed > 0, we may construct fθ so that for any collection of ordered pairs {(xj , yj)}j=0 with n, xj ∈ N and yj ∈ (0, 1), there exists a θ ∈ [0, 1] giving |fθ(xj)− yj | < for all j simultaneously. To achieve this, we apply prior results about the logistic map, an iterated map in dynamical systems theory that can be solved exactly. The existence of an equation fθ with this property highlights that “parameter counting” fails as a measure of model complexity when the class of models under consideration is only slightly broad.

The mathematician John von Neumann famously admonished that with four free parameters he could make an elephant, and with five he could make it wiggle its trunk [1].Indeed, the number of free parameters is often taken as a proxy of model complexity intuitively as well as in quantitative model comparison measures like AIC [2] and BIC [3].While these measures can be shown to be statistically principled or optimal for certain classes of models [4], they are often used to evaluate arbitrary models by practitioners in a given field.The aim of this short note is to show that, in fact, very simple, elementary models exist that are capable of fitting arbitrarily many points to an arbitrary precision using only a single real-valued parameter θ.This is not always due to severe pathologies-one such model, studied here, is infinitely continuously differentiable as a function of θ.The existence of this model has implications for statistical model comparison, and shows that great care must be taken in machine learning efforts to discover equations from data [5][6][7] since some simple models can fit any data set arbitrarily well.
We will consider the simple setting of a "scatter plot" of x-values at natural numbers 0, 1, 2, 3, . . ., n and y-values in (0, 1).We show how to construct an elementary function f θ that, as θ varies, may fit any collection of ordered pairs {(x i , y j )} n j=0 to an arbitrary precision > 0. The existence of a solution for x = 0, 1, 2, . . ., n implies the existence of solution for any subset of these integers.The general approach taken here will be to first find an initial condition of a chaotic dynamical system whose orbit comes close to values related to each y j .Then, an exact solution of this dynamical system yields an equation y = f θ (x) that, as x varies, recovers the system's dynamics with initial condition θ.This approach is related to attempts to encode computations into chaotic dynamical systems [8,9].The techniques deployed here are not novel mathematically, but this lesson from dynamical systems theory has not been explicitly articulated in the literature on statistics and model comparison.
We will make use of the logistic map m(z) = 4z(1 − z) whose iterated application can be solved exactly [10] for a given initial value θ as This solution follows from the double angle identity, and the requirement that m 0 (θ) = θ.The map m is chaotic [11] and it is well-established that m may be viewed as a shift map on θ through its conjugacy via ϕ(z) = sin 2 (2πz) to the Bernoulli map, S has the effect of removing the first bit of a binary expansion 0.z This property of S means that we may construct a point ω ∈ (0, 1) whose orbit under S will bring it arbitrarily close to each member of any collection of points.Specifically, let us fix > 0 and choose r ∈ N so that 2 −r < /2.We will define y j = ϕ −1 (y j ) and denote the binary expansion of y j as 0.y j1 y j2 y j3 • • • .Define the parameter value ω ∈ (0, 1) by concatenating the first r binary digits of each y j , Due to the construction of ω and the ability to interpret S as removing the leftmost bit, S rj (ω) agrees with y j on its first r bits, so, The ability to construct such an orbit relies ultimately on the fact that S is continuous and topologically mixing.Since ϕ is a homeomorphism between S and m, S rj = ϕ −1 • m rj • ϕ.
Putting these two facts together with (6) yields that for all j, where the last inequality follows the Lipschitz condition and application of ϕ to each term inside the absolute value.This presentation has elided one technical factor, which is that ϕ is not one-toone on (0, 1) and so ϕ −1 has two possible values.This has the consequence that in (7), m rj (ϕ(ω)))) may be close to either y j or 1 − y j , since ϕ is symmetrical about 1/2.To address this, we may always use the lower value for ϕ −1 and scale the y j so that they are always below 0.5.This scaling may then be inverted in the output of the final equation if desired.
Equation (7) shows that m rj will come close to each of the y j when started on value θ = ϕ(ω).
Thus, we may define a single parameter equation, where choosing θ = ϕ(ω) yields that |f θ (x j ) − y j | < for all j.Of course, the y j were freely chosen, showing that f θ can approximate any data set {(x j , y j )} n j=0 as θ varies in [0, 1].Note that for a fixed , the number of data points n that can be fit is not bounded.In addition, this f θ is continuous and differentiable for fixed r-indeed infinitely continuously differentiable-and so it will satisfy nearly all regularity conditions that would normally weed out such pathological functions in the context of parameter estimation.
To illustrate that (8) can fit an arbitrary data set as θ varies, Figure 1 shows the value f θ (0), f θ (1), f θ (2), . . ., for two values of θ.Each parameter value was created by following the construction above using target y j chosen at each x value from the black pixels of a line drawing of either an elephant (left) or signature (right).The implementation used the arbitrary precision library mpmath in python [12].Both are able to be fit well by f θ (x) if θ is appropriately tuned.This single parameter model provides a large improvement over the prior state of the art in fitting an elephant [13,14].Note that the only shown x values are integers, and between these integers are the rapidly oscillating sin patterns implied by (8).
The equation f θ (x) is extraordinarily sensitive to its single parameter θ and in fact will generalize to x > n in ways that depend only on the digits of θ are after the last digit of y n .Thus, while fitting the data, generalization behavior is completely determined by the free parameter's less significant digits.This implies that there can be no guarantees about the performance of f θ in extrapolation, despite its good fit.Thus, the construction shows that even a single parameter can overfit the data, and therefore it is not always preferable to use a model with fewer parameters.This fact is related to the observation, in a setting of classification, that f (x) = sin(x) has an infinite VC-dimension [15].
The existence of such a simple equation with such freedom in behavior illustrates a more basic problem that model complexity cannot be determined by counting parameters.More generally, uncritical use of a "parameter counting" approach ignores the fact that a single real-valued parameter potentially contains an unboundedly large amount of information since a real number requires an infinite number of bits to specify.Indeed, the set of real numbers that can even be described with finitely many bits (e.g. by a Turing machine) is countable and thus has measure zero.Given the existence of injective maps between R n and R [16], the number of parameters in a model cannot be a meaningful measure of its complexity once the class of models is large enough to implement these maps and effectively decode one single number into many.However, such embeddings are not continuous nor likely constructible as an ordinary looking equation that a scientist is likely to encounter.
The example provided in this paper shows that the infinite amount of information in a real valued parameter can be decoded quite simply, using just sin and exponentiation.The existence of such a simple yet problematic equation implies that attempts both at broad model comparison and automatic discovery of equations from data may often be ill-posed.Quantitatively, parameter-counting methods should be dispreferred relative to model comparisons based on measures that incorporate the precision required of real-valued parameters, including Minimum Description Length [17].The result also emphasizes the importance of constraints on scientific theories that are enforced independently from the measured data set, with a focus on careful a priori consideration of the class of models that should be compared [4].

FIG. 1 :
FIG. 1: A scatter plot of f θ for θ = 0.2446847266734745458227540656 • • • plotted at integer x values, showing that a single parameter can fit an elephant (left).The same model run with parameter θ = 0.0024265418055000401935387620 • • • showing a fit of a scatter plot to Joan Miró's signature (right).Both use r = 8 and require hundreds to thousands of digits of precision in θ.