Scientific Samsara

Samsara, as taught to me at highschool and stated by Wikipedia, refers to the Buddhist belief in a "cycle of reincarnation and ... mundane existence". To a Buddhist, then, all life, riches and success are pointless with the exception of attaining enlightenment.

We scientists should heed this lesson, for this logic applies equally well to our set of axioms. Our Nirvana consists of making the most accurate observations of nature, creating falsifiable hypotheses which best fit this data and developing a technology which is demonstrably better or more useful than all others in some way.

Let this sink in. Publishing countless highly-cited articles in Nature on the most mathematically elegant theory of the luminiferous aether is garbage science, if it fails to describe experiments in photonics better than our current leading theory. Controversially, I would even argue that such work advances science less than the discovery of some beetle by a retired hobby gardener, much as I am indoctrinated as a physicist to believe that my branch of science is superior to biology. Continuing to develop thermionic valves for computation, or - even more controversially - to work on supersymmetry, is Samsara.

Our cousins working in the tech industry, who have short-term profit and loss hanging over their innovation as the Sword of Damocles, have long known about technical debt, whereby it is ill-advised to struggle with something that would yield little benefit today and would in any case be more easily solved tomorrow. Consider that, assuming a Moore's Law with a doubling time of 2 years, delaying the ill-fated National Ignition Campaign from 2009 until now would have allowed scientists a theoretical factor of 16 improvement in computing power to help direct their fusion experiments. On the other hand, the technology to build LIGO, develop diode-pumped solid state lasers or launch a solar orbiter was just as mature back then as it is today. If we are honest, aside from making a quaint story, the work of Charles Babbage to build a steam-powered computer has had little practical impact on modern computer science.

In debates like "those coal miners made unemployed by their mine closure should find other jobs", I have heard the argument that it would be difficult to retrain to another career. I would question how scientists who intellectually cannot change research fields were able to reach the forefront of their profession in the first place.

Let me end with a different theology, the ending of Medea:
Our wishes do not always come to pass,
yet some god will find a way to make unexpected happen.
So with this story.

The challenges of equality: a mathematical perspective

There has been a lot of talk recently of equal representation in various fields. I would like to offer my mathematical opinion on the matter and I will illustrate these thoughts with an example I hope to be totally politically uncharged. I was recently lucky enough to learn - in person - about the formation of a new Overwatch League. For those over 40, Overwatch is a multiplayer videogame and the new OWL is a league of professional teams (each player is likely to be on at least 6 figures) with sponsorship deals and games at live stadiums.

So, let's imagine how statistically likely we are to have equality in this league, specifically that between two groups: professional players with glasses and those without. The examples below are, of course, completely hypothetical.

As a consequence of the Central Limit theorem, any meaningful probability is extremely likely to be Normally distributed. Not guaranteed, but extremely likely. The Normal distribution is characterized by the standard deviation (width) σ and mean μ.

Assume that two groups of precisely equal size play the videogame: those who wear glasses and those who do not. As expected, given that the total number of players is large enough to be statistically significant at 35 million worldwide (and therefore each group, since they're 50% of the total), then their skill is normally distributed. Suppose that the groups' standard deviations are equal, but the "Glasses" group for whatever reason has a small skill advantage in the form of a slightly higher mean:

The grey area represents the overlap of the two distribution functions - perhaps it is accurate to describe them by having more in common than not. That is true for the majority of players, but to qualify for a top-level place on a competitive team, a player must surely be at the highest levels of skill. Let's model this by making the (not very realistic) assumption that only the most skillful players are selected, until spots on every team are filled. 

Statistically, this is represented by taking an integral from infinity down to some threshold skill value T (the skill of worst players on a team). We can expect that the integral is larger for the Glasses population for the example above, but you can see it clearly, equally scaled below:

 

The definite integral of the Gaussian function is the error function (erf). The sum of the two integrals is the total fraction F of the playerbase playing on a team.

The inversion of this equation allows finding the threshold, given a set fraction. This becomes non-analytic for an arbitrary number of distributions, but is straightforwardly done with Brent's algorithm.

Let's imagine that we set a threshold - "the top 1% of all players are on a professional team" - and wish to know the proportion with glasses. The expected percentage is shown for three choices of threshold as a function of the shift in mean below:

The functions are symmetric about the point (50%, 0), because the two groups can effectively be "swapped" if Δμ is reversed in sign.

Similarly, assuming both populations have the same mean, but varying the standard deviation of Glasses compared to No Glasses is shown in the plot below:

So arguably the standard deviation has a greater effect than the mean, because when skimming only the very best off the top, it helps having a lot of statistical spread - having many highly skilled and equally many poorly skilled players - as opposed to having almost everyone "average".

To conclude:

• If there is a true disparity between two populations, then the mathematics is clear: one group will always tend to be over-represented. The solution to achieving parity of representation is to equalize the statistical distributions: perhaps there is a systematic poorer retention of the No Glasses group, which can be eliminated, or the game itself can be changed to be more friendly to both.
• Conversely, if the statistics suggest that 70% of players should have glasses, but the true figure is 95%, then there is some strong bias in the system. The situation is unfair, but also certain to be counter-productive: for the Glasses integral to make up such a large proportion, its threshold must be lower than for No Glasses; this means that the overall skill level of the players could be increased if the latter were represented fairly (making up 30% of professional teams).
• If measuring game skill directly is not possible, one should be very wary of proxy metrics like reaction time, etc. These may be only weakly correlated with the truly interesting parameter and that correlation itself may be poorly known, leading to bias in the analysis itself.