Can We Trust What We Can't See?, February 8

In the last three decades, astronomers have detected 6,000 exoplanets. However, we rarely know much about them and must rely on models to understand how they behave. In astronomy, developing these models to understand exoplanet formation and evolution is considered theoretical work.

The role of theory in science is well understood. Scientists tend to fall into three broad categories: theorists, experimentalists, or instrument builders. Particle physicists, for example, may develop theoretical frameworks, run collider experiments, or design more advanced detectors and accelerators. Theoretical frameworks inform which experiments to run and what tools are needed.

Astronomy works differently. Because we cannot run controlled experiments on stars or planets, we rely on observation rather than manipulation. Instead of designing experiments, astronomers propose specific targets to observe and use increasingly sophisticated instruments to collect data from them. Both theories driven by physicists writing on chalkboards and theories created by computational models can then be tested against these observations.

Computer models of astronomical systems sit in an ambiguous space: they can function both as theory and as experiment. A model encodes physical laws, but once run, this software explores outcomes that are not analytically obvious. In that sense, it behaves like a virtual laboratory.

However, models vary widely in fidelity. High-resolution simulations based on real data, such as the evolution of well-observed, nearby asteroids, resemble experiments. Other models are less tightly constrained by observations, such as those modelling the billions of years of evolution in an asteroid belt. These are closer to theory, prioritising conceptual understanding and producing results that observational data may take years to confirm.

These variations explain why researchers who design computer models, particularly those less connected to observations, are often considered theorists. It’s therefore unsurprising that models generate considerable controversy: they can be implemented in many ways and yield vastly different results.

Now, to return to exoplanets. Via observations of an exoplanet, we can see the spectrum revealed by an atmosphere. However, many possible interpretations could fit this atmosphere; for instance, an observation could potentially be explained by the presence of water or CO₂ in an atmosphere. To resolve this degeneracy, astronomers use a statistical method called an atmospheric "retrieval," varying hundreds of underlying parameters until they find the best fit to the observed spectrum.

However, assumptions built into the retrieval can drastically alter the results. For instance, clouds and hazes can flatten spectra; if an astronomer assumes a cloud-free atmosphere, their retrieval might incorrectly suggest a barren planet with no atmosphere at all. And clouds are just one parameter. For every choice in the retrievals there's a different controversy. Different research groups use incompatible assumptions, leading to contradictory results from the same data. Some retrievals function as black boxes, leaving observers wondering whether modellers can reproduce or even fully understand their own outputs. Some wonder whether publish-or-perish pressures drive overinterpretation of marginal signals.

Ultimately, though, the tension between theorists and observers remains healthy. Science progresses through uncomfortable conversations and unresolved questions. If we had all the answers, there'd be nothing left to discover.