Explained | Theory of everything: How a fear of failure is hampering physicists’ quest for the ultimate answer

Sam McKee for The Conversation

Manchester: It has been over a century since the boom period of physics exploded with Albert Einstein, Max Planck and others, sending us spinning into a new world of chaos from our previously ordered universe.

This brilliant generation of physicists ultimately peeled back the layers of the universe, as well as of the atom, to reveal a world stranger than fiction.

Ever since those earliest days of quantum mechanics, the theory ruling the microworld of atoms and particles, the holy grail of physics has been finding a theory of everything – uniting quantum mechanics with Einstein’s theory of general relativity, which applies to the universe on large scales.

But we still don’t have a tried and tested theory of everything. And I believe a fear of failure is a big part of the problem.

Creating a theory of everything isn’t exactly easy. It involves producing one framework uniting the fundamental forces of our universe, while accounting for all the underlying constants and quantities as well as every subatomic particle. The prize for whoever answers this ultimate question is eternal glory in the annals of humankind.

There was great hunger to solve it in Einstein’s generation. In fact, Einstein worked on a theory of everything on his very deathbed – work that he was ultimately ridiculed for. Einstein’s contribution to physics was so great he still remains a superstar. But physicists Arthur Eddington, Hermann Weyl and mathematician David Hilbert were not so fortunate, with some facing much worse consequences.

Take Eddington, for example, perhaps the greatest scientist you have never heard of. The Cambridge astronomer and physicist proved Einstein correct in his work analysing a 1919 eclipse – launching Einstein to superstardom. Eddington also wrote the first English books on relativity before doing the same on Georges Lemaitre’s Big Bang theory.

He also wrote a book on quantum physics, and became the greatest popular writer on science in the 1920s and 1930s, alongside his groundbreaking work on stellar physics (the physics of stars). Yet he is obscure today due to his intense pursuit of a fundamental theory.

Published posthumously, his attempt was immediately banished for its incredible failure. Mocked as numerology (the belief in a mystical relationship between a number and events), his strange interest in the power of certain numbers was made fun of by other scientists. And, as many notable astrophysicists have pointed out, it has produced no value since its publication.

Eddington’s stunning final failure served as a powerful warning of the price that comes with missing the mark. The final decade of his life spent pursuing a theory of everything ended in severe damage to his legacy.

A new generation

The generation of the physicist Richard Feynman (1918–88), following that of Einstein and Eddington, lost interest in a theory of everything. Feynman and his peers found their own glory in new subatomic discoveries and theories, and applications of physics to chemistry and biology, leading to several Nobel prizes. The ridicule endured by those who tried and failed before them may have been one of the reasons.

This inordinate cost for failure ultimately rose alongside the glory of interwar physics. In a period of unparalleled success, failure was more unforgiving. This was hardly an incentive for young and brilliant modern minds seeking to apply themselves to the largest question.

Even today, attempts at theories of everything get mocked. String theory, for example, is such an attempt, and has been scorned by Nobel laureate Roger Penrose as not being real science.

He is not alone. Physicist Stephen Hawking believed a version of string theory called M-theory was our best option for a theory of everything. But the theory has struggled in producing predictions that can be tested by experiments.

A young scientist today may wonder, if Einsteinx, Eddington and Hawking could not solve the problem, then who will? And indeed, many are doubtful that it can be achieved. Is it even necessary as, pragmatically, we can do without one?

It is no wonder, then, that many physicists prefer to avoid the term “theory of everything” these days, opting instead for less grandiose alternatives such as “quantum gravity”.

Funding and career progression

Alongside the heavy price of failure, other problems are lurking. A brilliant young mind could be staring at a career dead end in seeking out a theory of everything. What academic progression can one expect at the start of their career if this is desired? Who will give significant funding to young, unproven researchers pursuing a seemingly impossible goal in the short term?

It is likely that a theory of everything will ultimately require massive collaboration to be solved. Ironically, this may be a job for the older physicists, despite the warnings of Eddington and others. Francis Crick dedicated his attention to trying to solve the problem of consciousness in his later years, albeit without success.

We need collaboration. But we may be looking at the prospect of a theory of everything only coming from those who have accomplished so much they can afford the potential embarrassment and will be given the benefit of the doubt. This hardly stirs the enthusiasm of the vibrant, young minds that may otherwise tackle the problem.

In trying to solve the ultimate problem, we may have inadvertently created a monster. Our academic framework for research progression is not conducive to it, and history has presented an unkind picture of what happens to those who try.

And yet, our greatest progress has always come from those willing to take risks.

(Sam McKee is Tutor and Researcher in Philosophy of Science at Manchester Metropolitan University)