A trio of British-born researchers working in the U.S. won the Nobel Prize in physics for what one of them called a curious mathematical “toy” that to his surprise revolutionized the study of exotic matter suitable for quantum computers, new superconductors, and advanced designer materials.

Working separately, the three laureates conceived a new way to understand the topology of materials, or the study of shapes that change in increments. At its simplest, a sheet of paper can have many sides, depending upon the topology of its folds. At an atomic level, however, variations in the structural topology of electrons can yield materials with properties unknown among the commonplace solids, fluids and gases of the ordinary world.

“At the time, I thought it was of scientific interest and mathematical interest, but I didn’t think it would ever find a particular realization,” said Princeton University physicist F. Duncan M. Haldane, who shared the award for theoretical experiments he had conducted in the 1980s. “I basically stumbled on this playing with mathematics.”

Dr. Haldane shared the 2016 Nobel Prize with theoretical physicist David J. Thouless at the University of Washington in Seattle and physicist J. Michael Kosterlitz at Brown University in Providence, R.I. Dr. Thouless won half of the eight million Swedish kronor ($931,000) cash award, while Dr. Haldane and Dr. Kosterlitz split the remaining half.

Starting in the 1970s, the three scientists delved into the odder properties of the material world step-by-step.

Dr. Thouless, 82, explored the behavior of swirling vortexes in superfluids and other problems related to topological quantum numbers.

Dr. Kosterlitz, 73, sorted out an apparently impossible contradiction between theory and experiment by explaining the existence of the superfluid state, in which fluids flow without apparent friction in thin films of helium. He learned of his award while on his way to lunch in Helsinki, where he is a visiting professor at Aalto University. “I’m a little bit dazzled. I’m still trying to take it in,” Dr. Kosterlitz told the Associated Press.

Dr. Haldane, 65, discovered how topology can be used to understand chains of small magnets found in some materials.

“My work was a kind of sleeper,” said Dr. Haldane, who considered his work at the time a mathematical toy. “It was a very theoretical thing.…It didn’t become such a big deal until my work got extended [by other scientists]. You never set out to discover something new. You stumble on it and you have the luck to recognize that it is something new.”

Announcing the award in Stockholm Tuesday, the Royal Swedish Academy of Sciences said in its citation that “this area has boosted frontline research in condensed matter physics, not least because of the hope that topological materials could be used in new generations of electronics and super-conductors, or in future quantum computers.”

Florida State University physicist Laura Greene, who is director of the National High Magnetic Field Laboratory, said, “They cracked a crucial part of the problem—an under-sung fundamental question in physics today. Not only were their solutions enormously clever and inspiring, but they laid the foundations for today’s exploding field of topology matter.”

While their work was theoretical, “there is nothing more practical than understanding the theoretical basis of the properties of matter, said physicist Robbert Dijkgraaf, director of the Institute for Advanced Studies in Princeton, N.J. “Our whole world is a material world. They were really pioneers of a field that has blossomed and is essential in understanding the properties of modern materials.”