The stunning connection between entropy, time and information
In all of physics, Entropy is one of the most unique yet enigmatic concepts. And understandably so. what does it actually mean?
The common interpretation is that it is a measure of disorder in the universe. And the second law of thermodynamics states that entropy can never decrease, in other words, the universe is on an inexorable path to more and more disorder. And it is this one way property of entropy that some scientists contend is the reason time also flows one way – forward.
And If entropy is responsible for something like flow of time itself, then is it a fundamental property of nature? Or is there something deeper about entropy that really explains what is going on at the core of reality.
Scottish physicist, James Clark Maxwell came up with a thought experiment, called Maxwell’s demon, that showed a mechanism where entropy of a closed system could be reversed. It would violate the 2nd law. It was a paradox that no one could resolve it for over 100 years.
But when computer researchers at IBM finally did solve this paradox, they showed that entropy was related to something that could be even more fundamental, and if time depends on it, it could perhaps be the underlying principle that rules the universe – and that is information.
How is this possible? What is the link between entropy, information, and time?
If you observe the rotation of the planets around the sun. And then reverse the video, there is nothing about the reversed motion that is unusual. It would be permissible within the laws of physics.
Similarly, as far as is known, all physical laws are time reversible. The only law of physics which appears to be time irreversible is the law of increasing entropy – the 2nd law of thermodynamics.
Scrambling an egg, shattering a hard drive, putting sugar in coffee – these are all actions such that if you saw them in reverse, you would immediately know that you are watching something absurd, that could not happen. But the 2nd law of thermodynamics appears to be the only law that reversing these processes would violate. Running the movie backward would mean decreasing entropy.
The 2nd law says that in an isolated system the entropy can never decrease. But this doesn’t tell us what entropy actually is. So what is entropy? The common definition is that entropy is a measure of the disorder of a system. This is not quite precise. To get a better understanding, let look at a simple visual example.
Let’s take a container with gas isolated on one side with a barrier. If we remove the barrier, we expect the gas to expand into the full volume of the container. We have just increased the entropy of the gas.
Why is there more disorder if the gas expands into a larger volume? The reason is that there is a greater volume and thus more places that individual gas molecules could be. There are more ways to arrange the gas molecules in a larger volume than in a smaller volume. This greater possible number of arrangements is really a better description of entropy.
The insight was made by Austrian Physicist Ludwig Boltzman as a definition of entropy – that it is really a way of counting the number of arrangements of atoms inside a system
Now what if we had a plunger on the right wall and pushed the gas back to half the volume of the container. Wouldn’t this reduce the entropy back to what it was? No because compressing the gas like this requires putting work into the system. It would raise the temperature of the gas. The gas would heat up and the entropy would increase making up for any decrease in entropy because of the smaller volume.
Why does the entropy increase when the temperature increases? Again, this has to do with an increase in the number of ways that the motions of molecules can be described. An increase in temperature is due to an increase in motions or kinetic energy of molecules in the gas. As the molecules move faster, the temperature increases. There are more ways to describe this motion than if the molecules were moving slower.
Now the question is, could the gas spontaneously arrange itself so that it fills only half the volume. You might say, well that can’t happen because it would violate the 2nd law by decreasing the entropy. Yes that would violate the 2nd law, but the truth is that there is nothing in the laws of physics that really prevents this from happening, except that it is very very unlikely.
The possibility that all the multiple trillions of molecules in the gas could arrange themselves in one side of the container is about one in 10^150,000,000,000,000,000,000,000. This number is so enormous that it is statistically impossible. To give you an idea of scale, the total number of planck volumes, the smallest volume that could exist, that could fit in the volume of the known universe of the universe is only 10^183. So, needless to say, the 2nd law is not going to be violated anytime soon. But it also tells you that the 2nd law is not an absolute law but a statistical law. In other words, it could be violated, but it is extremely unlikely.
But in 1867, Scottish physicist, James Clerk Maxwell devised a thought experiment that showed that that the 2nd law could be violated, and entropy could indeed decrease. And this thought experiment seemed perfectly plausible to scientists, and they could not figure out why it was wrong for over a 100 years. British Physicist, Lord Kelvin later labeled this Maxwell’s Demon.
Imagine a chamber with a divider in the center. If we start out with a gas in one chamber, this gas will distribute itself evenly between the two chambers. Now what if we put a door between the two chambers that is controlled by a little demon.
The door is frictionless so adds no heat or energy in the enclosed system. The demon controls the door such that he only lets the molecules travel in one direction. When a molecule comes near the door from the right, he opens the door to let the molecule travel to the left chamber.
He doesn’t let any of the molecules from the left chamber travel over to the right chamber. You can imagine that over time, this demon makes it so all the gas is collected on the left chamber such that no molecules remain in the right chamber.
No work was added and the temperature remained the same, so the entropy decreased. This clearly seems to violate the 2nd law.
You might say well, it’s a thought experiment with an imaginary demon. It can’t really be done. But you have to remember, this was thought of at a time when the concept nano robots and microprocessors did not exist. It is conceivable that in the near future we could build a computer controlled device that could be the mechanized equivalent of the demon.
So scientists could not disprove the idea behind this thought experiment for over a hundred years. The solution finally came in 1982 by two scientists at, of all places, IBM research.
Rolf Landaur and Charles Bennett. They boiled down the demon to a more fundamental idea behind entropy. And that is information.
They showed that In order for the demon to reduce the entropy, he has to gather information about the movement of various molecules to determine when to open and close the door. He is building a memory record. The demon is increasing the amount of information in the system because he is part of the system. This increase in information is an increase in entropy.
They showed that the decrease in entropy due to the volume decrease is exactly counterbalanced by an increase in information in the demon’s brain. The 2nd law is not violated.
You might say, well, all I have to do is erase the memory of the demon. If this was a computer hard drive, we can just reformat the computer hard drive, and there, now the hard drive is low entropy again.
Aha, but Landower and Bennett showed that erasing information is not free. Erasing information produces heat. It increases entropy. So you can not gather information in the Demon and then erase it to reduce the overall entropy. The increase in entropy from the heat of the erasure is perfectly offset by the decrease in information in the hard drive.
This is not very much heat, only about 3 billionths of a trillionth of a joule per bit that is erased.
But this also gives us a clue about the true nature of entropy. And that nature is that it is really a measure of the information required to express the state of a system. A system with Higher entropy requires more information to describe its microstates.
When you increase temperature by adding energy for example, there are more ways that this energy can be shared with the gas molecules. This requires more information to figure out.
If we increase the volume, the molecules have more places where they can be, so this requires more information to describe the microstate of the gas. This information requirement is a better definition of entropy than simply disorder
A system tends to flow towards more ways that it can be arranged. And so the information needed to describe these arrangements is always increasing.
Now, how does time enter into the picture? How is time related to entropy and information?
Since all the laws of physics we know appear to be symmetrical with respect to time, there doesn’t seem to be any reason time should be asymmetrical…Except through the idea of entropy. Entropy is asymmetrical. It flows only in one direction – higher.
The feature of universe that changes with time Is entropy.
This inspired British Scientist Arthur Eddington to postulate that it is the increase in entropy that is responsible for the flow of time.
But does time go forward because entropy increases? Or does entropy increase because time goes forward. This is not well understood. It is not a good theory because the prediction it would make is that time should go backward when entropy decreases.
There are pockets of decreasing entropy all over the place. The inside of your refrigerator is a place where entropy is decreasing. Your food is getting colder, but time doesn’t flow backward inside your refrigerator.
Similarly, earth at night experiences decreasing entropy. But time does not flow backward at night.
So what actually causes time to flow forward? To answer that, let’s look at why entropy increases.
Entropy increases not because the fundamental laws of physics says it does, but because there are simply more ways that microstates can arrange themselves in the future than in the past.
For example, there are more ways to arrange checker pieces on a checker board than when neatly arranged at the start of the game.
We are much more likely to be in a state of higher entropy than lower entropy as time passes. But since entropy is fundamentally a statistical law, based on higher and higher amount of information needed to describe microstates, then could the flow of time also be a statistical phenomenon. That is, time is much more likely to flow forward than to flow backward.
If so, this would leave open the idea that time flowing backward is not forbidden, but just statistically impossible.
If entropy increases over time, that means that entropy must have been at its lowest possible state at the start of the universe, at the big bang. This is sometimes referred to as the past hypothesis.
This low entropy start is thought to be the primary reason why time flows forward.
But why was entropy so low at the beginning. Why did the universe start in a very ordered state? Given all the energy and matter in the universe, and all the states it could have been in, why did it start in this very small subset of all the different configurations it could have been in.
No one really knows the final answer but there are some leading ideas.
Maybe there is a fundamental reason, but we haven’t discovered it yet. And we are trying to explain something that is part of our ignorance.
Maybe It is just a fact. It is a property of the universe, like the fundamental constants.
Another idea is from physicists like as Alan Guth and Sean Carroll. if there is no upper limit to entropy, then no matter where the universe started, it would have been in the lowest entropy state. It has nowhere else to go except a higher entropy state.
If Time flows forward as entropy becomes higher, then we are a result of the 2nd law. We are a consequence of the increasing entropy of the universe. If the universe was in thermal equilibrium, we would not exist. The reason we have causality, evolution, biological processes, thoughts, memories and even consciousness is the 2nd law.
Alan Guth points out that If information was not increasing in our brain, we would not have memories or consciousness. This links entropy to consciousness itself.
And Entropy will continue to increase for billions, and probably trillions upon trillions of years, when all the galaxies will have spread apart. All the stars will have burnt out. Dark energy will have ripped apart all matter including all molecules, all atoms, and even subatomic particles. At some point we will be in thermal equilibrium, at least locally where ever we are in the universe.
And at this point, entropy may cease its inexorable forward march. And all will become quiet but Space will not go away and laws of physics probably won’t either.
What will remain is the vacuum energy, quantum fluctuations in the fabric of space time. Given enough time, there will be enough random energy fluctuation within the vast space of the universe that an atom forms spontaneously, a molecule forms spontaneously, interstellar gases could form, stars and galaxies could form. The bigger the thing, the longer you have to wait. But if you have an eternity to wait, it will happen at some point.
And statistics predicts that if you wait long enough, around 10^10^10^120 years, even a universe could form spontaneously.
