Entropy in physics vs information systems

Question

Can anyone please explain the equivalence or similarity of entropy in physics and entropy in information systems in layman terms? Sorry I'm no mathematician, but still I am trying ti understand the concepts so that I'll have a better understanding of the concepts. I have an idea of entropy in Physics, but I don't understand when someone says entropy in information systems and its uses and applications. Thanks for your time.

Answer 1

Information entropy (also called Shannon Information) is the measure of "surprise" about a new bit of information. A system with high entropy has a large surprise. Low entropy, little surprise.

Systems with high entropy are difficult to compress, because every bit is surprising and so has to be recorded.

Systems with low entropy are easy to compress, because you can predict what comes next given what you've seen before.

Counter-intuitively, this means that a TV showing static (white noise) is presenting a lot of information because each frame is random, while a TV show has comparatively little information because most frames can be mostly predicted based on the previous frame. Similarly, a good random number generator is defined by having very high entropy/information/surprise.

It also means that the amount of entropy is highly dependent on context. The digits of pi have very high entropy because an arbitrary one is impossible to predict (assuming pi is normal). But if I know that you will be sending me the digits of pi, then the digits themselves have zero information because I could have computed all of them myself.

The reason all of this plays into cryptography is because the goal of a cryptographic system is generate an output that is indistinguishable from random, which is to say that it takes low-entropy information and outputs high-entropy information. The output of a cryptographic algorithm can have no more entropy than its highest-entropy input. Systems whose highest-entropy input is a human chosen password are going to be very poor crypto systems because they are very predictable (have little information; low entropy). A good crypto system will include a high-entropy value like a well-seeded and unpredictable random number. To the extent that this random number is predictable (has low entropy), the system is weakened.

You must be careful at this point not to over-analogize between thermodynamic and information entropy. In particular, one is almost exclusively interested in entropy gradients in thermodynamics, while entropy is treated as an absolute value in information theory (measured in bits). Conversely, information entropy is sometimes incorrectly thought of as a form of energy that is "depleted" when generating random numbers. This is not true in any useful way, and certainly not like heat energy.

Also, how cryptographers use the word entropy isn't precisely the same as how Shannon used it. See Guesswork is not a substitute for Entropy for one discussion of this.

For how this does and doesn't apply to thermodynamics more broadly (and particularly how it applies to the famous Maxwell's Demon), I recommend the Wikipedia article comparing the two kinds of entropy.