Chapter 2

Information and entropy

Two terms travel under the name entropy and this chapter keeps them strictly apart. Thermodynamic entropy is chaos, whose opposite is the order we here call information: we recognise information because it carries patterns that chaos does not. Shannon entropy is surprise, where maximum information is maximum entropy – the meaning the computer-science curriculum uses, and a different one from the physicist’s. Compression and coding belong to this chapter because they lead directly to language: a word is a compressed pointer to a cluster of experience.