Information Theory through Toy Examples

Posted October 26, 2017 by Matthew Andres Moreno in Education / 0 Comments

Motivation

Clean the gutters, clear enough space in the garage to park the car, organize a laundry bin of family photos, respond to an email that’s been wallowing too long at the bottom of the inbox…we all have a list of things we’ve been meaning to get around to. Learning a bit of information theory has been one of those things I meant to get around to for longer than I would like to admit. A few weeks ago, I ran into Dr. Chris Adami at the BEACON Congress. He recommended a pair of introductory papers on information theory. In [Adami, 2016], he provides a lively introduction to information theory that pays special attention to guiding newcomers through commonly-encountered intellectual obstacles and dispelling common misconceptions about information theory. In [Adami, 2012], Adami presents an information theoretic perspective on genetic material and brains, ultimately demonstrating how these substrates fundamentally promote fitness by allowing organisms to exploit information about their environments.

With these papers on my desk, it was finally time to learn a little information theory. Now that I’ve spent a few hours brushing up on it, I’m glad to have this new addition to my scientific toolbelt. I found the papers Dr. Adami recommended to be accessible and useful. I really had my lightbulb moment, though, when I took a little time to cook up some toy examples and walk through the math on my own. I’d like to share them with you here. There will be a little math along the way, but it is just in service of the toy stories I want to tell. what I found most valuable about this exercise when I worked through it on my own was seeing how the results of the math tie in to the story. So, don’t let the math spook you. The math is just in service of the story. Ultimately, I hope these examples might strike a first spark of intuition, or at least serve as an invitation to dip a toe into information theory.

information-toy

(PDF viewer not working? Download the PDF here.)

References

[Adami, 2012] Adami, C. (2012). The use of information theory in evolutionary biology. Annals of the New York Academy of Sciences, 1256(1):49–65.

[Adami, 2016] Adami, C. (2016). What is information? Phil. Trans. R. Soc. A, 374(2063):20150230.

Matthew Andres Moreno

I am a doctoral student at the Michigan State University Department of Computer Science and Engineering. My research interest is in using digital evolution techniques to investigate scientific questions about evolution. I am currently investigating the relationship between plasticity and evolvability as well as major evolutionary transitions.

More Posts - Website - Twitter

Leave a Reply