סגור
This miniMOOC teaches the math behind Shannon's entropy. It was created by Dr. Rivki Gadot (Open University , LevAcademic Center) & Dvir Lanzberg (the lecturer). It was made for undergraduate students, but we believe that anyone with a basic knowledge in calculus and algebra can get over the math in this miniMOOC.
Shannon started by asking what is "information" and how it can be measured. Instead of giving a definition, he claimed that any function that measures information must have three properties, and proved that there is a single function that meets these three properties. This function become to be known as, Shannon's entropy.
There are six short clips in this miniMOOC. Each clip is accompanied by exercises or a quiz that let you deepen your understanding. The topics of the six clips are:
- The problem - states what was the problem that Shannon tried to solve.
- The solution - introduces Shannon's entropy.
- The three properties - explains what are the three properties that every function for the amount of information, must have.
- Equal probabilities - proves that Shannon's entropy is the only function that has the three properties, if the events' probabilities were equal.
- Rational probabilities - proves that Shannon's entropy is the only function that has the three properties, if the events' probabilities were rational numbers.
- Real probabilities - proves that Shannon's entropy is the only function that has the three properties, if the events' probabilities were real numbers.
1. The Problem
Information and surprise
This clip states what was the problem that Shannon tried to solve.
- How much surprise is there in finding out the outcome of a coin toss?
- How much surprise is there in finding out which permutation of an array is sorted?
- How much surprised can we get by knowing the weather in Tel-Aviv in August?
After watching the clip, try to solve try to solve these problems: Part 1 - Questions.
Want to check your answers? Download Part 1 - Answers for answers and explanations.
 
2. The Solution
Entropy
This clip introduces Shannon's entropy.
So given n events how much information (the surprise, the entropy) is there in knowing which event happened?
 
 
After watching the clip, try to solve these problems: Part 2 - Questions
Want to check your answers? Download Part 2 - Answers for answers and explanations.
 
3. Three Properties
Shannon required three properties of H:
This clip explains what are the three properties that every function for the amount of information, must have:
- H should be continuous in the pi
- H(1/n,…,1/n) should be a monotonic increasing function of n
- If a choice be broken down into two successive choices, the original H should be the weighted sum of the individual values of H
After watching the clip, try to solve these problems: Part 3 - Questions
Want to check your answers? Download Part 3 - Answers for answers and explanations.
 
4. Equal probabilities
Lets start by considering n events with equal probabilities.
This clip proves that Shannon's entropy is the only function that has the three properties, if the events' probabilities were equal.
- What are the other options?
- What does it mean equal probabilities?
- What is the value of H in this case?
After watching the clip, try to solve these problems: Part 4 - Questions
Want to check your answers? Download Part 4 - Answers for answers and explanations.
 
5. Rational Probabilities
Now consider n events with rational probabilities.
This clip proves that Shannon's entropy is the only function that has the three properties, if the events' probabilities were rational numbers.
- Is this the general case?
- What is the value of H in this case?
 
After watching the clip, try to solve these problems: Part 5 - Questions
Want to check your answers? Download Part 5 - Answers for answers and explanations.
 
6. Real Probabilities
Eventually consider n events with real probabilities.
This clip proves that Shannon's entropy is the only function that has the three properties, if the events' probabilities were rational numbers.
- Is this the general case?
- What is the value of H in this case?
 
After watching the clip, try to solve these problems: Part 6 - Questions
Want to check your answers? Download; Part 6 - Answers for answers and explanations.
 
Before we say goodbye...
Some Questions We Did Not Discuss (and You May Want to Research Yourself)
- Who was Shannon? What else did he discovered and invented?
- What is the meaning of "Entropy"?
- Why is Shannon's entropy so important?
 
Materials
| Part | Link2Clips | Questions | Answers |
|---|---|---|---|
| The Problem | |||
| The Solution | |||
| Three Properties | |||
| Equal Probabilities | |||
| Rational Probabilities | |||
| Real Probabilities |
miniMOOCs נוספים
15
מגדלי הנוי - miniMOOC
Shanon Entropy
אודות הקורס
מטרת הקורס להדגים רדוקציה בין בעיות, כלומר איך ניתן להמיר בעיה אחת לבעיה אחרת. רדוקציה יכולה לסייע לפתור בעיה בעזרת פתרון בעיה אחרת.
הקורס פותח ע"י דביר לנצברג והוא מורכב משלושה סרטונים, שאלונים ושאלות לחשיבה.
מרצה: דביר לנצברג
שפה: עברית
אודות הקורס
מטרת הקורס להדגים כיצד כתיבת אלגוריתם לפתרון בעיה
מאפשר להגיע לתובנות לגבי הבעיה ופתרונה.
הקורס פותח ע"י דביר לנצברג והוא מורכב משלושה סרטונים, שאלונים ושאלות לחשיבה.
מרצה: דביר לנצברג
שפה: עברית
About the Course
This miniMOOC teaches the math behind Shannon's entropy. The miniMOOC created by Rivki Gadot & Dvir Lanzberg and it requires a basic knowledge in calculus and algebra .
Lecturer - Dvir Lanzberg
Language - English
