ECE636

Information Theory
Spring
Catalog Data: 

Graduate Course Information

=======================================

ECE 636

Information Theory

Credits:

3.00

Course Website:

UA Catalog Description:

http://catalog.arizona.edu/allcats.html

Course Assessment:

Homework:  8 – 10 assignments.

Exams:  2 Midterm Exams, 1 Comprehensive Final Exam

Grading Policy:

Typically: 10% Homework,

                  30% Exam #1,

                  30% Exam #2,

                  30% Final Exam.

Course Summary:

Definition of a measure of information and study of its properties; introduction to entropy, mutual information, channel capacity, and rate-distortion theory.

Prerequisite(s): 
ECE503 - Probability and Random Processes
Textbook(s): 

“Elements of Information Theory,” by T.M. Cover and J.A. Thomas, Wiley.

Course Topics: 

1.     Entropy, Relative Entropy, Mutual Information

a. Intuitive interpretation of entropy

b. Joint Entropy

c. Conditional Entropy

d. Conditional Relative Entropy

e. Conditional Mutual Information

f. Sufficient statistics

g. Fano’s Inequality

2.     Asymptotic Equipartion Principle

a. Application of AEP: Simple minded data compression

3.     Entropy Rates of Random Processes

a. Hidden Markov Models

4.     Data Compression

a. Huffman codes

b. Optimality of Huffman codes

c. Shannon-Fano-Elias coding

d. Arithmetic coding

5.     Channel Capacity

a. Binary erasure channel (BEC)

b. Channel Coding Theorem

6.     Differential Entropy

7.     The Gaussian Channel

a. Binary amplitude modulation

b. Independent channels with average power constraint

c. Channels with dependent noise

8.     Rate Distortion Theory

Class/Laboratory Schedule: 

Lecture:  150 minutes/week

Prepared by: 
Michael Marcellin
Prepared Date: 
April 2013

University of Arizona College of Engineering