An Introduction to Information Theory

By Prof. Adrish Banerjee | IIT Kanpur

Learners enrolled: 1890

Information Theory answers two fundamental questions: what is the maximum data rate at which we can transmit over a communication link, and what is the fundamental limit of data compression. In this course we will explore answers to these two questions. We will study some practice source compression algorithms. We will also study how to compute channel capacity of simple channels.

Intended Audience : 3rd/4th year UG students in EC stream, 1st year PG students in communications and signal processing specialization

Prerequisites : Basic knowledge of probability theory and digital communications

Industries Support : Communication companies, defense laboratories

Summary

Course Status :	Completed
Course Type :	Elective
Language for course content :	English
Duration :	8 weeks
Category :	Electrical, Electronics and Communications Engineering Communication and Signal Processing
Credit Points :	2
Level :	Undergraduate/Postgraduate
Start Date :	24 Jan 2022
End Date :	18 Mar 2022
Enrollment Ends :	07 Feb 2022
Exam Date :	27 Mar 2022 IST

Note: This exam date is subject to change based on seat availability. You can check final exam date on your hall ticket.

Page Visits

Course layout

Week 1: Introduction: Entropy, Relative Entropy, Mutual Information;

Information Inequalities;

Week 2: Block to variable length coding-I: Prefix-free code

Block to variable length coding-II: Bounds on optimal codelength;

Block to variable length coding-III: Huffman coding.

Week 3: Variable to block length coding

The asymptotic equipartition property

Block to block coding of DMS

Week 4: Universal Source Coding-I: Lempel-Ziv Algorithm-LZ77

Universal source coding-II: Lempel-Ziv Welch Algorithm (LZW)

Week 5: Coding for sources with memory

Channel capacity of discrete memoryless channels.

Week 6: Joint typical sequences

Noisy channel coding theorem;

Differential entropy;

Week 7: Gaussian Channel;

Parallel Gaussian Channel.

Week 8: Rate Distortion Theory;

Blahut-Arimoto Algorithm for computation of channel capacity and rate- distortion function.

Books and references

James L. Massey, Lecture notes on ``Applied Digital Information Theory I''.

Thomas M. Cover, Joy A. Thomas, ``Elements of Information Theory'', 2nd Edition, John Wiley & Sons, 2006.

Robert G. Gallager, ``Information Theory and Reliable Communications'', John Wiley & Sons, 1968.

Raymond W. Yeung, ``Information Theory and Network Coding'', Springer, 2008.

David J. C. MacKay, ``Information Theory, Inference, and Learning Algorithms'', Cambridge University Press.

Robert Ash, ``Information Theory'', Dover Publications, 1965.

Imre Csiszar and Jonos Korner, ``Information Theory'', Second edition, Cambridge University Press, 2011.

Instructor bio

Prof. Adrish Banerjee

IIT Kanpur

Prof. Adrish Banerjee received his Bachelors degree from Indian Institute of Technology, Kharagpur and Masters and Ph.D. degree from University of Notre Dame, Indiana, USA. He is currently the Next Generation Broadcasting Chair Professor in the Department of Electrical Engineering at Indian Institute of Technology, Kanpur. He is a recipient of Microsoft Research India young faculty award, Institute of Engineers India young engineer award, and IETE Prof. K. Sreenivasan memorial award. His research interests are in the physical layer aspects of wireless communications, particularly green communications, and error control coding.

Course certificate

The course is free to enroll and learn from. But if you want a certificate, you have to register and write the proctored exam conducted by us in person at any of the designated exam centres.
The exam is optional for a fee of Rs 1000/- (Rupees one thousand only).
Date and Time of Exams: 27 March 2022 Morning session 9am to 12 noon; Afternoon Session 2pm to 5pm.
Registration url: Announcements will be made when the registration form is open for registrations.
The online registration form has to be filled and the certification exam fee needs to be paid. More details will be made available when the exam registration form is published. If there are any changes, it will be mentioned then.
Please check the form for more details on the cities where the exams will be held, the conditions you agree to when you fill the form etc.

CRITERIA TO GET A CERTIFICATE

Average assignment score = 25% of average of best 6 assignments out of the total 8 assignments given in the course.
Exam score = 75% of the proctored certification exam score out of 100

Final score = Average assignment score + Exam score

YOU WILL BE ELIGIBLE FOR A CERTIFICATE ONLY IF AVERAGE ASSIGNMENT SCORE >=10/25 AND EXAM SCORE >= 30/75. If one of the 2 criteria is not met, you will not get the certificate even if the Final score >= 40/100.

Certificate will have your name, photograph and the score in the final exam with the breakup.It will have the logos of NPTEL and IIT Kanpur .It will be e-verifiable at nptel.ac.in/noc.

Only the e-certificate will be made available. Hard copies will not be dispatched.

Once again, thanks for your interest in our online courses and certification. Happy learning.

- NPTEL team

SWAYAM Helpline / Support