Cloud Computing and Distributed Systems

By Prof. Rajiv Misra | IIT Patna

Learners enrolled: 8614

Cloud computing is the on-demand delivery of computations, storage, applications, and other IT resources through a cloud services platform over the internet with pay-as-you-go business model. Today's Cloud computing systems are built using fundamental principles and models of distributed systems. This course provides an in-depth understanding of distributed computing “concepts”, distributed algorithms, and the techniques, that underlie today's cloud computing technologies. The cloud computing and distributed systems concepts and models covered in course includes: virtualization, cloud storage: key-value/NoSQL stores, cloud networking,fault-tolerance cloud using PAXOS, peer-to-peer systems, classical distributed algorithms such as leader election, time, ordering in distributed systems, distributed mutual exclusion, distributed algorithms for failures and recovery approaches, emerging areas of big data and many more. And while discussing the concepts and techniques, we will also look at aspects of industry systems such as Apache Spark, Google’s Chubby, Apache Zookeeper, HBase, MapReduce, Apache Cassandra, Google’s B4, Microsoft’s Swan and many others. Upon completing this course, students will have intimate knowledge about the internals of cloud computing and how the distributed systems concepts work inside clouds.

INTENDED AUDIENCE : NIL
PREREQUISITES :
Minimum: Data Structures and Algorithms

Ideal: Computer Architecture, Basic OS and Networking concepts

INDUSTRY SUPPORT : Companies like Amazon, Microsoft, Google, IBM, Facebook and start-ups working on this field.

Summary

Course Status :	Completed
Course Type :	Elective
Language for course content :	English
Duration :	8 weeks
Category :	Computer Science and Engineering
Credit Points :	2
Level :	Undergraduate/Postgraduate
Start Date :	18 Jan 2021
End Date :	12 Mar 2021
Enrollment Ends :	01 Feb 2021
Exam Date :	21 Mar 2021 IST

Note: This exam date is subject to change based on seat availability. You can check final exam date on your hall ticket.

Page Visits

Course layout

Week 1: Introduction to Clouds, Virtualization and Virtual Machine

1. Introduction to Cloud Computing: Why Clouds, What is a Cloud,Whats new in todays Clouds, Cloud computing vs. Distributed computing, Utility computing, Features of today’s Clouds: Massive scale, AAS Classification: HaaS, IaaS, PaaS, SaaS, Data-intensive Computing, New Cloud Paradigms, Categories of Clouds: Private clouds, Public clouds

2. Virtualization: What’s virtualization, Benefits of Virtualization, Virtualization Models: Bare metal, Hosted hypervisor

3. Types of Virtualization: Processor virtualization, Memory virtualization, Full virtualization, Para virtualization, Device virtualization

4. Hotspot Mitigation for Virtual Machine Migration: Enterprise Data Centers, Data Center Workloads, Provisioning methods, Sandipiper Architecture, Resource provisioning, Black-box approach, Gray-box approach, Live VM Migration Stages, Hotspot Mitigation

Week 2: Network Virtualization and Geo-distributed Clouds

1. Server Virtualization: Methods of virtualization: Using Docker,Using Linux containers, Approaches for Networking of VMs: Hardware approach: Single-root I/O virtualization (SR-IOV), Software approach: Open vSwitch, Mininet and its applications

2. Software Defined Network: Key ideas of SDN, Evolution of SDN,SDN challenges, Multi-tenant Data Centers: The challenges, Network virtualization, Case Study: VL2, NVP

3. Geo-distributed Cloud Data Centers: Inter-Data Center Networking, Data center interconnection techniques: MPLS, Google’s B4 and Microsoft’s Swan

Week 3: Leader Election in Cloud, Distributed Systems and Industry Systems

1. Leader Election in Rings (Classical Distributed Algorithms): LeLann-Chang-Roberts (LCR) algorithm, The Hirschberg and Sinclair (HS) algorithm

2. Leader Election (Ring LE & Bully LE Algorithm): Leader Election Problem, Ring based leader election, Bully based leader election, Leader Election in Industry Systems: Google’s Chubby and Apache Zookeeper

3. Design of Zookeeper: Race condition, Deadlock, Coordination, Zookeeper design goals, Data model, Zookeeper architecture, Sessions, States, Usecases, Operations, Access Control List (ACL), Zookeeper applications: Katta, Yahoo! Message Broker

Week 4: Classical Distributed Algorithms and the Industry Systems

1. Time and Clock Synchronization in Cloud Data Centers: Synchronization in the cloud, Key challenges, Clock Skew, Clock Drift, External and Internal clock synchronization, Christians algorithm, Error bounds, Network time protocol (NTP), Berkley’s algorithm, Datacenter time protocol (DTP), Logical (or Lamport) ordering, Lamport timestamps, Vector timestamps

2. Global State and Snapshot Recording Algorithms: Global state, Issues in Recording a Global State, Model of Communication, Snapshot algorithm: Chandy-Lamport Algorithm

3. Distributed Mutual Exclusion: Mutual Exclusion in Cloud, Central algorithm, Ring-based Mutual Exclusion, Lamport’s algorithm, Ricart-Agrawala’s algorithm, Quorum-based Mutual Exclusion, Maekawa’s algorithm, Problem of Deadlocks, Handling Deadlocks, Industry Mutual Exclusion : Chubby

Week 5: Consensus, Paxos and Recovery in Clouds

1. Consensus in Cloud Computing and Paxos: Issues in consensus, Consensus in synchronous and asynchronous system, Paxos Algorithm

2. Byzantine Agreement: Agreement, Faults, Tolerance, Measuring Reliability and Performance, SLIs, SLOs, SLAs, TLAs, Byzantine failure, Byzantine Generals Problem, Lamport-Shostak-Pease Algorithm, Fischer-Lynch-Paterson (FLP) Impossibility

3. Failures & Recovery Approaches in Distributed Systems: Local checkpoint, Consistent states, Interaction with outside world, Messages, Domino effect, Problem of Livelock, Rollback recovery schemes, Checkpointing and Recovery Algorithms: Koo-Toueg Coordinated Checkpointing Algorithm

Week 6: Cloud Storage: Key-value stores/NoSQL

1. Design of Key-Value Stores: Key-value Abstraction, Key-value/NoSQL Data Model, Design of Apache Cassandra, Data Placement Strategies, Snitches, Writes, Bloom Filter, Compaction, Deletes, Read, Membership, CAP Theorem, Eventual Consistency, Consistency levels in Cassandra, Consistency Solutions

2. Design of HBase: What is HBase, HBase Architecture, Components, Data model, Storage Hierarchy, Cross-Datacenter Replication, Auto Sharding and Distribution, Bloom Filter, Fold, Store, and Shift

Week 7: P2P Systems and their use in Industry Systems

1. Peer to Peer Systems in Cloud Computing: Napster, Gnutella, FastTrack, BitTorrent, DHT, Chord, Pastry and Kelips.

Week 8: Cloud Applications: MapReduce, Spark and Apache Kafka

1. MapReduce: Paradigm, Programming Model, Applications, Scheduling, Fault-Tolerance, Implementation Overview, Examples

2. Introduction to Spark: Resilient Distributed Datasets (RDDs), RDD Operations, Spark applications: Page Rank Algorithm, GraphX, GraphX API, GraphX working

3.Introduction to Kafka: What is Kafka, Use cases for Kafka, Data model, Architecture, Types of messaging systems, Importance of brokers

Books and references

Text Books:

1. Distributed and Cloud Computing From Parallel Processing to the Internet of Things- Kai Hwang, Jack Dongarra, Geoffrey Fox

2. Cloud Computing: Principles and Paradigms, Editors: Rajkumar Buyya, James Broberg, Andrzej M. Goscinski, Wile, 2011

3. Distributed Computing: Principles, Algorithms, and Systems- Ajay D. Kshemkalyani and Mukesh Singhal

4. Distributed Computing: Fundamentals, Simulations and Advanced Topics-Hagit Attiya and Jennifer Welch

Reference Book:

1.Distributed Algorithms-Nancy Lynch

2.Cloud Computing Bible, Barrie Sosinsky, Wiley-India, 2010

3.Cloud Computing: Principles, Systems and Applications, Editors: Nikos Antonopoulos, Lee Gillam, Springer, 2012

Instructor bio

Prof. Rajiv Misra

IIT Patna

Dr. Rajiv Misra is working in Department of Computer Science and Engineering at Indian Institute of Technology Patna, India. He obtained his Ph.D degree from IIT Kharagpur, M.Tech degree in Computer Science and Engineering from the Indian Institute of Technology (IIT) Bombay, and Bachelor's of engineering degree in Computer Science from MNIT Allahabad. His research interests spanned a design of distributed algorithms for Mobile, Adhoc and Sensor Networks, Cloud Computing and Wireless Networks. He has contributed significantly to these areas and published more than 70 papers in high quality journals and conferences, and 2 book chapters. His h-index is 10 with more than 590 citations. He has authored papers in IEEE Transactions on Mobile Computing, IEEE Transaction on Parallel and Distributed Systems, IEEE Systems Journal, Adhoc Networks, Computer Network, Journal of Parallel and Distributed Computing. He has edited a book titled as “Smart Techniques for a Smarter Planet: Towards Smarter Algorithms” for the "Studies in Fuzziness and Soft Computing" book series, Springer (2018). He has supervised four Phd students and currently four Phd students working under his supervision in the area of big data, cloud computing, distributed computing, and sensor networks. He is a senior member of the IEEE and fellow of IETE. He has completed as the Principal Investigator of R&D Project Sponsored by DeiTY entitled as “Vehicular Sensor and Mesh Networks based Future ITS”. He has mentored the online courses on Cloud Computing, Advanced Graph Theory and Distributed Systems in the platform of NPTEL.

Course certificate

The course is free to enroll and learn from. But if you want a certificate, you have to register and write the proctored exam conducted by us in person at any of the designated exam centres.
The exam is optional for a fee of Rs 1000/- (Rupees one thousand only).
Date and Time of Exams: 21 March 2021 Morning session 9am to 12 noon; Afternoon Session 2pm to 5pm.
Registration url: Announcements will be made when the registration form is open for registrations.
The online registration form has to be filled and the certification exam fee needs to be paid. More details will be made available when the exam registration form is published. If there are any changes, it will be mentioned then.
Please check the form for more details on the cities where the exams will be held, the conditions you agree to when you fill the form etc.

CRITERIA TO GET A CERTIFICATE

Average assignment score = 25% of average of best 6 assignments out of the total 8 assignments given in the course.
Exam score = 75% of the proctored certification exam score out of 100

Final score = Average assignment score + Exam score

YOU WILL BE ELIGIBLE FOR A CERTIFICATE ONLY IF AVERAGE ASSIGNMENT SCORE >=10/25 AND EXAM SCORE >= 30/75. If one of the 2 criteria is not met, you will not get the certificate even if the Final score >= 40/100.

Certificate will have your name, photograph and the score in the final exam with the breakup.It will have the logos of NPTEL and IIT Kanpur .It will be e-verifiable at nptel.ac.in/noc.

Only the e-certificate will be made available. Hard copies will not be dispatched.

Once again, thanks for your interest in our online courses and certification. Happy learning.

- NPTEL team

SWAYAM Helpline / Support