This course is an introduction to cloud computing designed for the students who wish to use the cloud for data science applications. It covers the topics of how cloud computing can be used to support data science workflows, including data storage, processing, analysis, and visualization. It also includes security considerations for the entire pipeline. Overall, the course provides students with the skills and knowledge necessary to effectively use cloud computing for design, implementation, test, and deployment of data science applications.

Lecture Time : Tuesday/Thursday - 11AM -12:30PM

Classroom: CEME-Floor 1-Room 1202

Office hours: Tuesday, 1PM - 3PM

The course will rely mainly on the following textbook.

•  Learning Spark: Lightning-fast Data Analytics by: Jules Damji,                         
  Brooke Wenig, Tathagata Das 

• Cloud service delivery models
• Cloud storage systems
• Batch processing
• Stream processing
• Cloud security
  
  

Mahmoud Alkhatib, Email: makhatib@students.cs.ubc.ca 

Arman Moztarzadeh, Email: arman88@student.ubc.ca

Sopuruchi Chisom, Email: schisom@student.ubc.ca 

Chen ZHao, Email: laozc@student.ubc.ca  

Download the syllabus (v1.0)

Introduction to Data centres and Cloud [SLIDES]

Cloud Computing Service Models, Serverless Computing [SLIDES] 

 Cloud Computing Service Models, Containerization [SLIDES]

Cloud Service Computing Models, Virtualization [SLIDES]

Big Data [SLIDES]

Data Stores [SLIDES1] , [SLIDES2] , [SLIDES3]

Data Management Systems [SLIDES]

Data Processing [SLIDES1] , [SLIDES2]

Structured Data Processing [SLIDES]

Distributed Machine Learning [SLIDES]

Stream Processing [SLIDES1] , [SLIDES2]

Graph Processing - Resource Management [SLIDES1] , [SLIDES2]

Cloud Security- part1: [SLIDES]

Cloud Security- part2: [SLIDES]

Guest Speaker-Cloud Security [SLIDES]

Guest Speaker-Advanced topics [SLIDES]

• Assignment 0: Go Serverless (5%); [AWS] [Azure] [Rubric]
• Assignment 1:  Containerization Vs. Serverless (5%);  [AWS] [Azure] [Rubric]
• Assignment 2:  Running Image recognition on a Virtual Machine (5%);  [AWS] [Azure] [Rubric]
• Assignment 3: Running image recognition in a VM using Object Store (5%);  [AWS] [Azure] [Rubric]
• Assignment 4: Comparing Single Node vs Cluster Performance and Cost in Image Classification Using Amazon EMR (5%);  [AWS] [Azure] [Rubric]
• Assignment 5: Streaming Text Analysis using Spark (5%);  [AWS] [Azure] [Rubric]
• Assignment 6: Building a Machine Learning Pipeline through Jupyter Notebook (10%);  [Non-simplified] [Simplified] [Rubric]

• Tutorial 1:  [AWS] [Azure] 
• Tutorial 2:  [AWS] [Azure] 
• Tutorial 3:  [AWS] [Azure] 
• Tutorial 4:  [AWS] [Azure] 

In teams of up to 5, students will design and implement a cloud-based data processing pipeline using one storage system, one computing engine, and one database to store or visualize the results. The project topic is student-chosen but must cover the entire pipeline, giving students the opportunity to apply the practical skills learned in the course to solve a real-world problem. Evaluation will focus on the quality of the architecture, cost-efficiency, performance, and security. The project will culminate in a final presentation at the end of the semester [Project Description Document]

• A Survey on the Evolution of Stream Processing Systems
• Bigtable: A Distributed Storage System for Structured Data                                         
• Clipper: A Low-Latency Online Prediction Serving System | USENIX
• Lakehouse: A New Generation of Open Platforms that Unify Data Warehousing
  and Advanced Analytics 
• The Snowflake Elastic Data Warehouse
• The Datacenter as a Computer, An Introduction to the Design of Warehouse-Scale Machines
• Dominant Resource Fairness: Fair Allocation of Multiple Resource Types
• Dynamo: Amazon's Highly Available Key-value Store
• Apache Flink™: Stream and Batch Processing in a Single Engine
• GraphX: Graph Processing in a Distributed Dataflow Framework
• The Hadoop Distributed File System
• Twitter Heron: Stream Processing at Scale
• Hive - A Warehousing Solution Over a Map-Reduce Framework
• Cassandra - A Decentralized Structured Storage System
• MapReduce: Simplified Data Processing on Large Clusters
• Mesos: A Platform for Fine-Grained Resource Sharing in the Data Center
• Delta Lake: High-Performance ACID Table Storage over Cloud Obiect Stores
• Scaling Distributed Machine Learning with the Parameter Server 
• PowerGraph: Distributed Graph-Parallel Computation on Natural Graphs
• Pregel: A System for Large-Scale Graph Processing
• PyTorch Distributed: Experiences on Accelerating Data Parallel Training
• Ray: A Distributed Framework for Emerging Al Applications
• Locking the sky: a survey on laaS cloud security
• Discretized Streams: Fault-Tolerant Streaming Computation at Scale
• Resilient Distributed Datasets: A Fault-Tolerant Abstraction for
  In-Memory Cluster Computing
• Spark SQL: Relational Data Processing in Spark
• The Deep Learning Compiler: A Comprehensive Survey
• TVM: An Automated End-to-End Optimizing Compiler for Deep Learning