Welcome to this course on ETL and Data Pipelines with Shell, Airflow, and Kafka. This course is ideal for aspiring Big Data Engineers, Machine Learning Engineers, Data Warehousing Specialists as well as Developers. In this course you will master: Various ETL and Data Pipeline Tools and Techniques. You will utilize Bash Scripts as well as cutting-edge open-source tools such as Apache Airflow, and Apache Kafka to build Data Pipelines for processing and moving data between various systems in a data platform. You will practice what you learn through Hands-on Labs in each module, as well as demonstrate your newly acquired skills in a real-world inspired Final Project. This course has four instructors: Yan Luo, Jeff Grossman, Sabrina Spillner, and Ramesh Sannareddy. Yan Luo, Ph.D., is a data scientist and developer at IBM Canada. He has been building innovative AI and cognitive applications in various areas, such as mining software repositories, personalized health management, wireless networks, and digital banking. Yan received his Ph.D. in Machine Learning from the University of Western Ontario. Jeff Grossman, Ph.D., has a background in Pure Mathematics, Geophysical Signal and Image Processing, Medical Imaging, and Data Science and Engineering. He is the founder of 617 Data Solutions Inc. and serves as a Subject Matter Expert at Skill-Up Technologies, developing data-related educational content, and volunteers with CAMDEA Digital Forum as Associate Member, Alberta, Canada. Sabrina Spillner is a Senior Instructional Designer and Content Developer with Skill-up Technologies. She specializes in learning solutions across sectors. For the past 18 years, Sabrina has been a pathfinder, adopting new approaches and technologies to create innovative learning solutions. She has worked with Orbus Software, the EU, the University of Cambridge, KPMG UK, and KPMG LG. Ramesh Sannareddy holds a Bachelors' Degree in Information Systems from the Birla Institute of Technology, Pilani. He has two and a half decades of experience in Information Technology Infrastructure Management, Database Administration, Information Integration, and Automation. He worked for companies such as Intergraph, Genpact, HCL, and Microsoft. Currently, he is a freelancer and pursues his passion for teaching. He teaches Data Science, Machine Learning, Programming, and Databases. In this course, you will explore the fundamental principles and techniques behind ETL and ELT processes. In module 1, Data Processing Techniques, you will learn to: Describe what an ETL pipeline is. Describe why ELT is an emergent trend. Describe the trending shift from ETL to ELT. List examples of raw data sources. Name data loading techniques. Differentiate batch loading from stream loading. You will also explore how to construct a basic ETL data pipeline from scratch using Bash shell-scripting, and explore use cases for the two main paradigms within data pipeline engineering: batch and streaming data pipelines. In module 2, ETL and Data Pipelines Tools and Techniques, you will learn to: Describe how shell scripting can be used to implement an ETL pipeline. Describe what a data pipeline is. Describe data pipeline solutions for mitigating data flow bottlenecks. Differentiate between batch and streaming data pipelines. Discuss data pipeline technologies. You will also further cement this knowledge by exploring and applying a popular open-source data pipeline tool named Kafka. In module 3, Building Data Pipelines Using Airflow, you will learn to: List the main principles of Apache Airflow. Interpret Airflow pipelines as Python scripts that define Airflow DAG objects. List key advantages of defining workflows as code. Identify current DAGs in your environment and set up dependencies amongst tasks. Use logging capabilities to monitor the task status and diagnose problems with DAG runs. Further, you will explore Apache Kafka, another popular open-source data pipeline tool, and use it to get hands-on experience with streaming data pipelines, implementing Kafka’s message producers and consumers, and creating a Kafka weather topic. In module 4, Building Streaming Pipelines Using Kafka, you will learn to: List the main components of an ESP. Recognize Apache Kafka as an Event Streaming Platform. Describe an end-to-end event streaming pipeline example. Describe what the Kafka Streams API is. And finally, you can test your hands-on knowledge in module 5 – Final Project. Your assignments will be to complete a project to create an ETL pipeline using an Airflow DAG and build a streaming ETL pipeline using Kafka. To get the most from this course, watch every video and check your learning by taking all quizzes. Use the discussion forums to connect with your peers and the teaching assistants. And most importantly, make sure you complete the hands-on labs to practice your new skills and demonstrate your abilities. Congratulations on beginning the next step on this exciting journey. And good luck!
