Building ETL and Data Pipelines with Bash, Airflow and Kafka
About this Course
Well-designed and automated data pipelines and ETL processes are the foundation of a successful Business Intelligence platform. Defining your data workflows, pipelines and processes early in the platform design ensures the right raw data is collected, transformed and loaded into desired storage layers and available for processing and analysis as and when required. This course is designed to provide you the critical knowledge and skills needed by Data Engineers and Data Warehousing specialists to create and manage ETL, ELT, and data pipeline processes. Upon completing this course you’ll gain a solid understanding of Extract, Transform, Load (ETL), and Extract, Load, and Transform (ELT) processes; practice extracting data, transforming data, and loading transformed data into a staging area; create an ETL data pipeline using Bash shell-scripting, build a batch ETL workflow using Apache Airflow and build a streaming data pipeline using Apache Kafka. You’ll gain hands-on experience with practice labs throughout the course and work on a real-world inspired project to build data pipelines using several technologies that can be added to your portfolio and demonstrate your ability to perform as a Data Engineer. This course pre-requisites that you have prior skills to work with datasets, SQL, relational databases, and Bash shell scripts.Created by: IBM
Level: Introductory
Related Online Courses
Port cities are dynamic environments. They face ever-changing challenges and demands from port activities under continually evolving economic and environmental circumstances. They also offer a rich... more
Please Note: Learners who successfully complete this IBM course can earn a skill badge — a detailed, verifiable and digital credential that profiles the knowledge and skills you’ve acquired in thi... more
There is a rising demand for people with the skills to work with Big Data sets and this course can start you on your journey through our Big Data MicroMasters program towards a recognised... more
El mayor activo de las empresas actuales son sus datos, datos que, mayoritariamente se alojan en bases de datos relacionales en línea. Prácticamente todos los sistemas de gestión de bases de da... more
AI is transforming how we live, work, and play. By enabling new technologies like self-driving cars and recommendation systems or improving old ones like medical diagnostics and search engines, the... more