Apache Airflow: Building and Managing Data Pipelines Training Course

Apache Airflow serves as an open-source framework for designing, scheduling, and overseeing workflows. Through Airflow 2.x, organisations gain the capability to coordinate large-scale data pipelines, integrate seamlessly with cloud infrastructure, and maintain high levels of observability and security within production settings.

Delivered by an instructor either online or on-site, this training is tailored for data and DevOps professionals at beginner to intermediate levels who aim to utilise Apache Airflow for constructing, automating, and scaling comprehensive data pipelines.

Upon completion, participants will possess the ability to:

Deploy and configure Apache Airflow across diverse environments.
Design, schedule, and oversee DAGs leveraging the TaskFlow API.
Connect Airflow with various data sources, cloud platforms, and machine learning workflows.
Orchestrate ETL pipelines while ensuring data integrity.
Enhance security and scalability using various executors, queues, and Kubernetes.
Implement observability standards and best practices for live production deployments.

Course Format

Engaging lectures paired with interactive discussions.
Extensive exercises and practical application.
Real-time implementation within a live-lab setting.

Customisation Options

To request a tailored version of this training, please reach out to us to discuss arrangements.

This course is available as onsite live training in Netherlands or online live training.

Thank you for sending your enquiry! One of our team members will contact you shortly.

Thank you for sending your booking! One of our team members will contact you shortly.

Duration 21 hours (3 days)

Course Outline

Introduction to Apache Airflow

Defining workflow orchestration
Primary features and advantages of Apache Airflow
Enhancements in Airflow 2.x and an overview of its ecosystem

Architecture and Fundamental Concepts

Roles of the scheduler, web server, and worker processes
Understanding DAGs, tasks, and operators
Executors and backends (Local, Celery, Kubernetes)

Installation and Configuration

Deploying Airflow in local and cloud-based settings
Configuring Airflow with various executors
Establishing metadata databases and connections

Exploring the Airflow UI and CLI

Navigating the Airflow web interface
Tracking DAG runs, tasks, and logs
Utilising the Airflow CLI for administrative tasks

Creating and Managing DAGs

Building DAGs via the TaskFlow API
Applying operators, sensors, and hooks
Handling dependencies and defining scheduling intervals

Connecting Airflow with Data and Cloud Services

Linking databases, APIs, and message queues
Executing ETL pipelines through Airflow
Cloud integrations: AWS, GCP, and Azure operators

Monitoring and Observability

Task logs and real-time supervision
Utilising Prometheus and Grafana for metrics
Configuring alerts and notifications via email or Slack

Securing Apache Airflow

Implementing Role-Based Access Control (RBAC)
Authentication methods using LDAP, OAuth, and SSO
Managing secrets with Vault and cloud-based secret stores

Scaling Apache Airflow

Managing parallelism, concurrency, and task queues
Leveraging CeleryExecutor and KubernetesExecutor
Deploying Airflow on Kubernetes using Helm

Production Best Practices

Implementing version control and CI/CD for DAGs
Testing and debugging DAGs effectively
Ensuring reliability and performance at scale

Troubleshooting and Performance Tuning

Diagnosing failed DAGs and tasks
Enhancing DAG execution performance
Avoiding common pitfalls

Summary and Recommended Next Steps

Requirements

Proficiency in Python programming
General knowledge of data engineering or DevOps principles
Basic understanding of ETL processes or workflow orchestration

Intended Audience

Data scientists
Data engineers
DevOps and infrastructure engineers
Software developers

Custom Corporate Training

Training solutions designed exclusively for businesses.

Customized Content: We adapt the syllabus and practical exercises to the real goals and needs of your project.
Flexible Schedule: Dates and times adapted to your team's agenda.
Format: Online (live), In-company (at your offices), or Hybrid.

Investment

Price per private group, online live training, starting from 4800 € + VAT*

(*The final price may vary depending on the technical specialization of the course, the level of customization, the method of delivery and the number of learners)

Need help picking the right course?
opleidingen@nobleprog.com or +31 208 080 666

Testimonials (7)

The instructor adapted the training to the participants’ level and responded to all questions. He was very communicative, and it was easy to interact with him. I really appreciated the format of the training, which included many practical exercises. Overall, it was a very engaging and well-organized session.

Jacek Chlopik - ZAKLAD UBEZPIECZEN SPOLECZNYCH