Thank you for sending your enquiry! One of our team members will contact you shortly.
Thank you for sending your booking! One of our team members will contact you shortly.
Course Outline
Introduction to Apache Airflow
- What is workflow orchestration
- Key features and benefits of Apache Airflow
- Airflow 2.x improvements and ecosystem overview
Architecture and Core Concepts
- Scheduler, web server, and worker processes
- DAGs, tasks, and operators
- Executors and backends (Local, Celery, Kubernetes)
Installation and Setup
- Installing Airflow in local and cloud environments
- Configuring Airflow with different executors
- Setting up metadata databases and connections
Navigating the Airflow UI and CLI
- Exploring the Airflow web interface
- Monitoring DAG runs, tasks, and logs
- Using the Airflow CLI for administration
Authoring and Managing DAGs
- Creating DAGs with the TaskFlow API
- Using operators, sensors, and hooks
- Managing dependencies and scheduling intervals
Integrating Airflow with Data and Cloud Services
- Connecting to databases, APIs, and message queues
- Running ETL pipelines with Airflow
- Cloud integrations: AWS, GCP, Azure operators
Monitoring and Observability
- Task logs and real-time monitoring
- Metrics with Prometheus and Grafana
- Alerting and notifications with email or Slack
Securing Apache Airflow
- Role-based access control (RBAC)
- Authentication with LDAP, OAuth, and SSO
- Secrets management with Vault and cloud secret stores
Scaling Apache Airflow
- Parallelism, concurrency, and task queues
- Using CeleryExecutor and KubernetesExecutor
- Deploying Airflow on Kubernetes with Helm
Best Practices for Production
- Version control and CI/CD for DAGs
- Testing and debugging DAGs
- Maintaining reliability and performance at scale
Troubleshooting and Optimization
- Debugging failed DAGs and tasks
- Optimizing DAG performance
- Common pitfalls and how to avoid them
Summary and Next Steps
Requirements
- Experience with Python programming
- Familiarity with data engineering or DevOps concepts
- Understanding of ETL or workflow orchestration
Audience
- Data scientists
- Data engineers
- DevOps and infrastructure engineers
- Software developers
21 Hours
Delivery Options
Private Group Training
Our identity is rooted in delivering exactly what our clients need.
- Pre-course call with your trainer
- Customisation of the learning experience to achieve your goals -
- Bespoke outlines
- Practical hands-on exercises containing data / scenarios recognisable to the learners
- Training scheduled on a date of your choice
- Delivered online, onsite/classroom or hybrid by experts sharing real world experience
Private Group Prices RRP from €6840 online delivery, based on a group of 2 delegates, €2160 per additional delegate (excludes any certification / exam costs). We recommend a maximum group size of 12 for most learning events.
Contact us for an exact quote and to hear our latest promotions
Public Training
Please see our public courses
Testimonials (7)
The training was spot on. Very useful theory and exercices.
Vladimir - PUBLIC COURSE
Course - Apache Airflow
The training was spot on in all aspects. Usefull theoretical aspects and exercises.
Vladimir - PUBLIC COURSE
Course - Apache Airflow
The training was spot on in all aspects. Usefull theoretical aspects and exercises.
Vladimir - PUBLIC COURSE
Course - Apache Airflow
The training was spot on in all aspects. Usefull theoretical aspects and exercises.
Vladimir - PUBLIC COURSE
Course - Apache Airflow
The training was spot on in all aspects. Usefull theoretical aspects and exercises.
Vladimir - PUBLIC COURSE
Course - Apache Airflow
The training was spot on in all aspects. Usefull theoretical aspects and exercises.
Vladimir - PUBLIC COURSE
Course - Apache Airflow
The training was spot on in all aspects. Usefull theoretical aspects and exercises.
