Getting started with AirFlow on WSL

AirFlow

Apache Airflow is an open-source workflow management platform. It started at Airbnb in October 2014 as a solution to manage the company’s increasingly complex workflows. Airflow helps you to create workflows using Python programming language and these workflows can be scheduled and monitored easily with it.

Features of Airflow

• Pure Python
• Useful Ui
• Robust Integrations
• Easy to Use
• Open Source

Install Airflow on Ubuntu

Login to Ubuntu and install pip3 first in order to install airflow.

sudo apt-get -y install python3-pip

One pip3 is installed run below to install airflow.

pip3 install apache-airflow==1.10.12

Set airflow home path using

export AIRFLOW_HOME=~/airflow

Run airflow from terminal to create airflow folder and basic configuration file. Move to the airflow home folder, it would have airflow.cfg and other files. Verify airflow version using airflow version command.

airflow version

Create dags folder in airflow home folder if not available. Run airflow db init to setup sqlite database with needed database tables.

airflow db init

If you are using airflow version 2.0 and above then you need to create user to access UI.

airflow users create     --role Admin     --username admin     --firstname <first name>     --lastname <last name>     --email <emailid>

With this, we are all set to start airflow webserver and login to ui using above created user to have a look at ui.

airflow webserver

If all goes well then you can access airflow admin home on http://localhost:8080/home. Login to ui using created user. On the login page you can see lot of example dags loaded in in-active state.

Let us start with looking at code of example and create some dags to start playing with it. Best way to learn airflow is to copy any example dag code and modify the code according to your need and put back the files with new name and dag name in dags folder.

Important Links

Set of links which are handy while working with airflow.

• Install Airflow and scale using celery executor and message queuing service rabbitmq.
• Custom email alert on task failure.
• Examples on using xcom in airflow.
• Customize airflow using plugins.
• Dynamic workflow in airflow.
• Custom rest api using plugin with basic token authentication.
• Custom rest api using airflow plugins with basic authentication.
• Complete airflow documentation in detail.

Upgrade airflow to latest 2.*

Follow the below steps given on link to upgrade to airflow 2.*.

Note :- Some of the of the content are taken from wikipedia, airflow.