Category Archives: airflow

Apache Airflow – user mangement and multi-tenancy

I'm trying to create a single Apache Airflow instance that can host multiple users, that will be unable to affect one another negatively. This would include but not limited to:

  • Maintain their own python environments
  • Manage their own variables
  • Manage their own connections
  • Manage their own dags

Looking through the official airflow docs. I see a couple things that may help.

1) ability to create users and 2) the ability to be multi tenant.

1) If I follow the user creation process https://airflow.apache.org/security.html#web-authentication. All the users created seem to be admin, but how do I create a non admin user and control what can they do / not do? I can't seem to locate anymore documentation.

2) The link, https://airflow.apache.org/security.html#multi-tenancy, says that "You can filter the list of dags in webserver by owner name when authentication is turned on by setting ", but I don't see how I can assign dags to specific users.

Thanks for the help.

Unable to setup a DB2 / DashDB JDBC Connection in Apache Airflow

I'm trying to create a DB2 / DashDB connection using the Airflow UI. I have added the db2jcc4.jar driver and provided the path as well as the class name com.ibm.db2.jcc.DB2Driver.class

I tried to run a simple query (in the ad hoc UI) and always get the same error

java.lang.RuntimeException: Class com.ibm.db2.jcc.DB2Driver.class not found

Did anybody need to setup a DB2 / DashDB connection in Apache Airflow before?

Found nothing on the web about that.

Thanks

Airflow "This DAG isnt available in the webserver DagBag object "

when I put a new DAG python script in the dags folder, I can view a new entry of DAG in the DAG UI but it was not enabled automatically. On top of that, it seems does not loaded properly as well. I can only click on the Refresh button few times on the right side of the list and toggle the on/off button on the left side of the list to be able to schedule the DAG. These are manual process as I need to trigger something even though the DAG Script was put inside the dag folder.

Anyone can help me on this ? Did I missed something ? Or this is a correct behavior in airflow ?

By the way, as mentioned in the post title, there is an indicator with this message "This DAG isn't available in the webserver DagBag object. It shows up in this list because the scheduler marked it as active in the metdata database" tagged with the DAG title before i trigger all this manual process.