Prerequisites
- An Apache Airflow deployment (Self-Hosted, Amazon MWAA, or Astronomer)
- Admin access to your Airflow instance
- API access enabled on your Airflow deployment
Step-by-Step Setup
1. Navigate to Integrations
In your Corelayer dashboard, go to the Integrations page.2. Add Airflow Integration
Click the Add Integration button and select Apache Airflow from the list of available integrations.3. Select Deployment Type
Choose your Airflow deployment type:- Self-Hosted — Standard Airflow deployment managed by your team
- Amazon MWAA — Managed Workflows for Apache Airflow on AWS
- Astronomer — Airflow managed by Astronomer
4. Configure Credentials
Depending on your deployment type: Self-Hosted:- Enter your Airflow Base URL (e.g.,
https://airflow.yourcompany.com) - Provide authentication credentials (username/password or API token)
- Enter your MWAA Environment Name
- Specify the AWS Region
- Ensure Corelayer’s AWS integration has access to the MWAA environment
- Enter your Astronomer API URL
- Provide your Organization ID and Deployment ID
- Enter your Astronomer API Key
5. Test Connection
Click Test Connection to verify that Corelayer can access your Airflow instance.6. Select DAGs
Once connected, select which DAGs you want Corelayer to monitor:- Browse the list of available DAGs
- Check the boxes next to the DAGs you want to include
- Click Update Selection
7. Add DAGs to Your Service Maps
When configuring your Corelayer Service Maps, select which Airflow DAGs are relevant to each service.What Corelayer Can Do With Airflow
Once connected, Corelayer will:- Monitor selected DAGs for task failures and errors
- Pull DAG run and task execution context for debugging
- Analyze pipeline failures and suggest remediation steps
- Correlate Airflow issues with other integrations for root-cause analysis
Managing Your Integration
Disconnecting
To disconnect your Airflow integration:- Navigate to Settings > Integrations
- Find your Airflow integration and click Uninstall
- Confirm the uninstallation
- Optionally, revoke the API credentials you created for Corelayer
Troubleshooting
Integration Not Working?
- Verify your Airflow instance is accessible from the internet (or through your network configuration)
- Ensure the API credentials are correct and have not expired
- Check that the Airflow REST API is enabled on your instance
- For MWAA, confirm the IAM role has the required permissions
- For Astronomer, verify the API key and deployment ID are correct