In this IBM Databand product update, we’re excited to announce our new support data observability for Azure Data Factory (ADF).
Customers using ADF as their data pipeline orchestration and data transformation tool can now leverage Databand’s observability and incident management capabilities to ensure the reliability and quality of their data.
Why use Databand with ADF?
- End-to-end pipeline monitoring: collect metadata, metrics, and logs from all dependent systems.
- Trend analysis: build historical trends to proactively detect anomalies and alert on potential issues.
- Custom alerting: create custom alert types that go beyond native ADF alerting.
- Central logging: get a single pane of glass for metadata collection across tools (e.g., Databricks, Spark, Airflow).
Watch the video, book a Databand demo, or keep reading to see how to get started using Databand and ADF together.
Setting up your ADF integration
Unlike other data observability solutions, Databand has a supported, no-code ADF integration for cloud and self-hosted options. After you have set up your linked services, head on over to the Databand integrations tab to set up your connection.
Select the ADF integration option, and then enter the integration details from your Azure subscription.
Then select the resource group(s) and their corresponding data factories you want connected. In this example, we selected “adf” as the resource group and “databand-east-us-adf1” as the data factory.
After the connection is saved, you can start monitoring your ADF factories and create alerts for any ADF data incident.
Creating an alert
To create a new ADF alert, select the add alert button and then choose the option for pipelines, tasks, or datasets.
For this blog, we’re going to highlight three different ADF alerts in Databand
- Task custom metric: create an alert with custom metrics logged by ADF.
- Run duration: identify pipeline durations that fall outside of defined limits or are anomalous.
- Run state: know when your pipeline has failed, succeeded, or entered some other specific state.
Task custom metric
ADF reports certain metrics to us regarding the execution of the pipeline.
One of those is the throughput metric which tells us how many records were processed per second in our data copy activity. If throughput is too low, it could mean that resources need to be adjusted somewhere.
In this case, we created an alert to tell us whenever throughput drops below 4,000 records per second.
[Image 6]
When the alert is triggered, Databand shows the user that the throughput was 3,949 records per second.
Run duration
Databand uses ML anomaly detection to alert users when pipelines fall outside an acceptable range. In this alert, our acceptable range of completion is between 557 and 943 seconds. If an ADF pipeline executes faster or slower outside of this range, Databand will trigger an alert.
Run state
If an ADF pipeline fails to execute or fails after executing, Databand will notify users immediately. In this example, we have created a critical severity alert to notify users when an ADF pipeline fails.
Upon further investigation, we can see that the downstream Databricks process was the cause of the pipeline alert.
Viewing ADF pipelines and runs in Databand
Databand lets you filter by individual Azure data factories rather than the entire integration.
This helps your team isolate and troubleshoot data factory issues more easily based on role-based access controls.
If you want to drill into each pipeline run, simply select a pipeline to enter it’s run history. In this example, we’re using service_311_request_analysis pipeline.
Run duration trend
Databand displays the full history of your ADF run durations. This way you can see if there are any issues or anomalies in the run behavior.
Task insights
Task insights will filter any failed tasks to the top. However, if there are no failed tasks then Databand will show you the tasks in order of last completed.
Advanced details
Depending on which task you select, Databand will show advanced details about the parameters that were passed to the ADF activities.
ADF metrics
Just like other Databand integrations (e.g., Airflow, DataStage), you can report on various metrics from your ADF pipeline behavior. At the run level, we include metrics like pipeline state, pipeline duration, task state, and task duration that we can use for anomaly alerting.
ADF natively reports some other interesting metrics that could be used in Databand alerts. Beyond reporting on just the schema and record count, Databand users can on things like throughput, copyDuration, dataWritten, filesWritten, rowsCopied, usedDataIntegrationUnits, and many more ADF-specific metrics.
ADF logging
Databand will automatically collect the input and output logs from your ADF activities. If there is an error log, Databand will surface the task overview on the run details screen.
Data interactions
Databand will automatically collect the input and output logs from your ADF activities. If there is an error log, Databand will surface the task overview on the run details screen.
In this example, we’re showing two operators which allows us to display operation lineage. If other pipelines read from these operations, Databand will automatically connect the downstream connections.
Want more?
See how Databand delivers continuous observability for Azure Data Factory to help detect data pipeline incidents and resolve them fast.
If you’re ready to take a deeper look, book a demo today.
#DataIntegration