Every day, there seems to be a new integration use case for data observability. Whether it’s a shiny, modern data pipeline tool or that legacy workhorse that’s never going away, data observability spans both data stacks.
With that in mind, we’re excited to release Databand’s new public API, which will address all your data observability integration needs.
Our new API is based on the widely used Open Lineage schema, which makes it easier for users to post metadata directly to their Databand environment. We encourage customers to start building their own integrations, and we will continue to prioritize releasing supported integrations like Azure Data Factory based on common customer requests.
With your custom API integration, you can track and define alerts for the following metadata:
- State alerts for runs and tasks
- Duration alerts for runs and tasks
- Capturing logs for runs and tasks
- Visual representations of your pipelines and their tasks, such as DAGs
- Task parameters
Getting Started
To start a custom Databand integration, select the integration tab, click the Add Integration button, and select the custom integration option.
You will need to provide the name of the tool you are integrating
This will generate an access token for API credentials.
Click the link to start writing your custom integration, and check out this example.
Sample Integration with GitLab and Tekton
To give you an idea of how to use Databand’s API, we’ve built a custom integration with our internal CI/CD pipelines using GitLab and Tekton. Until now, we haven’t considered CI/CD pipelines part of a traditional data observability use case.
However, we can now monitor the status of our CI/CD pipelines using the Databand API. Our team has full visibility into their pipelines, access to error messages, and overviews of each pipeline's duration.
This integration was fully built with the custom integration APIs. Like our native integrations (e.g., Airflow, dbt), you’ll have a similar and familiar experience.
For example, we’re tracking the overall state and duration of the pipeline and collecting logs from each pipeline's task level.
Whatever logs we collect from each step of our GitLab and Tekton pipelines are also shown in Databand.