We are excited to share the release of watsonx.data integration SDK version 1.1.0. This update focuses on streamlining the developer experience and providing deeper control over data flows through improved object management.

Enhanced UX for Stages and Links

The core of this release is a significant overhaul of how stage and link objects are accessed and displayed. We have simplified the internal mapping to ensure that interacting with your flow topology feels intuitive and programmatic.

Accessing stages

You can now list all stages within batch flows and receive a clearly labeled list of objects. This makes it easier to identify specific connectors or operators within a complex pipeline.

Code: flow.stages

Output:

[PostgresqlStage(label='transactions_1'),
 PeekStage(label='Peek_1'),
 FilterStage(label='Filter_1'),
 TransformerStage(label='Transformer_1'),
 MysqlStage(label='TM_DS_DB_1_1'),
 PeekStage(label='Peek_2'),
 SequentialFileStage(label='Sequential_file_1')]

Improved stage configuration output display

When viewing configurations of a stage, it is now presented in a formatted, readable structure that allows for immediate inspection of parameters, connection details, and runtime settings. This clean representation helps developers quickly identify configuration values without parsing through dense blocks of raw data.

Code:

flow.stages.get(label = "transaction_1").configuration

Output:

{
    "buf_mode": "default",
    "max_mem_buf_size": 3145728,
    "buf_free_run": 50,
    "queue_upper_size": 0,
    "disk_write_inc": 1048576,
    "runtime_column_propagation": false,
    "connection": {
        "asset_id": null,
        "proj_id": null,
        "raw_properties": null,
        "name": "PostgreSQL_conn",
        "discover_data_assets": null,
        "cluster_access_token": null,
        "cluster_user_name": null,
        "database": "cpd",
        "defer_credentials": false,
        "hostname_or_ip_address": "52.116.198.152",
        "password": "DataDuck!",
        "port": "5432",
        "query_timeout": 300,
        "retry_limit": 2
    },
    "add_procedure_return_value_to_schema": false,
    "batch_size": 3000,
    "buffering_mode": "default"
    ...
}

Simplified Configuration Management

Managing stage properties is now a first class experience. The configuration output provides a comprehensive view of connection details, runtime parameters, and resource limits. Developers can now update specific parameters like batch size with just a few lines of code.

Code:

flow.stages.get(label = "transaction_1").configuration["batch_size"] = 2000
project.update_flow(flow)

Output: <Response [201]>

Deep Schema Visibility

Understanding the data moving through your links is critical for debugging and transformation logic. The 1.1.0 SDK provides a structured schema output for links, detailing types, nullability, and technical metadata.

Accessing Links

Code: flow.links

Output:

[Link_1 (src='transactions_1', dest='Peek_1'),
 Link_2 (src='Peek_1', dest='Filter_1'),
 Link_3 (src='Filter_1', dest='Transformer_1'),
 Link_4 (src='Transformer_1', dest='TM_DS_DB_1_1'),
 Link_5 (src='Transformer_1', dest='Peek_2'),
 Link_6 (src='Peek_2', dest='Sequential_file_1')]

Viewing Link Schema

Code: flow.links[0].schema

Output:

[
    {
        "name": "id",
        "type": "integer",
        "metadata": {
            "description": "",
            "max_length": 6,
            "decimal_precision": 100,
            "is_key": false
        },
        "nullable": false
    },
    ...
]

Topology Persistence and Consistency

A major functional improvement in this version is the persistence of flow metadata during duplication or editing. Whether you are working with streaming or batch processes, the SDK now ensures the following remain intact across operations:

• Topology and Structure: The arrangement and relationship between stages are preserved.
• Annotations and Comments: Critical documentation within the flow is no longer lost during updates.

Bug Fixes and Stability

In addition to these features, 1.1.0 includes numerous stability improvements and bug fixes. These updates resolve previous inconsistencies in how data assets were referenced and improve the reliability of flow updates via the project interface.

To get started with our SDK install the ibm-watsonx-data-integration via pip today

pip3 install ibm-watsonx-data-integration

To see more details and view code examples, visit our documentation here.

Try out watsonx.data integration for free today!