Skip to content

Download of URI folder is broken in 1.35.0 #48941

Description

@ChrisiPK
  • Package Name: azure-ai-ml
  • Package Version: 1.35.0
  • Operating System: macOS 26.6.2 (arm64); also reproduces in the python:3.14-slim (Linux) container used in production
  • Python Version: 3.14.6

Describe the bug

MLClient.jobs.download(name=job_name, output_name=<name>) silently downloads nothing for a named uri_folder output of a PipelineJob, without raising any error.

The root cause is in JobOperations._get_named_output_uri()get_job_output_uris_from_dataplane() (azure/ai/ml/operations/_job_ops_helper.py). That helper filters the RunHistory run_metadata.outputs dict to only the entries whose .type is a member of the SDK's own DataType enum:

dataset_ids = [
    run_outputs[output_name].asset_id
    for output_name in output_names
    if run_outputs[output_name].type in [o.value for o in DataType]
]

DataType's values are lowercase-with-underscore ("uri_file", "uri_folder", "mltable"), but for a pipeline job's outputs, the RunHistory dataplane reports the wire type as PascalCase ("UriFolder"), e.g.:

>>> client.jobs._runs_operations.get_run_data(job_name).run_metadata.outputs["forecast_data"].type
'UriFolder'
>>> 'UriFolder' in [o.value for o in DataType]
False

Because of this casing mismatch, every pipeline-job uri_folder/uri_file/mltable output is silently excluded from dataset_ids, get_batch_dataset_uris() is never called for it, and _get_named_output_uri() returns an empty dict for that output name. JobOperations.download() then just logs a debug message (Could not download output "<name>"...) and returns having downloaded nothing — no exception, no visible signal that anything went wrong.

The underlying data is completely fine: fetching the asset id directly and resolving it through the dataset dataplane manually (bypassing the broken filter) downloads it without any problem:

run_outputs = client.jobs._runs_operations.get_run_data(job_name).run_metadata.outputs
asset_id = run_outputs["forecast_data"].asset_id
uris = client.jobs._dataset_dataplane_operations.get_batch_dataset_uris([asset_id])
uri = uris.values_property[asset_id].uri
# uri resolves fine and download_artifact_from_aml_uri(uri=uri, ...) downloads the folder successfully

To Reproduce

  1. Deploy a PipelineJob (e.g. via a pipelineComponentBatchDeployment behind a batch endpoint, or any multi-step pipeline) with at least one uri_folder output at the top level (e.g. outputs = {"my_output": Output(type="uri_folder")}).
  2. Run the job to completion (status == "Completed").
  3. Call:
    client.jobs.download(name=job_name, download_path="./out", output_name="my_output")
  4. Observe that ./out is empty (or only contains unrelated artifacts), with no exception raised.
  5. To confirm the root cause directly:
    run_outputs = client.jobs._runs_operations.get_run_data(job_name).run_metadata.outputs
    print(run_outputs["my_output"].type)  # -> "UriFolder"
    from azure.ai.ml._restclient.arm_ml_service.models import DataType
    print(run_outputs["my_output"].type in [o.value for o in DataType])  # -> False

Expected behavior

client.jobs.download(output_name="my_output") should download the named output's contents, matching the output's actual type regardless of casing (or at minimum raise a clear error instead of silently succeeding with nothing downloaded). get_job_output_uris_from_dataplane()'s type filter should compare the wire value case/format-insensitively (e.g. normalize both sides, such as .lower().replace("_", "")), or use the DataType enum's actual wire values instead of the Python-side .value strings.

Screenshots

N/A

Additional context

  • This behaves correctly in azure-ai-ml 1.34.1, where the same call to jobs.download(output_name=...) for the same kind of pipeline job downloads correctly. Something in the 1.35.0 migration (changelog: "Migrated SDK entities and their consumers off the per-version msrest REST clients onto the shared arm_ml_service hybrid client") appears to have introduced or exposed this casing mismatch between the wire type and the DataType enum.
  • This is especially dangerous because it fails silently — no exception is raised, so callers have no signal that the download produced nothing unless they separately verify the expected files exist on disk afterward.
  • Confirmed against a real, long-completed batch-endpoint pipeline job in our workspace; the underlying blob data was present and fully intact the entire time — only the SDK's output resolution was broken.
  • Workaround we're using in application code: resolve the output's asset id via client.jobs._runs_operations.get_run_data(job_name).run_metadata.outputs[output_name].asset_id, resolve it to a URI via client.jobs._dataset_dataplane_operations.get_batch_dataset_uris([asset_id]), and download that URI with azure.ai.ml._artifacts._artifact_utilities.download_artifact_from_aml_uri. This relies on private/internal APIs and should not be necessary.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions