Python: [Bug]: Code Interpreter file citations are lost through FoundryToolbox hosting

[eamonoreilly]

Description

Summary

When an Agent Framework agent invokes Code Interpreter through a Foundry FoundryToolbox, Code Interpreter creates the requested file, but the hosted Responses API does not return a native container_file_citation. The customer therefore receives no container_id or file_id and cannot download the generated file through the container-files API.

Reproduction

1. Configure a Foundry Toolbox with a code_interpreter tool
2. Create an Agent Framework agent using FoundryToolbox and ResponsesHostServer
3. Ask the agent to write /mnt/data/result.txt and attach it
4. Invoke the hosted agent through its Responses endpoint
5. Inspect the response output and message annotations

Observed behavior

The hosted response exposes the tool result as a regular function_call_output, with no file citation. However, a direct MCP tools/call to the same Toolbox returns the required metadata on the text content block:

{
  "type": "text",
  "text": "...",
  "_meta": {
    "container_id": "cntr_...",
    "container_file_citations": "[{\"file_id\":\"cfile_...\",\"filename\":\"result.txt\",\"container_id\":\"cntr_...\"}]"
  }
}

This indicates that Code Interpreter and the Toolbox service produce the artifact identifiers, but Agent Framework does not carry them through to the hosted Responses output. In the Python path, per-content MCP _meta appears to be dropped during MCP result conversion. The hosting adapter also has no conversion from this metadata to a Responses container_file_citation annotation.

Expected behavior

Agent Framework should preserve the per-content metadata and emit an annotation equivalent to:

{
  "type": "container_file_citation",
  "container_id": "cntr_...",
  "file_id": "cfile_...",
  "filename": "result.txt"
}

Code Sample

import asyncio
import os

from agent_framework import Agent
from agent_framework.foundry import FoundryChatClient
from agent_framework_foundry_hosting import FoundryToolbox, ResponsesHostServer
from azure.identity import DefaultAzureCredential


async def main():
    credential = DefaultAzureCredential()
    agent = Agent(
        client=FoundryChatClient(
            project_endpoint=os.environ["FOUNDRY_PROJECT_ENDPOINT"],
            model=os.environ["AZURE_AI_MODEL_DEPLOYMENT_NAME"],
            credential=credential,
        ),
        instructions=(
            "Use Code Interpreter when asked to create a file. Write files to "
            "/mnt/data and attach them to the response."
        ),
        tools=FoundryToolbox(credential),
        default_options={"store": False},
    )
    await ResponsesHostServer(agent).run_async()


if __name__ == "__main__":
    asyncio.run(main())



services:
  agent-tools:
    host: azure.ai.toolbox
    uses:
      - ai-project
    tools:
      - type: code_interpreter
        name: code_interpreter

  repro-agent:
    host: azure.ai.agent
    project: src/repro-agent
    uses:
      - ai-project
      - agent-tools
    protocols:
      - protocol: responses
        version: 2.0.0
    environmentVariables:
      - name: AZURE_AI_MODEL_DEPLOYMENT_NAME
        value: ${AZURE_AI_MODEL_DEPLOYMENT_NAME}
      - name: TOOLBOX_NAME
        value: agent-tools

Error Messages / Stack Traces

Package Versions

agent-framework-core==1.10.0 agent-framework-foundry==1.8.2 agent-framework-foundry-hosting==1.0.0a260630 mcp==1.28.0

Python Version

Python 3.14

Additional Context

No response

[linhongyu510]

I can take this. I plan to keep the change focused and provider-independent:

1. Preserve per-content MCP _meta when converting CallToolResult.content into Agent Framework Content, using the existing additional_properties metadata surface rather than adding a new public type.
2. In the Foundry Responses hosting conversion, recognize validated container_file_citations metadata and emit native container_file_citation annotations.
3. Add unit tests for both conversion boundaries, including malformed/absent metadata remaining a normal function result.

I will avoid any live Foundry dependency in the tests. Please let me know if the team prefers a different metadata key or normalization boundary.

[linhongyu510]

The implementation is ready on my fork for review of the approach before I open the upstream PR: linhongyu510/agent-framework@main...linhongyu510:agent-framework:fix/7916-container-file-citations

The final implementation also accounts for two adjacent correctness boundaries:

• IFC metadata is merged conservatively (untrusted / private dominate), and the security proxy consumes only the ifc sub-key so file citation metadata remains available downstream.
• Citations wait for an assistant message that actually contains the filename; a newer citation for the same filename replaces an older pending one.

Validation completed locally:

• Core package: 4,169 passed, 116 skipped, 2 xfailed (88% coverage)
• Foundry hosting package: 243 passed (90% coverage)
• Core and Foundry syntax checks passed
• Changed source/tests pass strict Pyright; Foundry source and all five test type checkers pass

If this normalization boundary looks right to the core team, I will open the PR against main.