diff --git a/docs/specs/004-python-function-calling-loop.md b/docs/specs/004-python-function-calling-loop.md index 8381d6b65c..6b859b2716 100644 --- a/docs/specs/004-python-function-calling-loop.md +++ b/docs/specs/004-python-function-calling-loop.md @@ -434,6 +434,12 @@ that manually replay messages own the equivalent rule: do not resend an approval original shape and fails explicitly when that provider representation is unavailable or inconsistent. Locally generated shell outputs preserve stdout, stderr, truncation notices, non-zero exits, and timeout outcomes and are sent in both service-side and stateless continuation modes. + A `shell_call` for a configured local executor with valid item/call identifiers and `commands=[]` is a no-op: + return the call paired with a `shell_call_output` containing `output=[]`, without entering local execution or + approval. Preserve `max_output_length` when supplied. This applies to explicit local environments and omitted + environments resolved to the configured local executor; hosted, unconfigured, and malformed calls are unchanged. + Service-side continuation submits the locally generated empty output without replaying the stored call, including + when the same response also contains ordinary function calls whose results drive the next model iteration. ### Reasoning-bound calls @@ -592,6 +598,7 @@ that manually replay messages own the equivalent rule: do not resend an approval | Parallel calls | Results retain the corresponding call ids and execution count. | `test_max_function_calls_limits_parallel_invocations`, `test_streaming_multiple_function_calls_parallel_execution` | | Sequential calls | Calls execute one at a time in model order, including after approval replay; mixed local/Host-owned siblings retain that order, approval-only batches wait for every decision before executing, each call has isolated context, and middleware termination takes effect after the accepted batch finishes. | `packages/core/tests/core/test_tools.py::test_sequential_function_invocation_runs_calls_in_model_order`, `test_sequential_function_invocation_isolates_context`, `test_sequential_function_invocation_finishes_batch_after_termination`, `packages/core/tests/core/test_harness_tool_approval.py::test_mixed_batch_hides_already_approved_request_until_approval_replay`, `test_sequential_approval_replay_preserves_model_order_when_responses_are_reversed`, `test_sequential_approval_replay_waits_for_the_complete_batch`, `packages/core/tests/core/test_function_invocation_logic.py::test_sequential_mixed_batch_preserves_approval_and_host_model_order`, `packages/core/tests/workflow/test_agent_executor_tool_calls.py::test_workflow_cancels_host_member_of_mixed_batch_with_terminal_result`, `test_workflow_host_cancellation_preserves_occurrence_with_reused_call_id` | | Informational-only call | The call is returned but not executed or approved. | `test_informational_only_function_call_is_not_invoked`, `test_informational_only_function_call_does_not_request_approval`, `test_streaming_informational_only_function_call_is_not_invoked` | +| Empty local shell call | An empty command list for a configured local executor returns a paired empty shell output without execution or approval; both parsers preserve the pair and stateless replay preserves output limits. Response-ID and conversation continuation submit local outputs once, including mixed function-call batches, without replaying stored calls or hosted outputs. Hosted, unconfigured, and malformed calls do not receive synthetic results. | `packages/openai/tests/openai/test_openai_chat_client.py::test_empty_local_shell_call_replays_with_empty_output`, `test_empty_local_shell_call_parsing_emits_pair_without_execution`, `test_empty_local_shell_output_survives_mixed_tool_continuation`, `test_empty_shell_result_is_not_synthesized_for_hosted_unconfigured_or_malformed_calls` | | OpenAI hosted/local shell boundary | Hosted shell calls remain informational in streaming and non-streaming responses even when a local executor is configured; only valid explicit local-shell items or shell calls marked with a local environment can execute, local execution preserves its configured approval mode, stateless loops preserve the complete provider shell transcript or fail explicitly, and locally generated shell outputs preserve stdout, stderr, truncation notices, non-zero exits, and timeout outcomes in every continuation mode. | `packages/openai/tests/openai/test_openai_chat_client.py::test_response_content_creation_with_shell_call_remains_hosted_with_local_tool`, `test_parse_chunk_from_openai_shell_call_done_remains_hosted`, `test_parse_chunk_from_openai_local_environment_shell_call_done_emits_command`, `test_mixed_shell_calls_only_invoke_explicit_local_shell_call`, `test_stateless_shell_transcript_without_provider_item_fails`, `test_prepare_messages_keeps_local_shell_output_under_storage`, `test_shell_output_payloads_use_structured_function_result_items`, `test_malformed_local_environment_shell_call_is_not_executable`, `test_response_content_creation_with_local_shell_call_maps_to_function_call`, `test_malformed_local_shell_call_is_not_executable`, `test_response_function_call_named_local_shell_is_informational`, `test_parse_chunk_function_call_named_local_shell_is_informational`, `test_local_shell_tool_requires_approval_before_function_loop_execution` | | Declaration-only call | The call is surfaced as user input and is not executed; streaming arguments appear once while finalized request metadata remains available. | `test_declaration_only_tool`, `test_streaming_declaration_only_tool_preserves_metadata_without_duplicate_arguments` | | Declaration-only mixed batch | Known local siblings are never converted into Host pauses. With an authoritative session they remain deferred until every Host pause occurrence is answered, survive session serialization with a process-portable logical-run budget (including a reclassified approval continuation), reclassify against the current tool policy, execute exactly once using the configured concurrent or sequential invocation mode, replay results in original model order, and remain inert on replay; without one they execute on the same turn under the same configured invocation mode. Streaming updates and finalization preserve one Host pause and one result per local sibling. | `test_mixed_declaration_only_batch_defers_siblings_until_all_host_results`, `test_mixed_declaration_only_deferred_call_keeps_ordinary_middleware_semantics`, `test_mixed_declaration_only_deferred_call_reclassifies_changed_approval_policy`, `test_mixed_declaration_only_budget_survives_reclassification_to_approval`, `test_mixed_declaration_only_deferred_call_does_not_execute_removed_tool`, `test_mixed_declaration_only_budget_is_process_portable_without_approval_middleware`, `test_sessionless_mixed_declaration_only_batch_executes_siblings_sequentially`, `test_sessionless_mixed_declaration_only_batch_executes_siblings_concurrently`, `test_streaming_sessionless_mixed_declaration_only_batch_returns_pause_and_result` | diff --git a/python/packages/core/agent_framework/_tools.py b/python/packages/core/agent_framework/_tools.py index 0d607e4845..fe87e8868c 100644 --- a/python/packages/core/agent_framework/_tools.py +++ b/python/packages/core/agent_framework/_tools.py @@ -104,6 +104,7 @@ def _generate_function_call_occurrence_id() -> str: DEFAULT_MAX_ITERATIONS: Final[int] = 40 DEFAULT_MAX_CONSECUTIVE_ERRORS_PER_REQUEST: Final[int] = 3 SHELL_TOOL_KIND_VALUE: Final[str] = "shell" +_LOCAL_TOOL_RESULT_KEY: Final[str] = "agent_framework.local_tool_result" _TOOL_APPROVAL_STATE_KEY: Final[str] = "tool_approval" _APPROVAL_SESSION_IS_AUTHORITATIVE_KEY: Final[str] = "_approval_session_is_authoritative" _PARENT_TOOL_APPROVAL_SOURCE_IDS_CONTEXT_KEY: Final[str] = "_parent_tool_approval_source_ids" @@ -4543,10 +4544,21 @@ def _reset_required_tool_choice(options: dict[str, Any]) -> None: def _prepare_messages_for_next_iteration(prepared_messages: list[Message], response: ChatResponse[Any]) -> None: + from ._types import Message + if response.conversation_id is None: prepared_messages.extend(response.messages) return - prepared_messages[:] = response.messages[-1:] + local_results = [ + content + for message in response.messages[:-1] + for content in message.contents + if content.type in {"function_result", "shell_tool_result", "computer_tool_result"} + and content.additional_properties.get(_LOCAL_TOOL_RESULT_KEY) is True + ] + prepared_messages[:] = ( + [Message(role="tool", contents=local_results)] if local_results else [] + ) + response.messages[-1:] @dataclass diff --git a/python/packages/core/tests/core/test_sessions.py b/python/packages/core/tests/core/test_sessions.py index 1cf5d5adcf..e16f654ff2 100644 --- a/python/packages/core/tests/core/test_sessions.py +++ b/python/packages/core/tests/core/test_sessions.py @@ -835,6 +835,13 @@ def from_dict(cls, value: dict[str, Any]) -> "_RegisteredSessionState": class TestStateTypeRegistry: + @pytest.fixture(autouse=True) + def isolate_state_registry(self, monkeypatch: pytest.MonkeyPatch) -> None: + from agent_framework import _sessions + + monkeypatch.setattr(_sessions, "_STATE_TYPE_REGISTRY", _sessions._STATE_TYPE_REGISTRY.copy()) + monkeypatch.setattr(_sessions, "_STATE_CLASS_REGISTRY", _sessions._STATE_CLASS_REGISTRY.copy()) + def test_same_registration_is_idempotent(self) -> None: register_state_type(_RegisteredSessionState) diff --git a/python/packages/openai/agent_framework_openai/_chat_client.py b/python/packages/openai/agent_framework_openai/_chat_client.py index ea9b3e010e..19ec0fb3bf 100644 --- a/python/packages/openai/agent_framework_openai/_chat_client.py +++ b/python/packages/openai/agent_framework_openai/_chat_client.py @@ -49,6 +49,7 @@ from agent_framework._settings import SecretString from agent_framework._telemetry import USER_AGENT_KEY, mark_feature_used from agent_framework._tools import ( + _LOCAL_TOOL_RESULT_KEY, # pyright: ignore[reportPrivateUsage] SHELL_TOOL_KIND_VALUE, FunctionInvocationConfiguration, FunctionInvocationLayer, @@ -1949,7 +1950,10 @@ def _prepare_message_for_openai( args = {"type": "message", "role": message.role} all_messages.append(function_call) case "shell_tool_call" | "shell_tool_result": - if request_uses_service_side_storage: + if request_uses_service_side_storage and not ( + content.type == "shell_tool_result" + and content.additional_properties.get(_LOCAL_TOOL_RESULT_KEY) is True + ): continue if "content" in args or "tool_calls" in args: all_messages.append(args) @@ -2596,6 +2600,22 @@ def _shell_item_to_contents(self, item: Any, local_shell_tool_name: str | None) raw_representation=item, ) ) + if ( + local_shell_tool_name + and is_local_environment + and shell_call_id + and shell_call_item_id + and isinstance(raw_shell_commands, list) + and not raw_shell_commands + ): + contents.append( + Content.from_shell_tool_result( + call_id=shell_call_id, + outputs=[], + max_output_length=shell_max_output, + additional_properties={_LOCAL_TOOL_RESULT_KEY: True}, + ) + ) elif item_type == "local_shell_call": raw_local_call_id = getattr(item, "call_id", None) local_call_id = raw_local_call_id if isinstance(raw_local_call_id, str) else "" @@ -2853,6 +2873,16 @@ def _prepare_shell_transcript_item_for_openai( expected_type: Literal["shell_call", "shell_call_output", "local_shell_call"], ) -> dict[str, Any]: """Restore a provider-issued shell transcript item for stateless replay.""" + if ( + expected_type == "shell_call_output" + and content.raw_representation is None + and content.call_id + and content.outputs == [] + ): + output: dict[str, Any] = {"type": "shell_call_output", "call_id": content.call_id, "output": []} + if content.max_output_length is not None: + output["max_output_length"] = content.max_output_length + return output payload = cls._serialize_provider_payload(content.raw_representation) if isinstance(payload, Mapping): typed_payload = cast("Mapping[str, Any]", payload) diff --git a/python/packages/openai/tests/openai/test_openai_chat_client.py b/python/packages/openai/tests/openai/test_openai_chat_client.py index f0da16d8a6..d64e33065c 100644 --- a/python/packages/openai/tests/openai/test_openai_chat_client.py +++ b/python/packages/openai/tests/openai/test_openai_chat_client.py @@ -2589,6 +2589,303 @@ def test_malformed_local_environment_shell_call_is_not_executable( assert all(content.type != "function_call" for content in contents) +@pytest.mark.parametrize("max_output_length", [None, 4096]) +@pytest.mark.parametrize("round_trip", [False, True]) +@pytest.mark.parametrize("environment", [{"type": "local"}, None]) +@pytest.mark.parametrize("service_storage", [False, True]) +def test_empty_local_shell_call_replays_with_empty_output( + max_output_length: int | None, + round_trip: bool, + environment: dict[str, str] | None, + service_storage: bool, +) -> None: + """Zero local commands must replay with a matching empty result, not an orphan call.""" + client = OpenAIChatClient(model="test-model", api_key="test-key") + item = ResponseFunctionShellToolCall.model_validate({ + "type": "shell_call", + "id": "sh_empty", + "call_id": "call_empty", + "status": "completed", + "action": {"commands": [], "max_output_length": max_output_length}, + "environment": environment, + }) + + contents = client._shell_item_to_contents(item, "run_shell") + + assert all(content.type != "function_call" for content in contents) + message = Message("assistant", contents) + if round_trip: + restored_result = Message.from_json(Message("assistant", [contents[1]]).to_json()).contents[0] + message = Message("assistant", [contents[0], restored_result]) + replay = client._prepare_messages_for_openai( + [message, Message("user", ["Continue"])], + request_uses_service_side_storage=service_storage, + ) + + shell_calls = [entry for entry in replay if entry["type"] == "shell_call"] + assert len(shell_calls) == (0 if service_storage else 1) + if not service_storage: + assert shell_calls[0]["call_id"] == "call_empty" + assert shell_calls[0]["action"]["commands"] == [] + shell_outputs = [entry for entry in replay if entry["type"] == "shell_call_output"] + expected_output: dict[str, Any] = {"type": "shell_call_output", "call_id": "call_empty", "output": []} + if max_output_length is not None: + expected_output["max_output_length"] = max_output_length + assert shell_outputs == [expected_output] + + +@pytest.mark.parametrize("streaming", [False, True]) +def test_empty_local_shell_call_parsing_emits_pair_without_execution(streaming: bool) -> None: + client = OpenAIChatClient(model="test-model", api_key="test-key") + executed_commands: list[str] = [] + + def local_exec(command: str) -> str: + executed_commands.append(command) + return command + + local_tool = OpenAIChatClient.get_shell_tool(func=local_exec) + item = ResponseFunctionShellToolCall.model_validate({ + "type": "shell_call", + "id": "sh_empty", + "call_id": "call_empty", + "status": "completed", + "action": {"commands": []}, + "environment": {"type": "local"}, + }) + if streaming: + event = ResponseOutputItemDoneEvent( + type="response.output_item.done", item=item, output_index=0, sequence_number=1 + ) + update = client._parse_chunk_from_openai(event, options={"tools": [local_tool]}, function_call_ids={}) + finalized = ChatResponse.from_updates([update]) + assert [content.type for content in finalized.messages[0].contents] == ["shell_tool_call", "shell_tool_result"] + contents = update.contents + else: + response = MagicMock( + output=[item], + output_parsed=None, + metadata={}, + usage=None, + id="resp_empty", + model="test-model", + created_at=1000000000, + status="completed", + incomplete=None, + ) + parsed = client._parse_response_from_openai(response, options={"tools": [local_tool]}) + contents = parsed.messages[0].contents + + assert [content.type for content in contents] == ["shell_tool_call", "shell_tool_result"] + assert [content.call_id for content in contents] == ["call_empty", "call_empty"] + assert contents[1].outputs == [] + assert executed_commands == [] + + +@pytest.mark.parametrize( + ("environment", "tool_name", "item_id", "call_id", "commands"), + [ + ({"type": "container_reference"}, "run_shell", "sh_empty", "call_empty", []), + ({"type": "local"}, None, "sh_empty", "call_empty", []), + (None, None, "sh_empty", "call_empty", []), + ({"type": "local"}, "run_shell", None, "call_empty", []), + ({"type": "local"}, "run_shell", "sh_empty", None, []), + ({"type": "local"}, "run_shell", "sh_empty", "call_empty", None), + ({"type": "local"}, "run_shell", "sh_empty", "call_empty", [1]), + ], +) +def test_empty_shell_result_is_not_synthesized_for_hosted_unconfigured_or_malformed_calls( + environment: dict[str, str] | None, + tool_name: str | None, + item_id: str | None, + call_id: str | None, + commands: Any, +) -> None: + client = OpenAIChatClient(model="test-model", api_key="test-key") + item = SimpleNamespace( + type="shell_call", + id=item_id, + call_id=call_id, + environment=environment, + action=SimpleNamespace(commands=commands), + status="completed", + ) + + contents = client._shell_item_to_contents(item, tool_name) + + assert [content.type for content in contents] == ["shell_tool_call"] + + +@pytest.mark.parametrize("streaming", [False, True]) +@pytest.mark.parametrize("continuation", ["stateless", "response", "conversation"]) +async def test_empty_local_shell_output_survives_mixed_tool_continuation(streaming: bool, continuation: str) -> None: + """The next request submits local results once without replaying stored shell items.""" + requests: list[dict[str, Any]] = [] + executed: list[str] = [] + + def local_exec(command: str) -> str: + executed.append("shell") + return command + + @tool(approval_mode="never_require") + def ordinary_tool() -> str: + """Return the ordinary tool result.""" + executed.append("ordinary") + return "ordinary result" + + shell_call = { + "type": "shell_call", + "id": "sh_empty", + "call_id": "call_empty", + "status": "completed", + "environment": {"type": "local"}, + "action": {"commands": []}, + } + hosted_call = { + "type": "shell_call", + "id": "sh_hosted", + "call_id": "call_hosted", + "status": "completed", + "environment": {"type": "container_reference", "container_id": "cntr_fixture"}, + "action": {"commands": ["pwd"]}, + } + hosted_output = { + "type": "shell_call_output", + "id": "sh_hosted_output", + "call_id": "call_hosted", + "status": "completed", + "output": [{"stdout": "/workspace", "stderr": "", "outcome": {"type": "exit", "exit_code": 0}}], + } + + def respond(request: Any) -> Any: + requests.append(json.loads(request.content)) + first = len(requests) == 1 + output: list[dict[str, Any]] = ( + [ + hosted_call, + hosted_output, + shell_call, + { + "type": "function_call", + "id": "fc_ordinary", + "call_id": "call_ordinary", + "name": "ordinary_tool", + "arguments": "{}", + "status": "completed", + }, + ] + if first + else [ + { + "type": "message", + "id": "msg_done", + "role": "assistant", + "status": "completed", + "content": [{"type": "output_text", "text": "done", "annotations": [], "logprobs": []}], + } + ] + ) + response: dict[str, Any] = { + "id": "resp_first" if first else "resp_final", + "object": "response", + "created_at": 1000000000, + "status": "completed", + "model": "test-model", + "output": output, + "usage": None, + "metadata": {}, + "parallel_tool_calls": True, + "tools": [], + "tool_choice": "auto", + "error": None, + "incomplete_details": None, + "conversation": {"id": "conv_fixture"} if continuation == "conversation" else None, + } + if not streaming: + return _OPENAI_HTTPX.Response(200, json=response) + events: list[dict[str, Any]] = [ + { + "type": "response.created", + "response": {**response, "status": "in_progress", "output": []}, + } + ] + for index, item in enumerate(output): + events.append({"type": "response.output_item.added", "output_index": index, "item": item}) + if item["type"] == "function_call": + events.append({ + "type": "response.function_call_arguments.delta", + "output_index": index, + "item_id": item["id"], + "delta": "{}", + }) + elif item["type"] == "message": + events.append({ + "type": "response.output_text.delta", + "output_index": index, + "content_index": 0, + "item_id": item["id"], + "delta": "done", + "logprobs": [], + }) + events.append({"type": "response.output_item.done", "output_index": index, "item": item}) + events.append({"type": "response.completed", "response": response}) + body = "".join( + f"data: {json.dumps({**event, 'sequence_number': number})}\n\n" for number, event in enumerate(events) + ) + return _OPENAI_HTTPX.Response(200, content=body, headers={"content-type": "text/event-stream"}) + + async with AsyncOpenAI( + api_key="test-key", http_client=_OPENAI_HTTPX.AsyncClient(transport=_OPENAI_HTTPX.MockTransport(respond)) + ) as api: + client = OpenAIChatClient(model="test-model", async_client=api) + options: OpenAIChatOptions[None] = { + "store": continuation != "stateless", + "tools": [OpenAIChatClient.get_shell_tool(func=local_exec), ordinary_tool], + } + messages = [Message("user", ["Use the tools."])] + if streaming: + stream = client.get_response(messages, options=options, stream=True) + updates = [update async for update in stream] + result = await stream.get_final_response() + assert ( + sum( + content.type == "shell_tool_result" and content.call_id == "call_empty" + for update in updates + for content in update.contents + ) + == 1 + ) + else: + result = await client.get_response(messages, options=options) + + assert result.text == "done" + assert ( + sum( + content.type == "shell_tool_result" and content.call_id == "call_empty" + for message in result.messages + for content in message.contents + ) + == 1 + ) + assert executed == ["ordinary"] + assert len(requests) == 2 + next_input = requests[1]["input"] + assert [item for item in next_input if item["type"] == "shell_call_output" and item["call_id"] == "call_empty"] == [ + {"type": "shell_call_output", "call_id": "call_empty", "output": []} + ] + assert ( + sum(item["type"] == "function_call_output" and item["call_id"] == "call_ordinary" for item in next_input) == 1 + ) + if continuation != "stateless": + assert all(item["type"] not in {"shell_call", "function_call"} for item in next_input) + assert all(item.get("call_id") != "call_hosted" for item in next_input) + key, value = ( + ("conversation", "conv_fixture") + if continuation == "conversation" + else ("previous_response_id", "resp_first") + ) + assert requests[1][key] == value + + @pytest.mark.asyncio async def test_local_shell_tool_requires_approval_before_function_loop_execution() -> None: """An explicit local shell call executes only after approval."""