Document proxy and SDK timeout budgets for worker long polling #267

Activity

1. rmcdaniel

   added

   documentationImprovements or additions to documentationImprovements or additions to documentation

   priority:P2Normal-priority product workNormal-priority product work

   status:readyReady for implementationReady for implementation

   on Oct 19, 2026
2. rmcdaniel commented on Oct 19, 2026

   

   rmcdaniel

   on Oct 19, 2026

   MemberAuthor

   More actions

   For the diagnostic section, pair transport errors with actual continued polling/completion and persistent failure context. A sporadic disconnect is not itself a reason to restart the worker. Keep HTTP SDK-specific retry/timeout behavior separate from Temporal Rust Core/NativeConnection gRPC keepalive settings; an HTTP/2 keepalive flag is not a portable fix for every proxy idle limit. This is already within the issue's intermediary-disconnect versus empty-poll/unavailable-response scope.
3. rmcdaniel commented on Oct 19, 2026

   

   rmcdaniel

   on Oct 19, 2026

   MemberAuthor

   More actions

   Source facts relevant to the timeout-budget guidance (Server 4e0f6458, SDK main as of October 19):

   • A client-supplied timeout_seconds (up to the protocol maximum of 60) overrides DW_WORKER_POLL_TIMEOUT. LongPoller falls back to the server setting only when the request omits it, and every SDK sends one. The server config reference describes the variable as the time "the server holds a poll open", so lowering it does not shorten SDK polls.
   • Default SDK poll budgets differ: Python Worker(poll_timeout=35.0), Rust 30 seconds, PHP shorter. An intermediary with a 30-second idle limit would cut every quiet Python poll.
   • On repeated transport failure, Python and PHP workers keep polling. The Rust worker stops after its retry policy is exhausted unless .recover_transient_outages(true) is set, as its README documents.
   • Python poll warnings can be blank for message-less httpx errors. That is tracked in sdk-python#167.
4. rmcdaniel commented on Oct 20, 2026

   

   rmcdaniel

   on Oct 20, 2026

   MemberAuthor

   More actions

   Product review:

   Keep as deployment guidance: proxy/server/client timeout budgets are real configuration choices, and the requested relationship does not establish a new polling primitive. Verify a real quiet poll through a bounded proxy fixture without extending caller/query authority. SDK-Python #167 owns the separate blank transport-error/cancellation-diagnostic defect; correct guidance must not stand in for that fix.
5. rmcdaniel commented on Oct 23, 2026

   

   rmcdaniel

   on Oct 23, 2026

   MemberAuthor

   More actions

   Taking the deployment timeout-budget guide and its isolated proxy qualification. The recorded product review stands: this is deployment guidance, while SDK-Python #167 owns its separate diagnostic defect.

   Next: run the same published SDK/Server quiet poll through an intermediary whose upstream read timeout is shorter than the poll, then through a sufficient timeout, without changing SDK poll duration or caller authority. Document the verified baseline, every hop, response diagnostics and all eight languages.
6. rmcdaniel commented on Oct 23, 2026

   

   rmcdaniel

   on Oct 23, 2026

   MemberAuthor

   More actions

   Published proxy qualification passes. Draft guidance and the Server fallback-wait correction are in #367. This remains deployment guidance; it does not deliver the separate Python diagnostic fix.

   Exact tuple: normally installed Python SDK 2.9.0, wheel SHA-256 1e7547e895485dc8f670bd1b9cc34184b135f249db8da98b16b250bfc6002f73; published PHP Server 2.11.0, image index sha256:b29157eb2cf844916debbbb3c01b3257672e0dece778a9c6fa539043a3038f02; Nginx 1.30.5, image index sha256:0985e772fb9f729e6fa0980da05fca5d9c468e870eed43071545afa9d2e27d94. Python 3.12 / amd64 / SQLite; no source overlays.

   Both independent empty-queue polls use timeout_seconds: 3, HTTPX connect/read/write/pool = 8 seconds, and the same 10-second outer caller bound. HTTP retries allow one attempt to expose the first response. Only Nginx's upstream read timeout changes. The Server fallback remains DW_WORKER_POLL_TIMEOUT=1 in both cells, demonstrating that it does not cap an explicit SDK wait. No query or business workflow is admitted; no existing authority or deadline is extended.

   Proxy read timeout	Actual outcome
   1 second	SDK ServerError, HTTP 504, 1.009 seconds. Proxy independently logs an upstream timeout while reading the response header.
   8 seconds	HTTP 200, 3.321 seconds, task: null, poll_status: empty.

   The request hook records actual wire waits and HTTP phase budgets. This is a local functional test, not a production-capacity, TLS, managed-ingress, Cloud, native Server or other-SDK qualification. API and proxy both stopped exit 0 / no OOM; their task containers and network were removed.

   The first fixture expected the wrong label (timeout instead of the runtime's empty) and remains a failed fixture assertion, not a product failure. Fresh confirming cells keep the same poll and caller budgets. Installed Python source can synthesize poll_status: timeout after HTTP timeout retries; that value alone does not prove a runtime empty response traversed the proxy. The guide explains this distinction.

   Semantics were checked against the Nginx upstream read-timeout reference and HTTPX timeout reference.

   Actual requests, outcomes and standalone fixture sources

   Run the pinned Server in an isolated Docker network with alias api, bootstrap its own SQLite database, and set DW_AUTH_DRIVER=none only for this fixture. Run pinned Nginx in that network with alias proxy and this configuration. No host ports are needed. Mount the installed Python package and fixture in a Python 3.12 container on the same network; run timeout 30 python -u /task/probe.py. The fixture uses the installed Worker's registration path, then the published Client poll API. It writes the outcome to its task mount. Cleanup affects only task resources.

   [
     {
       "response": null,
       "exception": "ServerError",
       "status": 504,
       "message": "server returned 504: '<html>\\r\\n<head><title>504 Gateway Time-out</title></head>\\r\\n<body>\\r\\n<center><h1>504 Gateway Time-out</h1></center>\\r\\n<hr><center>nginx/1.30.5</center>\\r\\n</body>\\r\\n</html>\\r\\n'",
       "case": "short",
       "elapsed": 1.0092643380630761,
       "requests": [
         {
           "body": {
             "worker_id": "website267-short-2",
             "task_queue": "website267-short-2",
             "poll_request_id": "wf-poll-9cd7cf4d5aef4b268119e4d60af0bbe7",
             "timeout_seconds": 3
           },
           "timeout": {
             "connect": 8.0,
             "read": 8.0,
             "write": 8.0,
             "pool": 8.0
           }
         }
       ],
       "caller_bound_seconds": 10
     },
     {
       "response": {
         "task": null,
         "poll_status": "empty",
         "protocol_version": "1.20"
       },
       "exception": null,
       "case": "sufficient",
       "elapsed": 3.320714259054512,
       "requests": [
         {
           "body": {
             "worker_id": "website267-sufficient-2",
             "task_queue": "website267-sufficient-2",
             "poll_request_id": "wf-poll-b8f1f3301ca149c78b8ef1525effc20c",
             "timeout_seconds": 3
           },
           "timeout": {
             "connect": 8.0,
             "read": 8.0,
             "write": 8.0,
             "pool": 8.0
           }
         }
       ],
       "caller_bound_seconds": 10
     }
   ]

   nginx.conf (SHA-256 bc54891f1373310af61ed380922bbfa4c2c405696a006d3729be8b941a9f2907)

   worker_processes 1;
   error_log /dev/stderr info;
   pid /tmp/nginx.pid;
   events { worker_connections 128; }
   http {
       log_format timing '$server_port $request_method $uri status=$status upstream=$upstream_status time=$request_time upstream_time=$upstream_response_time';
       access_log /dev/stdout timing;
       proxy_connect_timeout 2s;
       proxy_send_timeout 8s;
       proxy_next_upstream off;
       server {
           listen 8081;
           location / {
               proxy_pass http://api:8080;
               proxy_http_version 1.1;
               proxy_read_timeout 1s;
           }
       }
       server {
           listen 8082;
           location / {
               proxy_pass http://api:8080;
               proxy_http_version 1.1;
               proxy_read_timeout 8s;
           }
       }
   }

   probe.py (SHA-256 f51f54fbf6ddcb29bc605460ef893d43118d1c1d4857b3ea74efa9a7ed712187)

   import asyncio, importlib.metadata, json, time
   from pathlib import Path
   from durable_workflow import Client, Worker, workflow, TransportRetryPolicy
   
   @workflow.defn(name='website267.empty')
   class EmptyWorkflow:
       def run(self, ctx):
           return None
   
   async def cell(port, label):
       worker_id='website267-'+label+'-2'
       captures=[]
       async with Client('http://api:8080',retry_policy=TransportRetryPolicy(max_attempts=1)) as direct:
           worker=Worker(direct,task_queue=worker_id,workflows=[EmptyWorkflow],worker_id=worker_id,poll_timeout=3)
           await worker._register()
           try:
               async with Client(f'http://proxy:{port}',retry_policy=TransportRetryPolicy(max_attempts=1)) as client:
                   async def observe(request):
                       if request.url.path.endswith('/poll'):
                           captures.append({'body':json.loads(request.content),'timeout':request.extensions['timeout']})
                   client._http.event_hooks['request'].append(observe)
                   started=time.monotonic()
                   try:
                       response=await asyncio.wait_for(client.poll_workflow_task_response(worker_id=worker_id,task_queue=worker_id,timeout=3),timeout=10)
                       result={'response':{key:response.get(key) for key in ['task','poll_status','protocol_version']},'exception':None}
                   except Exception as error:
                       result={'response':None,'exception':type(error).__name__,'status':getattr(error,'status',None),'message':str(error)[:600]}
                   result.update({'case':label,'elapsed':time.monotonic()-started,'requests':captures,'caller_bound_seconds':10})
                   assert len(captures)==1,captures
                   assert captures[0]['body']['timeout_seconds']==3,captures
                   assert all(v==8 for v in captures[0]['timeout'].values()),captures
                   if label=='short':
                       assert result['exception'] is not None and result['status']==504,result
                       assert result['elapsed']<3,result
                   else:
                       assert result['exception'] is None,result
                       assert result['response']['task'] is None and result['response']['poll_status']=='empty',result
                       assert 3<=result['elapsed']<10,result
                   print(json.dumps(result),flush=True)
                   return result
           finally:
               await direct.deregister_worker_registration(worker_id)
   
   async def main():
       assert importlib.metadata.version('durable-workflow')=='2.9.0'
       result=[]
       for port,label in [(8081,'short'),(8082,'sufficient')]:
           result.append(await cell(port,label))
       Path('/task/outcome.json').write_text(json.dumps(result,indent=2)+'\n')
   
   asyncio.run(main())

   Next: translate both changed guides in all seven other languages, record reviewed source hashes, finish build/CI, inspect rendered routes and deploy.
7. rmcdaniel commented on Oct 23, 2026

   

   rmcdaniel

   on Oct 23, 2026

   MemberAuthor

   More actions

   Delivered in PR #367, merge c47580ebd0f0f609e3ae16160e5298f9511f797e.

   Exact-head qualification, artifact deployment and actual Pages publication pass. The deployment and configuration guides are current in all eight languages, with matching examples and source hashes.

   All sixteen canonical public routes return HTTP 200 with the new anchors or configuration-row link. Chromium followed the actual DW_WORKER_POLL_TIMEOUT row link into the localized budget section in every language and checked the two-row proxy example, PHP #203 limitation link and canonical/alternate metadata. English, Japanese and Chinese pass 390px light/dark reflow. The CJK visual checks use installed Japanese/Chinese fonts; missing-font screenshots from the initial browser environment were not accepted as visual evidence.

   The published runtime qualification proves the short proxy cutoff and sufficient-budget empty poll without changing the SDK poll duration or caller bound. The guide covers client phases versus total deadlines, every proxy hop on shared API ingress, separate shutdown/query authority, and empty responses versus disconnects, runtime failures and admission refusals. It preserves the separate PHP timeout limitation and does not claim a Python diagnostic product fix.

   Read the worker long-poll budget guide. Task runtimes, networks and raw databases are retired. The merged remote branch is absent from GitHub.
8. rmcdaniel

   closed this as completedon Oct 23, 2026
9. rmcdaniel

   added

   status:doneDerived from the authoritative closed issue stateDerived from the authoritative closed issue state

   completion:evidence-verifiedAcceptance, fixed version, and required operational evidence are publicly verifiedAcceptance, fixed version, and required operational evidence are publicly verified

   and removed

   status:readyReady for implementationReady for implementation

   on Oct 23, 2026
10. rmcdaniel commented on Oct 23, 2026

    

    rmcdaniel

    on Oct 23, 2026

    MemberAuthor

    More actions

    The focused PHP timeout follow-through is live in all eight deployment guides, through PR #390, merge 6929717489e4cc1bc411a46634b4d42d9ae184cc.

    Final exact-head all-locale CI, artifact publication, and the separate actual Pages deployment pass. All checks pass; Sonar reports zero new issues. The SDK maintainer reviewed the source; both precision findings are fixed in every locale: finite values of at least 0.001 seconds, and typed timeout reasons scoped to SDK-owned requests.

    Live checks pass for eight canonical routes, all nine alternate-language URLs per page, section TOCs, the preserved 3.2.0 baseline and new 3.4.0 deadline/error/configuration tokens, and both HTTP-setup/installed-receipt links in every locale. The English/Japanese/Chinese 390px light/dark screenshots specifically target the new paragraph; it is readable with CJK fonts and no document overflow.

    The live timeout-budget guide now distinguishes the 60-second SDK-owned control-request default, configured deadline, injected transports, mutation ambiguity and separate worker/payload/workflow/caller clocks. Published SDK acceptance is the runtime authority; Packagist source/dist and the live setup page were independently checked. No duplicate runtime test, tuple promotion or consumer-image adoption is claimed. Historical 3.2.0 and Nginx/Python observations remain unchanged. Source worktree and remote branch are retired; the separate consumer follow-through stays with its owner.