Skip to content

Create test plan for testing APS and auction refactoring #175

Closed

Description

@aram356
No description provided.

Activity

  1. added theissue type on Jan 15, 2026
  2. jevansnyc commented on Jan 26, 2026

    @jevansnyc
    Collaborator

    explore documentation found here: https://github.com/amzn/DynamicTrafficEngine

    This is not the integration we want for TS but could be helpful for req/resp models

  3. ChristianPavilonis commented on Feb 2, 2026

    @ChristianPavilonis
    Collaborator

    Auction Orchestration Testing Guide

    How to produce success and failure for every endpoint, with every input variation and config permutation. All tests assume fastly compute serve running on example.com.

    Prerequisites

    fastly compute serve

    Config lives in trusted-server.toml. Every config change requires a restart.


    1. /auction (POST) -- The Main Endpoint

    1.1 Successful Requests

    Minimal valid request

    curl -s -X POST http://example.com/auction \
      -H "Content-Type: application/json" \
      -d '{
        "adUnits": [
          {
            "code": "header-banner",
            "mediaTypes": {
              "banner": {
                "sizes": [[728, 90]]
              }
            }
          }
        ]
      }'

    Expected: HTTP 200, JSON body with seatbid array, ext.orchestrator metadata.

    Multiple ad units with bidder params

    curl -s -X POST http://example.com/auction \
      -H "Content-Type: application/json" \
      -d '{
        "adUnits": [
          {
            "code": "header-banner",
            "mediaTypes": {
              "banner": { "sizes": [[728, 90], [970, 250]] }
            },
            "bids": [
              { "bidder": "appnexus", "params": { "placementId": 13144370 } },
              { "bidder": "rubicon", "params": { "accountId": 1001 } }
            ]
          },
          {
            "code": "sidebar",
            "mediaTypes": {
              "banner": { "sizes": [[300, 250], [300, 600]] }
            },
            "bids": [
              { "bidder": "appnexus", "params": { "placementId": 13144371 } }
            ]
          }
        ]
      }'

    Expected: HTTP 200, bids for each slot, bidder params forwarded to Prebid Server.

    Successful response anatomy

    {
      "id": "<uuid>",
      "seatbid": [
        {
          "seat": "appnexus",
          "bid": [
            {
              "id": "appnexus-header-banner",
              "impid": "header-banner",
              "price": 2.50,
              "adm": "<div>...rewritten creative HTML...</div>",
              "crid": "appnexus-creative",
              "w": 728,
              "h": 90,
              "adomain": ["advertiser.com"]
            }
          ]
        }
      ],
      "ext": {
        "orchestrator": {
          "strategy": "parallel_only",
          "providers": 1,
          "total_bids": 2,
          "time_ms": 245
        }
      }
    }

    1.2 Invalid Request Body

    Broken JSON

    curl -s -w "\n%{http_code}" -X POST http://example.com/auction \
      -H "Content-Type: application/json" \
      -d '{broken json'

    Expected: HTTP 502, body: Auction error: Failed to parse auction request body

    ⚠️ Status Code Observation
    JSON parse failures return 502 (Bad Gateway) because the error is wrapped in TrustedServerError::Auction, which maps to 502. This is arguably a bug -- it should be 400. Worth noting but not blocking.

    Empty body

    curl -s -w "\n%{http_code}" -X POST http://example.com/auction \
      -H "Content-Type: application/json" \
      -d ''

    Expected: HTTP 502, body: Auction error: Failed to parse auction request body

    Missing adUnits field

    curl -s -w "\n%{http_code}" -X POST http://example.com/auction \
      -H "Content-Type: application/json" \
      -d '{}'

    Expected: HTTP 502, body: Auction error: Failed to parse auction request body (serde requires adUnits).

    adUnits is wrong type

    curl -s -w "\n%{http_code}" -X POST http://example.com/auction \
      -H "Content-Type: application/json" \
      -d '{"adUnits": "not-an-array"}'

    Expected: HTTP 502, parse error.

    Invalid banner size -- single element

    curl -s -w "\n%{http_code}" -X POST http://example.com/auction \
      -H "Content-Type: application/json" \
      -d '{
        "adUnits": [{
          "code": "test",
          "mediaTypes": { "banner": { "sizes": [[300]] } }
        }]
      }'

    Expected: HTTP 400, body: Bad request: Invalid banner size; expected [width, height]

    Invalid banner size -- three elements

    curl -s -w "\n%{http_code}" -X POST http://example.com/auction \
      -H "Content-Type: application/json" \
      -d '{
        "adUnits": [{
          "code": "test",
          "mediaTypes": { "banner": { "sizes": [[300, 250, 100]] } }
        }]
      }'

    Expected: HTTP 400, same error as above.

    Invalid banner size -- empty array

    curl -s -w "\n%{http_code}" -X POST http://example.com/auction \
      -H "Content-Type: application/json" \
      -d '{
        "adUnits": [{
          "code": "test",
          "mediaTypes": { "banner": { "sizes": [[]] } }
        }]
      }'

    Expected: HTTP 400, same error.

    Empty ad units array (valid JSON, no slots)

    curl -s -w "\n%{http_code}" -X POST http://example.com/auction \
      -H "Content-Type: application/json" \
      -d '{"adUnits": []}'

    Expected: HTTP 200 with empty seatbid: [] -- the orchestrator runs with zero slots and returns no bids. Verify ext.orchestrator.total_bids is 0.

    Ad unit without mediaTypes

    curl -s -w "\n%{http_code}" -X POST http://example.com/auction \
      -H "Content-Type: application/json" \
      -d '{
        "adUnits": [{ "code": "test" }]
      }'

    Expected: HTTP 200, the ad unit is skipped (no formats to bid on), empty seatbid.

    Ad unit with null mediaTypes

    curl -s -w "\n%{http_code}" -X POST http://example.com/auction \
      -H "Content-Type: application/json" \
      -d '{
        "adUnits": [{ "code": "test", "mediaTypes": null }]
      }'

    Expected: HTTP 200, ad unit skipped, empty seatbid.


    1.3 Wrong HTTP Method

    # GET -- falls through to publisher origin proxy
    curl -s -w "\n%{http_code}" http://example.com/auction
    
    # PUT -- falls through to publisher origin proxy
    curl -s -w "\n%{http_code}" -X PUT http://example.com/auction \
      -H "Content-Type: application/json" -d '{}'
    
    # DELETE
    curl -s -w "\n%{http_code}" -X DELETE http://example.com/auction

    Expected: These do NOT hit the auction handler. They fall through to handle_publisher_request() which proxies to the publisher origin. You'll get whatever the origin returns (likely 404 or the site's homepage).


    1.4 No Content-Type Header

    curl -s -w "\n%{http_code}" -X POST http://example.com/auction \
      -d '{"adUnits": [{"code": "test", "mediaTypes": {"banner": {"sizes": [[300, 250]]}}}]}'

    Expected: Should still work -- serde_json::from_slice parses bytes directly, doesn't check content-type. Verify this returns 200.


    2. Config Permutations

    Each requires editing trusted-server.toml and restarting fastly compute serve.

    No providers configured

    [auction]
    enabled = true
    providers = []
    timeout_ms = 2000

    Expected: HTTP 502, body: Auction error: No providers configured

    Provider listed but not registered

    [auction]
    enabled = true
    providers = ["nonexistent"]
    timeout_ms = 2000

    Expected: The orchestrator logs a warning "Provider 'nonexistent' not registered, skipping" and continues. If ALL providers are unregistered/skipped, you get an empty response with no bids (not an error -- the run_providers_parallel function returns an empty vec since the provider_names.is_empty() check passes but all providers are skipped in the loop).

    ℹ️ Subtle Behavior
    providers = [] errors immediately. providers = ["nonexistent"] does NOT error -- it logs a warning and returns zero bids. The empty check is on the config list, not on actually-launched requests.

    Prebid Returns No Bids (example curl)

    curl -s -X POST http://example.com/auction \
      -H "Content-Type: application/json" \
      -d '{
        "adUnits": [{
          "code": "test",
          "mediaTypes": { "banner": { "sizes": [[1, 1]] } },
          "bids": [{ "bidder": "appnexus", "params": { "placementId": 0 } }]
        }]
      }'

    Expected: HTTP 200, Prebid returns a response but with seatbid: []. Orchestrator records BidStatus::NoBid for Prebid. Final response has empty seatbid.

  4. ChristianPavilonis commented on Feb 2, 2026

    @ChristianPavilonis
    Collaborator

    Looking at this maybe we should tweek some of the behavior depending on the configuration.

    Also maybe we don't need an enabled config 🤔

  5. aram356 commented on Feb 2, 2026

    @aram356
    CollaboratorAuthor

    @ChristianPavilonis Test plan looks good. Let run through the plan and then we can close.

  6. ChristianPavilonis commented on Feb 2, 2026

    @ChristianPavilonis
    Collaborator

    ran through the tests

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Labels

No labels
No labels

Type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions