Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 6 additions & 5 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -103,15 +103,15 @@ $HOME/fivetran-proxy-agent/
├── proxy-agent-manager.sh --> Management script
├── config/
│ └── config.json --> Agent configuration (permissions: 600)
├── logs/ --> Agent and manager logs
├── logs/ --> Agent, manager and doctor logs
└── version --> Pinned agent version (for example 1.2.9)

# Windows
%USERPROFILE%\fivetran-proxy-agent\
├── proxy-agent-manager.ps1 --> Management script
├── config\
│ └── config.json --> Agent configuration (owner read/write; SYSTEM and Administrators read)
├── logs\ --> Agent and manager logs
├── logs\ --> Agent, manager and doctor logs
└── version --> Pinned agent version (for example 1.2.9)
```

Expand All @@ -122,15 +122,15 @@ $HOME/fivetran-proxy-agent/
Use `proxy-agent-manager.sh` to control the agent:

```bash
./proxy-agent-manager.sh {start|stop|restart|upgrade|status|logs}
./proxy-agent-manager.sh {start|stop|restart|upgrade|status|logs|doctor [options]}
```

### Windows

Use `proxy-agent-manager.ps1` to control the agent:

```powershell
& "$env:USERPROFILE\fivetran-proxy-agent\proxy-agent-manager.ps1" {start|stop|restart|upgrade|status|logs}
& "$env:USERPROFILE\fivetran-proxy-agent\proxy-agent-manager.ps1" {start|stop|restart|upgrade|status|logs|doctor [options]}
```

| Command | Description |
Expand All @@ -141,6 +141,7 @@ Use `proxy-agent-manager.ps1` to control the agent:
| `upgrade` | Pull and start the latest version, with auto-rollback on failure |
| `status` | Show container name, image, and health status |
| `logs` | Stream live container logs |
| `doctor` | Check the installation and connectivity to Fivetran in a separate container. The doctor's log and archive are saved to `logs/`, or its output if the doctor saves no log, as in 1.2.10. Options are passed to the agent's `--doctor`, e.g. `doctor --region us-east-1` |

## Troubleshooting

Expand All @@ -163,7 +164,7 @@ CI (`.github/workflows/ci.yml`) runs on every pull request:

- `shellcheck` on the shell scripts and PSScriptAnalyzer (settings in `PSScriptAnalyzerSettings.psd1`) on the PowerShell scripts.
- Unit tests for the management scripts and the Windows installer: `tests/bash/manager.bats` (bats), and `tests/pwsh/Manager.Tests.ps1` and `tests/pwsh/Install.Tests.ps1` (Pester 5, under both Windows PowerShell 5.1 and PowerShell 7). They run the scripts with stubbed `docker`, registry and host calls, so they need no Docker daemon.
- Integration tests that run the installer and management script against a real Docker daemon: `tests/integration/linux.sh` on Ubuntu (amd64 and arm64), and `tests/integration/windows.ps1` on Windows Server 2022 and 2025 with Windows containers (on 2025 also under PowerShell 7). They use a stub agent image (`tests/stub/`) that only writes the heartbeat file, and a local stand-in for the registry, the script download and the configure endpoint (`tests/integration/registry_server.py`), so they need no Fivetran credentials. They cover installing from a config file, from a `TOKEN`, and from a legacy WebSocket config, then rollback, upgrade, restart and stop.
- Integration tests that run the installer and management script against a real Docker daemon: `tests/integration/linux.sh` on Ubuntu (amd64 and arm64), and `tests/integration/windows.ps1` on Windows Server 2022 and 2025 with Windows containers (on 2025 also under PowerShell 7). They use a stub agent image (`tests/stub/`) that only writes the heartbeat file, or stands in for the doctor, and a local stand-in for the registry, the script download and the configure endpoint (`tests/integration/registry_server.py`), so they need no Fivetran credentials. They cover installing from a config file, from a `TOKEN`, and from a legacy WebSocket config, then rollback, upgrade, doctor, restart and stop.

To run them locally you need Docker for the integration tests, `python3` (`python` on Windows) for the registry stand-in, [bats-core](https://github.com/bats-core/bats-core) for the bash tests and [Pester 5](https://pester.dev) for the PowerShell tests. The stub images pull `alpine` and `servercore` base images, so the integration tests need network access for that.

Expand Down
136 changes: 108 additions & 28 deletions proxy-agent-manager.ps1
Original file line number Diff line number Diff line change
@@ -1,7 +1,10 @@
#Requires -Version 5.1
param(
[Parameter(Position = 0)]
[string]$Command
[string]$Command,
# Passed to --doctor by the doctor command, e.g. --region us-east-1
[Parameter(ValueFromRemainingArguments)]
[string[]]$DoctorArgs
)

$ErrorActionPreference = 'Stop'
Expand Down Expand Up @@ -29,9 +32,9 @@ if (-not $WINDOWS_VERSION) { $WINDOWS_VERSION = $null }

# ── Startup validation ───────────────────────────────────────────────────────

$validCommands = @('start', 'stop', 'restart', 'upgrade', 'status', 'logs')
$validCommands = @('start', 'stop', 'restart', 'upgrade', 'status', 'logs', 'doctor')
if (-not $Command -or $Command -notin $validCommands) {
Write-Host "Usage: $($MyInvocation.MyCommand.Name) {start|stop|restart|upgrade|status|logs}"
Write-Host "Usage: $($MyInvocation.MyCommand.Name) {start|stop|restart|upgrade|status|logs|doctor [options]}"
exit 1
}

Expand Down Expand Up @@ -299,6 +302,46 @@ function Assert-MemoryAllocation {
}
}

# Returns the config path inside the container.
function Get-ContainerConfigPath {
param([Parameter(Mandatory)][ValidateSet('linux', 'windows')][string]$ContainerOsType)
if ($ContainerOsType -eq 'windows') { return 'C:/config/config.json' }
return '/config/config.json'
}

# Returns the log folder inside the container.
function Get-ContainerLogDir {
param([Parameter(Mandatory)][ValidateSet('linux', 'windows')][string]$ContainerOsType)
if ($ContainerOsType -eq 'windows') { return 'C:/app/logs' }
return $CONTAINER_LOG_DIR
}

# Returns the docker run arguments shared by the agent and doctor containers,
# so the doctor checks the agent's config, memory limit and log folder.
function Get-AgentRunArgs {
param([Parameter(Mandatory)][ValidateSet('linux', 'windows')][string]$ContainerOsType)

$null = New-Item -ItemType Directory -Force -Path (Join-Path $BASE_DIR 'logs')

# Docker Desktop for Windows requires forward-slash paths in volume mounts.
$logsMount = (Join-Path $BASE_DIR 'logs') -replace '\\', '/'
if ($ContainerOsType -eq 'windows') {
# Windows containers can only mount folders.
$configMount = "$((Join-Path $BASE_DIR 'config') -replace '\\', '/'):C:/config:ro"
} else {
$configMount = "$((Join-Path $BASE_DIR 'config\config.json') -replace '\\', '/'):/config/config.json:ro"
}
$containerLogDir = Get-ContainerLogDir -ContainerOsType $ContainerOsType

return @(
"--memory=${MEMORY_ALLOCATION_MB}m",
'--env', 'IS_DOCKER=true',
'--env', "LOG_FOLDER_PATH=$containerLogDir",
'-v', $configMount,
'-v', "${logsMount}:$containerLogDir"
)
}

function Stop-ProxyAgent {
try { docker inspect $CONTAINER_NAME 2>&1 | Out-Null } catch {}
if ($LASTEXITCODE -eq 0) {
Expand All @@ -317,31 +360,20 @@ function Start-ProxyAgent {

Stop-ProxyAgent

$null = New-Item -ItemType Directory -Force -Path (Join-Path $BASE_DIR 'logs')

# Docker Desktop for Windows requires forward-slash paths in volume mounts.
$configFileMount = (Join-Path $BASE_DIR 'config\config.json') -replace '\\', '/'
$configDirectoryMount = (Join-Path $BASE_DIR 'config') -replace '\\', '/'
$logsMount = (Join-Path $BASE_DIR 'logs') -replace '\\', '/'

# Build argument list to avoid PowerShell variable expansion inside the health check.
$dockerArgs = @(
'run', '-d',
'--name', $CONTAINER_NAME,
'--restart', 'unless-stopped',
"--memory=${MEMORY_ALLOCATION_MB}m",
'--label', 'fivetran=proxy-agent',
'--label', "proxy_agent_id=$AGENT_ID",
'--env', 'IS_DOCKER=true',
'--env', 'HEARTBEAT_EXPIRY_SECONDS=30',
'--health-interval', '10s',
'--health-retries', '3',
'--health-start-period', '30s'
)

if ($ContainerOsType -eq 'windows') {
$containerConfigPath = 'C:/config/config.json'
$containerLogDir = 'C:/app/logs'
$containerHeartbeatPath = 'C:/app/logs/proxy-agent-heartbeat.txt'
$windowsHealthCheckScript = @(
'$path = $env:HEARTBEAT_PATH'
Expand All @@ -352,29 +384,19 @@ function Start-ProxyAgent {
$windowsHealthCheckEncoded = [Convert]::ToBase64String([System.Text.Encoding]::Unicode.GetBytes($windowsHealthCheckScript))
$windowsHealthCheck = "powershell.exe -NoProfile -EncodedCommand $windowsHealthCheckEncoded"
$dockerArgs += @(
'--env', "LOG_FOLDER_PATH=$containerLogDir",
'--env', "HEARTBEAT_PATH=$containerHeartbeatPath",
'--health-cmd', $windowsHealthCheck,
'--health-timeout', '15s',
'-v', "${configDirectoryMount}:C:/config:ro",
'-v', "${logsMount}:$containerLogDir",
"${IMAGE}:${ImageTag}",
'-i', $containerConfigPath
'--health-timeout', '15s'
)
} else {
$containerConfigPath = '/config/config.json'
$containerLogDir = $CONTAINER_LOG_DIR
$dockerArgs += @(
'--env', "LOG_FOLDER_PATH=$containerLogDir",
'--env', 'HEARTBEAT_PATH=/tmp/proxy-agent-heartbeat.txt',
'--health-cmd', '[ ! -f $HEARTBEAT_PATH ] || { . $HEARTBEAT_PATH && [ $(date +%s) -lt $HEARTBEAT_EXPIRE_AT ]; }',
'--health-timeout', '3s',
'-v', "${configFileMount}:/config/config.json:ro",
'-v', "${logsMount}:$containerLogDir",
"${IMAGE}:${ImageTag}",
'-i', $containerConfigPath
'--health-timeout', '3s'
)
}
$dockerArgs += Get-AgentRunArgs -ContainerOsType $ContainerOsType
$dockerArgs += @("${IMAGE}:${ImageTag}", '-i', (Get-ContainerConfigPath -ContainerOsType $ContainerOsType))

& docker @dockerArgs | Out-Host
if ($LASTEXITCODE -ne 0) {
Expand Down Expand Up @@ -461,6 +483,59 @@ function Invoke-Upgrade {
}
}

# Runs the doctor in a throwaway container beside the agent's, passing
# $DoctorArgs to --doctor (e.g. --region us-east-1), and exits with its exit
# code. Newer doctors save their log and an archive to the mounted log folder,
# so they outlive the container. Otherwise, as with older doctors (e.g.
# 1.2.10), which only print their report, or when the doctor fails before
# saving its log, its output is saved there instead.
function Invoke-Doctor {
param([string[]]$DoctorArgs)

$containerOsType = Get-ContainerOsType
$windowsVersion = Resolve-WindowsVersion -ContainerOsType $containerOsType
$imageTag = Get-ImageTag -Version $CURRENT_VERSION -ContainerOsType $containerOsType -WindowsVersion $windowsVersion

# No terminal (-t), which would merge the doctor's progress line into the
# saved output. The working folder is where newer doctors save their log if
# they can't read the config.
$dockerArgs = @('run', '--rm', '--workdir', (Get-ContainerLogDir -ContainerOsType $containerOsType))
$dockerArgs += Get-AgentRunArgs -ContainerOsType $containerOsType
$dockerArgs += @("${IMAGE}:${imageTag}", (Get-ContainerConfigPath -ContainerOsType $containerOsType), '--doctor')
if ($DoctorArgs) { $dockerArgs += $DoctorArgs }

$logsDir = Join-Path $BASE_DIR 'logs'
$timestamp = [DateTime]::UtcNow.ToString("yyyy-MM-dd'T'HH-mm-ss")
$outputFile = Join-Path $logsDir "proxy-agent-manager-doctor-$timestamp-$([Guid]::NewGuid().ToString('N')).log"
$earlierDoctorLogs = @(Get-ChildItem -LiteralPath $logsDir -Filter 'doctor-*.log' | ForEach-Object Name)
# Tee-Object can't choose the encoding on Windows PowerShell 5.1.
$outputStream = [System.IO.File]::Open($outputFile, [System.IO.FileMode]::CreateNew, [System.IO.FileAccess]::Write, [System.IO.FileShare]::Read)
$writer = [System.IO.StreamWriter]::new($outputStream, [System.Text.UTF8Encoding]::new($false))
# Under 'Stop', Windows PowerShell 5.1 throws on the first line docker
# writes to stderr once it is redirected.
$ErrorActionPreference = 'Continue'
try {
# Stderr lines arrive as error records; "$_" is their text.
& docker @dockerArgs 2>&1 | ForEach-Object {
$line = "$_"
Write-Host $line
$writer.WriteLine($line)
}
$status = $LASTEXITCODE
} finally {
$writer.Dispose()
}
$newDoctorLogs = Get-ChildItem -LiteralPath $logsDir -Filter 'doctor-*.log' | Where-Object { $_.Name -notin $earlierDoctorLogs }
if ($newDoctorLogs) {
Remove-Item -LiteralPath $outputFile
# The doctor printed paths inside the container.
Write-Host "$(Get-ContainerLogDir -ContainerOsType $containerOsType) in the container is $logsDir on this host."
} else {
Write-Host "The doctor's output was saved to $outputFile"
}
exit $status
}

# ── Commands ─────────────────────────────────────────────────────────────────

switch ($Command) {
Expand Down Expand Up @@ -491,4 +566,9 @@ switch ($Command) {
'logs' {
docker logs -f $CONTAINER_NAME
}
'doctor' {
$CURRENT_VERSION = Get-ReleaseVersion -Tag $PINNED_VERSION
Assert-MemoryAllocation
Invoke-Doctor -DoctorArgs $DoctorArgs
}
}
80 changes: 72 additions & 8 deletions proxy-agent-manager.sh
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@
#
# Management script for Fivetran Proxy Agent (Docker)
#
# Usage: ./proxy-agent-manager.sh {start|stop|restart|upgrade|status|logs}
# Usage: ./proxy-agent-manager.sh {start|stop|restart|upgrade|status|logs|doctor [options]}
#
set -euo pipefail

Expand Down Expand Up @@ -49,6 +49,7 @@ fi

LOGFILE="$BASE_DIR/logs/proxy-agent-manager.log"
CONTAINER_LOG_DIR="/app/logs"
CONTAINER_CONFIG_FILE="/config/config.json"

# ── Functions ────────────────────────────────────────────────────────────────

Expand All @@ -73,6 +74,16 @@ check_memory_allocation() {
fi
}

# Shared by the agent and doctor containers, so the doctor checks the agent's
# config, memory limit and log folder.
AGENT_RUN_ARGS=(
--memory="${MEMORY_ALLOCATION_MB}m"
--env IS_DOCKER=true
--env LOG_FOLDER_PATH="$CONTAINER_LOG_DIR"
-v "$BASE_DIR/config/config.json:$CONTAINER_CONFIG_FILE:ro"
-v "$BASE_DIR/logs:$CONTAINER_LOG_DIR"
)

get_latest_version() {
local registry_host="${IMAGE%%/*}"
local repository_path="${IMAGE#*/}"
Expand Down Expand Up @@ -155,22 +166,18 @@ start_agent() {
docker run -d \
--name "${CONTAINER_NAME}" \
--restart unless-stopped \
--memory="${MEMORY_ALLOCATION_MB}m" \
--label fivetran=proxy-agent \
--label proxy_agent_id="$AGENT_ID" \
--env IS_DOCKER=true \
--env LOG_FOLDER_PATH="$CONTAINER_LOG_DIR" \
"${AGENT_RUN_ARGS[@]}" \
--env HEARTBEAT_PATH=/tmp/proxy-agent-heartbeat.txt \
--env HEARTBEAT_EXPIRY_SECONDS=30 \
--health-cmd '[ ! -f $HEARTBEAT_PATH ] || { . $HEARTBEAT_PATH && [ $(date +%s) -lt $HEARTBEAT_EXPIRE_AT ]; }' \
--health-interval 10s \
--health-timeout 3s \
--health-retries 3 \
--health-start-period 30s \
-v "$BASE_DIR/config/config.json:/config/config.json:ro" \
-v "$BASE_DIR/logs:$CONTAINER_LOG_DIR" \
"$(image_tag "$version")" \
-i /config/config.json
-i "$CONTAINER_CONFIG_FILE"

local timeout=60
log "Waiting for ${CONTAINER_NAME} to become healthy (timeout: ${timeout}s)..."
Expand Down Expand Up @@ -199,6 +206,56 @@ start_agent() {
fi
}

# Lists the logs doctors saved in the log folder, sorted for comm.
list_doctor_logs() {
find "$BASE_DIR/logs" -maxdepth 1 -name 'doctor-*.log' | LC_ALL=C sort
}

# Runs the doctor in a throwaway container beside the agent's, passing "$@" to
# --doctor (e.g. --region us-east-1), and exits with its exit code. Newer
# doctors save their log and an archive to the mounted log folder, so they
# outlive the container. Otherwise, as with older doctors (e.g. 1.2.10), which
# only print their report, or when the doctor fails before saving its log, its
# output is saved there instead.
run_doctor() {
mkdir -p "$BASE_DIR/logs"
local output_file temporary_output_file earlier_doctor_logs timestamp
timestamp=$(date -u +'%Y-%m-%dT%H-%M-%S')
temporary_output_file=$(mktemp "$BASE_DIR/logs/proxy-agent-manager-doctor-${timestamp}.XXXXXXXXXX")
output_file="${temporary_output_file}.log"
# macOS mktemp requires Xs at the end, so add the discoverable .log suffix
# with a hard link. ln creates it atomically and won't overwrite a report.
if ! ln "$temporary_output_file" "$output_file"; then
rm -f "$temporary_output_file"
return 1
fi
rm -f "$temporary_output_file"
earlier_doctor_logs=$(list_doctor_logs)

# No terminal (-t), which would merge the doctor's progress line into the
# saved output. The working folder is where newer doctors save their log if
# they can't read the config.
# set +e so a failing doctor doesn't exit the script, and PIPESTATUS keeps
# the doctor's exit code even if tee fails, e.g. on a full disk.
set +e
docker run --rm \
--workdir "$CONTAINER_LOG_DIR" \
"${AGENT_RUN_ARGS[@]}" \
"$(image_tag "$CURRENT_VERSION")" \
"$CONTAINER_CONFIG_FILE" --doctor "$@" 2>&1 \
| tee "$output_file"
local status="${PIPESTATUS[0]}"
set -e
if [ -n "$(LC_ALL=C comm -13 <(echo "$earlier_doctor_logs") <(list_doctor_logs))" ]; then
rm -f "$output_file"
# The doctor printed paths inside the container.
echo "$CONTAINER_LOG_DIR in the container is $BASE_DIR/logs on this host."
else
echo "The doctor's output was saved to $output_file"
fi
exit "$status"
}

# ── Commands ─────────────────────────────────────────────────────────────────

case "${1:-}" in
Expand Down Expand Up @@ -237,8 +294,15 @@ case "${1:-}" in
docker logs -f "${CONTAINER_NAME}"
;;

doctor)
CURRENT_VERSION=$(get_release_version "$PINNED_VERSION") || exit 1
check_memory_allocation
shift
run_doctor "$@"
;;

*)
echo "Usage: $0 {start|stop|restart|upgrade|status|logs}"
echo "Usage: $0 {start|stop|restart|upgrade|status|logs|doctor [options]}"
exit 1
;;
esac
Loading
Loading