Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 3 additions & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -23,6 +23,7 @@ This [Terraform](https://www.terraform.io/) module creates the required infrastr
- Tailored software, hardware and network configuration: Bring your own AMI, define the instance types and subnets to use.
- OS support: Linux (x64/arm64) and Windows
- Multi-Runner: Create multiple runner configurations with a single deployment
- [Multiple organizations](docs/multi-org.md): Opt-in organization-aware registration, scheduled pools, and cleanup
- GitHub cloud, GitHub Cloud with Data Residency and GitHub Enterprise Server (GHES) support.
- Org and repo level runners. enterprise level runners are not supported (yet).

Expand Down Expand Up @@ -128,6 +129,7 @@ Join our discord community via [this invite link](https://discord.gg/bxgXW8jJGh)
| <a name="input_enable_jit_config"></a> [enable\_jit\_config](#input\_enable\_jit\_config) | Overwrite the default behavior for JIT configuration. By default JIT configuration is enabled for ephemeral runners and disabled for non-ephemeral runners. In case of GHES check first if the JIT config API is available. In case you are upgrading from 3.x to 4.x you can set `enable_jit_config` to `false` to avoid a breaking change when having your own AMI. | `bool` | `null` | no |
| <a name="input_enable_job_queued_check"></a> [enable\_job\_queued\_check](#input\_enable\_job\_queued\_check) | Only scale if the job event received by the scale up lambda is in the queued state. By default enabled for non ephemeral runners and disabled for ephemeral. Set this variable to overwrite the default behavior. | `bool` | `null` | no |
| <a name="input_enable_managed_runner_security_group"></a> [enable\_managed\_runner\_security\_group](#input\_enable\_managed\_runner\_security\_group) | Enables creation of the default managed security group. Unmanaged security groups can be specified via `runner_additional_security_group_ids`. | `bool` | `true` | no |
| <a name="input_enable_multi_org_runners"></a> [enable\_multi\_org\_runners](#input\_enable\_multi\_org\_runners) | Enable organization-scoped runners across multiple GitHub organizations. Resolves app installations per organization, scopes runner-group caches and idle retention by organization, and enables pool\_config.org. | `bool` | `false` | no |
| <a name="input_enable_organization_runners"></a> [enable\_organization\_runners](#input\_enable\_organization\_runners) | Register runners to organization, instead of repo level | `bool` | `false` | no |
| <a name="input_enable_runner_bidirectional_label_match"></a> [enable\_runner\_bidirectional\_label\_match](#input\_enable\_runner\_bidirectional\_label\_match) | If set to true, the runner labels and workflow job labels must be an exact two-way match (same set, any order, no extras or missing labels). This is stricter than `enable_runner_workflow_job_labels_check_all` which only checks that workflow labels are a subset of runner labels. When false, if __any__ label matches it will trigger the webhook. | `bool` | `false` | no |
| <a name="input_enable_runner_binaries_syncer"></a> [enable\_runner\_binaries\_syncer](#input\_enable\_runner\_binaries\_syncer) | Option to disable the lambda to sync GitHub runner distribution, useful when using a pre-build AMI. | `bool` | `true` | no |
Expand Down Expand Up @@ -171,7 +173,7 @@ Join our discord community via [this invite link](https://discord.gg/bxgXW8jJGh)
| <a name="input_metrics"></a> [metrics](#input\_metrics) | Configuration for metrics created by the module, by default disabled to avoid additional costs. When metrics are enable all metrics are created unless explicit configured otherwise. | <pre>object({<br/> enable = optional(bool, false)<br/> namespace = optional(string, "GitHub Runners")<br/> metric = optional(object({<br/> enable_github_app_rate_limit = optional(bool, true)<br/> enable_job_retry = optional(bool, true)<br/> enable_spot_termination_warning = optional(bool, true)<br/> }), {})<br/> })</pre> | `{}` | no |
| <a name="input_minimum_running_time_in_minutes"></a> [minimum\_running\_time\_in\_minutes](#input\_minimum\_running\_time\_in\_minutes) | The time an ec2 action runner should be running at minimum before terminated, if not busy. | `number` | `null` | no |
| <a name="input_parameter_store_tags"></a> [parameter\_store\_tags](#input\_parameter\_store\_tags) | Map of tags that will be added to all the SSM Parameter Store parameters created by the Lambda function. | `map(string)` | `{}` | no |
| <a name="input_pool_config"></a> [pool\_config](#input\_pool\_config) | The configuration for updating the pool. The `pool_size` to adjust to by the events triggered by the `schedule_expression`. For example you can configure a cron expression for weekdays to adjust the pool to 10 and another expression for the weekend to adjust the pool to 1. Use `schedule_expression_timezone` to override the schedule time zone (defaults to UTC). | <pre>list(object({<br/> schedule_expression = string<br/> schedule_expression_timezone = optional(string)<br/> size = number<br/> }))</pre> | `[]` | no |
| <a name="input_pool_config"></a> [pool\_config](#input\_pool\_config) | The configuration for updating the pool. The `pool_size` to adjust to by the events triggered by the `schedule_expression`. For example you can configure a cron expression for weekdays to adjust the pool to 10 and another expression for the weekend to adjust the pool to 1. Use `schedule_expression_timezone` to override the schedule time zone (defaults to UTC). With `enable_multi_org_runners`, set `org` per schedule; omitted values use `pool_runner_owner`. | <pre>list(object({<br/> schedule_expression = string<br/> schedule_expression_timezone = optional(string)<br/> org = optional(string)<br/> size = number<br/> }))</pre> | `[]` | no |
| <a name="input_pool_include_busy_runners"></a> [pool\_include\_busy\_runners](#input\_pool\_include\_busy\_runners) | Include busy runners in the pool calculation. By default busy runners are not included in the pool. | `bool` | `false` | no |
| <a name="input_pool_lambda_memory_size"></a> [pool\_lambda\_memory\_size](#input\_pool\_lambda\_memory\_size) | Memory size limit for scale-up lambda. | `number` | `512` | no |
| <a name="input_pool_lambda_reserved_concurrent_executions"></a> [pool\_lambda\_reserved\_concurrent\_executions](#input\_pool\_lambda\_reserved\_concurrent\_executions) | Amount of reserved concurrent executions for the scale-up lambda function. A value of 0 disables lambda from being triggered and -1 removes any concurrency limitations. | `number` | `1` | no |
Expand Down
45 changes: 45 additions & 0 deletions docs/multi-org.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,45 @@
# Multiple organizations

Enable `enable_multi_org_runners = true` to share a control plane across organizations. Runners register in the organization that owns the repository in the webhook. This selects organization-level registration even if `enable_organization_runners` is false. The flag defaults to false, preserving existing registration, installation selection, pool ownership, and scale-down behavior.

Install the GitHub App in each target organization. An enterprise-owned app can be used through its organization installations. The app needs **Self-hosted runners: write** at organization scope for [organization JIT configuration](https://docs.github.com/en/rest/actions/self-hosted-runners#create-configuration-for-a-just-in-time-runner-for-an-organization), along with the existing workflow-job permissions and webhook subscriptions. This mode uses organization runner APIs.

## Scheduled pools

Add `org` to each `pool_config` schedule:

```hcl
enable_multi_org_runners = true
enable_ephemeral_runners = true

pool_config = [
{
org = "org-a"
schedule_expression = "cron(0 8 * * ? *)"
schedule_expression_timezone = "UTC"
size = 2
},
{
org = "org-b"
schedule_expression = "cron(0 8 * * ? *)"
schedule_expression_timezone = "UTC"
size = 5
},
]
```

An omitted `org` uses `pool_runner_owner`. Multi-org pools must have a valid organization login in one of those fields. Use the organization's login rather than its display name. Multi-org mode normalizes it to lowercase across pools, webhooks, retries, and cleanup. Without the flag, `org` is ignored and the existing default owner is used.

Pool reconciliation lists GitHub runners and compute instances for that organization only. `runners_maximum_count` applies separately to each organization within a runner configuration. Each organization shares that runner configuration's labels, runner-group name, compute settings, and maximum count. A schedule defines a target size, not an additive pool; avoid conflicting schedules for the same organization. Existing scale-up/pool concurrency limits still apply, and maximum checks are not atomic across concurrent invocations.

For the legacy `modules/multi-runner` interface, set `enable_multi_org_runners` and `pool_config` inside the entry's `runner_config`. For the v2 interface and `modules/runner-config`, set `orchestration_provider.webhook.github.multi_org_runners = true` and put the schedules under `orchestration_provider.webhook.lambda.pool.config`. Its default pool owner is `lambda.pool.runner_owner`.

## Installation and runner lifecycle

- Scale-up and job retry reuse the primary app's webhook installation ID. Additional apps, or events without an installation ID, resolve the selected app's installation for the target organization. Every configured app that can be selected must be installed in all target organizations.
- Pool and scale-down resolve an organization installation with the selected app. Preconfigured global installation IDs are ignored in multi-org mode because they cannot identify installations in several organizations.
- Runner-group IDs are cached by organization and group name. A group named `Default` in one organization cannot supply another organization's group ID. Existing unscoped entries are not reused in multi-org mode.
- EC2 already persists the organization in `ghr:Owner` alongside `ghr:Type = Org`. Scale-down, deregistration, and orphan checks use that ownership metadata; no additional tag is required. Capacity lookups include existing mixed-case owner tags, and scale-down groups those tags under the same lowercase organization. EC2 queries retain their environment and runner-type filters, then compare owner tags locally because AWS tag matching is case-sensitive. Other compute providers use the equivalent owner/type fields in their provider contract.
- Scale-down applies the existing idle configuration independently to each organization. After enabling multi-org on a repository-scoped deployment, legacy repository runners share the same organization allowance as new organization runners. Eviction ordering applies across both types, while cleanup continues to use each runner’s original repository or organization API. Pool sizes do not change scale-down idle settings; these remain separate controls. Orphan checks use the tagged owner's GitHub endpoints, including the final check before termination of a JIT orphan. A GitHub lookup failure does not establish that a runner is an orphan.

This feature does not verify enterprise membership. The organizations available to the GitHub Apps and the existing webhook repository allowlist define the accepted scope. Existing owner tags remain readable when toggling the flag; do not remove an app installation while it still has managed runners to clean up.
11 changes: 11 additions & 0 deletions lambdas/functions/control-plane/src/github/multi-org.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,11 @@
import yn from 'yn';

/** Opt-in organization-scoped installation selection and lifecycle accounting. */
export function multiOrgEnabled(): boolean {
return yn(process.env.ENABLE_MULTI_ORG_RUNNERS, { default: false });
}

/** Preserve legacy identity while using GitHub's case-insensitive org logins in multi-org mode. */
export function normalizeOrganization(owner: string): string {
return multiOrgEnabled() ? owner.toLowerCase() : owner;
}
27 changes: 27 additions & 0 deletions lambdas/functions/control-plane/src/github/octokit.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -190,3 +190,30 @@ describe('Test getOctokit stale installation fallback', () => {
expect(createGithubInstallationAuth).toHaveBeenCalledTimes(1);
});
});

describe('multi-org retry authentication', () => {
it.each([0, 1])('ignores global installation for app %s', async (appIndex) => {
vi.clearAllMocks();
vi.stubEnv('ENABLE_MULTI_ORG_RUNNERS', 'true');
try {
vi.mocked(createGithubAppAuth).mockResolvedValue({ token: 'token', appIndex } as Awaited<
ReturnType<typeof createGithubAppAuth>
>);
vi.mocked(getStoredInstallationId).mockResolvedValue(999);
mockOctokit.apps.getOrgInstallation.mockResolvedValue({ data: { id: 20 } });
await getOctokit('', true, {
eventType: 'workflow_job',
id: 1,
repositoryOwner: 'org-b',
repositoryName: 'repo',
repoOwnerType: 'Organization',
installationId: 10,
});
expect(getStoredInstallationId).not.toHaveBeenCalled();
expect(createGithubInstallationAuth).toHaveBeenCalledWith(appIndex === 0 ? 10 : 20, '', appIndex);
if (appIndex === 1) expect(mockOctokit.apps.getOrgInstallation).toHaveBeenCalledWith({ org: 'org-b' });
} finally {
vi.unstubAllEnvs();
}
});
});
5 changes: 3 additions & 2 deletions lambdas/functions/control-plane/src/github/octokit.ts
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,7 @@ import {
createOctokitClient,
getStoredInstallationId,
} from './auth';
import { multiOrgEnabled, normalizeOrganization } from './multi-org';

const logger = createChildLogger('octokit');

Expand All @@ -27,7 +28,7 @@ async function resolveInstallationIdFromApi(
return enableOrgLevel
? (
await githubClient.apps.getOrgInstallation({
org: payload.repositoryOwner,
org: normalizeOrganization(payload.repositoryOwner),
})
).data.id
: (
Expand All @@ -45,7 +46,7 @@ async function resolveInstallationId(
appIndex?: number,
): Promise<number> {
// Use pre-stored installation ID when available (avoids an API call)
if (appIndex !== undefined) {
if (!multiOrgEnabled() && appIndex !== undefined) {
const storedId = await getStoredInstallationId(appIndex);
if (storedId !== undefined) return storedId;
}
Expand Down
93 changes: 93 additions & 0 deletions lambdas/functions/control-plane/src/pool/pool.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -363,3 +363,96 @@ describe('pool adjustment', () => {
});
});
});

describe('multi-org pools', () => {
it('isolates organization installation, capacity lookup and runner registration for each schedule', async () => {
process.env.ENABLE_MULTI_ORG_RUNNERS = 'true';
vi.mocked(ghAuth.getStoredInstallationId).mockResolvedValueOnce(999);
for (const org of ['org-a', 'org-b']) {
await adjust({ poolSize: 3, org });
expect(githubClient.apps.getOrgInstallation).toHaveBeenLastCalledWith({ org });
expect(githubClient.paginate).toHaveBeenLastCalledWith(githubClient.actions.listSelfHostedRunnersForOrg, {
org,
per_page: 100,
});
expect(poolProvider.listRunners).toHaveBeenLastCalledWith({
environment: process.env.ENVIRONMENT,
runnerOwner: org,
runnerType: 'Org',
runnerOwnerIgnoreCase: true,
});
expect(poolProvider.createRunners).toHaveBeenLastCalledWith(
expect.objectContaining({
numberOfRunners: 1,
githubRunnerConfig: expect.objectContaining({ runnerOwner: org, runnerType: 'Org' }),
}),
);
}
expect(ghAuth.getStoredInstallationId).not.toHaveBeenCalled();
vi.mocked(ghAuth.getStoredInstallationId).mockReset().mockResolvedValue(undefined);
});

it('uses the default owner for a schedule without an org in multi-org mode', async () => {
process.env.ENABLE_MULTI_ORG_RUNNERS = 'true';
await adjust({ poolSize: 3 });
expect(githubClient.apps.getOrgInstallation).toHaveBeenCalledWith({ org: ORG });
});

it.each([undefined, 'owner/repo', ''])('rejects a missing or invalid owner %s before GitHub calls', async (org) => {
process.env.ENABLE_MULTI_ORG_RUNNERS = 'true';
delete process.env.RUNNER_OWNER;
await expect(adjust({ poolSize: 3, org })).rejects.toThrow('Multi-org pools require an organization');
expect(mockedAppAuth).not.toHaveBeenCalled();
});

describe.each(['event.org', 'RUNNER_OWNER'])('login validation for %s', (source) => {
it.each(['org-', 'org--name', '-org', 'a'.repeat(40), 'org_name', 'org\n'])(
'rejects invalid login %j before GitHub calls',
async (org) => {
process.env.ENABLE_MULTI_ORG_RUNNERS = 'true';
if (source === 'RUNNER_OWNER') process.env.RUNNER_OWNER = org;
await expect(adjust({ poolSize: 3, org: source === 'event.org' ? org : undefined })).rejects.toThrow(
'1-39 alphanumeric characters or single hyphens',
);
expect(mockedAppAuth).not.toHaveBeenCalled();
},
);

it.each(['a', 'Org-1', 'org-a-b', 'a'.repeat(39), `${'a'.repeat(37)}-1`])('accepts valid login %s', async (org) => {
process.env.ENABLE_MULTI_ORG_RUNNERS = 'true';
if (source === 'RUNNER_OWNER') process.env.RUNNER_OWNER = org;
await adjust({ poolSize: 3, org: source === 'event.org' ? org : undefined });
expect(githubClient.apps.getOrgInstallation).toHaveBeenCalledWith({ org: org.toLowerCase() });
});
});

it('ignores event.org when multi-org is disabled', async () => {
process.env.ENABLE_MULTI_ORG_RUNNERS = 'false';
await adjust({ poolSize: 3, org: 'org-b' });
expect(githubClient.apps.getOrgInstallation).toHaveBeenCalledWith({ org: ORG });
expect(poolProvider.createRunners).toHaveBeenCalledWith(
expect.objectContaining({
githubRunnerConfig: expect.objectContaining({ runnerOwner: ORG }),
}),
);
});
});

it('normalizes the pool owner for registration and capacity lookup only in multi-org mode', async () => {
process.env.ENABLE_MULTI_ORG_RUNNERS = 'true';
await adjust({ poolSize: 3, org: 'Org-A' });
expect(poolProvider.listRunners).toHaveBeenCalledWith(
expect.objectContaining({ runnerOwner: 'org-a', runnerOwnerIgnoreCase: true }),
);
expect(poolProvider.createRunners).toHaveBeenCalledWith(
expect.objectContaining({ githubRunnerConfig: expect.objectContaining({ runnerOwner: 'org-a' }) }),
);
process.env.ENABLE_MULTI_ORG_RUNNERS = 'false';
process.env.RUNNER_OWNER = 'Org-A';
await adjust({ poolSize: 3 });
expect(poolProvider.listRunners).toHaveBeenLastCalledWith({
environment: process.env.ENVIRONMENT,
runnerOwner: 'Org-A',
runnerType: 'Org',
});
});
Loading
Loading