diff --git a/.github/workflows/docs-negotiation.yml b/.github/workflows/docs-negotiation.yml index f0ffc918a23..9653b27d007 100644 --- a/.github/workflows/docs-negotiation.yml +++ b/.github/workflows/docs-negotiation.yml @@ -268,6 +268,7 @@ jobs: DOCS_GENERATORS_REPORT_PATH: ${{ github.workspace }}/apps/docs/test-results/generators/report.json DOCS_COMPATIBILITY_REPORT_PATH: ${{ github.workspace }}/apps/docs/test-results/compatibility.json DOCS_HEADER_REPORT_PATH: ${{ github.workspace }}/apps/docs/test-results/docs-headers.json + GITLAB_EXPORT_REPORT_PATH: ${{ github.workspace }}/apps/docs/test-results/gitlab-export.json steps: - name: Checkout code uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v6 @@ -292,6 +293,8 @@ jobs: - name: Verify native header edge cases if: ${{ !cancelled() && steps.generators.outcome != 'skipped' }} run: bun run --cwd apps/docs test:headers:e2e + - name: Verify GitLab CSV export over HTTPS + run: python3 apps/docs/scripts/test-gitlab-export-e2e.py - name: Bound the generated content cache if: always() run: bash .github/actions/docs-content-cache/prune.sh content diff --git a/apps/docs/content/docs/search/gitlab.mdx b/apps/docs/content/docs/search/gitlab.mdx index 7ed99eb6c4b..5b424809213 100644 --- a/apps/docs/content/docs/search/gitlab.mdx +++ b/apps/docs/content/docs/search/gitlab.mdx @@ -73,6 +73,37 @@ In this example, Alice can search `engineering/platform` because the files link Saving the CSVs grants the access listed in them. Changing a membership in GitLab alone does not change these grants. Replace the CSVs to add or revoke access; the change applies to the next search or read without re-indexing. +### Export both CSVs from GitLab + +Download [export-permissions.py](/static/examples/gitlab/export-permissions.py) and save it as `ci/export-gitlab-permissions.py` in your repository. It uses Python 3.10 or later with no extra packages and exports one project per run. + +Use GitLab 17.4 or later and an internal export token with `read_api` and permission to see all members of the target project and their primary emails. Keep this token in a masked, protected CI variable named `GITLAB_EXPORT_TOKEN`. It stays in your CI job; Sim receives the CSVs. The source's separate read-only PAT continues to retrieve content. + +The exporter makes only these GitLab API requests: + +| Endpoint | Purpose | +| --- | --- | +| `GET /api/v4/projects/:id` | Resolve the numeric ID and full project path. | +| `GET /api/v4/projects/:id/members/all` | Page through visible direct, inherited, and invited-group members, 100 at a time. | +| `GET /api/v4/users/:id` | Read each eligible member's account state and primary email. | + +GitLab documents [membership visibility](https://docs.gitlab.com/api/project_members/#list-all-members-of-a-project) and [user profile fields](https://docs.gitlab.com/api/users/#retrieve-a-single-user). The `read_api` scope alone does not grant access to private emails. The export token needs that directory permission too. If a primary email is unavailable, the script stops; it never substitutes a public email. Confirm the exported emails match your teammates' verified Sim emails. + +With `GITLAB_EXPORT_TOKEN` set in the environment, run: + +```bash +python3 ci/export-gitlab-permissions.py \ + --gitlab-url https://gitlab.example.com \ + --project engineering/platform \ + --output-dir gitlab-permissions +``` + +This creates `gitlab-permissions/users.csv` and `gitlab-permissions/project-permissions.csv`. Numeric project IDs are also accepted. The output directory must not already exist. Both files are published together after the export completes and passes validation. + +The export includes active, unlocked human members with Reporter access or higher. It excludes Guests, Planners, bots, and memberships expiring today or earlier in UTC. It does not infer access from public/internal visibility, administrator or Auditor privileges, or custom permissions below Reporter. GitLab policies such as SSO or external authorization are not represented by these CSVs. Review the grants against your organization's access requirements before uploading them. + +Requests run serially, paced at 60 per minute by default. Set `--requests-per-minute` to change this export-job ceiling; it does not configure live Search traffic. Each run uses one project request, one request per membership page, and one profile request per candidate reader. API errors, hidden emails, identity conflicts, inconsistent pagination totals (when GitLab provides them), and oversized exports stop the job without publishing either file. Run exports after bulk membership updates finish; paginated API reads are not a point-in-time snapshot. + ## Access checks The source token retrieves content. Administrator mode proves the reader's current GitLab identity from verified Sim email, then checks project/group membership and policy during the request. Unsupported restrictions fail closed. @@ -93,7 +124,7 @@ After an admin configures a source in **Non-admin token** mode, a job can [read Store an organization admin's [personal Sim API key](/api-reference/authentication) with [**Full access**](/api-reference/authentication#key-access) as a [masked, protected GitLab CI variable](https://docs.gitlab.com/ci/variables/) named `SIM_API_KEY`. A personal key inherits its owner's access; a read-only key can read the revision but not replace the CSVs. This endpoint can only read the revision or replace both CSVs; it cannot change the GitLab PAT, host, or project. -This example runs on a schedule against a protected branch. Replace `./ci/export-gitlab-permissions` with your existing export command. It must write a fresh, complete `users.csv` and `project-permissions.csv` and exit nonzero if it cannot finish. +Commit [the exporter above](#export-both-csvs-from-gitlab) at `ci/export-gitlab-permissions.py`. Set `GITLAB_EXPORT_TOKEN`, `GITLAB_PROJECT` (full path or numeric ID), and the Sim variables above. This example uses `CI_SERVER_URL` for the GitLab host and runs on a schedule against a protected branch. Choose the schedule based on how quickly permission changes must reach Sim. ```yaml title=".gitlab-ci.yml" sync_gitlab_permissions: @@ -102,22 +133,28 @@ sync_gitlab_permissions: rules: - if: '$CI_PIPELINE_SOURCE == "schedule" && $CI_COMMIT_REF_PROTECTED == "true"' before_script: - - apk add --no-cache curl jq + - apk add --no-cache ca-certificates curl jq python3 script: - | set -eu + umask 077 + export_root=$(mktemp -d) + trap 'rm -rf "$export_root"' EXIT url="${SIM_BASE_URL%/}/api/v2/organizations/$SIM_ORG_ID/search/sources/$SIM_SOURCE_ID/permissions" auth="X-API-Key: $SIM_API_KEY" current=$(curl --fail --silent --show-error -H "$auth" "$url") revision=$(printf '%s' "$current" | jq -er '.data.revision') - ./ci/export-gitlab-permissions + python3 ci/export-gitlab-permissions.py \ + --gitlab-url "$CI_SERVER_URL" --project "$GITLAB_PROJECT" \ + --output-dir "$export_root/permissions" --requests-per-minute 60 jq -n --argjson rev "$revision" \ - --rawfile users users.csv --rawfile grants project-permissions.csv \ + --rawfile users "$export_root/permissions/users.csv" \ + --rawfile grants "$export_root/permissions/project-permissions.csv" \ '{expectedRevision: $rev, permissions: {provider: "gitlab", userMapping: {filename: "users.csv", content: $users}, - projectPermissions: {filename: "project-permissions.csv", content: $grants}}}' > update.json + projectPermissions: {filename: "project-permissions.csv", content: $grants}}}' > "$export_root/update.json" curl --fail --silent --show-error --request PUT -H "$auth" \ - -H 'Content-Type: application/json' --data-binary @update.json "$url" + -H 'Content-Type: application/json' --data-binary "@$export_root/update.json" "$url" ``` A successful update returns the new revision. A `409 Conflict` means someone updated the source after the job read its revision. Rerun the job from the beginning so it exports current grants. An invalid or header-only export leaves the previous grants in place. Alert on failed jobs so revocations aren't missed. If the project should have no readers, choose **Pause search** in the source settings. You can update the grants while the source is paused, then resume search when the grants are correct. diff --git a/apps/docs/public/static/examples/gitlab/export-permissions.py b/apps/docs/public/static/examples/gitlab/export-permissions.py new file mode 100644 index 00000000000..d6891b16405 --- /dev/null +++ b/apps/docs/public/static/examples/gitlab/export-permissions.py @@ -0,0 +1,224 @@ +#!/usr/bin/env python3 +"""Export one GitLab project's readers to Sim's permission CSVs (Python 3.10+).""" + +import argparse +import csv +from datetime import date, datetime, timezone +import io +import json +import os +from pathlib import Path +import re +import ssl +import sys +import tempfile +import time +from urllib.error import HTTPError, URLError +from urllib.parse import quote, urlencode, urlsplit +from urllib.request import HTTPRedirectHandler, HTTPSHandler, Request, build_opener + + +MAX_BYTES = 4 * 1024 * 1024 +MAX_ROWS = 100_000 +EMAIL = re.compile( + r"[a-zA-Z0-9.!#$%&'*+/=?^_`{|}~-]+@" + r"[a-zA-Z0-9](?:[a-zA-Z0-9-]{0,61}[a-zA-Z0-9])?" + r"(?:\.[a-zA-Z0-9](?:[a-zA-Z0-9-]{0,61}[a-zA-Z0-9])?)*" +) + + +class ExportError(Exception): + """An incomplete or unsafe export must not be uploaded.""" + + +class NoRedirects(HTTPRedirectHandler): + def redirect_request(self, req, fp, code, msg, headers, newurl): + # A redirect must never forward the private token to another host. + return None + + +class GitLab: + def __init__(self, base_url, token, requests_per_minute): + parsed = urlsplit(base_url) + if (parsed.scheme != "https" or not parsed.hostname or parsed.username + or parsed.password or parsed.query or parsed.fragment + or any(ord(char) < 33 for char in base_url)): + raise ExportError("Use an HTTPS GitLab instance URL without credentials, query, or fragment.") + self.base = base_url.rstrip("/") + "/api/v4/" + self.token = token + self.interval = 60 / requests_per_minute + self.last_request = 0.0 + self.opener = build_opener(NoRedirects(), HTTPSHandler(context=ssl.create_default_context())) + + def get(self, path, query=None): + delay = self.interval - (time.monotonic() - self.last_request) + if delay > 0: + time.sleep(delay) + url = self.base + path + ("?" + urlencode(query) if query else "") + request = Request(url, headers={"PRIVATE-TOKEN": self.token, "Accept": "application/json"}) + self.last_request = time.monotonic() + try: + with self.opener.open(request, timeout=30) as response: + raw = response.read(8 * 1024 * 1024 + 1) + headers = response.headers + except HTTPError as error: + raise ExportError(f"GitLab GET {path} failed (HTTP {error.code}); no export published.") from None + except (URLError, TimeoutError, OSError): + raise ExportError("GitLab connection failed. Check the host, HTTPS trust, and network access.") from None + if len(raw) > 8 * 1024 * 1024: + raise ExportError("GitLab response exceeds the 8 MiB limit.") + try: + return json.loads(raw), headers + except (ValueError, UnicodeError): + raise ExportError("GitLab returned invalid JSON.") from None + + +def identifier(value): + if type(value) is not int or not 0 < value <= 9_007_199_254_740_991: + raise ExportError("GitLab returned an invalid numeric identity.") + return value + + +def active(record): + if not isinstance(record, dict) or not isinstance(record.get("state"), str): + raise ExportError("GitLab returned a user without an account state.") + return record["state"] == "active" and not record.get("locked", False) + + +def export(client, project): + record, _ = client.get("projects/" + quote(project, safe="")) + if not isinstance(record, dict): + raise ExportError("GitLab returned an invalid project.") + project_id = identifier(record.get("id")) + project_path = record.get("path_with_namespace") + if (not isinstance(project_path, str) or len(project_path) > 1024 + or "/" not in project_path + or any(not part or part in (".", "..") + or not all(char.isalnum() or char in "_.-" for char in part) + for part in project_path.split("/"))): + raise ExportError("GitLab returned an invalid project path.") + users, emails, seen_members = {}, {}, set() + today = datetime.now(timezone.utc).date() + page = 1 + expected_total = None + while page <= 1000: + members, headers = client.get(f"projects/{project_id}/members/all", + {"per_page": 100, "page": page}) + if not isinstance(members, list) or len(members) > 100: + raise ExportError("GitLab returned an invalid membership page.") + total = headers.get("X-Total") + if total is not None: + if not re.fullmatch(r"[0-9]{1,6}", total) or int(total) > MAX_ROWS: + raise ExportError("GitLab membership total is invalid or exceeds the 100,000-row limit.") + if expected_total is not None and int(total) != expected_total: + raise ExportError("GitLab membership total changed during pagination. Rerun the export.") + expected_total = int(total) + for member in members: + if not isinstance(member, dict): + raise ExportError("GitLab returned an invalid membership.") + user_id = identifier(member.get("id")) + if user_id in seen_members: + raise ExportError("GitLab membership pagination repeated a user. Rerun the export.") + seen_members.add(user_id) + if len(seen_members) > MAX_ROWS: + raise ExportError("Project membership exceeds the 100,000-row limit.") + role = member.get("access_level") + if type(role) is not int: + raise ExportError("GitLab returned a membership without an access level.") + if not active(member) or role < 20: + continue + expiry = member.get("expires_at") + if expiry is not None: + try: + expired = date.fromisoformat(expiry) <= today + except (TypeError, ValueError): + raise ExportError("GitLab returned an invalid membership expiry.") from None + if expired: + continue + profile, _ = client.get(f"users/{user_id}") + if not isinstance(profile, dict) or identifier(profile.get("id")) != user_id: + raise ExportError("GitLab returned a mismatched user identity.") + if not active(profile) or profile.get("bot", False): + continue + email = profile.get("email") + if not isinstance(email, str): + raise ExportError( + f"User {user_id}: primary email is unavailable. The CI token needs directory email access." + ) + email = email.strip().lower() + if len(email) > 254 or not EMAIL.fullmatch(email): + raise ExportError(f"User {user_id}: primary email is invalid.") + if email in emails and emails[email] != user_id: + raise ExportError("Multiple GitLab users share an email. Resolve the identity conflict first.") + emails[email] = user_id + users[user_id] = email + next_page = headers.get("X-Next-Page") + if next_page is None: + next_page = str(page + 1) if len(members) == 100 else "" + if not next_page: + break + if not re.fullmatch(r"[0-9]+", next_page) or int(next_page) != page + 1 or not members: + raise ExportError("GitLab returned invalid membership pagination.") + page = int(next_page) + else: + raise ExportError("GitLab membership pagination exceeds 1,000 pages.") + if expected_total is not None and len(seen_members) != expected_total: + raise ExportError("Exported membership count does not match GitLab's total. Rerun the export.") + if not users: + raise ExportError("No eligible readers. Pause search in Sim; empty CSVs cannot revoke all grants.") + return project_path, users + + +def write_export(output, project_path, users): + files = {} + ordered = sorted(users) + for name, header, rows in [ + ("users.csv", ["user_id", "email"], [(user_id, users[user_id]) for user_id in ordered]), + ("project-permissions.csv", ["project_path", "user_id"], + [(project_path, user_id) for user_id in ordered]), + ]: + buffer = io.StringIO(newline="") + writer = csv.writer(buffer, lineterminator="\n") + writer.writerow(header) + writer.writerows(rows) + files[name] = buffer.getvalue().encode("utf-8") + if len(files[name]) > MAX_BYTES: + raise ExportError(f"{name} exceeds Sim's 4 MiB CSV limit.") + output.parent.mkdir(parents=True, exist_ok=True) + # Publish the pair together, so an interrupted export cannot mix new and old CSVs. + with tempfile.TemporaryDirectory(dir=output.parent, prefix=".gitlab-export-") as temporary: + for name, content in files.items(): + path = Path(temporary) / name + path.write_bytes(content) + path.chmod(0o600) + os.rename(temporary, output) + + +def main(): + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument("--gitlab-url", required=True, help="HTTPS instance URL, without /api/v4") + parser.add_argument("--project", required=True, help="Project ID or full namespace/project path") + parser.add_argument("--output-dir", default="gitlab-permissions", help="New directory for both CSVs") + parser.add_argument("--requests-per-minute", type=int, default=60, help="Request-start ceiling (default: 60)") + args = parser.parse_args() + token = os.environ.get("GITLAB_EXPORT_TOKEN", "") + if not token or any(char.isspace() for char in token): + parser.error("Set GITLAB_EXPORT_TOKEN to the CI export token.") + if args.requests_per_minute <= 0: + parser.error("--requests-per-minute must be positive.") + output = Path(args.output_dir).absolute() + if output.exists() or output.is_symlink(): + raise ExportError("Output directory already exists. Choose a new directory for a fresh export.") + client = GitLab(args.gitlab_url, token, args.requests_per_minute) + project_path, users = export(client, args.project) + write_export(output, project_path, users) + print(f"Exported {len(users)} readers to {output}/users.csv and {output}/project-permissions.csv") + + +if __name__ == "__main__": + try: + main() + except (ExportError, OSError) as error: + message = str(error) if isinstance(error, ExportError) else "Could not publish the CSV files." + print(f"Export failed: {message}", file=sys.stderr) + sys.exit(1) diff --git a/apps/docs/scripts/fixtures/content-baseline.json b/apps/docs/scripts/fixtures/content-baseline.json index 0b72b79d282..41cdbd0b4f5 100644 --- a/apps/docs/scripts/fixtures/content-baseline.json +++ b/apps/docs/scripts/fixtures/content-baseline.json @@ -177203,21 +177203,28 @@ "meta": null, "sha256": "68cb9b0d31751b48e06f8d452f15821f1e68ee3816f17638a88294587a58908c", "bytes": 55, - "line": 48 + "line": 52 }, { "lang": "csv", "meta": null, "sha256": "5c2ebfcc044ebc01c17a4af2664a4fbc667ae4916ef21555d90cfd9f245394bd", "bytes": 70, - "line": 56 + "line": 60 + }, + { + "lang": "bash", + "meta": null, + "sha256": "473dea567e9f48d8b1fa1dded81bf8a30ce177ea12338164f05d3fd015570637", + "bytes": 154, + "line": 94 }, { "lang": "yaml", "meta": "title=\".gitlab-ci.yml\"", - "sha256": "bf3d6e7bec6b87f3966580713633eb8baed778c9dced4569a65cc62d790dee42", - "bytes": 1084, - "line": 98 + "sha256": "29261e0a66aac3cf52e7b9164aaa44f235765bcfd2cae4e68d744d31b29b57bf", + "bytes": 1441, + "line": 129 } ], "links": [ diff --git a/apps/docs/scripts/test-gitlab-export-e2e.py b/apps/docs/scripts/test-gitlab-export-e2e.py new file mode 100644 index 00000000000..9acac3079e0 --- /dev/null +++ b/apps/docs/scripts/test-gitlab-export-e2e.py @@ -0,0 +1,206 @@ +#!/usr/bin/env python3 +"""Exercise the downloadable exporter over HTTPS, in a separate process.""" + +import csv +from datetime import datetime, timezone +import json +import os +from pathlib import Path +import ssl +import subprocess +import sys +import tempfile +import threading +import time +from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer +from urllib.parse import parse_qs, urlsplit + + +SCRIPT = Path(__file__).resolve().parents[1] / "public/static/examples/gitlab/export-permissions.py" +TOKEN = "fixture-token-must-not-appear-in-output" +SIM_ROOT = Path(__file__).resolve().parents[2] / "sim" + + +def member(user_id, **overrides): + return {"id": user_id, "state": "active", "access_level": 30, + "expires_at": None, **overrides} + + +class GitLabFixture(BaseHTTPRequestHandler): + calls = [] + + def log_message(self, *_args): + pass + + def do_GET(self): + parsed = urlsplit(self.path) + scenario, endpoint = parsed.path.lstrip("/").split("/api/v4/", 1) + self.calls.append((scenario, endpoint, parse_qs(parsed.query), time.monotonic())) + if self.headers.get("PRIVATE-TOKEN") != TOKEN: + self.reply(401, {"message": "Missing token"}) + return + if scenario == "denied": + self.reply(403, {"message": TOKEN}) + return + if endpoint in ("projects/engineering%2Fplatform", "projects/91"): + if scenario == "redirect": + self.reply(302, {}, {"Location": "/redirect/api/v4/trap"}) + else: + self.reply(200, {"id": 91, "path_with_namespace": "engineering/platform", + "repository_access_level": "disabled" if scenario == "issues-only" else "enabled", + "issues_access_level": "enabled", "wiki_access_level": "enabled"}) + elif endpoint == "projects/91/members/all": + page = parse_qs(parsed.query).get("page", ["1"])[0] + total = "11" if scenario in ("truncated-membership", "changing-total") else "10" + if scenario == "changing-total" and page == "2": + total = "10" + headers = {} if scenario == "missing-pagination" else {"X-Total": total} + if scenario == "page-failure" and page == "2": + self.reply(500, {"message": TOKEN}) + elif scenario == "pagination-loop": + self.reply(200, [member(1)], {"X-Next-Page": "1"}) + elif scenario == "missing-pagination" and page == "1": + self.reply(200, [member(i, access_level=10) for i in range(100, 200)]) + elif scenario == "empty": + self.reply(200, [member(3, access_level=10)]) + elif page == "1": + self.reply(200, [member(2), member(1)], {**headers, "X-Next-Page": "2"}) + else: + self.reply(200, [member(3, access_level=10), member(4, state="blocked"), + member(5), member(6, expires_at="2000-01-01"), + member(7), member(8, access_level=15), member(9), + member(10, expires_at=datetime.now(timezone.utc).date().isoformat())], + {**headers, "X-Next-Page": ""}) + elif endpoint.startswith("users/"): + user_id = int(endpoint.split("/")[1]) + email = {1: "ALICE@example.com", 2: "bob@example.com", 7: "carol@example.com"}.get( + user_id, f"service-{user_id}@example.com") + if scenario == "duplicate-email" and user_id == 2: + email = "alice@example.com" + profile = {"id": user_id, "state": "active", "locked": user_id == 9, + "bot": user_id == 5, "email": email} + if scenario == "hidden-email": + profile.pop("email") + profile["public_email"] = f"public-alias-{user_id}@example.com" + if scenario == "wrong-identity": + profile["id"] = 999 + self.reply(200, profile) + else: + self.reply(404, {"message": "Unexpected endpoint"}) + + def reply(self, status, body, headers=None): + data = json.dumps(body).encode() + self.send_response(status) + self.send_header("Content-Type", "application/json") + self.send_header("Content-Length", str(len(data))) + for key, value in (headers or {}).items(): + self.send_header(key, value) + self.end_headers() + self.wfile.write(data) + + +def run(): + report_path = Path(os.environ["GITLAB_EXPORT_REPORT_PATH"]) + report = {"boundary": "CLI subprocess against a local HTTPS GitLab API fixture", "checks": []} + try: + with tempfile.TemporaryDirectory() as temporary: + root = Path(temporary) + cert, key = root / "cert.pem", root / "key.pem" + subprocess.run([ + "openssl", "req", "-x509", "-newkey", "rsa:2048", "-nodes", "-days", "1", + "-keyout", str(key), "-out", str(cert), "-subj", "/CN=localhost", + "-addext", "subjectAltName=IP:127.0.0.1", + ], check=True, capture_output=True) + server = ThreadingHTTPServer(("127.0.0.1", 0), GitLabFixture) + tls = ssl.SSLContext(ssl.PROTOCOL_TLS_SERVER) + tls.load_cert_chain(cert, key) + server.socket = tls.wrap_socket(server.socket, server_side=True) + thread = threading.Thread(target=server.serve_forever, daemon=True) + thread.start() + try: + for scenario, expected_error in [ + ("success", None), ("numeric-project", None), ("missing-pagination", None), + ("issues-only", None), ("paced", None), + ("truncated-membership", "total"), ("changing-total", "changed"), + ("hidden-email", "email"), ("duplicate-email", "email"), + ("wrong-identity", "identity"), ("denied", "403"), + ("page-failure", "500"), ("pagination-loop", "pagination"), + ("redirect", "302"), ("empty", "readers"), + ("existing-output", "exists"), + ]: + output = root / scenario + if scenario == "existing-output": + output.mkdir() + (output / "users.csv").write_text("keep the previous export") + command = [sys.executable, str(SCRIPT), "--gitlab-url", + f"https://127.0.0.1:{server.server_port}/{scenario}", + "--project", "91" if scenario == "numeric-project" else "engineering/platform", + "--output-dir", str(output), + "--requests-per-minute", "120" if scenario == "paced" else "600000"] + result = subprocess.run(command, capture_output=True, text=True, timeout=20, + env={**os.environ, "GITLAB_EXPORT_TOKEN": TOKEN, + "SSL_CERT_FILE": str(cert)}) + assert TOKEN not in result.stdout + result.stderr, "Token leaked in output" + if expected_error: + assert result.returncode != 0, f"{scenario}: export unexpectedly succeeded" + assert expected_error in result.stderr.lower(), (scenario, result.stderr) + if scenario == "existing-output": + assert (output / "users.csv").read_text() == "keep the previous export" + else: + assert not output.exists(), f"{scenario}: published a partial export" + else: + assert result.returncode == 0, (scenario, result.stderr) + with (output / "users.csv").open() as handle: + users = list(csv.reader(handle)) + with (output / "project-permissions.csv").open() as handle: + grants = list(csv.reader(handle)) + expected = [["7", "carol@example.com"]] if scenario == "missing-pagination" else [ + ["1", "alice@example.com"], ["2", "bob@example.com"], + ["7", "carol@example.com"]] + assert users == [["user_id", "email"], *expected], users + assert grants == [["project_path", "user_id"], *[ + ["engineering/platform", row[0]] for row in expected]], grants + parsed = subprocess.run([ + "bun", "-e", """ +import { parseGitLabCsv, gitLabCsvSubjects } from './connectors/gitlab/permission-config/parser' +const directory = process.argv[1] +const users = parseGitLabCsv({filename: 'users.csv', content: await Bun.file(`${directory}/users.csv`).text()}, 'userMapping') +const grants = parseGitLabCsv({filename: 'project-permissions.csv', content: await Bun.file(`${directory}/project-permissions.csv`).text()}, 'projectPermissions') +process.stdout.write(JSON.stringify({ + subjects: gitLabCsvSubjects(users, grants, 'engineering/platform'), + otherProject: gitLabCsvSubjects(users, grants, 'engineering/other'), +})) +""", str(output), + ], cwd=SIM_ROOT, capture_output=True, text=True, timeout=20) + assert parsed.returncode == 0, parsed.stderr + assert json.loads(parsed.stdout) == { + "subjects": sorted("u:" + row[1] for row in expected), + "otherProject": [], + }, parsed.stdout + if scenario == "paced": + arrivals = [arrived for case, _, _, arrived in GitLabFixture.calls if case == scenario] + # The first request's transit time can consume one pacing interval. + minimum_span = (len(arrivals) - 2) * 0.5 + assert arrivals[-1] - arrivals[0] >= minimum_span, "Requests exceeded the configured rate" + report["pacing"] = {"requests": len(arrivals), "requestsPerMinute": 120, + "observedSeconds": arrivals[-1] - arrivals[0], + "minimumSeconds": minimum_span} + report["checks"].append({"name": scenario, "passed": True}) + assert not any(endpoint == "trap" for _, endpoint, _, _ in GitLabFixture.calls) + report["passed"] = True + finally: + server.shutdown() + server.server_close() + thread.join() + except Exception as error: + report["passed"] = False + report["error"] = str(error) + raise + finally: + report_path.parent.mkdir(parents=True, exist_ok=True) + report_path.write_text(json.dumps(report, indent=2) + "\n") + print(f"Passed {len(report['checks'])} exporter checks. Report: {report_path}") + + +if __name__ == "__main__": + run()