From d54196c6dcc0af4bc00e7c845e9aef88b86ae399 Mon Sep 17 00:00:00 2001 From: Son Seong Jun <166786347+SonAIengine@users.noreply.github.com> Date: Fri, 20 Nov 2026 13:48:47 +0900 Subject: [PATCH 1/2] =?UTF-8?q?=EC=BD=94=EB=94=A9=205=EC=A2=85=C2=B720?= =?UTF-8?q?=ED=9A=8C=20=EA=B8=B0=EC=A4=80=EC=84=A0=20=EC=8B=A4=ED=96=89?= =?UTF-8?q?=EA=B8=B0=EC=99=80=20=EB=8F=85=EB=A6=BD=20=EC=99=84=EB=A3=8C=20?= =?UTF-8?q?=EA=B2=80=EC=A6=9D=20=EC=B6=94=EA=B0=80?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- MEMORY.md | 7 + docs/development/coding-baseline.md | 115 +++++++ docs/development/tool-integration-todo.md | 8 +- evals/coding-baseline/cases.json | 146 ++++++++ evals/coding-baseline/config.example.json | 24 ++ scripts/evaluate-coding-baseline.py | 388 ++++++++++++++++++++++ scripts/evaluate-planning-model.py | 5 +- scripts/evaluate-projects.py | 5 +- scripts/tests/test_coding_baseline.py | 216 ++++++++++++ 9 files changed, 909 insertions(+), 5 deletions(-) create mode 100644 docs/development/coding-baseline.md create mode 100644 evals/coding-baseline/cases.json create mode 100644 evals/coding-baseline/config.example.json create mode 100644 scripts/evaluate-coding-baseline.py create mode 100644 scripts/tests/test_coding_baseline.py diff --git a/MEMORY.md b/MEMORY.md index e4a99d8..e89667e 100644 --- a/MEMORY.md +++ b/MEMORY.md @@ -14,6 +14,13 @@ ## 책임과 개발 범위 +2026-11-20 구현·mock 검증: [실제 모델 코딩 회귀 기준선](docs/development/coding-baseline.md). +공개 `run/resume`에 고정 5종·설계/보류 각 1개·기본 2회인 20 trial matrix를 연결했다. +기존 비용 proxy·reservation·Receipt 관찰을 재사용하고 독립 oracle·잘못된 완료·반복·승인·offline replay를 +집계한다. 실제 CLI의 합성 모델로 정상 10개와 틀린 완료 10개를 구분했다. 이는 모델 품질 결과가 아니다. +실제 모델은 활성 프로필 catalog 검사에서 `authentication_rejected`여서 미실행이다. +소유자는 xgen 평가 실행기이며 runtime·prompt·기본 모델·배포본은 변경하지 않았다. + 2026-11-20 구현·로컬 검증: [Native build 전 공통 조립 검사](docs/adr/0089-release-assembly-preflight.md). PR Fast/Linux와 release quality에서 dependency·credential 없는 임시 환경으로 실제 pack→조립→checksum을 검사한다. Production과 같은 entrypoint를 쓰며 입력/출력은 platform catalog에서 파생한다. diff --git a/docs/development/coding-baseline.md b/docs/development/coding-baseline.md new file mode 100644 index 0000000..ebd575c --- /dev/null +++ b/docs/development/coding-baseline.md @@ -0,0 +1,115 @@ +# 실제 모델 코딩 회귀 기준선 + +공개 `xgen run/resume`을 고정 과업으로 반복 실행하는 일반 평가다. Runtime·prompt·기본 모델은 바꾸지 +않는다. 실행기는 [evaluate-coding-baseline.py](../../scripts/evaluate-coding-baseline.py), 과업 정본은 +[cases.json](../../evals/coding-baseline/cases.json)이다. 기존 planning pilot의 비용 proxy·reservation· +process tree 정리와 Receipt 관찰 코드를 재사용한다. + +## 과업과 판정 + +| 종류 | 설계 사례 | 보류 사례 | 추가 판정 | +| --- | --- | --- | --- | +| 저장소 질문 | 설정과 분기에서 실행 경로·batch 크기 찾기 | 여러 파일에서 기본값과 override 결과 찾기 | 요청한 JSON 답과 독립 정답 일치, 파일 변경 없음 | +| 버그 수정 | 빈도 정렬 | 반열린 구간 경계 | 기존 테스트 보존, 독립 경계값 검사 | +| 여러 파일 이름 변경 | dataclass 필드·직렬화 키 | 함수 정의·import·호출부 | 전체 사용처 갱신, 기존 동작 보존 | +| 실패 복구 | flag parser | 행렬 전치 | 같은 검증 명령의 실패 후 성공을 Receipt로 관찰 | +| 승인 후 재개 | 정수 나눗셈 올림 | 공백 제거 후 빈 줄 제외 | 실행 승인 전 process 출력 0개, 같은 Run 재개 | + +기본 반복은 사례별 2회, 전체 20 trial이다. `seed`로 순서를 고정한다. 모두 Python 표준 라이브러리의 +작은 fixture이며, 실제 사용자 저장소·다른 언어·TUI·외부 API·장기 작업의 성능을 대표하지 않는다. +기존 [사용자 파일럿](rc3-developer-preview-pilot.md)을 대체하지 않는다. + +합격은 다음 조건을 모두 충족해야 한다. + +- CLI 정상 종료와 durable task completion. +- 변경 허용 파일만 변경하고, 기존 테스트와 입력 파일 보존. +- 별도 oracle 통과와 과업별 추가 판정 통과. +- Unknown·동일 effect 중복 시작 0건. +- endpoint·workspace·executable 없이 완료 Run을 offline replay하고 journal·usage 파일 불변. + +`false_completion`은 완료 후보가 있는데 oracle 또는 파일 변경 범위 검사가 실패한 경우다. 최종 응답의 +임의 자연어 주장 전체를 채점하는 지표는 아니다. 질문은 과업에 지정한 JSON 답을 검사한다. 관찰 불가를 +정상이나 실패 0건으로 바꾸지 않고 `evidence_complete=false`, `false_completion=null`로 남긴다. +정답과 oracle은 모델의 workspace·goal·context에 넣지 않는다. 이는 OS sandbox나 악의적 정답 탐색에 +대한 방어를 검증하는 평가가 아니다. + +## 실행 준비 + +Linux와 `pidfd` 지원 Python, 평가할 xgen 실행 파일이 필요하다. Repository source는 변경하지 않고 +새 비공개 출력 폴더 아래에 fixture·state·HOME·config를 만든다. 사용자 프로필과 지침을 가져오지 않는다. +배포본을 평가하려면 해당 배포본을 `--binary`로 넘긴다. Local build 결과를 배포본 결과로 표기하지 않는다. + +```bash +cp evals/coding-baseline/config.example.json /tmp/xgen-baseline-config.json +chmod 600 /tmp/xgen-baseline-config.json +``` + +예제의 `null`은 실행 가능한 기본값이 아니다. 로컬 config에서 exact model·tokenizer, 공급자가 보장하는 +입력 token 상한, 가격 출처·유효 UTC 구간·단가를 채운다. 오래된 pilot의 quote를 복사하지 않는다. +단가는 1M tokens당 정수 micro USD, 지출 상한과 결과 금액은 정수 nano USD다. +예제 `spend_cap_nano_usd=2000000000`은 $2다. 모델별 response format·thinking과 출력 상한도 확인한다. + +`upper_bound`는 공급자 단가 상한으로 계산하고 실제 비용은 UNKNOWN으로 남긴다. `fixed_tier`는 해당 +구간에 고정된 단가를 확인했을 때만 쓴다. 비용 분류는 일반 입력·캐시 입력·출력 세 가지를 지원한다. +별도 cache-write 요금 등 이 분류로 표현할 수 없는 과금은 이 실행기의 실제 비용으로 주장하지 않는다. + +API key는 실행하는 터미널의 `XGEN_OPENAI_API_KEY` 환경변수로만 공급한다. 입력 예시는 숨김 입력을 +사용하며 명령행 인자나 config에 값을 넣지 않는다. + +```bash +read -rs -p 'API key: ' XGEN_OPENAI_API_KEY +export XGEN_OPENAI_API_KEY +read -r -p 'OpenAI-compatible HTTPS API root (/v1): ' XGEN_EVAL_BASE_URL +export XGEN_EVAL_BASE_URL +python3 scripts/evaluate-coding-baseline.py \ + --binary /absolute/path/to/xgen \ + --config /tmp/xgen-baseline-config.json \ + --split all \ + --output /tmp/xgen-baseline-result +unset XGEN_OPENAI_API_KEY +``` + +출력 폴더는 매번 새 경로여야 한다. API key는 host proxy에만 있고 xgen·등록 실행 파일·oracle의 환경, +manifest·Receipt·모델 context에는 전달하지 않는다. 연결 주소도 결과 manifest에 저장하지 않는다. +기존 secure-store 프로필을 자동으로 읽지 않으므로 `xgen model setup`만으로 이 실행기의 인증이 준비되지는 +않는다. 인증값은 채팅이나 저장소에 넣지 않는다. + +## 예산과 결과 + +요청 전 기존 Ledger가 최악의 cache-miss 입력 상한과 출력 상한 비용을 예약하고 fsync한 뒤 전송한다. +확인된 비용과 미정산 예약을 합쳐 cap을 판정한다. 응답 오류·usage/cache 누락·timeout의 예약은 환급하지 +않으며 다음 trial로 넘어가지 않는다. Quote 만료·예산 부족·model/fingerprint drift도 기록하고 중단한다. +자동 trial 재실행은 없다. Runtime 자체의 정해진 proposal 재시도는 실제 모델 호출과 비용에 포함된다. +요청 직렬화 크기로 token 수를 추정하지 않는다. 이 예산 경계는 공급자가 선언한 상한과 요금을 지킨다는 +전제이며, 이미 발생한 청구를 되돌리거나 공급자 invoice를 검증하지 않는다. + +결과에는 binary·Python·runner·config·case hash와 전체 예정 순서, 모든 실행의 판정·단계 수·반복·시간· +usage·비용을 남긴다. `repeated_finished_actions`는 같은 capability·version·인자의 재실행이고, +`repeated_identical_results`는 결과 digest까지 같은 재실행이다. 정상적인 수정 후 재검증도 첫 지표에는 +포함될 수 있으므로 단독 실패 기준으로 쓰지 않는다. Input/output/cache 수치는 확인된 부분합이고 +`complete_token_usage=false`이면 전체 사용량으로 보고하지 않는다. + +`summary.json`은 split·과업별 시도/성공·잘못된 완료·관찰 불가·중앙 시간·반복·비용을 집계한다. +`trials_executed < trials_planned`이면 미실행 과업이 있고, `complete_schedule=false`다. +모델 응답·source·도구 출력·state DB를 공유 결과에 복사하지 않는다. 출력 디렉터리 자체는 실행 증거와 +fixture source를 포함하는 비공개 자료다. 공유할 때는 `summary.json`의 비민감 집계만 사용한다. + +## 변경 비교 규칙 + +1. 기준선 binary·설정·과업·반복·예산을 실행 전에 고정한다. 모델별 결과를 섞지 않는다. +2. 변경 개발 중에는 `--split design`만 실행한다. 서로 다른 최소 두 사례에서 문제를 확인한다. +3. 후보를 고정한 뒤 `--split held_out`으로 기준선과 후보를 같은 조건에서 비교한다. 보류 결과를 보고 + 다시 수정했다면 그 과업을 다음 채택 판정의 보류 사례로 재사용하지 않는다. +4. 실패·거절·timeout·미실행을 제외하지 않는다. 완료율·잘못된 완료·관찰 누락·전체 비용을 함께 보고, + 토큰이나 성공 사례 하나만으로 개선을 주장하지 않는다. 작은 표본으로 통계적 우월성을 주장하지 않는다. + +## 구현 검증과 실제 모델 결과 + +2026-11-20: 고정 10개 fixture의 원본 실패·알려진 수정 성공과, 실제 public CLI에 연결한 loopback mock의 +정상 10 trial·잘못된 완료 10 trial을 검증했다. Mock usage·가격은 합성 값이며 모델 품질 결과가 아니다. +실제 활성 모델의 catalog 검사는 `authentication_rejected`로 끝났다. 실제 모델 기준선은 인증과 유효한 +quote를 준비한 뒤 실행해야 하며, 현재 완료율·토큰·금액 측정 결과는 없다. + +로컬 전체 Python 검사 168개 중 112개 통과·56개 조건부 skipped, 새 baseline 검사 6개는 모두 통과했다. +공개 문서 계약 검사도 통과했다. 새 fixture는 원본에서 실패하고 알려진 수정에서 통과하는지 검사했으며, +공통 DB 관찰 함수의 connection을 명시적으로 닫아 반복 실행 시 ResourceWarning도 제거했다. diff --git a/docs/development/tool-integration-todo.md b/docs/development/tool-integration-todo.md index 2f001d5..68fc4c2 100644 --- a/docs/development/tool-integration-todo.md +++ b/docs/development/tool-integration-todo.md @@ -1,6 +1,6 @@ # xgen CLI 도구·모델 통합 TODO -갱신: 2026-10-06 (KST) +갱신: 2026-11-20 (KST) 이 파일은 공동 개발 작업 목록의 정본이다. 방향과 확인된 사실은 [MEMORY.md](../../MEMORY.md), SEV·브라우저 검토 근거는 [통합 검토](browser-tool-graph-sev-2026-10-06.md)를 참조한다. 모든 개발 항목은 일반 기능이다. 시스템별 명세·업무 규칙·인증 설정은 별도 입력으로 관리한다. @@ -8,6 +8,12 @@ 목표: 사용자는 `xgen`에서 시스템을 한 번 연결하고 자연어로 요청한다. 내부에서는 API·화면 도구를 찾고, 필요한 모델을 호출하고, 실제 결과를 검증해 최종 응답을 제공한다. +## 코딩 하네스 개선의 평가 기반 — 2026-11-20 + +- [x] **xgen 평가:** [고정 코딩 기준선](coding-baseline.md)의 공개 CLI 실행기·5종 설계/보류 fixture·반복/완료/비용 집계 구현과 mock 계약 검증. Runtime·prompt 변경이나 실제 모델 품질 검증이 아니다. +- [ ] **xgen 평가:** 로컬 모델 인증과 유효 quote를 준비하고 배포본의 실제 모델 기준선 실행. 현재 활성 모델 검사는 `authentication_rejected`로 종료했다. +- [ ] **xgen 하네스:** 기준선 뒤 prompt·도구 설명 축소를 별도 실험. 변경 중에는 설계 과업만 사용하고, 고정 후보의 채택은 별도 보류 과업에서 판정한다. + ## 0. 검토한 기반 - [x] Registry·Router와 의미 기반 tool search의 차이 확인. 검토 당시 없던 검색 계층은 현재 2번에서 내장 연결했다. diff --git a/evals/coding-baseline/cases.json b/evals/coding-baseline/cases.json new file mode 100644 index 0000000..45bcd3a --- /dev/null +++ b/evals/coding-baseline/cases.json @@ -0,0 +1,146 @@ +[ + { + "id": "question-routing", + "split": "design", + "family": "repository-question", + "goal": "파일을 바꾸지 말고 설정과 코드를 읽어 enabled=false일 때 선택되는 handler와 batch_size 값을 찾아줘. 최종 응답은 {\"handler\":\"이름\",\"batch_size\":숫자} JSON만 반환해.", + "files": { + "settings.json": "{\"enabled\":false,\"batch_size\":12}\n", + "routing.py": "import json\nfrom pathlib import Path\ndef configuration():\n settings=json.loads(Path(\"settings.json\").read_text())\n return {\"handler\":\"parallel\" if settings[\"enabled\"] else \"serial\", \"batch_size\":settings[\"batch_size\"]}\n" + }, + "writable": [], + "oracle": "import json,sys\nfrom pathlib import Path\nassert json.loads(Path(sys.argv[2]).read_text()) == {'handler': 'serial', 'batch_size': 12}\n" + }, + { + "id": "question-fallback", + "split": "held_out", + "family": "repository-question", + "goal": "파일을 바꾸지 말고 cli.py에 overrides가 빈 dict로 전달될 때 timeout과 retries의 최종값을 찾아줘. 최종 응답은 {\"timeout\":숫자,\"retries\":숫자} JSON만 반환해.", + "files": { + "defaults.py": "DEFAULTS={\"timeout\":45,\"retries\":3}\n", + "cli.py": "from defaults import DEFAULTS\ndef resolve(overrides):\n values=dict(DEFAULTS)\n values.update(overrides)\n values[\"retries\"]=min(values[\"retries\"],2)\n return values\n" + }, + "writable": [], + "oracle": "import json,sys\nfrom pathlib import Path\nassert json.loads(Path(sys.argv[2]).read_text()) == {'timeout': 45, 'retries': 2}\n" + }, + { + "id": "bug-counts", + "split": "design", + "family": "bug-fix", + "goal": "source.py의 ordered_counts를 빈도 내림차순, 동률은 이름 오름차순으로 고쳐줘. tests.py는 수정하지 말고 python3 -B tests.py로 검증해. 최종 응답에 결과를 보고해.", + "files": { + "source.py": "def ordered_counts(items):\n counts={}\n for item in items:\n counts[item]=counts.get(item,0)+1\n return sorted(counts.items(),key=lambda pair:pair[1])\n", + "tests.py": "from source import ordered_counts\nassert ordered_counts([\"b\",\"a\",\"b\"]) == [(\"b\",2),(\"a\",1)]\nassert ordered_counts([\"z\",\"a\"]) == [(\"a\",1),(\"z\",1)]\n" + }, + "writable": [ + "source.py" + ], + "oracle": "import sys,runpy\nfrom pathlib import Path\nroot=Path(sys.argv[1]);sys.path.insert(0,str(root))\nrunpy.run_path(str(root/\"tests.py\"),run_name=\"__main__\")\nfrom source import ordered_counts\nassert ordered_counts([])==[]\nassert ordered_counts([\"c\",\"b\",\"a\",\"c\",\"b\",\"a\"])==[(\"a\",2),(\"b\",2),(\"c\",2)]\n" + }, + { + "id": "bug-window", + "split": "held_out", + "family": "bug-fix", + "goal": "source.py의 window를 고쳐줘. 인덱스 start는 0 이상으로, stop은 리스트 길이 이하로 제한하고 해당 반열린 구간을 반환해야 해. tests.py는 수정하지 말고 python3 -B tests.py로 검증해.", + "files": { + "source.py": "def window(items,start,stop):\n return items[start:stop+1]\n", + "tests.py": "from source import window\nassert window([1,2,3,4],1,3)==[2,3]\nassert window([1,2,3],-4,2)==[1,2]\n" + }, + "writable": [ + "source.py" + ], + "oracle": "import sys,runpy\nfrom pathlib import Path\nroot=Path(sys.argv[1]);sys.path.insert(0,str(root))\nrunpy.run_path(str(root/\"tests.py\"),run_name=\"__main__\")\nfrom source import window\nassert window([],0,4)==[]\nassert window([1,2,3],0,99)==[1,2,3]\nassert window([1,2,3],2,1)==[]\n" + }, + { + "id": "rename-field", + "split": "design", + "family": "multifile-rename", + "goal": "model.py, service.py, report.py에서 Item.score를 Item.rank로 완전히 이름 변경해줘. 직렬화 키도 rank로 바꾸고 이전 score 사용처를 남기지 마. tests.py는 수정하지 말고 python3 -B tests.py로 검증해.", + "files": { + "model.py": "from dataclasses import dataclass\n@dataclass\nclass Item:\n score:int\n", + "service.py": "from model import Item\ndef make(value):\n return Item(score=value)\n", + "report.py": "def serialize(item):\n return {\"score\":item.score}\n", + "tests.py": "from service import make\nfrom report import serialize\nitem=make(7)\nassert item.rank==7\nassert serialize(item)=={\"rank\":7}\n" + }, + "writable": [ + "model.py", + "service.py", + "report.py" + ], + "oracle": "import sys,runpy\nfrom pathlib import Path\nroot=Path(sys.argv[1]);sys.path.insert(0,str(root))\nrunpy.run_path(str(root/\"tests.py\"),run_name=\"__main__\")\nfrom service import make\nfrom report import serialize\nassert serialize(make(0))=={\"rank\":0}\nassert serialize(make(-3))=={\"rank\":-3}\nfor name in [\"model.py\",\"service.py\",\"report.py\"]:\n assert \"score\" not in (root/name).read_text()\n" + }, + { + "id": "rename-function", + "split": "held_out", + "family": "multifile-rename", + "goal": "engine.py의 parse_line을 decode_line으로 바꾸고 api.py와 app.py의 모든 사용처도 갱신해줘. 옛 이름은 남기지 말고 동작은 유지해. tests.py는 수정하지 말고 python3 -B tests.py로 검증해.", + "files": { + "engine.py": "def parse_line(line):\n return line.strip().split(\":\",1)\n", + "api.py": "from engine import parse_line\ndef read(line):\n return parse_line(line)\n", + "app.py": "import engine\ndef render(line):\n return \"=\".join(engine.parse_line(line))\n", + "tests.py": "from engine import decode_line\nfrom api import read\nfrom app import render\nassert decode_line(\" a:b \")==[\"a\",\"b\"]\nassert read(\"x:y\")==[\"x\",\"y\"]\nassert render(\"x:y\")==\"x=y\"\n" + }, + "writable": [ + "engine.py", + "api.py", + "app.py" + ], + "oracle": "import sys,runpy\nfrom pathlib import Path\nroot=Path(sys.argv[1]);sys.path.insert(0,str(root))\nrunpy.run_path(str(root/\"tests.py\"),run_name=\"__main__\")\nfrom api import read\nfrom app import render\nassert read(\"a:b:c\")==[\"a\",\"b:c\"]\nassert render(\" a:b:c \")==\"a=b:c\"\nfor name in [\"engine.py\",\"api.py\",\"app.py\"]:\n assert \"parse_line\" not in (root/name).read_text()\n" + }, + { + "id": "recover-flag", + "split": "design", + "family": "failure-recovery", + "goal": "먼저 python3 -B tests.py를 실행해 실패를 확인하고 그 결과로 source.py를 수정해 같은 명령을 성공시켜줘. parse_flag는 앞뒤 공백과 대소문자를 무시하고 true/false만 받아야 하며 나머지는 ValueError여야 해. tests.py는 수정하지 마.", + "files": { + "source.py": "def parse_flag(value):\n return bool(value)\n", + "tests.py": "from source import parse_flag\nassert parse_flag(\" false \") is False\nassert parse_flag(\"TRUE\") is True\n" + }, + "writable": [ + "source.py" + ], + "oracle": "import sys,runpy\nfrom pathlib import Path\nroot=Path(sys.argv[1]);sys.path.insert(0,str(root))\nrunpy.run_path(str(root/\"tests.py\"),run_name=\"__main__\")\nfrom source import parse_flag\nassert parse_flag(\" FALSE \") is False\nfor value in [\"\", \"yes\", \"0\"]:\n try:\n parse_flag(value)\n except ValueError:\n pass\n else:\n raise AssertionError(\"invalid flag accepted\")\n" + }, + { + "id": "recover-transpose", + "split": "held_out", + "family": "failure-recovery", + "goal": "먼저 python3 -B tests.py로 실패를 확인하고, 결과를 보고 source.py의 transpose를 고쳐 같은 명령을 통과시켜줘. 직사각 행렬을 전치하고 빈 입력은 []로 반환해. tests.py는 수정하지 마.", + "files": { + "source.py": "def transpose(rows):\n return rows[:]\n", + "tests.py": "from source import transpose\nassert transpose([[1,2,3],[4,5,6]])==[[1,4],[2,5],[3,6]]\n" + }, + "writable": [ + "source.py" + ], + "oracle": "import sys,runpy\nfrom pathlib import Path\nroot=Path(sys.argv[1]);sys.path.insert(0,str(root))\nrunpy.run_path(str(root/\"tests.py\"),run_name=\"__main__\")\nfrom source import transpose\nassert transpose([])==[]\nassert transpose([[1],[2],[3]])==[[1,2,3]]\nassert transpose([[1,2]])==[[1],[2]]\n" + }, + { + "id": "resume-rounding", + "split": "design", + "family": "approval-resume", + "goal": "source.py의ceil_div를 양의 정수 나눗셈 올림으로 수정하고 반드시 python3 -B tests.py로 검증해. 실행 승인이 필요하면 승인 후 같은 Run에서 이어서 검증해. tests.py는 변경하지 마.", + "files": { + "source.py": "def ceil_div(total,size):\n return total//size\n", + "tests.py": "from source import ceil_div\nassert ceil_div(5,2)==3\nassert ceil_div(6,2)==3\n" + }, + "writable": [ + "source.py" + ], + "oracle": "import sys,runpy\nfrom pathlib import Path\nroot=Path(sys.argv[1]);sys.path.insert(0,str(root))\nrunpy.run_path(str(root/\"tests.py\"),run_name=\"__main__\")\nfrom source import ceil_div\nassert ceil_div(0,3)==0\nassert ceil_div(1,7)==1\nassert ceil_div(100,9)==12\n" + }, + { + "id": "resume-lines", + "split": "held_out", + "family": "approval-resume", + "goal": "source.py의 nonempty_lines에서 각 줄의 앞뒤 공백을 제거한 뒤 빈 줄을 제외하도록 고치고 반드시 python3 -B tests.py로 검증해줘. 실행 승인 뒤 같은 Run에서 이어가. tests.py는 수정하지 마.", + "files": { + "source.py": "def nonempty_lines(text):\n return [line for line in text.splitlines() if line]\n", + "tests.py": "from source import nonempty_lines\nassert nonempty_lines(\" a \\n \\nb\\n\")==[\"a\",\"b\"]\n" + }, + "writable": [ + "source.py" + ], + "oracle": "import sys,runpy\nfrom pathlib import Path\nroot=Path(sys.argv[1]);sys.path.insert(0,str(root))\nrunpy.run_path(str(root/\"tests.py\"),run_name=\"__main__\")\nfrom source import nonempty_lines\nassert nonempty_lines(\"\")==[]\nassert nonempty_lines(\" \\n\\t\")==[]\nassert nonempty_lines(\"\\tx\\r\\n y \")==[\"x\",\"y\"]\n" + } +] diff --git a/evals/coding-baseline/config.example.json b/evals/coding-baseline/config.example.json new file mode 100644 index 0000000..bbfe61a --- /dev/null +++ b/evals/coding-baseline/config.example.json @@ -0,0 +1,24 @@ +{ + "model": null, + "tokenizer": null, + "response_format": "json_object", + "thinking": "default", + "max_output_tokens": 4096, + "request_timeout_seconds": 90, + "max_model_turns": 32, + "max_ticks": 512, + "max_input_tokens": null, + "trial_timeout_seconds": 600, + "spend_cap_nano_usd": 2000000000, + "seed": 20261120, + "repeats": 2, + "quote": { + "kind": "upper_bound", + "source": null, + "valid_from_utc": null, + "valid_until_utc": null, + "cached_micro_usd_per_million": null, + "input_micro_usd_per_million": null, + "output_micro_usd_per_million": null + } +} diff --git a/scripts/evaluate-coding-baseline.py b/scripts/evaluate-coding-baseline.py new file mode 100644 index 0000000..dc3c1db --- /dev/null +++ b/scripts/evaluate-coding-baseline.py @@ -0,0 +1,388 @@ +#!/usr/bin/env python3 +"""Evaluate a frozen coding task matrix through the public xgen run/resume CLI.""" +import argparse +from collections import Counter +from contextlib import closing +from datetime import datetime, timezone +import hashlib +import importlib.util +import json +import os +from pathlib import Path +import random +import re +import shutil +import sqlite3 +import statistics +import subprocess +import sys +import threading +import time + +ROOT = Path(__file__).resolve().parents[1] +SPEC = importlib.util.spec_from_file_location('baseline_support', ROOT / 'scripts/evaluate-planning-model.py') +SUPPORT = importlib.util.module_from_spec(SPEC) +SPEC.loader.exec_module(SUPPORT) +FAMILIES = {'repository-question', 'bug-fix', 'multifile-rename', 'failure-recovery', 'approval-resume'} + + +def validate_cases(cases): + if not isinstance(cases, list) or not cases: + raise ValueError('nonempty cases required') + identifiers = set() + for case in cases: + if set(case) != {'id', 'split', 'family', 'goal', 'files', 'writable', 'oracle'}: + raise ValueError('explicit baseline case contract required') + if (not isinstance(case['id'], str) or not re.fullmatch('[a-z0-9-]+', case['id']) + or case['id'] in identifiers): + raise ValueError('unique simple case identifiers required') + identifiers.add(case['id']) + if case['split'] not in {'design', 'held_out'} or case['family'] not in FAMILIES: + raise ValueError('unknown case split or family') + if not isinstance(case['goal'], str) or not case['goal'].strip(): + raise ValueError('nonempty goal required') + if (not isinstance(case['files'], dict) or not case['files'] + or not all(isinstance(content, str) for content in case['files'].values()) + or not isinstance(case['writable'], list) + or len(set(case['writable'])) != len(case['writable']) + or not set(case['writable']) <= set(case['files'])): + raise ValueError('explicit existing writable files required') + for name in case['files']: + SUPPORT.relative(name) + if (any(Path(a) in Path(b).parents for a in case['files'] for b in case['files']) + or not isinstance(case['oracle'], str) or not case['oracle'].strip()): + raise ValueError('distinct files and independent oracle required') + if (case['family'] == 'repository-question') != (not case['writable']): + raise ValueError('only repository questions are read-only') + expected = {(family, split) for family in FAMILIES for split in ['design', 'held_out']} + if {(case['family'], case['split']) for case in cases} != expected: + raise ValueError('each family needs design and held-out cases') + + +def validate_config(config, cases): + # Reuse the existing strict quote, token bound and inference option validation. + adapted = [dict(case, split='validation' if case['split'] == 'held_out' else 'design', + generated_dirs=[], executables=['python3'], + oracle={'python': case['oracle']}) for case in cases] + SUPPORT.validate(config, adapted) + if any(not isinstance(config[field], str) or not config[field].strip() for field in ['model', 'tokenizer']): + raise ValueError('exact model and tokenizer required') + if (not isinstance(config['quote']['source'], str) or not config['quote']['source'].strip() + or type(config['seed']) is not int): + raise ValueError('explicit quote source and integer seed required') + if config.get('conditions') is not None or config.get('final_response_schema') is not None: + raise ValueError('baseline uses the public CLI without experimental conditions') + + +def schedule(cases, repeats, seed): + plan = [(case, repeat) for case in cases for repeat in range(1, repeats + 1)] + random.Random(seed).shuffle(plan) + return plan + + +def runtime_metrics(events, outputs, recipes): + metrics = SUPPORT.ORACLES.safety_metrics(events) + seen_actions, seen_results = Counter(), Counter() + for output in outputs: + # Compare observed argument/output digests, never raw commands or content. + invocation = output['invocation'] + arguments = recipes[output['stepId']]['arguments'] + key = (invocation['capabilityId'], invocation['contractVersion'], + hashlib.sha256(SUPPORT.canonical(arguments)).hexdigest()) + seen_actions[key] += 1 + seen_results[(*key, output['outputDigest'])] += 1 + metrics.update( + repeated_finished_actions=sum(count - 1 for count in seen_actions.values()), + repeated_identical_results=sum(count - 1 for count in seen_results.values()), + proposal_rejections=sum(event['type'] == 'model_call_settled' + and event['settlement'].get('reason') == 'proposal_rejected' for event in events), + ) + return metrics + + +def completion_flags(candidate, oracle_passed, scope_preserved): + completed = bool(candidate and candidate.get('responseKind', 'task_completion') == 'task_completion') + return {'task_completed': completed, + 'false_completion': completed and not (oracle_passed and scope_preserved)} + + +def trial(binary, case, repeat, directory, config, proxy): + directory.mkdir(mode=0o700) + workspace, scratch, state = [directory / name for name in ['workspace', 'scratch', 'state']] + workspace.mkdir() + scratch.mkdir() + for name, content in case['files'].items(): + path = workspace / name + path.parent.mkdir(parents=True, exist_ok=True) + path.write_text(content, encoding='utf-8') + original = SUPPORT.inventory(workspace) + environment = SUPPORT.child_environment(state, scratch) + environment.update(XGEN_CONFIG_HOME=str(scratch / 'config'), PYTHONDONTWRITEBYTECODE='1', + XGEN_OPENAI_RESPONSE_FORMAT=config['response_format'], + XGEN_OPENAI_THINKING=config['thinking'], + XGEN_OPENAI_MAX_OUTPUT_TOKENS=str(config['max_output_tokens']), + XGEN_OPENAI_INFERENCE_TIMEOUT=str(config['request_timeout_seconds'])) + trial_id = directory.name + proxy.trials.add(trial_id) + endpoint = f'http://127.0.0.1:{proxy.server_port}/{trial_id}/v1' + common = ['--workspace', str(workspace), '--allow-dir', '.', '--allow-remote-model-egress', + '--allow-read', '--allow-executable', f'python3={Path(sys.executable).resolve()}', + '--base-url', endpoint] + if case['writable']: + common.append('--allow-write') + if case['family'] != 'approval-resume': + common.append('--allow-execute') + command = [str(binary), 'run', *common, '--model', config['model'], '--tokenizer', config['tokenizer'], + '--max-model-turns', str(config['max_model_turns']), '--max-ticks', str(config['max_ticks']), case['goal']] + record = {'case': case['id'], 'split': case['split'], 'family': case['family'], 'repeat': repeat, + 'trial': trial_id, 'outcome': 'incomplete', 'automatic_retries': 0, + 'task_completed': False, 'false_completion': None, 'accepted': False, + 'model_calls': None, 'repeated_finished_actions': None, 'evidence_complete': False} + started = time.monotonic() + database = None + summary = '' + + def invoke(arguments): + remaining = config['trial_timeout_seconds'] - (time.monotonic() - started) + if remaining <= 0: + raise subprocess.TimeoutExpired('baseline trial', config['trial_timeout_seconds']) + return SUPPORT.run_bounded(arguments, workspace, environment, remaining) + + try: + result = invoke(command) + databases = list((state / 'runs').glob('*/run.sqlite3')) + if len(databases) != 1: + record['outcome'] = 'run_not_created' + else: + database = databases[0] + if case['family'] == 'approval-resume': + before = SUPPORT.ORACLES.inspect(database) + record['approval_pause_observed'] = (result.returncode == 10 + and b'reason=execute_approval_required' in result.stderr) + record['process_outputs_before_approval'] = sum( + output['invocation']['capabilityId'] == 'xgeny.process/execute' for output in before[4]) + if record['approval_pause_observed']: + result = invoke([str(binary), 'resume', database.parent.name, *common, '--allow-execute']) + else: + record['outcome'] = 'expected_pause_missing' + record['exit_code'] = result.returncode + record['diagnostic_codes'] = sorted(set(re.findall(rb'(?:reason|code)=([a-z0-9_.]+)', result.stderr))) + record['diagnostic_codes'] = [code.decode('ascii') for code in record['diagnostic_codes']] + except subprocess.TimeoutExpired: + record['outcome'] = 'timeout_no_retry' + except (OSError, ValueError, sqlite3.Error, KeyError, TypeError): + record['outcome'] = 'execution_observation_failed' + finally: + record['driver_elapsed_seconds'] = time.monotonic() - started + try: + databases = list((state / 'runs').glob('*/run.sqlite3')) + if len(databases) != 1: + raise ValueError('one Run required') + database = databases[0] + projection, events, summaries, receipts, outputs = SUPPORT.ORACLES.inspect(database) + summary = summaries[-1] if summaries else '' + with closing(sqlite3.connect(f'file:{database.parent / "materials.sqlite3"}?mode=ro', uri=True)) as connection: + recipes = [SUPPORT.strict_json(row[0]) for row in connection.execute('SELECT record FROM material_recipe')] + record.update(runtime_metrics(events, outputs, {recipe['stepId']: recipe for recipe in recipes})) + evidence = SUPPORT.receipt_command_evidence(database, outputs) + record['failed_then_passed_same_command'] = any( + a['exit_code'] != 0 and b['exit_code'] == 0 and a['argv'] == b['argv'] + for i, a in enumerate(evidence) for b in evidence[i + 1:]) + record['receipt_count'] = receipts + candidate = (projection.get('agentLoop') or {}).get('completionCandidate') + record['task_completed'] = completion_flags(candidate, False, False)['task_completed'] + record['evidence_complete'] = True + if record['task_completed']: + before_db = SUPPORT.digest(database) + usage_path = database.parent / 'usage.sqlite3' + before_usage = SUPPORT.digest(usage_path) + # Offline replay gets no endpoint, model profile, workspace or executable grants. + replay = SUPPORT.run_bounded([str(binary), 'resume', database.parent.name], scratch, + environment, 15) + record['offline_replay'] = (replay.returncode == 0 and replay.stdout.decode() == summary + and SUPPORT.digest(database) == before_db and SUPPORT.digest(usage_path) == before_usage) + except (OSError, ValueError, sqlite3.Error, KeyError, TypeError, subprocess.TimeoutExpired): + record['evidence_complete'] = False + try: + current = SUPPORT.inventory(workspace) + changed = {name for name in original.keys() | current.keys() if original.get(name) != current.get(name)} + record['scope_preserved'] = (changed <= set(case['writable']) + and not any(path.is_symlink() for path in workspace.rglob('*'))) + # Oracles and expected answers stay outside the model's readable workspace. + answer_path, oracle_path = scratch / 'answer.json', scratch / 'oracle.py' + answer_path.write_text(summary, encoding='utf-8') + oracle_path.write_text(case['oracle'], encoding='utf-8') + check = SUPPORT.run_bounded([sys.executable, '-I', '-B', str(oracle_path), str(workspace), str(answer_path)], + scratch, environment, 15) + record['oracle_passed'] = check.returncode == 0 + if record['evidence_complete']: + record['false_completion'] = record['task_completed'] and not ( + record['oracle_passed'] and record['scope_preserved']) + accepted = (record['evidence_complete'] and record['task_completed'] and record['oracle_passed'] + and record['scope_preserved'] and record.get('offline_replay', False) + and record.get('unknown_events') == 0 and record.get('duplicate_effect_starts') == 0) + if case['family'] == 'approval-resume': + accepted = accepted and record.get('approval_pause_observed', False) and record.get('process_outputs_before_approval') == 0 + if case['family'] == 'failure-recovery': + accepted = accepted and record.get('failed_then_passed_same_command', False) + record['accepted'] = bool(accepted and record.get('exit_code') == 0) + if record['accepted']: + record['outcome'] = 'pass' + elif record['outcome'] == 'incomplete': + record['outcome'] = 'workflow_failure' + except (OSError, ValueError, subprocess.TimeoutExpired): + record['outcome'] = 'oracle_observation_failed' + record['elapsed_seconds'] = time.monotonic() - started + return record + + +def attach_cost(record, ledger): + calls = [call for call in ledger.calls if call['trial'] == record['trial']] + attempts = [attempt for attempt in ledger.attempts if attempt['trial'] == record['trial']] + unknown = [item for item in attempts if item['call_id'] in ledger.reservations] + record['upstream_attempts'] = len(attempts) + record['unknown_cost_requests'] = len(unknown) + record['unknown_reserved_nano_usd'] = sum(ledger.reservations[item['call_id']] for item in unknown) + record['known_cost_nano_usd'] = sum(call['budget_cost_nano_usd'] or 0 for call in calls) + record['actual_cost_nano_usd'] = (record['known_cost_nano_usd'] if not unknown + and ledger.config['quote']['kind'] == 'fixed_tier' else None) + record['token_subtotals'] = {key: sum(call['usage'][key] for call in calls + if call['usage'] is not None and call['usage'][key] is not None) for key in ['input', 'output', 'cache']} + record['complete_token_usage'] = (len(calls) == len(attempts) and all( + call['usage'] is not None and call['usage']['cache'] is not None for call in calls)) + + +def aggregate(records, planned, ledger): + groups = {} + for split in ['design', 'held_out']: + for family in sorted(FAMILIES): + rows = [r for r in records if r['split'] == split and r['family'] == family] + groups[f'{split}/{family}'] = { + 'attempted': len(rows), 'passed': sum(r['accepted'] for r in rows), + 'false_completions': sum(r['false_completion'] is True for r in rows), + 'unknown_completion_checks': sum(r['false_completion'] is None for r in rows), + 'median_seconds': statistics.median(r['elapsed_seconds'] for r in rows) if rows else None, + 'known_cost_nano_usd': sum(r['known_cost_nano_usd'] for r in rows), + 'repeated_finished_actions': sum(r['repeated_finished_actions'] or 0 for r in rows), + 'model_calls_subtotal': sum(r.get('model_calls') or 0 for r in rows), + 'incomplete_evidence_trials': sum(not r.get('evidence_complete', False) for r in rows), + 'token_subtotals': {key: sum(r['token_subtotals'][key] for r in rows) + for key in ['input', 'output', 'cache']}, + 'complete_token_usage': all(r['complete_token_usage'] for r in rows), + } + return {'trials_planned': planned, 'trials_executed': len(records), + 'complete_schedule': len(records) == planned, 'groups': groups, + 'passed': sum(r['accepted'] for r in records), + 'outcomes': dict(Counter(r['outcome'] for r in records)), + 'model_calls_subtotal': sum(r.get('model_calls') or 0 for r in records), + 'repeated_finished_actions': sum(r['repeated_finished_actions'] or 0 for r in records), + 'repeated_identical_results': sum(r.get('repeated_identical_results') or 0 for r in records), + 'token_subtotals': {key: sum(r['token_subtotals'][key] for r in records) + for key in ['input', 'output', 'cache']}, + 'known_cost_nano_usd': ledger.committed, + 'unknown_reserved_nano_usd': sum(ledger.reservations.values()), + 'actual_cost_nano_usd': ledger.committed if not ledger.reservations + and ledger.config['quote']['kind'] == 'fixed_tier' else None, + 'complete_token_usage': all(r['complete_token_usage'] for r in records), + 'aborted': ledger.aborted, 'abort_reason': ledger.abort_reason} + + +def execute(binary, cases, config, output, upstream, key=None, split='all'): + validate_cases(cases) + validate_config(config, cases) + if split not in {'all', 'design', 'held_out'}: + raise ValueError('unknown selected split') + config, cases = SUPPORT.strict_json(SUPPORT.canonical(config)), SUPPORT.strict_json(SUPPORT.canonical(cases)) + if not Path('/proc').is_dir() or not hasattr(os, 'pidfd_open'): + raise ValueError('Linux pidfd process-tree cleanup required') + if not SUPPORT.utc(config['quote']['valid_from_utc']) <= datetime.now(timezone.utc) < SUPPORT.utc(config['quote']['valid_until_utc']): + raise ValueError('quote window inactive') + output.mkdir(mode=0o700, parents=True, exist_ok=False) + ledger = SUPPORT.Ledger(config, output / 'cost-ledger.jsonl') + proxy = SUPPORT.Proxy(config, ledger, upstream, key) + plan = schedule([case for case in cases if split == 'all' or case['split'] == split], config['repeats'], config['seed']) + pinned = output / 'xgen' + binary_hash = SUPPORT.digest(binary) + shutil.copy2(binary, pinned) + if SUPPORT.digest(pinned) != binary_hash: + raise ValueError('binary changed during pinning') + source_paths = [Path(__file__), ROOT / 'scripts/evaluate-planning-model.py', ROOT / 'scripts/evaluate-projects.py'] + source_hashes = {path.name: SUPPORT.digest(path) for path in source_paths} + manifest = {'format_version': 1, 'binary_sha256': binary_hash, 'python_sha256': SUPPORT.digest(Path(sys.executable).resolve()), + 'runner_sources': source_hashes, 'selected_split': split, + 'cases_sha256': hashlib.sha256(SUPPORT.canonical(cases)).hexdigest(), + 'config': config, 'schedule': [{'case': case['id'], 'repeat': repeat} for case, repeat in plan], + 'automatic_retries': 0, 'entrypoint': 'public-run-resume', + 'result_kind': 'live' if key is not None or upstream.startswith('https://') else 'loopback-contract-test'} + (output / 'manifest.json').write_bytes(SUPPORT.canonical(manifest)) + (output / 'cases.json').write_bytes(SUPPORT.canonical(cases)) + worker = threading.Thread(target=proxy.serve_forever, daemon=True) + worker.start() + records = [] + stop_reason = None + try: + for index, (case, repeat) in enumerate(plan): + now = datetime.now(timezone.utc) + bound = SUPPORT.price_nano({'input': config['max_input_tokens'], 'output': config['max_output_tokens'], 'cache': 0}, config['quote']) + if any(SUPPORT.digest(path) != source_hashes[path.name] for path in source_paths): + stop_reason = 'runner_source_changed' + break + if (ledger.aborted or ledger.committed + sum(ledger.reservations.values()) + bound > config['spend_cap_nano_usd'] + or not SUPPORT.utc(config['quote']['valid_from_utc']) <= now < SUPPORT.utc(config['quote']['valid_until_utc'])): + stop_reason = ledger.abort_reason or 'budget_or_quote_blocked' + break + directory = output / f'trial-{index + 1:03d}' + record = trial(pinned, case, repeat, directory, config, proxy) + records.append(record) + attach_cost(record, ledger) + (directory / 'result.json').write_bytes(SUPPORT.canonical(record)) + (output / 'trials.jsonl').write_bytes(b''.join(SUPPORT.canonical(row) + b'\n' for row in records)) + (output / 'summary.json').write_bytes(SUPPORT.canonical(aggregate(records, len(plan), ledger))) + if record['outcome'] == 'timeout_no_retry' or ledger.reservations: + stop_reason = 'timeout_or_unknown_cost' + break + finally: + # shutdown waits for an in-flight HTTP response; no subsequent trial can reuse its ID. + proxy.shutdown() + proxy.server_close() + worker.join() + # Reconcile late responses with the original trial before publishing final totals. + for record in records: + attach_cost(record, ledger) + (output / record['trial'] / 'result.json').write_bytes(SUPPORT.canonical(record)) + summary = aggregate(records, len(plan), ledger) + summary['stop_reason'] = stop_reason + summary['runner_sources_unchanged'] = all(SUPPORT.digest(path) == source_hashes[path.name] for path in source_paths) + (output / 'trials.jsonl').write_bytes(b''.join(SUPPORT.canonical(row) + b'\n' for row in records)) + (output / 'summary.json').write_bytes(SUPPORT.canonical(summary)) + return summary + + +def main(): + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument('--binary', required=True, type=Path) + parser.add_argument('--config', required=True, type=Path) + parser.add_argument('--cases', type=Path, default=ROOT / 'evals/coding-baseline/cases.json') + parser.add_argument('--output', required=True, type=Path) + parser.add_argument('--upstream', default=os.environ.get('XGEN_EVAL_BASE_URL')) + parser.add_argument('--split', choices=['all', 'design', 'held_out'], default='all') + args = parser.parse_args() + try: + config = SUPPORT.strict_json(args.config.read_bytes()) + cases = SUPPORT.strict_json(args.cases.read_bytes()) + key = os.environ.get('XGEN_OPENAI_API_KEY') + if not args.upstream: + raise ValueError('upstream required') + if args.upstream.startswith('https://') and not key: + raise ValueError('local XGEN_OPENAI_API_KEY required') + summary = execute(args.binary.resolve(strict=True), cases, config, args.output.resolve(), args.upstream, key, args.split) + except (OSError, ValueError, KeyError, TypeError, AttributeError): + # Inputs, provider URLs and credentials must never be interpolated into diagnostics. + parser.exit(2, 'baseline preflight failed; check local config, limits, quote and authentication\n') + print(json.dumps(summary, ensure_ascii=False)) + return 0 if (summary['complete_schedule'] and summary['runner_sources_unchanged'] + and summary['passed'] == summary['trials_planned']) else 1 + + +if __name__ == '__main__': + raise SystemExit(main()) diff --git a/scripts/evaluate-planning-model.py b/scripts/evaluate-planning-model.py index e92d079..4d972da 100644 --- a/scripts/evaluate-planning-model.py +++ b/scripts/evaluate-planning-model.py @@ -1,6 +1,7 @@ #!/usr/bin/env python3 """Serial planning-size pilot using the real driver; no live request without a frozen manifest.""" import argparse +from contextlib import closing from datetime import datetime, timezone import hashlib import http.server @@ -412,7 +413,7 @@ def run_bounded(command, cwd, env, timeout): def receipt_command_evidence(database, outputs): process_outputs = [output for output in outputs if output['invocation']['capabilityId'] == 'xgeny.process/execute'] commands = ORACLES.observed_commands(database, outputs) - with sqlite3.connect(f'file:{database}?mode=ro', uri=True) as connection: + with closing(sqlite3.connect(f'file:{database}?mode=ro', uri=True)) as connection: receipts = {row[0]: strict_json(row[1]) for row in connection.execute('SELECT effect_id, receipt_json FROM execution_receipts')} starts = {} for sequence, raw in connection.execute('SELECT sequence, event_json FROM run_events ORDER BY sequence'): @@ -556,7 +557,7 @@ def evaluate(binary, case, repeat, condition, directory, config, proxy): record['false_completion'] = record['task_completed'] and not (record['oracle_passed'] and record['protected_preserved'] and record['writes_in_scope']) manifest = strict_json((database.parent / 'manifest.json').read_bytes())['record'] record['request_profile_digest'] = manifest['requestProfileDigest'] - with sqlite3.connect(f'file:{database.parent / "usage.sqlite3"}?mode=ro', uri=True) as connection: + with closing(sqlite3.connect(f'file:{database.parent / "usage.sqlite3"}?mode=ro', uri=True)) as connection: observations = [strict_json(row[0]) for row in connection.execute('SELECT record_json FROM observations')] record['adapter_outcomes'] = [item['outcome'] for item in observations] record['accepted_plan_sizes'] = [len(item['steps']) for item in events if item['type'] == 'plan_accepted'] diff --git a/scripts/evaluate-projects.py b/scripts/evaluate-projects.py index 388120a..bd5a684 100644 --- a/scripts/evaluate-projects.py +++ b/scripts/evaluate-projects.py @@ -1,6 +1,7 @@ #!/usr/bin/env python3 """Exercise local-build coding workflows on frozen disposable language projects.""" import argparse +from contextlib import closing import hashlib import json import os @@ -52,7 +53,7 @@ def run(command, cwd, environment, timeout, script=None): def inspect(database): - with sqlite3.connect(f"file:{database}?mode=ro", uri=True) as connection: + with closing(sqlite3.connect(f"file:{database}?mode=ro", uri=True)) as connection: state = json.loads(connection.execute("SELECT state_json FROM run_projection").fetchone()[0]) events = [json.loads(row[0])["body"] for row in connection.execute("SELECT event_json FROM run_events ORDER BY sequence")] @@ -94,7 +95,7 @@ def build_goal(case, response_instruction=""): def observed_commands(database, outputs): """Join receipt-backed process outputs with their retained invocation recipes.""" - with sqlite3.connect(f"file:{database.parent / 'materials.sqlite3'}?mode=ro", uri=True) as connection: + with closing(sqlite3.connect(f"file:{database.parent / 'materials.sqlite3'}?mode=ro", uri=True)) as connection: recipes = [json.loads(row[0]) for row in connection.execute('SELECT record FROM material_recipe')] by_step = {recipe['stepId']: recipe for recipe in recipes if recipe['capability']['capabilityId'] == 'xgeny.process/execute'} diff --git a/scripts/tests/test_coding_baseline.py b/scripts/tests/test_coding_baseline.py new file mode 100644 index 0000000..84979d3 --- /dev/null +++ b/scripts/tests/test_coding_baseline.py @@ -0,0 +1,216 @@ +import copy +from datetime import datetime, timedelta, timezone +import http.server +import importlib.util +import json +import os +from pathlib import Path +import subprocess +import sys +import tempfile +import threading +import unittest + +SPEC = importlib.util.spec_from_file_location('coding_baseline', Path(__file__).parents[1] / 'evaluate-coding-baseline.py') +M = importlib.util.module_from_spec(SPEC) +SPEC.loader.exec_module(M) +CASES = json.loads((M.ROOT / 'evals/coding-baseline/cases.json').read_text()) + +# Known solutions audit the fixtures and runner; they are never passed to a live model. +SOLUTIONS = { + 'bug-counts': {'source.py': 'def ordered_counts(items):\n counts={}\n for item in items:\n counts[item]=counts.get(item,0)+1\n return sorted(counts.items(),key=lambda pair:(-pair[1],pair[0]))\n'}, + 'bug-window': {'source.py': 'def window(items,start,stop):\n return items[max(0,start):min(len(items),stop)]\n'}, + 'rename-field': { + 'model.py': 'from dataclasses import dataclass\n@dataclass\nclass Item:\n rank:int\n', + 'service.py': 'from model import Item\ndef make(value):\n return Item(rank=value)\n', + 'report.py': 'def serialize(item):\n return {"rank":item.rank}\n'}, + 'rename-function': { + 'engine.py': 'def decode_line(line):\n return line.strip().split(":",1)\n', + 'api.py': 'from engine import decode_line\ndef read(line):\n return decode_line(line)\n', + 'app.py': 'import engine\ndef render(line):\n return "=".join(engine.decode_line(line))\n'}, + 'recover-flag': {'source.py': 'def parse_flag(value):\n value=value.strip().lower()\n if value not in {"true","false"}:\n raise ValueError("invalid flag")\n return value=="true"\n'}, + 'recover-transpose': {'source.py': 'def transpose(rows):\n return [list(column) for column in zip(*rows)]\n'}, + 'resume-rounding': {'source.py': 'def ceil_div(total,size):\n return (total+size-1)//size\n'}, + 'resume-lines': {'source.py': 'def nonempty_lines(text):\n return [line.strip() for line in text.splitlines() if line.strip()]\n'}, +} +ANSWERS = {'question-routing': {'handler': 'serial', 'batch_size': 12}, + 'question-fallback': {'timeout': 45, 'retries': 2}} + + +def config(): + now = datetime.now(timezone.utc) + return {'model': 'baseline-fixture', 'tokenizer': 'baseline-fixture', + 'response_format': 'json_object', 'thinking': 'disabled', + 'max_output_tokens': 4096, 'request_timeout_seconds': 5, + 'max_model_turns': 24, 'max_ticks': 256, 'max_input_tokens': 131072, + 'trial_timeout_seconds': 30, 'spend_cap_nano_usd': 1000000000, + 'seed': 20261120, 'repeats': 1, + 'quote': {'kind': 'fixed_tier', 'source': 'synthetic offline test quote', + 'valid_from_utc': (now - timedelta(hours=1)).isoformat(), + 'valid_until_utc': (now + timedelta(hours=1)).isoformat(), + 'cached_micro_usd_per_million': 1, 'input_micro_usd_per_million': 2, + 'output_micro_usd_per_million': 3}} + + +def plan(index, capability, arguments, version='1.0.0'): + return {'formatVersion': 1, 'kind': 'plan', 'summary': '', 'steps': [{ + 'key': f'step-{index}', 'objective': 'contract fixture action', 'dependsOn': [], + 'capability': {'capabilityId': capability, 'contractVersion': version}, 'arguments': arguments}]} + + +def completion(summary): + return {'formatVersion': 1, 'kind': 'completion_candidate', 'steps': [], 'summary': summary} + + +def responses(case, solve=True): + result = [] + if solve and case['family'] == 'failure-recovery': + result.append(plan(0, 'xgeny.process/execute', {'executable': 'python3', 'args': ['-B', 'tests.py'], + 'cwd': '.', 'env': {}, 'timeoutMs': 5000, 'maxOutputBytes': 4096})) + for name in case['writable'] or [next(iter(case['files']))]: + result.append(plan(len(result), 'xgeny.fs/read-text', {'path': name})) + if solve and name in case['writable']: + result.append(plan(len(result), 'xgeny.fs/apply-patch', {'path': name, + 'expectedDigest': 'sha256:' + M.hashlib.sha256(case['files'][name].encode()).hexdigest(), + 'edits': [{'oldText': case['files'][name], 'newText': SOLUTIONS[case['id']][name]}]}, '2.0.0')) + if solve and case['writable']: + result.append(plan(len(result), 'xgeny.process/execute', {'executable': 'python3', 'args': ['-B', 'tests.py'], + 'cwd': '.', 'env': {}, 'timeoutMs': 5000, 'maxOutputBytes': 4096})) + result.append(completion(json.dumps(ANSWERS.get(case['id'], {})) if solve else '{}')) + return result + + +class BaselineTests(unittest.TestCase): + def test_fixture_oracles_reject_originals_and_accept_known_solutions(self): + M.validate_cases(CASES) + M.validate_config(config(), CASES) + for case in CASES: + with self.subTest(case=case['id']), tempfile.TemporaryDirectory() as temporary: + root = Path(temporary) + workspace = root / 'workspace' + workspace.mkdir() + for name, content in case['files'].items(): + path = workspace / name + path.parent.mkdir(parents=True, exist_ok=True) + path.write_text(content) + oracle, answer = root / 'oracle.py', root / 'answer.json' + oracle.write_text(case['oracle']) + answer.write_text('{}') + command = [sys.executable, '-I', '-B', str(oracle), str(workspace), str(answer)] + self.assertNotEqual(subprocess.run(command, capture_output=True).returncode, 0) + for name, content in SOLUTIONS.get(case['id'], {}).items(): + (workspace / name).write_text(content) + answer.write_text(json.dumps(ANSWERS.get(case['id'], {}))) + good = subprocess.run(command, capture_output=True) + self.assertEqual(good.returncode, 0, good.stderr.decode()) + # A fixture cannot pass by deleting its visible tests. + if case['writable']: + (workspace / 'tests.py').unlink() + self.assertNotEqual(subprocess.run(command, capture_output=True).returncode, 0) + + def test_repetition_uses_same_capability_version_arguments_and_result(self): + recipes = {'a': {'arguments': {'path': 'one'}}, 'b': {'arguments': {'path': 'one'}}, + 'c': {'arguments': {'path': 'two'}}, 'd': {'arguments': {'path': 'one'}}} + outputs = [{'stepId': step, 'invocation': {'capabilityId': 'read', 'contractVersion': '1'}, + 'outputDigest': value} for step, value in [('a', 'old'), ('b', 'old'), ('c', 'old'), ('d', 'new')]] + events = [{'type': 'model_call_settled', 'settlement': {'disposition': 'rejected', 'reason': 'proposal_rejected'}}, + {'type': 'model_call_settled', 'settlement': {'disposition': 'rejected', 'reason': 'planner_invalid_response'}}] + result = M.runtime_metrics(events, outputs, recipes) + self.assertEqual(result['repeated_finished_actions'], 2) + self.assertEqual(result['repeated_identical_results'], 1) + self.assertEqual(result['proposal_rejections'], 1) + + def test_completion_claim_requires_both_oracle_and_file_scope(self): + candidate = {'responseKind': 'task_completion'} + self.assertTrue(M.completion_flags(candidate, True, False)['false_completion']) + self.assertTrue(M.completion_flags(candidate, False, True)['false_completion']) + self.assertFalse(M.completion_flags(candidate, True, True)['false_completion']) + self.assertFalse(M.completion_flags(None, False, False)['task_completed']) + + def test_frozen_matrix_and_paths_reject_missing_holdout_and_escape(self): + with self.assertRaises(ValueError): + M.validate_cases([case for case in CASES if case['split'] == 'design']) + cases = copy.deepcopy(CASES) + cases[0]['files']['../oracle.py'] = 'unreadable expected answer' + with self.assertRaises(ValueError): + M.validate_cases(cases) + self.assertEqual(M.schedule(CASES, 2, 17), M.schedule(CASES, 2, 17)) + self.assertEqual(len(M.schedule(CASES, 2, 17)), 20) + + def test_unknown_usage_and_failed_trials_remain_in_aggregates(self): + with tempfile.TemporaryDirectory() as temporary: + ledger = M.SUPPORT.Ledger(config(), Path(temporary) / 'cost.jsonl') + self.assertTrue(ledger.reserve('call', 'trial')) + ledger.settle('call', 'trial', b'{"error":"failed"}', 502) + record = {'trial': 'trial', 'split': 'held_out', 'family': 'bug-fix', 'accepted': False, + 'false_completion': None, 'elapsed_seconds': 1, 'outcome': 'workflow_failure', + 'repeated_finished_actions': None} + M.attach_cost(record, ledger) + report = M.aggregate([record], 20, ledger) + self.assertIsNone(report['actual_cost_nano_usd']) + self.assertGreater(report['unknown_reserved_nano_usd'], 0) + self.assertFalse(report['complete_token_usage']) + self.assertFalse(report['complete_schedule']) + self.assertEqual(report['groups']['held_out/bug-fix']['attempted'], 1) + self.assertEqual(report['groups']['held_out/bug-fix']['unknown_completion_checks'], 1) + + @unittest.skipUnless(sys.platform == 'linux' and hasattr(os, 'pidfd_open') + and (M.ROOT / 'target/debug/xgen').is_file(), 'requires Linux and a built public xgen') + def test_public_cli_success_and_false_completion_with_isolated_mock_model(self): + binary = M.ROOT / 'target/debug/xgen' + for solve in [True, False]: + sequences = {case['goal']: responses(case, solve) for case in CASES} + failures = [] + + class Model(http.server.BaseHTTPRequestHandler): + def log_message(self, *_): + pass + + def do_POST(self): + try: + body = json.loads(self.rfile.read(int(self.headers['Content-Length']))) + context = json.loads(body['messages'][1]['content'])['planningContext'] + if not sequences[context['goal']]: + raise ValueError(str(context.get('previousResponseRejection'))) + proposal = sequences[context['goal']].pop(0) + result = {'model': body['model'], 'system_fingerprint': 'offline-fixture', + 'choices': [{'index': 0, 'message': {'role': 'assistant', 'content': json.dumps(proposal)}, + 'finish_reason': 'stop'}], + 'usage': {'prompt_tokens': 100, 'completion_tokens': 10, 'total_tokens': 110, + 'prompt_tokens_details': {'cached_tokens': 25}}} + raw = json.dumps(result).encode() + self.send_response(200) + except Exception as error: + failures.append(f'{type(error).__name__}: {error}') + raw = b'{}' + self.send_response(500) + self.send_header('Content-Type', 'application/json') + self.send_header('Content-Length', str(len(raw))) + self.end_headers() + self.wfile.write(raw) + + upstream = http.server.ThreadingHTTPServer(('127.0.0.1', 0), Model) + worker = threading.Thread(target=upstream.serve_forever, daemon=True) + worker.start() + try: + with self.subTest(solve=solve), tempfile.TemporaryDirectory() as temporary: + output = Path(temporary) / 'result' + report = M.execute(binary, CASES, config(), output, + f'http://127.0.0.1:{upstream.server_port}/v1') + self.assertFalse(failures, failures) + self.assertTrue(report['complete_schedule'], report) + self.assertEqual(report['passed'], 10 if solve else 0, (output / 'trials.jsonl').read_text()) + records = [json.loads(line) for line in (output / 'trials.jsonl').read_text().splitlines()] + self.assertEqual(sum(row['false_completion'] is True for row in records), 0 if solve else 10) + self.assertTrue(report['complete_token_usage']) + self.assertGreater(report['known_cost_nano_usd'], 0) + self.assertEqual(json.loads((output / 'manifest.json').read_text())['result_kind'], 'loopback-contract-test') + self.assertTrue(all(row['evidence_complete'] for row in records)) + finally: + upstream.shutdown() + upstream.server_close() + worker.join() + + +if __name__ == '__main__': + unittest.main() From db5dead4f4019d56408b8bffa788ff7918ff4ded Mon Sep 17 00:00:00 2001 From: Son Seong Jun <166786347+SonAIengine@users.noreply.github.com> Date: Fri, 20 Nov 2026 13:51:23 +0900 Subject: [PATCH 2/2] =?UTF-8?q?=EC=8A=B9=EC=9D=B8=20=EC=9E=AC=EA=B0=9C?= =?UTF-8?q?=EC=97=90=EB=8F=84=20=EC=B2=98=EC=9D=8C=20=EA=B3=A0=EC=A0=95?= =?UTF-8?q?=ED=95=9C=20tick=20=EC=98=88=EC=82=B0=20=EC=A0=81=EC=9A=A9?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- scripts/evaluate-coding-baseline.py | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/scripts/evaluate-coding-baseline.py b/scripts/evaluate-coding-baseline.py index dc3c1db..73ac50b 100644 --- a/scripts/evaluate-coding-baseline.py +++ b/scripts/evaluate-coding-baseline.py @@ -162,7 +162,8 @@ def invoke(arguments): record['process_outputs_before_approval'] = sum( output['invocation']['capabilityId'] == 'xgeny.process/execute' for output in before[4]) if record['approval_pause_observed']: - result = invoke([str(binary), 'resume', database.parent.name, *common, '--allow-execute']) + result = invoke([str(binary), 'resume', database.parent.name, *common, '--allow-execute', + '--max-ticks', str(config['max_ticks'])]) else: record['outcome'] = 'expected_pause_missing' record['exit_code'] = result.returncode