openlogos verify

Read test results from a JSONL file, compare against defined test case IDs in logos/resources/test/, validate three layers of coverage, and generate an acceptance report. This is the Gate 3.5 quality check.

Synopsis

openlogos verify
openlogos verify --format json

Must be run from the project root.

Options

Option	Description
--format json	Output machine-readable JSON envelope (includes pre_run and sandbox diagnostics).

Sandbox execution (verify / smoke standardization)

openlogos verify supports sandboxed pre-run execution through logos/logos.config.json:

{
  "verify": {
    "result_path": "logos/resources/verify/test-results.jsonl",
    "pre_run_command": "npm test",
    "sandbox_mode": "auto",
    "sandbox_root": "/private/tmp",
    "sandbox_deny_workspace_write": true
  }
}

Behavior:

• sandbox_mode: "off": keep historical behavior, no isolation.
• sandbox_mode: "auto": prefer isolation; if isolation is unavailable, downgrade with warning and keep command compatibility.
• sandbox_mode: "always": isolation is mandatory; if setup fails or non-whitelisted writes are detected, verify fails.

When sandbox is enabled, pre-run commands only allow result-file write-back (verify.result_path, optional regression_result_path, incremental_result_path) plus report output.

What it checks

The verification runs three layers of validation:

Layer 1: Runtime test results

Reads logos/resources/verify/test-results.jsonl (or a custom path from logos.config.json) and counts pass/fail/skip/uncovered.

Layer 2: Design-time coverage checklist

Parses the “覆盖度校验” (Coverage Validation) checklist from *-test-cases.md files. These are assertions made by AI at test-design time — e.g., “All S02 exception cases are covered.”

Layer 3: Acceptance criteria traceability

Parses the “验收条件追溯” (Acceptance Criteria Traceability) table from test case files. Each acceptance criterion (e.g., S01-AC-01) is linked to specific test case IDs, and the runtime results are checked to confirm they pass.

Gate 3.5 pass criteria

All four conditions must be met:

Condition	Description
Zero failed tests	No test result has status: "fail"
100% coverage	Every ID defined in test specs has a corresponding result
Checklist complete	All - [x] items in the design-time checklist are checked
AC traceability	All acceptance criteria have linked test cases that pass

If any condition fails, the command exits with code 1.

Proposal lifecycle integration

When there is an active change proposal (.openlogos-guard exists), openlogos verify automatically writes a marker file to the proposal directory after the gate result is determined:

Gate result	Marker written
PASS	logos/changes/<slug>/VERIFY_PASS
FAIL	logos/changes/<slug>/VERIFY_FAIL

These markers advance the proposal step:

• VERIFY_PASS → proposal step becomes verify-passed (ready to archive)
• VERIFY_FAIL → proposal step becomes verify-failed (fix and re-run verify)

openlogos status reads these markers to show the correct next action.

Test results format (JSONL)

Each line in test-results.jsonl is a JSON object:

{"id":"UT-S01-01","status":"pass","duration_ms":12,"timestamp":"2026-04-10T10:00:00Z"}
{"id":"UT-S01-02","status":"fail","duration_ms":5,"error":"Expected 201, got 400"}
{"id":"ST-S02-01","status":"skip","scenario":"S02"}

Field	Required	Values	Description
id	Yes	UT-SXX-XX or ST-SXX-XX	Test case ID matching the test spec
status	Yes	pass, fail, skip	Test result
duration_ms	No	number	Execution time
timestamp	No	ISO 8601	When the test ran
error	No	string	Error message (for failed tests)
scenario	No	string	Scenario identifier

If the same id appears multiple times (e.g., from re-runs), the last occurrence wins.

Test case ID detection

The command scans all *-test-cases.md files in logos/resources/test/ for IDs matching the pattern:

UT-S01-01    (Unit Test, Scenario 01, Case 01)
ST-S02-03    (Scenario Test, Scenario 02, Case 03)

The regex: /\b(UT|ST)-S\d{2}-\d{2,3}\b/

Example output (PASS)

🔍 OpenLogos Test Verification

Reading test results: logos/resources/verify/test-results.jsonl
Reading test cases: logos/resources/test/

──────────────────────────────────────────────────
📊 Results Summary
──────────────────────────────────────────────────
  Total defined:  111 cases (79 UT + 32 ST)
  Total executed: 111 cases
  ✅ Passed:      86
  ❌ Failed:       0
  ⏭️  Skipped:     25
──────────────────────────────────────────────────
  Coverage:  100%  (111/111)
  Pass rate: 77%  (86/111)
──────────────────────────────────────────────────

📋 Design-time Coverage (Layer 1)
  Checklist: 8/8 assertions confirmed

🔗 Acceptance Criteria Traceability (Layer 3)
  AC traceability: 16/16 criteria passed

✅ Gate 3.5: PASS

📄 Report: logos/resources/verify/acceptance-report.md

Example output (FAIL)

🔍 OpenLogos Test Verification

Reading test results: logos/resources/verify/test-results.jsonl
Reading test cases: logos/resources/test/

──────────────────────────────────────────────────
📊 Results Summary
──────────────────────────────────────────────────
  Total defined:  111 cases (79 UT + 32 ST)
  Total executed: 109 cases
  ✅ Passed:      84
  ❌ Failed:       2
  ⏭️  Skipped:     23
──────────────────────────────────────────────────
  Coverage:  98%  (109/111)
  Pass rate: 77%  (84/109)
──────────────────────────────────────────────────

❌ Failed cases:
  UT-S02-04  Expected HTTP 302, got 301
  UT-S04-12  Token verification returned false for valid token

⚠️  Uncovered cases (2):
  ST-S03-05
  ST-S03-06

❌ Gate 3.5: FAIL

📄 Report: logos/resources/verify/acceptance-report.md

Acceptance report

The command generates logos/resources/verify/acceptance-report.md containing:

• Summary table — defined/executed/passed/failed/skipped/uncovered/coverage/pass-rate/gate result
• Failed cases — ID and error message for each
• Uncovered cases — IDs not found in test results
• Skipped cases — IDs with status: "skip"
• Design-time coverage — checklist assertion table with ✅/❌
• AC traceability — acceptance criteria → linked test cases → runtime status

JSON output highlights

With --format json, data includes:

• pre_run: execution mode (none / pre_run_command / two_phase), command statuses, result paths, diagnostics and suggestions.
• sandbox: sandbox mode, whether execution was isolated, workspace-write policy, status (pass / warn / fail / skipped), diagnostics and suggestions.

Custom result path

Override the default JSONL path in logos.config.json:

{
  "verify": {
    "result_path": "test-output/results.jsonl"
  }
}

Errors

Error	Cause	Fix
No test results found at ...	JSONL file doesn’t exist	Run your tests first (they should output to test-results.jsonl)
No test case specs found	logos/resources/test/ is empty	Run test design (Phase 3-3a) first
logos/logos.config.json not found	Not in project root	cd to project root
Exit code 1	Gate 3.5 FAIL	Fix failing tests, add missing coverage, or complete the checklist

Related commands

• status — Check if Phase 3-5 (verification) is complete
• launch — Typically run after verify passes, to activate change management