Run Enterprise-Bench L1–L2 with Grok Build + grok-4.5 (Harbor harness) #3

Summary

Track an evaluation of Grok Build (-a grok-build) with model grok-4.5 against Enterprise-Bench/l1-l2-bench on Harbor Hub.

Harness: Harbor (job runner, Docker trials, judge, jobs artifacts)
Agent under test: Grok Build (xAI CLI, installed per trial container)
Dataset: Enterprise-Bench/l1-l2-bench (14 tasks)
Judge: OpenAI GPT-5 via OPENAI_API_KEY (required for all agents)

Why this run

• Add a Grok Build + grok-4.5 data point next to existing Claude Code / DevRev Computer jobs in the README.
• Validate that Harbor's grok-build adapter works end-to-end with Enterprise-Bench MCP servers (crm / pm / file-server).
• Produce submit-ready results per docs/submit-results.md.

Architecture

Harbor Hub dataset (Enterprise-Bench/l1-l2-bench)
        ↓
Harbor job runner (Docker trials, scoring, jobs/)
        ↓  -a grok-build -m grok-4.5
Grok Build agent (CLI installed in each trial container)
        ↓  MCP (host.docker.internal:8011–8013)
Maple Payments synthetic data
        ↓  OPENAI_API_KEY
GPT-5 judge → reward 0.0 | 1.0

Prerequisites

Auth note (important for team)

• Interactive Grok Build on a laptop can use SuperGrok / OAuth (grok login).
• Harbor evals run headless in Docker and the upstream adapter requires XAI_API_KEY.
• Subscription alone is not sufficient for multi-trial container runs.

Known gotchas (from field SKILL + prior local runs)

1. make install before make setup — otherwise setuptools flat-layout errors on data/ / images/.
2. Base image built locally — pull access denied means image missing, not registry auth.
3. MCP 421 Misdirected Request — FastMCP host check rejects Host: host.docker.internal. Local fix: allowed_hosts=["*"] on each mcp.run(...), then restart servers.
4. Ports 801x (MCP) vs 900x (REST) are both correct.
5. First trial is slower: Grok CLI installs inside the container (curl https://x.ai/cli/install.sh).
6. Grok Build disables web search by default (closed-book eval integrity). Keep that unless intentionally changing protocol.

Execution plan

Phase 0 — Tooling

uv tool upgrade harbor
harbor --version   # expect ≥ 0.20
harbor auth status

Phase 1 — Dataset + environment

Prefer a clean Hub download or the maintained local tree.

harbor download enterprise-bench/l1-l2-bench -o ./enterprise-bench
cd ./enterprise-bench/l1-l2-bench

make install
make setup
# apply MCP allowed_hosts fix if Host-header probe returns 421
make build-image
make start-servers

Verify MCP from agent Host header:

curl -s -o /dev/null -w "%{http_code}\n" -X POST http://localhost:8012/mcp \
  -H "Host: host.docker.internal:8012" \
  -H "Accept: application/json, text/event-stream" \
  -H "Content-Type: application/json" \
  -d '{"jsonrpc":"2.0","id":1,"method":"initialize","params":{"protocolVersion":"2025-06-18","capabilities":{},"clientInfo":{"name":"t","version":"1"}}}'
# expect 200

Phase 2 — Smoke (1 task, 1 attempt)

export XAI_API_KEY=...
export OPENAI_API_KEY=...

harbor run \
  -p eng-l1-a \
  -a grok-build \
  -m grok-4.5 \
  --mcp-config mcp.json \
  --ae XAI_API_KEY=$XAI_API_KEY \
  --ae OPENAI_API_KEY=$OPENAI_API_KEY \
  -k 1 -n 1 \
  --yes \
  --jobs-dir jobs/grok-build-grok-4.5-smoke

Smoke pass criteria: agent starts, MCP tools appear in trajectory, judge writes reward.txt, no auth failure in first minutes of setup.

Phase 3 — Reliability pass (compare to prior local baseline)

Prior local claude-code / claude-opus-4-8 full job (2026-07-09__09-26-22): 70 trials, mean reward ~0.74, pass@5 ~0.93.

# all 14 tasks × 5 attempts, 3 concurrent (safe on ~8 GiB Docker)
harbor run \
  -p . \
  -a grok-build \
  -m grok-4.5 \
  --mcp-config mcp.json \
  --ae XAI_API_KEY=$XAI_API_KEY \
  --ae OPENAI_API_KEY=$OPENAI_API_KEY \
  -k 5 -n 3 \
  --yes \
  --jobs-dir jobs/grok-build-grok-4.5-k5

Run long jobs under tmux. Prefer explicit harbor run over Makefile defaults (Makefile still centers on claude-code).

Phase 4 — Optional methodology-grade + submit

Paper protocol: 14 × 10 = 140 observations.

harbor run -p . -a grok-build -m grok-4.5 --mcp-config mcp.json \
  --ae XAI_API_KEY=$XAI_API_KEY --ae OPENAI_API_KEY=$OPENAI_API_KEY \
  -k 10 -n 3 --yes --jobs-dir jobs/grok-build-grok-4.5-k10

Submission package (see docs/submit-results.md):

• Agent name + version (Grok Build / CLI version)
• Model: grok-4.5 (provider xAI)
• Harbor version
• Dataset ref: enterprise-bench/l1-l2-bench (or commit SHA)
• Full command, -k, -n
• OS, Docker memory/CPU, any agent kwargs
• Harbor Hub job link or archived jobs/
• Known failures / task exclusions

Do not submit modified tasks without disclosure, scrubbed failed trials, or logs containing secrets.

Phase 5 — Teardown

make stop-servers
# avoid make clean unless intentionally wiping jobs + re-applying MCP fix

Success metrics to report

• Mean reward / success rate
• pass@k (k=2,4,5 and/or 10)
• Per-task reliability (note: eng-l1-a criterion 6 has been hard across agents)
• Infra errors vs genuine task fails
• Rough wall-clock and cost notes (xAI agent + OpenAI judge)

Checklist

• Harbor ≥ 0.20; harbor run -h lists grok-build
• XAI_API_KEY + OPENAI_API_KEY available to the runner
• Base image built; MCP servers healthy (Host-header probe 200)
• Smoke: eng-l1-a k=1 complete
• Reliability: 14 tasks × k=5 complete
• Results summarized vs Claude baseline
• Optional: k=10 methodology run
• Optional: Hub upload / result submission
• Servers stopped; keys rotated if pasted into shell history

Time / cost rough order

Task agent timeout in task.toml is typically 600s; verifier 300s.

References

• Dataset: https://hub.harborframework.com/datasets/enterprise-bench/l1-l2-bench
• Harbor: https://github.com/harbor-framework/harbor
• Grok Build docs: https://docs.x.ai/build/overview
• Methodology: https://devrev.ai/enterprise-bench-methodology
• This repo: README.md, SKILL.md, docs/submit-results.md
• Existing public job links in README (Claude Code / DevRev Computer)

Owner / next step

Execute Phase 0–2 as soon as API keys are available, then Phase 3 for a submit-ready Grok data point.

/cc @jeff-smith (adjust if handle differs)