Community benchmark: Gemini 3.8 Flash on Enterprise-Bench (frontier) #89

Goal

Run Gemini 3.8 Flash on Enterprise-Bench L1-L2 and submit a public, reproducible leaderboard result. Category: Frontier / proprietary. Open for a community contributor to claim.

Why this model

Google's release emphasizes multi-step reasoning, iterative tool use, and long-horizon agentic work, while noting that higher effort can consume more tokens.[6] Enterprise-Bench can test that accuracy/efficiency tradeoff and add Google model coverage missing from the current board.

Coverage checked (2026-09-28)

This model is absent from the live Harbor leaderboard and the repository's merged entry files.[8][9] Existing issue and PR titles/bodies were checked before opening this request. Related but different: #40 targets Gemini 3.5 Flash and #81 broadly names Gemini 3 without a 3.8 configuration. Coordinate with those contributors if they want to take this version-specific run.

Suggested starting point

• Model: gemini-3.8-flash through the Gemini API. This request is for regular Flash, not the restricted Flash Cyber variant.[6]
• Suggested harness: Cline via Harbor's cline-cli adapter, configured for the Gemini provider. Confirm model selection and MCP tool support in a smoke test; the new model/provider combination has not been validated by this issue.
• Access: Gemini API credentials with adequate quota plus the benchmark judge credentials. Google links AI Studio and developer documentation from the announcement.[6]
• Record: exact model ID, API version/provider, harness and Harbor versions, thinking/effort setting, context limits, token usage and rate-limit retries. A Gemini-native harness is welcome if you share a reproducible Harbor adapter instead.

The harness is a recommendation, not a restriction. Share a different reproducible pairing here if it is easier to run. This issue does not imply that a new adapter or provider integration has already been tested.

Run and submission checklist

Start with the repository's runbook. The live Harbor dataset slug is case-sensitive: Enterprise-Bench/l1-l2-bench.[9]

• Comment with your proposed harness, provider, access needs and rough timeline before starting, so we can avoid duplicate runs. No benchmark run is needed to express interest.
• Pin the unmodified L1-L2 dataset revision or repo commit, harness/Harbor versions, and model configuration. Connect the standard CRM, PM and file-server MCP tools.
• Smoke-test one task and inspect its tool calls before committing to a full run. Share any adapter or access blocker here.
• Run all 14 L1-L2 tasks with 10 independent trials per task, targeting 140 trials. Keep the judge and tool surface consistent, record concurrency, and disclose every failure, retry or exclusion. Do not discard failed trials.
• Report accuracy, pass@k, cached/uncached/output tokens, runtime, and cost. Record self-hosted infrastructure cost separately where relevant, rather than presenting free API usage as zero total cost.
• Upload a public Harbor job with trajectories that reviewers can open while logged out. Remove credentials, not failure evidence.
• Submit one result PR using leaderboard/row-template.yaml under leaderboard/entries/. Fill every required field with observed results, ensure n_trials equals the unique trial_ids count, and run make validate-leaderboard.[7]

Done means a reviewed, reproducible result submission, not an intent or a placeholder score. This issue requests a run; it does not claim a benchmark result or reserve a leaderboard rank. Maintainers handle publication after review.[7]

Want to pick this up?

Comment here with your harness + provider and any setup or access blocker. If compute/API cost is a blocker, flag it before spending. Leave secrets out of comments and logs. You can also join the DevRev Community Slack to discuss setup.

Sources

[6] https://blog.google/innovation-and-ai/models-and-research/gemini-models/3-8-flash-and-3-8-flash-cyber
[7] https://github.com/devrev/enterprise-bench/blob/main/docs/submit-results.md
[8] https://github.com/devrev/enterprise-bench/tree/main/leaderboard/entries
[9] https://hub.harborframework.com/datasets/Enterprise-Bench/l1-l2-bench?tab=leaderboard&leaderboard=main