Community benchmark: DeepSeek-V4.1-Flash on Enterprise-Bench (open-weight) #85

Goal

Run DeepSeek-V4.1-Flash on Enterprise-Bench L1-L2 and submit a public, reproducible leaderboard result. Category: Open-weight. Open for a community contributor to claim.

Why this model

DeepSeek's new open-weight model targets input-heavy agentic workloads with a redesigned KV-cache architecture and controllable reasoning effort.[2] Enterprise-Bench is a useful test of its retrieval accuracy, latency, and cost on repeated multi-tool workflows. This is a different target from the existing V4 Pro rows and V4 Flash intent.

Coverage checked (2026-09-28)

This model is absent from the live Harbor leaderboard and the repository's merged entry files.[8][9] Existing issue and PR titles/bodies were checked before opening this request. Related but different: #49 proposes DeepSeek V4-Flash; this issue targets V4.1-Flash. Coordinate shared setup work, but do not relabel V4 results as V4.1.

Suggested starting point

• Checkpoint: deepseek-ai/DeepSeek-V4.1-Flash, MIT-licensed weights.[2]
• Suggested harness: Claude Code, to make a same-harness comparison with the existing DeepSeek V4 Pro row possible. Verify the provider's Messages-compatible endpoint or bridge and MCP tool round trips before the full run.
• Access: Use an endpoint explicitly serving V4.1 Flash, or the official model-card inference instructions for self-hosting. Do not silently substitute V4 Flash or a moving alias.
• Record: exact checkpoint/API model ID, provider, reasoning effort, sampling and token limits. The model card supports numeric reasoning effort 1–100 and notes that prompt encoding is supplied as code rather than a Jinja chat template; self-hosted deployments must preserve tool-call and thinking serialization.[2]

The harness is a recommendation, not a restriction. Share a different reproducible pairing here if it is easier to run. This issue does not imply that a new adapter or provider integration has already been tested.

Run and submission checklist

Start with the repository's runbook. The live Harbor dataset slug is case-sensitive: Enterprise-Bench/l1-l2-bench.[9]

• Comment with your proposed harness, provider, access needs and rough timeline before starting, so we can avoid duplicate runs. No benchmark run is needed to express interest.
• Pin the unmodified L1-L2 dataset revision or repo commit, harness/Harbor versions, and model configuration. Connect the standard CRM, PM and file-server MCP tools.
• Smoke-test one task and inspect its tool calls before committing to a full run. Share any adapter or access blocker here.
• Run all 14 L1-L2 tasks with 10 independent trials per task, targeting 140 trials. Keep the judge and tool surface consistent, record concurrency, and disclose every failure, retry or exclusion. Do not discard failed trials.
• Report accuracy, pass@k, cached/uncached/output tokens, runtime, and cost. Record self-hosted infrastructure cost separately where relevant, rather than presenting free API usage as zero total cost.
• Upload a public Harbor job with trajectories that reviewers can open while logged out. Remove credentials, not failure evidence.
• Submit one result PR using leaderboard/row-template.yaml under leaderboard/entries/. Fill every required field with observed results, ensure n_trials equals the unique trial_ids count, and run make validate-leaderboard.[7]

Done means a reviewed, reproducible result submission, not an intent or a placeholder score. This issue requests a run; it does not claim a benchmark result or reserve a leaderboard rank. Maintainers handle publication after review.[7]

Want to pick this up?

Comment here with your harness + provider and any setup or access blocker. If compute/API cost is a blocker, flag it before spending. Leave secrets out of comments and logs. You can also join the DevRev Community Slack to discuss setup.

Sources

[2] https://huggingface.co/deepseek-ai/DeepSeek-V4.1-Flash
[7] https://github.com/devrev/enterprise-bench/blob/main/docs/submit-results.md
[8] https://github.com/devrev/enterprise-bench/tree/main/leaderboard/entries
[9] https://hub.harborframework.com/datasets/Enterprise-Bench/l1-l2-bench?tab=leaderboard&leaderboard=main