Issues

• [BUG] Belebele cf tasks fail for Latvian, Lithuanian, Maltese and Slovenian: missing translation literals

  bug
  huggingface/lighteval#1471 · Filter by author HannanJaved opened on Nov 2, 2026

• HLE calibration_error compares confidence in percent (0-100) with correctness in 0/1, so a model that is always right at 100% confidence scores 70

  huggingface/lighteval#1469 · Filter by author damienen opened on Nov 2, 2026

• [BUG] Stderr of _average and all rows is the mean of the subtask stderrs (about 7x too large for MMLU)

  huggingface/lighteval#1466 · Filter by author Andreas-Hum opened on Nov 1, 2026

• [BUG] LCB grader fails correct solutions that write via sys.stdout.buffer

  huggingface/lighteval#1464 · Filter by author abhishekcpp007 opened on Nov 1, 2026

• [EVAL] Add support for IFEval-Ko

  huggingface/lighteval#1463 · Filter by author SubJeeLee opened on Nov 1, 2026

• [BUG] LiveCodeBench grader runs generated code with no memory limit

  huggingface/lighteval#1459 · Filter by author williamcaban opened on Oct 29, 2026

• [BUG] Reasoning tag removal fails when <think> is part of the input prompt

  bug
  huggingface/lighteval#1453 · Filter by author Tialo opened on Oct 27, 2026
  1

• [EVAL] HAE-RAE Bench: Korean-native knowledge and vocabulary

  huggingface/lighteval#1448 · Filter by author jhwanseok opened on Oct 23, 2026
  2

• [BUG] TruthfulQA MC2 drops true answers after a gap in gold indexes

  huggingface/lighteval#1442 · Filter by author salma2vec opened on Oct 20, 2026

• [BUG] Token F1 ignores repeated words in predictions and references

  huggingface/lighteval#1441 · Filter by author salma2vec opened on Oct 20, 2026

• Bayes@N metric for repeated generative evaluation (existing PR #1219)

  huggingface/lighteval#1440 · Filter by author mohsenhariri opened on Oct 20, 2026

• [EVAL] CLIcK: Korean cultural and language understanding

  huggingface/lighteval#1437 · Filter by author ojm6135 opened on Oct 20, 2026

• [BUG] Inspect math scorer rejects equivalent numerical answers by comparing parser fallback strings

  huggingface/lighteval#1430 · Filter by author surgan12 opened on Oct 16, 2026

• [EVAL] KMMLU (Korean-native MMLU, 45 subjects): would a PR be welcome?

  huggingface/lighteval#1424 · Filter by author CY-HYUN opened on Oct 10, 2026
  2

• Importing swiss_legal metrics sets float32 matmul precision to "medium" process-wide, breaking long transformers generations

  huggingface/lighteval#1414 · Filter by author dzautner opened on Oct 2, 2026

• PassAtK returns 1.0 for all-wrong samples when k > n

  huggingface/lighteval#1410 · Filter by author caiotheodoro opened on Sep 30, 2026
  2

• [BUG] 187 multilingual/QA tasks can't load on datasets>=4 (script-based datasets); ifeval-fr/gpqa-fr/bac-fr repos are gone

  huggingface/lighteval#1406 · Filter by author David-Wu1119 opened on Sep 28, 2026
  1

• [BUG] LiveCodeBench scores every generation as -2 when multiprocessing uses spawn or forkserver

  huggingface/lighteval#1400 · Filter by author David-Wu1119 opened on Sep 27, 2026

• [FT] Add a format-neutral post-run export hook

  huggingface/lighteval#1397 · Filter by author halvrenofviryel opened on Sep 23, 2026
  1

• Metrics.drop aggregates em/f1 across documents with max, so one correct sample reports 1.0

  huggingface/lighteval#1395 · Filter by author shaurya416 opened on Sep 23, 2026

• avg_at_n silently ignores strip_strings/normalize (preprocess not applied), deflating scores

  huggingface/lighteval#1384 · Filter by author Abelo9996 opened on Sep 10, 2026

• Normalized multiple-choice probability underflows for long sequence log-probabilities

  huggingface/lighteval#1375 · Filter by author YusefSyed opened on Sep 2, 2026

• SamplingMetric string normalize argument always raises: getmembers() list treated as a dict

  huggingface/lighteval#1365 · Filter by author Abelo9996 opened on Aug 29, 2026

• Prediction cache identity is the model config, not the model that ran: weights moving under the same revision name silently reuse the old responses

  huggingface/lighteval#1364 · Filter by author AUTHENSOR opened on Aug 29, 2026

• Sample cache: doc.id collides across evaluation splits, crashing runs and silently serving one split's answers for another split's questions

  huggingface/lighteval#1363 · Filter by author AUTHENSOR opened on Aug 29, 2026
  1