Skip to content

Bug: max_completion_tokens in llm_pool.defaults ignored when judge_panel is not configured #348

Open

Description

@xmican10

Root cause

  • No judge_panel → manager.py:256 uses system_config.llm (legacy default LLMConfig), bypassing llm_pool entirely
  • resolve_llm_config() — which merges defaults → model params — is never called
  • Ragas falls back to InstructorModelArgs.max_tokens = 1024
    Works when: model-level parameters set + judge_panel configured (triggers resolve_llm_config())

Reproduce
Set llm_pool.defaults.parameters.max_completion_tokens: 500, no judge_panel
Run eval → effective limit is 1024, not 500

Fix: When llm_pool is present but no judge_panel, resolve primary model via resolve_llm_config() instead of falling back to system_config.llm.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions