Code validation repeats the first result for subsequent target columns #1092

Priority Level

Medium (Annoying but has workaround)

Describe the bug

When a code-validation column has multiple target_columns, the first target receives its per-row results correctly, but every later target repeats its first validation result across all rows.

This can silently mark invalid code as valid when the first record passes, or mark valid code as invalid when the first record fails. Validation metadata is repeated along with is_valid.

Steps/Code to reproduce bug

Run this in a DataDesigner development environment. It uses the real Python validator/Ruff; only the resource provider is stubbed. No model endpoint or credentials are required.

from __future__ import annotations

from unittest.mock import Mock

import pandas as pd

from data_designer.config.column_configs import ValidationColumnConfig
from data_designer.config.run_config import RunConfig
from data_designer.config.utils.code_lang import CodeLang
from data_designer.config.validator_params import CodeValidatorParams, ValidatorType
from data_designer.engine.column_generators.generators.validation import ValidationColumnGenerator
from data_designer.engine.resources.resource_provider import ResourceProvider

provider = Mock(spec=ResourceProvider)
provider.run_config = RunConfig()

data = pd.DataFrame(
    {
        "first_code": ["a = 1", "b = 2", "c = 3"],
        "second_code": ["a = 1", "def f(:", "c = 3"],
    }
)
config = ValidationColumnConfig(
    name="check",
    validator_type=ValidatorType.CODE,
    validator_params=CodeValidatorParams(code_lang=CodeLang.PYTHON),
    target_columns=["first_code", "second_code"],
    batch_size=2,
)
result = ValidationColumnGenerator(config=config, resource_provider=provider).generate(data)
actual = [row["second_code"]["is_valid"] for row in result["check"]]
print(actual)
assert actual == [True, False, True]

Actual output: [True, True, True]; the assertion fails.

As a control, change target_columns to ["second_code"] and read row["is_valid"] instead. That correctly returns [True, False, True]. Both the failure and control reproduce with batch_size=2 and batch_size=10.

If a restricted environment prevents Ruff cache writes, run with RUFF_NO_CACHE=true.

Expected behavior

Each target column's nested validation result should correspond to that target's code in the same input row. The second target above should return [True, False, True], including a failed validation and the matching diagnostics for the syntax-error row.

Agent Diagnostic / Prior Investigation

The shared Python/SQL result-assembly branch in validation.py uses:

for dict_output in outputs_as_dicts:
    dict_output[cols[0]] = concatenated_outputs[0].model_dump(mode="json")

For targets after the first, this always selects output zero instead of pairing each row with its corresponding result.

• Reproduced with actual Python and SQL validators during review; the minimal reproduction above uses Python.
• Confirmed the bug in both PR fix: align validation results with input index #1078 (e1968077) and its parent implementation. The current default-branch source also retains this loop. This is an existing bug, independent of the index-alignment fix in fix: align validation results with input index #1078 / ValidationColumnConfig misaligns rows and discards the whole batch after an upstream row is dropped #1073.
• The reproduction and single-target control were rerun with Python 3.13.3, pandas 3.0.5, and Ruff 0.15.5.
• Searched existing issues for multiple code validators, repeated validation results, target_columns, and concatenated_outputs; no matching issue was found.

Suggested fix: check each target's output count, then pair output dictionaries with the corresponding validation outputs instead of selecting concatenated_outputs[0]. Add Python/SQL regression coverage with multiple targets and different validity/metadata between rows.

Additional context

Workaround: define a separate validation column for each code target, with one target_columns entry per validation column.

AI assistance: Codex investigated the bug, executed the reproduction and controls, and prepared this report.

Checklist

• I reproduced this issue or provided a minimal example
• I searched the docs/issues myself, or had my agent do so
• If I used an agent, I included its diagnostics above