You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Interest check: to_openeval()/from_openeval() bridge between Dataset/Datapoint and the EvalPort portable format #78
I know from your README that this library is Fern-generated and PRs against the generated code can't land as-is ("we suggest opening an issue first to discuss with us") — so this is exactly that: an interest check before I'd sketch anything, not a PR against generated types.
Looking at src/humanloop/types/dataset_response.py and datapoint_response.py, DatasetResponse.datapoints is a list[DatapointResponse], and each DatapointResponse already carries messages: list[ChatMessage] and a target: dict[str, DatapointResponseTargetValue] — which maps quite directly onto an EvalPort test case:
Given a Dataset can already be round-tripped through client.prompts.log(...) / client.datasets, a converter like this would let a Humanloop Dataset be exported as a portable, framework-agnostic regression suite (or vice versa). Given the generation constraint, I'd guess this would live as a small hand-written helper module rather than touching the generated types/ — happy to sketch that if there's interest, or to hear if this is better raised against your API/generation pipeline instead. No pressure either way — for context, related precedent is forming in openai/openai-python#3619 (openai/openai-python#3619, still open) where native to_openeval()/from_openeval() helpers are being added to that SDK for the same reason.
Activity