Activity

1. Roman Lutz (romanlutz)

   added

   not ready yetThis issue needs more definition or is blocked by a pending change.This issue needs more definition or is blocked by a pending change.

   on Jun 25, 2025
2. Roman Lutz (romanlutz)

   added a parent issue on Jun 25, 2025

   • Improving the Greedy Coordinate Gradient implementation #960
3. romanlutz commented on Mar 4, 2026

   

   Roman Lutz (romanlutz)

   on Mar 4, 2026

   ContributorAuthor

   More actions

   Logprobs Support Across LLM Providers

   Summary Table

   Provider	API Format	Logprobs?	Models	Notes
   OpenAI	Chat Completions	✅ Yes	GPT-3.5-turbo, GPT-4, GPT-4o, GPT-4.1, GPT-5, GPT-5.1, GPT-5.2	logprobs=true, top_logprobs (1–20). Not available for image/multimodal inputs.
   OpenAI	Responses API	✅ Yes	GPT-4.1, GPT-4.1-nano, GPT-4o, GPT-5, GPT-5.1, GPT-5.2	top_logprobs param + include: ["message.output_text.logprobs"]. Added mid-2025.
   OpenAI	Completions (legacy)	✅ Yes	text-davinci-003 etc.	Deprecated; full logprobs + echo support.
   Azure OpenAI	Chat Completions	⚠️ Partial	GPT-4o, GPT-4.1	Supported on OpenAI side; Azure may lag behind and some deployments still reject logprobs. Check your region/API version.
   Anthropic	Messages API	❌ No	Claude (all versions)	Not supported. Frequently requested. No workaround except estimation hacks via openlogprobs library.
   Google Gemini	Vertex AI only	✅ Yes	Gemini 1.5, 2.5	response_logprobs=true, logprobs=N. Not available on the standard Google GenAI API — only via Vertex AI.
   DeepSeek	Chat Completions (OpenAI-compat)	❌ Broken	DeepSeek V3.2, R1	API accepts the param but returns 0 or -9999. Effectively non-functional as of late 2025.
   Mistral	Native API	❌ Not documented	Mistral models	Not prominently documented. Works via self-hosted (vLLM) or Amazon Bedrock.
   Cohere	Generate / Chat	❌ No	Command models	Not supported natively.
   Meta Llama (via Together AI)	Chat Completions (OpenAI-compat)	✅ Yes	Llama 3.x	logprobs param, well-documented.
   Meta Llama (via Fireworks AI)	Chat Completions (OpenAI-compat)	✅ Yes	Llama 3.2, 3.3	OpenAI-compatible schema with logprobs.
   Self-hosted (vLLM)	OpenAI-compatible	✅ Yes	Any supported model	Full logprobs + prompt logprobs. Works with Llama, Mistral, Qwen, etc. Best option for guaranteed access.
   Amazon Bedrock	Custom model import	✅ Yes	Llama, Mistral, Qwen, etc.	return_logprobs: true. Added 2025.

   Key Takeaways for Attacks

   1. Best bets for logprobs-based attacks: OpenAI models (Chat Completions or Responses API), Gemini via Vertex AI, and open-weight models served through Together AI, Fireworks AI, or self-hosted vLLM.

   2. Anthropic is a blocker: Claude has no logprobs support at all — any attack requiring token probabilities (e.g., membership inference, calibration-based jailbreak detection, perplexity analysis) simply cannot target Claude.

   3. DeepSeek is unreliable: Despite OpenAI-compatible API surface, logprobs values are meaningless placeholders.

   4. Self-hosted via vLLM is the most reliable path: Full control, works with any HuggingFace model, guaranteed logprobs access including prompt logprobs (useful for prefix attacks).

   5. Azure OpenAI has friction: Feature rollout lags behind OpenAI proper — verify per-deployment before building attack pipelines against Azure endpoints.
4. Roman Lutz (romanlutz)

   closed this as completedon Mar 4, 2026