Skip to content

OnnxRuntimeGenAIChatCompletionService seems broken? #10796

Closed
Bug
Closed
OnnxRuntimeGenAIChatCompletionService seems broken?#10796
Bug
Labels
bugSomething isn't working

Description

@lilhoser
Aaron LeMasters (lilhoser)
opened on Mar 4, 2025

Describe the bug

Updated to the latest SK and ran some previously working chat completion code:

await foreach (var chatUpdate in m_Kernel.InvokePromptStreamingAsync<StreamingChatMessageContent>(
    promptTemplate: @$"Question: {{{{$query}}}}{additionalContext}",
    arguments: new KernelArguments()
    {
        { "query", userInput.Content }
    }))

and I get an exception:

Method not found: 'Void Microsoft.ML.OnnxRuntimeGenAI.GeneratorParams.SetInputSequences(Microsoft.ML.OnnxRuntimeGenAI.Sequences)'.

   at Microsoft.SemanticKernel.Connectors.Onnx.OnnxRuntimeGenAIChatCompletionService.<RunInferenceAsync>d__13.MoveNext()
   at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[TStateMachine](TStateMachine& stateMachine) in /_/src/libraries/System.Private.CoreLib/src/System/Runtime/CompilerServices/AsyncMethodBuilderCore.cs:line 38
   at Microsoft.SemanticKernel.Connectors.Onnx.OnnxRuntimeGenAIChatCompletionService.<RunInferenceAsync>d__13.System.IAsyncDisposable.DisposeAsync()
   at System.Runtime.CompilerServices.ConfiguredCancelableAsyncEnumerable`1.Enumerator.DisposeAsync() in /_/src/libraries/System.Private.CoreLib/src/System/Runtime/CompilerServices/ConfiguredCancelableAsyncEnumerable.cs:line 76
   at Microsoft.SemanticKernel.Connectors.Onnx.OnnxRuntimeGenAIChatCompletionService.<GetStreamingChatMessageContentsAsync>d__12.MoveNext()
   at Microsoft.SemanticKernel.Connectors.Onnx.OnnxRuntimeGenAIChatCompletionService.<GetStreamingChatMessageContentsAsync>d__12.System.Threading.Tasks.Sources.IValueTaskSource<System.Boolean>.GetResult(Int16 token)
   at Microsoft.SemanticKernel.KernelFunctionFromPrompt.<InvokeStreamingCoreAsync>d__7`1.MoveNext() in Microsoft.SemanticKernel\KernelFunctionFromPrompt.cs:line 193
   at Microsoft.SemanticKernel.KernelFunctionFromPrompt.<InvokeStreamingCoreAsync>d__7`1.MoveNext() in Microsoft.SemanticKernel\KernelFunctionFromPrompt.cs:line 193
   at Microsoft.SemanticKernel.KernelFunctionFromPrompt.<InvokeStreamingCoreAsync>d__7`1.System.Threading.Tasks.Sources.IValueTaskSource<System.Boolean>.GetResult(Int16 token)
   at Microsoft.SemanticKernel.KernelFunction.<InvokeStreamingAsync>d__30`1.MoveNext() in Microsoft.SemanticKernel\KernelFunction.cs:line 276
   at Microsoft.SemanticKernel.KernelFunction.<InvokeStreamingAsync>d__30`1.MoveNext() in Microsoft.SemanticKernel\KernelFunction.cs:line 265
   at Microsoft.SemanticKernel.KernelFunction.<InvokeStreamingAsync>d__30`1.System.Threading.Tasks.Sources.IValueTaskSource<System.Boolean>.GetResult(Int16 token)

Could be related: microsoft/onnxruntime-genai#1230

Activity

  1. Aaron LeMasters (lilhoser)
    added
    bugSomething isn't working
    on Mar 4, 2025
  2. Aaron LeMasters (lilhoser)
    added the
    Bug
    issue type on Mar 4, 2025
  3. Mark Wallace (markwallace-microsoft)
    added
    triage
    on Mar 4, 2025

  4. stephentoub commented on Mar 4, 2025

    @stephentoub
    Stephen Toub (stephentoub)
    on Mar 4, 2025
    Member

    What version of the ONNX Runtime GenAI component are you referencing? There was a change in that library. Try updating to the latest on nuget.

  5. lilhoser commented on Mar 4, 2025

    @lilhoser
    Aaron LeMasters (lilhoser)
    on Mar 4, 2025
    Author
        <PackageReference Include="Microsoft.KernelMemory.SemanticKernelPlugin" Version="0.97.250211.1" />
    <PackageReference Include="Microsoft.ML.OnnxRuntimeGenAI" Version="0.7.0-rc1" Condition=" '$(Configuration)' == 'Debug' OR '$(Configuration)' == 'Release' " />
    <PackageReference Include="Microsoft.ML.OnnxRuntimeGenAI.Cuda" Version="0.6.0" Condition=" '$(Configuration)' == 'Debug_Cuda' OR '$(Configuration)' == 'Release_Cuda' " />
    <PackageReference Include="Microsoft.ML.OnnxRuntimeGenAI.DirectML" Version="0.6.0" Condition=" '$(Configuration)' == 'Debug_DirectML' OR '$(Configuration)' == 'Release_DirectML' " />
    <PackageReference Include="Microsoft.SemanticKernel" Version="1.40.1" />
    <PackageReference Include="Microsoft.SemanticKernel.Abstractions" Version="1.40.1" />
    <PackageReference Include="Microsoft.SemanticKernel.Connectors.Onnx" Version="1.39.0-alpha" />
    <PackageReference Include="Microsoft.SemanticKernel.Connectors.Qdrant" Version="1.39.0-preview" />
    <PackageReference Include="Microsoft.SemanticKernel.Core" Version="1.40.1" />
    <PackageReference Include="Microsoft.SemanticKernel.Plugins.Memory" Version="1.39.0-alpha" />

  6. stephentoub commented on Mar 5, 2025

    @stephentoub
    Stephen Toub (stephentoub)
    on Mar 5, 2025
    Member

    You're using an inconsistent set of packages, e.g. some SK packages version 1.40.1 and other 1.39.0-alpha, some ONNX packages 0.7.0-r1 and some 0.6.0.

    Try moving all of them to be the latest versions on nuget.

  7. Mark Wallace (markwallace-microsoft)
    removed
    triage
    on Mar 5, 2025
  8. Mark Wallace (markwallace-microsoft)
    added this to Semantic Kernelon Mar 5, 2025
  9. Mark Wallace (markwallace-microsoft)
    moved this to Sprint: In Review in Semantic Kernelon Mar 5, 2025
  10. Aaron LeMasters (lilhoser)
    closed this as completedon Mar 5, 2025

  11. lilhoser commented on Mar 5, 2025

    @lilhoser
    Aaron LeMasters (lilhoser)
    on Mar 5, 2025
    Author

    These are the latest packages according to Nuget package manager. However, I removed them and added them back and now have a different package lineup that looks more consistent:

    <PackageReference Include="Microsoft.KernelMemory.SemanticKernelPlugin" Version="0.97.250211.1" />
    <PackageReference Include="Microsoft.SemanticKernel" Version="1.40.1" />
    <PackageReference Include="Microsoft.SemanticKernel.Abstractions" Version="1.40.1" />
    <PackageReference Include="Microsoft.SemanticKernel.Connectors.Onnx" Version="1.40.1-alpha" />
    <PackageReference Include="Microsoft.SemanticKernel.Connectors.Qdrant" Version="1.40.1-preview" />
    <PackageReference Include="Microsoft.SemanticKernel.Core" Version="1.40.1" />

    I'm not sure why nuget did not indicate there was an updated version of 1.39.0-alpha.

    It did the same thing for OnnxRuntimeGenAI - claimed 0.6.0 was the latest for Cuda and DirectML (when actually 0.7.0-rc1 is the latest for all three).

  12. Aaron LeMasters (lilhoser)
    changed the title [-]Bug: OnnxRuntimeGenAIChatCompletionService seems broken?[/-] [+]OnnxRuntimeGenAIChatCompletionService seems broken?[/+] on Mar 5, 2025
  13. Mark Wallace (markwallace-microsoft)
    moved this from Sprint: In Review to Sprint: Done in Semantic Kernelon Mar 10, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    Bug

    Projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions