Repository navigation
.NET Allow ISKFunction to return IAsyncEnumerable #1298
Description
Important
Labeled High because it will not require a breaking change, but it's very important to complete by v1.0.0
Is there any way currently to allow ISKFunction to return an IAsyncEnumerable? If not is there any plan to?
Reacted by Mads Bolaris
Metadata
Metadata
Assignees
Type
Projects
- Status
Activity
wayne-o commentedon Jun 1, 2023
Or to put it another way, is there any way to get the gpt client to use the streaming endpoint and emit events on each message?
craigomatic commentedon Jun 2, 2023
We added support for streaming chat messages semi-recently, although I'm not sure this is what you are asking for: https://github.com/microsoft/semantic-kernel/blob/main/samples/dotnet/kernel-syntax-examples/Example33_StreamingChat.cs
Can you be a little more specific about the types of events you are looking to consume?
wayne-o commentedon Jun 2, 2023
awharrison-28 commentedon Jun 2, 2023
Roger Barreto (@rogerbarreto) has some ideas in the works to make the requested behavior possible. TBD - when we get those changes in, let's make sure to tag this issue.
wayne-o commentedon Jun 3, 2023
MithrilMan commentedon Jun 13, 2023
Since you don't have this yet in Semantic Kernel, let me give my view on this:
I'm working on my own implementation of generative LLM integration here: https://github.com/MithrilMan/AIdentities
since a couple of months (I didn't knew about semantic kernel and when I found it it was already late and anyway wasn't adhering 100% to my vision) and I implemented what Wayne Douglas (@wayne-o) is asking in my design: I've a concept of Skills and when a skill is executed, this is the signature of the call
IAsyncEnumerable<Thought> ExecuteAsync(Prompt prompt, SkillExecutionContext context, CancellationToken cancellationToken);The core of my skill/brain thing is here (readme isn't updated) https://github.com/MithrilMan/AIdentities/tree/25e60f875979a0d595561fb7d0f1f5e82e5ab7a1/src/AIdentities.Shared/Features/CognitiveEngine
It works, but I'm sick of a problem that arises around the use of IAsyncEnumerable: when the API fails in the middle of generating a streamed content, it fails badly because it fails when you start to consume the iterator and .Net doesn't have a nice way to handle such cases, so if you want to implement a retry pattern for example by using polly (and we've seen how this is important with OpenAI that lately had big problems serving properly their API), you can't just wrap the call to the stream method because the exception of course arise when you materialize the enumerator.
Unluckily .Net doesn't allow to call a
yield returnwithin a try/catch block, so you have to create all sort of workaround to have a proper retry method.Also when you stream chunks of text, what would you do if something goes wrong in the API ?
In a simple chat application you can just ask the user to send again the message, but if it's part of a skill (or skfunction as you call it) It's very likely that You'd need to inform the consumer that you have to start over and that it should forget the partial generated text (= higher complexity)
I've yet not figured out how to properly and affectively handle such scenarios, in my case when it fails, fails badly, but this makes me wonder if it's worth to use streaming calls in skills at all.
Of course it's cool to see that my Skill is producing a text as a stream and the user can start reading as soon as it's generating, also what I called AIdentity can generate different kind of "thoughts" that can be streemed too (think of it like a kind of log in some scanrio) but at the same time if it breaks in the middle you'll have big troubles to rollback what you eventually did with the partial generated text, without mentioning minor problems it can give if you try to parse that text on a markdown viewer for example (that's why recently in my chat section I'm building the message on a normal div and then swith to markdown once completed), and I'm starting to think that maybe all this complexity isn't worth the effort.
I'm a bit torn as to whether or not implementing stream at the function level is something I'd want to deal with. Maybe just being able to signal to consumer events like "StartingTextGeneration" / "EndingTextGeneration" could be enough
3 remaining items
markwallace-microsoft commentedon Oct 10, 2023
Make child of #1649
[-]Allow ISKFunction to return IAsyncEnumerable[/-][+].NET Allow ISKFunction to return IAsyncEnumerable[/+].Net Kernel and ISKFunction Streaming Option #1 (Phase 1) (#3496)