[vmpuri]

Implement JSON formatted responses using OpenAI API types for server completion requests. Rather than giving single tokens at a time, the server will respond with a JSON following the API dataclasses corresponding to OpenAI API types.

Testing:
Server

python3 torchchat.py server stories15M   

Request (chunked)

curl http://127.0.0.1:5000/chat \
  -H "Content-Type: application/json" \
  -d '{
    "model": "stories15M",
    "stream": "true",
    "messages": [
      {
        "role": "system",
        "content": "You are a helpful assistant."
      },
      {
        "role": "user",
        "content": "Story about a dog."
      }
    ]

Request (synchronous)

curl http://127.0.0.1:5000/chat \
  -H "Content-Type: application/json" \
  -d '{
    "model": "stories15M",
    "stream": "false",
    "messages": [
      {
        "role": "system",
        "content": "You are a helpful assistant."
      },
      {
        "role": "user",
        "content": "Story about a dog."
      }
    ]

Screen.Recording.2024-08-02.at.1.22.21.PM.mov

[pytorch-bot]

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/torchchat/995

• 📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

✅ No Failures

As of commit 0d3a5c3 with merge base a3bf37d ():
💚 Looks good so far! There are no failures yet. 💚

This comment was automatically generated by Dr. CI and updates every 15 minutes.

[Jack-Khuu]

Can you add a video for non-chunked as well for record purposes

[Jack-Khuu]

Some nits, but the content lgtm

Remember to verify the behavior lines up with how people plan on using OpenAI

[Jack-Khuu]

Let's add a comment that this field doesn't match the spec, but is populated

We'll fix in a separate PR

[Jack-Khuu]

Suggested change

	nextok = chunk.choices[0].delta.content
	nextok = nextok if nextok is not None else ""
	if (next_tok := chunk.choices[0].delta.content) is None:
	next_tok = ""