OpenAI streaming chat completions in Python

<p>Using OpenAI chat completions but responses have high latency before any text appears. Need to stream the response token-by-token so users see text as it's generated, rather than waiting for the full response.</p>

<p>Use the streaming API with async generators in FastAPI:</p> <div class="highlight"><pre><span></span><code><span class="kn">from</span><span class="w"> </span><span class="nn">openai</span><span class="w"> </span><span class="kn">import</span> <span class="n">AsyncOpenAI</span> <span class="kn">from</span><span class="w"> </span><span class="nn">fastapi</span><span class="w"> </span><span class="kn">import</span> <span class="n">APIRouter</span> <span class="kn">from</span><span class="w"> </span><span class="nn">fastapi.responses</span><span class="w"> </span><span class="kn">import</span> <span class="n">StreamingResponse</span> <span class="n">router</span> <span class="o">=</span> <span class="n">APIRouter</span><span class="p">()</span> <span class="n">client</span> <span class="o">=</span> <span class="n">AsyncOpenAI</span><span class="p">()</span> <span class="nd">@router</span><span class="o">.</span><span class="n">post</span><span class="p">(</span><span class="s1">'/chat/stream'</span><span class="p">)</span> <span class="k">async</span> <span class="k">def</span><span class="w"> </span><span class="nf">stream_chat</span><span class="p">(</span><span class="n">request</span><span class="p">:</span> <span class="n">ChatRequest</span><span class="p">)</span> <span class="o">-&gt;</span> <span class="n">StreamingResponse</span><span class="p">:</span> <span class="k">async</span> <span class="k">def</span><span class="w"> </span><span class="nf">generate</span><span class="p">():</span> <span class="n">stream</span> <span class="o">=</span> <span class="k">await</span> <span class="n">client</span><span class="o">.</span><span class="n">chat</span><span class="o">.</span><span class="n">completions</span><span class="o">.</span><span class="n">create</span><span class="p">(</span> <span class="n">model</span><span class="o">=</span><span class="s1">'gpt-4o-mini'</span><span class="p">,</span> <span class="n">messages</span><span class="o">=</span><span class="n">request</span><span class="o">.</span><span class="n">messages</span><span class="p">,</span> <span class="n">stream</span><span class="o">=</span><span class="kc">True</span><span class="p">,</span> <span class="p">)</span> <span class="k">async</span> <span class="k">for</span> <span class="n">chunk</span> <span class="ow">in</span> <span class="n">stream</span><span class="p">:</span> <span class="n">delta</span> <span class="o">=</span> <span class="n">chunk</span><span class="o">.</span><span class="n">choices</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span><span class="o">.</span><span class="n">delta</span> <span class="k">if</span> <span class="n">delta</span><span class="o">.</span><span class="n">content</span><span class="p">:</span> <span class="c1"># Server-Sent Events format</span> <span class="k">yield</span> <span class="sa">f</span><span class="s1">'data: </span><span class="si">{</span><span class="n">json</span><span class="o">.</span><span class="n">dumps</span><span class="p">({</span><span class="s2">"content"</span><span class="p">:</span><span class="w"> </span><span class="n">delta</span><span class="o">.</span><span class="n">content</span><span class="p">})</span><span class="si">}</span><span class="se">\n\n</span><span class="s1">'</span> <span class="k">if</span> <span class="n">chunk</span><span class="o">.</span><span class="n">choices</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span><span class="o">.</span><span class="n">finish_reason</span> <span class="o">==</span> <span class="s1">'stop'</span><span class="p">:</span> <span class="k">yield</span> <span class="s1">'data: [DONE]</span><span class="se">\n\n</span><span class="s1">'</span> <span class="k">return</span> <span class="n">StreamingResponse</span><span class="p">(</span> <span class="n">generate</span><span class="p">(),</span> <span class="n">media_type</span><span class="o">=</span><span class="s1">'text/event-stream'</span><span class="p">,</span> <span class="n">headers</span><span class="o">=</span><span class="p">{</span><span class="s1">'Cache-Control'</span><span class="p">:</span> <span class="s1">'no-cache'</span><span class="p">,</span> <span class="s1">'X-Accel-Buffering'</span><span class="p">:</span> <span class="s1">'no'</span><span class="p">},</span> <span class="p">)</span> <span class="c1"># Client-side consumption (Next.js)</span> <span class="k">async</span> <span class="n">function</span> <span class="n">streamChat</span><span class="p">(</span><span class="n">messages</span><span class="p">:</span> <span class="n">Message</span><span class="p">[])</span> <span class="p">{</span> <span class="n">const</span> <span class="n">response</span> <span class="o">=</span> <span class="k">await</span> <span class="n">fetch</span><span class="p">(</span><span class="s1">'/chat/stream'</span><span class="p">,</span> <span class="p">{</span> <span class="n">method</span><span class="p">:</span> <span class="s1">'POST'</span><span class="p">,</span> <span class="n">headers</span><span class="p">:</span> <span class="p">{</span> <span class="s1">'Content-Type'</span><span class="p">:</span> <span class="s1">'application/json'</span> <span class="p">},</span> <span class="n">body</span><span class="p">:</span> <span class="n">JSON</span><span class="o">.</span><span class="n">stringify</span><span class="p">({</span> <span class="n">messages</span> <span class="p">}),</span> <span class="p">});</span> <span class="n">const</span> <span class="n">reader</span> <span class="o">=</span> <span class="n">response</span><span class="o">.</span><span class="n">body</span><span class="err">!</span><span class="o">.</span><span class="n">getReader</span><span class="p">();</span> <span class="n">const</span> <span class="n">decoder</span> <span class="o">=</span> <span class="n">new</span> <span class="n">TextDecoder</span><span class="p">();</span> <span class="k">while</span> <span class="p">(</span><span class="n">true</span><span class="p">)</span> <span class="p">{</span> <span class="n">const</span> <span class="p">{</span> <span class="n">done</span><span class="p">,</span> <span class="n">value</span> <span class="p">}</span> <span class="o">=</span> <span class="k">await</span> <span class="n">reader</span><span class="o">.</span><span class="n">read</span><span class="p">();</span> <span class="k">if</span> <span class="p">(</span><span class="n">done</span><span class="p">)</span> <span class="k">break</span><span class="p">;</span> <span class="n">const</span> <span class="n">text</span> <span class="o">=</span> <span class="n">decoder</span><span class="o">.</span><span class="n">decode</span><span class="p">(</span><span class="n">value</span><span class="p">);</span> <span class="n">const</span> <span class="n">lines</span> <span class="o">=</span> <span class="n">text</span><span class="o">.</span><span class="n">split</span><span class="p">(</span><span class="s1">'</span><span class="se">\n</span><span class="s1">'</span><span class="p">)</span><span class="o">.</span><span class="n">filter</span><span class="p">(</span><span class="n">l</span> <span class="o">=&gt;</span> <span class="n">l</span><span class="o">.</span><span class="n">startsWith</span><span class="p">(</span><span class="s1">'data: '</span><span class="p">));</span> <span class="k">for</span> <span class="p">(</span><span class="n">const</span> <span class="n">line</span> <span class="n">of</span> <span class="n">lines</span><span class="p">)</span> <span class="p">{</span> <span class="n">const</span> <span class="n">data</span> <span class="o">=</span> <span class="n">line</span><span class="o">.</span><span class="n">slice</span><span class="p">(</span><span class="mi">6</span><span class="p">);</span> <span class="k">if</span> <span class="p">(</span><span class="n">data</span> <span class="o">===</span> <span class="s1">'[DONE]'</span><span class="p">)</span> <span class="k">break</span><span class="p">;</span> <span class="n">const</span> <span class="n">parsed</span> <span class="o">=</span> <span class="n">JSON</span><span class="o">.</span><span class="n">parse</span><span class="p">(</span><span class="n">data</span><span class="p">);</span> <span class="n">onChunk</span><span class="p">(</span><span class="n">parsed</span><span class="o">.</span><span class="n">content</span><span class="p">);</span> <span class="o">//</span> <span class="n">update</span> <span class="n">UI</span> <span class="p">}</span> <span class="p">}</span> <span class="p">}</span> </code></pre></div> <p>Key: <code>stream=True</code> returns an async iterable. The <code>X-Accel-Buffering: no</code> header prevents nginx from buffering the stream. Always handle <code>[DONE]</code> sentinel to know when streaming is complete.</p>