OpenAI streaming chat completions with FastAPI SSE

<p>I want to stream OpenAI chat completion responses to users so tokens appear in real-time. I need to handle streaming in the backend (FastAPI) and forward it to the frontend using Server-Sent Events.</p>

<p>Stream completions with StreamingResponse:</p> <div class="highlight"><pre><span></span><code><span class="kn">import</span><span class="w"> </span><span class="nn">json</span> <span class="kn">from</span><span class="w"> </span><span class="nn">openai</span><span class="w"> </span><span class="kn">import</span> <span class="n">AsyncOpenAI</span> <span class="kn">from</span><span class="w"> </span><span class="nn">fastapi.responses</span><span class="w"> </span><span class="kn">import</span> <span class="n">StreamingResponse</span> <span class="n">client</span> <span class="o">=</span> <span class="n">AsyncOpenAI</span><span class="p">()</span> <span class="k">async</span> <span class="k">def</span><span class="w"> </span><span class="nf">stream_completion</span><span class="p">(</span><span class="n">prompt</span><span class="p">:</span> <span class="nb">str</span><span class="p">):</span> <span class="w"> </span><span class="sd">"""Async generator yielding SSE events."""</span> <span class="k">try</span><span class="p">:</span> <span class="k">async</span> <span class="k">with</span> <span class="n">client</span><span class="o">.</span><span class="n">chat</span><span class="o">.</span><span class="n">completions</span><span class="o">.</span><span class="n">stream</span><span class="p">(</span> <span class="n">model</span><span class="o">=</span><span class="s1">'claude-opus-4-6'</span><span class="p">,</span> <span class="n">messages</span><span class="o">=</span><span class="p">[{</span><span class="s1">'role'</span><span class="p">:</span> <span class="s1">'user'</span><span class="p">,</span> <span class="s1">'content'</span><span class="p">:</span> <span class="n">prompt</span><span class="p">}],</span> <span class="n">max_tokens</span><span class="o">=</span><span class="mi">1000</span><span class="p">,</span> <span class="p">)</span> <span class="k">as</span> <span class="n">stream</span><span class="p">:</span> <span class="k">async</span> <span class="k">for</span> <span class="n">event</span> <span class="ow">in</span> <span class="n">stream</span><span class="p">:</span> <span class="k">if</span> <span class="n">event</span><span class="o">.</span><span class="n">type</span> <span class="o">==</span> <span class="s1">'content.delta'</span><span class="p">:</span> <span class="k">yield</span> <span class="sa">f</span><span class="s1">'data: </span><span class="si">{</span><span class="n">json</span><span class="o">.</span><span class="n">dumps</span><span class="p">({</span><span class="s2">"text"</span><span class="p">:</span><span class="w"> </span><span class="n">event</span><span class="o">.</span><span class="n">delta</span><span class="p">})</span><span class="si">}</span><span class="se">\n\n</span><span class="s1">'</span> <span class="k">yield</span> <span class="s1">'data: [DONE]</span><span class="se">\n\n</span><span class="s1">'</span> <span class="k">except</span> <span class="ne">Exception</span> <span class="k">as</span> <span class="n">e</span><span class="p">:</span> <span class="k">yield</span> <span class="sa">f</span><span class="s1">'data: </span><span class="si">{</span><span class="n">json</span><span class="o">.</span><span class="n">dumps</span><span class="p">({</span><span class="s2">"error"</span><span class="p">:</span><span class="w"> </span><span class="nb">str</span><span class="p">(</span><span class="n">e</span><span class="p">)})</span><span class="si">}</span><span class="se">\n\n</span><span class="s1">'</span> <span class="nd">@router</span><span class="o">.</span><span class="n">post</span><span class="p">(</span><span class="s1">'/complete'</span><span class="p">)</span> <span class="k">async</span> <span class="k">def</span><span class="w"> </span><span class="nf">complete</span><span class="p">(</span><span class="n">body</span><span class="p">:</span> <span class="n">CompletionRequest</span><span class="p">):</span> <span class="k">return</span> <span class="n">StreamingResponse</span><span class="p">(</span> <span class="n">stream_completion</span><span class="p">(</span><span class="n">body</span><span class="o">.</span><span class="n">prompt</span><span class="p">),</span> <span class="n">media_type</span><span class="o">=</span><span class="s1">'text/event-stream'</span><span class="p">,</span> <span class="n">headers</span><span class="o">=</span><span class="p">{</span> <span class="s1">'Cache-Control'</span><span class="p">:</span> <span class="s1">'no-cache'</span><span class="p">,</span> <span class="s1">'X-Accel-Buffering'</span><span class="p">:</span> <span class="s1">'no'</span><span class="p">,</span> <span class="c1"># Disable Nginx buffering</span> <span class="p">},</span> <span class="p">)</span> </code></pre></div> <p>Frontend reader:</p> <div class="highlight"><pre><span></span><code><span class="kd">const</span><span class="w"> </span><span class="nx">response</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="k">await</span><span class="w"> </span><span class="nx">fetch</span><span class="p">(</span><span class="s1">'/api/complete'</span><span class="p">,</span><span class="w"> </span><span class="p">{</span> <span class="w"> </span><span class="nx">method</span><span class="o">:</span><span class="w"> </span><span class="s1">'POST'</span><span class="p">,</span><span class="w"> </span><span class="nx">body</span><span class="o">:</span><span class="w"> </span><span class="kt">JSON.stringify</span><span class="p">({</span><span class="w"> </span><span class="nx">prompt</span><span class="w"> </span><span class="p">}),</span> <span class="w"> </span><span class="nx">headers</span><span class="o">:</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="s1">'Content-Type'</span><span class="o">:</span><span class="w"> </span><span class="s1">'application/json'</span><span class="w"> </span><span class="p">},</span> <span class="p">});</span> <span class="kd">const</span><span class="w"> </span><span class="nx">reader</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="nx">response</span><span class="p">.</span><span class="nx">body</span><span class="o">!</span><span class="p">.</span><span class="nx">getReader</span><span class="p">();</span> <span class="kd">const</span><span class="w"> </span><span class="nx">decoder</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="ow">new</span><span class="w"> </span><span class="nx">TextDecoder</span><span class="p">();</span> <span class="k">while</span><span class="w"> </span><span class="p">(</span><span class="kc">true</span><span class="p">)</span><span class="w"> </span><span class="p">{</span> <span class="w"> </span><span class="kd">const</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="nx">done</span><span class="p">,</span><span class="w"> </span><span class="nx">value</span><span class="w"> </span><span class="p">}</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="k">await</span><span class="w"> </span><span class="nx">reader</span><span class="p">.</span><span class="nx">read</span><span class="p">();</span> <span class="w"> </span><span class="k">if</span><span class="w"> </span><span class="p">(</span><span class="nx">done</span><span class="p">)</span><span class="w"> </span><span class="k">break</span><span class="p">;</span> <span class="w"> </span><span class="kd">const</span><span class="w"> </span><span class="nx">chunk</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="nx">decoder</span><span class="p">.</span><span class="nx">decode</span><span class="p">(</span><span class="nx">value</span><span class="p">).</span><span class="nx">replace</span><span class="p">(</span><span class="s1">'data: '</span><span class="p">,</span><span class="w"> </span><span class="s1">''</span><span class="p">).</span><span class="nx">trim</span><span class="p">();</span> <span class="w"> </span><span class="k">if</span><span class="w"> </span><span class="p">(</span><span class="nx">chunk</span><span class="w"> </span><span class="o">!==</span><span class="w"> </span><span class="s1">'[DONE]'</span><span class="p">)</span><span class="w"> </span><span class="nx">setOutput</span><span class="p">(</span><span class="nx">prev</span><span class="w"> </span><span class="p">=&gt;</span><span class="w"> </span><span class="nx">prev</span><span class="w"> </span><span class="o">+</span><span class="w"> </span><span class="nb">JSON</span><span class="p">.</span><span class="nx">parse</span><span class="p">(</span><span class="nx">chunk</span><span class="p">).</span><span class="nx">text</span><span class="p">);</span> <span class="p">}</span> </code></pre></div> <p>Key points: - X-Accel-Buffering: no prevents Nginx buffering SSE responses - SSE format: data: {json}\n\n (double newline terminates each event) - Always yield [DONE] sentinel so clients know stream ended cleanly - Handle errors inside the generator -- StreamingResponse cannot send HTTP errors after headers sent</p>