OpenAI API error handling with rate limit backoff

<p>I am hitting OpenAI rate limits and getting RateLimitError exceptions. I need robust retry logic with exponential backoff specifically for OpenAI API calls, and I need to handle different error types differently.</p>

<p>OpenAI error handling with tenacity:</p> <div class="highlight"><pre><span></span><code><span class="c1"># pip install tenacity</span> <span class="kn">from</span><span class="w"> </span><span class="nn">openai</span><span class="w"> </span><span class="kn">import</span> <span class="n">AsyncOpenAI</span><span class="p">,</span> <span class="n">RateLimitError</span><span class="p">,</span> <span class="n">APITimeoutError</span><span class="p">,</span> <span class="n">APIConnectionError</span> <span class="kn">from</span><span class="w"> </span><span class="nn">tenacity</span><span class="w"> </span><span class="kn">import</span> <span class="p">(</span> <span class="n">retry</span><span class="p">,</span> <span class="n">stop_after_attempt</span><span class="p">,</span> <span class="n">wait_exponential</span><span class="p">,</span> <span class="n">retry_if_exception_type</span><span class="p">,</span> <span class="n">before_sleep_log</span> <span class="p">)</span> <span class="kn">import</span><span class="w"> </span><span class="nn">logging</span> <span class="n">client</span> <span class="o">=</span> <span class="n">AsyncOpenAI</span><span class="p">()</span> <span class="n">log</span> <span class="o">=</span> <span class="n">logging</span><span class="o">.</span><span class="n">getLogger</span><span class="p">(</span><span class="vm">__name__</span><span class="p">)</span> <span class="nd">@retry</span><span class="p">(</span> <span class="n">retry</span><span class="o">=</span><span class="n">retry_if_exception_type</span><span class="p">((</span><span class="n">RateLimitError</span><span class="p">,</span> <span class="n">APITimeoutError</span><span class="p">,</span> <span class="n">APIConnectionError</span><span class="p">)),</span> <span class="n">wait</span><span class="o">=</span><span class="n">wait_exponential</span><span class="p">(</span><span class="n">multiplier</span><span class="o">=</span><span class="mi">1</span><span class="p">,</span> <span class="nb">min</span><span class="o">=</span><span class="mi">2</span><span class="p">,</span> <span class="nb">max</span><span class="o">=</span><span class="mi">60</span><span class="p">),</span> <span class="n">stop</span><span class="o">=</span><span class="n">stop_after_attempt</span><span class="p">(</span><span class="mi">5</span><span class="p">),</span> <span class="n">before_sleep</span><span class="o">=</span><span class="n">before_sleep_log</span><span class="p">(</span><span class="n">log</span><span class="p">,</span> <span class="n">logging</span><span class="o">.</span><span class="n">WARNING</span><span class="p">),</span> <span class="p">)</span> <span class="k">async</span> <span class="k">def</span><span class="w"> </span><span class="nf">generate_embedding_with_retry</span><span class="p">(</span><span class="n">text</span><span class="p">:</span> <span class="nb">str</span><span class="p">)</span> <span class="o">-&gt;</span> <span class="nb">list</span><span class="p">[</span><span class="nb">float</span><span class="p">]:</span> <span class="n">response</span> <span class="o">=</span> <span class="k">await</span> <span class="n">client</span><span class="o">.</span><span class="n">embeddings</span><span class="o">.</span><span class="n">create</span><span class="p">(</span> <span class="n">model</span><span class="o">=</span><span class="s1">'text-embedding-3-small'</span><span class="p">,</span> <span class="nb">input</span><span class="o">=</span><span class="n">text</span><span class="p">,</span> <span class="p">)</span> <span class="k">return</span> <span class="n">response</span><span class="o">.</span><span class="n">data</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span><span class="o">.</span><span class="n">embedding</span> <span class="c1"># Handle non-retryable errors separately:</span> <span class="k">async</span> <span class="k">def</span><span class="w"> </span><span class="nf">safe_embed</span><span class="p">(</span><span class="n">text</span><span class="p">:</span> <span class="nb">str</span><span class="p">)</span> <span class="o">-&gt;</span> <span class="nb">list</span><span class="p">[</span><span class="nb">float</span><span class="p">]</span> <span class="o">|</span> <span class="kc">None</span><span class="p">:</span> <span class="k">try</span><span class="p">:</span> <span class="k">return</span> <span class="k">await</span> <span class="n">generate_embedding_with_retry</span><span class="p">(</span><span class="n">text</span><span class="p">)</span> <span class="k">except</span> <span class="n">RateLimitError</span> <span class="k">as</span> <span class="n">e</span><span class="p">:</span> <span class="n">log</span><span class="o">.</span><span class="n">error</span><span class="p">(</span><span class="s1">'Rate limit exhausted after retries'</span><span class="p">,</span> <span class="n">error</span><span class="o">=</span><span class="nb">str</span><span class="p">(</span><span class="n">e</span><span class="p">))</span> <span class="k">return</span> <span class="kc">None</span> <span class="k">except</span> <span class="ne">Exception</span> <span class="k">as</span> <span class="n">e</span><span class="p">:</span> <span class="n">log</span><span class="o">.</span><span class="n">error</span><span class="p">(</span><span class="s1">'Unexpected embedding error'</span><span class="p">,</span> <span class="n">error</span><span class="o">=</span><span class="nb">str</span><span class="p">(</span><span class="n">e</span><span class="p">))</span> <span class="k">return</span> <span class="kc">None</span> </code></pre></div> <p>Rate limit tiers (as of 2024): - Tier 1: 500 RPM, 200K TPM for embeddings - Each batch of 100 texts uses ~1 request</p> <p>Key points: - Retry on RateLimitError, TimeoutError, ConnectionError (transient) - Do NOT retry on AuthenticationError, InvalidRequestError (permanent failures) - tenacity wait_exponential: starts at 2s, doubles, caps at 60s - Log before_sleep to monitor retry frequency in production - Track token usage to stay within tier limits proactively</p>