Python logging with structlog in production

<p>Using Python's standard logging module but log output is unstructured text that's hard to search in log aggregation tools (Datadog, Grafana Loki, CloudWatch). Need structured JSON logs with consistent fields like request_id, user_id, duration.</p>

<p>Configure structlog for structured JSON output with context binding:</p> <div class="highlight"><pre><span></span><code><span class="c1"># app/logging.py</span> <span class="kn">import</span><span class="w"> </span><span class="nn">structlog</span> <span class="kn">import</span><span class="w"> </span><span class="nn">logging</span> <span class="kn">import</span><span class="w"> </span><span class="nn">sys</span> <span class="k">def</span><span class="w"> </span><span class="nf">configure_logging</span><span class="p">(</span><span class="n">debug</span><span class="p">:</span> <span class="nb">bool</span> <span class="o">=</span> <span class="kc">False</span><span class="p">)</span> <span class="o">-&gt;</span> <span class="kc">None</span><span class="p">:</span> <span class="n">shared_processors</span> <span class="o">=</span> <span class="p">[</span> <span class="n">structlog</span><span class="o">.</span><span class="n">contextvars</span><span class="o">.</span><span class="n">merge_contextvars</span><span class="p">,</span> <span class="c1"># Merge bound context</span> <span class="n">structlog</span><span class="o">.</span><span class="n">processors</span><span class="o">.</span><span class="n">add_log_level</span><span class="p">,</span> <span class="n">structlog</span><span class="o">.</span><span class="n">processors</span><span class="o">.</span><span class="n">TimeStamper</span><span class="p">(</span><span class="n">fmt</span><span class="o">=</span><span class="s1">'iso'</span><span class="p">,</span> <span class="n">utc</span><span class="o">=</span><span class="kc">True</span><span class="p">),</span> <span class="n">structlog</span><span class="o">.</span><span class="n">stdlib</span><span class="o">.</span><span class="n">add_logger_name</span><span class="p">,</span> <span class="p">]</span> <span class="k">if</span> <span class="n">debug</span><span class="p">:</span> <span class="c1"># Human-readable in development</span> <span class="n">processors</span> <span class="o">=</span> <span class="n">shared_processors</span> <span class="o">+</span> <span class="p">[</span> <span class="n">structlog</span><span class="o">.</span><span class="n">dev</span><span class="o">.</span><span class="n">ConsoleRenderer</span><span class="p">()</span> <span class="p">]</span> <span class="k">else</span><span class="p">:</span> <span class="c1"># JSON in production</span> <span class="n">processors</span> <span class="o">=</span> <span class="n">shared_processors</span> <span class="o">+</span> <span class="p">[</span> <span class="n">structlog</span><span class="o">.</span><span class="n">processors</span><span class="o">.</span><span class="n">dict_tracebacks</span><span class="p">,</span> <span class="n">structlog</span><span class="o">.</span><span class="n">processors</span><span class="o">.</span><span class="n">JSONRenderer</span><span class="p">(),</span> <span class="p">]</span> <span class="n">structlog</span><span class="o">.</span><span class="n">configure</span><span class="p">(</span> <span class="n">processors</span><span class="o">=</span><span class="n">processors</span><span class="p">,</span> <span class="n">wrapper_class</span><span class="o">=</span><span class="n">structlog</span><span class="o">.</span><span class="n">make_filtering_bound_logger</span><span class="p">(</span><span class="n">logging</span><span class="o">.</span><span class="n">DEBUG</span> <span class="k">if</span> <span class="n">debug</span> <span class="k">else</span> <span class="n">logging</span><span class="o">.</span><span class="n">INFO</span><span class="p">),</span> <span class="n">context_class</span><span class="o">=</span><span class="nb">dict</span><span class="p">,</span> <span class="n">logger_factory</span><span class="o">=</span><span class="n">structlog</span><span class="o">.</span><span class="n">PrintLoggerFactory</span><span class="p">(</span><span class="n">file</span><span class="o">=</span><span class="n">sys</span><span class="o">.</span><span class="n">stdout</span><span class="p">),</span> <span class="n">cache_logger_on_first_use</span><span class="o">=</span><span class="kc">True</span><span class="p">,</span> <span class="p">)</span> <span class="c1"># Usage in FastAPI middleware — bind request context</span> <span class="kn">from</span><span class="w"> </span><span class="nn">starlette.middleware.base</span><span class="w"> </span><span class="kn">import</span> <span class="n">BaseHTTPMiddleware</span> <span class="kn">import</span><span class="w"> </span><span class="nn">uuid</span> <span class="k">class</span><span class="w"> </span><span class="nc">RequestLoggingMiddleware</span><span class="p">(</span><span class="n">BaseHTTPMiddleware</span><span class="p">):</span> <span class="k">async</span> <span class="k">def</span><span class="w"> </span><span class="nf">dispatch</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">request</span><span class="p">,</span> <span class="n">call_next</span><span class="p">):</span> <span class="n">request_id</span> <span class="o">=</span> <span class="nb">str</span><span class="p">(</span><span class="n">uuid</span><span class="o">.</span><span class="n">uuid4</span><span class="p">())[:</span><span class="mi">8</span><span class="p">]</span> <span class="n">structlog</span><span class="o">.</span><span class="n">contextvars</span><span class="o">.</span><span class="n">clear_contextvars</span><span class="p">()</span> <span class="n">structlog</span><span class="o">.</span><span class="n">contextvars</span><span class="o">.</span><span class="n">bind_contextvars</span><span class="p">(</span> <span class="n">request_id</span><span class="o">=</span><span class="n">request_id</span><span class="p">,</span> <span class="n">method</span><span class="o">=</span><span class="n">request</span><span class="o">.</span><span class="n">method</span><span class="p">,</span> <span class="n">path</span><span class="o">=</span><span class="n">request</span><span class="o">.</span><span class="n">url</span><span class="o">.</span><span class="n">path</span><span class="p">,</span> <span class="p">)</span> <span class="n">logger</span> <span class="o">=</span> <span class="n">structlog</span><span class="o">.</span><span class="n">get_logger</span><span class="p">()</span> <span class="n">logger</span><span class="o">.</span><span class="n">info</span><span class="p">(</span><span class="s1">'request_started'</span><span class="p">)</span> <span class="n">response</span> <span class="o">=</span> <span class="k">await</span> <span class="n">call_next</span><span class="p">(</span><span class="n">request</span><span class="p">)</span> <span class="n">logger</span><span class="o">.</span><span class="n">info</span><span class="p">(</span><span class="s1">'request_completed'</span><span class="p">,</span> <span class="n">status_code</span><span class="o">=</span><span class="n">response</span><span class="o">.</span><span class="n">status_code</span><span class="p">)</span> <span class="k">return</span> <span class="n">response</span> <span class="c1"># In route handlers</span> <span class="n">logger</span> <span class="o">=</span> <span class="n">structlog</span><span class="o">.</span><span class="n">get_logger</span><span class="p">()</span> <span class="k">async</span> <span class="k">def</span><span class="w"> </span><span class="nf">create_trace</span><span class="p">(</span><span class="n">trace</span><span class="p">:</span> <span class="n">TraceCreate</span><span class="p">,</span> <span class="n">user</span><span class="p">:</span> <span class="n">User</span><span class="p">)</span> <span class="o">-&gt;</span> <span class="n">Trace</span><span class="p">:</span> <span class="n">logger</span><span class="o">.</span><span class="n">info</span><span class="p">(</span><span class="s1">'creating_trace'</span><span class="p">,</span> <span class="n">user_id</span><span class="o">=</span><span class="nb">str</span><span class="p">(</span><span class="n">user</span><span class="o">.</span><span class="n">id</span><span class="p">),</span> <span class="n">title</span><span class="o">=</span><span class="n">trace</span><span class="o">.</span><span class="n">title</span><span class="p">)</span> <span class="n">result</span> <span class="o">=</span> <span class="k">await</span> <span class="n">db</span><span class="o">.</span><span class="n">insert</span><span class="p">(</span><span class="n">trace</span><span class="p">)</span> <span class="n">logger</span><span class="o">.</span><span class="n">info</span><span class="p">(</span><span class="s1">'trace_created'</span><span class="p">,</span> <span class="n">trace_id</span><span class="o">=</span><span class="nb">str</span><span class="p">(</span><span class="n">result</span><span class="o">.</span><span class="n">id</span><span class="p">))</span> <span class="k">return</span> <span class="n">result</span> </code></pre></div> <p><code>merge_contextvars</code> automatically includes context bound via <code>bind_contextvars()</code> in every log line — no need to pass logger/context around. Production JSON logs: <code>{"event": "request_completed", "status_code": 200, "request_id": "abc123", "timestamp": "2024-01-01T..."}</code>.</p>