Why I stopped rotating active log files in Python

Log rotation is usually treated as a solved problem.

Write to app.log.

When it gets too large, or when the day changes, rename it to something else.

Then open a new app.log.

That model works well enough in many deployments. If it works for yours, I would not replace it just for the sake of replacing it.

But while building a local-file logging library for Python, I became more interested in a narrower problem:

What happens when the file you rotate is still the file being written to?

A log file is not just a path. It is also an open file handle, a writer state, and sometimes a coordination point between processes.

That is the detail that changed how I think about local-file logging.

The Windows case: rename can fail

On Windows, renaming an active log file can fail if another handle still has it open in a way that blocks the rename.

That can happen more often than people expect:

• a service process still owns the file
• another process is tailing or collecting it
• antivirus or backup software touches it at the wrong time
• multiple workers are writing to what looks like the same log destination

In a rotation-based design, the fragile moment is the boundary where the logger tries to rename, truncate, or reopen the active file.

If that operation fails, the logger now has to decide what failure means. Should the application fail? Should the log record be dropped? Should the handler retry? Should the old file continue receiving records? Should the error be visible or hidden?

The practical issue is that a logging policy decision has become a file-handle coordination problem.

The POSIX case: rename can succeed and still surprise you

On POSIX systems, the failure mode is different.

A rename usually succeeds even if another process still has the old file open. That sounds better, and often it is.

But pathnames and file descriptors are not the same thing.

If a writer still holds the old file descriptor, it can keep appending to the file that has already been moved away from the active path.

So the filesystem operation can succeed while new records still go to a file you no longer think of as current.

For a one-process script, that may not matter much. For long-running services, collectors, external rotation tools, or multiprocess workloads, it is the kind of behavior I would rather not build around unless I have to.

The alternative: route instead of rotate

The model I ended up preferring is:

Do not rename the active file. Open the next destination file instead.

I call this append-only routing.

At a routing point, such as a size boundary or a date boundary, the logger opens the next destination file and appends there. The active file is not renamed, truncated, or reopened as part of that transition.

For example, with daily routing:

• today writes to one dated log file
• tomorrow writes to the next dated log file
• no midnight rename of the active file is required

This is the core idea behind a Python library I built called D-SafeLogger:

https://github.com/nightmarewalker/D-SafeLogger

The README has the full feature overview, so I will keep this article focused on the design reason: avoid making the active file itself the thing that must be mutated at the routing boundary.

The broader idea is not new

I do not think the broad idea of switching future log output to a different file is new.

Other logging ecosystems have treated file sets and time-based destination names as normal production logging concerns for a long time. Logback is a good example on the Java side. Serilog and NLog are common examples on the .NET side.

I do not mean that these libraries all use the same implementation model as D-SafeLogger. They do not.

The shared idea is narrower:

at a log boundary, future records can be written to a newly selected destination file instead of treating the currently active file as something that must be renamed afterward.

That boundary is the part I wanted to make central in Python.

The part I wanted was more specific:

make rename-free routing the normal file-lifecycle model of a stdlib-compatible Python logger.

D-SafeLogger still provides a compatibility path for external rotators. If an existing deployment really needs an external rename/create cycle, the application can configure routing_mode="none" and call ReopenLogFiles() afterward.

But that is not the design center.

The design center is that D-SafeLogger-owned routing does not rename, truncate, or reopen the active file at the routing boundary. It opens the next destination file and appends there.

So the novelty is not "nobody has ever thought about switching log files before." The novelty is putting rename-free routing at the center of a zero-dependency, stdlib-compatible Python logger that keeps stdlib logging.getLogger() / logger.info() call sites and extends the same ownership model into multiprocess logging.

Minimal example

The basic call site is intentionally close to normal Python logging style.

from dsafelogger import ConfigureLogger, GetLogger

ConfigureLogger(log_path="./logs", pg_name="MyApp")

logger = GetLogger(__name__)
logger.info("Application started")

With the default routing_mode="none", this writes to:

./logs/MyApp.log

Here, pg_name is the application name used as the log-file prefix: MyApp.log in this default setup, and later MyApp_20260403.log when daily routing is enabled.

The distribution name is d-safelogger; the import name is dsafelogger.

A typical text line looks like this:

2026-04-03 09:15:22.738 [INF][app.py:6:<module>] Application started

The application code keeps ordinary logging-style calls. The file lifecycle policy lives behind the logger configuration.

That was important to me. I did not want a logging library that required every application and dependency to move to a new logging API before it could participate.

What this changes for application code

The practical benefit is not only the file-routing model.

It is that application code does not have to grow along with the logging policy.

In development, the setup can stay small:

ConfigureLogger(log_path="./logs", pg_name="MyApp")

That writes to the plain local file:

./logs/MyApp.log

Later, the startup configuration can add local-file policy:

ConfigureLogger(
    log_path="./logs",
    pg_name="MyApp",
    routing_mode="daily",
    structured=True,
    enable_hash=True,
)

With daily routing, the destination becomes a dated file such as:

./logs/MyApp_20260403.log

The call sites do not change:

logger = GetLogger(__name__)
logger.info("Application started")

That is the part I care about as a developer.

I want logging to start as something I can add without ceremony, but I do not want the early call sites to become throwaway code. If the application later needs JSON Lines, daily routing, environment overrides, integrity sidecars, masking, diagnostics, async handoff, or multiprocess file ownership, those should be startup policy changes, not a rewrite of every logger.info(...) call.

Why this matters more with multiple processes

Single-process logging can already have lifecycle problems, but multiprocess logging makes ownership much more important.

If each worker process opens the same shared log file, then the file is no longer owned by one clear writer. You now have several processes with their own file handles, buffering behavior, failure modes, and shutdown timing.

That is exactly the kind of situation where I do not want each worker making file lifecycle decisions independently.

D-SafeLogger separates the design into capture, transport, and sinks.

Capture is the application-facing side. It stays close to Python's stdlib logging model: logger.info(...), log records, and call sites that look familiar.

Transport moves records from the caller to the writer. In a simple process, that can be direct. In a multiprocess setup, it goes over IPC.

Sinks own the files. They own file handles, routing, append-only destination selection, optional sidecars, and manifests.

In multiprocess mode, workers keep normal logging-style calls, but a parent-side Writer owns the sinks. Workers do not each open the shared log files.

The point is not that every application needs this extra structure. The point is that, once multiple processes are involved, "who owns the log file?" becomes a real design question.

A missing log line should not be the only signal

Another thing I wanted to avoid is this failure mode:

A record was attempted, but the only visible result is that there is no line in the file.

That is not enough information.

There are different kinds of missing records. A record may be rejected for a known policy or setup reason. It may be dropped under a bounded delivery rule. Or the system may have expected delivery and still be unable to account for the record.

D-SafeLogger names these outcomes:

• KnownRejected
• KnownDropped
• UnexplainedLost

The names are intentionally plain.

KnownRejected means the record was refused due to a known policy, setup, or sink condition.

KnownDropped means the system knowingly dropped the record under a bounded delivery rule.

UnexplainedLost is the category I want to be rare and suspicious. It means the system expected delivery but cannot account for the record.

This is mostly a multiprocess concern. In a single-process crash, the application, transport, and sink can all die together. There may be no surviving authority left to classify what happened.

In a multiprocess setup, the shape is different. A worker can die while the parent-side Writer continues. A queue can hit backpressure. A sink can reject writes. Some workers can detach cleanly while others fail.

That is where delivery-state accounting becomes useful.

Benchmarks: performance sanity and resilience evidence

I included benchmarks in the project, but D-SafeLogger is not designed as a speed-first logger.

The main design target is file-lifecycle clarity: stdlib-compatible capture, append-only routing, explicit sink ownership, bounded delivery behavior, and visible delivery-state classification.

That said, performance still matters. A clean design should not be unnecessarily slow.

The single-process async results are useful in that sense. They show that the design does not require a large performance penalty in ordinary local-file logging. In the selected benchmark run, D-SafeLogger's async mode was very competitive and was among the strongest results in several cells.

I read that result as a consequence of the architecture, not as the purpose of the architecture. Capture and transport are separated cleanly, the caller hands off records quickly, and file output is handled behind that boundary.

The result matters because it shows that the cleaner handoff model does not make ordinary local-file logging impractically slow.

The multiprocess benchmark should be read differently.

There, D-SafeLogger intentionally pays costs for IPC, parent-side Writer ownership, route dispatch, delivery accounting, and bounded shutdown behavior. The multiprocess claim is not raw throughput.

The more important multiprocess benchmark is the resilience profile.

The question there is:

Can the backend account for what happened to attempted records under operational stress?

The resilience benchmark includes cases such as temporary sink unavailability, burst backpressure, worker crash, mixed worker behavior, and shutdown behavior.

In the selected resilience run, D-SafeLogger produced classified loss/reject/drop fields for 16/16 selected summary rows and fully explained 16/16 of them. For example, when the sink was temporarily unavailable, attempted records were classified as KnownRejected rather than disappearing into an ambiguous gap.

That is the benchmark claim I care about most:

not "this logger is the fastest," but "the design remains observable when the logging path is stressed."

Benchmark details are here:

https://github.com/nightmarewalker/D-SafeLogger/blob/main/BENCHMARK.md

Where this model fits

Append-only routing is a local-file logging model.

It fits best when local log files are part of the operational artifact: services, scheduled jobs, long-running processes, audit-oriented logs, or multiprocess workers where file ownership and delivery-state visibility matter.

It is less relevant when stdout or stderr is already the operational boundary and an external collector owns routing, retention, aggregation, durability, and shipping.

It also does not make unusual storage semantics safe. I would not use it as an argument that NFS, SMB, FUSE, cloud-synced folders, or unreliable storage are suddenly good places for active logs.

For audit-oriented deployments, the model I prefer is:

1. write active logs to a durable local filesystem
2. close the file
3. verify, archive, or transfer the closed file afterward

D-SafeLogger is not a log shipper, tracing backend, metrics pipeline, or access-control system. It is focused on the local-file boundary before those systems take over.

The boundary I chose

Python already gives us many useful logging primitives: handlers, filters, QueueHandler, QueueListener, dictConfig, and so on.

D-SafeLogger is not a rejection of those primitives. It is a local-file logging policy built on top of stdlib logging.

The design boundary I chose is this:

do not mutate the file currently being written to as part of normal routing.

When D-SafeLogger owns routing, the next record goes to the next selected file. The active file is not renamed, truncated, or reopened at that boundary.

That does not make logging universally safe, and it does not replace collectors, backups, storage policy, or operational discipline.

But it removes one lifecycle boundary from the critical path: the moment where a logger tries to change the identity of the file it is actively writing.

Project links:

• GitHub / README: https://github.com/nightmarewalker/D-SafeLogger
• PyPI: https://pypi.org/project/d-safelogger/
• Landing page: https://nightmarewalker.github.io/D-SafeLogger/
• Examples: https://github.com/nightmarewalker/D-SafeLogger/tree/main/examples

I would be especially interested in feedback on the append-only routing model.

Note: I am a non-native English speaker. I used AI assistance to polish the English and structure of this article, but the project, design decisions, benchmarks, and responsibility for the content are mine.