Even a Single Log Line Needs a Convention

Half of the teams that adopt structured logging collapse along the same path. They swap the log library for something like pino or slog, and mark the moment the first JSON line is emitted as their "success point." A month later, the moment they query the logs to build a monitoring dashboard, they realize that memory was wrong. JSON only provides structure; what goes inside that structure is entirely up to the people using it.

The library swap takes a day. Level criteria and field conventions still haven't been agreed on two months later. Four points keep teams stuck the longest during the migration: level, fields, masking, and cost. Each looks independent, but they're all connected.

There is no boundary line between error and warn

Most level debates erupt over "errors that are recoverable." If a payment retry fails once and automatically enters a retry loop, it's not an error even if a status 500 is logged. Conversely, a path that returns 200 but responds three seconds late is worth a warn. Status code and level are not directly proportional. Unless that premise is established, error quickly converges to "a slightly worse warn."

Most of the problem dissolves if you set level criteria by "whether a human needs to act" rather than "the size of the problem." info is a line you read and have nothing to do with, warn is a line the system recovers from on its own but deserves attention, and error is a line where a person needs to intervene. If you hold a session where everyone reviews each other's logs against these criteria, roughly half of the lines classified as error drop down to warn. A few real examples build the criteria better than a written convention ever will.

For cases that are still ambiguous, you can temporarily use mechanical thresholds. Something like warn = status code 400+ or P99 latency exceeded. That's better than having no rule at all. A month later, adjust based on the actual noise. The point that's easy to miss is the control of debug. Send all library-internal logs to debug, and keep the default level in production at info, so whatever ends up in debug is safe from a cost standpoint.

How far should the request ID flow

The first task in field design is settling on a common field set. Fix timestamp to ISO 8601 in UTC, and make service, level, and msg required. Many teams add environment and version on top. What collapses before the schema is naming. The moment user_id and userId coexist in one service, log search splits into two branches. Create a single field dictionary, and require that new fields be added only by registering them in that dictionary. If msg also follows a rule of starting with a verb, like "action: details," the search syntax stays simple.

Request IDs are the trickiest field convention to propagate. The part where the HTTP middleware creates the ID isn't hard. The real problem comes after. The ID breaks when work moves to async handlers, message queues, and batch jobs. The request ID must be carried in the message at the moment it's enqueued, and batch jobs are cleaner off with their own batch ID. Logs with a broken ID can't be traced through correlations even at a 99.9% collection rate. This item should be a top priority at the design stage.

Mask at the data entry point, not in the logger

The principle is to strip sensitive information before it reaches the log. Attaching a masking option to the logger is only a second line of defense. Because the call sites are not singular. If a query string lands whole inside an exception message, a logger filter won't remove it, and you also can't stop an external library from printing a request body at debug level.

The most common leak is the URL. When access tokens ride along as query parameters, that's not a problem to solve with log filters. Establish a rule that query parameters are never recorded in logs, and note the pattern of tokens appearing in queries in the incident manual. And masking verification should be automated. A single script that generates real password strings in a test environment and checks whether they appear in the logs closes the gap between "believing masking exists" and "masking actually working."

You also have to decide which layer performs the masking. The recommendation is to do it in the layer that creates the log. Leave it to the log output layer and the filter list gets split across two places for management. And masking rules need a designated "owner for verification." A rule with no owner never gets updated.

JSON is more expensive than text

The first thing you feel after the migration is the weight of the logs. A single JSON line with 20 fields runs around 500 bytes, and at thousands of requests per second that piles up to tens of GB per day. It's several times heavier than text logs, and it costs money at every stage: collection, storage, and search.

Half of the cost comes from logs that shouldn't be emitted at all. Even one retry loop that logs at info level repeats the same line every 500ms. Health checks and keep-alive requests should be excluded from logging by default, and high-volume paths need a judgment call on applying sampling. Settle on the rule of one event per line ahead of time. If you stuff a raw stack trace into a JSON value, parsers cut lines in the wrong places, and merge logic gets added to the collection pipeline. At that point, log format is no longer a developer-team problem; it's an infrastructure problem.

Storage-layer design is also part of the agreement. The common structure keeps the last 7–30 days on fast search and moves the rest to low-cost storage. Unlimited retention is a liability, not an asset. If you put per-service log volume and level distribution on a dashboard, when one service suddenly jumps 10x, you can spot the cause before diving into code analysis. The recurring story of bumping to debug temporarily for bug tracking and never turning it back down, sending costs through the roof, gets caught in large part if level changes are part of code review.

Logs get reviewed along with the code

Validating the migration doesn't end after two or three deploys. Pick one core transaction and trace it end to end to confirm it carries a single request ID across every stage. Confirm that field autocomplete shows up in the search tool and that the time-range filter behaves as intended. The criterion for a successful migration isn't "JSON is being emitted"; it's "queries return the answer you want."

The last item to agree on is about people. Log lines are easy to gloss over in a PR. A team that ignores log additions in code review has a scattered field convention within six months. This is a process problem, not a tool problem.

The chain is: ambiguous level criteria inflate error counts, more errors bury the alerts, and logs nobody reads become the clue in a security incident. Start by writing a single document — level criteria, common fields, masking layer, and retention period — before you install the library. The log library is just a tool for executing that document.

Comments

Loading comments.