DevOps
Log Rotation in Linux: Managing Production Logs Before They Fill Your Disk
February 20269 min read
DevOps
Log files grow without bound until they fill your disk and crash your server. On a busy NestJS API at Commsult Indonesia, a single day of verbose logging can generate several gigabytes of log files. An unmanaged /var/log directory on a DigitalOcean Droplet has ended production services for many developers — the disk fills silently, write operations start failing, and applications crash in confusing ways. Logrotate, the standard Linux log rotation tool, prevents this entirely and requires about 10 minutes to configure properly.
Logrotate is a utility designed to ease administration of systems that generate large numbers of log files. It runs as a daily cron job (via /etc/cron.daily/logrotate) and processes configuration files in /etc/logrotate.d/. For each log file, it applies the configured rotation policy: renaming the current log, creating a new empty one, optionally compressing old rotations, and deleting old rotations beyond the retention count. Most system services (Nginx, syslog, PostgreSQL) install their own logrotate configs in /etc/logrotate.d/ automatically.
Key logrotate options: daily/weekly/monthly (rotation frequency), rotate N (keep N rotated copies), compress (gzip old logs, typically 5-10x size reduction), delaycompress (compress the previous rotation but not the most recent — important for services that may still be writing to the rotated file), copytruncate (copy then truncate instead of rename/create — needed for apps that cannot reopen log files), and postrotate/endscript (run a command after rotation, e.g., nginx -s reopen).
Place per-application logrotate configs in /etc/logrotate.d/. For a NestJS application writing to /var/log/nestjs/app.log: configure daily rotation, keep 30 days of rotations, compress all but the most recent, and send SIGUSR1 to the Node.js process to reopen the log file (using the postrotate directive). Without the SIGUSR1 signal, Node.js continues writing to the old (renamed) file handle and the new log file stays empty — a common misconfiguration.
┌─────────────────────────────────────────────────────┐
│ LOG ROTATION LIFECYCLE │
└─────────────────────────────────────────────────────┘
app.log (current, 500MB)
│
▼ [logrotate triggers]
app.log.1 (renamed) ← app writes here briefly
app.log (new, empty) ← postrotate: send SIGUSR1
│
▼ [next day]
app.log.2.gz (compressed) ← delaycompress kicks in
app.log.1 (yesterday)
app.log (current)
│
▼ [after rotate 30]
app.log.30.gz (oldest)
... older files DELETED
/var/lib/logrotate/status tracks last rotation timeFrom my experience managing log rotation on Commsult Indonesia servers, always use size-based rotation in addition to time-based rotation for high-traffic services. A daily rotation is fine for a 10 MB/day log, but if traffic spikes and you are generating 2 GB/day, you want rotation at 500MB regardless of time. Use size 500M in your logrotate config as an upper bound trigger — it rotates if the file exceeds 500MB even if a daily rotation already happened. This prevents a traffic spike from filling your disk between daily rotation windows.
Always test logrotate configuration before relying on it in production. Run logrotate -d /etc/logrotate.d/your-app for a dry run that shows exactly what would happen without performing any actions. Check the output carefully for errors and verify the correct files are targeted. Force an immediate rotation with logrotate -vf /etc/logrotate.d/your-app (-f forces rotation even if the file does not meet the rotation criteria). Check that the log file is rotated, compressed (after delaycompress runs on the next cycle), and that your application is writing to the new log file correctly.
Docker containers write logs to Docker's own log driver, not to files that logrotate can manage. Configure Docker's JSON file log driver with size and rotation limits in /etc/docker/daemon.json: set log-driver to json-file with log-opts max-size of 50m and max-file of 5. This tells Docker to rotate container logs when they reach 50MB and keep 5 rotations. For Docker Compose, configure the logging driver per service. Do not rely on Docker's default unbounded log accumulation in production.
# /etc/logrotate.d/nestjs-app
/var/log/nestjs/app.log {
daily
size 500M # rotate if exceeds 500MB regardless of time
rotate 30
compress
delaycompress
missingok
notifempty
create 0644 nestjs nestjs
postrotate
kill -USR1 $(cat /var/run/nestjs-app.pid) 2>/dev/null || true
endscript
}
# /etc/docker/daemon.json — Docker log limits
{
"log-driver": "json-file",
"log-opts": {
"max-size": "50m",
"max-file": "5"
}
}
# Test configuration (dry run)
logrotate -d /etc/logrotate.d/nestjs-app
# Force immediate rotation
logrotate -vf /etc/logrotate.d/nestjs-app
# Emergency: disk full recovery
du -sh /var/log/* | sort -rh | head -10
truncate -s 0 /var/log/nestjs/app.logLog rotation failures are silent by default — logrotate does not send alerts when something goes wrong. Add monitoring: check /var/lib/logrotate/status to see when each log file was last rotated, set up a cron job or systemd timer that checks log file sizes and alerts if any single log file exceeds 1GB (indicates rotation is failing), and add disk usage alerts in Grafana/Prometheus to catch growing log directories before they fill the filesystem.
I once configured copytruncate for a NestJS service because I did not want to implement a SIGUSR1 handler. Copytruncate works by copying the log file to a new name and then truncating the original — but there is a race condition where log lines written between the copy and the truncate are lost. For a low-write log, this is acceptable. For a high-traffic API logging every request, you can lose thousands of log entries per rotation. The correct approach is to implement proper log file reopening in your application and use the postrotate signal.
Logrotate manages file-level rotation but good application logging practices reduce the volume that needs rotating. Use structured JSON logging (Winston, Pino) with appropriate log levels — debug logs should not exist in production. Route different severity levels to different files: info to access.log, errors to error.log, enabling different rotation policies per level (keep 30 days of info, 90 days of errors). Integrate application logs with a central log aggregation system (Loki, Elasticsearch) for long-term retention without filling local disk.
When your disk is full and services are crashing, immediate steps: find the largest files with du -sh /var/log/* sorted by size, truncate (not delete) the largest log files with truncate -s 0 /path/to/huge.log (truncating preserves the file descriptor so running applications do not crash), then force logrotate immediately to properly handle the rotation. Fix the root cause: either configure logrotate correctly or add more disk space. On DigitalOcean, expanding a Droplet volume takes under a minute as a temporary measure.