# Systemd Services for Background Jobs: The Right Way to Run Node.js in Production

> How to write production-ready systemd service files for Node.js, NestJS, and background workers — with restart policies, resource limits, and environment variable management.

URL: https://www.matthewswong.com/en/blog/systemd-services-jobs/

- Author: Matthews Wong (https://www.matthewswong.com/en/about/)
- Published: 2025-06-10
- Updated: 2026-10-17
- Category: DevOps

Systemd manages over 70% of Linux systems in production and is the standard init system for Ubuntu, Debian, CentOS, and every major Linux distribution. Yet most tutorials for running Node.js apps in production still recommend pm2 — a userspace process manager that duplicates functionality already built into systemd. Migrating NestJS background workers from pm2 to native systemd services gives you automatic boot startup, structured logging via journald, resource limit enforcement, and better integration with monitoring tools.

## Why Systemd Over pm2

pm2 is excellent for development and prototyping but adds an unnecessary layer in production. Systemd handles: automatic startup at boot, automatic restart on crash with configurable backoff, CPU and memory limits enforced by the kernel, structured logs via journald accessible with journalctl, and integration with systemctl for status and control. The single advantage pm2 retains is cluster mode for multi-process Node.js — but for NestJS, this is better handled by running multiple Docker containers behind Nginx.

### Anatomy of a Service File

A systemd service file lives in /etc/systemd/system/your-service.service and has three sections: [Unit] describes the service and its dependencies (After=network.target ensures the network is up before starting), [Service] defines how to run the service, and [Install] determines when the service starts (WantedBy=multi-user.target is the production default that starts the service in normal multi-user mode). Restart=on-failure restarts on crashes but not on clean exits — the right behavior for production.

### Environment Variables in Systemd

Never hardcode secrets in service files. Use EnvironmentFile=/etc/your-service/.env to load environment variables from a file with 640 permissions owned by root and your service user. This file is readable by the service user and root only, not world-readable like a .env file in the application directory. Alternatively, use Environment= directives for non-sensitive config: Environment=NODE_ENV=production. Combine both approaches: EnvironmentFile for secrets, Environment for configuration.

> When running NestJS services under systemd, use StandardOutput=journal+console and StandardError=journal+console directives to send both stdout/stderr to journald AND to the console simultaneously. This means you get structured log storage with journalctl while still seeing output in systemctl status. Add a SyslogIdentifier=your-service-name directive and logs are tagged with your service name — critical for filtering when you have 10+ services all writing to journald.

## Restart Policies and Backoff

Systemd restart behavior is controlled by Restart=, RestartSec=, and StartLimitIntervalSec= / StartLimitBurst=. For production services: Restart=on-failure restarts on non-zero exit codes, RestartSec=5s waits 5 seconds between restarts (prevents tight crash loops from hammering the CPU), StartLimitIntervalSec=60 and StartLimitBurst=5 means if the service restarts 5 times in 60 seconds, systemd gives up and puts it in a failed state — triggering an alert and requiring manual intervention. This is safer than infinite restart loops that can mask underlying issues.

### Resource Limits and Security Hardening

Systemd can enforce resource limits directly in the service file: MemoryMax=512M (OOM-kills the service if it exceeds 512MB), CPUQuota=50% (limits to half of one CPU core), LimitNOFILE=65535 (file descriptor limit for the service), and security hardening directives: PrivateTmp=true (isolated /tmp), ProtectSystem=strict (read-only filesystem except /var /run /tmp), NoNewPrivileges=true (prevents privilege escalation), and User=nestjs (run as a dedicated non-root user). These hardening directives reduce the blast radius of a compromised service.

> **Systemd After= Does Not Mean Ready=**

> I burned several hours debugging a service that was starting before its PostgreSQL dependency was ready. After=postgresql.service only guarantees ordering — it does not check if PostgreSQL is actually ready to accept connections. For a NestJS app that needs PostgreSQL, use After=postgresql.service plus application-level retry logic: configure TypeORM or Prisma with retryAttempts and retryDelay to handle temporary connection failures on startup. Do not rely on systemd dependency ordering alone for service readiness — it only controls start order, not readiness.

## Managing Services with systemctl

Essential systemctl commands: systemctl start/stop/restart your-service, systemctl enable your-service (starts on boot), systemctl status your-service (shows last 10 log lines plus PID and resource usage), systemctl daemon-reload (required after editing service files), and journalctl -u your-service -f (tail logs for the service). For debugging crashes: journalctl -u your-service --since today shows all log output since midnight, and journalctl -u your-service -p err shows only error-level logs.

## Systemd Timers: Cron Replacement

Systemd timers are a modern replacement for cron jobs, with better logging, dependency management, and missed-job handling. Create a .service file describing what to run and a .timer file describing when to run it. Timers appear in systemctl status output, missed jobs are tracked, and output goes to journald. Use OnCalendar=daily for once-daily jobs, OnBootSec=15min OnUnitActiveSec=1h for hourly jobs starting 15 minutes after boot. The main advantage over cron: timer logs tell you exactly when the job ran, how long it took, and whether it succeeded.

## Production Service Template for NestJS

My production NestJS systemd service template includes: User=nestjs (dedicated service account), WorkingDirectory=/opt/nestjs/app, ExecStart=/usr/bin/node dist/main.js, EnvironmentFile=/etc/nestjs/.env, Restart=on-failure with RestartSec=10s and StartLimitBurst=3, MemoryMax=512M, LimitNOFILE=65535, NoNewPrivileges=true, PrivateTmp=true, StandardOutput=journal+console, and SyslogIdentifier=nestjs-app. This template handles 95% of NestJS production deployments and provides restart resilience, resource limits, and proper logging.

## Sources & Further Reading

- Linux Expert — 7 Essential Advanced Systemd Service Unit Best Practices
- DEV Community — How to Manage Background Services with systemctl and systemd
- Akash Rajpurohit — Keep Your Services Running in the Background with SystemD

## Source links

- https://www.linuxoperatingsystem.net/systemd-service-unit-best-practices
- https://dev.to/chetansingh63/how-to-manage-background-services-with-systemctl-and-systemd-with-celery-example-320f
- https://akashrajpurohit.com/blog/keep-your-services-running-in-the-background-with-systemd/

## FAQ

### Why use systemd instead of pm2 to manage Node.js and NestJS processes in production?

pm2 is well-suited for development and prototyping, but in production it adds an unnecessary userspace layer that duplicates functionality already built into systemd. Systemd natively handles automatic startup at boot, crash restarts with configurable backoff, kernel-enforced CPU and memory limits, and structured logging via journald — without any additional dependency.

### How should environment variables and secrets be managed in a systemd service file?

Never hardcode secrets directly in the service file. Instead, use the EnvironmentFile directive pointing to a file (e.g. /etc/your-service/.env) with 640 permissions, owned by root and the service user, so it is not world-readable like a .env file inside the application directory. Non-sensitive config values can use the Environment= directive directly, and both approaches can be combined in the same service file.

### What restart policy settings are recommended for a production NestJS service?

The recommended combination is Restart=on-failure (restarts only on non-zero exit codes, not clean exits), RestartSec=5s (adds a 5-second delay to prevent tight crash loops from hammering the CPU), and StartLimitIntervalSec=60 with StartLimitBurst=5 (if the service restarts 5 times within 60 seconds, systemd stops trying and puts it in a failed state, requiring manual intervention). This prevents infinite restart loops that can mask underlying problems.

### Does After=postgresql.service in a systemd unit guarantee that PostgreSQL is ready to accept connections?

No — After=postgresql.service only controls start order, not service readiness. It ensures your service starts after the PostgreSQL unit, but PostgreSQL may not yet be ready to accept connections at that point. The correct approach is to combine After=postgresql.service with application-level retry logic, such as configuring TypeORM or Prisma with retryAttempts and retryDelay to handle temporary connection failures at startup.

### How do systemd timers compare to traditional cron jobs for scheduled background tasks?

Systemd timers offer better logging, dependency management, and missed-job handling compared to cron. Each timer is paired with a .service file (what to run) and a .timer file (when to run it), and output goes to journald — so you can see exactly when the job ran, how long it took, and whether it succeeded. Timers also appear in systemctl status output and track missed jobs, which cron does not do natively.

---

- HTML version: https://www.matthewswong.com/en/blog/systemd-services-jobs/
- Bahasa Indonesia: https://www.matthewswong.com/id/blog/systemd-services-jobs/
- Site index for agents: https://www.matthewswong.com/llms.txt
- Sitemap: https://www.matthewswong.com/sitemap.xml
- API description (OpenAPI): https://www.matthewswong.com/openapi.json