Skip to main content
Web app skills · Lesson 11 of 11 · 13 min

Playwright Smoke Test in GitHub Actions: Serve, Wait, Fail

Add a Playwright smoke test to GitHub Actions: serve the built Next.js app, wait until it answers, install headless Chromium and print the log on failure.

By Matthews WongReviewed on

Key takeaway

A smoke test opens the built app in a real browser and checks that the few things that must work, do. It catches what unit and contract tests cannot: a build that compiles but renders a broken page. In CI you build, start the server in the background, wait for it to answer, run the test, and print the server log if anything fails.

You will be able to

  • Explain what a browser smoke test guards that unit and contract tests cannot.
  • Write a workflow that builds, serves, waits for readiness, tests and shows the server log on failure.
  • Keep the Playwright version and its browser build in step.
Watch it work

Step through the diagram

Press Play, or step with the arrows. Every step is written out below the diagram.

From build to a failing smoke test

A workflow builds the app, starts it in the background with its output in a log file, polls until it answers, runs the browser test, and prints the log when the test fails.

Step 1 of 6. One job runs five things in order: the build, a background server, a wait loop, a headless browser test, and a log file that is only read if something fails.

Keys: ← → step · Home / End · K play or pause

  1. One job runs five things in order: the build, a background server, a wait loop, a headless browser test, and a log file that is only read if something fails.

  2. The build runs first, then the server starts in the background with its output sent to a log file, so the next step can run while it stays up.

    PORT=3000 npm start > server.log 2>&1 &

  3. Starting is not ready. A loop requests the page every two seconds and moves on only when it gets a success response.

    if curl -sf -o /dev/null http://localhost:3000/en/play/; then exit 0; fiPredict the next step The first request arrives before the server is listening. What should the loop do? (Fail the job at once · Sleep, then try again · Start the tests anyway)

  4. With the server answering, a script opens the page in headless Chromium, clicks through the key path and records every check that does not hold.

    BASE_URL=http://localhost:3000 npm run game:smokePredict the next step A check fails, say the canvas never appears. What does the script do? (Passes anyway · Prints FAIL and exits with code 1 · Stops the server)

  5. The script sets a non-zero exit code. The workflow step fails, and later steps that only run on success are skipped.

    process.exitCode = 1;

  6. A final step marked to run on failure prints the log file. The browser says what broke on screen, and the log says why the server broke.

    if: failure()

What does a smoke test guard that unit tests cannot?

A smoke test answers one question: does the thing we are about to ship basically work? It opens the real built app in a real browser and exercises the shortest path through it. Unit and contract tests read data and call functions, so they pass even when the bundle throws on load, a route returns an error page, or a script fails in the browser. Only a run of the built output notices that.

Read more (2)

This repo's smoke script, written for its 3D game page, shows the shape. It checks that the consent notice renders, that no heavy asset files are requested before the visitor accepts, that accepting mounts a canvas that actually produces frames, that a deep link opens the right card, and that the page logged no console errors. Those are user-visible outcomes, which is what the Playwright docs say tests should verify.

Keep it small. A smoke test is a tripwire, not a regression suite: a few checks on the path that matters most, so a failure means something is clearly broken and the run stays short enough to run on every pull request.

How do you start the built app in a workflow and wait for it?

A CI step is a shell command, and a server never exits, so you start it in the background with a trailing ampersand and send its output to a file. Then you wait. Starting a process is not the same as it being ready, so the next step must not assume the port is open. This workflow polls the real page with curl every two seconds, up to sixty times, and treats the first success as ready.

Read more (2)

Polling a real page beats sleeping a fixed time: it continues as soon as the server is ready and never guesses. If the loop runs out, the step prints a message, dumps the server log and exits 1, so a server that never came up fails with its own error message in view instead of a vague timeout in the test step after it.

Why does a failing command stop the job at all? GitHub documents that by default the bash shell for a step runs with fail-fast behaviour, so a non-zero exit code fails the step, and a failed step skips the ones after it unless they say otherwise.

How do you install headless Chromium and keep versions in step?

Playwright downloads its own browser builds. The docs show npx playwright install --with-deps chromium for installing one browser together with the operating system libraries it needs on Linux, and advise installing only the browsers you use, especially on CI. By default Playwright launches browsers headless, so a plain ubuntu runner needs no display.

Read more (3)

The docs also state that each version of Playwright needs specific versions of browser binaries. That is the pinning rule: the browser build is a function of the Playwright package version, so install the browser after the dependencies, from the same lockfile, in the same job. This repo's package.json allows a range, while the lockfile records the exact release the workflow actually gets, and a changed version means a different browser build to download.

What about caching the browser? The CI page of the docs advises against it, saying restoring the cache takes comparable time to a download, and that the Linux system dependencies are not cacheable anyway. If you cache regardless, key the cache on the Playwright version. This workflow caches the npm packages through setup-node and downloads Chromium fresh on each run.

One practical note for 3D pages: with no GPU on the runner, the script launches Chromium with flags that select a software renderer, so WebGL works. That is specific to this game, and a normal page needs no extra flags.

How do you make a failed run explain itself?

The server ran in the background, so its output went to a file and a failed test shows only the browser side. A step with if: failure() fixes that. GitHub documents that failure() is true when any previous step of the job failed, and that a step runs only after success unless you add a status check function to its condition. The last step of this workflow uses it to print the log.

Read more (2)

The test script cooperates. Instead of throwing at the first problem, it records each failure with a clear message, sets the exit code to 1, and carries on, so one run reports every broken check. For failures you cannot reproduce from text, the Playwright docs recommend the trace viewer on CI over videos and screenshots, which you can add on top of the log.

Finally, fence the job in. The workflow cancels an older run of the same branch when a new push arrives, grants only read access to the repository contents, and runs only when relevant paths change. A smoke test that nobody waits for is a smoke test people keep.

How this site does it

The real config behind this lesson

Excerpts from the repository that builds and deploys this website, quoted as they are.

.github/workflows/game-smoke.yml:53-62
      - name: Serve
        run: |
          PORT=3000 npm start > server.log 2>&1 &
          for i in $(seq 1 60); do
            if curl -sf -o /dev/null http://localhost:3000/en/play/; then exit 0; fi
            sleep 2
          done
          echo "server did not come up"
          cat server.log
          exit 1

Start, wait, give up loudly. The trailing ampersand backgrounds the server and 2>&1 sends both streams to server.log. The loop returns on the first successful request, and if sixty tries pass, it prints the log and exits 1 so the job fails here, not later.

.github/workflows/game-smoke.yml:73-75
      - name: Server log on failure
        if: failure()
        run: cat server.log || true

The safety net. failure() overrides the default success-only condition, so this runs after any earlier step failed. The trailing || true keeps the step itself from failing if the log file does not exist yet.

.github/workflows/game-smoke.yml:50-51
      - name: Install Chromium for Playwright
        run: npx playwright install --with-deps chromium

One browser plus its operating system libraries, installed after npm ci so the browser build matches the installed Playwright version. Only Chromium is installed because the test only launches Chromium.

scripts/game/smoke-test.mjs:29-32
function fail(message) {
  console.error(`smoke-test: FAIL — ${message}`);
  process.exitCode = 1;
}

A check that fails without stopping the run. It prints a labelled message and sets the exit code, so every broken check shows up in one pass and the process still ends with a failing status that the workflow step understands.

scripts/game/smoke-test.mjs:35-41
  const browser = await chromium.launch({
    headless: true,
    args: [
      "--use-gl=angle",
      "--use-angle=swiftshader",
      "--enable-unsafe-swiftshader",
    ],
  });

Headless Chromium with a software GL renderer, because a CI runner has no GPU and this page draws WebGL. The args are for this 3D game only; an ordinary page would call launch with headless alone.

package.json
"playwright": "^1.60.0",

The dependency allows a range. The package-lock.json beside it records the exact release that gets installed, and the browser build to download follows that version. Tighten the range if you need the same browser build on every machine.

What breaks

Mistakes that cost real time

  • Tests start before the server is ready

    Launching the server and running the test in consecutive lines races the port opening, and the first run passes or fails by luck. Poll a real URL until it answers, and fail the wait step with the log if it never does.

  • Browser build missing after a version bump

    Each Playwright version needs specific browser binaries. If a dependency range moves Playwright to a release whose browser build is not on the machine, launching fails until you run the install command again. In CI the install step runs every time, which is why it works there.

  • A failure with no server log

    The server's output went to a file, so without a step that prints it a red run says only that a browser check failed. Add a step with if: failure() that prints the log.

  • A smoke test that grows into a flaky suite

    Every extra path adds more ways to fail for reasons unrelated to the change. Keep the smoke test to the shortest path that proves the app is alive, and test depth elsewhere. Stub third-party responses so an outside site cannot turn the run red.

What each kind of test can see
Runs againstSource modules and dataThe built, served app
CatchesWrong logic, broken data rulesA page that errors, a script that throws, a missing asset
NeedsNode onlyA build, a running server and a browser
Try it yourself

Do it on your own machine

Add a smoke test to a Next.js project and run it the way CI would, on your own machine first.

  1. Install Playwright as a dependency and download only Chromium with its system libraries.

    npm install --save-dev playwright
npx playwright install --with-deps chromium

  2. Write scripts/smoke.mjs. Open the home page, assert one visible heading, and set process.exitCode = 1 with a clear message when it is missing.

    import { chromium } from "playwright";

const browser = await chromium.launch({ headless: true });
const page = await browser.newPage();
await page.goto(process.env.BASE_URL ?? "http://localhost:3000");
if (!(await page.getByRole("heading", { level: 1 }).isVisible())) {
  console.error("smoke: FAIL - no h1");
  process.exitCode = 1;
}
await browser.close();

  3. Build, start the server in the background with its output in a file, and wait with a polling loop before you run the script.

    npm run build
PORT=3000 npm start > server.log 2>&1 &
for i in $(seq 1 30); do curl -sf -o /dev/null http://localhost:3000/ && break; sleep 1; done
node scripts/smoke.mjs

  4. Copy the same commands into a workflow. Add a last step with if: failure() that runs cat server.log, then break the page on purpose and read the log in the run.

    - name: Server log on failure
  if: failure()
  run: cat server.log || true
Quick quiz

Three questions before you move on

Pick an answer to see why it is right or wrong. Nothing is saved or graded, so guess freely.

  1. 1.A build compiles and every unit test passes, yet the page renders blank in a browser. Which check catches it?
  2. 2.Why does the workflow poll the page instead of running the test right after starting the server?
  3. 3.Which condition makes the log step run when an earlier step failed?
Think it through

Answer in your head, then check

  1. 1.Why install the browser after the npm dependencies, in the same job?Show the answer

    Answer:Each Playwright version needs specific browser binaries. Installing after the dependencies means the install command reads the version that is actually installed, so the browser build matches it.

  2. 2.The smoke script records a failure and keeps going instead of throwing. What is the benefit?Show the answer

    Answer:One run reports every broken check, and the exit code is still non-zero so the workflow step fails. You fix several problems per push instead of one.

  3. 3.Should you cache the Playwright browser in CI?Show the answer

    Answer:The Playwright CI docs advise against it, because restoring the cache takes comparable time to downloading, and system dependencies are not cacheable. If you do, key it on the Playwright version.

Go deeper

Related deep dives on this site

Sources

Primary sources checked on the research date. Follow them for the current version.

Previous lessonSelf-Host Next.js: Standalone Output, Docker and Env Vars