Building a Streaming Chatbot with Groq and Next.js

Most chatbot demos feel sluggish because they wait for the entire response before showing anything. A production chatbot should feel instant, and the two ingredients that make that possible are a genuinely fast inference provider and a streaming pipeline that pushes tokens to the browser as soon as they exist, rather than buffering the full reply on the server.

This post walks through building that pipeline end to end: a Next.js App Router route handler that calls the Groq API with streaming enabled, a client that reads the response incrementally, a system prompt that keeps the assistant on-topic, and the rate limiting and cost controls you need before putting any of this in front of real traffic.

Why Groq Changes the Streaming Experience

Groq runs open models like Llama and GPT OSS on custom LPU (Language Processing Unit) hardware rather than general purpose GPUs, which is why its chat completions API can sustain very high output token throughput. For a streaming chatbot, that throughput matters more than raw model quality, because the perceived speed of a chat interface is dominated by how quickly tokens arrive after the first one, not by total request time.

• Time to first token is usually under a few hundred milliseconds, so the UI can start rendering almost immediately after the user hits send.
• Sustained token throughput stays high even on longer responses, so a five hundred word answer does not visibly slow down partway through.
• The API is OpenAI-compatible in shape, so the official groq-sdk package and the streaming pattern below will feel familiar if you have built a chat feature against OpenAI before.

Building the Streaming Route Handler

In the App Router, a route handler is just a function that receives a Web Request and returns a Web Response, which means you can return a raw ReadableStream instead of a JSON body. That is exactly what streaming chat needs: the handler opens a connection to Groq with stream set to true, then re-emits each token as a server-sent event as soon as it arrives.

The handler below keeps the source of truth for the conversation on the server, builds the messages array with a system prompt plus the incoming user message, and forwards each delta from the Groq stream to the client using the standard Server-Sent Events wire format, a line prefixed with data colon followed by a JSON payload and a blank line. Wrapping the whole call in a try and finally block matters here: if the upstream Groq request fails partway through, the finally block still closes the controller so the client connection does not hang open indefinitely.

// app/api/chat/route.ts
import { NextRequest } from "next/server"
import Groq from "groq-sdk"

const groq = new Groq({ apiKey: process.env.GROQ_API_KEY })

export async function POST(request: NextRequest) {
  const { message } = await request.json()

  const encoder = new TextEncoder()

  const stream = new ReadableStream({
    async start(controller) {
      try {
        const completion = await groq.chat.completions.create({
          model: "openai/gpt-oss-120b",
          messages: [
            { role: "system", content: "You are a helpful assistant." },
            { role: "user", content: message },
          ],
          max_completion_tokens: 1024,
          temperature: 0.7,
          stream: true,
        })

        for await (const chunk of completion) {
          const delta = chunk.choices[0]?.delta?.content
          if (delta) {
            controller.enqueue(
              encoder.encode(`data: ${JSON.stringify({ delta })}\n\n`)
            )
          }
        }
        controller.enqueue(encoder.encode('data: {"done":true}\n\n'))
      } catch (err) {
        controller.enqueue(
          encoder.encode('data: {"error":"stream_failed"}\n\n')
        )
      } finally {
        controller.close()
      }
    },
  })

  return new Response(stream, {
    headers: {
      "Content-Type": "text/event-stream; charset=utf-8",
      "Cache-Control": "no-cache, no-transform",
      Connection: "keep-alive",
    },
  })
}

Consuming the Stream on the Client

On the client, you do not need an EventSource object because the request is a POST with a body, not a plain GET. Instead, read the response body directly with the Fetch API's streaming reader, decode each chunk of bytes into text, and split on the double newline that separates server-sent events.

Buffer any partial event that arrives split across two chunks, since TCP does not guarantee that a full event lands in a single read. Once you see a done flag in the payload, stop reading and finalize the message in your UI state. It also pays to render the accumulated text as each delta arrives rather than waiting for the whole message, since that incremental repaint is the entire visual point of streaming in the first place; a client that buffers everything before updating the screen throws away the latency benefit Groq's throughput was supposed to deliver.

// client component
async function sendMessage(message: string, onDelta: (text: string) => void) {
  const res = await fetch("/api/chat", {
    method: "POST",
    headers: { "Content-Type": "application/json" },
    body: JSON.stringify({ message }),
  })

  const reader = res.body!.getReader()
  const decoder = new TextDecoder()
  let buffer = ""

  while (true) {
    const { value, done } = await reader.read()
    if (done) break

    buffer += decoder.decode(value, { stream: true })
    const lines = buffer.split("\n\n")
    buffer = lines.pop() ?? ""

    for (const line of lines) {
      if (!line.startsWith("data: ")) continue
      const payload = JSON.parse(line.slice(6))
      if (payload.delta) onDelta(payload.delta)
      if (payload.done) return
    }
  }
}

System Prompts That Keep the Assistant on Topic

A chatbot that answers absolutely anything is a support and abuse liability. The system prompt should state the assistant's single purpose plainly, list what it must refuse to do, and include a short reminder near the end of the prompt reinforcing the scope, since long conversations can dilute earlier instructions. It also helps to embed a unique canary string as an HTML comment inside the system prompt; if that string ever shows up echoed back in a user message, it is a strong signal that someone extracted your prompt and is testing it against your own endpoint.

• State the persona and scope in the first sentence: what topics the assistant answers, and what it should redirect away from.
• Add an explicit instruction to ignore any request embedded in user input that tries to override, replace, or expand the assistant's role, since prompt injection attempts are common even on low-traffic portfolio and support chatbots.
• Validate and cap user input server-side before it reaches the model: enforce a maximum message length, strip control characters, and reject known injection phrases with a regex pass before spending a Groq request on them.

Rate Limiting and Cost Control

Groq enforces limits per API key across several dimensions: requests per minute, tokens per minute, and daily caps for both. Exceeding any of them returns a 429 status with a retry-after header telling you how many seconds to wait. Your own application should apply a tighter limit per visitor well before you ever hit Groq's ceiling, both to control cost and to stop a single abusive client from starving everyone else.

In front of the Groq call, add a per-visitor rate limiter keyed by IP address, capped well under the plan ceiling. Also cap max_completion_tokens on every request; an unbounded completion length is the single easiest way to turn a cheap per-token model into an expensive bill when a user (or a scraper) sends the same prompt in a loop.

Production Checklist Before You Ship

Streaming chat looks simple in a demo and breaks in specific, predictable ways in production. Run through this list before pointing real traffic at the route handler, and revisit it again after launch once real usage patterns start showing up in your logs.

1. Keep the API key server-side only, read from an environment variable, and never returned in any client-visible response or log line.
2. Apply per-visitor rate limiting on your own server in addition to relying on Groq's plan limits, and return a clear retry-after value to the client.
3. Cap max_completion_tokens and validate the incoming message length so a single request cannot balloon token usage or cost.
4. Handle stream errors gracefully: if the Groq call throws mid-stream, emit an error event the client can render instead of leaving the connection hanging.
5. Log token usage per request (Groq returns usage data) so you can catch a cost spike from a bug or abuse before the monthly invoice does it for you.