The Big Beautiful Bot

A streaming LLM chat demo that shows how to do token-by-token streaming properly. The backend pushes the model's reply to the browser over Server-Sent Events (SSE) as the tokens arrive, and it does so with the resilience you actually need in production: comment heartbeats to keep the connection alive and surface dead clients, Last-Event-ID resume so a dropped connection replays from where it left off instead of starting over, client-side cancellation that propagates all the way to the OpenAI call, and partial persistence so a half-finished answer is still saved.

The stack is Spring Boot 4.1 (Java 25) with Spring AI talking to OpenAI on the backend, and Next.js 16 with React 19 on the frontend. It accompanies a tutorial, so the code is commented heavily and the moving parts are kept deliberately small.

Prerequisites

• JDK 25 (the Gradle wrapper builds against a Java 25 toolchain).
• Node.js 20 or newer for the Next.js frontend.
• An OpenAI API key. The demo defaults to the gpt-5-mini model.

Running it

There is no Docker here. You run the backend, then run the frontend. Start the backend first: the browser only ever talks to the Next.js proxy routes, which forward to the backend.

1. Backend

Set your OpenAI key as an environment variable, then start the app with the Gradle wrapper.

On macOS or Linux:

cd backend
export OPENAI_API_KEY=sk-...
./gradlew bootRun

On Windows PowerShell:

cd backend
$env:OPENAI_API_KEY="sk-..."
.\gradlew bootRun

The backend serves on http://localhost:8080. It uses an in-memory H2 database with ddl-auto: create-drop, seeded from import.sql on every startup, so the data resets each time you restart. The model is set in backend/src/main/resources/application.yaml if you want to change it.

2. Frontend

In a second terminal:

cd frontend
npm install
npm run dev

Then open http://localhost:3000. If your backend is not on port 8080, point the frontend at it with the BACKEND_URL environment variable (it defaults to http://localhost:8080).

What to try

• Send a message and watch the reply stream in token by token rather than appearing all at once.
• Click Stop mid-stream. The cancellation propagates through the proxy to the backend, which aborts the OpenAI call and saves the partial answer it had received so far. Reload the conversation and you will see the truncated reply preserved.
• Browse the seeded conversations in the sidebar. These come from import.sql and reappear on every backend restart.
• Use the suggested prompts on the empty state to get going quickly.

How it works

The browser never talks to Spring directly. It calls the Next.js proxy routes under frontend/app/api/, which forward to the backend. This hides the backend origin, sidesteps CORS, and gives auth a natural home in a real deployment.

• The streaming endpoint is POST /api/chat/stream in backend/.../controller/ChatStreamController.java. It returns a Flux<ServerSentEvent<String>>, which Spring MVC adapts onto its async machinery and writes as text/event-stream. The streaming, heartbeat and resume logic lives in backend/.../service/ChatService.java.
• The conversation list and history come from GET /api/conversations and GET /api/conversations/{id}/messages in backend/.../controller/ConversationController.java, served from H2.
• The frontend proxy is frontend/app/api/chat/route.ts (streaming, with request.signal forwarded so a Stop cancels the upstream call) plus frontend/app/api/conversations/route.ts and frontend/app/api/conversations/[id]/messages/route.ts.
• The client-side stream handling, including Last-Event-ID resume on a dropped connection, is in frontend/app/hooks/useChat.ts.

Resilience behaviours worth noting: each SSE event carries an id of the form messageId:charOffset, so on reconnect the client sends the last id it saw and the server replays from that offset rather than regenerating. Heartbeats are emitted as SSE comments on a fixed interval to keep writes flowing, since the servlet API only notices a dead client on write. Both knobs live under app.sse.* in application.yaml (heartbeat-interval, default 15s; timeout, default 5m).

Observability

The backend registers a gauge for the number of currently open SSE streams. With the app running, fetch it from the metrics endpoint:

http://localhost:8080/actuator/metrics/sse.connections.active

Open a chat, start a long reply, and watch the count rise and fall. The H2 web console is enabled in application.yaml and available at http://localhost:8080/h2-console (JDBC URL jdbc:h2:mem:streamingllm, user sa, blank password) if you want to inspect the seeded tables directly.

Optional: see proxy buffering for yourself

This is not required to run the demo. If you put a reverse proxy in front of the backend, a buffering proxy will collect the whole response before forwarding it, which destroys the live token-by-token effect: the user waits, sees nothing, then gets the entire answer at once. The fix is to turn buffering off for the stream. An Nginx server block in front of the backend would look like this:

server {
    listen 80;

    location /api/chat/stream {
        proxy_pass http://localhost:8080;

        # Without these two lines Nginx buffers the whole SSE response
        # and the live streaming effect is lost.
        proxy_buffering off;
        proxy_set_header X-Accel-Buffering no;

        proxy_http_version 1.1;
        proxy_read_timeout 5m;
    }
}

The backend already sets X-Accel-Buffering: no and Cache-Control: no-cache on the stream response, which is the application-level half of the same fix.

Project layout

springbootnextllm/
├── backend/                         Spring Boot 4 / Java 25 / Spring AI
│   ├── build.gradle
│   └── src/main/
│       ├── java/com/tucanoo/streamingllmdemo/
│       │   ├── controller/          ChatStreamController, ConversationController
│       │   ├── service/             ChatService (streaming, heartbeats, resume)
│       │   ├── entity/              Conversation, Message, Role
│       │   ├── repository/          JPA repositories
│       │   ├── dto/                 request/response shapes
│       │   ├── config/              chat client, SSE properties, CORS
│       │   └── metrics/             SseConnectionMetrics (active-stream gauge)
│       └── resources/
│           ├── application.yaml     model, H2, actuator, app.sse.* knobs
│           └── import.sql           seeded conversations (reloaded each run)
└── frontend/                        Next.js 16 / React 19
    └── app/
        ├── api/                     proxy routes to the backend
        │   ├── chat/                streaming proxy
        │   └── conversations/       list + history proxies
        ├── components/              ChatApp UI
        ├── hooks/                   useChat (stream + resume client)
        └── lib/                     backend URL, SSE parsing, types

Taking it further

• Add authentication with Spring Security and pass the identity through the Next.js proxy.
• Swap H2 for Postgres by changing the datasource and the ddl-auto strategy so data persists.
• Deploy it, putting the Nginx config above (or your platform's equivalent) in front so streaming survives the proxy.