Streaming AI Responses with Server-Sent Events

This is the part that made me leave PHP behind.

In PHP, streaming is a no-go. The language was built for the request-response cycle — receive a request, process it, send the full response, done. There are workarounds, sure, but they're hacks. Output buffering tricks, flush calls that may or may not work depending on your server config. It's not what PHP was designed for.

And for a chat app, streaming is everything. You don't want to stare at a loading spinner for ten seconds while the AI generates a full response. You want to see words appear in real-time, like someone is typing back to you. That's what makes it feel alive.

Enter Server-Sent Events

Photo by Scott Rodgerson on Unsplash

Server-Sent Events (SSE) is a simple protocol. The server keeps an HTTP connection open and pushes data to the client in chunks. The browser receives these chunks as they arrive. No polling, no websockets, no complexity.

In FastAPI, the entire streaming endpoint came down to this: a StreamingResponse that yields chunks as they come in from the LLM. That's it. The framework handles the connection management for you.

After years of fighting PHP to do anything resembling real-time, this felt like magic.

How It Works

Photo by Ling App on Unsplash

The flow is straightforward but took me a while to get right:

1. The frontend sends a POST request to /api/v1/chat/stream with the character ID, message, and optional LLM parameters
2. The endpoint loads the character's system prompt from a markdown file
3. The OpenRouter client opens an async HTTP connection and streams the response
4. Each chunk is yielded as an SSE data: line back to the frontend
5. The frontend reads the stream and appends text to the chat bubble as it arrives

The beauty of async/await in Python is that the server isn't blocked while waiting for the next chunk from OpenRouter. It can handle other requests in between. Coming from PHP's synchronous world, this was a revelation.

from fastapi import FastAPI
from fastapi.responses import StreamingResponse
import json

@app.post("/api/v1/chat/stream")
async def stream_chat(req: ChatRequest):
    async def event_generator():
        response = await client.chat.completions.create(
            model="meta-llama/llama-3-8b-instruct:free",
            messages=[
                {"role": "system", "content": load_prompt(req.character_id)},
                {"role": "user", "content": req.message}
            ],
            stream=True,
            temperature=req.temperature,
        )
        async for chunk in response:
            content = chunk.choices[0].delta.content or ""
            if content:
                yield f"data: {json.dumps({'content': content})}\n\n"
        yield "data: [DONE]\n\n"

    return StreamingResponse(event_generator(), media_type="text/event-stream")

OpenRouter: Free Models, Real Learning

Photo by Igor Omilaev on Unsplash

I used OpenRouter because I didn't want to spend money while experimenting. They offer free models — not the best ones, but good enough to learn with and more than enough to power a fun side project.

The integration was simple: it's just an HTTP client calling their API. The streaming response comes back as SSE-formatted lines, and you parse each line to extract the content delta. When the model is done, you get a [DONE] marker.

Tunable Parameters

Photo by Oleksandr Latkun on Unsplash

One of the features I'm most happy with is the parameter sidebar. Users can tune the LLM in real-time:

• Temperature — How creative the responses are
• Top-p — How focused the response is
• Max tokens — Response length limit
• Presence penalty — Encourages new topics
• Frequency penalty — Reduces repetition

This was fun to build because it makes the app feel like a sandbox, not just a chat window. Crank the temperature to max and watch Yoda produce beautiful nonsense. Set it to zero and get cold, literal responses. It's a toy, and that's the point.

Streaming Bugs

Photo by Shutter Speed on Unsplash

Here's something that bit me: SSE sends each line as a separate event. So when the LLM response includes a newline character, it gets split across multiple data: lines on the frontend. The text would arrive jumbled — words out of order, line breaks in weird places.

The fix was to handle multiline chunks explicitly in the endpoint — splitting on newlines and re-emitting each line as its own SSE event, with an explicit empty-line marker when the original chunk ended with a newline. It's the kind of detail that seems obvious in hindsight but cost me hours.

# Sanitizing multiline text chunks for Server-Sent Events
def encode_sse_payload(raw_text: str) -> list[str]:
    events = []
    # Preserve explicit newline boundaries within LLM responses
    lines = raw_text.split("\n")
    for idx, line in enumerate(lines):
        payload = json.dumps({"content": line})
        events.append(f"data: {payload}\n\n")
        # Re-inject newline separator if chunk has trailing newlines
        if idx  len(lines) - 1:
            events.append(f"data: {json.dumps({'content': '\\n'})}\n\n")
    return events

Known Issues I'm Honest About

Photo by Mitchell Griest on Unsplash

I'll be upfront: the streaming implementation has limitations. Characters lose context after a few messages because conversation history isn't persisted — it only lives in the frontend's state. And personality fade is real — after 2-3 messages, the character starts sounding generic as the system prompt's influence wanes.

I listed these as known bugs on the site itself. I'd rather be transparent about what's broken than pretend it's production-ready. It's a learning project, and the bugs are part of the story.

Why This Mattered

Photo by Logan Voss on Unsplash

Streaming was the feature that justified the entire stack choice. If I'd stayed in PHP, I could have built the API, the database layer, the character system — but the streaming experience would have been a hack. Python's async support and FastAPI's StreamingResponse made it natural.

Next up: how I created the character personalities using another LLM to write the prompts for me.