[Feature] Streaming #10

New issue

Closed

opened 2026-08-25 10:27:36 +00:00 by epistemophiliac · 0 comments

epistemophiliac commented

2026-08-25 10:27:36 +00:00

Owner

Problem

LLMAPI currently buffers every upstream response server-side and returns a single JSON payload once inference completes. For API consumers — which is how our own employees use this service — this creates two concrete issues:

Latency: a 30s inference means 30s of silence before any byte reaches the client. Streaming would deliver the first token in ~200ms.
Memory pressure: the gateway holds the full response in memory before releasing it. Long outputs (code generation, long-form) inflate the server heap for no reason.

Current state

/v1/chat/completions accepts stream: true and proxies SSE correctly.
/v1/responses (OpenAI Responses API) does not stream — it always buffers.
The response cache (RESPONSE_CACHE) only stores non-streaming responses, so there is no architectural reason to keep buffering as the default.
PROVIDER_STREAM_STALL_TIMEOUT_MS already exists to guard against stalled streams.

Proposal

Make streaming the default inference mode across all surfaces. Non-streaming becomes an explicit opt-out.

Default behavior

Client sends	Behavior
`stream: true` or `Accept: text/event-stream`	Stream (SSE)
`stream: false`	Buffer and return single JSON
`stream` omitted	Stream (new default)

Surfaces affected

/v1/chat/completions
/v1/responses
/v1beta/chat/completions
/mcp
Ollama /api/chat
Gemini emulation

Opt-out

Clients that need a single JSON payload (batch jobs, webhook consumers, legacy integrations) send stream: false explicitly. No breaking change for them — they already set it.

Response cache interaction

The existing response cache already excludes streaming responses. No change needed there. If a future "streaming cache" is desired, it can be layered on later.

Backwards compatibility

Clients that already send stream: false → unchanged.
Clients that already send stream: true → unchanged.
Clients that omit stream → behavior changes from buffer to stream. This is the intended migration. A Sunset-style warning header on the first N responses can smooth the transition.

Why this matters for us

Our employees consume LLMAPI exclusively through the API. They need streaming to:

Get interactive-feeling responses in internal tools.
Reduce perceived latency in chat interfaces.
Allow early cancellation of expensive long-running inferences.

This is an API feature, not a UX feature — the dashboard is unaffected.

Response cache (RESPONSE_CACHE) — already streaming-aware.
PROVIDER_STREAM_STALL_TIMEOUT_MS — already guards mid-stream stalls.
FALLBACK_TIME_BUDGET_MS — failover budget applies per-chunk, not per-request.

## Problem LLMAPI currently buffers every upstream response server-side and returns a single JSON payload once inference completes. For API consumers — which is how our own employees use this service — this creates two concrete issues: 1. **Latency**: a 30s inference means 30s of silence before any byte reaches the client. Streaming would deliver the first token in ~200ms. 2. **Memory pressure**: the gateway holds the full response in memory before releasing it. Long outputs (code generation, long-form) inflate the server heap for no reason. ## Current state - `/v1/chat/completions` accepts `stream: true` and proxies SSE correctly. - `/v1/responses` (OpenAI Responses API) does **not** stream — it always buffers. - The response cache (`RESPONSE_CACHE`) only stores non-streaming responses, so there is no architectural reason to keep buffering as the default. - `PROVIDER_STREAM_STALL_TIMEOUT_MS` already exists to guard against stalled streams. ## Proposal Make streaming the **default** inference mode across all surfaces. Non-streaming becomes an explicit opt-out. ### Default behavior | Client sends | Behavior | |---|---| | `stream: true` or `Accept: text/event-stream` | Stream (SSE) | | `stream: false` | Buffer and return single JSON | | `stream` omitted | **Stream** (new default) | ### Surfaces affected - `/v1/chat/completions` - `/v1/responses` - `/v1beta/chat/completions` - `/mcp` - Ollama `/api/chat` - Gemini emulation ### Opt-out Clients that need a single JSON payload (batch jobs, webhook consumers, legacy integrations) send `stream: false` explicitly. No breaking change for them — they already set it. ### Response cache interaction The existing response cache already excludes streaming responses. No change needed there. If a future "streaming cache" is desired, it can be layered on later. ### Backwards compatibility - Clients that **already** send `stream: false` → unchanged. - Clients that **already** send `stream: true` → unchanged. - Clients that **omit** `stream` → behavior changes from buffer to stream. This is the intended migration. A `Sunset`-style warning header on the first N responses can smooth the transition. ## Why this matters for us Our employees consume LLMAPI exclusively through the API. They need streaming to: - Get interactive-feeling responses in internal tools. - Reduce perceived latency in chat interfaces. - Allow early cancellation of expensive long-running inferences. This is an **API feature**, not a UX feature — the dashboard is unaffected. ## Related - Response cache (`RESPONSE_CACHE`) — already streaming-aware. - `PROVIDER_STREAM_STALL_TIMEOUT_MS` — already guards mid-stream stalls. - `FALLBACK_TIME_BUDGET_MS` — failover budget applies per-chunk, not per-request.

epistemophiliac referenced this issue from a commit

2026-08-25 11:28:25 +00:00

feat(streaming): streaming becomes the default when stream is omitted (#10)

epistemophiliac referenced this issue from a commit

2026-08-25 11:29:02 +00:00

merge test/issue-10-streaming-default: streaming default when omitted (#10)

epistemophiliac referenced this issue

2026-08-25 11:29:33 +00:00

Release v1.2 — Budgets route + streaming default #11

epistemophiliac closed this issue

2026-08-25 21:41:47 +00:00