AI runtime limits and Pi comparison¶

Normal agent behavior follows Pi v1.0.0 at a13d35a742c6ef8462812a28fbe1d8c8b7431c32. Separate defensive limits protect the hosted service from malformed streams, runaway work, and resource exhaustion. These limits have substantial headroom for schedule workflows. They do not determine when normal conversation compaction starts.

The tables show code defaults and fixed bounds. Defensive defaults are constants in core/nurse_scheduling/ai/config.py. Optional overrides appear below. Environment examples contain the normal deployment settings. MB and GB use decimal bytes. MiB and GiB use powers of 1,024. N/A marks a limit or component absent in the pinned Pi implementation.

Conversation and generation¶

Each history or turn-prefix section is summarized in one request. Complete, successful summaries are accepted, including empty text, as in Pi. Summary failure preserves the original context and does not disable later automatic compaction attempts in the same run. Compaction skips when the retention boundary leaves no older messages to summarize. An immediately repeated compaction skips when the last transcript entry is already a summary. Summary text is preserved, including whitespace. A large text estimate clamps requested output to at least one token, as in Pi. The provider decides whether the request fits. A provider rejection preserves the original context. A recovery attempt can summarize a single oversized final input after omitting the failed response, as in Pi. Manual compaction keeps that input when there is no omitted recovery attempt. A split conversation turn can require two summary requests, as in Pi. Model token limits bound generated text and reasoning.

Integration differences¶

Pi's design guides the whole agent. These implementation differences remain:

• The provider adapter uses OpenAI-compatible messages. Fallback estimates use characters divided by four, count readable reasoning and tool arguments, and include the system prompt and tool definitions. Opaque signatures and encrypted replay data do not inflate these estimates. Message serialization and image representation differ from Pi's native message format.
• Requests default to a 32,768-token response allowance so underestimated large tool responses leave room in the context window. Explicit request limits replace this default. Model output and available-context limits still apply.
• The loop checks context before tool continuations as well as after a final response, so growing tool results can be summarized before the next request. Length-stop recovery compares reported completion usage with the normal output allowance, which applies model and explicit request limits before the remaining context limit. Filling a smaller context cap can trigger recovery. Filling the normal allowance does not. If usage is absent, recovery requires a known request cap below the known normal allowance. If recovery compaction skips or fails, normal output-limit handling retains the response and refuses its partial tool calls. Manual compaction uses shared run admission, Stop, replay, and persistence. It requires an idle chat instead of implicitly stopping an active schedule operation. Use Stop before compacting an active chat. The UI combines Pi's two ineligible cases into one skipped notice.
• Status messages stay with their user question. They describe the schedule workspace and cannot replace the request that starts a conversation turn. Summaries from failed or cancelled runs are invalidated because those runs discard their temporary workspace.
• Model discovery must supply a positive context window before paid generation. Pi supplies model definitions directly. Numeric reasoning settings must leave the provider's minimum thinking budget and room for an answer. Invalid settings fail before the request, rather than sending a known invalid payload.
• Pi's extension hooks and custom compaction instructions are not exposed by this service. HTTP, memory, storage, and runtime bounds below protect the shared deployment and can still refuse work at capacity. Manual compaction bypasses the active-history quota and summarizes the full retained history, so reaching that quota does not prevent reducing it. Shared memory and concurrency limits still apply to the compaction request.

Defensive execution and input bounds¶

The provider counts decoded HTTP bytes before buffering SSE lines or parsing JSON. An oversized or unterminated line cannot bypass the response bound. The byte budget applies separately to each HTTP attempt.

User messages have no separate character limit. The model context budget bounds the accepted provider request. The HTTP body bound protects request parsing.

A completed exchange remains available even if its output crosses a history or memory bound. Further work is refused until capacity is available, or the user exports the chat and starts another. A refused optimizer review appears in the chat as a failed review with the same message. Allow memory headroom for output from every admitted run.

Parallelism applies to independent read-only batches. A batch containing a mutating tool remains sequential. More slots can help large I/O batches. Set concurrency according to sandbox capacity.

Files, service capacity, and optimizer¶

These components have no corresponding hosted-service quota in Pi.

Shared memory, concurrency, deadlines, and storage must fit deployment capacity. They remain operator controls, even when legitimate traffic can fill the service. Raising a count alone cannot provide more CPU, model capacity, or storage. The default 50 MB attachment bound stays below NGINX's 64 MiB request bound.

Optional defensive overrides¶

Normal operation requires none of these settings. Set a positive integer in docker/.env when a legitimate workload needs another bound. Both backend Compose variants pass these values only when set, using Compose's environment resolution. The service uses its code defaults when an override is absent. Empty, invalid, zero, and negative values fail startup. Keep attachment bytes plus multipart overhead below the proxy request bound when changing upload capacity.

For failure ordering, persistence, and event behavior, see AI Assistant Backend. The environment example and both backend Compose variants expose the runtime controls. In Compose, the optimizer timeout follows OPTIMIZE_DEFAULT_TIMEOUT_SECONDS so both services share the same default.

Provider streaming measurements¶

Turn details show estimated provider cost, output tokens per second, call count, and missing prices. Provider call records retain streaming duration and time to first output when measurable. Streaming duration runs from the first nonempty output chunk to the last. Output includes text, reasoning, and tool-call generation. Time to first output starts when the backend begins the HTTP request.

Speed uses provider-reported completion tokens, including reasoning tokens once. It excludes input tokens, time to first output, tool execution, VM work, retry backoff, and gaps between calls. A call needs reported positive output usage and at least two output chunks with distinct arrival times. Missing or incomplete measurements remain unavailable. Cost availability is independent of speed. When usage reports reasoning tokens without reasoning stream events, speed is unavailable because their generation time cannot be matched to the observed interval.

Run and conversation totals divide measured output tokens by measured streaming seconds. They do not average individual speeds. Turn details report how many calls contributed. Recovery and exports retain the same measurements. These are backend-observed delivery speeds. Provider buffering, hidden reasoning, network delays, and stream-consumer delays can affect the estimate.

Provider protocol failures log the underlying exception and gateway call ID when available. Use the call ID in turn details to find the matching backend log. Diagnostic logs omit provider response content and credentials.