Backend Server¶

The backend server is an asynchronous FastAPI service. It accepts scheduling YAML, retains job state and events, runs jobs in the background, and serves the resulting XLSX artifact.

This page covers only the HTTP server and its job infrastructure. The CLI, scheduling model, solver implementations, and frontend are out of scope.

Run Locally¶

After installing the core dependencies, start the development server from the repository root:

./scripts/start_backend.sh

Verify the process and its dependencies:

export API_URL="${API_URL:-http://localhost:8000}"

curl "$API_URL/ready"
curl "$API_URL/info"

/ready returns a minimal readiness result. /info also includes the API and application versions plus current worker status and activities.

Interactive OpenAPI documentation is available at $API_URL/docs, with the schema at $API_URL/openapi.json.

Architecture¶

flowchart TB
    Client[<b>HTTP client</b><br/>Submit YAML<br/>Receive JSON, SSE, XLSX]

    subgraph Process[FastAPI process]
        API[<b>API routes</b><br/>HTTP job endpoints]
        Controller[<b>JobController</b><br/>Job use cases<br/>Lifecycle policy]
        Worker[<b>JobWorker</b><br/>Claim and execute jobs]
        Executor[<b>Process runner</b><br/>Spawn and supervise direct child]
        Store[<b>JobStore</b><br/>Atomic job, lease,<br/>event, and artifact storage]
        Maintenance[<b>JobMaintenance</b><br/>Expire claims and jobs]

        subgraph Child[Per-job child process]
            Runner[<b>OptimizationRunner</b><br/>Call scheduling engine<br/>Create XLSX]
            Engine[<b>Scheduling engine</b><br/>Build and solve]
        end
    end

    Memory[<b>In-memory store</b><br/>Process-local]
    Redis[(<b>Redis store</b><br/>Cross-process)]

    Client -->|Submit and control| API
    API -->|JSON, SSE, XLSX| Client
    API -->|Commands| Controller
    Worker -->|Commands and outcomes| Controller
    Worker -->|Run job| Executor
    Executor <-->|Events, controls, result| Runner
    Runner -->|Schedule| Engine
    Controller -->|Persist| Store
    Maintenance -->|Cleanup| Controller
    Store -->|Either| Memory
    Store -->|or| Redis
    Memory ~~~ Redis

nurse_scheduling.serve:app is the public ASGI entry point. Each application process owns one worker thread and one maintenance thread. The worker renews its shared presence lease whether idle or running. Each optimization runs in a separate child process.

The controller owns job lifecycle policy. The worker owns execution orchestration. The process executor owns child process supervision. The runner owns one scheduler invocation and its output conversion.

Stores claim jobs in successful admission order. Equal timestamps and clock adjustments do not change queue positions. Redis appends after the largest existing queue score, including timestamp scores written by older versions.

Capacity pruning removes jobs in the order their first terminal transitions were stored. Updating terminal metadata does not change that order. Redis keeps this order across restarts. Jobs retained from older versions have no completion index, so Redis prunes them first, ordered by finish time and then job ID.

Job Lifecycle¶

stateDiagram-v2
    [*] --> queued: POST /optimize
    queued --> running: worker claims job
    queued --> cancelled: cancel
    running --> completed: result
    running --> failed: failure
    running --> cancelling: cancel
    cancelling --> cancelled: process ends
    cancelling --> cancelled: worker lease expires
    completed --> [*]
    cancelled --> [*]
    failed --> [*]

    classDef completedState fill:transparent,stroke:#43a047,stroke-width:3px
    classDef cancelledState fill:transparent,stroke:#78909c,stroke-width:3px
    classDef failedState fill:transparent,stroke:#e53935,stroke-width:3px
    class completed completedState
    class cancelled cancelledState
    class failed failedState

completed, cancelled, and failed are terminal states. A completed job may be optimal, feasible, or infeasible. An XLSX artifact is available only when a schedule was produced.

Cancellation is available for every running job. It immediately terminates the optimization process tree, uses error code cancelled, and discards the result and artifact. Early completion requires solver support and sets a control flag without adding another lifecycle state. If a current result is available, the job later becomes completed.

Timeout enforcement¶

The server accepts a timeout for every solver and enforces it at two levels. The selected solver runs in a child process and first receives the requested limit so it can stop cleanly and return any result it supports. The hard server watchdog starts when the child process starts and does not depend on a solving phase event. Its deadline combines the requested timeout and a 90-second timeout grace period by default. The grace period covers startup, model construction, and shutdown while still bounding a job that becomes stuck before solving. If the process has not returned by the deadline, the server terminates its optimization process tree. Tree cleanup is required for PuLP command-line backends because they launch external solver executables. OR-Tools runs inside the direct optimization child and does not require descendant cleanup. A thread is not sufficient because Python cannot safely force-stop an arbitrary worker thread.

A process that returns before the hard deadline follows its normal result path. A feasible result returned at the solver limit uses termination reason solver_timeout. Forced termination marks the job as failed with error code process_timeout and produces no artifact, even if an incumbent score was reported earlier. The error message records the requested timeout, timeout grace, and forced termination. Preserving the last schedule would require checkpointing it outside the child process.

{
  "error": {
    "code": "process_timeout",
    "message": "The optimization process did not return within the requested 300-second timeout and 90-second timeout grace period. The server terminated the process."
  }
}

HTTP API¶

The POST /optimize response includes backend with the accepting API instance's app and API versions, service name, deployment ID, instance ID, and claimed performance. Capture it from that response, since a separate /info request can reach another instance. Later job responses set backend to null. This identifies the accepting API instance. A shared-store worker may execute the job on another instance.

Authentication¶

A deployment that sets API_AUTH_TOKEN or API_AUTH_TOKENS requires a bearer key on every application route except /info and /ready. Those two stay public so clients and deployment probes can discover the server without credentials. /info reports the requirement:

{ "auth": { "required": true, "scheme": "bearer" } }

Backends that predate this field omit it, and clients treat a missing descriptor as an open server. Send the token as a bearer credential:

Use an https:// API URL when authentication is enabled. A trusted reverse proxy may terminate TLS before forwarding requests to the backend. TLS protects both the bearer credential and the stream token from interception.

export API_AUTH_TOKEN="<token>"
curl -H "Authorization: Bearer $API_AUTH_TOKEN" "$API_URL/optimize/options"

GET /optimize/{job_id}/events also accepts a short-lived token in the URL, because EventSource cannot send an Authorization header. Job responses embed one in links.events, so a client opens the stream with the link as given:

{ "links": { "events": "/optimize/<job_id>/events?token=<stream_token>" } }

The stream token is an HMAC of the job ID and an expiry signed with the matched bearer key. It authorizes only that job's stream and is rejected on every other route, which keeps the deployment key out of URLs, proxy logs, and referrer headers. Its lifetime is OPTIMIZE_MAX_TIMEOUT_SECONDS plus OPTIMIZE_TIMEOUT_GRACE_SECONDS plus a few seconds of slack, so it outlives the longest run the deployment allows and expires shortly after. An expired stream token makes the frontend fall back to polling the job.

A missing or incorrect key returns 401 with a WWW-Authenticate: Bearer header. Configure multiple static keys with a JSON object mapping IDs to keys:

API_AUTH_TOKENS='{"institution-a":"first-key","person-b":"second-key"}'

IDs may contain letters, numbers, underscores, and hyphens. The legacy ID is reserved for API_AUTH_TOKEN. IDs are recorded for administration, while clients send only the key and never receive the ID. Requests use a keyed fingerprint map for direct lookup followed by a constant-time comparison. Stream links carry no credential hint, so the server verifies a stream token against each configured key rather than putting a stable key-derived value in a URL. Remove a pair and restart the server to revoke its bearer and stream tokens. API_AUTH_TOKEN remains supported and may be used alongside identified keys during migration.

Identified keys attribute a request, they do not isolate one. Every configured key reaches every protected route, so any key may read, cancel, finish, and stream a job created with another key. Use separate deployments when callers must not see each other's jobs.

When authentication is configured, the generated /openapi.json, /docs, and /redoc routes are disabled and return 404.

API_AUTH_REQUIRED makes authentication mandatory rather than optional. The images built for deployment set it, so a container started without either key setting fails with API_AUTH_REQUIRED is set, so API_AUTH_TOKEN or API_AUTH_TOKENS must not be empty instead of serving openly. A server run outside those images leaves it unset, so setting either key setting is enough to turn authentication on for local use.

Prepare the input as YAML. The repository includes a minimal scheduling example. Submit either a YAML file or a YAML string, but not both:

curl -i \
  -H "Authorization: Bearer $API_AUTH_TOKEN" \
  -F file=@core/tests/testcases/basics/01_1nurse_1shift_1day.yaml \
  -F timeout=60 \
  "$API_URL/optimize"

The server returns 202 Accepted, the job representation, a Location header, and Retry-After: 1. Use the returned job ID to follow events and download the result:

export JOB_ID="<job_id>"
export EVENTS_URL="/optimize/<job_id>/events?token=<stream_token>"

curl -N "$API_URL$EVENTS_URL"
curl -OJ -H "Authorization: Bearer $API_AUTH_TOKEN" \
  "$API_URL/optimize/$JOB_ID/xlsx"

# Delete retained data after the job reaches a terminal state.
curl -i -X DELETE -H "Authorization: Bearer $API_AUTH_TOKEN" \
  "$API_URL/optimize/$JOB_ID"

The server persists and replays job.state_changed, job.phase_changed, job.progressed, job.control_changed, and job.result_available events. Send Last-Event-ID when reconnecting to continue after the last received event. Disconnecting from the stream does not stop the job.

Job submission sets a seven-day, HTTP-only client correlation cookie for diagnostics. It does not control access to a job or its lifetime, though a browser reaching a job that a different browser created is reported. Browser CORS access is limited to local origins and nursescheduling.org subdomains.

Lifecycle and storage errors use a stable JSON envelope:

{
  "error": {
    "code": "job_not_found",
    "message": "Job was not found"
  }
}

Request parsing and validation errors retain FastAPI's standard error format. Common status codes include 404 for missing resources, 409 for invalid job operations, 413 for oversized YAML, and 429 when job capacity is exhausted or one address holds too many open event streams.

Every job runs on the same workers in submission order. Without another limit, one address could fill the queue with long jobs and delay every other user for hours. An unauthenticated request therefore counts against its address's share, JOB_MAX_PENDING_PER_CLIENT. Past that share, the request gets 429 with the code client_job_capacity_exceeded. A request with a bearer key is not counted, because keys belong to trusted clients such as the diagnostic. The store keeps a salted digest of the address, not the address itself. The Redis store shares its salt through Redis, so every API process derives the same digest. The AI service forwards the browser's address when it submits optimizer jobs and opens their event streams, so assistant users do not share the AI container's address. Many people behind one network address share one budget. Raise the limit if a site's users run several jobs at once.

Each open job event stream also reads the job store once per second. One address may therefore hold at most JOB_MAX_EVENT_STREAMS_PER_CLIENT open streams in each API process. Past that limit, the request gets 429 with the code event_stream_capacity_exceeded, and the address can retry after one of its streams closes. A browser holds one or two streams. Requests with a bearer key are not counted. Requests with a stream token are counted, because the token is part of the URL and can leak through logs or shared links. Each Uvicorn worker counts its own streams, so the limit per address grows with the number of workers.

The frontend and CLI serialize plain, valid YAML, so aliases and parse failures both indicate a client that built its request by hand. Both are reported without changing how the request is answered. Submitted YAML is also bounded by how far its aliases expand and how deeply it nests, not only by its byte size. An alias is a reference, so a small document can name hundreds of millions of nodes that every later traversal pays for, and parsing costs grow faster than nesting depth. Nesting is measured from parser events, so brackets in text do not count, and data passing either bound is refused with 400 before a job is queued. Reading happens on a worker thread and only after the checks that cost nothing, so a request that was going to be rejected never pays for it.

Suspicious request reporting¶

Missing routes and unauthenticated probes are internet background noise and are not reported. Client errors that instead require knowledge of this API's contract are sent to Sentry, because a scanner cannot produce them:

A reported request answers exactly like an unreported one, so its body, status, and headers reveal nothing. Reporting still costs a little time, so a caller measuring closely can infer that something happened. Each signal groups into its own Sentry issue.

Repeats of one signal from one address are counted within a fixed window, and a signal that reaches SUSPICION_ESCALATE_COUNT is reported as an error rather than a warning, carrying its occurrences count. A signal naming a job also carries distinct_job_ids, and escalates only when that count reaches the threshold, so a client retrying one job stays a warning however often it repeats while a caller working through identifiers does not. Only a signal whose every request names a job counts them, so mixing one job request into another signal cannot hold that signal's escalation down. Further repeats within that window keep counting but are not reported, so one address cannot spend the project's event quota. Because the window is fixed rather than sliding, repeats spread across a boundary can stay below the threshold. Addresses are counted as a salted digest, so the counters hold no record of who connected, and the salt is derived from bearer keys and the deployment ID when authentication is enabled. Authenticated Redis deployments share counters across worker processes. Open deployments use a random salt per process, so their counters are process-local even with Redis. They can reach the threshold later. Counting is advisory. A Redis failure leaves a report unescalated and does not suppress it based on incomplete counts.

A stale browser tab can produce job_id_probe after its job is deleted or expires, and a mistyped token produces rejected_bearer_token, so both are reported as warnings rather than errors. Neither reaches an error by repeating, because a stale tab names one job and a mistyped token names none. Changing API_AUTH_TOKEN invalidates every stream token already handed out, so expect forged_stream_token from real clients until the longest outstanding one expires.

Every event carries a client.address tag holding the address Uvicorn resolved for its request. Sentry's own attribution is left alone. In Compose, NGINX replaces X-Forwarded-For with Cloudflare's single CF-Connecting-IP value, and Uvicorn trusts that forwarded header. Both the tag and Sentry therefore receive the same public client address. The original caller-supplied forwarding chain is intentionally discarded, so Compose no longer reports a claimed address disagreement.

Docker assigns all network addresses. This trust model assumes local service containers are not hostile, because Uvicorn accepts forwarded headers from any peer. Keep the API and AI ports unpublished and route public traffic only through Cloudflare Tunnel and NGINX. Cloudflare recommends CF-Connecting-IP for the original visitor address because it has one consistent address rather than the variable-length X-Forwarded-For chain.

Storage and Scaling¶

Do not use memory mode with multiple Uvicorn workers. A later request may reach a different process that does not contain the job. Redis mode coordinates job claims across processes. Each process still executes at most one job at a time. Opaque lease tokens fence stale workers. Stores validate the job revision, lease, and active-job association together before accepting worker updates. Each execution retains the exact lease used to claim its job.

Example with three server processes:

cd core
JOB_BACKEND=redis \
JOB_REDIS_URL=redis://localhost:6379/0 \
uvicorn nurse_scheduling.serve:app \
  --workers 3 \
  --host 0.0.0.0 \
  --port 8000 \
  --no-access-log

Configuration¶

All server settings are read once when the application is constructed.

Numeric values must be positive. JOB_MAX_RETAINED must be at least JOB_MAX_PENDING. The default solver must be advertised, and the timeout default must remain within the configured minimum and maximum.

Both key settings are optional and unset by default, so a locally run server needs no credentials. Setting either one turns on authentication for that deployment. Use at least 16 characters per key. When API_AUTH_REQUIRED=true, the backend rejects a shorter key. When it is false, a shorter key is accepted with a warning for local testing.

The images under docker/ set API_AUTH_REQUIRED=true, so a deployment that publishes the backend refuses to start without a key. Serving one without authentication requires API_AUTH_REQUIRED=false. See Authentication.

The three CLAIMED_PERFORMANCE_* values are optional, but they must be set together. A complete compute benchmark writes them to claimed-performance.env beside its report. The API publishes the result at GET /info as claimed_performance. The frontend displays it as Claimed performance when that backend is selected.

Inspect Redis with RedisInsight¶

The Redis-backed Compose deployment includes an optional RedisInsight service. It listens only on the backend host's loopback interface and does not retain its own settings. Start it from docker/:

docker compose -f compose.backend.yml --profile inspection run --rm --service-ports redisinsight

For a remote backend, forward the loopback port over SSH:

ssh -L 5540:127.0.0.1:5540 user@backend-host

Open http://127.0.0.1:5540 and select the preconfigured Nurse Scheduling Redis database. Job-store keys start with nurse_scheduling:jobs:v0:. Usage-reporting keys start with nurse_scheduling:usage:v0:. Use those prefixes in the key browser to narrow the results.

RedisInsight connects with the same unrestricted access as the backend. Its browser and CLI can change or delete production data, so use them only for inspection. Press Ctrl+C when finished. Compose removes the temporary RedisInsight container while the Redis service and its redis-data volume remain intact.

For staging, pass --env-file .env.staging to the command. The compose.backend.memory.yml variant has no Redis service and therefore no RedisInsight inspector.

Tests¶

Run the primary server tests from core/:

pytest --log-cli-level=INFO tests/test_serve.py
pytest --log-cli-level=INFO tests/test_optimize_job_backends.py

To include Redis integration coverage, start a local Redis instance and use a dedicated database:

JOB_REDIS_TEST_URL=redis://localhost:6379/15 \
pytest --log-cli-level=INFO tests/test_optimize_job_backends.py