S3 HTTP API

Stow supports the native S3 operations and limits listed below. Unlisted behavior is unsupported unless marked provider-tolerant, such as SDK headers that Stow ignores.

Scope: local S3-compatible storage with a Go server and client wrappers.

Non-goals (v1): Versioning, Object Lock, ACLs, bucket policies, IAM, KMS/SSE, lifecycle rules, replication, notifications, S3 Select, batch operations.

0. SDK compatibility profile

0.1 Compatibility profile

The supported external profile is the version-pinned AWS SDK v3 (Node) and AWS SDK for Go v2. Stow accepts the safe union of behavior observable through those SDKs. It does not require undocumented Amazon-service strictness. Benign unknown headers are ignored; semantic markers for unsupported operations fail before they can fall through to a supported operation.

0.2 Naming and path decoding

Bucket names use 3–63 characters from [a-z0-9._-]. This local extension permits leading/trailing hyphens and underscores but never path separators, control characters, or traversal syntax. S3 reserved-name and IP-address rules remain rejected. Object keys are opaque strings; . and .. are valid content and are encoded reversibly by persistent storage. HTTP paths are decoded exactly once.

0.3 Authentication compatibility

Header authentication may use the SDK-compatible Date header when X-Amz-Date is absent, provided the selected date header is in SignedHeaders and the signature validates. Presigned URLs still require X-Amz-Date. Presigned methods are GET, PUT, and HEAD; X-Amz-Expires must be in 1..604800.

Every supplied x-amz-* header must appear in SignedHeaders, including copy source, source conditions, user metadata and checksum headers. Adding an unsigned copy-source header to a presigned PUT cannot change the authenticated operation.

Stow validates authentication material without rewriting it. Noncanonical SignedHeaders, repeated or unknown Authorization components, and duplicate or differently spelled authentication query parameters return 403. Sections 4.1 and 4.2 define the accepted grammar.

0.4 Policies and propagation

The policies are readThroughCache and mirrorWrites. local is a mode. Both policies require separate live-write consent for upstream propagation. mirrorWrites emits a startup warning. The legacy proxy policy returns a migration error.

Local mutations commit before propagation. A durable per-key outbox records an immutable local version and upstream baseline. It retries transient failures with bounded backoff and retains deterministic failures for inspection. Filesystem workers coordinate through expiring claims.

Before retrying an attempted put, recovery checks its upstream ETag, content type, user metadata and requested checksum. A match acknowledges the write without another mutation. An attempted delete is acknowledged when the upstream object is absent. Recovery may need another conditional attempt if it cannot confirm the result.

Outbox files have schema versions. Newer schemas fail at open; unversioned files migrate. Retry and discard routes always require the admin token, including on loopback.

0.5 Conditional operations and checksums

The SDK profile includes atomic If-None-Match: * and If-Match conditional PUTs, conditional GET/HEAD validators, Content-MD5, CRC32, CRC32C, CRC64NVME, SHA-1, and SHA-256. Unknown checksum algorithms return an error.

COPY supports source conditions. Destination If-Match and If-None-Match conditions return 501 NotImplemented before mutation; atomic destination comparison is not exposed by the current copy-store interface. GET/HEAD apply ETag conditions before their corresponding date conditions, and 304 responses retain the configured CORS headers without an object body or checksum.

CRC64NVME is supported for single-object PUT and full-object GET/HEAD, including upstream read/write conversion. The checksum is the base64 encoding of the big-endian NVMe CRC64 value; known vectors and the installed AWS CLI default-upload profile cover its variant. A mismatching value fails with 400 BadDigest. Partial (206) and unchanged (304) responses omit whole-object checksum headers.

CRC64NVME multipart initiation returns 501 NotImplemented; completion requests carrying CRC64NVME or an explicit checksum type are also refused. Multipart full-object/composite checksum completion is not implemented. A part checksum may be checked independently, but that does not establish a completed-object checksum. Upstream composite checksums are not relabeled as full-object checksums in the local cache. Existing multipart ETag behavior remains supported.

Conditional writes and reads return different statuses:

A matching If-None-Match read returns 304 without a body or S3 error code. SDKs may report different local error names: Go reports NotModified; the TypeScript SDK reports Unknown.

0.6 Backends and versioning

Filesystem is the default. Memory is temporary storage with the same supported behavior. The old sidecar format is not migrated; startup warns and uses the atomic format.

1. Supported Operations

All S3 operations use AWS Signature Version 4 (SigV4) unless served via a presigned URL (query-string auth). Requests without valid auth receive 403 Forbidden with an S3-shaped XML error body (see §5).

1.1 Bucket Operations

Notes:

• Bucket names MUST satisfy the amended local grammar in §0.2. Invalid names: 400 InvalidBucketName.
• CreateBucket MUST NOT require LocationConstraint for v1 (single implicit region).
• x-amz-acl, x-amz-grant-*, and policy headers are ignored on supported bucket ops (no ACL enforcement).

1.2 Object Operations

Object metadata (v1):

• Content-Type: Stored and returned on GET/HEAD. Default application/octet-stream if omitted on PUT.
• Content-Length: Required on PUT; enforced. Mismatch between declared length and body: 400 Bad Request.
• ETag: Strong validator; quoted hex MD5 for single-part objects; multipart ETag format per S3 ({md5}-{partCount}).
• x-amz-meta-*: Arbitrary user metadata keys (case-insensitive key normalization per S3). Returned on GET/HEAD. Max 2 KB total user metadata per object (enforce 400 if exceeded).
• Metadata adapters: HTTP metadata is emitted with lowercase x-amz-meta- names; SDK maps expose lowercase bare user keys such as project. Existing persisted/embedded maps using full prefixed header names remain readable through the compatibility normalizer. Upstream writes convert these names to SDK keys, avoiding accidental double prefixes. Conflicting case/prefix aliases are refused; the adapter never chooses a value based on map iteration order. Invalid names and CR/LF values are refused. PUT/HEAD/GET, COPY/REPLACE, multipart completion, reopen, and upstream-cache-offline paths have real SDK regression coverage.
• Range requests: GET with Range: bytes={start}-{end} returns 206 Partial Content with Content-Range header. An end past the last byte is clamped to the last byte, not refused — 416 is for a range that cannot be satisfied at all, which means a start at or past the size, an end before its start, or an unparseable spec. Refusals answer 416 Range Not Satisfiable with Content-Range: bytes */{size}. The distinction matters in both directions: refusing a clamped end breaks resumable clients, and clamping a start turns a 416 into a silently short 206.

CopyObject constraints (v1):

• Same-bucket and cross-bucket copy supported locally.
• x-amz-metadata-directive supported: COPY (default) or REPLACE.
• x-amz-copy-source-if-match, if-none-match, if-modified-since, if-unmodified-since supported; failed precondition → 412 Precondition Failed.
• SSE/KMS copy headers ignored (no encryption at rest in v1).

1.3 Listing — ListObjectsV2

Required response fields: Name, Prefix, KeyCount, MaxKeys, IsTruncated, Contents[] (when non-empty: Key, LastModified, ETag, Size, StorageClass = STANDARD), optional CommonPrefixes[] (Prefix), ContinuationToken (echo if sent), NextContinuationToken (if truncated), Delimiter, EncodingType.

Pagination contract:

• Default max-keys = 1000; server MAY use a lower internal page size but MUST honor IsTruncated + NextContinuationToken semantics.
• continuation-token is opaque; clients MUST round-trip the token verbatim.
• Listing is lexicographic UTF-8 byte order by key.

Prefix / delimiter:

• prefix filters keys by prefix.
• delimiter (typically /) groups keys; keys containing delimiter after prefix appear under CommonPrefixes, not Contents.

URL encoding (encoding-type=url):

• When encoding-type=url, keys and prefixes in XML MUST be URL-encoded; clients decode per AWS SDK behavior.

1.4 Multipart Upload Lifecycle

Multipart rules:

• Part numbers: integers 1–10000.
• Minimum part size: 5 MiB for all parts except the last (S3 rule); smaller intermediate parts → 400 EntityTooSmall.
• UploadPart MAY accept Content-MD5 for validation; mismatch → 400 Bad Request.
• Incomplete uploads do not appear in ListObjectsV2 until completed.
• Aborted or expired uploads MUST NOT leave ghost objects.

1.5 Presigned URLs

Presigned URLs bypass the Authorization header; auth is carried in query parameters (X-Amz-Algorithm, X-Amz-Credential, X-Amz-Date, X-Amz-Expires, X-Amz-SignedHeaders, X-Amz-Signature).

Expired presigned URL → 403 AccessDenied (Request has expired). Invalid signature → 403 AccessDenied.

1.6 CORS

Stow MUST respond to browser preflight and cross-origin requests for presigned uploads/downloads.

Bucket-specific CORS XML configuration APIs are out of scope; CORS is a fixed permissive dev policy in v1.

3. URL Style Support

Routing rules:

• Extract bucket from first path segment (path-style) or leftmost Host label before base host (virtual-hosted).
• Keys are the remaining path after bucket, URL-decoded per RFC 3986; preserve / in keys.
• ListBuckets uses / only (no bucket in Host).
• TLS termination is out of scope for local dev; plain HTTP on configurable port (default documented in package README).

Invalid bucket in Host (virtual-hosted): 400 Bad Request with InvalidBucketName or NoSuchBucket as appropriate.

4. Authentication Requirements

4.1 SigV4 Header Auth (SDK default)

Failure responses:

• Missing / malformed auth → 403 Forbidden, code AccessDenied or SignatureDoesNotMatch
• Wrong secret → 403 SignatureDoesNotMatch
• Request time skew > 15 minutes → 403 RequestTimeTooSkewed

Stow does not trim, lowercase, deduplicate or sort a malformed SignedHeaders value before verification. Doing so would verify a different header list from the one supplied. Repeated Authorization components are also rejected rather than choosing one value.

4.2 Presigned URLs

• Same local dev credentials used to sign.
• X-Amz-Expires maximum: 604800 seconds (7 days).
• Supported signed operations: GET, PUT, HEAD (HEAD via GET presign with SDK options).
• Unsigned query params not in signature MAY be ignored unless they alter signed headers/body.
• Authentication parameters (X-Amz-Algorithm, X-Amz-Credential, X-Amz-Date, X-Amz-Expires, X-Amz-SignedHeaders, X-Amz-Signature) are read by exact name. A parameter appearing twice, or under another spelling such as x-amz-signature, is refused rather than resolved: the signed query and the values the verifier consumes are two readings of one string, so a query carrying two spellings could be checked against a value that was not the one signed. Ordinary duplicate query parameters are deliberately unaffected — AWS canonicalizes those by sorting, and they are not authentication material.
• Far-edge skew. A presigned URL is refused when its signing time is further than the configured MaxSkew (default 15 minutes) in the future, as well as after its expiry. The expiry is derived from the signing time, which is inside the URL and therefore the client's to choose, so without this bound a URL signed for a date next year would never expire. The bound is the verifier's own allowance rather than a constant of the presigned branch, so both kinds of request are limited by the same number. A client whose host clock is more than 15 minutes fast will see 403 RequestTimeTooSkewed; that is the intended behaviour and matches what an ordinary signed request already did.

4.3 Anonymous Access

Not supported. Unsigned requests (except presigned) → 403 AccessDenied.

4.4 Upstream Credentials (Run-Through Only)

Upstream STOW_* / S3_* / AWS_* credentials are used only by the run-through adapter to reach live providers. They MUST NOT authenticate requests to the local stow endpoint.

5. Unsupported Features — Expected SDK Behavior

When a client invokes an unsupported API, stow MUST return S3-compatible XML errors so SDKs surface predictable exceptions.

Error XML shape (all errors):

<?xml version="1.0" encoding="UTF-8"?>
<Error>
  <Code>{Code}</Code>
  <Message>{human-readable message}</Message>
  <Resource>/{bucket}/{key}</Resource>
  <RequestId>{uuid}</RequestId>
</Error>

Every response MUST include x-amz-request-id header matching RequestId in body when body is XML.

6. Run-Through Mode Behavior Contract

Enable run-through with --mode run-through or STOW_MODE=run-through and configured upstream credentials. Environment precedence is STOW_* > S3_* > AWS_*. Credentials alone do not change local mode. STOW_MODE=local forces local-only storage.

6.1 Policies

A mirrorWrites policy configures routing; it does not authorize propagation. Within run-through mode, upstream writes additionally require live-write consent (STOW_ALLOW_LIVE_WRITES=true or --allow-live-writes), and without it the effective policy reports mirrorWrites-disabled and writes stay local.

6.2 Read-Through Cache Semantics

GET/HeadObject flow (readThroughCache):
1. If object exists locally → return local bytes/metadata
   - If upstream configured AND revalidation enabled (default): compare ETag/Last-Modified with upstream HEAD
     - If upstream newer → refresh local copy, then serve
     - If upstream 404 → evict only an entry marked as upstream-derived; never evict a local-only write
2. If local miss → HEAD/GET upstream
   - 404 → pass through 404 to client
   - 200 → persist to local backend, then serve

ListObjectsV2 in run-through:

• Returns union of local keys and upstream keys (deduplicated by key name).
• For duplicate keys, local metadata wins for ETag/Size/LastModified in listing (local is authoritative for dev).
• The merged result is ordered by key, then max-keys and continuation-token are applied to the merged set. Continuation tokens are opaque; a client resumes by returning a token it received and MUST NOT construct one.
• Consistency: eventual. Local and upstream listings occur at different times, without a shared transaction. Concurrent writes can make a key appear on two consecutive pages or neither page. A key deleted upstream during listing can still appear. Clients MUST tolerate these results.
• Concurrent writes: propagation uses the upstream ETag captured during a prior read, or an observation before preparing the mutation. Confirmed absence becomes an expected-absence condition. If upstream changes, the local mutation remains committed and the outbox reports a terminal conflict. A precondition failure MUST NOT be retried as a transport failure. If observation fails, the local mutation commits but propagation is refused. Existing v2 entries without a baseline also fail rather than writing unconditionally. Put, copy, multipart completion and delete use the captured condition. Stow does not merge conflicts or choose a winner.
• Under readThroughCache and mirrorWrites, listing is the merged local/upstream result described above; the legacy proxy policy is not supported.

Object size / buffering (v1, intentional):

• Single-part Put and multipart Complete buffer object/part bytes in memory to compute MD5 ETags. Stow is a dev/test bucket service — not production object storage. Large-object streaming without full buffering is out of scope for v1.

ListBuckets / HeadBucket / CreateBucket / DeleteBucket:

• Local bucket namespace is authoritative.
• CreateBucket creates locally only; does not create upstream bucket unless allowLiveWrites and explicit bucket-mirror option (v1: no auto-create upstream).
• DeleteBucket deletes locally; upstream bucket untouched.

CopyObject / Multipart / DeleteObject / DeleteObjects:

• Execute against the local store first.
• With readThroughCache + allowLiveWrites or mirrorWrites, supported mutations propagate to upstream through the durable outbox and use the same key path.

Presigned URLs:

• Signed against local endpoint; reads/writes hit local policy layer (not direct upstream bypass).

6.3 Startup Contract

On every Stow.start(), log to stdout (and expose via /_stow/status):

• Mode: local-only | run-through
• Upstream endpoint (host only; no secrets)
• Cache policy: readThroughCache | mirrorWrites | none
• Write policy: local-only | allowLiveWrites | mirrorWrites
• Override hints: STOW_MODE=local forces local; in local mode, how to ask for run-through (--mode run-through or STOW_MODE=run-through); allowLiveWrites flag

7. Admin Routes

Admin routes use separate authentication from SigV4. Read-only routes allow loopback requests without a token. Other addresses require STOW_ADMIN_TOKEN, sent as X-Stow-Admin. Outbox mutation routes require the token on every address. Unauthorized requests return 404. Without a configured token, outbox mutations are unavailable.

--allow-public-admin is deprecated and ignored. It does not grant access.

Errors: Unknown /_stow/* paths → 404 JSON { "error": "not found" }.

Appendix A — Common HTTP Status Summary

Appendix C — Request and upstream transport boundaries (0.3.0)

Native serve and workspace serve accept --max-request-bytes. Omitting the flag retains the 8 MiB default; an explicit value must be a positive integer. Zero and negative overrides are rejected before serving. The native readiness capability maxRequestBytes, and workspace readiness field max_request_bytes, report the actual configured limit. TypeScript startup/session/workspace helpers accept maxRequestBytes; Python session/workspace helpers accept max_request_bytes. The Go workspace facade has FacadeWithOptions and WorkspaceFacadeOptions.MaxRequestBytes (zero means the Go option was omitted).

The limit applies to bytes read from the HTTP request body, before SigV4 payload verification, checksum validation and object/multipart handling. HTTP transfer framing is removed by the HTTP server; an AWS signed-chunked encoding inside the body counts toward this bound. It is a per-request bound, including each multipart part, independent of stored-data quotas. Oversized bodies return the S3 EntityTooLarge error. The limit does not provide process-memory isolation.

Upstream endpoints must use HTTPS. HTTP is permitted by default only for literal loopback IP addresses (including IPv6 ::1). DNS names such as localhost require an explicit opt-in even when they resolve to loopback. Operators can use --allow-insecure-upstream; TypeScript startup exposes allowInsecureUpstream. The internal Go upstream configuration exposes AllowInsecureHTTP. Opt-in permits HTTP transport for the selected endpoint; it does not grant upstream writes. URLs containing user information or fragments, relative URLs, and other schemes are rejected before creating the credential-bearing upstream client.

The upstream HTTP client follows only 307/308 redirects within the original request origin (scheme, host and port), with a ten-request redirect bound. Cross-origin redirects, including HTTPS-to-HTTP redirects, are refused before a request reaches the destination. This preserves signed method/body handling and prevents a redirect from forwarding authorization or session-token headers to another origin. Existing write consent, bucket scope and offline rules still apply independently.

Native serve and workspace serve also bound active HTTP handlers to 16 by default, configurable with positive --max-concurrent-requests. The Go server config and workspace facade options expose MaxConcurrentRequests (zero selects 16). Admission precedes body buffering and authentication; excess requests fail with S3 SlowDown, HTTP 503, and Retry-After: 1. Native readiness reports maxConcurrentRequests; workspace facade readiness reports max_concurrent_requests. Clients should retry with backoff. This bounds active handlers, not accepted TCP connections or total process memory.

For workspace runtime sessions, staged multipart part bytes share MaxBytes accounting with committed objects, including reconciliation on reopen. Upload initiation also has a separate 1,024 outstanding upload session default, counting every upload even for the same or an existing target. Go runtime and workspace options expose MaxMultipartUploads (zero selects 1,024; negative is invalid). Successful abort/completion frees a slot; failed operations retain it. Opening existing data above the cap returns a quota error and preserves all uploads. Go capabilities, native readiness (maxMultipartUploads), and workspace facade readiness (max_multipart_uploads) report the effective cap. The native server's unlimited byte/object storage profile remains unlimited unless explicit storage quotas are configured; its upload count still defaults to 1,024. Neither request limit replaces storage quotas or bounds operating-system memory.

Workspace file reads and checkpoint input scans/copies pin an os.OpenRoot directory, refuse symbolic-link components and root identity replacement, and bound captured bytes to the measured file size plus one change-detection byte. Host writers must still be quiescent for a consistent capture. This read-side confinement is not an OS sandbox: inherited path-based workspace mutations and metadata publication require a trusted, cooperative host filesystem.

Local listing token bounds

Nonempty continuation-token values must be valid UTF-8, at most 1024 bytes, and contain no NUL; invalid values return 400 InvalidArgument. Current tokens are lexical key markers, not authenticated query-bound provider tokens. Clients should replay returned tokens.

Resource-policy listings paginate permitted objects and prefixes before choosing their lexical continuation marker. Backend cursors and denied names stay internal; truncation describes further permitted entries. A page may scan multiple backend pages, with cancellation checked between calls. Policy-free listings retain their backend pagination behavior.

Committed local publication errors

A workspace PUT that successfully publishes content but cannot persist its object index returns a committed-mutation error, not a refusal. The native SDK exposes ErrMutationCommitted, returns the known object identity with that error and accounts for the committed bytes. HTTP returns 500 InternalError, asking the caller to verify the outcome before retrying. The object file and metadata index are separate publications: this response does not promise rollback or crash-safe metadata recovery. Stop writers and preserve both files before repairing an index obstruction. A live handle retains the intended metadata and can persist it on a successful close after repair; reopening an invalid index fails closed. If that handle was lost, current bytes alone cannot reconstruct all original metadata.