Develop

cloudbuild.yaml

@@ -23,6 +23,7 @@ steps:
           pubsub.googleapis.com \
           storage.googleapis.com \
           translate.googleapis.com \
+          iamcredentials.googleapis.com \
           --project=$PROJECT_ID
   - name: 'node:24'
     id: 'install-deps'
@@ -77,6 +78,20 @@ steps:
   - name: ghcr.io/lessify/firebase:edge
     id: 'firebase-deploy'
     args: [ 'deploy', '--config=firebase.$PROJECT_ID.json', '--project=$PROJECT_ID', '--only=hosting,functions,storage,firestore,auth', '--force' ]
+  # let the functions sign Storage URLs as themselves: assets over 31 MiB are redirected to one,
+  # because Hosting truncates function responses at 32 MiB (mirrors scripts/localess/url-signing.mjs)
+  - name: 'gcr.io/google.com/cloudsdktool/cloud-sdk'
+    id: 'grant-url-signing'
+    entrypoint: 'bash'
+    allowFailure: true
+    args:
+      - '-c'
+      - |
+        SA="$PROJECT_NUMBER-compute@developer.gserviceaccount.com"
+        gcloud iam service-accounts add-iam-policy-binding "$$SA" \
+          --member="serviceAccount:$$SA" \
+          --role=roles/iam.serviceAccountTokenCreator \
+          --project=$PROJECT_ID
 options:
   logging: CLOUD_LOGGING_ONLY
 substitutions:

docs/cdn-caching.md

@@ -37,6 +37,8 @@ Whenever the marker is rewritten, its generation increments. The content marker
 | Asset response (incl. `/original`, `/download`) | 365 days | 365 days | browser / CDN |
 | Asset `304 Not Modified` | 365 days | 365 days | browser / CDN |
 | Asset canonical-size redirect (`w`/`h` above source) | 365 days | 365 days | browser / CDN |
+| Asset over 31 MiB — `302` to a signed Storage URL (valid 24 h) | 1 hour | 1 hour | browser / CDN |
+| Asset over 31 MiB that could not be signed — `500` | no-cache | no-cache | — |
 | 400 — invalid asset transform param | 1 hour | 1 hour | browser / CDN |
 | 404 — space not found | 7 days | 7 days | browser / CDN |
 | 404 — content cache marker missing / content not found on disk (content routes only) | 10 min | 10 min | browser / CDN |

docs/deployment/check.md

@@ -136,6 +136,20 @@ invoked by Google's own service agents, so making them public would widen access
 
 ---
 
+## URL signing
+
+Firebase Hosting buffers a rewritten function response and cuts it off at 32 MiB, still answering
+`200` with a matching `Content-Length` — so a large video "downloads" as its first 32 MiB. The
+asset API therefore redirects anything over 31 MiB to a short-lived signed Storage URL. The
+functions have no key file, so they sign through the IAM Credentials `signBlob` API as their own
+runtime service account (the default compute account, `<project-number>-compute@developer.gserviceaccount.com`),
+which needs `roles/iam.serviceAccountTokenCreator` **on itself**. Nothing grants that by default.
+
+`localess:deploy` grants it after each functions deploy; `--fix` grants it here. The account only
+exists once functions have been deployed, so before that the row reads `?` and points at a deploy.
+
+---
+
 ## `--fix`
 
 ```bash
@@ -151,6 +165,7 @@ Applies only the repairs that are **free and reversible**:
 | Project labels | Writes the three markers |
 | Local files / region | A full `syncLocalFiles`, propagating the live Firestore location everywhere |
 | Invoker bindings | Adds `allUsers` to `roles/run.invoker`, preserving every other binding |
+| URL signing | Grants the runtime service account `roles/iam.serviceAccountTokenCreator` on itself |
 | Admin user | Prompts for credentials, then creates the account, its admin claim and its documents |
 
 Everything else is reported with the command that does it, because it is not safe to do as a side

docs/deployment/firebase-setup.md

@@ -92,7 +92,7 @@ In order:
 1.  Preflight        firebase-tools present and >= 15.29.0, CLI authenticated
 2.  Project          adopt --project, or pick from a list / create one
 3.  Billing          link a billing account, then wait for it to become active
-4.  APIs             check the 15 required APIs, enable whichever are off
+4.  APIs             check the 16 required APIs, enable whichever are off
 5.  Firestore        adopt the existing location, or ask and create in --region
 6.  Storage          create and link the default bucket in --region
 7.  Storage CORS     allow GET and HEAD from any origin, when the bucket has no rules
@@ -116,8 +116,15 @@ firestore             identitytoolkit        firebasestorage
 firebaseextensions    cloudfunctions         cloudbuild
 artifactregistry      run                    eventarc
 pubsub                storage                translate
+iamcredentials
 ```
 
+`iamcredentials.googleapis.com` is what the functions sign Storage URLs through — assets over
+31 MiB are redirected to a signed URL, because Firebase Hosting truncates function responses at
+32 MiB. Signing also needs the runtime service account to hold
+`roles/iam.serviceAccountTokenCreator` on itself; deploy grants that after the functions are up
+(the account does not exist before), and `localess:check` reports it.
+
 ### Authentication
 
 `firebase.json` carries an `auth` block, which the `auth` deploy target sends to Google's

docs/v1-functions-api.md

@@ -37,7 +37,7 @@ Content delivery with cache-busting and asset transformation. All content/transl
 - **`thumbnail` param** — Collapses an animated WebP/GIF to its first frame, and extracts a video frame via FFmpeg (requires `w`). Has no effect on other image types. Since animations now resize with every frame intact, `thumbnail` is how you ask for a *still* rather than how you make resizing work.
 - **EXIF orientation is applied** — Sharp strips the orientation tag on re-encode, so a rotated source is baked into the pixels. Without it a portrait phone photo returned landscape with its aspect ratio transposed.
 - **Embedded colour profiles are carried through** — via `keepIccProfile()`. Sources without one gain nothing; tagging every response as sRGB would add ~506 bytes each.
-- **`/original` and `/download`** — serve the stored bytes exactly as uploaded, inline and as an `attachment` respectively. Since a bare transform request re-encodes, these are the only way to retrieve the original file. Neither enters Sharp, and both reject `w`/`h`/`q`/`f`/`fit`/`thumbnail`/`download` with `400` rather than ignoring them.
+- **`/original` and `/download`** — serve the stored bytes exactly as uploaded, inline and as an `attachment` respectively. Since a bare transform request re-encodes, these are the only way to retrieve the original file. Neither enters Sharp, and both reject `w`/`h`/`q`/`f`/`fit`/`thumbnail`/`download` with `400` rather than ignoring them. Objects over 31 MiB are answered with a `302` to a signed Storage URL (24 h validity, redirect cached 1 h) carrying the same `Content-Type` and `Content-Disposition`, because Firebase Hosting truncates function responses at 32 MiB; smaller ones are streamed from Storage with `Range` support. The same applies to a bare `/assets/:assetId` request for a non-image type (video, audio, PDF). Signing needs the runtime service account to hold `roles/iam.serviceAccountTokenCreator` on itself — see [check.md](deployment/check.md#url-signing).
 - **Removed in v4** — the `?download` flag and `f=original`. Both return `400` with a message naming the replacement route. Responses already cached under the old spellings keep serving for the remainder of their 365-day TTL.
 
 #### Asset resize combinations (`w` / `h`)

functions/src/utils/stream-object.ts

@@ -3,13 +3,26 @@ import { logger } from 'firebase-functions';
 import type { Readable } from 'stream';
 
 /**
- * Cloud Run rejects a response larger than 32 MiB with a bare `500` unless it is streamed with
- * chunked transfer encoding — and a response carrying `Content-Length` is not streamed in that
- * sense. Bodies at or under this size keep their `Content-Length` (so clients show progress);
- * anything larger is sent without one, which is what makes Node fall back to chunked encoding.
- * Kept a little under the hard limit to leave room for the headers.
+ * Largest object served through the function; anything bigger is redirected to Storage.
+ *
+ * Firebase Hosting buffers a rewritten function response and cuts it off at 32 MiB — a 104 MiB
+ * video arrived as its first 32 MiB with a `200` and a matching `Content-Length`, so the browser
+ * reported a finished download. Chunked encoding does not help: it gets past Cloud Run's own
+ * 32 MiB limit, but not past Hosting's. Kept a little under the limit to leave room for headers.
+ */
+export const MAX_PROXIED_BYTES = 31 * 1024 * 1024;
+
+/**
+ * How long a signed Storage URL stays valid. Long enough that a video player can still seek
+ * (each seek is a fresh `Range` request against the signed URL) well after the page loaded.
  */
-export const MAX_CONTENT_LENGTH_BYTES = 31 * 1024 * 1024;
+export const SIGNED_URL_TTL_SECONDS = 24 * 60 * 60;
+
+/**
+ * How long the redirect itself may be cached, by the browser and by the Hosting CDN. Well under
+ * the signed URL's lifetime, so a cached redirect never points at an expired signature.
+ */
+export const SIGNED_REDIRECT_MAX_AGE = 60 * 60;
 
 export type ByteRange = { start: number; end: number };
 
@@ -45,24 +58,99 @@ export function parseByteRange(header: string | undefined, size: number): ByteRa
 }
 
 /** The slice of a Storage `File` this needs, so tests can hand in a fake. */
-export interface StreamableFile {
+export interface StoredFile {
   createReadStream(options?: { start?: number; end?: number; validation?: false }): Readable;
+  getSignedUrl(config: {
+    version: 'v4';
+    action: 'read';
+    expires: number;
+    responseDisposition?: string;
+    responseType?: string;
+  }): Promise<[string]>;
+}
+
+/** The headers the stored object must be served with, wherever it is served from. */
+export interface StoredObjectHeaders {
+  contentType: string;
+  contentDisposition: string;
+}
+
+/**
+ * Serves a stored object: streamed through the function when Hosting can carry it, otherwise a
+ * `302` to a short-lived signed Storage URL. The caller sets the type, disposition, cache and
+ * security headers first; the redirect overrides the cache header and repeats type and
+ * disposition as signed response overrides, so Storage serves the same filename.
+ *
+ * Signing calls the IAM Credentials `signBlob` API, so the runtime service account needs
+ * `iam.serviceAccounts.signBlob` on itself (`npm run localess:check -- --fix` grants it). When
+ * that is missing this answers an uncached `500` — never the truncated body Hosting would make
+ * of a streamed one.
+ * @param {Request} req Incoming request, read for `Range` and the method
+ * @param {Response} res Response to write
+ * @param {StoredFile} file Storage object to serve
+ * @param {number} size Object size in bytes, from its metadata
+ * @param {StoredObjectHeaders} headers Type and disposition to serve the object with
+ * @param {string} tag Log prefix of the calling route
+ * @return {Promise<void>} resolves once the response is handed off
+ */
+export async function serveStoredObject(
+  req: Request,
+  res: Response,
+  file: StoredFile,
+  size: number,
+  headers: StoredObjectHeaders,
+  tag: string
+): Promise<void> {
+  if (size <= MAX_PROXIED_BYTES) {
+    streamStoredObject(req, res, file, size, tag);
+    return;
+  }
+  let url: string;
+  try {
+    [url] = await file.getSignedUrl({
+      version: 'v4',
+      action: 'read',
+      expires: Date.now() + SIGNED_URL_TTL_SECONDS * 1000,
+      responseDisposition: headers.contentDisposition,
+      responseType: headers.contentType,
+    });
+  } catch (error) {
+    logger.error(`${tag} could not sign a Storage URL for a ${size}-byte object`, error);
+    res.removeHeader('Content-Disposition');
+    res
+      .status(500)
+      .header('Cache-Control', 'no-cache')
+      .contentType('text/plain')
+      .send(
+        'This asset is too large to serve through Hosting, and the function could not sign a Storage URL for it. ' +
+          'Grant the functions service account roles/iam.serviceAccountTokenCreator on itself: npm run localess:check -- --fix'
+      );
+    return;
+  }
+  logger.info(`${tag} ${size} bytes, redirecting to Storage`);
+  res.removeHeader('Content-Disposition');
+  res.header('Cache-Control', `public, max-age=${SIGNED_REDIRECT_MAX_AGE}, s-maxage=${SIGNED_REDIRECT_MAX_AGE}`).redirect(302, url);
 }
 
 /**
  * Streams a stored object to the response, honouring a single `Range`. Headers other than
  * length and range (type, disposition, cache) must already be set by the caller.
  *
- * Replaces `download()` to `/tmp` + `sendFile`: on Cloud Functions `/tmp` is RAM, so that held
- * the whole object in memory, and `sendFile` always sets `Content-Length` — which is what made
- * every object over 32 MiB fail.
+ * Replaces `download()` to `/tmp` + `sendFile`, which held the whole object in memory: `/tmp`
+ * is RAM on Cloud Functions.
  * @param {Request} req Incoming request, read for `Range` and the method
  * @param {Response} res Response to stream into
- * @param {StreamableFile} file Storage object to read
+ * @param {StoredFile} file Storage object to read
  * @param {number} size Object size in bytes, from its metadata
  * @param {string} tag Log prefix of the calling route
  */
-export function streamStoredObject(req: Request, res: Response, file: StreamableFile, size: number, tag: string): void {
+export function streamStoredObject(
+  req: Request,
+  res: Response,
+  file: Pick<StoredFile, 'createReadStream'>,
+  size: number,
+  tag: string
+): void {
   res.header('Accept-Ranges', 'bytes');
   const range = parseByteRange(req.headers.range, size);
   if (range === 'unsatisfiable') {
@@ -74,9 +162,7 @@ export function streamStoredObject(req: Request, res: Response, file: Streamable
   if (range) {
     res.status(206).header('Content-Range', `bytes ${start}-${end}/${size}`);
   }
-  if (length <= MAX_CONTENT_LENGTH_BYTES) {
-    res.header('Content-Length', String(length));
-  }
+  res.header('Content-Length', String(length));
   if (req.method === 'HEAD' || length === 0) {
     res.end();
     return;

functions/src/v1/cdn-assets.test.ts

@@ -19,6 +19,9 @@ import { beforeAll, beforeEach, describe, expect, it, vi } from 'vitest';
  * The env has to be set before the module graph loads, which is why the imports are dynamic.
  */
 const MD5 = 'abc123==';
+const SIGNED_URL = 'https://storage.googleapis.com/test-project.appspot.com/signed';
+/** Past what Firebase Hosting can carry, so the object is redirected rather than streamed. */
+const LARGE = 40 * 1024 * 1024;
 
 let CDN: express.Router;
 let bucket: { file: (path: string) => unknown };
@@ -82,13 +85,24 @@ interface StoredAsset {
   storageMissing?: boolean;
   /** Simulates a deleted asset still referenced by published content. */
   documentMissing?: boolean;
+  /** Overrides the stored object's size, to exercise the large-object path without the bytes. */
+  size?: number;
+  /** Simulates a runtime service account that is not allowed to sign URLs. */
+  signingFails?: boolean;
 }
 
-function given(asset: StoredAsset): { download: ReturnType<typeof vi.fn>; createReadStream: ReturnType<typeof vi.fn> } {
+function given(asset: StoredAsset): {
+  download: ReturnType<typeof vi.fn>;
+  createReadStream: ReturnType<typeof vi.fn>;
+  getSignedUrl: ReturnType<typeof vi.fn>;
+} {
   const file = {
     getMetadata: asset.storageMissing
       ? vi.fn().mockRejectedValue(new Error('not found'))
-      : vi.fn().mockResolvedValue([{ md5Hash: MD5, size: String(asset.bytes.length) }]),
+      : vi.fn().mockResolvedValue([{ md5Hash: MD5, size: String(asset.size ?? asset.bytes.length) }]),
+    getSignedUrl: asset.signingFails
+      ? vi.fn().mockRejectedValue(new Error('Permission iam.serviceAccounts.signBlob denied'))
+      : vi.fn().mockResolvedValue([SIGNED_URL]),
     createReadStream: vi
       .fn()
       .mockImplementation((options?: { start?: number; end?: number }) =>
@@ -188,6 +202,52 @@ describe('asset routes — the passthrough pair', () => {
     expect(res.headers['content-range']).toBe('bytes */10');
   });
 
+  it('/download redirects an object too large for Hosting to a signed Storage URL', async () => {
+    const file = given({ bytes: Buffer.from('x'), size: LARGE, type: 'video/quicktime', extension: '.mov' });
+
+    const res = await request(app()).get(`${URL}/download`);
+
+    expect(res.status).toBe(302);
+    expect(res.headers['location']).toBe(SIGNED_URL);
+    expect(res.headers['cache-control']).toContain('max-age=3600');
+    expect(res.headers['content-disposition']).toBeUndefined();
+    expect(file.createReadStream).not.toHaveBeenCalled();
+    // Storage must hand back the same filename and type the function would have sent.
+    const config = file.getSignedUrl.mock.calls[0][0];
+    expect(config).toMatchObject({ version: 'v4', action: 'read', responseType: 'video/quicktime' });
+    expect(config.responseDisposition).toContain('attachment');
+    expect(config.responseDisposition).toContain('photo.mov');
+  });
+
+  it('/original signs an inline disposition for a large object', async () => {
+    const file = given({ bytes: Buffer.from('x'), size: LARGE, type: 'video/mp4', extension: '.mp4' });
+
+    const res = await request(app()).get(`${URL}/original`);
+
+    expect(res.status).toBe(302);
+    expect(file.getSignedUrl.mock.calls[0][0].responseDisposition).toContain('inline');
+  });
+
+  it('a large object that cannot be signed fails loudly and uncached, never truncated', async () => {
+    const file = given({ bytes: Buffer.from('x'), size: LARGE, type: 'video/quicktime', extension: '.mov', signingFails: true });
+
+    const res = await request(app()).get(`${URL}/download`);
+
+    expect(res.status).toBe(500);
+    expect(res.headers['cache-control']).toBe('no-cache');
+    expect(res.text).toContain('localess:check');
+    expect(file.createReadStream).not.toHaveBeenCalled();
+  });
+
+  it('a bare transform-route URL on a large video redirects too', async () => {
+    given({ bytes: Buffer.from('x'), size: LARGE, type: 'video/mp4', extension: '.mp4' });
+
+    const res = await request(app()).get(URL);
+
+    expect(res.status).toBe(302);
+    expect(res.headers['location']).toBe(SIGNED_URL);
+  });
+
   it('a bare transform-route URL on a video streams the stored bytes', async () => {
     const bytes = Buffer.from('video bytes');
     const file = given({ bytes, type: 'video/mp4', extension: '.mp4' });