Skip to content

feat(knowledge): index Confluence image attachments - #8949

Merged
waleedlatif1 merged 13 commits into
stagingfrom
feat/kb-image-documents
Oct 13, 2026
Merged

waleedlatif1 merged 13 commits into
stagingfrom
feat/kb-image-documents

Conversation

@mzxchandra

@mzxchandra mzxchandra commented Oct 12, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Confluence image attachments (PNG, JPEG, WebP, GIF) were dropped at listing, so tables, scans and screenshots attached to pages never reached knowledge-base search.
This PR indexes them: each image is read once by a Gemini vision model, which writes out what it holds (a table becomes one line per row with its headers, a scan becomes its full text, a screenshot becomes its visible text plus a description), and that text is chunked and embedded like any other document.

  • Per-source policy. Connectors declare an image policy on their meta (imageExtraction). Confluence is semantic; Google Drive and every other source keep today's OCR path unchanged.
  • Dedicated Gemini adapter. Transient server and network errors are retried within the extraction's time budget (retryWithExponentialBackoff, as OCR does); throttling and overload defer the pass through provider admission. Error bodies are reduced to Gemini's status and reason codes.
  • Adapter boundary. image-extraction-gemini.ts calls generateContent with a JSON-schema response, the same way the embedding adapters call their providers. Knowledge processing never goes through the workflow provider layer (which would pull the tool registry into the knowledge page). Keys resolve like embeddings: workspace BYOK, then GEMINI_API_KEY, then the GEMINI_API_KEY_1..3 pool.
  • Image Attachments setting, opt-in for existing connectors. A new dropdown on the Confluence connector ("Read with AI" / "Don't index"). New connectors are created with it on, through a server-side defaultSourceConfig on the connector meta that every create surface (UI, v2 API, CLI, Sim) applies. An unset value means off, so connectors created before this change never start reading (and billing) images until their owner turns the setting on. No migration. Turning it off removes indexed images on the next sync, outside the soft-delete and mass-deletion hold that guard real source deletions, for both service-account and member-credential connectors.
  • Decorative images. Logos, icons and images under 64 px complete with 0 chunks and show a "Nothing to index" empty state. Files under 2 KB, SVG, HEIC and AVIF are never listed.
  • Per-sync cap. KB_IMAGE_EXTRACTIONS_PER_SYNC (default 500) bounds new or changed images read per sync; the rest wait for the next sync, which reports partial. Held-back existing images stay indexed (they are still marked seen). A member sync shares one budget across members, the count survives a listing restart and is saved per batch, and a full resync does not re-read unchanged images.
  • No key, no failures. Without any Gemini key (self-hosted), images are left out of the sync instead of becoming failed documents.
  • Retry checkpoint. A vision checkpoint on a new shared processing-checkpoint-store (OCR and embedding checkpoints moved onto it, with keys and format unchanged) means a retried pass reuses the extraction instead of paying twice.
  • Billing. Each extraction is a separate recordUsage model entry (knowledge-document-vision:<docId>:<passId>), zero for BYOK. Extractions whose document then fails are not billed, matching embeddings today.
  • Stale embeddings. Completing a pass now always replaces the previous embeddings, so a document that re-indexes to zero chunks drops its old ones.
  • Runtime. sharp is added to the Trigger.dev bundle for image preparation (orientation, downscale to 2048 px, first GIF frame).

Product update (if relevant)

Teams with Confluence knowledge bases can now search what lives inside images on their pages: schedules and tables pasted as screenshots, scanned letters, UI screenshots in how-to pages, and diagrams.
It is on by default for new Confluence connectors and billed per image as a model call. Existing connectors keep images off until their owner sets Image Attachments to Read with AI.
Docs: knowledgebase/connectors.mdx (Confluence section) and the self-hosting environment variables page.

Type of Change

  • Bug fix
  • New feature
  • Breaking change
  • Documentation
  • Other: ___________

Testing

  • Unit tests for the extractor (request shape, schema validation, error mapping, token-limit cut-off, deadline reserve, undecodable images, model fallback), the Gemini adapter, the vision checkpoint, document routing (Confluence images to vision, Drive images still to OCR), billing, the per-sync cap (new and changed images, no-key hold-back, full resync), the Image Attachments opt-out removal, and the Confluence listing and hydration rules.
    The connector, sync, document-processing, connector application and outbox suites pass (86 files, 1,085 tests); check:audits and lint are clean.
  • New integration suite confluence-images.integration.ts against real PostgreSQL, file storage and in-process indexing (synthetic connector listing behind the real Confluence meta; synthetic extraction and embeddings): the cap across syncs with held-back changed images staying indexed, full resync not re-reading unchanged images, opt-out removal and re-enable, opt-in for connectors created before the setting, zero-chunk re-indexing dropping old embeddings, and the no-key hold-back. Breaking the held-back seen stamp on purpose fails the suite. ocr-input-failures.integration.ts now seeds a Google Drive source for its image cases, since Confluence images no longer go to OCR.
  • End-to-end on a local stack against a synthetic Confluence space (11 attachments: a table, a scanned letter, a UI screenshot, a flowchart, a photo, a logo, an extensionless PNG, an animated GIF, a tiny icon, an SVG and a PDF).
    Each of 8 search questions ranked the right image document first; the logo completed with 0 chunks and the empty state; the icon and SVG were never listed; the PDF was unchanged; the cap produced partial syncs until all images were read; a re-sync made no new charges.
    This run predates the review follow-ups in the last commit (opt-out removal, cap on changed images, no-key hold-back), which are covered by unit tests.
  • Comparison of Gemini and OpenAI vision models on a sample of document images favored Gemini on quality and cost, so this PR is Gemini only.

Reviewers, please focus on:

  • lib/knowledge/connectors/sync-primitives.ts (processDocOps): images held back by the cap or a missing key are never fetched; held-back existing images are still marked seen so reconciliation keeps them.
  • lib/knowledge/connectors/sync-content-pass.ts (removeDisabledImages): the opt-out deletes image documents directly by media type.
  • lib/knowledge/documents/service.ts: the unconditional embedding replacement on completion and the separate vision usage entry.
  • lib/knowledge/documents/processing-checkpoint-store.ts: the refactor should be behavior-preserving for OCR and embedding checkpoints.

To verify on staging: knowledge-processing runs for image documents load sharp in the Trigger.dev worker (local runs processed in-process).

Checklist

  • Code follows project style guidelines
  • Self-reviewed my changes
  • Tests added/updated and passing
  • No new warnings introduced
  • I confirm that I have read and agree to the terms outlined in the Contributor License Agreement (CLA)

Screenshots/Videos

A completed decorative image shows a "Nothing to index" empty state instead of bare table headers; the Confluence connector form gains an Image Attachments dropdown between Content Type and Filter by Label.

OCR and embedding checkpoints each carried their own copy of the framed
header + body format, the outbox cleanup reservation and the cleanup
handler factory. Move them into processing-checkpoint-store so a third
checkpoint kind can reuse them.
Confluence images (PNG, JPEG, WebP, GIF) were dropped at listing, so
tables, scans and screenshots attached to pages never reached search.

- Connectors declare an image policy; Confluence reads images with a
  dedicated Gemini adapter that returns the image's text, every other
  source keeps OCR unchanged.
- A new Image Attachments setting on the Confluence connector, on by
  default, so existing knowledge bases pick images up on their next sync.
- Decorative images (logos, icons, images under 64 px) complete with no
  chunks; files under 2 KB and SVGs are never listed.
- Each sync admits at most KB_IMAGE_EXTRACTIONS_PER_SYNC new images
  (default 500); the rest wait for the next sync, which reports partial.
- A vision checkpoint lets a retried pass reuse the extraction instead
  of paying for a second call; usage is billed per image.
- Completing a pass now always replaces old embeddings, so a document
  that re-indexes to zero chunks drops its stale ones.
- sharp is added to the Trigger.dev bundle for image preparation.
A completed image with no chunks shows an empty state instead of bare
table headers, and the connector and self-hosting docs cover image
attachments and the two new environment variables.
Review follow-ups for Confluence image indexing.

- The per-sync cap now counts changed images as well as new ones, and a
  member sync shares one budget across its members. Held-back images
  that already exist are still marked seen, so they are never reconciled
  away.
- A full resync no longer re-reads unchanged images; their bytes are
  pinned to the attachment version.
- Without any Gemini key, images are left out of the sync instead of
  becoming failed documents.
- Turning Image Attachments off removes the indexed images on the next
  sync, outside the soft-delete and mass-deletion hold meant for source
  deletions.
- Gemini errors: a response cut off at the token limit, a rejected key
  or region, and an unknown model fail permanently with an accurate
  message instead of retrying or blaming the image; an image whose pixel
  data cannot be decoded fails as invalid.
- The admission wait keeps the post-call time reserve, Retry-After uses
  the shared parser, and a configured model must be a catalog Gemini
  model so its pricing is known.
@vercel

vercel Bot commented Oct 12, 2026 •

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated
docs Ready Ready Preview Oct 13, 2026 2:59am UTC

Request Review

@cubic-dev-ai cubic-dev-ai Bot left a comment •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

1 issue found across 38 files

Confidence score: 3/5

  • In apps/sim/lib/knowledge/connectors/sync-content-pass.ts, a Gemini key resolution failure can leave images unindexed while the pass reports success. Mark the checkpoint deferred when image operations are held.
Prompt for AI agents (unresolved issues)

Check if these issues are valid — if so, understand the root cause of each and fix them. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. If appropriate, use sub-agents to investigate and fix each issue separately.


<file name="apps/sim/lib/knowledge/connectors/sync-content-pass.ts">

<violation number="1" location="apps/sim/lib/knowledge/connectors/sync-content-pass.ts:284">
P2: When Gemini key resolution fails, `processDocOps` holds image operations without setting `cycle.imagesDeferred`, so this pass can report success while leaving images unindexed. Mark the checkpoint deferred for key-unavailable images so the sync status reflects the pending extraction.</violation>
</file>

Heads up: you’ve reached your flex budget. Increase your flex budget or wait for usage to reset.

Turn on auto-fix | Re-trigger cubic

Comment thread apps/sim/lib/knowledge/connectors/member-sync-engine.ts
Comment thread apps/sim/lib/knowledge/documents/image-extraction-gemini.ts Outdated
Comment thread apps/sim/lib/knowledge/documents/service.ts
Comment thread apps/sim/lib/knowledge/connectors/listing-checkpoint.ts
Comment thread apps/sim/lib/knowledge/documents/document-processor.ts Outdated
Comment thread apps/sim/lib/knowledge/connectors/sync-content-pass.ts
Comment thread apps/sim/lib/knowledge/connectors/sync-content-pass.ts Outdated
Comment thread apps/sim/lib/knowledge/documents/image-extraction.ts Outdated
Comment thread apps/sim/lib/knowledge/connectors/sync-primitives.ts Outdated
@greptile-apps

greptile-apps Bot commented Oct 12, 2026 •

Copy link
Copy Markdown
Contributor

RetriggerConfidence Score: 5/5

[High impact] The PR appears safe to merge; no actionable blocking issues remain.

Summary

This PR makes Confluence image attachments searchable through Gemini, with per-connector settings, extraction limits, retry checkpoints, and separate usage charges.

  • Confluence connectors can list supported image attachments for indexing.
  • Confluence images can become searchable text through Gemini vision.
  • Image retries reuse saved readings, and completed passes replace old chunks.
  • Each sync limits image reads and removes images disabled by the owner.

Diagram

%%{init: {'theme': 'neutral'}}%%
flowchart TD
  A[Confluence attachment listing] --> B{Image reading enabled?}
  B -->|No| C[Leave images out]
  B -->|Yes| D{Key and budget available?}
  D -->|No| E[Hold back image and keep existing document seen]
  D -->|Yes| F[Download image]
  F --> G{Saved extraction available?}
  G -->|Yes| I[Chunk extracted text]
  G -->|No| H[Reserve capacity and call Gemini]
  H --> J[Save extraction checkpoint]
  J --> I
  I --> K[Replace old embeddings]
  K --> L[Record successful extraction usage]
Loading

Reviews (7) · Last reviewed commit: "refactor(knowledge): tighten image extra..." · Reviewed by Greptile

Comment thread apps/sim/lib/knowledge/connectors/sync-content-pass.ts
Comment thread apps/sim/lib/knowledge/connectors/sync-content-pass.ts
Comment thread apps/sim/lib/knowledge/documents/image-extraction-gemini.ts
…nnectors

- New connectors are created with Image Attachments on, through a
  server-side defaultSourceConfig on the connector meta that every create
  surface applies; an unset value now means off, so existing connectors
  only read images once their owner turns the setting on.
- The Gemini call retries transient server and network errors within the
  extraction's time budget; throttling still defers the pass.
- Gemini error bodies are reduced to their status and reason codes.
- Images under the byte floor complete as decorative even when the source
  reported no size, and the extraction model must be a generateContent
  Gemini model.
- A member sync that held images back forces a full listing next time,
  the image count survives a listing restart and is saved per batch, and
  turning images off removes them for member-credential connectors too.
- The disabled-image removal skips rows the delete would refuse.
- An integration suite covers the cap, held-back images staying indexed,
  full resync, opt-out removal, opt-in for existing connectors, zero-chunk
  re-indexing and the no-key hold-back against real PostgreSQL; the OCR
  input-failure suite now uses a source that still sends images to OCR.
@mzxchandra

Copy link
Copy Markdown
Contributor Author

@greptile

@mzxchandra

Copy link
Copy Markdown
Contributor Author

@cubic-dev-ai review this PR

@cubic-dev-ai

cubic-dev-ai Bot commented Oct 12, 2026

Copy link
Copy Markdown
Contributor

@cubic-dev-ai review this PR

@mzxchandra I have started the AI code review. It will take a few minutes to complete.

@cubic-dev-ai cubic-dev-ai Bot left a comment •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All reported issues were addressed across 43 files

Heads up: you’ve reached your flex budget. Increase your flex budget or wait for usage to reset.

Turn on auto-fix | Re-trigger cubic

Comment thread apps/sim/lib/knowledge/documents/image-extraction.ts Outdated
Comment thread apps/sim/lib/knowledge/connectors/member-sync-engine.ts Outdated
…es on every member run

- A Gemini reply that is not valid JSON is retried within the extraction's
  budget and fails permanently once the attempts are spent, instead of
  costing a whole document retry each time.
- A member sync that skips its content pass (content not due yet) still
  removes images once Image Attachments is turned off.
- The reconciliation integration suite runs with images on, so it keeps
  exercising reconciliation alone.
@mzxchandra

Copy link
Copy Markdown
Contributor Author

@greptile

@mzxchandra

Copy link
Copy Markdown
Contributor Author

@cubic-dev-ai review this PR

@cubic-dev-ai

cubic-dev-ai Bot commented Oct 12, 2026

Copy link
Copy Markdown
Contributor

@cubic-dev-ai review this PR

@mzxchandra I have started the AI code review. It will take a few minutes to complete.

Comment thread apps/sim/lib/knowledge/connectors/member-sync-engine.ts Outdated
Comment thread apps/sim/lib/knowledge/documents/processing-checkpoint-store.ts Outdated
Comment thread apps/sim/lib/knowledge/application/connectors.test.ts Outdated

@cubic-dev-ai cubic-dev-ai Bot left a comment •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All reported issues were addressed across 44 files

Heads up: you’ve reached your flex budget. Increase your flex budget or wait for usage to reset.

Turn on auto-fix | Re-trigger cubic

Comment thread apps/sim/lib/knowledge/documents/image-extraction.ts Outdated
Comment thread apps/sim/lib/knowledge/connectors/member-sync-engine.ts
Comment thread apps/sim/lib/knowledge/documents/image-extraction.ts Outdated
Comment thread apps/sim/lib/knowledge/connectors/member-sync-engine.ts Outdated
Comment thread apps/sim/lib/knowledge/connectors/member-sync-engine.ts Outdated
…traction attempt

- Member runs add the lifecycle's purges to the run's deletion counts
  instead of overwriting them, so image removals after Image Attachments
  is turned off are reported and logged as purges.
- A retried extraction reports the tokens of every attempt Gemini
  answered, since each one was billed.
- The checkpoint header check uses isRecordLike.
- The create-time default is tested through the real creation path in
  the integration suite, replacing a mock-only unit test.
@mzxchandra

Copy link
Copy Markdown
Contributor Author

@greptile

@mzxchandra

Copy link
Copy Markdown
Contributor Author

@cubic-dev-ai review this PR

@cubic-dev-ai

cubic-dev-ai Bot commented Oct 12, 2026

Copy link
Copy Markdown
Contributor

@cubic-dev-ai review this PR

@mzxchandra I have started the AI code review. It will take a few minutes to complete.

@cubic-dev-ai cubic-dev-ai Bot left a comment •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All reported issues were addressed across 43 files

Heads up: you’ve reached your flex budget. Increase your flex budget or wait for usage to reset.

Turn on auto-fix | Re-trigger cubic

Comment thread apps/sim/lib/knowledge/connectors/member-sync-engine.ts Outdated
An unfinished disabled-image removal now keeps the member run marked as
having work remaining, so the next run starts promptly instead of waiting
a full sync interval with images still searchable.
@mzxchandra

Copy link
Copy Markdown
Contributor Author

@greptile

@mzxchandra

Copy link
Copy Markdown
Contributor Author

@cubic-dev-ai review this PR

@cubic-dev-ai

cubic-dev-ai Bot commented Oct 13, 2026

Copy link
Copy Markdown
Contributor

@cubic-dev-ai review this PR

@mzxchandra I have started the AI code review. It will take a few minutes to complete.

@cubic-dev-ai cubic-dev-ai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No issues found across 43 files

Confidence score: 5/5

  • Automated review surfaced no issues in the provided summaries.
  • No files require special attention.

Heads up: you’ve reached your flex budget. Increase your flex budget or wait for usage to reset.

You've manually re-run cubic several times on this PR. Each manual re-review checks the full PR again and counts toward your usage quota. To preserve your usage limits, we recommend letting cubic automatically review new commits.

Turn on auto-fix | Re-trigger cubic

…only when queued

- Connectors declare image extraction once on their meta
  (`imageExtraction: { settingId }`), read by the shared
  `isImageExtractionEnabled`; drops `ConnectorConfig.imageExtractionEnabled`
  and the Confluence-only helper.
- Turning the image setting off sets `knowledge_connector.image_removal_pending`
  (migration 0412); the next sync removes the indexed images and clears it, so
  ordinary syncs no longer scan a connector's documents for images.
- Gemini specifics (key, Gemini 3 model resolution, error classification) live
  in the adapter. Requests send `thinkingLevel: LOW`, and any finish reason other
  than STOP fails without paid retries.
- Image reads get their own admission budget (`vision`,
  KB_CONFIG_VISION_REQUESTS_PER_MINUTE, default 300) instead of sharing OCR's.
- One zod schema validates the reply and generates the request's JSON schema.
- The prompt is source-neutral and treats text in an image as content, never
  instructions; sharp loads lazily so it can only fail image documents.
- Full relisting after held-back work is one named rule
  (`noticeRequiresFullListing`).
@waleedlatif1

Copy link
Copy Markdown
Collaborator

@greptile

@waleedlatif1

Copy link
Copy Markdown
Collaborator

@cubic-dev-ai review this PR

@cubic-dev-ai

cubic-dev-ai Bot commented Oct 13, 2026

Copy link
Copy Markdown
Contributor

@cubic-dev-ai review this PR

@waleedlatif1 I have started the AI code review. It will take a few minutes to complete.

@cubic-dev-ai cubic-dev-ai Bot left a comment •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All reported issues were addressed across 51 files

Heads up: you’ve reached your flex budget. Increase your flex budget or wait for usage to reset.

You've manually re-run cubic several times on this PR. Each manual re-review checks the full PR again and counts toward your usage quota. To preserve your usage limits, we recommend letting cubic automatically review new commits.

Turn on auto-fix | Re-trigger cubic

Comment thread apps/sim/lib/knowledge/documents/vision-checkpoints.ts
Comment thread apps/sim/connectors/confluence/meta.ts
Comment thread apps/sim/lib/knowledge/documents/image-extraction.ts Outdated
Comment thread apps/sim/lib/knowledge/connectors/member-sync-engine.ts
Comment thread apps/sim/lib/knowledge/documents/image-extraction.ts Outdated
- Every Gemini attempt takes its own admission, and 429/503 records a provider
  cooldown for the key; the retry delay is read from the body's RetryInfo when
  no Retry-After header is sent.
- The image is base64-encoded once per extraction, rotated upright even when it
  already fits, and flattened onto white before a JPEG fallback.
- Cost goes through calculateBillableModelCost; both indexing charges share one
  recordIndexingUsage helper.
- Reconciliation waits until queued image removal finishes, and the removal walks
  by id instead of rescanning from the start.
- The per-sync image cap is at least 1, so a sync can always finish.
- canExtractImages lives with the extractor; isSemanticImage treats an undeclared
  policy as OCR, so callers drop their fallbacks.
- Tests: unique embedding vectors, restored pool keys, waits for dispatched syncs,
  fake timers in finally, native noise fixtures; drops mock-only assertions.
@waleedlatif1

Copy link
Copy Markdown
Collaborator

@greptile

@waleedlatif1

Copy link
Copy Markdown
Collaborator

@cubic-dev-ai review this PR

@cubic-dev-ai

cubic-dev-ai Bot commented Oct 13, 2026

Copy link
Copy Markdown
Contributor

@cubic-dev-ai review this PR

@waleedlatif1 I have started the AI code review. It will take a few minutes to complete.

@cubic-dev-ai cubic-dev-ai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No issues found across 52 files

Confidence score: 5/5

  • Automated review surfaced no issues in the provided summaries.
  • No files require special attention.

Heads up: you’ve reached your flex budget. Increase your flex budget or wait for usage to reset.

You've manually re-run cubic several times on this PR. Each manual re-review checks the full PR again and counts toward your usage quota. To preserve your usage limits, we recommend letting cubic automatically review new commits.

Turn on auto-fix | Re-trigger cubic

This branch was successfully deployed

1 active deployment
Preview — aa204463 Deployed Oct 13, 2026 by vercel[bot]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants