Skip to content

feat: capture agent-stated intent via MCP tool into session.jsonl #27

Open

Description

@codeprakhar25

Problem

Traces currently record prompt excerpts, line ranges, and files changed — but the intent (the why behind the change) is almost never populated. The intent field exists in AgentdiffMetadata and flows through the full pipeline, but it defaults to "unspecified" because there is no reliable acquisition path.

The prompt excerpt is used as a proxy, but prompts describe what to do, not why:

  • "fix the auth bug" — was this a security incident? a test failure? a refactor?
  • "do the thing we talked about" — completely opaque
  • Multi-step sessions produce many tool calls under one vague prompt

This is the bottleneck for making PR comments and review context meaningful.

Approach

Have the agent state its own intent as a session.jsonl event right before it commits (or at session end). The agent has full context of what it did and why — instead of reverse-engineering intent from diffs/prompts, just ask it.

Mechanism: MCP tool call (Option A)

  1. Add a set_intent MCP tool (or extend record_context) that writes an intent event directly to session.jsonl
  2. Agent rules (.cursor/rules/, CLAUDE.md, etc.) instruct agents: "Before committing, call set_intent with a 1-2 sentence description of why you made these changes"
  3. prepare-ledger.py reads the most recent type: "intent" event from session.jsonl for the matching session and uses it as the intent field
  4. Everything downstream (finalize, report, PR comments) works as-is

New session.jsonl event format

{
  "timestamp": "2026-05-21T12:00:00Z",
  "type": "intent",
  "agent": "cursor",
  "session_id": "conv-xyz",
  "description": "Extract auth middleware into shared module to fix duplicate token validation across 3 route handlers"
}

Implementation Steps

  1. MCP tool: Add set_intent tool to agentdiff-mcp.rs that writes intent events to session.jsonl
  2. prepare-ledger.py: Read type: "intent" events from session.jsonl, use matching session's intent as the intent field in the payload
  3. Agent rules: Add default rules for Cursor and Claude Code that instruct agents to call set_intent before committing
  4. Optional: intent_type field: Add a taxonomy classifier (bugfix/feature/refactor/test/docs/security/config) derived from the diff, as a fallback when no agent-stated intent exists
  5. Report update: Show intent_type + intent together in PR comment output

Why this works

  • Zero latency — no external API calls
  • Zero cost — no LLM inference
  • High quality — the agent has full session context
  • Fits existing architecture — session.jsonl → prepare-ledger → finalize → traces
  • Works across all MCP-capable agents

Acceptance Criteria

  • set_intent MCP tool writes intent events to session.jsonl
  • prepare-ledger.py picks up intent events and populates the intent field
  • PR comments show agent-stated intent instead of raw prompt excerpts
  • Agent rules exist for at least Cursor and Claude Code
  • Traces without agent-stated intent fall back to prompt-derived or "unspecified"

Activity

  1. MHASK commented on Jul 17, 2026

    @MHASK

    This is a strong direction. I would treat agent-stated intent as useful review evidence, but not as the only source of truth. The agent has context, but it is also the actor being audited.

    A few schema/design suggestions that may make this more robust:

    • Store intent.source, e.g. agent_stated, human_declared, commit_message, issue_title, fallback_classifier. That lets reports distinguish intent captured from the agent from intent declared by a human.
    • Store intent.scope or intent_target, e.g. files, symbols, issue number, task id, or PR id the intent claims to address. This helps detect drift later.
    • Keep both agent_stated_intent and observed_change_summary. If they disagree, surface that mismatch instead of overwriting one with the other.
    • Add risk_categories[] separately from intent_type, e.g. auth, payments, ci, dependencies, infra, secrets, migrations. A change can be a refactor and still touch auth.
    • Consider a confidence or evidence field for derived values. Human-declared or issue-linked intent is different from a classifier guess.
    • Record the commit/session boundary carefully for multi-step sessions. A session-level intent can become misleading if one long agent session produces several commits with different purposes.

    For PR comments, the most useful output may be:

    Claimed intent: <agent/human stated intent>
    Observed change: <diff-derived summary>
    Risk categories: <auth, deps, ci, ...>
    Potential drift: <none / touched billing while intent mentions login>
    

    That makes intent capture actionable for maintainers rather than just another metadata field.

  2. codeprakhar25 commented on Jul 18, 2026

    @codeprakhar25
    OwnerAuthor

    Thanks, this is very useful!

    So the current status is: set_intent shipped in v0.1.29. intent + intent_type (bugfix/feature/refactor/test/docs/security/performance/config/dependency) flow through session.jsonl → prepare-ledger → finalize-ledger → traces → reports today.

    Taking two of your suggestions as follow-up work, since they're cheap and add real signal without another schema layer:

    • intent.source (agent_stated | fallback) — right now intent can come from a set_intent event or from pending-context payload with no marker distinguishing them. Small addition, closes that gap.
    • risk_categories[] — derivable deterministically from touched paths (auth/, payments/, .github/workflows/, Cargo.toml/package.json, migrations/) with no LLM call, consistent with the zero-cost design goal already in this issue.
    • Issue/ task ID can be a great addition here intent.scope/intent_target

    Already implemented/thoughts:

    • separate confidence/evidence field — redundant with source; the source tag already implies confidence tier.
    • observed_change_summary + drift detection against claimed intent — agree this is the highest-value idea here, but it's a real feature (diff summarizer + comparison), not a field. Worth its own issue rather than folding into this one.

    Session/commit boundary: already handled — finalize-ledger.py strips consumed type: intent events from session.jsonl right after the commit they belong to, so long sessions with multiple commits don't bleed intent across boundaries.

    The main challenge with more increasing this is how scalably and eff. we can store this in large codebase!

  3. MHASK commented on Jul 18, 2026

    @MHASK

    That breakdown makes sense, especially keeping source as the confidence tier instead of adding another field.

    For large repos, I would probably avoid storing symbol/file-level derived data directly on every session event. A lighter shape could be:

    • keep the raw intent event small: intent, intent_type, source, scope/task ref
    • compute risk_categories from the touched paths during ledger/finalize
    • store a compact per-commit summary keyed by commit/session id
    • keep any expensive observed-change summary as a generated artifact, not mandatory ledger state

    So the durable ledger stays cheap, and richer drift detection can be opt-in or cached by commit hash. That also leaves room for large monorepos where the same touched-path classification may be enough for review routing without persisting a full semantic summary every time.

  4. codeprakhar25 commented on Jul 28, 2026

    @codeprakhar25
    OwnerAuthor

    Got late, missed this!

    But yeah, this should be the path, opt-in for large repos, especially monorepos. Will double check for the scope/task-ref definition. Also the definition will be evolved as it is developed. Would you be open to contribute to the repo with this one?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions