Skip to content

Add Python language support via Tree-sitter (consolidates #21–#25) - #27

Open
awe-srush wants to merge 7 commits into
google:mainfrom
awe-srush:consolidated-tree-sitter
Open

awe-srush wants to merge 7 commits into
google:mainfrom
awe-srush:consolidated-tree-sitter

Conversation

@awe-srush

Copy link
Copy Markdown
Contributor

Summary

Adds Python language support to Vanir via a tree-sitter-backed parser, along with the shared infrastructure required to plug additional tree-sitter languages into the parser pipeline in the future.

This PR consolidates #21–#25 into a single atomic change as requested on the original PRs, so CI can validate the complete Tree-sitter Python integration in one pass. Each commit preserves its original author for attribution.

What this adds

Vanir can now scan .py files for missing security patches. Once merged, no changes to any upstream Vanir logic are required - the new PythonParser is auto-discovered by get_parser_class() and dispatched for .py files on Python 3.10+. On Python 3.9, tree-sitter dependencies are not installed and .py files are silently skipped.

Commits (in dependency order)

1. feat(language_parsers): add TreeSitterParserBase — @awe-srush

Introduces a shared abstract base class between AbstractLanguageParser and future tree-sitter-backed parsers. Handles the full ParseResults assembly pipeline (function discovery via tree-sitter queries, FunctionChunkBase / LineChunkBase construction, ParseError collection) using iterative cursor-based DFS to avoid Python recursion limits. Language-agnostic — all grammar-specific logic is delegated to three abstract methods (_extract_param_names, _extract_annotations, _collect_locals_calls) that concrete subclasses implement. Tree-sitter imports are lazy so the module remains importable on Python 3.9.

2. bug(language_parsers): fix subclass discovery to support multi-level parser hierarchies — @awe-srush

Fixes a bug in get_parser_class() where parsers registered two or more levels deep in the AbstractLanguageParser hierarchy were silently ignored. The existing implementation used __subclasses__(), which only returns direct subclasses — meaning any parser inheriting through TreeSitterParserBase was invisible. Replaces it with a recursive _all_subclasses() helper that walks the full hierarchy and filters out abstract intermediates via __abstractmethods__. Existing C++ and Java parser behavior is unchanged.

3. feat(language_parsers): add Python parser via tree-sitter — @awe-srush

Concrete PythonParser on top of TreeSitterParserBase. Implements all three abstract methods against Python's tree-sitter grammar, handling the full range of Python parameter forms (typed, defaulted, *args, **kwargs, positional-only markers), return/parameter type annotations, and local/call collection through a compiled query covering standard assignments, augmented assignments, for-loop targets, walrus expressions, and direct calls. Nested-function byte ranges are excluded so inner-scope names don't leak into the outer function's signature.

4. feat(requirements): add tree-sitter dependencies for Python parser — @avshar15

Adds tree-sitter and tree-sitter-python to requirements.txt, both guarded with python_version>='3.10' so Python 3.9 environments remain unaffected.

5. fix(requirements): bound tree-sitter version to <0.26 — @avshar15

Pins the upper bound after API changes in tree-sitter 0.26.

6. feat(language_parsers): add Bazel build targets for Python parser — @ishijo

Adds tree_sitter_base and python_parser build targets and wires python_parser into the language_parsers aggregator so the dispatcher picks it up automatically.

7. chore(language_parsers): drop py_test target from python BUILD.bazel — @ishijo

Removes an unnecessary py_test target.

Note for maintainers

Lock files (requirements_lock_3.10.txt through requirements_lock_3.13.txt) need to be regenerated after merge:
bazel run //requirements:requirements_3.10.update
bazel run //requirements:requirements_3.11.update
bazel run //requirements:requirements_3.12.update
bazel run //requirements:requirements_3.13.update

Validation

  • Detects GHSA-xjgw-4wvw-rgm4 (path traversal in mcp-atlassian) end-to-end on Python source
  • Clean runs on Django and Scrapy at ~290 files/sec
  • Multi-version Docker matrix passing on Python 3.9–3.13

Author attribution

Per the review guidance, each commit preserves its original author. Base commit is mine (@awe-srush); additional contributors @avshar15 and @ishijo are attributed on their own commits and will be picked up as Co-authored-by: trailers on import.

Supersedes

copybara-service Bot pushed a commit that referenced this pull request Oct 22, 2026
Add pure python Tree-sitter base parser framework.

This change adapts the core Tree-sitter AST traversal routines from GitHub PR #27 (#27) and refactors them into `GenericTreesitterParser` with unit tests.

Co-authored-by: Awe Srush <srushti.erudite@gmail.com>
Co-authored-by: Avantika Sharma <avantikaspare15@gmail.com>
Co-authored-by: Ishika Johari <ishijohari@gmail.com>
PiperOrigin-RevId: 978013230
Change-Id: I8e5edd0f3e44185d878f3b1c953227d38d53f589
GitOrigin-RevId: 0b3f04926bfe3ad782f39725b558fc6f57dc7b94
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants