Skip to content

Implement Unicode-aware word motion (w/e/b/diw/ciw/de/db) - #296

Merged
panarch merged 2 commits into
mainfrom
unicode-word-motion
Feb 8, 2026
Merged

panarch merged 2 commits into
mainfrom
unicode-word-motion

Conversation

@panarch

@panarch panarch commented Feb 8, 2026 •

Copy link
Copy Markdown
Member

Summary

  • edtui's built-in word motions use is_ascii_alphanumeric(), causing CJK characters to each be treated as a separate word boundary. This replaces them with custom Unicode-aware implementations using char::is_alphanumeric().
  • Fixes vim-incompatible behaviors: de deleting trailing whitespace, db including cursor char, w not skipping leading whitespace on line crossings, b going to last char instead of word start on line crossings.
  • Removes insert-mode Alt+F/Alt+B word motions and their keymap help entries.

Test plan

  • cargo clippy --all-targets -- -D warnings passes
  • cargo test --workspace — 71 tests pass (19 new: 8 CJK + 11 word motion)
  • Manual testing with Korean text for w/e/b/diw/ciw

Summary by CodeRabbit

  • Improvements

    • Enhanced Unicode/CJK-aware word navigation and selection for more accurate cursor movement across languages and lines.
  • Changes

    • Removed Alt+F and Alt+B word-navigation shortcuts in insert mode and updated the displayed keymap accordingly.
  • Tests

    • Added extensive automated tests covering word-motion, cross-line behavior, and CJK scenarios.

edtui's built-in word motions use is_ascii_alphanumeric(), causing CJK
characters to be classified as Unknown (each character treated as its own
word boundary). Replace with custom Unicode-aware implementations using
char::is_alphanumeric() so Korean/CJK text is handled correctly.

- Add char_class, skip_ws (cross-line), skip_ws_rev helpers
- Add move_word_forward, move_word_backward, move_word_forward_to_end
- Add select_inner_word for diw/ciw
- Fix de to not delete trailing whitespace (remove extra MoveForward)
- Fix db to use exclusive motion (don't include cursor char)
- Fix w to skip leading whitespace when crossing lines
- Fix b to find word start (not last char) when crossing lines
- Remove insert-mode Alt+F/Alt+B word motions and keymap help entries
@panarch panarch self-assigned this Feb 8, 2026
@coderabbitai

coderabbitai Bot commented Feb 8, 2026 •

Copy link
Copy Markdown
📝 Walkthrough

Walkthrough

Refactors editor word-motion into Unicode-aware helper functions, removes Alt+F/Alt+B insert-mode bindings and keymap entries, updates normal/visual mode transitions to call the new helpers, and adds extensive word-motion tests (including CJK) plus small Tester helpers for assertions.

Changes

Cohort / File(s) Summary
Word Motion Infrastructure
tui/src/transitions/notebook/textarea.rs
Added Unicode-aware word-motion subsystem: char_class, skip_ws/skip_ws_rev, max_col_for_mode, update_visual_selection, forward/backward motions (move_word_forward, move_word_forward_to_end, move_word_backward), and select_inner_word. Exposed as pub(super) for local use.
Mode Handler Refactors
tui/src/transitions/notebook/editing_normal_mode.rs, tui/src/transitions/notebook/editing_visual_mode.rs
Replaced direct MoveWord* action executions with calls to the new helper functions; adjusted delete/selection flows (e.g., DeleteInsideWord, DeleteWordBack) to use selection helpers. Removed references to removed MoveWord* enum variants.
Insert-Mode Keybindings & Keymap Display
tui/src/context/notebook.rs, tui/src/views/dialog/editor_keymap.rs
Removed Alt+F / Alt+B handling and the corresponding keymap display entries for Alt-word navigation in insert mode; deleted related imports/usages.
Tests & Test Utilities
tui/tests/editor_word_motion.rs, tui/tests/editor_cjk.rs, tui/tests/tester.rs
Added comprehensive word-motion tests (including CJK and cross-line scenarios) and two Tester helpers: cursor() and editor_text() for inspecting editor state in tests.

Estimated code review effort

🎯 3 (Moderate) | ⏱️ ~25 minutes

Possibly related PRs

Suggested labels

codex

Poem

🐰 I hop through words in Unicode light,
Skipping spaces, left and right,
Alt-keys trimmed, helpers sing,
Tests verify each CJK spring,
Little rabbit cheers the night ✨

🚥 Pre-merge checks | ✅ 3
✅ Passed checks (3 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title accurately and specifically summarizes the main change: implementing Unicode-aware word motion with specific vim commands (w/e/b/diw/ciw/de/db).
Docstring Coverage ✅ Passed Docstring coverage is 100.00% which is sufficient. The required threshold is 80.00%.

✏️ Tip: You can configure your own custom pre-merge checks in the settings.

✨ Finishing touches
  • 📝 Generate docstrings
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Post copyable unit tests in a comment
  • Commit unit tests in branch unicode-word-motion

No actionable comments were generated in the recent review. 🎉

🧹 Recent nitpick comments
tui/src/transitions/notebook/textarea.rs (1)

121-132: skip_ws_rev relies on implicit synchronization between the iterator position and idx.col.

The rev().skip(…) pattern works correctly but is fragile: the correctness depends on skip being exactly line.len() - (idx.col + 1) so the first yielded character corresponds to idx.col, and each subsequent character lines up with the decrement. Consider a simpler formulation that makes the index tracking explicit:

♻️ Optional: explicit index loop
 fn skip_ws_rev(lines: &Lines, idx: &mut Index2) {
     if let Some(line) = lines.get(RowIndex::new(idx.row)) {
-        let skip = line.len().saturating_sub(idx.col + 1);
-        for ch in line.iter().rev().skip(skip) {
-            if !ch.is_whitespace() {
+        while idx.col > 0 {
+            if !line[idx.col].is_whitespace() {
                 break;
             }
             idx.col = idx.col.saturating_sub(1);
         }
     }
 }

Note: this simplification omits the check for the initial position (idx.col) itself — you'd need to decide whether the current position should also be tested. The existing code does test it, which matches the caller's expectation (cursor is already at col - 1 when skip_ws_rev is called). Either way, making the index explicit avoids a subtle coupling.


Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands and usage tips.

@gluesql-coverage-bot

gluesql-coverage-bot Bot commented Feb 8, 2026 •

Copy link
Copy Markdown

Glues Coverage Report

  • Commit: a9221f6e36b59ef9d82d72c54f02d1c6c0aac29c
  • Timestamp: 2026-02-08T162924Z
  • Report: View report

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🤖 Fix all issues with AI agents
In `@tui/src/transitions/notebook/textarea.rs`:
- Around line 235-290: The recursive upward moves in move_word_backward_once
(calls to itself from the col==0 branch and the whitespace-at-col-0 branch after
skip_ws_rev) can cause deep recursion on many blank lines; refactor this
function to an iterative loop: use a while loop that moves idx/ editor.cursor up
by one row when at col 0 or when whitespace at col 0 instead of recursing,
reusing skip_ws_rev and the same char_class/backward-scan logic (Index2,
editor.lines, editor.cursor) until you find the word start or reach the buffer
start; ensure each iteration updates idx/cursor and continues rather than
calling move_word_backward_once.
🧹 Nitpick comments (2)
tui/tests/editor_cjk.rs (1)

8-23: setup() helper is duplicated from editor_word_motion.rs.

This helper is identical to the one in editor_word_motion.rs (lines 8–21). Consider extracting it into tester.rs as a shared method (e.g., Tester::setup_editor_with_text) to avoid the duplication. Not blocking since test-file-level helpers are common.

tui/src/transitions/notebook/textarea.rs (1)

122-132: skip_ws_rev iterates beyond the logical stop point when idx.col saturates at 0.

When all characters from idx.col down to 0 are whitespace, idx.col saturates at 0 but the iterator continues visiting remaining characters (all of which are whitespace), calling saturating_sub(1) as a no-op each time. The result is still correct (caller handles the "still on whitespace" case at lines 259-266), but the extra iterations are wasted work.

An early break when idx.col == 0 would short-circuit cleanly:

Proposed fix
         for ch in line.iter().rev().skip(skip) {
             if !ch.is_whitespace() {
                 break;
             }
+            if idx.col == 0 {
+                break;
+            }
             idx.col = idx.col.saturating_sub(1);
         }

Avoids theoretical stack overflow on documents with many consecutive
blank or whitespace-only lines.
@panarch
panarch merged commit ee4467b into main Feb 8, 2026
6 checks passed
@panarch
panarch deleted the unicode-word-motion branch February 8, 2026 16:32
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant