Skip to content

Add tfidf_transformer - #3178

Merged
davisking merged 3 commits into
davisking:masterfrom
arrufat:tfidf-transformer
Oct 31, 2026
Merged

davisking merged 3 commits into
davisking:masterfrom
arrufat:tfidf-transformer

Conversation

@arrufat

@arrufat arrufat commented Oct 31, 2026 •

Copy link
Copy Markdown
Contributor

As discussed in #3176, this PR adds tfidf_transformer, which learns TF-IDF weights from sparse vectors of term counts and turns documents into TF-IDF vectors that the sparse linear trainers can use directly.

I needed it for a page classifier that trains dlib's linear SVM on hashed n-grams, where it replaced our hand-written TF-IDF code with identical results. The defaults follow scikit-learn's TfidfTransformer (smoothed idf, raw counts, unit length).

In a separate commit, I also added normalize() and normalize_inplace() for sparse vectors, since the transformer needs it. Let me know if you'd rather not have it, and I'll drop it.

The only doubt I have is around the name normalize_inplace: I followed make_sparse_vector_inplace, but maybe there's a better one.

Closes #3176

A tool that learns TF-IDF weights from sparse term-count vectors.
@arrufat

arrufat commented Oct 31, 2026

Copy link
Copy Markdown
Contributor Author

If you're ok with it, I might submit a couple more PRs after this one:

  • feature hashing for text n-grams, using the existing murmur_hash3
  • a text classification example showing how all these pieces work together

@davisking

Copy link
Copy Markdown
Owner

Nice, this is a great PR.

@davisking
davisking merged commit 6f27b54 into davisking:master Oct 31, 2026
13 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Sparse TF-IDF transformer?

2 participants