Repository navigation
Sparse TF-IDF transformer? #3176
Description
I've been doing text classification with dlib's linear tools (svm_c_linear_dcd_trainer on sparse vectors, Platt scaling, cross-validation), and the only thing I had to write myself was the TF-IDF weighting. Would you take a small tfidf_transformer for sparse vectors, in the style of vector_normalizer?
template <typename sample_type> // dlib sparse vector
class tfidf_transformer
{
public:
void train(const std::vector<sample_type>& counts);
sample_type operator()(const sample_type& counts) const; // idf-weighted, unit length
void set_min_document_frequency(unsigned long); // default 1
void set_max_features(unsigned long); // default: all
void use_sublinear_tf(bool); // 1 + log(tf), default false
void use_smooth_idf(bool); // log((1+n)/(1+df)) + 1, default true
void normalize_to_unit_length(bool); // default true
unsigned long num_features() const;
};Output indices are renumbered to the kept features, so max_index_plus_one stays small for the trainers. Defaults match scikit-learn's TfidfTransformer. No tokenizing or hashing: the input is counts keyed by whatever indices the user has (word ids, or n-grams hashed with murmur_hash3).
If text features are out of scope, no problem.
Activity
davisking commentedon Oct 31, 2026
Sure, go for it.