Previously downloaded MLX models not available for use

@cyrilzakka

Really like the app, great idea for Mac AI integration! I have around 200GB of MLX models already downloaded through mlx_lm.server that I use with LibreChat - however, there appears to be no way to use these with HuggingChat. Please let me know if it would be possible to add this functionality.

According to this post MLX is supported and indeed works but only with the two models available in the HC Settings panel (I've tried the Qwen2.5-3B-Instruct model).

Now supports MLX inference. Press CMD+Shift+\ to switch to local inference

Originally posted by @cyrilzakka in #32

This is probably because of the difference where HC and MLX store their models. In HC they're at ~/Documents/huggingface, but with MLX they're at ~/.cache/huggingface. I had hoped to symlink the two but it's not possible as the directory structure is very different. See below for the structure. Thanks!

HuggingChat

~/Documents
    │
    └── huggingface
        │
        └── models
            │
            ├── argmaxinc
            │   └── whisperkit-coreml
            │       └── openai_whisper-large-v3-v20240930
            │
            ├── mlx-community
            │   └── Qwen2.5-3B-Instruct-bf16
            │
            ├── openai
            │   └── whisper-large-v3
            │
            └── ...

mlx_lm saved models

~/.cache
    │
    └── huggingface
        │
        └── hub
            │
            ├── models--mlx-community--Llama-3.2-3B-Instruct-4bit
            │   ├── blobs
            │   ├── refs
            │   └── snapshots
            │
            ├── models--mlx-community--Llama-3.3-70B-Instruct-4bit
            │   ├── blobs
            │   ├── refs
            │   └── snapshots
            │
            └── ...

I had AI write this script to reformat and link the existing MLX models into same directory where HuggingChat stores its downloaded models. (This method worked to enable support for local cached MLX models with LMStudio)

However, the two inference models HC shows in Settings are hardcoded on lines 86 and 87 in ModelManager.swift.
I attempted to add a local model following the same format, but I am not a developer and failed miserably. The app built but did not recognize Qwen2.5-3B-Instruct-bf16 which had been previously downloaded through Settings. Hopefully this can help point someone with the capabilities in the right direction.

Model copy script:

#!/bin/bash

HF_BASE="/Users/vpnwiz/.cache/huggingface/hub"
MLX_BASE="/Users/vpnwiz/Documents/huggingface/models/mlx-community"

for model_dir in "$HF_BASE"/models--mlx-community--*; do
    # Extract model name from directory
    model_name=$(basename "$model_dir" | sed 's/models--mlx-community--//')
    
    # Create target directory if it doesn't exist
    mkdir -p "$MLX_BASE/$model_name"
    
    # Find the snapshot directory (assumes latest)
    snapshot_dir=$(find "$model_dir/snapshots" -maxdepth 1 -type d | sort | tail -n 1)
    
    # Create symbolic links for each model file
    for file in "$snapshot_dir"/*.json "$snapshot_dir"/*.safetensors; do
        if [ -f "$file" ]; then
            ln -sf "$file" "$MLX_BASE/$model_name/$(basename "$file")"
        fi
    done
done

@vpnwiz

@vpnwiz thank you for the effort, this is a great first start! The app is currently in its last major revision until v1.0 and will try to incorporate your first point by then.

Navigation Menu

Repository navigation

Previously downloaded MLX models not available for use #39

HuggingChat

mlx_lm saved models

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Previously downloaded MLX models not available for use #39

Description

HuggingChat

mlx_lm saved models

Activity

vpnwiz commented on Jan 22, 2025

cyrilzakka commented on Jan 22, 2025

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions