Instructions to use Irfanuruchi/Phi-2-MLX-4bit with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- MLX
How to use Irfanuruchi/Phi-2-MLX-4bit with MLX:
# Make sure mlx-lm is installed # pip install --upgrade mlx-lm # if on a CUDA device, also pip install mlx[cuda] # Generate text with mlx-lm from mlx_lm import load, generate model, tokenizer = load("Irfanuruchi/Phi-2-MLX-4bit") prompt = "Once upon a time in" text = generate(model, tokenizer, prompt=prompt, verbose=True) - Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- LM Studio
- MLX LM
How to use Irfanuruchi/Phi-2-MLX-4bit with MLX LM:
Generate or start a chat session
# Install MLX LM uv tool install mlx-lm # Generate some text mlx_lm.generate --model "Irfanuruchi/Phi-2-MLX-4bit" --prompt "Once upon a time"
- Atomic Chat
Update README.md
Browse files
README.md
CHANGED
|
@@ -11,3 +11,67 @@ tags:
|
|
| 11 |
base_model: microsoft/phi-2
|
| 12 |
library_name: mlx
|
| 13 |
---
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 11 |
base_model: microsoft/phi-2
|
| 12 |
library_name: mlx
|
| 13 |
---
|
| 14 |
+
|
| 15 |
+
|
| 16 |
+
# Phi-2 MLX 4-bit
|
| 17 |
+
|
| 18 |
+
This repository provides a **4-bit MLX-quantized version** of
|
| 19 |
+
[Microsoft Phi-2](https://huggingface.co/microsoft/phi-2),
|
| 20 |
+
optimized for **fast, low-memory local inference** on Apple Silicon.
|
| 21 |
+
|
| 22 |
+
This variant prioritizes **speed and minimal RAM usage**, making it ideal
|
| 23 |
+
for laptops and on-device experimentation.
|
| 24 |
+
|
| 25 |
+
---
|
| 26 |
+
|
| 27 |
+
## Model Details
|
| 28 |
+
|
| 29 |
+
- **Base model:** microsoft/phi-2
|
| 30 |
+
- **Architecture:** Decoder-only Transformer
|
| 31 |
+
- **License:** MIT
|
| 32 |
+
- **Quantization:** MLX static quantization (≈4.5 bits per weight)
|
| 33 |
+
- **Target hardware:** Apple Silicon (M1 / M2 / M3)
|
| 34 |
+
|
| 35 |
+
---
|
| 36 |
+
|
| 37 |
+
## Performance Characteristics
|
| 38 |
+
|
| 39 |
+
| Metric | Value |
|
| 40 |
+
|-----|------|
|
| 41 |
+
| Disk size | ~1.5–1.7 GB |
|
| 42 |
+
| Peak RAM usage | ~1.6–1.8 GB |
|
| 43 |
+
| Inference speed | Fast |
|
| 44 |
+
| Instruction quality | Good |
|
| 45 |
+
|
| 46 |
+
---
|
| 47 |
+
|
| 48 |
+
## Usage
|
| 49 |
+
|
| 50 |
+
```bash
|
| 51 |
+
mlx_lm.generate \
|
| 52 |
+
--model /path/to/Phi-2-MLX-4bit \
|
| 53 |
+
--prompt "Explain the FFT in simple terms." \
|
| 54 |
+
--max-tokens 120
|
| 55 |
+
```
|
| 56 |
+
|
| 57 |
+
---
|
| 58 |
+
|
| 59 |
+
## Notes
|
| 60 |
+
|
| 61 |
+
- This is a **quantized conversion**, not a fine-tune.
|
| 62 |
+
- The 4-bit version is best for:
|
| 63 |
+
- faster inference
|
| 64 |
+
- lower memory usage
|
| 65 |
+
- interactive local testing
|
| 66 |
+
- For higher-quality reasoning and instruction-following, see the **5-bit** variant.
|
| 67 |
+
|
| 68 |
+
---
|
| 69 |
+
|
| 70 |
+
## License
|
| 71 |
+
|
| 72 |
+
This repository redistributes a quantized MLX conversion of **Microsoft Phi-2**.
|
| 73 |
+
|
| 74 |
+
- **Original model license:** MIT
|
| 75 |
+
- **MLX conversion:** MIT
|
| 76 |
+
|
| 77 |
+
See `LICENSE` for details.
|