Irfanuruchi commited on
Commit
031e8f8
·
verified ·
1 Parent(s): 2b0af8d

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +64 -0
README.md CHANGED
@@ -11,3 +11,67 @@ tags:
11
  base_model: microsoft/phi-2
12
  library_name: mlx
13
  ---
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
11
  base_model: microsoft/phi-2
12
  library_name: mlx
13
  ---
14
+
15
+
16
+ # Phi-2 MLX 4-bit
17
+
18
+ This repository provides a **4-bit MLX-quantized version** of
19
+ [Microsoft Phi-2](https://huggingface.co/microsoft/phi-2),
20
+ optimized for **fast, low-memory local inference** on Apple Silicon.
21
+
22
+ This variant prioritizes **speed and minimal RAM usage**, making it ideal
23
+ for laptops and on-device experimentation.
24
+
25
+ ---
26
+
27
+ ## Model Details
28
+
29
+ - **Base model:** microsoft/phi-2
30
+ - **Architecture:** Decoder-only Transformer
31
+ - **License:** MIT
32
+ - **Quantization:** MLX static quantization (≈4.5 bits per weight)
33
+ - **Target hardware:** Apple Silicon (M1 / M2 / M3)
34
+
35
+ ---
36
+
37
+ ## Performance Characteristics
38
+
39
+ | Metric | Value |
40
+ |-----|------|
41
+ | Disk size | ~1.5–1.7 GB |
42
+ | Peak RAM usage | ~1.6–1.8 GB |
43
+ | Inference speed | Fast |
44
+ | Instruction quality | Good |
45
+
46
+ ---
47
+
48
+ ## Usage
49
+
50
+ ```bash
51
+ mlx_lm.generate \
52
+ --model /path/to/Phi-2-MLX-4bit \
53
+ --prompt "Explain the FFT in simple terms." \
54
+ --max-tokens 120
55
+ ```
56
+
57
+ ---
58
+
59
+ ## Notes
60
+
61
+ - This is a **quantized conversion**, not a fine-tune.
62
+ - The 4-bit version is best for:
63
+ - faster inference
64
+ - lower memory usage
65
+ - interactive local testing
66
+ - For higher-quality reasoning and instruction-following, see the **5-bit** variant.
67
+
68
+ ---
69
+
70
+ ## License
71
+
72
+ This repository redistributes a quantized MLX conversion of **Microsoft Phi-2**.
73
+
74
+ - **Original model license:** MIT
75
+ - **MLX conversion:** MIT
76
+
77
+ See `LICENSE` for details.