htdemucs core for one 7.8 s segment, float32, coremltools 9.0

README.md

@@ -0,0 +1,37 @@
+---
+license: mit
+library_name: coreml
+pipeline_tag: audio-to-audio
+tags: [coreml, core-ml, source-separation, stem-separation, demucs, htdemucs, apple, on-device, slurper]
+---
+
+# htdemucs on Core ML
+
+[htdemucs](https://github.com/facebookresearch/demucs) (Meta, MIT) as a Core ML `.mlpackage`, the drums, bass and other model behind [slurper](https://github.com/TrevorS/slurper). Runs on the GPU on macOS 15 and later.
+
+## Graph
+
+The model's real-valued core for one 7.8 s segment at 44.1 kHz, `HTDemucs.forward` from the normalization to just before `_mask`, in float32 because float16 overflows:
+
+| name | shape | meaning |
+|---|---|---|
+| `mix` (input) | `[1, 2, 343980]` | stereo segment |
+| `spec` (input) | `[1, 4, 2048, 336]` | `_magnitude(_spec(mix))`: left real, left imaginary, right real, right imaginary |
+| `time` (output) | `[1, 8, 343980]` | denormalized time branch, `[source, channel]` |
+| `freq` (output) | `[1, 16, 2048, 336]` | denormalized frequency branch, `[source, channel, real/imaginary]` |
+
+Sources are drums, bass, other, vocals. The host computes the STFT (`_spec`: 4096-point periodic Hann, hop 1024, reflect padding, normalized), inverts `freq` with `_ispec`, adds `time`, and crossfades segments overlapping by a quarter as `apply_model` does. `Sources/SlurperKit/Demucs.swift` in slurper is a reference host.
+
+## Conversion
+
+`scripts/convert_htdemucs.py` in slurper: `torch.jit.trace` of the core with the attention fast path disabled, then coremltools 9.0 to an ML program with float32 precision. Before saving, the Core ML output on the GPU is compared with PyTorch on a synthetic segment: drums 114 dB, bass 128 dB, other 114 dB, vocals 100 dB SDR.
+
+## Files
+
+- `htdemucs_fp32.mlpackage/Manifest.json`
+- `htdemucs_fp32.mlpackage/Data/com.apple.CoreML/model.mlmodel`
+- `htdemucs_fp32.mlpackage/Data/com.apple.CoreML/weights/weight.bin` (209 MB)
+
+## Attribution
+
+Hybrid Transformer Demucs by Simon Rouard, Francisco Massa and Alexandre Défossez (Meta AI), MIT. This conversion is MIT as well.

htdemucs_fp32.mlpackage/Data/com.apple.CoreML/model.mlmodel

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:1b97e7f457df338347ee0049246f2701e95977e25cdfb12896e00b0a26d2e28c
+size 538090

htdemucs_fp32.mlpackage/Data/com.apple.CoreML/weights/weight.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:a901ad380afd63e364931b20174392063146ebdb56e33c4e06afb265a651e5ec
+size 209160960

htdemucs_fp32.mlpackage/Manifest.json

@@ -0,0 +1,18 @@
+{
+    "fileFormatVersion": "1.0.0",
+    "itemInfoEntries": {
+        "4A30A2E9-CA49-4CF1-8F6A-092F75867083": {
+            "author": "com.apple.CoreML",
+            "description": "CoreML Model Specification",
+            "name": "model.mlmodel",
+            "path": "com.apple.CoreML/model.mlmodel"
+        },
+        "8A4F554E-F7A4-46C4-9905-69363B79EF47": {
+            "author": "com.apple.CoreML",
+            "description": "CoreML Model Weights",
+            "name": "weights",
+            "path": "com.apple.CoreML/weights"
+        }
+    },
+    "rootModelIdentifier": "4A30A2E9-CA49-4CF1-8F6A-092F75867083"
+}