Release Alberich-mini-v1 adapter and full benchmark results

README.md

@@ -60,7 +60,7 @@ Download this repository into a local directory and run from that directory. Ins
 ```python
 from inference import MultiDecider
 
-decider = MultiDecider()  # adapter in this directory; downloads pinned base separately
+decider = MultiDecider()  # downloads pinned base, then merges local adapter in RAM
 answers = decider.decide(
     context="Anna hat drei rote Ordner. Ben hat zwei blaue Ordner.",
     questions=[
@@ -109,21 +109,28 @@ BF16 backbone computation, gradient checkpointing, no LoRA dropout. Adapter tens
 
 The main stage contains 75114 English and 24886 German examples, including 21494 image examples and 2000 synthetic examples. Tasks cover natural-language inference, coreference, paraphrase, intent, sentiment, commonsense, numeric/temporal/rule reasoning, diagrams, visual relations and adapted multiple-choice OCR/document questions. See [training_summary.json](training_summary.json) for exact task counts and corpus hashes, and [DATA_SOURCES.md](DATA_SOURCES.md) for source and licensing notes.
 
-## Evaluation
+### In-memory merging (default)
+
+The inference helper now defaults to `merge_adapter=True`: it loads the separate base and adapter, then calls PEFT `merge_and_unload(safe_merge=True)`. This removes the separate LoRA operations during inference. No base or adapter files are modified, and this repository still distributes only the adapter.
+
+Use `MultiDecider(merge_adapter=False)` to reproduce the historical unmerged evaluation mode. Merging into BF16 weights is not numerically identical: in the 1000-question paired test, two answers changed, both from incorrect to correct. Individual option probabilities changed by up to 0.311; probability thresholds may therefore behave differently. This small accuracy increase does not establish a general quality improvement.
+
+## Evaluation
 
 These are **local subset results**, not official benchmark leaderboard scores. The fixed comparison has 1000 questions from 27 task families, with 37 per family and 38 for AI2D: 703 text and 297 image questions. It was sampled from held-out pools used during development, after training. It is not a newly collected blind evaluation. The earlier 200 pilot comparison cases were excluded. No further model tuning or calibration was performed on this 1000-question comparison.
 
 | System, local implementation | Correct | Accuracy | Mean ms/question | Peak allocated GPU GiB |
 |---|---:|---:|---:|---:|
-| **This adapter + Qwen3.5-2B** | **865/1000** | **86.5%** | 88 | 4.32 |
+| **This adapter + Qwen3.5-2B, merged in RAM (default)** | **867/1000** | **86.7%** | **63** | See paired test below |
+| This adapter + Qwen3.5-2B, unmerged (original run) | 865/1000 | 86.5% | 88 | 4.32 |
 | Qwen3.5-2B base, same readout without adapter | 700/1000 | 70.0% | 65 | 4.26 |
 | BARD-VL-4B, local OpenJev port | 809/1000 | 80.9% | 86 | 8.43 |
 | DiffusionVL-3B, local OpenJev port | 737/1000 | 73.7% | 86 | 7.24 |
 | Qwen3.5-4B, LitJev | 771/1000 | 77.1% | 164 | 8.78 |
 
-This adapter scored **587/703 (83.5%) on text**, **278/297 (93.6%) on images**, and **685/815 (84.0%) after excluding synthetic tasks**. Full aggregate per-task results are in [eval_results.json](eval_results.json).
+The original unmerged adapter scored **587/703 (83.5%) on text**, **278/297 (93.6%) on images**, and **685/815 (84.0%) after excluding synthetic tasks**. The merged variant scored **589/703 (83.8%) on text**, **278/297 (93.6%) on images**, and **687/815 (84.3%) after excluding synthetic tasks**. Original aggregate per-task results are in [eval_results.json](eval_results.json); the merge comparison is in [merged_adapter_results.json](merged_adapter_results.json).
 
-The unchanged Qwen3.5-2B base scored **700/1000 (70.0%)** using the same prompt, readout, question order and image settings, with LoRA disabled. The adapter improves overall accuracy by **+16.5 percentage points**. Base-only scores: **459/703 (65.3%) text** and **241/297 (81.1%) images**. On paired questions, the adapter fixes 224 base errors and introduces 59 errors on cases the base answered correctly.
+The unchanged Qwen3.5-2B base scored **700/1000 (70.0%)** using the same prompt, readout, question order and image settings, with LoRA disabled. The original unmerged adapter improves overall accuracy by **+16.5 percentage points**; the merged default by **+16.7 points**. Base-only scores: **459/703 (65.3%) text** and **241/297 (81.1%) images**. On paired questions, the original unmerged adapter fixes 224 base errors and introduces 59 errors on cases the base answered correctly.
 
 The base-only speed measurement was repeated as a standalone run with no concurrent model inference, using the same 1000 questions and timing boundaries. All 1000 selected answers matched the earlier base evaluation. Peak allocated memory is per benchmark process, not total GPU usage.
 
@@ -131,7 +138,20 @@ All systems received the same questions, options and option order. Images were c
 
 The OCR subsets are local multiple-choice adaptations with distractors, not open-ended DocVQA/TextVQA evaluations. Near-ceiling scores should not be read as general OCR accuracy. Dataset SHA256: `88cc14e7201b39b16545cd1e7578358a1747d528138d458c4351b2d27ed4e974`.
 
-## Limitations and intended use
+### Paired merge speed test
+
+A fresh sequential comparison in the same process measured **85.9 ms/question unmerged versus 63.5 ms merged**, a **26.1% latency reduction**, on the same 1000 questions. Accuracy was 865 versus 867 correct; 998 answers were identical. Merging took 0.22 seconds, excluding model loading. Peak allocated memory over the benchmark and the additional ten-question tests was 4.56 GiB unmerged and 4.50 GiB merged; these peaks have a different scope from the original single-question table.
+
+| Input | Questions per request | Unmerged median ms | Merged median ms |
+|---|---:|---:|---:|
+| Text | 1 | 57.7 | 49.1 |
+| Text | 10 | 141.7 | 107.1 |
+| Image | 1 | 112.0 | 98.1 |
+| Image | 10 | 256.7 | 235.4 |
+
+These latency examples use one fixed text context and one fixed image with distinct questions, three warmup requests and 20 measured requests per setting. They are not a representative multiquestion accuracy benchmark. All 22 selected answers matched between modes. Default image budget: 262144 pixels; the 1000-question comparison uses its original 1048576-pixel budget after a 768-pixel thumbnail. Timings include preprocessing and inference, not image-file reading or model loading. Ten questions take roughly 2.2–2.4 times the latency of one in these merged examples; they share a prefix pass and use one batched suffix pass.
+
+## Limitations and intended use
 
 This is a decision-support prototype for reviewing text and images with explicit answer choices. A human should inspect the source material before acting on consequential findings. It has not been validated to establish criminal intent, legal suspicion or evidentiary conclusions.
 
@@ -146,8 +166,10 @@ English and German were directly trained and evaluated. Other languages inherite
 
 The weights are unchanged from the evaluated short-repair checkpoint. The adapter configuration replaces the original local base path with the public base ID and pinned revision. `release_manifest.json` records every distributed file's size and hash; `validation.json` records local loading and inference checks.
 
-## Full public benchmark splits (zero-shot product readout)
-
+## Full public benchmark splits (zero-shot product readout)
+
+These results and their domain breakdowns were measured with the **unmerged adapter** (`merge_adapter=False`). The full MMStar/MMLU suites have not been rerun with the merged default; the paired merge validation above covers the separate 1000-question comparison.
+
 Complete splits, original option order, 0-shot, product chat prompt, direct A-J logits in FP32, BF16 backbone, no generation. MMStar RGB thumbnail <=768x768 then image budget 262144; full prompt <=1536 tokens, no truncation. Failures counted incorrect. Not standard MMLU 5-shot or a leaderboard submission.
 
 | Dataset | N per model | Base | Adapter | Delta pp | Errors base / adapter |

inference.py

@@ -33,7 +33,7 @@ def clone_cache(cache, count):
     return result
 
 class MultiDecider:
-    def __init__(self, adapter=True, dtype=torch.bfloat16, adapter_path=None, model_path=None, local_files_only=False):
+    def __init__(self, adapter=True, dtype=torch.bfloat16, adapter_path=None, model_path=None, local_files_only=False, merge_adapter=True):
         torch.set_num_threads(4)
         if not torch.cuda.is_available():
             raise RuntimeError('This inference implementation requires a CUDA GPU')
@@ -44,7 +44,13 @@ class MultiDecider:
         base = Qwen3_5ForConditionalGeneration.from_pretrained(path, local_files_only=local_files_only, revision=revision,
             dtype=dtype, attn_implementation='sdpa').cuda().eval()
         self.model = PeftModel.from_pretrained(base, adapter_path or Path(__file__).resolve().parent).eval() if adapter else base
-        self.base = base
+        self.base = base
+        # Merge only the in-memory model. Adapter/base files are never modified.
+        # BF16 rounding can change probabilities; disable for historical scores.
+        self.adapter_merged = bool(adapter and merge_adapter)
+        if self.adapter_merged:
+            self.base = self.model.merge_and_unload(safe_merge=True).eval()
+            self.model = self.base
         ids = [self.processor.tokenizer.encode(c, add_special_tokens=False) for c in 'ABCDEFGHIJ']
         assert all(len(x) == 1 for x in ids)
         self.letters = torch.tensor([x[0] for x in ids], device='cuda')

merged_adapter_results.json

@@ -0,0 +1,388 @@
+{
+  "dataset_sha256": "88cc14e7201b39b16545cd1e7578358a1747d528138d458c4351b2d27ed4e974",
+  "adapter_sha256": "dc9dbc8cd11c65e5d8dcff8df7ceeb39d8f73fe454e7fa51b5c155790c506bdd",
+  "protocol": "Same loaded BF16 base, active FP32 LoRA then safe merge_and_unload into BF16 weights. No weights saved. 1000 fixed questions, same option order, 768px thumbnail, max_image_pixels=1048576. Timed calls include preprocessing; image loading excluded. Separate 1/10-question fixtures use default 262144 image budget, 3 warmups and 20 measured calls. Multiquestion fixtures measure latency and parity, not benchmark accuracy.",
+  "modes": {
+    "active": {
+      "correct": 865,
+      "mean_ms": 85.93854070002271,
+      "peak_gib": 4.557392120361328,
+      "fixtures": {
+        "text-1": {
+          "mean_ms": 59.040699998877244,
+          "median_ms": 57.732200002647005,
+          "answers": [
+            {
+              "choice": 1,
+              "probabilities": [
+                0.00025641079992055893,
+                0.9991825222969055,
+                0.0005610074149444699
+              ]
+            }
+          ]
+        },
+        "text-10": {
+          "mean_ms": 144.78395000114688,
+          "median_ms": 141.6638500013505,
+          "answers": [
+            {
+              "choice": 1,
+              "probabilities": [
+                0.00025614723563194275,
+                0.9991936087608337,
+                0.0005502948770299554
+              ]
+            },
+            {
+              "choice": 1,
+              "probabilities": [
+                0.00039496851968578994,
+                0.9996050000190735
+              ]
+            },
+            {
+              "choice": 1,
+              "probabilities": [
+                0.001627324498258531,
+                0.9983727335929871
+              ]
+            },
+            {
+              "choice": 0,
+              "probabilities": [
+                0.9996798038482666,
+                0.0003202491207048297
+              ]
+            },
+            {
+              "choice": 0,
+              "probabilities": [
+                0.9997703433036804,
+                0.00022963015362620354
+              ]
+            },
+            {
+              "choice": 1,
+              "probabilities": [
+                0.00017448891594540328,
+                0.9998255372047424
+              ]
+            },
+            {
+              "choice": 0,
+              "probabilities": [
+                0.9996844530105591,
+                0.00031555391615256667
+              ]
+            },
+            {
+              "choice": 0,
+              "probabilities": [
+                0.9997870326042175,
+                0.00021302947425283492
+              ]
+            },
+            {
+              "choice": 0,
+              "probabilities": [
+                0.9998362064361572,
+                0.00016377581050619483
+              ]
+            },
+            {
+              "choice": 1,
+              "probabilities": [
+                0.0001576126232976094,
+                0.9998424053192139
+              ]
+            }
+          ]
+        },
+        "image-1": {
+          "mean_ms": 114.17969499889296,
+          "median_ms": 112.01630000141449,
+          "answers": [
+            {
+              "choice": 0,
+              "probabilities": [
+                0.9978511333465576,
+                0.0021488636266440153
+              ]
+            }
+          ]
+        },
+        "image-10": {
+          "mean_ms": 259.21306499876664,
+          "median_ms": 256.6619999925024,
+          "answers": [
+            {
+              "choice": 0,
+              "probabilities": [
+                0.9977619647979736,
+                0.002238077810034156
+              ]
+            },
+            {
+              "choice": 1,
+              "probabilities": [
+                0.0005537447286769748,
+                0.9994462132453918
+              ]
+            },
+            {
+              "choice": 1,
+              "probabilities": [
+                0.02532758191227913,
+                0.9746724367141724
+              ]
+            },
+            {
+              "choice": 1,
+              "probabilities": [
+                0.016182688996195793,
+                0.9838173389434814
+              ]
+            },
+            {
+              "choice": 1,
+              "probabilities": [
+                0.0012587311211973429,
+                0.9987413287162781
+              ]
+            },
+            {
+              "choice": 1,
+              "probabilities": [
+                0.018032928928732872,
+                0.9819670915603638
+              ]
+            },
+            {
+              "choice": 1,
+              "probabilities": [
+                0.0004362701438367367,
+                0.9995637536048889
+              ]
+            },
+            {
+              "choice": 0,
+              "probabilities": [
+                0.9969595670700073,
+                0.003040406620129943
+              ]
+            },
+            {
+              "choice": 1,
+              "probabilities": [
+                0.001857470953837037,
+                0.9981424808502197
+              ]
+            },
+            {
+              "choice": 1,
+              "probabilities": [
+                0.0006009377539157867,
+                0.9993990659713745
+              ]
+            }
+          ]
+        }
+      }
+    },
+    "merged": {
+      "correct": 867,
+      "mean_ms": 63.48714179956005,
+      "peak_gib": 4.495811462402344,
+      "fixtures": {
+        "text-1": {
+          "mean_ms": 52.594800000224495,
+          "median_ms": 49.0979000023799,
+          "answers": [
+            {
+              "choice": 1,
+              "probabilities": [
+                0.0002519080589991063,
+                0.9992098808288574,
+                0.0005381316295824945
+              ]
+            }
+          ]
+        },
+        "text-10": {
+          "mean_ms": 109.39867499910179,
+          "median_ms": 107.06689999642549,
+          "answers": [
+            {
+              "choice": 1,
+              "probabilities": [
+                0.00026706536300480366,
+                0.9991580247879028,
+                0.0005748923285864294
+              ]
+            },
+            {
+              "choice": 1,
+              "probabilities": [
+                0.00041133916238322854,
+                0.999588668346405
+              ]
+            },
+            {
+              "choice": 1,
+              "probabilities": [
+                0.0016603628173470497,
+                0.9983396530151367
+              ]
+            },
+            {
+              "choice": 0,
+              "probabilities": [
+                0.9996787309646606,
+                0.0003212429292034358
+              ]
+            },
+            {
+              "choice": 0,
+              "probabilities": [
+                0.9997716546058655,
+                0.00022835074923932552
+              ]
+            },
+            {
+              "choice": 1,
+              "probabilities": [
+                0.00017402767844032496,
+                0.9998260140419006
+              ]
+            },
+            {
+              "choice": 0,
+              "probabilities": [
+                0.9996775388717651,
+                0.0003225244290661067
+              ]
+            },
+            {
+              "choice": 0,
+              "probabilities": [
+                0.9997808337211609,
+                0.00021916748664807528
+              ]
+            },
+            {
+              "choice": 0,
+              "probabilities": [
+                0.9998308420181274,
+                0.00016914297884795815
+              ]
+            },
+            {
+              "choice": 1,
+              "probabilities": [
+                0.00016031514678616077,
+                0.9998396635055542
+              ]
+            }
+          ]
+        },
+        "image-1": {
+          "mean_ms": 99.02526999794645,
+          "median_ms": 98.12989999045385,
+          "answers": [
+            {
+              "choice": 0,
+              "probabilities": [
+                0.9978779554367065,
+                0.0021220268681645393
+              ]
+            }
+          ]
+        },
+        "image-10": {
+          "mean_ms": 236.76687000115635,
+          "median_ms": 235.40734999551205,
+          "answers": [
+            {
+              "choice": 0,
+              "probabilities": [
+                0.9979866743087769,
+                0.002013334073126316
+              ]
+            },
+            {
+              "choice": 1,
+              "probabilities": [
+                0.0005104709998704493,
+                0.9994895458221436
+              ]
+            },
+            {
+              "choice": 1,
+              "probabilities": [
+                0.029700670391321182,
+                0.9702993035316467
+              ]
+            },
+            {
+              "choice": 1,
+              "probabilities": [
+                0.01697063446044922,
+                0.983029305934906
+              ]
+            },
+            {
+              "choice": 1,
+              "probabilities": [
+                0.001263170037418604,
+                0.9987367987632751
+              ]
+            },
+            {
+              "choice": 1,
+              "probabilities": [
+                0.018061287701129913,
+                0.9819387197494507
+              ]
+            },
+            {
+              "choice": 1,
+              "probabilities": [
+                0.0004473220615182072,
+                0.9995526671409607
+              ]
+            },
+            {
+              "choice": 0,
+              "probabilities": [
+                0.9970535039901733,
+                0.002946516964584589
+              ]
+            },
+            {
+              "choice": 1,
+              "probabilities": [
+                0.0018149319803342223,
+                0.9981850981712341
+              ]
+            },
+            {
+              "choice": 1,
+              "probabilities": [
+                0.0005827727727591991,
+                0.9994171857833862
+              ]
+            }
+          ]
+        }
+      }
+    }
+  },
+  "merge_seconds": 0.22089949999644887,
+  "parity": {
+    "changed_answers": 2,
+    "max_probability_delta": 0.31059931218624115,
+    "active_only_correct": 0,
+    "merged_only_correct": 2
+  }
+}

release_manifest.json

@@ -1,7 +1,11 @@
 {
   "README.md": {
-    "bytes": 18391,
-    "sha256": "6dffb3724e8f3d669cf691c1ebc7bf9aadb401faf69d683f1a09ba3be3159f44"
+    "bytes": 21236,
+    "sha256": "06089f6d1fb01a1fbb0fdaaa2cd226165435a99ef154efbe81ffac67d2ca1240"
+  },
+  "LICENSE": {
+    "bytes": 11582,
+    "sha256": "a8928f06d5568b968f2710597053449e1dc350dc9e0406e23ca83cf33a303a45"
   },
   "LICENSING.md": {
     "bytes": 2469,
@@ -16,8 +20,8 @@
     "sha256": "83269f17bab3b0aa418787f9e1ecc458f517693210378dd89fa3aa5cf87062ff"
   },
   "inference.py": {
-    "bytes": 7945,
-    "sha256": "4548693fcf4925fc1cef2ff03070730c817f1b178eebf0c333ab6672259b2952"
+    "bytes": 8330,
+    "sha256": "3a700072864f2ac922b8008f70bf4f1c55ddff1fb212afecbf1648ab053212a5"
   },
   "adapter_config.json": {
     "bytes": 1395,
@@ -40,8 +44,8 @@
     "sha256": "b9f2f1bd4788930ab4fe73cfc0e621072615c9256ee0b783ddd72a6ce928a8bf"
   },
   "validation.json": {
-    "bytes": 1775,
-    "sha256": "c71a837998e332a6990ccb2de00643f623edeec35b60489ea5394e339d1f0bde"
+    "bytes": 1805,
+    "sha256": "d22ab4d10e03c5cef195a04867fc989bf4017a4bb1e0af92b80fd7ff291550af"
   },
   "dataset_licenses.json": {
     "bytes": 17366,
@@ -55,6 +59,22 @@
     "bytes": 4378,
     "sha256": "915a4d1dbc675dc7914d9de525f8e94df0a9e604ff16ef57fc8a90f744fc340d"
   },
+  "standard_benchmark_results.json": {
+    "bytes": 15995,
+    "sha256": "74d154a4ba44db04e39b1f727ec623d6793028127a5e4a46c152ced54b4082c9"
+  },
+  "domain_results.json": {
+    "bytes": 15535,
+    "sha256": "e44559d9185d7c90d0580b196bb1ea899b601c78855c83c5bc8c3da4565010ba"
+  },
+  "merged_adapter_results.json": {
+    "bytes": 10729,
+    "sha256": "52ff993a3b59f4dbba163b0b83fae9e179a2fd020a7b7406de9061a4f01cd8c0"
+  },
+  "alberich-mini.png": {
+    "bytes": 1326272,
+    "sha256": "f8227e1913ec81807ad7899b6adbc96923bd1ce11477709dd89be7f2374170bc"
+  },
   "NOTICE-XNLI.txt": {
     "bytes": 19332,
     "sha256": "1d8ca66080a2701508a70be4ddad4be28c509bcc3397a682fb8edf767fdd7a00"
@@ -114,21 +134,5 @@
   "NOTICE-CC-BY-SA-4.0.txt": {
     "bytes": 20138,
     "sha256": "28a9529c7d0bb4dc51f4bf5c116a3d16ef247a052f7591466768ddf563fd1cf5"
-  },
-  "standard_benchmark_results.json": {
-    "bytes": 15995,
-    "sha256": "74d154a4ba44db04e39b1f727ec623d6793028127a5e4a46c152ced54b4082c9"
-  },
-  "domain_results.json": {
-    "bytes": 15535,
-    "sha256": "e44559d9185d7c90d0580b196bb1ea899b601c78855c83c5bc8c3da4565010ba"
-  },
-  "LICENSE": {
-    "bytes": 11582,
-    "sha256": "a8928f06d5568b968f2710597053449e1dc350dc9e0406e23ca83cf33a303a45"
-  },
-  "alberich-mini.png": {
-    "bytes": 1326272,
-    "sha256": "f8227e1913ec81807ad7899b6adbc96923bd1ce11477709dd89be7f2374170bc"
   }
 }

validation.json

@@ -4,6 +4,7 @@
   "adapter_dtypes": [
     "torch.float32"
   ],
+  "adapter_merged": true,
   "gpu": "NVIDIA GeForce RTX 4070 SUPER",
   "checks": [
     {
@@ -18,14 +19,14 @@
       "questions": 3,
       "original_code_max_probability_delta": 0.0,
       "shared_independent_choices_match": true,
-      "shared_independent_max_probability_delta": 0.002172231674194336
+      "shared_independent_max_probability_delta": 0.0006584310904145241
     },
     {
       "modality": "text",
       "questions": 10,
       "original_code_max_probability_delta": 0.0,
       "shared_independent_choices_match": true,
-      "shared_independent_max_probability_delta": 0.001797497272491455
+      "shared_independent_max_probability_delta": 0.0019329637289047241
     },
     {
       "modality": "image",
@@ -39,14 +40,14 @@
       "questions": 3,
       "original_code_max_probability_delta": 0.0,
       "shared_independent_choices_match": true,
-      "shared_independent_max_probability_delta": 0.002624034881591797
+      "shared_independent_max_probability_delta": 0.0008810162544250488
     },
     {
       "modality": "image",
       "questions": 10,
       "original_code_max_probability_delta": 0.0,
       "shared_independent_choices_match": true,
-      "shared_independent_max_probability_delta": 0.0009383559226989746
+      "shared_independent_max_probability_delta": 9.277090430259705e-05
     }
   ],
   "note": "Smoke tests verify packaging and implementation parity, not general accuracy. Original and portable implementations use the same loaded model; weight identity is independently checked by SHA256."