GeoLIP
Collection
A series of useful models expert-trained using the GEOLIP distillation and constellation process. • 7 items • Updated
Return to the chalkboard.
Yes, it happened again. It's time to return to the chalkboard and make some new components.
Building from linear, baseline 20% accuracy on cifar10, Blues Brothers reference, it's midnight, and we're wearing sunglasses.
Hit it.
We're going clean in, and making these omega tokens sing. Every single will have a purpose, and a tracking point as to WHY they have a purpose.
Nothing ambiguous. Nothing left to up another mechanism.
MLP testing
I'm extracting and preserving the 4096 omega tokens [b, 4, 64, 64] from the fairly tanky Freckles-4096
With these extracted cifar10 features it will allow for much more rapid prototyping.
Freckles-4096 epoch 1 battery is ran
Check the results in the directory https://huggingface.co/AbstractPhil/geolip-SVAE/tree/main/v41_freckles_256
This is a patchwork tokenization size of 4096 related directly to the cross-attention, so be aware I'm not bloating numbers here.
This is literally how many patches exist, and this many patches exist specifically in the spectrum of 256x256. As per the scaling rule applied by the internal architectural resonance.
The sizes are internally very very different, but the actual information and parameters are identical.
enc_in [262144, 384]
enc_block_0 [262144, 384]
enc_block_1 [262144, 384]
enc_block_2 [262144, 384]
enc_block_3 [262144, 384]
enc_out_raw [262144, 192]
cross_attn_0_qkv [64, 4096, 12]
cross_attn_0_in [64, 4096, 4]
cross_attn_0_out [64, 4096, 4]
cross_attn_1_qkv [64, 4096, 12]
cross_attn_1_in [64, 4096, 4]
cross_attn_1_out [64, 4096, 4]
dec_in [262144, 384]
dec_block_0 [262144, 384]
dec_block_1 [262144, 384]
dec_block_2 [262144, 384]
dec_block_3 [262144, 384]
dec_out [262144, 48]
boundary_in [64, 3, 256, 256]
boundary_out [64, 3, 256, 256]
svd_U [64, 4096, 48, 4]
svd_S_orig [64, 4096, 4]
svd_S [64, 4096, 4]
svd_Vt [64, 4096, 4, 4]
svd_M [64, 4096, 48, 4]
recon [64, 3, 256, 256]
input [64, 3, 256, 256]
Keep in mind when I test Freckles with images, Freckles has never trained on a single image.
Freckles has ONLY SEEN NOISE. 16 types of noise. That's it, historically Freckles is image ignorant.
Meaning when I train heads and classification models with Freckles features, I'm training models with what Freckles turns pure images into.
Freckles has zero knowledge of any real data, ever. Just noise.
Freckles-4096 epoch 1 is complete
The MSE shows roughly the same as the freckles-256
Battery testing begins soon.
Little freckles couldn't beat pixel-in transformers
With just her 256 patches condensed into 64 dim features.
However... We're just getting started. The little 64 dim features were good, but not the full capacity of freckles, not by a long shot.
Next up is the 384 dim hidden features, no more pulling punches.
======================================================================
HEAD-TO-HEAD COMPARISON
======================================================================
Omega Processor (Freckles features): 72.7%
Baseline (raw patches): 77.2%
Delta: -4.5%
Random chance: 10.0%
======================================================================
Well fought little bottlenecked Freckles, it's time to unlock the limit gates now.
First Omega Processor Tests
Omega Processor Baseline (raw)
Final accuracy: 92.8% 82.0%
Params: 837,008 867,824
Epoch time: 34s 19s
PER-CLASS FINAL:
Omega Baseline
gaussian 100% 86%
uniform 99% 51% ← baseline can't separate
uniform_sc 100% 49% ← from its sibling
poisson 94% 97%
pink 33% 84% ← baseline does BETTER here
brown 64% 16% ← but worse here
salt_pepper 100% 100%
sparse 100% 100%
block 100% 100%
gradient 100% 100%
checker 100% 78%
mixed 95% 46% ← massive gap
structural 100% 100%
cauchy 100% 100%
exponential 100% 100%
laplace 100% 100%
UNSOLVABLE PAIRS:
Baseline: uniform/uniform_sc (51/49), pink/brown (84/16), mixed (46%), checker (78%)
Omega: pink/brown (33/64) — one remaining confusion pair
The omega processor features in this state are more robust than dataset introduced features. The convergence is faster, the data learns faster, and the output is clearly pretrained to a direct extreme.
Now... lets try Cifar10 shall we? I'll just feed it into our little Johanna-256, no image training ever seen.
4/9/2026 Omega Processer Prototype
Freckles encoder (frozen)
↓
SVD: U, S, Vt (exact, frozen)
↓
Feature Extractor (tiny, learned):
scalar_features(S) → 16 dims
relational_features(S, grid) → 16 dims
basis_features(U, Vt) → 32 dims
──────────────────────────────────────
concat → 64 dims per patch
↓
LayerNorm + Linear(64, d_model) → project to transformer dim
↓
Standard transformer encoder/decoder
↓
Task head (classification, generation, text, etc.)
Due to the model being so deviant from transformers, Claude and I have come up with a potential scaffold.
It aligns with the geometry and has a very small learning curve, which should coalesce into the necessary behavior.
This SHOULD allow any model to be able to utilize the information without needing a massive architectural change.
4/9/2026 Freckles - Resolution Scaling!?!
Freckles handles noise at massive ratios if I piecemeal the information together.
I'm going to try massive sizes, and test every single noise spectrum.
Currently Freckles handles MSE 0.000005 noise error from 16 types of noise, the tests show rigidity in the patches so they can't be slid around.
The manifold shows you can just... aim it at any noise target and it will solve via the patches, and the patches can be easily calculated to which is most likely the noise type with very low compute cost.
Noise type, noise location within structure, structure of that encapsulated system's image space... Alright yeah this is, well beyond expectation.
These are resolution independent models, resolution means nothing to them. I believe as well, that Freckles being 2m params, is substantially more powerful than each of the others at their 17m params.
The Omega Processor
The plan now is to wrap the center of the VAE structure with a memory bank driven constellation observer with hundreds of thousands of anchors.
We will see, exactly what this model is doing, why this model is doing that behavior, what we need to do to replicate this model, what steps we need to take to build the necessary optimizations to build the scaffolding for those tools, and everything between here and there.
We have TOUCHED Omega now, and I will NOT let this vapor dissolve. I will not let these numbers simply get mashed into a transformer without full observation and understanding of WHAT we are seeing. I MUST know.
IF THIS IS NOT TRULY THE SELF SOLVING FRAME, we will know VERY soon. VERY VERY soon. I will redact my claims of Omega and continue towards the legitimate self-solving quantum state that we are all looking for.
I TRULY BELIEVE this has ENOUGH potential for me to spend a good amount of time investigating.
If I can even find a SINGLE PIECE of truly high-dimensional scaffolding that doesn't directly conform to the dimensions; that somehow survived the battery of CM volume, that somehow formed knots upon knots, upon knots of impossible conjecture - that can be turned into a viable utility. Then I will have finished my task. Everything will have been proven and I can simply build.
All we need is JUST ONE. JUST ONE, and it will be enough to create the universe.
4/8/2025 Prototype V13, 14, 15, etc
Johanna - Noise variant 48:1
Patchworks Available:
- 64x64 0.06 mse
- 128x128 0.002 mse
- 256x256 0.0002 mse
17m params all
48:1 compression
[16, 8, 8] omega tokens
fair mse decoding on 16 noise types
This model can handle noise, images, audio, whatever - semi okay.
This is the one you want to finetune if you have a task you want to try.
Fresnel - Image variant 48:1
Patchworks Available:
- 64x64 0.00001~ mse
- 128x128 0.000002 mse
- 256x256 0.000001 mse
17m params
48:1 compression
[16, 8, 8] omega tokens
0.000002 mse on clean image latents
Don't feed fresnel noise, the model does not understand noise
Johanna is the noise model.
Grandmaster - Denoise Variant 48:1
Finetuned Johanna-128
Patchworks Available:
- 128x128 mse 0.0042 mse
17m params
48:1 compression
[16, 8, 8] omega tokens
This model accepts a noisy image input, and returns denoised output.
Trained using Fresnel's omega tokens as training targets, has a fair SNR denoising capacity already at 0.0042 mse.
This is part of a prototype meant to skip diffusion steps entirely.
Freckles - Noise Variant 12:1
Patchworks Available:
- 4x4 0.0002 mse
- gaus=0.001 unif=0.000 unif=0.001 pois=0.000
- pink=0.000 brow=0.000 salt=0.004 spar=0.000
- bloc=0.000 grad=0.000 chec=0.000 mixe=0.000
- stru=0.000 cauc=0.002 expo=0.001 lapl=0.001
2.7m params
12:1 compression
Uhhhh.. yeah I uhh... didn't expect this one to recon so well. I didn't set the measures to register lower than that.
I'll report later.
Indev:
Alexandria - Text variant
17m params
48:1 compression
DOES NOT WORK YET!
Prototype v12 - Patch 16 - 128x128 images - 1.2m imagenet images
Well the 64x64 image set worked just fine, so it's time to upgrade and test the limits of the architecture.
Can it simply... scale? ooooor do we need more solvers along the way to compensate?
benjamin-paine/imagenet-1k-128x128
Can we actually solve it
YES WE DID!
It seems geometric manifolds learn... differently than standard manifolds, don't they.
Prototype V11 - Patch16 - MSE 0.0005 - 64x64 tiny imagenet
I'd say it works.
The images show it works. It works.
Using geolip-core SVD (fp64 Gram+eigh (FL=available, N<=12))
PatchSVAE - 16 patches of 16×16
Dataset: tiny_imagenet (64×64, 200 classes)
Per-patch: (256, 16) = 4096 elements, rows on S^15
Encoder/Decoder: hidden=768, depth=4 (residual blocks)
Cross-attention: 2 layers on S vectors (2,272 params)
Soft hand: boost=1.5x near CV=0.125, penalty=0.3 far
Total params: 16,942,419
===============================================================================================
ep | loss recon t/ep | t_rec | S0 SD ratio erank | row_cv prox rw | S_delta
-----------------------------------------------------------------------------------------------
1 | 0.2595 0.1806 12.2 | 0.1024 | 5.036 3.254 1.55 15.87 | 0.2007 0.905 1.45 | 0.09694 a:0.0242/0.0247
2 | 0.1216 0.0845 12.3 | 0.0675 | 5.071 3.298 1.54 15.88 | 0.2018 0.885 1.44 | 0.17411 a:0.0251/0.0257
3 | 0.0847 0.0587 12.3 | 0.0470 | 5.093 3.312 1.54 15.88 | 0.2046 0.869 1.43 | 0.19894 a:0.0258/0.0265
4 | 0.0623 0.0432 12.3 | 0.0430 | 5.115 3.323 1.54 15.88 | 0.2006 0.864 1.43 | 0.20848 a:0.0264/0.0272
6 | 0.0359 0.0248 12.3 | 0.0198 | 5.129 3.332 1.54 15.88 | 0.2006 0.907 1.45 | 0.21832 a:0.0273/0.0281
8 | 0.0225 0.0155 12.2 | 0.0196 | 5.149 3.341 1.54 15.87 | 0.2017 0.876 1.44 | 0.22351 a:0.0279/0.0287
10 | 0.0170 0.0116 12.3 | 0.0100 | 5.151 3.352 1.54 15.88 | 0.2035 0.924 1.46 | 0.22671 a:0.0283/0.0290
12 | 0.0141 0.0096 12.3 | 0.0114 | 5.159 3.354 1.54 15.88 | 0.2009 0.909 1.45 | 0.22924 a:0.0285/0.0293
14 | 0.0121 0.0082 12.3 | 0.0073 | 5.156 3.362 1.53 15.88 | 0.2018 0.855 1.43 | 0.23137 a:0.0288/0.0296
16 | 0.0105 0.0072 12.3 | 0.0108 | 5.161 3.363 1.53 15.88 | 0.2003 0.860 1.43 | 0.23316 a:0.0290/0.0298
18 | 0.0094 0.0064 12.3 | 0.0055 | 5.158 3.365 1.53 15.88 | 0.2017 0.879 1.44 | 0.23467 a:0.0292/0.0300
20 | 0.0086 0.0058 12.3 | 0.0050 | 5.157 3.367 1.53 15.88 | 0.2023 0.805 1.40 | 0.23601 a:0.0293/0.0301
22 | 0.0079 0.0054 12.4 | 0.0045 | 5.157 3.369 1.53 15.88 | 0.1996 0.872 1.44 | 0.23726 a:0.0295/0.0303
24 | 0.0074 0.0050 12.2 | 0.0064 | 5.146 3.380 1.52 15.88 | 0.2044 0.879 1.44 | 0.23848 a:0.0296/0.0305
26 | 0.0068 0.0046 12.4 | 0.0039 | 5.155 3.372 1.53 15.88 | 0.2036 0.884 1.44 | 0.23955 a:0.0297/0.0306
28 | 0.0063 0.0042 12.3 | 0.0036 | 5.155 3.378 1.53 15.88 | 0.2077 0.841 1.42 | 0.24057 a:0.0299/0.0307
30 | 0.0058 0.0038 12.3 | 0.0038 | 5.155 3.380 1.53 15.88 | 0.2027 0.911 1.46 | 0.24149 a:0.0300/0.0309
32 | 0.0055 0.0036 12.2 | 0.0032 | 5.150 3.383 1.52 15.88 | 0.2045 0.807 1.40 | 0.24239 a:0.0301/0.0310
34 | 0.0054 0.0036 12.3 | 0.0037 | 5.145 3.388 1.52 15.88 | 0.1996 0.875 1.44 | 0.24329 a:0.0302/0.0311
36 | 0.0049 0.0032 12.3 | 0.0031 | 5.154 3.385 1.52 15.88 | 0.2054 0.828 1.41 | 0.24409 a:0.0303/0.0312
38 | 0.0046 0.0030 12.3 | 0.0027 | 5.152 3.390 1.52 15.88 | 0.2038 0.847 1.42 | 0.24490 a:0.0304/0.0313
40 | 0.0044 0.0029 12.3 | 0.0032 | 5.155 3.392 1.52 15.89 | 0.2046 0.855 1.43 | 0.24566 a:0.0305/0.0314
42 | 0.0043 0.0028 12.3 | 0.0024 | 5.152 3.395 1.52 15.89 | 0.2064 0.905 1.45 | 0.24637 a:0.0305/0.0315
44 | 0.0042 0.0027 12.3 | 0.0023 | 5.150 3.395 1.52 15.89 | 0.2084 0.844 1.42 | 0.24705 a:0.0306/0.0316
46 | 0.0039 0.0025 12.3 | 0.0022 | 5.149 3.400 1.51 15.89 | 0.2057 0.868 1.43 | 0.24776 a:0.0307/0.0317
48 | 0.0037 0.0024 12.3 | 0.0024 | 5.152 3.403 1.51 15.89 | 0.2138 0.831 1.42 | 0.24843 a:0.0308/0.0318
50 | 0.0038 0.0024 12.3 | 0.0025 | 5.149 3.406 1.51 15.89 | 0.2078 0.810 1.40 | 0.24906 a:0.0309/0.0319
52 | 0.0034 0.0021 12.3 | 0.0019 | 5.154 3.405 1.51 15.89 | 0.2082 0.872 1.44 | 0.24965 a:0.0309/0.0320
54 | 0.0033 0.0020 12.2 | 0.0019 | 5.156 3.406 1.51 15.89 | 0.2085 0.894 1.45 | 0.25022 a:0.0310/0.0320
56 | 0.0033 0.0020 12.4 | 0.0019 | 5.150 3.412 1.51 15.89 | 0.2058 0.866 1.43 | 0.25079 a:0.0311/0.0321
58 | 0.0031 0.0019 12.3 | 0.0033 | 5.147 3.416 1.51 15.89 | 0.2071 0.774 1.39 | 0.25135 a:0.0311/0.0322
60 | 0.0030 0.0018 12.4 | 0.0017 | 5.153 3.415 1.51 15.89 | 0.2134 0.840 1.42 | 0.25187 a:0.0312/0.0323
62 | 0.0030 0.0018 12.3 | 0.0016 | 5.155 3.416 1.51 15.89 | 0.2080 0.764 1.38 | 0.25235 a:0.0313/0.0323
64 | 0.0028 0.0017 12.2 | 0.0014 | 5.156 3.416 1.51 15.89 | 0.2100 0.666 1.33 | 0.25285 a:0.0313/0.0324
66 | 0.0028 0.0017 12.3 | 0.0017 | 5.151 3.419 1.51 15.89 | 0.2101 0.865 1.43 | 0.25333 a:0.0314/0.0324
68 | 0.0026 0.0015 12.3 | 0.0014 | 5.158 3.419 1.51 15.89 | 0.2078 0.838 1.42 | 0.25381 a:0.0314/0.0325
70 | 0.0025 0.0015 12.4 | 0.0021 | 5.160 3.422 1.51 15.89 | 0.2112 0.806 1.40 | 0.25428 a:0.0315/0.0326
72 | 0.0026 0.0015 12.2 | 0.0013 | 5.158 3.422 1.51 15.89 | 0.2126 0.835 1.42 | 0.25471 a:0.0316/0.0326
74 | 0.0024 0.0014 12.2 | 0.0015 | 5.154 3.427 1.50 15.89 | 0.2143 0.838 1.42 | 0.25514 a:0.0316/0.0327
76 | 0.0024 0.0014 12.2 | 0.0012 | 5.161 3.424 1.51 15.89 | 0.2151 0.847 1.42 | 0.25553 a:0.0317/0.0327
78 | 0.0023 0.0013 12.3 | 0.0014 | 5.157 3.428 1.50 15.89 | 0.2121 0.686 1.34 | 0.25592 a:0.0317/0.0328
80 | 0.0024 0.0013 12.3 | 0.0012 | 5.160 3.428 1.51 15.89 | 0.2068 0.824 1.41 | 0.25630 a:0.0317/0.0328
82 | 0.0027 0.0016 12.2 | 0.0015 | 5.146 3.394 1.52 15.89 | 0.2065 0.899 1.45 | 0.25687 a:0.0318/0.0329
84 | 0.0022 0.0013 12.2 | 0.0013 | 5.156 3.405 1.51 15.89 | 0.2092 0.875 1.44 | 0.25709 a:0.0319/0.0329
86 | 0.0022 0.0012 12.3 | 0.0022 | 5.154 3.413 1.51 15.89 | 0.2091 0.835 1.42 | 0.25726 a:0.0319/0.0329
88 | 0.0021 0.0012 12.3 | 0.0014 | 5.154 3.417 1.51 15.89 | 0.2074 0.840 1.42 | 0.25740 a:0.0319/0.0329
90 | 0.0022 0.0013 12.3 | 0.0030 | 5.147 3.416 1.51 15.89 | 0.2191 0.848 1.42 | 0.25753 a:0.0319/0.0329
92 | 0.0021 0.0011 12.3 | 0.0012 | 5.157 3.418 1.51 15.89 | 0.2123 0.775 1.39 | 0.25766 a:0.0319/0.0329
94 | 0.0021 0.0011 12.3 | 0.0010 | 5.156 3.419 1.51 15.89 | 0.2117 0.710 1.35 | 0.25779 a:0.0319/0.0329
96 | 0.0020 0.0011 12.3 | 0.0013 | 5.154 3.420 1.51 15.89 | 0.2166 0.940 1.47 | 0.25793 a:0.0319/0.0330
98 | 0.0019 0.0011 12.2 | 0.0010 | 5.156 3.421 1.51 15.89 | 0.2143 0.762 1.38 | 0.25807 a:0.0320/0.0330
100 | 0.0020 0.0010 12.3 | 0.0009 | 5.155 3.422 1.51 15.89 | 0.2173 0.642 1.32 | 0.25821 a:0.0320/0.0330
102 | 0.0020 0.0010 12.2 | 0.0009 | 5.156 3.423 1.51 15.89 | 0.2165 0.868 1.43 | 0.25835 a:0.0320/0.0330
104 | 0.0019 0.0010 12.4 | 0.0009 | 5.157 3.423 1.51 15.89 | 0.2125 0.788 1.39 | 0.25850 a:0.0320/0.0330
106 | 0.0019 0.0009 12.3 | 0.0009 | 5.156 3.424 1.51 15.89 | 0.2219 0.666 1.33 | 0.25866 a:0.0320/0.0331
108 | 0.0019 0.0009 12.3 | 0.0009 | 5.153 3.425 1.50 15.89 | 0.2202 0.671 1.34 | 0.25881 a:0.0321/0.0331
110 | 0.0020 0.0009 12.3 | 0.0011 | 5.153 3.427 1.50 15.89 | 0.2163 0.726 1.36 | 0.25896 a:0.0321/0.0331
112 | 0.0019 0.0009 12.4 | 0.0009 | 5.155 3.427 1.50 15.89 | 0.2205 0.837 1.42 | 0.25911 a:0.0321/0.0331
114 | 0.0019 0.0008 12.3 | 0.0008 | 5.155 3.427 1.50 15.89 | 0.2220 0.803 1.40 | 0.25926 a:0.0321/0.0332
116 | 0.0018 0.0008 12.3 | 0.0009 | 5.155 3.427 1.50 15.89 | 0.2211 0.852 1.43 | 0.25942 a:0.0321/0.0332
118 | 0.0019 0.0008 12.3 | 0.0008 | 5.153 3.429 1.50 15.89 | 0.2207 0.694 1.35 | 0.25957 a:0.0321/0.0332
120 | 0.0018 0.0008 12.2 | 0.0008 | 5.156 3.429 1.50 15.89 | 0.2271 0.664 1.33 | 0.25972 a:0.0322/0.0332
122 | 0.0018 0.0008 12.3 | 0.0008 | 5.154 3.429 1.50 15.89 | 0.2266 0.658 1.33 | 0.25986 a:0.0322/0.0333
124 | 0.0017 0.0007 12.3 | 0.0008 | 5.154 3.431 1.50 15.89 | 0.2201 0.771 1.39 | 0.26000 a:0.0322/0.0333
126 | 0.0017 0.0007 12.4 | 0.0008 | 5.157 3.430 1.50 15.89 | 0.2253 0.862 1.43 | 0.26014 a:0.0322/0.0333
128 | 0.0018 0.0007 12.3 | 0.0008 | 5.153 3.431 1.50 15.89 | 0.2222 0.638 1.32 | 0.26027 a:0.0322/0.0333
130 | 0.0018 0.0007 12.3 | 0.0007 | 5.155 3.431 1.50 15.89 | 0.2255 0.786 1.39 | 0.26040 a:0.0322/0.0333
132 | 0.0018 0.0007 12.3 | 0.0007 | 5.153 3.432 1.50 15.89 | 0.2325 0.778 1.39 | 0.26053 a:0.0323/0.0333
134 | 0.0017 0.0007 12.3 | 0.0007 | 5.154 3.433 1.50 15.89 | 0.2250 0.876 1.44 | 0.26065 a:0.0323/0.0334
136 | 0.0017 0.0006 12.3 | 0.0007 | 5.157 3.432 1.50 15.89 | 0.2269 0.866 1.43 | 0.26077 a:0.0323/0.0334
138 | 0.0017 0.0006 12.3 | 0.0007 | 5.156 3.433 1.50 15.89 | 0.2247 0.760 1.38 | 0.26088 a:0.0323/0.0334
140 | 0.0016 0.0006 12.3 | 0.0006 | 5.157 3.433 1.50 15.89 | 0.2242 0.725 1.36 | 0.26099 a:0.0323/0.0334
142 | 0.0016 0.0006 12.3 | 0.0006 | 5.157 3.433 1.50 15.89 | 0.2241 0.909 1.45 | 0.26109 a:0.0323/0.0334
144 | 0.0017 0.0006 12.3 | 0.0006 | 5.157 3.433 1.50 15.89 | 0.2287 0.815 1.41 | 0.26119 a:0.0323/0.0334
146 | 0.0016 0.0006 12.3 | 0.0007 | 5.158 3.434 1.50 15.90 | 0.2205 0.722 1.36 | 0.26128 a:0.0324/0.0334
148 | 0.0016 0.0006 12.3 | 0.0008 | 5.157 3.434 1.50 15.90 | 0.2286 0.691 1.35 | 0.26137 a:0.0324/0.0335
150 | 0.0016 0.0006 12.3 | 0.0006 | 5.158 3.434 1.50 15.90 | 0.2259 0.845 1.42 | 0.26146 a:0.0324/0.0335
152 | 0.0017 0.0006 12.3 | 0.0006 | 5.158 3.434 1.50 15.90 | 0.2295 0.757 1.38 | 0.26154 a:0.0324/0.0335
154 | 0.0016 0.0005 12.3 | 0.0006 | 5.159 3.435 1.50 15.90 | 0.2304 0.751 1.38 | 0.26162 a:0.0324/0.0335
156 | 0.0018 0.0005 12.3 | 0.0006 | 5.159 3.435 1.50 15.90 | 0.2264 0.796 1.40 | 0.26169 a:0.0324/0.0335
158 | 0.0017 0.0005 12.3 | 0.0006 | 5.160 3.434 1.50 15.90 | 0.2282 0.788 1.39 | 0.26176 a:0.0324/0.0335
160 | 0.0017 0.0005 12.3 | 0.0005 | 5.161 3.434 1.50 15.90 | 0.2291 0.766 1.38 | 0.26183 a:0.0324/0.0335
162 | 0.0016 0.0005 12.3 | 0.0005 | 5.161 3.434 1.50 15.90 | 0.2282 0.716 1.36 | 0.26189 a:0.0324/0.0335
164 | 0.0016 0.0005 12.3 | 0.0005 | 5.161 3.435 1.50 15.90 | 0.2344 0.792 1.40 | 0.26196 a:0.0324/0.0335
166 | 0.0016 0.0005 12.3 | 0.0006 | 5.162 3.434 1.50 15.90 | 0.2305 0.707 1.35 | 0.26202 a:0.0324/0.0335
168 | 0.0016 0.0005 12.3 | 0.0005 | 5.162 3.434 1.50 15.90 | 0.2353 0.816 1.41 | 0.26207 a:0.0324/0.0335
170 | 0.0016 0.0005 12.3 | 0.0005 | 5.163 3.434 1.50 15.90 | 0.2296 0.756 1.38 | 0.26213 a:0.0325/0.0335
172 | 0.0018 0.0005 12.4 | 0.0005 | 5.163 3.434 1.50 15.90 | 0.2391 0.742 1.37 | 0.26218 a:0.0325/0.0336
174 | 0.0016 0.0005 12.4 | 0.0005 | 5.163 3.434 1.50 15.90 | 0.2307 0.863 1.43 | 0.26224 a:0.0325/0.0336
176 | 0.0016 0.0005 12.3 | 0.0005 | 5.163 3.434 1.50 15.90 | 0.2329 0.854 1.43 | 0.26228 a:0.0325/0.0336
178 | 0.0017 0.0005 12.3 | 0.0005 | 5.164 3.434 1.50 15.90 | 0.2287 0.803 1.40 | 0.26233 a:0.0325/0.0336
180 | 0.0017 0.0005 12.3 | 0.0005 | 5.164 3.434 1.50 15.90 | 0.2361 0.819 1.41 | 0.26237 a:0.0325/0.0336
182 | 0.0018 0.0005 12.3 | 0.0005 | 5.164 3.434 1.50 15.90 | 0.2360 0.729 1.36 | 0.26241 a:0.0325/0.0336
184 | 0.0016 0.0005 12.3 | 0.0005 | 5.164 3.434 1.50 15.90 | 0.2395 0.774 1.39 | 0.26245 a:0.0325/0.0336
186 | 0.0015 0.0005 12.3 | 0.0005 | 5.165 3.434 1.50 15.90 | 0.2325 0.777 1.39 | 0.26248 a:0.0325/0.0336
188 | 0.0018 0.0005 12.3 | 0.0005 | 5.165 3.434 1.50 15.90 | 0.2430 0.669 1.33 | 0.26250 a:0.0325/0.0336
190 | 0.0016 0.0005 12.3 | 0.0005 | 5.165 3.434 1.50 15.90 | 0.2293 0.781 1.39 | 0.26252 a:0.0325/0.0336
192 | 0.0017 0.0005 12.3 | 0.0005 | 5.165 3.434 1.50 15.90 | 0.2246 0.763 1.38 | 0.26254 a:0.0325/0.0336
194 | 0.0019 0.0005 12.3 | 0.0005 | 5.165 3.434 1.50 15.90 | 0.2324 0.764 1.38 | 0.26255 a:0.0325/0.0336
196 | 0.0016 0.0005 12.2 | 0.0005 | 5.165 3.434 1.50 15.90 | 0.2301 0.868 1.43 | 0.26256 a:0.0325/0.0336
198 | 0.0016 0.0005 12.3 | 0.0005 | 5.165 3.434 1.50 15.90 | 0.2332 0.696 1.35 | 0.26256 a:0.0325/0.0336
200 | 0.0017 0.0005 12.3 | 0.0005 | 5.165 3.434 1.50 15.90 | 0.2397 0.808 1.40 | 0.26256 a:0.0325/0.0336
==========================================================================================
FINAL ANALYSIS
==========================================================================================
PatchSVAE: 16 patches × (256, 16)
Target CV: 0.125
Recon MSE: 0.000489 +/- 0.000743
Row CV: 0.2397
Cross-attention S delta: 0.26256
Learned alpha per mode (coordination strength):
Layer 0: mean=0.0327 max=0.0336 min=0.0323
α[ 0]: 0.0324 ######################################
α[ 1]: 0.0323 ######################################
α[ 2]: 0.0327 ######################################
α[ 3]: 0.0326 ######################################
α[ 4]: 0.0325 ######################################
α[ 5]: 0.0326 ######################################
α[ 6]: 0.0332 #######################################
α[ 7]: 0.0336 #######################################
α[ 8]: 0.0326 ######################################
α[ 9]: 0.0324 ######################################
α[10]: 0.0326 ######################################
α[11]: 0.0325 ######################################
α[12]: 0.0328 #######################################
α[13]: 0.0324 ######################################
α[14]: 0.0331 #######################################
α[15]: 0.0326 ######################################
Layer 1: mean=0.0323 max=0.0327 min=0.0315
α[ 0]: 0.0324 #######################################
α[ 1]: 0.0326 #######################################
α[ 2]: 0.0323 #######################################
α[ 3]: 0.0326 #######################################
α[ 4]: 0.0327 #######################################
α[ 5]: 0.0326 #######################################
α[ 6]: 0.0320 #######################################
α[ 7]: 0.0315 ######################################
α[ 8]: 0.0324 #######################################
α[ 9]: 0.0327 #######################################
α[10]: 0.0325 #######################################
α[11]: 0.0324 #######################################
α[12]: 0.0321 #######################################
α[13]: 0.0324 #######################################
α[14]: 0.0317 ######################################
α[15]: 0.0322 #######################################
Coordinated singular value profile:
S[ 0]: 5.1650 cum= 9.2% #############################
S[ 1]: 4.9525 cum= 17.6% ############################
S[ 2]: 4.8142 cum= 25.6% ###########################
S[ 3]: 4.6335 cum= 32.9% ##########################
S[ 4]: 4.5199 cum= 40.0% ##########################
S[ 5]: 4.4203 cum= 46.7% #########################
S[ 6]: 4.3376 cum= 53.2% #########################
S[ 7]: 4.2448 cum= 59.3% ########################
S[ 8]: 4.1641 cum= 65.3% ########################
S[ 9]: 4.0915 cum= 71.1% #######################
S[10]: 4.0086 cum= 76.6% #######################
S[11]: 3.9144 cum= 81.9% ######################
S[12]: 3.7995 cum= 86.8% ######################
S[13]: 3.6926 cum= 91.5% #####################
S[14]: 3.5949 cum= 95.9% ####################
S[15]: 3.4336 cum=100.0% ###################
Saving reconstruction grid...
Saved to /content/svae_patch_recon.png
4/7/2025 Prototype V10.3 - Patch16 - The VIT size.
Might need some tweaks but I don't think so. We're approaching actual vit prototype accuracy now.
Lets see how the SVAE performs.
Prototype V10.2 - Patch32 - Patchwork Cross-Attention with Edge Smoothing
This eliminates the edge cutting of the last version, and in the process the recon accuracy has gone up.
Model still escapes the discharge within 2 epochs and has robust recon.
Defeated the last version.
Prototype V10.1 Patchwork Cross-Attention - Stabilized
The patchwork has stabilized, and the output is more accurate than the original now that it supports SVD 32 with more accuracy and higher speed
Epoch 28 hit the unstable point, but the gradient clipped attention was the ticket that ensured solidity.
The discharge recovered immediately.
Give or take 97% accurate recall, lets get those numbers up before we move onto more powerful image sets. Roughly 28m params.
174 | 0.0471 0.0315 8.0 | 0.0318 | 4.038 2.092 1.93 31.52 | 0.1271 0.995 1.50 | 0.26864 a:0.0471/0.0476
176 | 0.0471 0.0314 7.9 | 0.0318 | 4.038 2.092 1.93 31.52 | 0.1343 1.000 1.50 | 0.26874 a:0.0471/0.0476
178 | 0.0471 0.0314 7.9 | 0.0318 | 4.038 2.093 1.93 31.52 | 0.1313 0.995 1.50 | 0.26883 a:0.0471/0.0477
180 | 0.0471 0.0314 8.0 | 0.0317 | 4.038 2.093 1.93 31.52 | 0.1312 1.000 1.50 | 0.26892 a:0.0471/0.0477
182 | 0.0471 0.0314 7.9 | 0.0317 | 4.038 2.093 1.93 31.52 | 0.1310 0.995 1.50 | 0.26899 a:0.0471/0.0477
184 | 0.0471 0.0314 7.9 | 0.0317 | 4.038 2.092 1.93 31.52 | 0.1350 0.993 1.50 | 0.26906 a:0.0472/0.0477
186 | 0.0470 0.0314 7.9 | 0.0317 | 4.038 2.092 1.93 31.52 | 0.1338 1.000 1.50 | 0.26911 a:0.0472/0.0477
188 | 0.0470 0.0314 8.0 | 0.0317 | 4.038 2.093 1.93 31.52 | 0.1305 0.999 1.50 | 0.26916 a:0.0472/0.0477
190 | 0.0470 0.0314 8.0 | 0.0317 | 4.038 2.093 1.93 31.52 | 0.1358 0.999 1.50 | 0.26919 a:0.0472/0.0477
192 | 0.0470 0.0314 8.0 | 0.0317 | 4.038 2.092 1.93 31.52 | 0.1354 0.999 1.50 | 0.26922 a:0.0472/0.0477
194 | 0.0470 0.0314 7.9 | 0.0317 | 4.038 2.093 1.93 31.52 | 0.1301 0.992 1.50 | 0.26923 a:0.0472/0.0477
196 | 0.0470 0.0314 7.9 | 0.0317 | 4.038 2.093 1.93 31.52 | 0.1330 0.998 1.50 | 0.26924 a:0.0472/0.0477
198 | 0.0470 0.0314 7.9 | 0.0317 | 4.038 2.093 1.93 31.52 | 0.1312 1.000 1.50 | 0.26925 a:0.0472/0.0477
200 | 0.0470 0.0314 7.9 | 0.0317 | 4.038 2.093 1.93 31.52 | 0.1300 1.000 1.50 | 0.26925 a:0.0472/0.0477
==========================================================================================
FINAL ANALYSIS
==========================================================================================
PatchSVAE: 4 patches × (256, 32)
Target CV: 0.125
Recon MSE: 0.031701 +/- 0.024789
Row CV: 0.1300
Cross-attention S delta: 0.26925
Learned alpha per mode (coordination strength):
Layer 0: mean=0.0471 max=0.0477 min=0.0466
α[ 0]: 0.0470 #######################################
α[ 1]: 0.0473 #######################################
α[ 2]: 0.0474 #######################################
α[ 3]: 0.0473 #######################################
α[ 4]: 0.0471 #######################################
α[ 5]: 0.0474 #######################################
α[ 6]: 0.0469 #######################################
α[ 7]: 0.0472 #######################################
α[ 8]: 0.0470 #######################################
α[ 9]: 0.0475 #######################################
α[10]: 0.0467 #######################################
α[11]: 0.0471 #######################################
α[12]: 0.0477 #######################################
α[13]: 0.0466 #######################################
α[14]: 0.0471 #######################################
α[15]: 0.0472 #######################################
α[16]: 0.0472 #######################################
α[17]: 0.0471 #######################################
α[18]: 0.0470 #######################################
α[19]: 0.0475 #######################################
α[20]: 0.0466 #######################################
α[21]: 0.0477 #######################################
α[22]: 0.0470 #######################################
α[23]: 0.0469 #######################################
α[24]: 0.0472 #######################################
α[25]: 0.0472 #######################################
α[26]: 0.0471 #######################################
α[27]: 0.0471 #######################################
α[28]: 0.0474 #######################################
α[29]: 0.0472 #######################################
α[30]: 0.0466 #######################################
α[31]: 0.0475 #######################################
Layer 1: mean=0.0472 max=0.0477 min=0.0466
α[ 0]: 0.0474 #######################################
α[ 1]: 0.0472 #######################################
α[ 2]: 0.0470 #######################################
α[ 3]: 0.0471 #######################################
α[ 4]: 0.0474 #######################################
α[ 5]: 0.0470 #######################################
α[ 6]: 0.0474 #######################################
α[ 7]: 0.0473 #######################################
α[ 8]: 0.0473 #######################################
α[ 9]: 0.0470 #######################################
α[10]: 0.0477 #######################################
α[11]: 0.0472 #######################################
α[12]: 0.0466 #######################################
α[13]: 0.0477 #######################################
α[14]: 0.0473 #######################################
α[15]: 0.0471 #######################################
α[16]: 0.0472 #######################################
α[17]: 0.0471 #######################################
α[18]: 0.0476 #######################################
α[19]: 0.0470 #######################################
α[20]: 0.0475 #######################################
α[21]: 0.0470 #######################################
α[22]: 0.0472 #######################################
α[23]: 0.0475 #######################################
α[24]: 0.0472 #######################################
α[25]: 0.0471 #######################################
α[26]: 0.0475 #######################################
α[27]: 0.0474 #######################################
α[28]: 0.0472 #######################################
α[29]: 0.0469 #######################################
α[30]: 0.0476 #######################################
α[31]: 0.0466 #######################################
Coordinated singular value profile:
S[ 0]: 4.0376 cum= 5.3% #############################
S[ 1]: 3.9321 cum= 10.3% #############################
S[ 2]: 3.8501 cum= 15.1% ############################
S[ 3]: 3.7785 cum= 19.8% ############################
S[ 4]: 3.7092 cum= 24.2% ###########################
S[ 5]: 3.6414 cum= 28.5% ###########################
S[ 6]: 3.5771 cum= 32.7% ##########################
S[ 7]: 3.5158 cum= 36.7% ##########################
S[ 8]: 3.4554 cum= 40.6% #########################
S[ 9]: 3.3961 cum= 44.3% #########################
S[10]: 3.3371 cum= 48.0% ########################
S[11]: 3.2788 cum= 51.5% ########################
S[12]: 3.2230 cum= 54.8% #######################
S[13]: 3.1681 cum= 58.1% #######################
S[14]: 3.1141 cum= 61.2% #######################
S[15]: 3.0607 cum= 64.3% ######################
S[16]: 3.0088 cum= 67.2% ######################
S[17]: 2.9568 cum= 70.1% #####################
S[18]: 2.9075 cum= 72.8% #####################
S[19]: 2.8572 cum= 75.5% #####################
S[20]: 2.8067 cum= 78.0% ####################
S[21]: 2.7584 cum= 80.5% ####################
S[22]: 2.7075 cum= 82.9% ####################
S[23]: 2.6574 cum= 85.2% ###################
S[24]: 2.6060 cum= 87.4% ###################
S[25]: 2.5535 cum= 89.5% ##################
S[26]: 2.4991 cum= 91.5% ##################
S[27]: 2.4413 cum= 93.5% ##################
S[28]: 2.3770 cum= 95.3% #################
S[29]: 2.2906 cum= 97.0% #################
S[30]: 2.2012 cum= 98.6% ################
S[31]: 2.0926 cum=100.0% ###############
Saving reconstruction grid...
Saved to /content/svae_patch_recon.png
Prototype V10 Patchwork Cross-Attention - Unstable
Tiny Imagenet can't draw enough information from a single monotonic MLP projection, so I'm breaking the structure into quadrant-based mlp patches with cross-attention for a prototype.
Each patch is 32x32 and they have svd 24 independently represented each with patchwork cross-attention. Similar to a vit, so I'm building it to a full vit structure over time to ensure solidity and solidarity.
Current proto is more stable but requires a bit more oomph.
The CV is enjoying it's drift a BIT too much
I'll try attention alpha rather than rigid alpha. 4 patches is a bit unstable, so lets get some stability.
Prototype V9 prod
Should run on colab. Install the necessary repos.
https://huggingface.co/AbstractPhil/geolip-SVAE/blob/main/prototype_v9_prod.py
Prototype V8 Soft Hand Loss
Stable prototype found. Scaling with the CV ratio within this band is a stable attractor to the structural response.
The soft hand loss is acting like a stable attractant. Correct utilization of this behavior can directly attenuate a model's structural internals to align to certain trajectory-based routes.
The alignment can be directly tuned at runtime, shifted to learn implicit rules, altered to teach specific behaviors, and more.
0.034 mse, which is a different gauge of loss entirely.
Prototype V7
Normalized spherical without magnitude, expected considerably faster with less accuracy at first stages.
What happens if you train with the wrong CV value?
Using geolip-core SVD (Gram + eigh)
SVAE - V=96, D=24 (Validated: CV=0.3668)
Matrix: (96, 24) = 2304 elements
SVD: geolip-core Gram+eigh
Losses: recon + CV(w=0.1, target=0.3668)
Params: 6,036,736
=====================================================================================
ep | loss recon cv_l t/ep | t_rec | S0 SD ratio erank | row_cv
-------------------------------------------------------------------------------------
1 | 0.4174 0.4169 0.0037 7.3 | 0.2843 | 5.39 1.977 2.73 23.15 | 0.3039
2 | 0.2492 0.2489 0.0031 7.3 | 0.2286 | 5.43 1.978 2.75 23.14 | 0.3148
3 | 0.2096 0.2093 0.0030 7.3 | 0.1946 | 5.50 1.982 2.77 23.13 | 0.3352
4 | 0.1858 0.1855 0.0021 7.2 | 0.1812 | 5.48 1.980 2.77 23.13 | 0.3460
6 | 0.1586 0.1581 0.0046 7.3 | 0.1541 | 5.31 1.873 2.83 23.09 | 0.3938
8 | 0.1419 0.1407 0.0096 7.3 | 0.1377 | 5.33 1.815 2.93 23.03 | 0.4565
10 | 0.1314 0.1283 0.0385 7.3 | 0.1279 | 5.42 1.778 3.05 22.97 | 0.5373
12 | 0.1226 0.1160 0.0599 7.2 | 0.1162 | 5.67 1.738 3.26 22.86 | 0.6060
14 | 0.1189 0.1087 0.0847 7.1 | 0.1109 | 5.78 1.705 3.39 22.79 | 0.6643
16 | 0.1175 0.1014 0.1935 7.1 | 0.0996 | 6.17 1.701 3.63 22.67 | 0.7598
18 | 0.1170 0.0952 0.2238 7.2 | 0.0974 | 6.50 1.671 3.89 22.52 | 0.8211
20 | 0.1173 0.0905 0.1539 7.2 | 0.0907 | 6.69 1.649 4.06 22.43 | 0.8383
22 | 0.1200 0.0852 0.3335 7.1 | 0.0903 | 7.11 1.655 4.30 22.30 | 0.9128
24 | 0.1233 0.0817 0.2770 7.2 | 0.0831 | 7.51 1.646 4.56 22.15 | 0.9654
26 | 0.1286 0.0785 0.3243 7.1 | 0.0778 | 7.71 1.646 4.68 22.09 | 1.0196
28 | 0.1328 0.0752 0.4244 7.2 | 0.0780 | 7.84 1.636 4.80 22.02 | 1.1002
30 | 0.1373 0.0726 0.8786 7.1 | 0.0752 | 8.24 1.631 5.05 21.87 | 1.1243
32 | 0.1437 0.0703 0.6946 7.2 | 0.0704 | 8.52 1.631 5.23 21.76 | 1.2061
34 | 0.6025 0.6020 0.0062 7.1 | 0.5194 | 28.25 10.261 2.75 23.14 | 0.2935
36 | 0.4995 0.4990 0.0062 7.2 | 0.4949 | 29.82 10.939 2.73 23.15 | 0.2982
38 | 0.4947 0.4942 0.0058 7.2 | 0.4915 | 28.37 10.433 2.72 23.15 | 0.2988
40 | 0.4579 0.4574 0.0053 7.2 | 0.4557 | 26.31 9.585 2.74 23.14 | 0.3041
42 | 0.4333 0.4328 0.0051 7.1 | 0.4259 | 22.03 7.996 2.75 23.14 | 0.2984
44 | 0.4057 0.4054 0.0038 7.1 | 0.3880 | 21.15 7.656 2.76 23.15 | 0.3177
46 | 0.3670 0.3667 0.0024 7.2 | 0.3634 | 19.33 6.943 2.78 23.13 | 0.3280
48 | 0.3495 0.3493 0.0005 7.1 | 0.3457 | 18.34 6.569 2.79 23.13 | 0.3336
50 | 0.3341 0.3340 0.0010 7.3 | 0.3326 | 17.55 6.298 2.79 23.13 | 0.3424
52 | 0.3205 0.3204 0.0003 7.2 | 0.3182 | 16.94 6.069 2.79 23.12 | 0.3549
SNAP. right there at epoch 34. The tension was too strong, the model simply snapped. I had it set to around 0.366, and it requires that value there where it snapped to. 0.2935
The actual value as of the bulk embedding tests show; CV=0.2992 is the stable attractor, almost precisely where the model snapped to.
The effect was so strong, that the entire model had a forced reset when it realized the fundamental invalidity.
Why? I don't know yet.
Models
V4 5m SVD+EIGH 100 epochs 48x24
V4 111m 200x24 SVD+EIGH KL_DIV - Undercooked, needs more epochs -> sequel faulty, collapse
V3 v1024 - SVD 24
V2 16 modes
V1 8 modes
Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support