---
license: other
license_name: bananamind-community-license-1.0
license_link: LICENSE
language:
- en
library_name: transformers
pipeline_tag: text-generation
datasets:
- HuggingFaceFW/fineweb-edu
- mlfoundations/dclm-baseline-1.0
- HuggingFaceTB/smollm-corpus
- HuggingFaceTB/finemath
tags:
- causal-lm
- language-model
- base-model
- small-language-model
- bananamind
- bananamind2
- bananamind2-pro
- pretrained
- 100b-tokens
- digit-tokenizer
- pytorch
- safetensors
- custom-code
- trust-remote-code
- custom-architecture
---

![BananaMind 2 Pro](banner.png)

# BananaMind-2-Pro

BananaMind-2-Pro is a **138,971,520-parameter** decoder-only base language model trained from scratch by BananaMind. The completed pretraining run processed **99,999,449,088 tokens**—the scheduled realization of a 100B-token curriculum—over **184,954 optimizer steps**.

The model has a **3,072-token context window** and a custom **32,768-token digit-aware byte-level BPE tokenizer**. Its architecture uses grouped-query attention, QK normalization, RoPE, SwiGLU, RMSNorm, tied input/output embeddings, and a KV cache for generation.

This is a base model, not an instruction-tuned or chat model. Use continuation-style prompts and load the repository with `trust_remote_code=True`.

## Release Status

| Field | Value |
|---|---:|
| Release type | Final base checkpoint |
| Checkpoint step | 184,953 |
| Optimizer steps completed | 184,954 |
| Tokens seen | 99,999,449,088 |
| Training target | 100B tokens |
| Final phase | Quality finish |
| Training status | Complete |
| Evaluation status | Complete |

## Model Details

| Field | Value |
|---|---:|
| Parameters | 138,971,520 |
| Architecture | BananaMind2Pro decoder-only Transformer |
| Layers | 24 |
| Hidden size | 640 |
| Intermediate size | 1,920 |
| Attention heads | 8 |
| KV heads | 4 |
| Head dimension | 80 |
| Attention style | Grouped-query attention with QK norm |
| MLP | SwiGLU |
| Position embeddings | RoPE |
| RoPE theta | 100,000 |
| Normalization | RMSNorm |
| RMSNorm epsilon | 1e-6 |
| Vocabulary size | 32,768 |
| Context length | 3,072 |
| Embeddings | Tied input/output embeddings |
| Generation cache | KV cache supported |
| Weight format | safetensors |
| HF architecture | `BananaMind2ProForCausalLM` |
| HF model type | `bananamind2_pro` |

### Architecture Overview

The model maps tokens through a tied input embedding, 24 pre-normalized Transformer blocks, a final RMSNorm, and the tied language-model head. Each block contains causal grouped-query self-attention followed by a SwiGLU feed-forward network, with residual connections around both sublayers.

Eight query heads share four key/value heads, reducing KV-cache size while retaining multi-head query capacity. QK normalization stabilizes attention logits, and RoPE supplies position information without learned absolute-position embeddings. During autoregressive generation, each layer can reuse cached keys and values instead of recomputing the full prefix.

## Evaluation

![BananaMind-2-Pro benchmark comparison](benchmarks.png)

These results describe the exact final checkpoint at step 184,953. ARC Easy, ARC Challenge, PIQA, and HellaSwag use zero-shot `acc_norm,none`. ArithMark 3 uses length-normalized continuation accuracy, while ArithMark 2 uses raw continuation accuracy. Code Elo is the Base Bench 1.1 code-completion category; Base Bench 1.1 Elo is the complete 350-item suite.

| Benchmark | BananaMind-2-Pro | BananaMind-2-Pro-Preview | GPT-X2.5-135M | BananaMind-2-Medium | GPT-2 |
|---|---:|---:|---:|---:|---:|
| Trainable parameters | 139M | 139M | 135M | 49.6M | 124M |
| ARC Easy | **53.58%** | 51.01% | 51.81% | 43.81% | 39.35% |
| ARC Challenge | 27.82% | 27.13% | **29.18%** | 25.34% | 22.35% |
| PIQA | 67.52% | 66.76% | **69.42%** | 61.86% | 62.08% |
| HellaSwag | **42.78%** | 39.83% | 40.57% | 32.43% | 31.26% |
| ArithMark 3 | 38.20% | **38.90%** | 38.10% | 36.20% | 35.70% |
| ArithMark 2 | **32.08%** | 28.60% | N/A | 28.20% | 26.48% |
| INT Index | 24.96 | 23.04 | **25.17** | 15.37 | N/A |
| Code Elo | **1407** | 1295 | 1253 | 1034 | 996 |
| Base Bench 1.1 Elo | **1124** | 1106 | 1106 | 1034 | 996 |

The INT Index chance-normalizes HellaSwag, the mean of ARC Easy and ARC Challenge, PIQA, and ArithMark 3. ArithMark 2 is not included:

```text
N(s, c) = 100 * (s - c) / (100 - c)
INT = [N(H,25) + N((ARC_E+ARC_C)/2,25) + N(P,50) + 0.65*N(A3,25)] / 3.65
```

### INT Index vs Training Compute

![INT Index versus estimated training compute](int_index_vs_compute.png)

Training compute is estimated consistently as `6 × parameters × training tokens`. Using the exact 138,971,520-parameter count gives **83,382.91 PFLOPs** for Pro Final at 100B tokens and **43,279.49 PFLOPs** for Pro Preview at 51.904512B tokens. GPT-X2.5 uses 135M parameters, 75B tokens, and a 2,048-token context window; context length is not a separate term in this 6ND approximation.

| Model | Estimated training compute | INT Index |
|---|---:|---:|
| BananaMind-2-Pro Final | 83,382.91 PFLOPs | 24.96 |
| BananaMind-2-Pro Preview | 43,279.49 PFLOPs | 23.04 |
| GPT-X2.5-135M | 60,750.00 PFLOPs | 25.17 |
| GPT-X2-125M | 56,286.75 PFLOPs | 23.36 |
| GPT-X-125M | 11,210.56 PFLOPs | 19.94 |
| Supra2-100M | 18,000.00 PFLOPs | 19.41 |
| SmolLM-135M | 484,254.03 PFLOPs | 25.74 |
| BananaMind-2-Medium | 14,867.33 PFLOPs | 15.37 |
| OPT-125M | 135,000.00 PFLOPs | 13.80 |

### Base Bench Checkpoint Progression

![BananaMind 2 Pro Base Bench checkpoint progression](base_bench_progression.png)

This progression contains **41 complete 350-item evaluations** from 2.70B tokens through the final 99.999B-token checkpoint. Every point uses CUDA, bfloat16, and batch size 1 for consistency. The final batch-1 endpoint is **1132 Elo**, **236/350 correct (67.43%)**, and **64.69% weighted accuracy**. The primary comparison table above retains the separately measured batch-32 result.

### Base Bench 1.1 Category Results

The final model passed 233 of 350 cases for 66.57% raw accuracy and 63.73% weighted accuracy. Its overall fixed-item rating is 1124 Elo.

| Category | Elo | Correct | Accuracy | Weighted accuracy |
|---|---:|---:|---:|---:|
| Language completion | 1570 | 50/50 | 100.00% | 100.00% |
| Commonsense | 1185 | 40/50 | 80.00% | 78.86% |
| World knowledge | 1168 | 40/50 | 80.00% | 77.12% |
| Context tracking | 938 | 22/50 | 44.00% | 41.72% |
| Quantitative | 886 | 15/50 | 30.00% | 28.71% |
| Logical reasoning | 1059 | 24/50 | 48.00% | 44.51% |
| Code completion | 1407 | 42/50 | 84.00% | 85.67% |
| **Overall** | **1124** | **233/350** | **66.57%** | **63.73%** |

Base Bench was evaluated locally with CUDA, bfloat16, and batch size 32. ArithMark 3 used CUDA bfloat16 with batch size 64; ArithMark 2 used CUDA bfloat16 with batch size 1. Results can vary with harness versions, tokenizer handling, dtype, and scoring configuration. All values are self-reported evaluations.

## Tokenizer

BananaMind-2-Pro uses a custom 32,768-token byte-level BPE tokenizer trained on 75 GiB of representative FineWeb-Edu, DCLM, Cosmopedia-v2, FineMath-4+, and NPSet-2 Python educational data. It uses NFKC normalization and digit-aware pre-tokenization.

Digits are isolated before byte-level BPE, preventing complete numbers from being merged into large number tokens.

| Token | ID |
|---|---:|
| `0` | 19 |
| `1` | 20 |
| `2` | 21 |
| `3` | 22 |
| `4` | 23 |
| `5` | 24 |
| `6` | 25 |
| `7` | 26 |
| `8` | 27 |
| `9` | 28 |

Special token IDs:

| Token | ID |
|---|---:|
| `<|pad|>` | 0 |
| `<|bos|>` | 1 |
| `<|eos|>` | 2 |
| `<|unk|>` | 3 |

## Training Data

The 100B-token curriculum combines educational web text, broad web text, synthetic textbook material, mathematics, and Python educational code.

| Dataset | Target tokens | Aggregate share |
|---|---:|---:|
| FineWeb-Edu | 50.166B | 50.17% |
| DCLM | 26.125B | 26.13% |
| Cosmopedia-v2 | 13.525B | 13.53% |
| FineMath-4+ | 7.875B | 7.88% |
| NPSet-2 Python Edu | 2.309B | 2.31% |
| **Total** | **100.000B** | **100.00%** |

Training used a capacity-aware curriculum:

| Phase | Token range | Purpose |
|---|---:|---|
| Breadth foundation | 0B to 25B | Web-heavy language and knowledge foundation |
| Knowledge ramp | 25B to 40B | Gradual increase in synthetic, mathematics, and code data |
| Reasoning core | 40B to 75B | Sustained reasoning-oriented mixture |
| Synthesis ramp | 75B to 90B | Transition toward the finishing distribution |
| Quality finish | 90B to 100B | Final quality-focused mixture |

## Training Setup

| Field | Value |
|---|---:|
| Sequence length | 3,072 |
| Micro batch | 4 |
| Gradient accumulation | 44 |
| Effective batch | 176 sequences |
| Tokens per optimizer step | 540,672 |
| Optimizer steps | 184,954 |
| Scheduled training tokens | 99,999,449,088 |
| Optimizer | AdamW |
| Betas | 0.9, 0.95 |
| Peak learning rate | 1.5e-3 |
| Warmup steps | 2,000 |
| LR schedule | Warmup-stable-decay with cosine decay |
| Decay ratio | 0.15 |
| Weight decay | 0.1, then 0.01 after 40B tokens |
| Gradient clipping | 1.0 |
| Z-loss coefficient | 1e-4 until 40B tokens, then off |
| Compile | PyTorch compile enabled |
| Seed | 1337 |

The schedule used a 2,000-step linear warmup, a long stable learning-rate region, and a final cosine decay over the last 15% of training. Weight decay dropped from 0.1 to 0.01 after 40B tokens, and the auxiliary z-loss was disabled at the same boundary. The final optimizer update completed the planned 99,999,449,088-token schedule without rounding up to an extra sequence.

## Energy and Carbon Estimate

The following is an engineering estimate for the complete pretraining run, not a wall-meter measurement. Runtime is extrapolated from the observed 52,438-token/s run average. GPU and CPU package power were measured during training; motherboard, memory, storage, PSU loss, electricity use, and emissions are estimates.

| Item | Basis | Value |
|---|---|---:|
| Derived training time | 99.999449B tokens / 52,438 tokens/s | 529.72 hours (22.07 days) |
| GPU power | Measured 12-second `nvidia-smi` average at 99–100% utilization | 262 W |
| CPU package power | Two Intel RAPL samples of 42 W and 38 W | 40 W |
| Motherboard, memory, storage, and fans | Estimated combined power | 50 W |
| DC system load | Estimated | 352 W |
| PSU efficiency | Assumed | 90% |
| Wall power | Estimated | 391 W |
| Electricity use | Estimated | 207.1 kWh |
| Austrian grid intensity used | Recent daily estimate | 140 gCO2e/kWh |
| **Complete-run emissions** | **Estimated** | **29.0 kg CO2e** |

Using the cited 2024 and 2025 Austrian flow-traced annual means of 125.5 and 169.4 gCO2e/kWh gives an estimated range of **26.0–35.1 kg CO2e**. The grid figures come from [Electricity Maps](https://app.electricitymaps.com/map/zone/AT/3mo/daily). This estimate excludes embodied hardware emissions, tokenizer training, evaluation, external networking, and remote storage.

## Usage

Install the runtime dependencies:

```bash
pip install -U torch transformers safetensors
```

Load the custom architecture with remote code enabled:

```python
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "BananaMind/BananaMind-2-Pro"

tokenizer = AutoTokenizer.from_pretrained(
    model_id,
    trust_remote_code=True,
)

device = "cuda" if torch.cuda.is_available() else "cpu"
dtype = (
    torch.bfloat16
    if torch.cuda.is_available() and torch.cuda.is_bf16_supported()
    else torch.float32
)

model = AutoModelForCausalLM.from_pretrained(
    model_id,
    trust_remote_code=True,
    dtype=dtype,
).to(device).eval()

prompt = "The capital of France is"
inputs = tokenizer(prompt, return_tensors="pt").to(device)

with torch.no_grad():
    output = model.generate(
        **inputs,
        max_new_tokens=80,
        do_sample=True,
        temperature=0.7,
        top_p=0.9,
        repetition_penalty=1.1,
        pad_token_id=tokenizer.eos_token_id,
        eos_token_id=tokenizer.eos_token_id,
        use_cache=True,
    )

print(tokenizer.decode(output[0], skip_special_tokens=True))
```

For deterministic continuation scoring, use `do_sample=False`. For free-form sampling, a temperature from `0.6` to `0.8`, `top_p=0.9`, and `repetition_penalty=1.1` are reasonable starting points.

## Intended Use

BananaMind-2-Pro is intended for base-model research, local experimentation, text continuation, tokenizer research, arithmetic evaluation, checkpoint analysis, and small-language-model comparisons.

Suitable uses include studying training dynamics, testing compact-model architectures, comparing tokenizers, measuring continuation likelihoods, and using the checkpoint as a starting point for task-specific fine-tuning.

## Limitations

- This is a base model and does not reliably follow conversational instructions.
- The 3,072-token context window limits long-document use.
- At 139M parameters, factual recall and multi-step reasoning remain capacity-constrained.
- The training mixture is English-heavy, so performance in other languages is not characterized.
- The model has not received dedicated safety alignment and can produce inaccurate, biased, repetitive, or undesirable text.
- Loading requires repository-provided custom Transformers code with `trust_remote_code=True`.

Do not rely on the model for medical, legal, financial, safety-critical, or other high-stakes decisions.

## License

This repository is released under the [BananaMind Community License 1.0](LICENSE). Commercial products or services exceeding either threshold in Section 1 require a separate commercial license from Banaxi-Tech.