respinosamena/Helios-Nova-306M

Helios Nova 306M

Helios Nova 306M is a 306M-parameter, dense, decoder-only language model pre-trained from scratch on 50B tokens of FineWeb-Edu. It is the base model of the Helios Nova family; the architecture, tokenizer, pre-training, and evaluation were developed independently and end-to-end by a single author.

The model was built to study capability per unit of compute at small scale. At roughly 80× less pre-training data, it reaches 96% of SmolLM2-360M on commonsense reasoning (Winogrande + PIQA), measured on an identical evaluation harness. Pre-training cost under USD 190 of compute on a single GPU.

This is a base (pre-trained) model intended for text completion and as a starting point for fine-tuning. For instruction following and chat, use the instruction-tuned releases linked below.

Highlights

• 306M dense decoder, custom architecture and 16k tokenizer, trained from scratch.
• Data-efficient: 96% of SmolLM2-360M commonsense reasoning at ~80× fewer pre-training tokens; ties it on Winogrande.
• Low cost: 50B tokens on a single NVIDIA H100 in under 120 hours, for under USD 190.
• Modern recipe: Grouped-Query Attention, SwiGLU, RoPE, QK-Norm, RMSNorm pre-norm, tied embeddings, Warmup-Stable-Decay schedule.

Usage

import torch
from transformers import AutoTokenizer
from HeliosNova import HeliosNova   # from github.com/rafaelespinosamena/Helios-Nova-306M

tok = AutoTokenizer.from_pretrained("respinosamena/Helios-Nova-306M")
model = HeliosNova.from_pretrained("respinosamena/Helios-Nova-306M").eval()

ids = [tok.bos_token_id] + tok.encode("The history of computing began with", add_special_tokens=False)
out = model.generate(torch.tensor([ids]), max_new_tokens=64, temperature=0.8, top_k=50)
print(tok.decode(out[0], skip_special_tokens=True))

The model definition (HeliosNova.py) and full pre-training code are in the GitHub repository.

Model architecture

Component	Value
Parameters	305.8M (dense)
Layers / hidden size	24 / 1024 (depth-over-width, following the MobileLLM finding for sub-500M models)
Attention	Grouped-Query Attention — 16 query heads, 4 key-value heads, head dimension 64
Feed-forward	SwiGLU, intermediate size 3072
Positional encoding / norm	RoPE (theta 10,000), QK-Norm, RMSNorm (pre-norm), tied input/output embeddings
Tokenizer / context	Custom 16k BPE / 2048 tokens

Pre-training

Helios Nova 306M was pre-trained on 50B tokens of FineWeb-Edu on a single NVIDIA H100 in under 120 hours, for under USD 190. It uses a Warmup-Stable-Decay (WSD) learning-rate schedule with fused AdamW, bf16, and torch.compile. FineWeb-Edu (the educationally filtered subset of FineWeb) was chosen deliberately: the goal was to test whether architecture and a clean corpus could carry data efficiency at a fraction of the usual token budget. The validation loss decreases throughout the stable phase and drops sharply during the final decay.

Setting	Value
Tokens	50B (FineWeb-Edu)
Hardware	1 × NVIDIA H100, < 120 h
Cost	< USD 190
Optimizer	AdamW (fused), betas 0.9 / 0.95, weight decay 0.1, grad clip 1.0
Schedule	Warmup-Stable-Decay, peak LR 3e-4
Precision	bf16 + torch.compile

Evaluation

All models below were re-run through one identical lm-evaluation-harness configuration (0-shot), so the comparison is internally consistent; these figures therefore differ slightly from each model's published numbers.

Metric (0-shot)	Helios-306M (50B tok)	SmolLM2-360M (~4T)	Qwen2.5-0.5B (~18T)
Winogrande	57.2	57.9	56.3
PIQA	68.1	72.6	70.6
OpenBookQA	34.4	37.6	35.4
HellaSwag	44.7	52.5	49.5
ARC (avg)	42.8	53.4	45.5
MMLU	24.3	25.3	47.6
Commonsense reasoning (Winogrande + PIQA)	62.65	65.25	63.45

Helios reaches 96.0% of SmolLM2-360M on commonsense reasoning (Winogrande + PIQA) at roughly 80× less pre-training data, and ties it on Winogrande (99%). It trails on tasks bounded by data volume — broad factual recall (TriviaQA) and exam-style knowledge, where Qwen2.5-0.5B's much larger curated corpus is decisive. Helios Nova is data-efficient, not knowledge-rich.

Intended use and limitations

This is a base model: it performs next-token continuation and is intended for text completion and as a foundation for fine-tuning (instruction tuning, preference optimization, domain adaptation). It is not instruction-tuned and will not reliably follow prompts; for that, use the instruction-tuned releases below.

A 306M-parameter model trained on 50B tokens of educational text has limited world knowledge and performs near chance on broad factual recall (TriviaQA) and exam-style benchmarks (MMLU). Outputs may be inaccurate; verify before use. The model is English-only and has received no safety alignment.

The Helios Nova family

Model	Description
Helios-Nova-306M (this model)	From-scratch base model (50B tokens)
Helios-Nova-306M-Instruct	SFT instruction model (PyTorch)
Helios-Nova-306M-Instruct-GGUF	GGUF build of the SFT instruction model
Helios-Nova-306M-Instruct-2606	GRPO-aligned instruction model (GGUF and safetensors)

Citation

@misc{espinosamena2026heliosnova,
  title  = {Helios Nova 306M: a data-efficient language model pre-trained from scratch on a single GPU},
  author = {Espinosa Mena, Rafael},
  year   = {2026},
  howpublished = {\url{https://huggingface.co/respinosamena/Helios-Nova-306M}}
}

Contact

Rafael Espinosa Mena — rafaelespinosamena@gmail.com

License

Released under the Apache-2.0 license. Copyright 2026 Rafael Espinosa Mena.