a2d-gpt-neox-160M

Base autoregressive → MLM diffusion models derived from the EleutherAI Pythia family.
These are base models only (no supervised fine-tuning / instruction tuning).

Model description

• Model type: Decoder-only Transformer converted to Masked Language Modeling (MLM) diffusion architecture
• Base architecture: Pythia (GPT-NeoX)

Variants

Size	HF Base	Checkpoint	Total Params	Non-embedding Params	Tokens seen (pretrain)	Tokens / Param
70M	EleutherAI/pythia-70m	step 512	70.4M	18.9M	~1.07B	~15 + continued
160M	EleutherAI/pythia-160m	step 1000	~162M	~125M	~2.1B	~13 + continued
410M	EleutherAI/pythia-410m	step 3000	405.3M	~350M	~6.29B	~15.5 + continued

Original Pythia pretraining & checkpoint selection (EleutherAI):
All models were initialized from Pythia checkpoints selected according to Chinchilla scaling laws (Hoffmann et al., 2022) targeting ~15 tokens per parameter (slightly undertrained regime) for consistency across the model family.

• Dataset: The Pile
• Total tokens in Pythia run: 299.9B
• Checkpoint interval: every 2,097,152 tokens (~2M batch size)

Selected checkpoints:

• 70M: step 512 (~1.07B tokens, ~15 tok/param)
• 160M: step 1000 (~2.1B tokens, ~13 tok/param)
• 410M: step 3000 (~6.29B tokens, ~15.5 tok/param)

Intended uses & limitations

• Research on autoregressive-to-diffusion conversion
• Continued pre-training and domain adaptation experiments
• Analysis of scaling behavior in hybrid MLM diffusion models
• Not intended for direct deployment or chat/instruction use (base models only)

Training and evaluation data

Continued pre-training: All models were further trained on the same subset of ~3B tokens sampled from FineWeb-Edu using (masked) causal language modeling objective.

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 0.001
• train_batch_size: 64
• eval_batch_size: 64
• seed: 46
• optimizer: Use OptimizerNames.ADAMW_TORCH_FUSED with betas=(0.85,0.94) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
• lr_scheduler_type: cosine
• lr_scheduler_warmup_steps: 468
• training_steps: 4687

How to Get Started with the Model

Use the following code to get started with the model:

from transformers import AutoModelForCausalLM, AutoTokenizer

model_name = "avgJo3/a2d-gpt-neox-160M"  # or 70m / 160m variant
revision = "main"  # or specific checkpoint tag if uploaded separately

model = AutoModelForCausalLM.from_pretrained(
    model_name, 
    revision=revision, 
    trust_remote_code=True
)
tokenizer = AutoTokenizer.from_pretrained(
    model_name, 
    revision=revision, 
    trust_remote_code=True
)

Framework versions

• Transformers 5.0.0
• Pytorch 2.10.0+cu128
• Datasets 4.8.5
• Tokenizers 0.22.2

Citation

@article{biderman2023pythia,
  title={Pythia: A Suite for Analyzing Large Language Models Across Training and Scaling},
  author={Biderman, Stella and others},
  journal={ICML},
  year={2023}
}