🚀 Caca Transformers by Lyon

Custom Transformer architecture with modern features for efficient language modeling.

🎯 Features

• 30+ Model Variants (10M - 1T parameters)
• Grouped Query Attention (GQA)
• RoPE Positional Encoding
• SwiGLU Activation
• Flash Attention Support

📦 Installation

pip install caca-transformers

🤗 Available Models

Browse all models below ↓

🔗 Links

• PyPI: https://pypi.org/project/caca-transformers/
• GitHub: https://github.com/Lyon-28/caca-transformers
• Email: [email protected]