nepetai/Fern-gear-200M-73M-A-IND

Fern Gear 200M (73M Active) - Industrial AI

Company: Nepetai Creator: Al-Hasan Al-Mazhani Model: Fern-gear-200M-73M-A-IND

Architecture

• Base: Fern
• Mixture of Experts (8 experts, top-2 routing)
• ViT encoder for visual inputs
• Bidirectional LSTM for sequence context
• Rotary Position Embeddings
• SwiGLU activation
• ChatML conversation format

Parameters

• 214.4M total parameters
• 70.5M active parameters (MoE sparse routing)
• 611MB safetensors

Specialization

Industrial Programming & Automation Expert:

• PLC (Ladder Logic, Structured Text, FBD, IL)
• Arduino & Embedded Systems (C/C++)
• IoT & SCADA Systems
• C++, Python, Bash scripting
• PID Control & Mechatronics
• Modbus, VFD, HMI, Sensors
• English, Arabic, Japanese

Usage

import torch
from fern_3b_config import Fern3BConfig
from fern_3b_model import Fern3BModel

config = Fern3BConfig(
    vocab_size=64000, max_seq_len=1024, d_model=512, n_heads=8,
    n_layers=12, d_ff=2048, num_experts=8, top_k_experts=2,
    image_size=224, patch_size=16, vit_d_model=256, vit_n_heads=4,
    vit_n_layers=4, vit_num_classes=256, lstm_hidden_size=512,
    lstm_num_layers=1, lstm_dropout=0.0
)
model = Fern3BModel(config)
model.load_state_dict(torch.load('model.safetensors'))
model.eval()

# Generate
from transformers import AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained('gpt2')
prompt = "<|im_start|>system\nYou are Fern Gear...\n<|im_end|>\n<|im_start|>user\nWhat is PLC?\n<|im_end|>\n<|im_start|>assistant\n"
tokens = tokenizer.encode(prompt, return_tensors='pt')
output = model.generate(tokens, max_new_tokens=200, temperature=0.7)
print(tokenizer.decode(output[0]))

Training Details

• Dataset: 6,256 real industrial programming samples
• Languages: English, Arabic, Japanese
• Epochs: 5
• Final Loss: 0.40
• Device: Macbook Pro M3
• Format: ChatML