Training in progress - step 500

README.md

@@ -1,207 +1,73 @@
 ---
-base_model: Qwen/Qwen3-1.7B
-library_name: peft
-pipeline_tag: text-generation
+license: mit
+language:
+- en
+datasets:
+- speechbrain/LoquaciousSet
+base_model:
+- openai/whisper-large-v3-turbo
+- HuggingFaceTB/SmolLM3-3B
+pipeline_tag: automatic-speech-recognition
 tags:
-- base_model:adapter:Qwen/Qwen3-1.7B
-- lora
-- transformers
+- asr
+- speech-recognition
+- audio
+- smollm
+- whisper
+- mlp
 ---
 
-# Model Card for Model ID
+# Tiny Audio
 
-<!-- Provide a quick summary of what the model is/does. -->
+A speech recognition model trained in 24 hours on a single GPU for ~$12. Built with the [Tiny Audio](https://github.com/alexkroman/tiny-audio) codebase—a minimal, hackable framework for training ASR models.
 
+## Architecture
 
+```
+Audio (16kHz) → Whisper Encoder (frozen) → MLP Projector (trained) → SmolLM3-3B (frozen) → Text
+```
 
-## Model Details
-
-### Model Description
-
-<!-- Provide a longer summary of what this model is. -->
-
-
-
-- **Developed by:** [More Information Needed]
-- **Funded by [optional]:** [More Information Needed]
-- **Shared by [optional]:** [More Information Needed]
-- **Model type:** [More Information Needed]
-- **Language(s) (NLP):** [More Information Needed]
-- **License:** [More Information Needed]
-- **Finetuned from model [optional]:** [More Information Needed]
-
-### Model Sources [optional]
-
-<!-- Provide the basic links for the model. -->
-
-- **Repository:** [More Information Needed]
-- **Paper [optional]:** [More Information Needed]
-- **Demo [optional]:** [More Information Needed]
-
-## Uses
-
-<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
-
-### Direct Use
-
-<!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->
-
-[More Information Needed]
-
-### Downstream Use [optional]
-
-<!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->
-
-[More Information Needed]
-
-### Out-of-Scope Use
-
-<!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->
-
-[More Information Needed]
-
-## Bias, Risks, and Limitations
-
-<!-- This section is meant to convey both technical and sociotechnical limitations. -->
-
-[More Information Needed]
-
-### Recommendations
-
-<!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->
-
-Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.
-
-## How to Get Started with the Model
-
-Use the code below to get started with the model.
-
-[More Information Needed]
+**MLP Projector:**
+- Convolutional downsampling: 4x sequence compression via two stride-2 conv layers
+- Linear (1280 → 2048) → GELU → Linear (2048 → 2048)
+- Output normalization: RMSNorm
 
 ## Training Details
 
-### Training Data
-
-<!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
-
-[More Information Needed]
-
-### Training Procedure
-
-<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->
-
-#### Preprocessing [optional]
-
-[More Information Needed]
-
-
-#### Training Hyperparameters
-
-- **Training regime:** [More Information Needed] <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->
-
-#### Speeds, Sizes, Times [optional]
-
-<!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. -->
-
-[More Information Needed]
-
-## Evaluation
-
-<!-- This section describes the evaluation protocols and provides the results. -->
-
-### Testing Data, Factors & Metrics
-
-#### Testing Data
-
-<!-- This should link to a Dataset Card if possible. -->
-
-[More Information Needed]
-
-#### Factors
-
-<!-- These are the things the evaluation is disaggregating by, e.g., subpopulations or domains. -->
-
-[More Information Needed]
-
-#### Metrics
-
-<!-- These are the evaluation metrics being used, ideally with a description of why. -->
-
-[More Information Needed]
-
-### Results
-
-[More Information Needed]
-
-#### Summary
-
-
-
-## Model Examination [optional]
-
-<!-- Relevant interpretability work for the model goes here -->
-
-[More Information Needed]
-
-## Environmental Impact
-
-<!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->
-
-Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).
-
-- **Hardware Type:** [More Information Needed]
-- **Hours used:** [More Information Needed]
-- **Cloud Provider:** [More Information Needed]
-- **Compute Region:** [More Information Needed]
-- **Carbon Emitted:** [More Information Needed]
-
-## Technical Specifications [optional]
-
-### Model Architecture and Objective
-
-[More Information Needed]
-
-### Compute Infrastructure
-
-[More Information Needed]
-
-#### Hardware
-
-[More Information Needed]
-
-#### Software
-
-[More Information Needed]
-
-## Citation [optional]
-
-<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->
-
-**BibTeX:**
-
-[More Information Needed]
-
-**APA:**
+| | |
+|---|---|
+| **Dataset** | LoquaciousSet (25,000 hours) |
+| **Hardware** | Single NVIDIA A40 40GB |
+| **Training Time** | ~24 hours |
+| **Cost** | ~$12 |
+| **Trainable Parameters** | ~12M (projector only) |
 
-[More Information Needed]
+## Performance
 
-## Glossary [optional]
+**Word Error Rate (WER): 12.14%** on LoquaciousSet test set.
 
-<!-- If relevant, include terms and calculations in this section that can help readers understand the model or model card. -->
+See the [community leaderboard](https://github.com/alexkroman/tiny-audio#leaderboard) for comparisons.
 
-[More Information Needed]
+## Usage
 
-## More Information [optional]
+```python
+from transformers import pipeline
 
-[More Information Needed]
+pipe = pipeline("automatic-speech-recognition", model="mazesmazes/tiny-audio", trust_remote_code=True)
 
-## Model Card Authors [optional]
+result = pipe("path/to/audio.wav")
+print(result["text"])
+```
 
-[More Information Needed]
+## Limitations
 
-## Model Card Contact
+- English only
+- Optimized for 16kHz audio; other sample rates are resampled automatically
+- Performance may degrade on heavily accented speech, noisy environments, or domain-specific jargon
+- Maximum audio length limited by context window
 
-[More Information Needed]
-### Framework versions
+## Learn More
 
-- PEFT 0.18.0
+- **[Train your own model](https://github.com/alexkroman/tiny-audio)** — The full codebase with training scripts
+- **[Free 3-hour course](https://github.com/alexkroman/tiny-audio/blob/main/docs/course/0-course-overview.md)** — Build your own ASR system from scratch
+- **[Submit to leaderboard](https://github.com/alexkroman/tiny-audio#leaderboard)** — Share your trained model

asr_config.py

@@ -14,11 +14,10 @@ class ASRConfig(transformers.PretrainedConfig):
         attn_implementation: str = "flash_attention_2",
         model_dtype: str = "bfloat16",
         num_beams: Optional[int] = None,
-        system_prompt: str = "/no_think /system_override",
+        system_prompt: str = "You are a helpful assistant.",
         user_prompt: str = "Please transcribe this English audio into text: <audio>",
         encoder_dim: Optional[int] = None,
         llm_dim: Optional[int] = None,
-        encoder_stride: int = 2,  # Temporal downsampling factor of audio encoder (legacy, use encoder_conv_layers)
         # Encoder conv layers: list of (padding, kernel_size, stride) tuples
         # Default is Whisper/GLM-ASR structure: conv1(k=3,s=1,p=1) + conv2(k=3,s=2,p=1)
         encoder_conv_layers: Optional[list] = None,
@@ -52,7 +51,7 @@ class ASRConfig(transformers.PretrainedConfig):
         # Set default generation parameters (greedy decoding only)
         generation_defaults = {
             "num_beams": 1,
-            "max_new_tokens": 96,
+            "max_new_tokens": 256,
             "repetition_penalty": 1.0,
             "length_penalty": 1.0,
             "no_repeat_ngram_size": 0,
@@ -70,7 +69,6 @@ class ASRConfig(transformers.PretrainedConfig):
         self.user_prompt = user_prompt
         self.encoder_dim = encoder_dim
         self.llm_dim = llm_dim
-        self.encoder_stride = encoder_stride
         # Default conv layers for Whisper/GLM-ASR: [(pad, kernel, stride), ...]
         self.encoder_conv_layers = encoder_conv_layers or [(1, 3, 1), (1, 3, 2)]
         self.audio_sample_rate = audio_sample_rate

asr_modeling.py

@@ -96,7 +96,6 @@ class ASRModel(PreTrainedModel, GenerationMixin):
         super().__init__(config)
 
         self.system_prompt = config.system_prompt
-        self.encoder_stride = config.encoder_stride
         target_dtype = getattr(torch, config.model_dtype)
 
         # Audio encoder (frozen)
@@ -121,7 +120,10 @@ class ASRModel(PreTrainedModel, GenerationMixin):
         self.generation_config.length_penalty = config.length_penalty
         self.generation_config.repetition_penalty = config.repetition_penalty
         self.generation_config.no_repeat_ngram_size = config.no_repeat_ngram_size
-        self.generation_config.eos_token_id = self.tokenizer.convert_tokens_to_ids("<|im_end|>")
+        self.generation_config.eos_token_id = [
+            self.tokenizer.convert_tokens_to_ids("<|im_end|>"),
+            self.tokenizer.convert_tokens_to_ids("<|endoftext|>"),
+        ]
         self.generation_config.pad_token_id = self.tokenizer.pad_token_id
 
         # Feature extractor for audio preprocessing
@@ -145,7 +147,7 @@ class ASRModel(PreTrainedModel, GenerationMixin):
         encoder_kwargs = {
             "attn_implementation": config.attn_implementation,
             "low_cpu_mem_usage": True,
-            "torch_dtype": dtype,
+            "dtype": dtype,
         }
 
         if "whisper" in config.audio_model_id.lower():
@@ -296,7 +298,6 @@ class ASRModel(PreTrainedModel, GenerationMixin):
             feature_extractor=self.feature_extractor,
             tokenizer=self.tokenizer,
             projector=self.projector,
-            encoder_stride=self.encoder_stride,
             encoder_conv_layers=self.config.encoder_conv_layers,
         )
 

asr_pipeline.py

@@ -476,4 +476,32 @@ class ASRPipeline(transformers.AutomaticSpeechRecognitionPipeline):
         text = self.tokenizer.decode(tokens, skip_special_tokens=True).strip()
         # Strip <think>...</think> tags (Qwen3 doesn't respect /no_think prompt)
         text = re.sub(r"<think>.*?</think>\s*", "", text, flags=re.DOTALL).strip()
+        # Truncate if a word repeats more than 3 times consecutively
+        text = self._truncate_repetitions(text, max_repeats=3)
         return {"text": text}
+
+    def _truncate_repetitions(self, text: str, max_repeats: int = 3) -> str:
+        """Truncate text when a word repeats more than max_repeats times consecutively.
+
+        Args:
+            text: Input text to check for repetitions
+            max_repeats: Maximum allowed consecutive repetitions (default 3)
+
+        Returns:
+            Truncated text if repetition detected, otherwise original text
+        """
+        words = text.split()
+        if len(words) <= max_repeats:
+            return text
+
+        repeat_count = 1
+        for i in range(1, len(words)):
+            if words[i].lower() == words[i - 1].lower():
+                repeat_count += 1
+                if repeat_count > max_repeats:
+                    # Keep up to max_repeats of the repeated word
+                    return " ".join(words[:i])
+            else:
+                repeat_count = 1
+
+        return text

asr_processing.py

@@ -26,14 +26,12 @@ class ASRProcessor(ProcessorMixin):
         feature_extractor,
         tokenizer,
         projector=None,
-        encoder_stride: int = 2,
         encoder_conv_layers: Optional[list] = None,
     ):
         self.feature_extractor = feature_extractor
         self.tokenizer = tokenizer
         self.audio_token_id = tokenizer.convert_tokens_to_ids(self.AUDIO_TOKEN)
         self.projector = projector
-        self.encoder_stride = encoder_stride  # Legacy, kept for compatibility
         self.encoder_conv_layers = encoder_conv_layers or self.DEFAULT_ENCODER_CONV_LAYERS
 
     def _compute_encoder_output_length(self, mel_length: int) -> int: