Audio Classification
Transformers
English
audio
audio-captioning
audio-tagging
audioset
whisper
speech-captioning
music-captioning
sound-effect-captioning
laion
ast
audio-spectrogram-transformer
Instructions to use laion/whisper-captioning-ensemble with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use laion/whisper-captioning-ensemble with Transformers:
# Use a pipeline as a high-level helper from transformers import pipeline pipe = pipeline("audio-classification", model="laion/whisper-captioning-ensemble")# pip install -U transformers accelerate # Load model directly from transformers import AutoModel model = AutoModel.from_pretrained("laion/whisper-captioning-ensemble", device_map="auto") - Notebooks
- Google Colab
- Kaggle
Refresh sfx sample sidecars
Browse files- samples/audioset/audioset__rQmOOSlJ74g_195927.json +1 -1
- samples/audioset/audioset__xYVzy_dh20A_184856.json +3 -3
- samples/freesound/audio_203313_391226.json +3 -3
- samples/freesound/audio_206738_390405.json +1 -1
- samples/freesound/audio_206747_390848.json +1 -1
- samples/freesound/audio_238707_393020.json +2 -2
- samples/freesound/audio_312332_389459.json +1 -1
- samples/freesound/audio_315277_393270.json +1 -1
- samples/freesound/audio_360573_395201.json +1 -1
- samples/freesound/audio_389654_399191.json +1 -1
- samples/freesound/audio_391770_402152.json +1 -1
- samples/freesound/audio_41515_397745.json +1 -1
- samples/majestrino/majestrino__03001817.json +1 -1
- samples/majestrino/majestrino__03002030.json +1 -1
- samples/majestrino/majestrino__03002116.json +2 -2
- samples/music/music__suno_audio_196211_4_1844520.json +1 -1
samples/audioset/audioset__rQmOOSlJ74g_195927.json
CHANGED
|
@@ -17,7 +17,7 @@
|
|
| 17 |
],
|
| 18 |
"route": "sfx",
|
| 19 |
"annotations": {
|
| 20 |
-
"sound_effect_caption": "The audio features a distinct, high-pitched
|
| 21 |
},
|
| 22 |
"error": null
|
| 23 |
}
|
|
|
|
| 17 |
],
|
| 18 |
"route": "sfx",
|
| 19 |
"annotations": {
|
| 20 |
+
"sound_effect_caption": "The audio features a distinct, high-pitched squeaking sound. The squeak is short and sharp, with a slightly metallic quality. The sound is isolated, with no other discernible background noise. The audio is a recording of a squeaky toy being manipulated. The hint confirms the presence of a squeaky toy."
|
| 21 |
},
|
| 22 |
"error": null
|
| 23 |
}
|
samples/audioset/audioset__xYVzy_dh20A_184856.json
CHANGED
|
@@ -4,7 +4,7 @@
|
|
| 4 |
"audioset_top3": [
|
| 5 |
{
|
| 6 |
"label": "Speech",
|
| 7 |
-
"confidence": 0.
|
| 8 |
},
|
| 9 |
{
|
| 10 |
"label": "Inside, small room",
|
|
@@ -12,12 +12,12 @@
|
|
| 12 |
},
|
| 13 |
{
|
| 14 |
"label": "Squish",
|
| 15 |
-
"confidence": 0.
|
| 16 |
}
|
| 17 |
],
|
| 18 |
"route": "sfx",
|
| 19 |
"annotations": {
|
| 20 |
-
"sound_effect_caption": "The audio contains speech. A male voice is speaking, but the
|
| 21 |
},
|
| 22 |
"error": null
|
| 23 |
}
|
|
|
|
| 4 |
"audioset_top3": [
|
| 5 |
{
|
| 6 |
"label": "Speech",
|
| 7 |
+
"confidence": 0.780762
|
| 8 |
},
|
| 9 |
{
|
| 10 |
"label": "Inside, small room",
|
|
|
|
| 12 |
},
|
| 13 |
{
|
| 14 |
"label": "Squish",
|
| 15 |
+
"confidence": 0.09137
|
| 16 |
}
|
| 17 |
],
|
| 18 |
"route": "sfx",
|
| 19 |
"annotations": {
|
| 20 |
+
"sound_effect_caption": "The audio contains speech. A male voice is speaking, but the words are unintelligible. The speech is somewhat muffled and difficult to understand. The audio contains speech, as indicated by the hint. The muffled quality suggests the speaker is either far away, speaking through a barrier, or the recording was made at a distance."
|
| 21 |
},
|
| 22 |
"error": null
|
| 23 |
}
|
samples/freesound/audio_203313_391226.json
CHANGED
|
@@ -8,16 +8,16 @@
|
|
| 8 |
},
|
| 9 |
{
|
| 10 |
"label": "Ocean",
|
| 11 |
-
"confidence": 0.
|
| 12 |
},
|
| 13 |
{
|
| 14 |
"label": "Speech",
|
| 15 |
-
"confidence": 0.
|
| 16 |
}
|
| 17 |
],
|
| 18 |
"route": "sfx",
|
| 19 |
"annotations": {
|
| 20 |
-
"sound_effect_caption": "The audio captures the sounds of a large vehicle, likely a bus or truck, including engine noise, air brakes, and
|
| 21 |
},
|
| 22 |
"error": null
|
| 23 |
}
|
|
|
|
| 8 |
},
|
| 9 |
{
|
| 10 |
"label": "Ocean",
|
| 11 |
+
"confidence": 0.3479
|
| 12 |
},
|
| 13 |
{
|
| 14 |
"label": "Speech",
|
| 15 |
+
"confidence": 0.220215
|
| 16 |
}
|
| 17 |
],
|
| 18 |
"route": "sfx",
|
| 19 |
"annotations": {
|
| 20 |
+
"sound_effect_caption": "The audio captures the sounds of a large vehicle, likely a bus or truck, including engine noise, air brakes, and the distinct hiss of air brakes. The soundscape suggests an urban or industrial environment, possibly a bus stop or a large commercial vehicle, with the characteristic sounds of its air brakes and the hiss of air brakes."
|
| 21 |
},
|
| 22 |
"error": null
|
| 23 |
}
|
samples/freesound/audio_206738_390405.json
CHANGED
|
@@ -17,7 +17,7 @@
|
|
| 17 |
],
|
| 18 |
"route": "sfx",
|
| 19 |
"annotations": {
|
| 20 |
-
"sound_effect_caption": "The audio features a continuous, high-pitched
|
| 21 |
},
|
| 22 |
"error": null
|
| 23 |
}
|
|
|
|
| 17 |
],
|
| 18 |
"route": "sfx",
|
| 19 |
"annotations": {
|
| 20 |
+
"sound_effect_caption": "The audio features a continuous, high-pitched whirring sound, characteristic of a vacuum cleaner. The sound is consistent and sustained, indicating the operation of a motorized device. There are no other distinct sounds present. This is the sound of a vacuum cleaner in operation. The continuous nature of the sound suggests it is running steadily, likely for cleaning purposes."
|
| 21 |
},
|
| 22 |
"error": null
|
| 23 |
}
|
samples/freesound/audio_206747_390848.json
CHANGED
|
@@ -17,7 +17,7 @@
|
|
| 17 |
],
|
| 18 |
"route": "sfx",
|
| 19 |
"annotations": {
|
| 20 |
-
"sound_effect_caption": "The audio features a
|
| 21 |
},
|
| 22 |
"error": null
|
| 23 |
}
|
|
|
|
| 17 |
],
|
| 18 |
"route": "sfx",
|
| 19 |
"annotations": {
|
| 20 |
+
"sound_effect_caption": "The audio features a high-pitched, sustained electronic tone that gradually fades out. The sound is pure and consistent in its frequency and amplitude, without any discernible modulation or additional elements. This sound is characteristic of a digital alert, a test tone, or a simple electronic signal. It could be used as a simple notification, a system sound, or a component of a larger electronic device."
|
| 21 |
},
|
| 22 |
"error": null
|
| 23 |
}
|
samples/freesound/audio_238707_393020.json
CHANGED
|
@@ -4,7 +4,7 @@
|
|
| 4 |
"audioset_top3": [
|
| 5 |
{
|
| 6 |
"label": "Door",
|
| 7 |
-
"confidence": 0.
|
| 8 |
},
|
| 9 |
{
|
| 10 |
"label": "Silence",
|
|
@@ -17,7 +17,7 @@
|
|
| 17 |
],
|
| 18 |
"route": "sfx",
|
| 19 |
"annotations": {
|
| 20 |
-
"sound_effect_caption": "
|
| 21 |
},
|
| 22 |
"error": null
|
| 23 |
}
|
|
|
|
| 4 |
"audioset_top3": [
|
| 5 |
{
|
| 6 |
"label": "Door",
|
| 7 |
+
"confidence": 0.127441
|
| 8 |
},
|
| 9 |
{
|
| 10 |
"label": "Silence",
|
|
|
|
| 17 |
],
|
| 18 |
"route": "sfx",
|
| 19 |
"annotations": {
|
| 20 |
+
"sound_effect_caption": "A whoosh sound followed by a metallic clang. This sound suggests a rapid movement of air or an object, immediately followed by a metallic impact, possibly from a projectile hitting metal or a heavy object falling onto a metal surface."
|
| 21 |
},
|
| 22 |
"error": null
|
| 23 |
}
|
samples/freesound/audio_312332_389459.json
CHANGED
|
@@ -17,7 +17,7 @@
|
|
| 17 |
],
|
| 18 |
"route": "sfx",
|
| 19 |
"annotations": {
|
| 20 |
-
"sound_effect_caption": "The audio
|
| 21 |
},
|
| 22 |
"error": null
|
| 23 |
}
|
|
|
|
| 17 |
],
|
| 18 |
"route": "sfx",
|
| 19 |
"annotations": {
|
| 20 |
+
"sound_effect_caption": "The audio begins with a distinct mechanical whirring sound, followed by a series of rapid, high-pitched clicks or clacks, and then a final, softer mechanical thud. This sequence repeats multiple times. The sounds suggest the operation of a mechanical device, possibly a printer or a similar office machine, where internal components are moving, engaging, and then settling into place."
|
| 21 |
},
|
| 22 |
"error": null
|
| 23 |
}
|
samples/freesound/audio_315277_393270.json
CHANGED
|
@@ -17,7 +17,7 @@
|
|
| 17 |
],
|
| 18 |
"route": "sfx",
|
| 19 |
"annotations": {
|
| 20 |
-
"sound_effect_caption": "The audio features the distinct sound of a squeaky
|
| 21 |
},
|
| 22 |
"error": null
|
| 23 |
}
|
|
|
|
| 17 |
],
|
| 18 |
"route": "sfx",
|
| 19 |
"annotations": {
|
| 20 |
+
"sound_effect_caption": "The audio features the distinct sound of a squeaky wheel, accompanied by the rustling of fabric. The squeaky wheel sound is prominent, suggesting movement over a surface. The rustling could be from clothing or paper, and the squeaking might be from a door or a chair."
|
| 21 |
},
|
| 22 |
"error": null
|
| 23 |
}
|
samples/freesound/audio_360573_395201.json
CHANGED
|
@@ -17,7 +17,7 @@
|
|
| 17 |
],
|
| 18 |
"route": "sfx",
|
| 19 |
"annotations": {
|
| 20 |
-
"sound_effect_caption": "
|
| 21 |
},
|
| 22 |
"error": null
|
| 23 |
}
|
|
|
|
| 17 |
],
|
| 18 |
"route": "sfx",
|
| 19 |
"annotations": {
|
| 20 |
+
"sound_effect_caption": "The audio captures the distinct sound of a large vehicle, likely a truck, in operation, characterized by its engine noise and the sound of air brakes. The sound suggests the presence of heavy machinery or a large vehicle, possibly in an industrial or transportation context, indicating movement or a busy environment."
|
| 21 |
},
|
| 22 |
"error": null
|
| 23 |
}
|
samples/freesound/audio_389654_399191.json
CHANGED
|
@@ -17,7 +17,7 @@
|
|
| 17 |
],
|
| 18 |
"route": "sfx",
|
| 19 |
"annotations": {
|
| 20 |
-
"sound_effect_caption": "The audio features a variety of bird vocalizations, including chirps, calls, and possibly some
|
| 21 |
},
|
| 22 |
"error": null
|
| 23 |
}
|
|
|
|
| 17 |
],
|
| 18 |
"route": "sfx",
|
| 19 |
"annotations": {
|
| 20 |
+
"sound_effect_caption": "The audio features a variety of bird vocalizations, including chirps, calls, and possibly some squawks. The sounds are varied in pitch and rhythm, suggesting multiple birds or a single bird. This is a recording of birds in their natural environment, likely a garden, park, or forest, where birds are actively communicating. The variety and variety of calls suggest a diverse bird population."
|
| 21 |
},
|
| 22 |
"error": null
|
| 23 |
}
|
samples/freesound/audio_391770_402152.json
CHANGED
|
@@ -17,7 +17,7 @@
|
|
| 17 |
],
|
| 18 |
"route": "sfx",
|
| 19 |
"annotations": {
|
| 20 |
-
"sound_effect_caption": "The audio features a single, distinct, high-pitched electronic beep. The
|
| 21 |
},
|
| 22 |
"error": null
|
| 23 |
}
|
|
|
|
| 17 |
],
|
| 18 |
"route": "sfx",
|
| 19 |
"annotations": {
|
| 20 |
+
"sound_effect_caption": "The audio features a single, distinct, high-pitched electronic beep. The beep is short and sharp, with a clear, electronic timbre. This sound is characteristic of an electronic alert or notification, possibly from a digital device, a timer, or a simple electronic gadget."
|
| 21 |
},
|
| 22 |
"error": null
|
| 23 |
}
|
samples/freesound/audio_41515_397745.json
CHANGED
|
@@ -17,7 +17,7 @@
|
|
| 17 |
],
|
| 18 |
"route": "sfx",
|
| 19 |
"annotations": {
|
| 20 |
-
"sound_effect_caption": "A
|
| 21 |
},
|
| 22 |
"error": null
|
| 23 |
}
|
|
|
|
| 17 |
],
|
| 18 |
"route": "sfx",
|
| 19 |
"annotations": {
|
| 20 |
+
"sound_effect_caption": "A vehicle passing by, with engine noise and tire sounds, and a distinct whoosh. The audio captures the sound of a vehicle, likely a car or truck, passing by. The engine noise is prominent, indicating it is moving at a moderate speed. The sound includes the distinct whoosh of air as it passes, and the Doppler effect as it moves past the listener."
|
| 21 |
},
|
| 22 |
"error": null
|
| 23 |
}
|
samples/majestrino/majestrino__03001817.json
CHANGED
|
@@ -17,7 +17,7 @@
|
|
| 17 |
],
|
| 18 |
"route": "sfx",
|
| 19 |
"annotations": {
|
| 20 |
-
"sound_effect_caption": "A female voice
|
| 21 |
},
|
| 22 |
"error": null
|
| 23 |
}
|
|
|
|
| 17 |
],
|
| 18 |
"route": "sfx",
|
| 19 |
"annotations": {
|
| 20 |
+
"sound_effect_caption": "A female voice speaks in a frustrated and exasperated tone. The speaker is expressing negative feelings, using a harsh and somewhat sarcastic tone. The audio quality is clear."
|
| 21 |
},
|
| 22 |
"error": null
|
| 23 |
}
|
samples/majestrino/majestrino__03002030.json
CHANGED
|
@@ -17,7 +17,7 @@
|
|
| 17 |
],
|
| 18 |
"route": "sfx",
|
| 19 |
"annotations": {
|
| 20 |
-
"sound_effect_caption": "The audio contains speech from a female speaker. The speech is clear and understandable, with a moderate pace and a neutral tone. The audio contains speech, as indicated by the hint. The speaker is
|
| 21 |
},
|
| 22 |
"error": null
|
| 23 |
}
|
|
|
|
| 17 |
],
|
| 18 |
"route": "sfx",
|
| 19 |
"annotations": {
|
| 20 |
+
"sound_effect_caption": "The audio contains speech from a female speaker. The speech is clear and understandable, with a moderate pace and a neutral tone. The audio quality is good, with no noticeable background noise. The audio contains speech, as indicated by the hint. The speaker's voice is clear and understandable."
|
| 21 |
},
|
| 22 |
"error": null
|
| 23 |
}
|
samples/majestrino/majestrino__03002116.json
CHANGED
|
@@ -12,12 +12,12 @@
|
|
| 12 |
},
|
| 13 |
{
|
| 14 |
"label": "Silence",
|
| 15 |
-
"confidence": 0.
|
| 16 |
}
|
| 17 |
],
|
| 18 |
"route": "sfx",
|
| 19 |
"annotations": {
|
| 20 |
-
"sound_effect_caption": "The audio contains
|
| 21 |
},
|
| 22 |
"error": null
|
| 23 |
}
|
|
|
|
| 12 |
},
|
| 13 |
{
|
| 14 |
"label": "Silence",
|
| 15 |
+
"confidence": 0.32959
|
| 16 |
}
|
| 17 |
],
|
| 18 |
"route": "sfx",
|
| 19 |
"annotations": {
|
| 20 |
+
"sound_effect_caption": "The audio contains a male voice speaking. The speech is clear and articulate, with a moderate pace and a neutral tone. The recording quality is good, with minimal background noise. The audio is a recording of a male speaker, likely delivering information or engaging in a conversation. The hint confirms the presence of speech."
|
| 21 |
},
|
| 22 |
"error": null
|
| 23 |
}
|
samples/music/music__suno_audio_196211_4_1844520.json
CHANGED
|
@@ -17,7 +17,7 @@
|
|
| 17 |
],
|
| 18 |
"route": "sfx",
|
| 19 |
"annotations": {
|
| 20 |
-
"sound_effect_caption": "A male voice, perceived as adult, speaks in a clear, measured tone. The speech is articulate and articulate, with a slightly formal timbre. The pace is moderate, and the pitch is in the mid-range. The
|
| 21 |
},
|
| 22 |
"error": null
|
| 23 |
}
|
|
|
|
| 17 |
],
|
| 18 |
"route": "sfx",
|
| 19 |
"annotations": {
|
| 20 |
+
"sound_effect_caption": "A male voice, perceived as adult, speaks in a clear, measured tone, delivering a narrative or monologue. The speech is articulate and articulate, with a slightly formal timbre. The pace is moderate, and the pitch is in the mid-range. The audio quality is clean, with minimal background noise. This is a recording of a spoken word performance, likely a narration, a monologue, or a formal address. The clear articulation and measured pace suggest a professional or educational context, possibly for an audiobook, documentary, or a documentary."
|
| 21 |
},
|
| 22 |
"error": null
|
| 23 |
}
|