Object detection

Object detection is the computer vision task of detecting instances (such as humans, buildings, or cars) in an image. Object detection models receive an image as input and output
coordinates of the bounding boxes and associated labels of the detected objects. An image can contain multiple objects,
each with its own bounding box and a label (e.g. it can have a car and a building), and each object can
be present in different parts of an image (e.g. the image can have several cars).
This task is commonly used in autonomous driving for detecting things like pedestrians, road signs, and traffic lights.
Other applications include counting objects in images, image search, and more.
In this guide, you will learn how to:
Finetune DETR, a model that combines a convolutional
backbone with an encoder-decoder Transformer, on the C


Object detection

PPE-5
dataset. Use your finetuned model for inference.
The task illustrated in this tutorial is supported by the following model architectures:
 Conditional DETR, Deformable DETR, DETA, DETR, Table Transformer, YOLOS
Before you begin, make sure you have all the necessary libraries installed:
  Copied pip install -q datasets transformers evaluate timm albumentations
You’ll use 🤗 Datasets to load a dataset from the Hugging Face Hub, 🤗 Transformers to train your model,
and albumentations to augment the data. timm is currently required to load a convolutional backbone for the DETR model.
We encourage you to share your model with the community. Log in to your Hugging Face account to upload it to the Hub.
When prompted, enter your token to log in:
  Copied >>> from huggingface_hub import notebook_login
>>> notebook_login()


Load the CPPE-5 dataset

The CPPE-5 dataset contains images with
annotations identifying medical personal protective equipment (PPE) in the context of the COVID-19 pandemic.
Start by loading the dataset:
  Copied >>> from datasets import load_dataset
>>> cppe5 = load_dataset("cppe-5")
>>> cppe5
DatasetDict({
    train: Dataset({
        features: ['image_id', 'image', 'width', 'height', 'objects'],
        num_rows: 1000
    })
    test: Dataset({
        features: ['image_id', 'image', 'width', 'height', 'objects'],
        num_rows: 29
    })
})
You’ll see that this dataset already comes with a training set containing 1000 images and a test set with 29 images.
To get familiar with the data, explore what the examples look like.
  Copied >>> cppe5["train"][0]
{'image_id': 15,
 'image': <PIL.JpegImagePlugin.JpegImageFile image mode=RGB size=943x663 at 0x7F9EC9E77C10>,
 'width': 943,
 'height': 663,
 'objects': {'id': [114, 115, 116, 117],
  'area': [3796, 1


Load the CPPE-5 dataset

596, 152768, 81002],
  'bbox': [[302.0, 109.0, 73.0, 52.0],
   [810.0, 100.0, 57.0, 28.0],
   [160.0, 31.0, 248.0, 616.0],
   [741.0, 68.0, 202.0, 401.0]],
  'category': [4, 4, 0, 0]}}
The examples in the dataset have the following fields:
image_id: the example image id image: a PIL.Image.Image object containing the image width: width of the image height: height of the image objects: a dictionary containing bounding box metadata for the objects in the image:id: the annotation id area: the area of the bounding box bbox: the object’s bounding box (in the COCO format ) category: the object’s category, with possible values including Coverall (0), Face_Shield (1), Gloves (2), Goggles (3) and Mask (4)
You may notice that the bbox field follows the COCO format, which is the format that the DETR model expects.
However, the grouping of the fields inside objects differs from the annotation format DETR requires. You will
need to apply some pr


Load the CPPE-5 dataset

eprocessing transformations before using this data for training.
To get an even better understanding of the data, visualize an example in the dataset.
  Copied >>> import numpy as np
>>> import os
>>> from PIL import Image, ImageDraw
>>> image = cppe5["train"][0]["image"]
>>> annotations = cppe5["train"][0]["objects"]
>>> draw = ImageDraw.Draw(image)
>>> categories = cppe5["train"].features["objects"].feature["category"].names
>>> id2label = {index: x for index, x in enumerate(categories, start=0)}
>>> label2id = {v: k for k, v in id2label.items()}
>>> for i in range(len(annotations["id"])):
...     box = annotations["bbox"][i]
...     class_idx = annotations["category"][i]
...     x, y, w, h = tuple(box)
...     draw.rectangle((x, y, x + w, y + h), outline="red", width=1)
...     draw.text((x, y), id2label[class_idx], fill="white")
>>> image
To visualize the bounding boxes with associated labels, you can get the labels from the da


Load the CPPE-5 dataset

taset’s metadata, specifically
the category field.
You’ll also want to create dictionaries that map a label id to a label class (id2label) and the other way around (label2id).
You can use them later when setting up the model. Including these maps will make your model reusable by others if you share
it on the Hugging Face Hub.
As a final step of getting familiar with the data, explore it for potential issues. One common problem with datasets for
object detection is bounding boxes that “stretch” beyond the edge of the image. Such “runaway” bounding boxes can raise
errors during training and should be addressed at this stage. There are a few examples with this issue in this dataset.
To keep things simple in this guide, we remove these images from the data.
  Copied >>> remove_idx = [590, 821, 822, 875, 876, 878, 879]
>>> keep = [i for i in range(len(cppe5["train"])) if i not in remove_idx]
>>> cppe5["train"] = cppe5["train"].select(ke


Preprocess the data

To finetune a model, you must preprocess the data you plan to use to match precisely the approach used for the pre-trained model.
AutoImageProcessor takes care of processing image data to create pixel_values, pixel_mask, and
labels that a DETR model can train with. The image processor has some attributes that you won’t have to worry about:
image_mean = [0.485, 0.456, 0.406 ] image_std = [0.229, 0.224, 0.225]
These are the mean and standard deviation used to normalize images during the model pre-training. These values are crucial
to replicate when doing inference or finetuning a pre-trained image model.
Instantiate the image processor from the same checkpoint as the model you want to finetune.
  Copied >>> from transformers import AutoImageProcessor
>>> checkpoint = "facebook/detr-resnet-50"
>>> image_processor = AutoImageProcessor.from_pretrained(checkpoint)
Before passing the images to the image_processor, apply two preprocessing transf


Preprocess the data

ormations to the dataset:
Augmenting images Reformatting annotations to meet DETR expectations
First, to make sure the model does not overfit on the training data, you can apply image augmentation with any data augmentation library. Here we use Albumentations …
This library ensures that transformations affect the image and update the bounding boxes accordingly.
The 🤗 Datasets library documentation has a detailed guide on how to augment images for object detection,
and it uses the exact same dataset as an example. Apply the same approach here, resize each image to (480, 480),
flip it horizontally, and brighten it:
  Copied >>> import albumentations
>>> import numpy as np
>>> import torch
>>> transform = albumentations.Compose(
...     [
...         albumentations.Resize(480, 480),
...         albumentations.HorizontalFlip(p=1.0),
...         albumentations.RandomBrightnessContrast(p=1.0),
...     ],
...     bbox_params=albumentations.Bbox


Preprocess the data

Params(format="coco", label_fields=["category"]),
... )
The image_processor expects the annotations to be in the following format: {'image_id': int, 'annotations': List[Dict]},
where each dictionary is a COCO object annotation. Let’s add a function to reformat annotations for a single example:
  Copied >>> def formatted_anns(image_id, category, area, bbox):
...     annotations = []
...     for i in range(0, len(category)):
...         new_ann = {
...             "image_id": image_id,
...             "category_id": category[i],
...             "isCrowd": 0,
...             "area": area[i],
...             "bbox": list(bbox[i]),
...         }
...         annotations.append(new_ann)
...     return annotations
Now you can combine the image and annotation transformations to use on a batch of examples:
  Copied >>> # transforming a batch
>>> def transform_aug_ann(examples):
...     image_ids = examples["image_id"]
...     images, bboxes, area,


Preprocess the data

 categories = [], [], [], []
...     for image, objects in zip(examples["image"], examples["objects"]):
...         image = np.array(image.convert("RGB"))[:, :, ::-1]
...         out = transform(image=image, bboxes=objects["bbox"], category=objects["category"])
...         area.append(objects["area"])
...         images.append(out["image"])
...         bboxes.append(out["bboxes"])
...         categories.append(out["category"])
...     targets = [
...         {"image_id": id_, "annotations": formatted_anns(id_, cat_, ar_, box_)}
...         for id_, cat_, ar_, box_ in zip(image_ids, categories, area, bboxes)
...     ]
...     return image_processor(images=images, annotations=targets, return_tensors="pt")
Apply this preprocessing function to the entire dataset using 🤗 Datasets with_transform method. This method applies
transformations on the fly when you load an element of the dataset.
At this point, you can check what an example from the


Preprocess the data

dataset looks like after the transformations. You should see a tensor
with pixel_values, a tensor with pixel_mask, and labels.
  Copied >>> cppe5["train"] = cppe5["train"].with_transform(transform_aug_ann)
>>> cppe5["train"][15]
{'pixel_values': tensor([[[ 0.9132,  0.9132,  0.9132,  ..., -1.9809, -1.9809, -1.9809],
          [ 0.9132,  0.9132,  0.9132,  ..., -1.9809, -1.9809, -1.9809],
          [ 0.9132,  0.9132,  0.9132,  ..., -1.9638, -1.9638, -1.9638],
          ...,
          [-1.5699, -1.5699, -1.5699,  ..., -1.9980, -1.9980, -1.9980],
          [-1.5528, -1.5528, -1.5528,  ..., -1.9980, -1.9809, -1.9809],
          [-1.5528, -1.5528, -1.5528,  ..., -1.9980, -1.9809, -1.9809]],
         [[ 1.3081,  1.3081,  1.3081,  ..., -1.8431, -1.8431, -1.8431],
          [ 1.3081,  1.3081,  1.3081,  ..., -1.8431, -1.8431, -1.8431],
          [ 1.3081,  1.3081,  1.3081,  ..., -1.8256, -1.8256, -1.8256],
          ...,
          [-1.3179, -1.3179


Preprocess the data

, -1.3179,  ..., -1.8606, -1.8606, -1.8606],
          [-1.3004, -1.3004, -1.3004,  ..., -1.8606, -1.8431, -1.8431],
          [-1.3004, -1.3004, -1.3004,  ..., -1.8606, -1.8431, -1.8431]],
         [[ 1.4200,  1.4200,  1.4200,  ..., -1.6476, -1.6476, -1.6476],
          [ 1.4200,  1.4200,  1.4200,  ..., -1.6476, -1.6476, -1.6476],
          [ 1.4200,  1.4200,  1.4200,  ..., -1.6302, -1.6302, -1.6302],
          ...,
          [-1.0201, -1.0201, -1.0201,  ..., -1.5604, -1.5604, -1.5604],
          [-1.0027, -1.0027, -1.0027,  ..., -1.5604, -1.5430, -1.5430],
          [-1.0027, -1.0027, -1.0027,  ..., -1.5604, -1.5430, -1.5430]]]),
 'pixel_mask': tensor([[1, 1, 1,  ..., 1, 1, 1],
         [1, 1, 1,  ..., 1, 1, 1],
         [1, 1, 1,  ..., 1, 1, 1],
         ...,
         [1, 1, 1,  ..., 1, 1, 1],
         [1, 1, 1,  ..., 1, 1, 1],
         [1, 1, 1,  ..., 1, 1, 1]]),
 'labels': {'size': tensor([800, 800]), 'image_id': tensor([756]), 'cla


Preprocess the data

ss_labels': tensor([4]), 'boxes': tensor([[0.7340, 0.6986, 0.3414, 0.5944]]), 'area': tensor([519544.4375]), 'iscrowd': tensor([0]), 'orig_size': tensor([480, 480])}}
You have successfully augmented the individual images and prepared their annotations. However, preprocessing isn’t
complete yet. In the final step, create a custom collate_fn to batch images together.
Pad images (which are now pixel_values) to the largest image in a batch, and create a corresponding pixel_mask
to indicate which pixels are real (1) and which are padding (0).
  Copied >>> def collate_fn(batch):
...     pixel_values = [item["pixel_values"] for item in batch]
...     encoding = image_processor.pad(pixel_values, return_tensors="pt")
...     labels = [item["labels"] for item in batch]
...     batch = {}
...     batch["pixel_values"] = encoding["pixel_values"]
...     batch["pixel_mask"] = encoding["pixel_mask"]
...     batch["labels"] = labels
...     return batc


Training the DETR model

You have done most of the heavy lifting in the previous sections, so now you are ready to train your model!
The images in this dataset are still quite large, even after resizing. This means that finetuning this model will
require at least one GPU.
Training involves the following steps:
Load the model with AutoModelForObjectDetection using the same checkpoint as in the preprocessing. Define your training hyperparameters in TrainingArguments. Pass the training arguments to Trainer along with the model, dataset, image processor, and data collator. Call train() to finetune your model.
When loading the model from the same checkpoint that you used for the preprocessing, remember to pass the label2id
and id2label maps that you created earlier from the dataset’s metadata. Additionally, we specify ignore_mismatched_sizes=True to replace the existing classificat


Training the DETR model

ion head with a new one.
  Copied >>> from transformers import AutoModelForObjectDetection
>>> model = AutoModelForObjectDetection.from_pretrained(
...     checkpoint,
...     id2label=id2label,
...     label2id=label2id,
...     ignore_mismatched_sizes=True,
... )
In the TrainingArguments use output_dir to specify where to save your model, then configure hyperparameters as you see fit.
It is important you do not remove unused columns because this will drop the image column. Without the image column, you
can’t create pixel_values. For this reason, set remove_unused_columns to False.
If you wish to share your model by pushing to the Hub, set push_to_hub to True (you must be signed in to Hugging
Face to upload your model).
  Copied >>> from transformers import TrainingArguments
>>> training_args = TrainingArguments(
...     output_dir="detr-resnet-50_fin


Training the DETR model

etuned_cppe5",
...     per_device_train_batch_size=8,
...     num_train_epochs=10,
...     fp16=True,
...     save_steps=200,
...     logging_steps=50,
...     learning_rate=1e-5,
...     weight_decay=1e-4,
...     save_total_limit=2,
...     remove_unused_columns=False,
...     push_to_hub=True,
... )
Finally, bring everything together, and call train():
  Copied >>> from transformers import Trainer
>>> trainer = Trainer(
...     model=model,
...     args=training_args,
...     data_collator=collate_fn,
...     train_dataset=cppe5["train"],
...     tokenizer=image_processor,
... )
>>> trainer.train()
If you have set push_to_hub to True in the training_args, the training checkpoints are pushed to the
Hugging Face Hub. Upon training completion, push the final model to the Hub as well by calling the push_to_hub() method.
  Copied >>> trainer.push_to_hub(


Evaluate

Object detection models are commonly evaluated with a set of COCO-style metrics.
You can use one of the existing metrics implementations, but here you’ll use the one from torchvision to evaluate the final
model that you pushed to the Hub.
To use the torchvision evaluator, you’ll need to prepare a ground truth COCO dataset. The API to build a COCO dataset
requires the data to be stored in a certain format, so you’ll need to save images and annotations to disk first. Just like
when you prepared your data for training, the annotations from the cppe5["test"] need to be formatted. However, images
should stay as they are.
The evaluation step requires a bit of work, but it can be split in three major steps.
First, prepare the cppe5["test"] set: format the annotations and save the data to disk.
  Copied >>> import json
>>> # format annotations the same as for training, no need for data augmentation
>>> def val_formatted_a


Evaluate

nns(image_id, objects):
...     annotations = []
...     for i in range(0, len(objects["id"])):
...         new_ann = {
...             "id": objects["id"][i],
...             "category_id": objects["category"][i],
...             "iscrowd": 0,
...             "image_id": image_id,
...             "area": objects["area"][i],
...             "bbox": objects["bbox"][i],
...         }
...         annotations.append(new_ann)
...     return annotations
>>> # Save images and annotations into the files torchvision.datasets.CocoDetection expects
>>> def save_cppe5_annotation_file_images(cppe5):
...     output_json = {}
...     path_output_cppe5 = f"{os.getcwd()}/cppe5/"
...     if not os.path.exists(path_output_cppe5):
...         os.makedirs(path_output_cppe5)
...     path_anno = os.path.join(path_output_cppe5, "cppe5_ann.json")
...     categories_json = [{"supercategory": "none", "id": id, "name": id2label[id]} for id i


Evaluate

n id2label]
...     output_json["images"] = []
...     output_json["annotations"] = []
...     for example in cppe5:
...         ann = val_formatted_anns(example["image_id"], example["objects"])
...         output_json["images"].append(
...             {
...                 "id": example["image_id"],
...                 "width": example["image"].width,
...                 "height": example["image"].height,
...                 "file_name": f"{example['image_id']}.png",
...             }
...         )
...         output_json["annotations"].extend(ann)
...     output_json["categories"] = categories_json
...     with open(path_anno, "w") as file:
...         json.dump(output_json, file, ensure_ascii=False, indent=4)
...     for im, img_id in zip(cppe5["image"], cppe5["image_id"]):
...         path_img = os.path.join(path_output_cppe5, f"{img_id}.png")
...         im.save(path_img)
...     return path_output_cppe5, pat


Evaluate

h_anno
Next, prepare an instance of a CocoDetection class that can be used with cocoevaluator.
  Copied >>> import torchvision
>>> class CocoDetection(torchvision.datasets.CocoDetection):
...     def __init__(self, img_folder, image_processor, ann_file):
...         super().__init__(img_folder, ann_file)
...         self.image_processor = image_processor
...     def __getitem__(self, idx):
...         # read in PIL image and target in COCO format
...         img, target = super(CocoDetection, self).__getitem__(idx)
...         # preprocess image and target: converting target to DETR format,
...         # resizing + normalization of both image and target)
...         image_id = self.ids[idx]
...         target = {"image_id": image_id, "annotations": target}
...         encoding = self.image_processor(images=img, annotations=target, return_tensors="pt")
...         pixel_values = encoding["pixel_values"].squeeze()


Evaluate

# remove batch dimension
...         target = encoding["labels"][0]  # remove batch dimension
...         return {"pixel_values": pixel_values, "labels": target}
>>> im_processor = AutoImageProcessor.from_pretrained("devonho/detr-resnet-50_finetuned_cppe5")
>>> path_output_cppe5, path_anno = save_cppe5_annotation_file_images(cppe5["test"])
>>> test_ds_coco_format = CocoDetection(path_output_cppe5, im_processor, path_anno)
Finally, load the metrics and run the evaluation.
  Copied >>> import evaluate
>>> from tqdm import tqdm
>>> model = AutoModelForObjectDetection.from_pretrained("devonho/detr-resnet-50_finetuned_cppe5")
>>> module = evaluate.load("ybelkada/cocoevaluate", coco=test_ds_coco_format.coco)
>>> val_dataloader = torch.utils.data.DataLoader(
...     test_ds_coco_format, batch_size=8, shuffle=False, num_workers=4, collate_fn=collate_fn
... )
>>> with torch.no_grad():
...     for idx, batch in enumerate(tq


Evaluate

dm(val_dataloader)):
...         pixel_values = batch["pixel_values"]
...         pixel_mask = batch["pixel_mask"]
...         labels = [
...             {k: v for k, v in t.items()} for t in batch["labels"]
...         ]  # these are in DETR format, resized + normalized
...         # forward pass
...         outputs = model(pixel_values=pixel_values, pixel_mask=pixel_mask)
...         orig_target_sizes = torch.stack([target["orig_size"] for target in labels], dim=0)
...         results = im_processor.post_process(outputs, orig_target_sizes)  # convert outputs of model to COCO api
...         module.add(prediction=results, reference=labels)
...         del batch
>>> results = module.compute()
>>> print(results)
Accumulating evaluation results...
DONE (t=0.08s).
IoU metric: bbox
 Average Precision  (AP) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.352
 Average Precision  (AP) @[ IoU=0.50      | area=   all |


Evaluate

maxDets=100 ] = 0.681
 Average Precision  (AP) @[ IoU=0.75      | area=   all | maxDets=100 ] = 0.292
 Average Precision  (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.168
 Average Precision  (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.208
 Average Precision  (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.429
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=  1 ] = 0.274
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets= 10 ] = 0.484
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.501
 Average Recall     (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.191
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.323
 Average Recall     (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.590
These results can be further improved by adjusting the hyperparameters in TrainingArguments. Give it a go


Inference

Now that you have finetuned a DETR model, evaluated it, and uploaded it to the Hugging Face Hub, you can use it for inference.
The simplest way to try out your finetuned model for inference is to use it in a Pipeline. Instantiate a pipeline
for object detection with your model, and pass an image to it:
  Copied >>> from transformers import pipeline
>>> import requests
>>> url = "https://i.imgur.com/2lnWoly.jpg"
>>> image = Image.open(requests.get(url, stream=True).raw)
>>> obj_detector = pipeline("object-detection", model="devonho/detr-resnet-50_finetuned_cppe5")
>>> obj_detector(image)
You can also manually replicate the results of the pipeline if you’d like:
  Copied >


Inference

>> image_processor = AutoImageProcessor.from_pretrained("devonho/detr-resnet-50_finetuned_cppe5")
>>> model = AutoModelForObjectDetection.from_pretrained("devonho/detr-resnet-50_finetuned_cppe5")
>>> with torch.no_grad():
...     inputs = image_processor(images=image, return_tensors="pt")
...     outputs = model(**inputs)
...     target_sizes = torch.tensor([image.size[::-1]])
...     results = image_processor.post_process_object_detection(outputs, threshold=0.5, target_sizes=target_sizes)[0]
>>> for score, label, box in zip(results["scores"], results["labels"], results["boxes"]):
...     box = [round(i, 2) for i in box.tolist()]
...     print(
...         f"Detected {mo


Inference

del.config.id2label[label.item()]} with confidence "
...         f"{round(score.item(), 3)} at location {box}"
...     )
Detected Coverall with confidence 0.566 at location [1215.32, 147.38, 4401.81, 3227.08]
Detected Mask with confidence 0.584 at location [2449.06, 823.19, 3256.43, 1413.9]
Let’s plot the result:
  Copied >>> draw = ImageDraw.Draw(image)
>>> for score, label, box in zip(results["scores"], results["labels"], results["boxes"]):
...     box = [round(i, 2) for i in box.tolist()]
...     x, y, x2, y2 = tuple(box)
...     draw.rectangle((x, y, x2, y2), outline="red", width=1)
...     draw.text((x, y), model.config.id2label[label.item()], fill="white")
>>> imag


Question answering

Question answering tasks return an answer given a question. If you’ve ever asked a virtual assistant like Alexa, Siri or Google what the weather is, then you’ve used a question answering model before. There are two common types of question answering tasks:
Extractive: extract the answer from the given context. Abstractive: generate an answer from the context that correctly answers the question.
This guide will show you how to:
Finetune DistilBERT on the SQuAD dataset for extractive question answering. Use your finetuned model for inference.
The task illustrated in this tutorial is supported by the following model architectures:
 ALBERT, BART, BERT, BigBird, BigBird-Pegasus, BLOOM, CamemBERT, CANINE, ConvBERT, Data2VecText, DeBERTa, DeBERTa-v2, DistilBERT, ELECTRA, ERNIE, Erni


Question answering

eM, Falcon, FlauBERT, FNet, Funnel Transformer, OpenAI GPT-2, GPT Neo, GPT NeoX, GPT-J, I-BERT, LayoutLMv2, LayoutLMv3, LED, LiLT, Longformer, LUKE, LXMERT, MarkupLM, mBART, MEGA, Megatron-BERT, MobileBERT, MPNet, MPT, MRA, MT5, MVP, Nezha, Nyströmformer, OPT, QDQBert, Reformer, RemBERT, RoBERTa, RoBERTa-PreLayerNorm, RoCBert, RoFormer, Splinter, SqueezeBERT, T5, UMT5, XLM, XLM-RoBERTa, XLM-RoBERTa-XL, XLNet, X-MOD, YOSO
Before you begin, make sure you have all the necessary libraries installed:
  Copied pip install transformers datasets evaluate
We encourage you to login to your Hugging Face account so you can upload and share your model with the community. When prompted, enter your token to login:
  Copied >>> from huggingface_hub import notebook_login
>>> notebook_login()


Load SQuAD dataset

Start by loading a smaller subset of the SQuAD dataset from the 🤗 Datasets library. This’ll give you a chance to experiment and make sure everything works before spending more time training on the full dataset.
  Copied >>> from datasets import load_dataset
>>> squad = load_dataset("squad", split="train[:5000]")
Split the dataset’s train split into a train and test set with the train_test_split method:
  Copied >>> squad = squad.train_test_split(test_size=0.2)
Then take a look at an example:
  Copied >>> squad["train"][0]
{'answers': {'answer_start': [515], 'text': ['Saint Bernadette Soubirous']},
 'context': 'Architecturally, the school has a Catholic character. Atop the Main Building\'s gold dome is a golden statue of the Virgin Mary. Immediately in front of the Main Building and facing it, is a copper statue of Christ with arms upraised with th


Load SQuAD dataset

e legend "Venite Ad Me Omnes". Next to the Main Building is the Basilica of the Sacred Heart. Immediately behind the basilica is the Grotto, a Marian place of prayer and reflection. It is a replica of the grotto at Lourdes, France where the Virgin Mary reputedly appeared to Saint Bernadette Soubirous in 1858. At the end of the main drive (and in a direct line that connects through 3 statues and the Gold Dome), is a simple, modern stone statue of Mary.',
 'id': '5733be284776f41900661182',
 'question': 'To whom did the Virgin Mary allegedly appear in 1858 in Lourdes France?',
 'title': 'University_of_Notre_Dame'
}
There are several important fields here:
answers: the starting location of the answer token and the answer text. context: background information from which the model needs to extract the answer. question: the question a model should answer


Preprocess

The next step is to load a DistilBERT tokenizer to process the question and context fields:
  Copied >>> from transformers import AutoTokenizer
>>> tokenizer = AutoTokenizer.from_pretrained("distilbert-base-uncased")
There are a few preprocessing steps particular to question answering tasks you should be aware of:
Some examples in a dataset may have a very long context that exceeds the maximum input length of the model. To deal with longer sequences, truncate only the context by setting truncation="only_second". Next, map the start and end positions of the answer to the original context by setting
return_offset_mapping=True. With the mapping in hand, now you can find the start and end tokens of the answer. Use the sequence_ids method to
find which part of the offset corresponds to the question and which corresponds to the context.
Here is how you can create a function to truncate and map the start


Preprocess

and end tokens of the answer to the context:
  Copied >>> def preprocess_function(examples):
...     questions = [q.strip() for q in examples["question"]]
...     inputs = tokenizer(
...         questions,
...         examples["context"],
...         max_length=384,
...         truncation="only_second",
...         return_offsets_mapping=True,
...         padding="max_length",
...     )
...     offset_mapping = inputs.pop("offset_mapping")
...     answers = examples["answers"]
...     start_positions = []
...     end_positions = []
...     for i, offset in enumerate(offset_mapping):
...         answer = answers[i]
...         start_char = answer["answer_start"][0]
...         end_char = answer["answer_start"][0] + len(answer["text"][0])
...         sequence_ids = inputs.sequence_ids(i)
...         # Find the start and end of the context
...         idx = 0
...         while sequence_ids[idx] != 1:


Preprocess

...             idx += 1
...         context_start = idx
...         while sequence_ids[idx] == 1:
...             idx += 1
...         context_end = idx - 1
...         # If the answer is not fully inside the context, label it (0, 0)
...         if offset[context_start][0] > end_char or offset[context_end][1] < start_char:
...             start_positions.append(0)
...             end_positions.append(0)
...         else:
...             # Otherwise it's the start and end token positions
...             idx = context_start
...             while idx <= context_end and offset[idx][0] <= start_char:
...                 idx += 1
...             start_positions.append(idx - 1)
...             idx = context_end
...             while idx >= context_start and offset[idx][1] >= end_char:
...                 idx -= 1
...             end_positions.append(idx + 1)
...     inputs["start_positions"] = start_posi


Preprocess

tions
...     inputs["end_positions"] = end_positions
...     return inputs
To apply the preprocessing function over the entire dataset, use 🤗 Datasets map function. You can speed up the map function by setting batched=True to process multiple elements of the dataset at once. Remove any columns you don’t need:
  Copied >>> tokenized_squad = squad.map(preprocess_function, batched=True, remove_columns=squad["train"].column_names)
Now create a batch of examples using DefaultDataCollator. Unlike other data collators in 🤗 Transformers, the DefaultDataCollator does not apply any additional preprocessing such as padding.
  Pytorch  Hide Pytorch content    Copied >>> from transformers import DefaultDataCollator
>>> data_collator = DefaultDataCollator()  TensorFlow  Hide TensorFlow content    Copied >>> from transformers import DefaultDataCollator
>>> data_collator = DefaultDataCollator(return_tensors="tf")


Train

Pytorch  Hide Pytorch content  If you aren’t familiar with finetuning a model with the Trainer, take a look at the basic tutorial here! You’re ready to start training your model now! Load DistilBERT with AutoModelForQuestionAnswering:   Copied >>> from transformers import AutoModelForQuestionAnswering, TrainingArguments, Trainer
>>> model = AutoModelForQuestionAnswering.from_pretrained("distilbert-base-uncased") At this point, only three steps remain: Define your training hyperparameters in TrainingArguments. The only required parameter is output_dir which specifies where to save your model. You’ll push this model to the Hub by setting push_to_hub=True (you need to be signed in to Hugging Face to upload your model). Pass the training arguments to Trainer along with the model, dataset, tokenizer, and data collator. Call train() to finetune your model.   Copied >>> training_args = TrainingArguments(
...     output_dir="my_awesome_qa_model",
...     e


Train

valuation_strategy="epoch",
...     learning_rate=2e-5,
...     per_device_train_batch_size=16,
...     per_device_eval_batch_size=16,
...     num_train_epochs=3,
...     weight_decay=0.01,
...     push_to_hub=True,
... )
>>> trainer = Trainer(
...     model=model,
...     args=training_args,
...     train_dataset=tokenized_squad["train"],
...     eval_dataset=tokenized_squad["test"],
...     tokenizer=tokenizer,
...     data_collator=data_collator,
... )
>>> trainer.train() Once training is completed, share your model to the Hub with the push_to_hub() method so everyone can use your model:   Copied >>> trainer.push_to_hub()  TensorFlow  Hide TensorFlow content  If you aren’t familiar with finetuning a model with Keras, take a look at the basic tutorial here!
To finetune a model in TensorFlow, start by setting up an optimizer function, learning rate schedule, and some training hyperparameters:
  Copied >>> from transformers import create_optimizer


Train

>>> batch_size = 16
>>> num_epochs = 2
>>> total_train_steps = (len(tokenized_squad["train"]) // batch_size) * num_epochs
>>> optimizer, schedule = create_optimizer(
...     init_lr=2e-5,
...     num_warmup_steps=0,
...     num_train_steps=total_train_steps,
... ) Then you can load DistilBERT with TFAutoModelForQuestionAnswering:   Copied >>> from transformers import TFAutoModelForQuestionAnswering
>>> model = TFAutoModelForQuestionAnswering("distilbert-base-uncased") Convert your datasets to the tf.data.Dataset format with prepare_tf_dataset():   Copied >>> tf_train_set = model.prepare_tf_dataset(
...     tokenized_squad["train"],
...     shuffle=True,
...     batch_size=16,
...     collate_fn=data_collator,
... )
>>> tf_validation_set = model.prepare_tf_dataset(
...     tokenized_squad["test"],
...     shuffle=False,
...     batch_size=16,
...     collate_fn=data_collator,
... ) Configure the model for training with compile:   Copied >>> import t


Train

ensorflow as tf
>>> model.compile(optimizer=optimizer) The last thing to setup before you start training is to provide a way to push your model to the Hub. This can be done by specifying where to push your model and tokenizer in the PushToHubCallback:   Copied >>> from transformers.keras_callbacks import PushToHubCallback
>>> callback = PushToHubCallback(
...     output_dir="my_awesome_qa_model",
...     tokenizer=tokenizer,
... ) Finally, you’re ready to start training your model! Call fit with your training and validation datasets, the number of epochs, and your callback to finetune the model:   Copied >>> model.fit(x=tf_train_set, validation_data=tf_validation_set, epochs=3, callbacks=[callback]) Once training is completed, your model is automatically uploaded to the Hub so everyone can use it!
For a more in-depth example of how to finetune a model for question answering, take a look at the corresponding
PyTorch notebook
or TensorFlow notebook


Evaluate

Evaluation for question answering requires a significant amount of postprocessing. To avoid taking up too much of your time, this guide skips the evaluation step. The Trainer still calculates the evaluation loss during training so you’re not completely in the dark about your model’s performance.
If have more time and you’re interested in how to evaluate your model for question answering, take a look at the Question answering chapter from the 🤗 Hugging Face Course!


Inference

Great, now that you’ve finetuned a model, you can use it for inference!
Come up with a question and some context you’d like the model to predict:
  Copied >>> question = "How many programming languages does BLOOM support?"
>>> context = "BLOOM has 176 billion parameters and can generate text in 46 languages natural languages and 13 programming languages."
The simplest way to try out your finetuned model for inference is to use it in a pipeline(). Instantiate a pipeline for question answering with your model, and pass your text to it:
  Copied >>> from transformers import pipeline
>>> question_answerer = pipeline("question-answering", model="my_awesome_qa_model")
>>> question_answerer(question=question, context=context)
{'score': 0.2058267742


Inference

395401,
 'start': 10,
 'end': 95,
 'answer': '176 billion parameters and can generate text in 46 languages natural languages and 13'}
You can also manually replicate the results of the pipeline if you’d like:
  Pytorch  Hide Pytorch content  Tokenize the text and return PyTorch tensors:   Copied >>> from transformers import AutoTokenizer
>>> tokenizer = AutoTokenizer.from_pretrained("my_awesome_qa_model")
>>> inputs = tokenizer(question, context, return_tensors="pt") Pass your inputs to the model and return the logits:   Copied >>> import torch
>>> from transformers import AutoModelForQuestionAnswering
>>> model = AutoModelForQuestionAnswering.from_pretrained("my_awesome_qa_model")
>>> with torch.no_grad():
...     outputs = model(**inputs)


Inference

Get the highest probability from the model output for the start and end positions:   Copied >>> answer_start_index = outputs.start_logits.argmax()
>>> answer_end_index = outputs.end_logits.argmax() Decode the predicted tokens to get the answer:   Copied >>> predict_answer_tokens = inputs.input_ids[0, answer_start_index : answer_end_index + 1]
>>> tokenizer.decode(predict_answer_tokens)
'176 billion parameters and can generate text in 46 languages natural languages and 13'  TensorFlow  Hide TensorFlow content  Tokenize the text and return TensorFlow tensors:   Copied >>> from transformers import AutoTokenizer
>>> tokenizer = AutoTokenizer.from_pretrained("my_awesome_qa_model")
>>> inputs = tokenizer(question, text, return_tensors="tf") Pass y


Inference

our inputs to the model and return the logits:   Copied >>> from transformers import TFAutoModelForQuestionAnswering
>>> model = TFAutoModelForQuestionAnswering.from_pretrained("my_awesome_qa_model")
>>> outputs = model(**inputs) Get the highest probability from the model output for the start and end positions:   Copied >>> answer_start_index = int(tf.math.argmax(outputs.start_logits, axis=-1)[0])
>>> answer_end_index = int(tf.math.argmax(outputs.end_logits, axis=-1)[0]) Decode the predicted tokens to get the answer:   Copied >>> predict_answer_tokens = inputs.input_ids[0, answer_start_index : answer_end_index + 1]
>>> tokenizer.decode(predict_answer_tokens)
'176 billion parameters and can generate text in 46 languages natural languages and


Video classification

Video classification is the task of assigning a label or class to an entire video. Videos are expected to have only one class for each video. Video classification models take a video as input and return a prediction about which class the video belongs to. These models can be used to categorize what a video is all about. A real-world application of video classification is action / activity recognition, which is useful for fitness applications. It is also helpful for vision-impaired individuals, especially when they are commuting.
This guide will show you how to:
Fine-tune VideoMAE on a subset of the UCF101 data


Video classification

set. Use your fine-tuned model for inference.
The task illustrated in this tutorial is supported by the following model architectures:
 TimeSformer, VideoMAE, ViViT
Before you begin, make sure you have all the necessary libraries installed:
  Copied pip install -q pytorchvideo transformers evaluate
You will use PyTorchVideo (dubbed pytorchvideo) to process and prepare the videos.
We encourage you to log in to your Hugging Face account so you can upload and share your model with the community. When prompted, enter your token to log in:
  Copied >>> from huggingface_hub import notebook_login
>>> notebook_login(


Load UCF101 dataset

Start by loading a subset of the UCF-101 dataset. This will give you a chance to experiment and make sure everything works before spending more time training on the full dataset.
  Copied >>> from huggingface_hub import hf_hub_download
>>> hf_dataset_identifier = "sayakpaul/ucf101-subset"
>>> filename = "UCF101_subset.tar.gz"
>>> file_path = hf_hub_download(repo_id=hf_dataset_identifier, filename=filename, repo_type="dataset")
After the subset has been downloaded, you need to extract the compressed archive:
  Copied >>> import tarfile
>>> with tarfile.open(file_path) as t:
...      t.extractall(".")
At a high level, the dataset is organized like so:
  Copied UCF101_subset/
    train/
        BandMarching/
            video_1.mp4
            video_2.mp4
            ...
        Archery
            video_1.mp4
            video_2.mp4
            ...
        ...
    val/
        BandMarching/
            video_1.mp4
            video_


Load UCF101 dataset

2.mp4
            ...
        Archery
            video_1.mp4
            video_2.mp4
            ...
        ...
    test/
        BandMarching/
            video_1.mp4
            video_2.mp4
            ...
        Archery
            video_1.mp4
            video_2.mp4
            ...
        ...
The (sorted) video paths appear like so:
  Copied ...
'UCF101_subset/train/ApplyEyeMakeup/v_ApplyEyeMakeup_g07_c04.avi',
'UCF101_subset/train/ApplyEyeMakeup/v_ApplyEyeMakeup_g07_c06.avi',
'UCF101_subset/train/ApplyEyeMakeup/v_ApplyEyeMakeup_g08_c01.avi',
'UCF101_subset/train/ApplyEyeMakeup/v_ApplyEyeMakeup_g09_c02.avi',
'UCF101_subset/train/ApplyEyeMakeup/v_ApplyEyeMakeup_g09_c06.avi'
...
You will notice that there are video clips belonging to the same group / scene where group is denoted by g in the video file paths. v_ApplyEyeMakeup_g07_c04.avi and v_ApplyEyeMakeup_g07_c06.avi, for example.
For the validation and evaluation splits,


Load UCF101 dataset

you wouldn’t want to have video clips from the same group / scene to prevent data leakage. The subset that you are using in this tutorial takes this information into account.
Next up, you will derive the set of labels present in the dataset. Also, create two dictionaries that’ll be helpful when initializing the model:
label2id: maps the class names to integers. id2label: maps the integers to class names.
  Copied >>> class_labels = sorted({str(path).split("/")[2] for path in all_video_file_paths})
>>> label2id = {label: i for i, label in enumerate(class_labels)}
>>> id2label = {i: label for label, i in label2id.items()}
>>> print(f"Unique classes: {list(label2id.keys())}.")
# Unique classes: ['ApplyEyeMakeup', 'ApplyLipstick', 'Archery', 'BabyCrawling', 'BalanceBeam', 'BandMarching', 'BaseballPitch', 'Basketball', 'BasketballDunk', 'BenchPress'].
There are 10 unique classes. For each class, there are 30 videos in the training set.


Load a model to fine-tune

Instantiate a video classification model from a pretrained checkpoint and its associated image processor. The model’s encoder comes with pre-trained parameters, and the classification head is randomly initialized. The image processor will come in handy when writing the preprocessing pipeline for our dataset.
  Copied >>> from transformers import VideoMAEImageProcessor, VideoMAEForVideoClassification
>>> model_ckpt = "MCG-NJU/videomae-base"
>>> image_processor = VideoMAEImageProcessor.from_pretrained(model_ckpt)
>>> model = VideoMAEForVideoClassification.from_pretrained(
...     model_ckpt,
...     label2id=label2id,
...     id2label=id2label,
...     ignore_mismatched_sizes=True,  # provide this in case you're planning to fine-tune an already fine-tuned checkpoint
... )
While the model is loading, you might notice the following warning:
  Copied Some weights o


Load a model to fine-tune

f the model checkpoint at MCG-NJU/videomae-base were not used when initializing VideoMAEForVideoClassification: [..., 'decoder.decoder_layers.1.attention.output.dense.bias', 'decoder.decoder_layers.2.attention.attention.key.weight']
- This IS expected if you are initializing VideoMAEForVideoClassification from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).
- This IS NOT expected if you are initializing VideoMAEForVideoClassification from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).
Some weights of VideoMAEForVideoClassification were not initialized from the model checkpoint at MCG-NJU/videomae-base and are newly initialized


Load a model to fine-tune

: ['classifier.bias', 'classifier.weight']
You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.
The warning is telling us we are throwing away some weights (e.g. the weights and bias of the classifier layer) and randomly initializing some others (the weights and bias of a new classifier layer). This is expected in this case, because we are adding a new head for which we don’t have pretrained weights, so the library warns us we should fine-tune this model before using it for inference, which is exactly what we are going to do.
Note that this checkpoint leads to better performance on this task as the checkpoint was obtained fine-tuning on a similar downstream task having considerable domain overlap. You can check out this checkpoint which was obtained by fine-tuning MCG-NJU/videomae-base-finetuned-kinetic


Prepare the datasets for training

For preprocessing the videos, you will leverage the PyTorchVideo library. Start by importing the dependencies we need.
  Copied >>> import pytorchvideo.data
>>> from pytorchvideo.transforms import (
...     ApplyTransformToKey,
...     Normalize,
...     RandomShortSideScale,
...     RemoveKey,
...     ShortSideScale,
...     UniformTemporalSubsample,
... )
>>> from torchvision.transforms import (
...     Compose,
...     Lambda,
...     RandomCrop,
...     RandomHorizontalFlip,
...     Resize,
... )
For the training dataset transformations, use a combination of uniform temporal subsampling, pixel normalization, random cropping, and random horizontal flipping. For the validation and evaluation dataset transformations, keep the same transformation chain except for random cropping and horizontal flipping. To learn more about the details of these transformations check out the official documentation of PyT


Prepare the datasets for training

orchVideo.
Use the image_processor associated with the pre-trained model to obtain the following information:
Image mean and standard deviation with which the video frame pixels will be normalized. Spatial resolution to which the video frames will be resized.
Start by defining some constants.
  Copied >>> mean = image_processor.image_mean
>>> std = image_processor.image_std
>>> if "shortest_edge" in image_processor.size:
...     height = width = image_processor.size["shortest_edge"]
>>> else:
...     height = image_processor.size["height"]
...     width = image_processor.size["width"]
>>> resize_to = (height, width)
>>> num_frames_to_sample = model.config.num_frames
>>> sample_rate = 4
>>> fps = 30
>>> clip_duration = num_frames_to_sample * sample_rate / fps
Now, define the dataset-specific transformations and the datasets respectively. Starting with the training set:
  Copied >>> train_transform = Com


Prepare the datasets for training

pose(
...     [
...         ApplyTransformToKey(
...             key="video",
...             transform=Compose(
...                 [
...                     UniformTemporalSubsample(num_frames_to_sample),
...                     Lambda(lambda x: x / 255.0),
...                     Normalize(mean, std),
...                     RandomShortSideScale(min_size=256, max_size=320),
...                     RandomCrop(resize_to),
...                     RandomHorizontalFlip(p=0.5),
...                 ]
...             ),
...         ),
...     ]
... )
>>> train_dataset = pytorchvideo.data.Ucf101(
...     data_path=os.path.join(dataset_root_path, "train"),
...     clip_sampler=pytorchvideo.data.make_clip_sampler("random", clip_duration),
...     decode_audio=False,
...     transform=train_transform,
... )
The same sequence of workflow can be applied to the validation and evaluation sets:
  Copied >>> val_tran


Prepare the datasets for training

sform = Compose(
...     [
...         ApplyTransformToKey(
...             key="video",
...             transform=Compose(
...                 [
...                     UniformTemporalSubsample(num_frames_to_sample),
...                     Lambda(lambda x: x / 255.0),
...                     Normalize(mean, std),
...                     Resize(resize_to),
...                 ]
...             ),
...         ),
...     ]
... )
>>> val_dataset = pytorchvideo.data.Ucf101(
...     data_path=os.path.join(dataset_root_path, "val"),
...     clip_sampler=pytorchvideo.data.make_clip_sampler("uniform", clip_duration),
...     decode_audio=False,
...     transform=val_transform,
... )
>>> test_dataset = pytorchvideo.data.Ucf101(
...     data_path=os.path.join(dataset_root_path, "test"),
...     clip_sampler=pytorchvideo.data.make_clip_sampler("uniform", clip_duration),
...     decode_audio=False,
...     transf


Prepare the datasets for training

orm=val_transform,
... )
Note: The above dataset pipelines are taken from the official PyTorchVideo example. We’re using the pytorchvideo.data.Ucf101() function because it’s tailored for the UCF-101 dataset. Under the hood, it returns a pytorchvideo.data.labeled_video_dataset.LabeledVideoDataset object. LabeledVideoDataset class is the base class for all things video in the PyTorchVideo dataset. So, if you want to use a custom dataset not supported off-the-shelf by PyTorchVideo, you can extend the LabeledVideoDataset class accordingly. Refer to the data API documentation to learn more. Also, if your dataset follows a similar structure (as shown above), then using the pytorchvideo.data.Ucf101() should work just fine.
You can access the num_videos argument to know the number of videos in the dataset.
  Copied >>> print(train_dataset.num_videos, val_dataset.num_videos, test_dataset.num_videos)
# (300, 30,


Visualize the preprocessed video for better debugging

Copied >>> import imageio
>>> import numpy as np
>>> from IPython.display import Image
>>> def unnormalize_img(img):
...     """Un-normalizes the image pixels."""
...     img = (img * std) + mean
...     img = (img * 255).astype("uint8")
...     return img.clip(0, 255)
>>> def create_gif(video_tensor, filename="sample.gif"):
...     """Prepares a GIF from a video tensor.
...
...     The video tensor is expected to have the following shape:
...     (num_frames, num_channels, height, width).
...     """
...     frames = []
...     for video_frame in video_tensor:
...         frame_unnormalized = un


normalize_img(video_frame.permute(1, 2, 0).numpy())
...         frames.append(frame_unnormalized)
...     kargs = {"duration": 0.25}
...     imageio.mimsave(filename, frames, "GIF", **kargs)
...     return filename
>>> def display_gif(video_tensor, gif_name="sample.gif"):
...     """Prepares and displays a GIF from a video tensor."""
...     video_tensor = video_tensor.permute(1, 0, 2, 3)
...     gif_filename = create_gif(video_tensor, gif_name)
...     return Image(filename=gif_filename)
>>> sample_video = next(iter(train_dataset))
>>> video_tensor = sample_video["video"]
>>> display_gif(video_tensor


Train the model

Leverage Trainer from  🤗 Transformers for training the model. To instantiate a Trainer, you need to define the training configuration and an evaluation metric. The most important is the TrainingArguments, which is a class that contains all the attributes to configure the training. It requires an output folder name, which will be used to save the checkpoints of the model. It also helps sync all the information in the model repository on 🤗 Hub.
Most of the training arguments are self-explanatory, but one that is quite important here is remove_unused_columns=False. This one will drop any features not used by the model’s call function. By default it’s True because usually it’s ideal to drop unused feature columns, making it easier to unpack inputs into the model’s call function. But, in this case, you need the unused features (‘video’ in particular) in order to create pixel_values (which is a mandatory key our model expects in its inputs).
  Copied >>> from tra


Train the model

nsformers import TrainingArguments, Trainer
>>> model_name = model_ckpt.split("/")[-1]
>>> new_model_name = f"{model_name}-finetuned-ucf101-subset"
>>> num_epochs = 4
>>> args = TrainingArguments(
...     new_model_name,
...     remove_unused_columns=False,
...     evaluation_strategy="epoch",
...     save_strategy="epoch",
...     learning_rate=5e-5,
...     per_device_train_batch_size=batch_size,
...     per_device_eval_batch_size=batch_size,
...     warmup_ratio=0.1,
...     logging_steps=10,
...     load_best_model_at_end=True,
...     metric_for_best_model="accuracy",
...     push_to_hub=True,
...     max_steps=(train_dataset.num_videos // batch_size) * num_epochs,
... )
The dataset returned by pytorchvideo.data.Ucf101() doesn’t implement the __len__ method. As such, we must define max_steps when instantiating TrainingArguments.
Next, you need to define a function to compute the metrics from the predictions, which will use the metric you’ll load now. T


Train the model

he only preprocessing you have to do is to take the argmax of our predicted logits:
  Copied import evaluate
metric = evaluate.load("accuracy")
def compute_metrics(eval_pred):
    predictions = np.argmax(eval_pred.predictions, axis=1)
    return metric.compute(predictions=predictions, references=eval_pred.label_ids)
A note on evaluation:
In the VideoMAE paper, the authors use the following evaluation strategy. They evaluate the model on several clips from test videos and apply different crops to those clips and report the aggregate score. However, in the interest of simplicity and brevity, we don’t consider that in this tutorial.
Also, define a collate_fn, which will be used to batch examples together. Each batch consists of 2 keys, namely pixel_values and labels.
  Copied >>> def collate_fn(examples):
...     # permute to (num_frames, num_channels, height, width)
...     pixel_values = torch.stack(
...         [example["video"].permute(1, 0, 2, 3) for exam


Train the model

ple in examples]
...     )
...     labels = torch.tensor([example["label"] for example in examples])
...     return {"pixel_values": pixel_values, "labels": labels}
Then you just pass all of this along with the datasets to Trainer:
  Copied >>> trainer = Trainer(
...     model,
...     args,
...     train_dataset=train_dataset,
...     eval_dataset=val_dataset,
...     tokenizer=image_processor,
...     compute_metrics=compute_metrics,
...     data_collator=collate_fn,
... )
You might wonder why you passed along the image_processor as a tokenizer when you preprocessed the data already. This is only to make sure the image processor configuration file (stored as JSON) will also be uploaded to the repo on the Hub.
Now fine-tune our model by calling the train method:
  Copied >>> train_results = trainer.train()
Once training is completed, share your model to the Hub with the push_to_hub() method so everyone can use your model:
  Copied >>> trainer.push_to_hub()


Inference

Great, now that you have fine-tuned a model, you can use it for inference!
Load a video for inference:
  Copied >>> sample_test_video = next(iter(test_dataset))
The simplest way to try out your fine-tuned model for inference is to use it in a pipeline. Instantiate a pipeline for video classification with your model, and pass your video to it:
  Copied >>> from transformers import pipeline
>>> video_cls = pipeline(model="my_awesome_video_cls_model")
>>> video_cls("https://huggingface.co/datasets/sayakpaul/ucf101-subset/resolve/main/v_BasketballDunk_g14_c06.avi")
[{'score': 0.9272987842559814, 'label': 'BasketballDunk'},
 {'score': 0.017777055501937866, 'label':


Inference

'BabyCrawling'},
 {'score': 0.01663011871278286, 'label': 'BalanceBeam'},
 {'score': 0.009560945443809032, 'label': 'BandMarching'},
 {'score': 0.0068979403004050255, 'label': 'BaseballPitch'}]
You can also manually replicate the results of the pipeline if you’d like.
  Copied >>> def run_inference(model, video):
...     # (num_frames, num_channels, height, width)
...     perumuted_sample_test_video = video.permute(1, 0, 2, 3)
...     inputs = {
...         "pixel_values": perumuted_sample_test_video.unsqueeze(0),
...         "labels": torch.tensor(
...             [sample_test_video["label"]]
...         ),  # this can be skipped if you don't have labels avail


Inference

able.
...     }
...     device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
...     inputs = {k: v.to(device) for k, v in inputs.items()}
...     model = model.to(device)
...     # forward pass
...     with torch.no_grad():
...         outputs = model(**inputs)
...         logits = outputs.logits
...     return logits
Now, pass your input to the model and return the logits:
  Copied >>> logits = run_inference(trained_model, sample_test_video["video"])
Decoding the logits, we get:
  Copied >>> predicted_class_idx = logits.argmax(-1).item()
>>> print("Predicted class:", model.config.id2label[predicted_class_idx])
# Predicted class: BasketballDun


Visual Question Answering

Visual Question Answering (VQA) is the task of answering open-ended questions based on an image.
The input to models supporting this task is typically a combination of an image and a question, and the output is an
answer expressed in natural language.
Some noteworthy use case examples for VQA include:
Accessibility applications for visually impaired individuals. Education: posing questions about visual materials presented in lectures or textbooks. VQA can also be utilized in interactive museum exhibits or historical sites.


Visual Question Answering

 Customer service and e-commerce: VQA can enhance user experience by letting users ask questions about products. Image retrieval: VQA models can be used to retrieve images with specific characteristics. For example, the user can ask “Is there a dog?” to find all images with dogs from a set of images.
In this guide you’ll learn how to:
Fine-tune a classification VQA model, specifically ViLT, on the Graphcore/vqa dataset. Use your fine-tuned ViLT for inference. Run zero-shot VQA inference with a generative model, like BLIP-2


Fine-tuning ViLT

ViLT model incorporates text embeddings into a Vision Transformer (ViT), allowing it to have a minimal design for
Vision-and-Language Pre-training (VLP). This model can be used for several downstream tasks. For the VQA task, a classifier
head is placed on top (a linear layer on top of the final hidden state of the [CLS] token) and randomly initialized.
Visual Question Answering is thus treated as a classification problem.
More recent models, such as BLIP, BLIP-2, and InstructBLIP, treat VQA as a generative task. Later in this gui


Fine-tuning ViLT

de we
illustrate how to use them for zero-shot VQA inference.
Before you begin, make sure you have all the necessary libraries installed.
  Copied pip install -q transformers datasets
We encourage you to share your model with the community. Log in to your Hugging Face account to upload it to the 🤗 Hub.
When prompted, enter your token to log in:
  Copied >>> from huggingface_hub import notebook_login
>>> notebook_login()
Let’s define the model checkpoint as a global variable.
  Copied >>> model_checkpoint = "dandelin/vilt-b32-mlm"


Load the data

For illustration purposes, in this guide we use a very small sample of the annotated visual question answering Graphcore/vqa dataset.
You can find the full dataset on 🤗 Hub.
As an alternative to the Graphcore/vqa dataset, you can download the
same data manually from the official VQA dataset page. If you prefer to follow the
tutorial with your custom data, check out how to Create an image dataset
guide in the 🤗 Datasets documentation.
Let’s load the first 200 examples from the validation split and explore the dataset’s features:
  Copied >>> from datasets import load_dataset
>>> dataset = load_dataset("Graphcore/vqa", split="validation[:200]")
>>> dataset
Dataset({
    features: ['question', 'question_type', 'question_id', 'image_id', 'answer_type', 'label'],
    num_rows: 200
})
Let’s take a look at an example to understand the dataset’s features:
  Copied >>> dataset[0]
{'question': 'Where is he looking?',
 'question_type': 'none of the above',
 'qu


Load the data

estion_id': 262148000,
 'image_id': '/root/.cache/huggingface/datasets/downloads/extracted/ca733e0e000fb2d7a09fbcc94dbfe7b5a30750681d0e965f8e0a23b1c2f98c75/val2014/COCO_val2014_000000262148.jpg',
 'answer_type': 'other',
 'label': {'ids': ['at table', 'down', 'skateboard', 'table'],
  'weights': [0.30000001192092896,
   1.0,
   0.30000001192092896,
   0.30000001192092896]}}
The features relevant to the task include:
question: the question to be answered from the image image_id: the path to the image the question refers to label: the annotations
We can remove the rest of the features as they won’t be necessary:
  Copied >>> dataset = dataset.remove_columns(['question_type', 'question_id', 'answer_type'])
As you can see, the label feature contains several answers to the same question (called ids here) collected by different human annotators.
This is because the answer to a question can be subjective. In this case, the question is “where is he looking?“


Load the data

. Some people
annotated this with “down”, others with “at table”, another one with “skateboard”, etc.
Take a look at the image and consider which answer would you give:
  Copied >>> from PIL import Image
>>> image = Image.open(dataset[0]['image_id'])
>>> image
Due to the questions’ and answers’ ambiguity, datasets like this are treated as a multi-label classification problem (as
multiple answers are possibly valid). Moreover, rather than just creating a one-hot encoded vector, one creates a
soft encoding, based on the number of times a certain answer appeared in the annotations.
For instance, in the example above, because the answer “down” is selected way more often than other answers, it has a
score (called weight in the dataset) of 1.0, and the rest of the answers have scores < 1.0.
To later instantiate the model with an appropriate classification head, let’s create two dictionaries: one that maps
the label name to an integer and vice versa:
  Copi


Load the data

ed >>> import itertools
>>> labels = [item['ids'] for item in dataset['label']]
>>> flattened_labels = list(itertools.chain(*labels))
>>> unique_labels = list(set(flattened_labels))
>>> label2id = {label: idx for idx, label in enumerate(unique_labels)}
>>> id2label = {idx: label for label, idx in label2id.items()}
Now that we have the mappings, we can replace the string answers with their ids, and flatten the dataset for a more convenient further preprocessing.
  Copied >>> def replace_ids(inputs):
...   inputs["label"]["ids"] = [label2id[x] for x in inputs["label"]["ids"]]
...   return inputs
>>> dataset = dataset.map(replace_ids)
>>> flat_dataset = dataset.flatten()
>>> flat_dataset.features
{'question': Value(dtype='string', id=None),
 'image_id': Value(dtype='string', id=None),
 'label.ids': Sequence(feature=Value(dtype='int64', id=None), length=-1, id=None),
 'label.weights': Sequence(feature=Value(dtype='float64', id=None), length=-1, id=None)


Preprocessing data

The next step is to load a ViLT processor to prepare the image and text data for the model.
ViltProcessor wraps a BERT tokenizer and ViLT image processor into a convenient single processor:
  Copied >>> from transformers import ViltProcessor
>>> processor = ViltProcessor.from_pretrained(model_checkpoint)
To preprocess the data we need to encode the images and questions using the ViltProcessor. The processor will use
the BertTokenizerFast to tokenize the text and create input_ids, attention_mask and token_type_ids for the text data.
As for images, the processor will leverage ViltImageProcessor to resize and normalize the image, and create pixel_values and pixel_mask.
All these preprocessing steps are done under the hood, we only need to call the processor. However, we still need to
prepare the target labels. In this representation, each element corresponds to a possible answer


Preprocessing data

(label). For correct answers, the element holds
their respective score (weight), while the remaining elements are set to zero.
The following function applies the processor to the images and questions and formats the labels as described above:
  Copied >>> import torch
>>> def preprocess_data(examples):
...     image_paths = examples['image_id']
...     images = [Image.open(image_path) for image_path in image_paths]
...     texts = examples['question']
...     encoding = processor(images, texts, padding="max_length", truncation=True, return_tensors="pt")
...     for k, v in encoding.items():
...           encoding[k] = v.squeeze()

...     targets = []
...     for labels, scores in zip(examples['label.ids'], examples['label.weights']):
...         target = torch.zeros(len(id2label))
...         for label, score in zip(labels, scores):
...             target[label] = sco


Preprocessing data

re

...         targets.append(target)
...     encoding["labels"] = targets

...     return encoding
To apply the preprocessing function over the entire dataset, use 🤗 Datasets map function. You can speed up map by
setting batched=True to process multiple elements of the dataset at once. At this point, feel free to remove the columns you don’t need.
  Copied >>> processed_dataset = flat_dataset.map(preprocess_data, batched=True, remove_columns=['question','question_type',  'question_id', 'image_id', 'answer_type', 'label.ids', 'label.weights'])
>>> processed_dataset
Dataset({
    features: ['input_ids', 'token_type_ids', 'attention_mask', 'pixel_values', 'pixel_mask', 'labels'],
    num_rows: 200
})
As a final step, create a batch of examples using DefaultDataCollator:
  Copied >>> from transformers import DefaultDataCollator
>>> data_collator = DefaultDataCollator()


Train the model

You’re ready to start training your model now! Load ViLT with ViltForQuestionAnswering. Specify the number of labels
along with the label mappings:
  Copied >>> from transformers import ViltForQuestionAnswering
>>> model = ViltForQuestionAnswering.from_pretrained(model_checkpoint, num_labels=len(id2label), id2label=id2label, label2id=label2id)
At this point, only three steps remain:
Define your training hyperparameters in TrainingArguments:
  Copied >>> from transformers import TrainingArguments
>>> repo_id = "MariaK/vilt_finetuned_200"
>>> training_args = TrainingArguments(
...     output_dir=repo_id,
...     per_device_train_batch_size=4,
...     num_train_epochs=20,
...     save_steps=200,
...


Train the model

     logging_steps=50,
...     learning_rate=5e-5,
...     save_total_limit=2,
...     remove_unused_columns=False,
...     push_to_hub=True,
... )
Pass the training arguments to Trainer along with the model, dataset, processor, and data collator.
  Copied >>> from transformers import Trainer
>>> trainer = Trainer(
...     model=model,
...     args=training_args,
...     data_collator=data_collator,
...     train_dataset=processed_dataset,
...     tokenizer=processor,
... )
Call train() to finetune your model.
  Copied >>> trainer.train()
Once training is completed, share your model to the Hub with the push_to_hub() method to share your final model on the 🤗 Hub:
  Copied >>> trainer.push_to_hub(


Inference

Now that you have fine-tuned a ViLT model, and uploaded it to the 🤗 Hub, you can use it for inference. The simplest
way to try out your fine-tuned model for inference is to use it in a Pipeline.
  Copied >>> from transformers import pipeline
>>> pipe = pipeline("visual-question-answering", model="MariaK/vilt_finetuned_200")
The model in this guide has only been trained on 200 examples, so don’t expect a lot from it. Let’s see if it at least
learned something from the data and take the first example from the dataset to illustrate inference:
  Copied >>> example = dataset[0]
>>> image = Image.open(example['image_id'])
>>> question = example['question']
>>> print(question)
>>> pipe(image, question, top_k=1)
"Where is he looking?"
[{'score': 0.5498199462890625, 'answer': 'down'}]
Even though not very confident, the model indeed has learned something. With more examples and longer training,


Inference

you’ll get far better results!
You can also manually replicate the results of the pipeline if you’d like:
Take an image and a question, prepare them for the model using the processor from your model. Forward the result or preprocessing through the model. From the logits, get the most likely answer’s id, and find the actual answer in the id2label.
  Copied >>> processor = ViltProcessor.from_pretrained("MariaK/vilt_finetuned_200")
>>> image = Image.open(example['image_id'])
>>> question = example['question']
>>> # prepare inputs
>>> inputs = processor(image, question, return_tensors="pt")
>>> model = ViltForQuestionAnswering.from_pretrained("MariaK/vilt_finetuned_200")
>>> # forward pass
>>> with torch.no_grad():
...     outputs = model(**inputs)
>>> logits = outputs.logits
>>> idx = logits.argmax(-1).item()
>>> print("Predicted answer:", model.config.id2label[idx])
Predicted answer: down


Zero-shot VQA

The previous model treated VQA as a classification task. Some recent models, such as BLIP, BLIP-2, and InstructBLIP approach
VQA as a generative task. Let’s take BLIP-2 as an example. It introduced a new visual-language pre-training
paradigm in which any combination of pre-trained vision encoder and LLM can be used (learn more in the BLIP-2 blog post).
This enables achieving state-of-the-art results on multiple visual-language tasks including visual question answering.
Let’s illustrate how you can use this model for VQA. First, let’s load the model. Here we’ll explicitly send the model to a
GPU, if available, which we didn’t need to do earlier when training, as Trainer handles this automatically:
  Copied >>> from transformers import A


Zero-shot VQA

utoProcessor, Blip2ForConditionalGeneration
>>> import torch
>>> processor = AutoProcessor.from_pretrained("Salesforce/blip2-opt-2.7b")
>>> model = Blip2ForConditionalGeneration.from_pretrained("Salesforce/blip2-opt-2.7b", torch_dtype=torch.float16)
>>> device = "cuda" if torch.cuda.is_available() else "cpu"
>>> model.to(device)
The model takes image and text as input, so let’s use the exact same image/question pair from the first example in the VQA dataset:
  Copied >>> example = dataset[0]
>>> image = Image.open(example['image_id'])
>>> question = example['question']
To use BLIP-2 for visual question answering task, the textual prompt has to follow a specific format: Question: {} Answer:.
  Copied >>> prompt = f"Question: {question}


Zero-shot VQA

Answer:"
Now we need to preprocess the image/prompt with the model’s processor, pass the processed input through the model, and decode the output:
  Copied >>> inputs = processor(image, text=prompt, return_tensors="pt").to(device, torch.float16)
>>> generated_ids = model.generate(**inputs, max_new_tokens=10)
>>> generated_text = processor.batch_decode(generated_ids, skip_special_tokens=True)[0].strip()
>>> print(generated_text)
"He is looking at the crowd"
As you can see, the model recognized the crowd, and the direction of the face (looking down), however, it seems to miss
the fact the crowd is behind the skater. Still, in cases where acquiring human-annotated datasets is not feasible, this
approach can quickly produce useful result


Semantic segmentation

Semantic segmentation assigns a label or class to each individual pixel of an image. There are several types of segmentation, and in the case of semantic segmentation, no distinction is made between unique instances of the same object. Both objects are given the same label (for example, “car” instead of “car-1” and “car-2”). Common real-world applications of semantic segmentation include training self-driving cars to identify pedestrians and important traffic information, identifying cells and abnormalities in medical imagery, and monitoring environmental changes from satellite imagery.
This guide will show you how to:


Semantic segmentation

Finetune SegFormer on the SceneParse150 dataset. Use your finetuned model for inference.
The task illustrated in this tutorial is supported by the following model architectures:
 BEiT, Data2VecVision, DPT, MobileNetV2, MobileViT, MobileViTV2, SegFormer, UPerNet
Before you begin, make sure you have all the necessary libraries installed:
  Copied pip install -q datasets transformers evaluate
We encourage you to log in to your Hugging Face account so you can upload and share your model with the community. When prompted, enter your token to log in:
  Copied >>> from huggingface_hub import notebook_login
>>> notebook_login(


Load SceneParse150 dataset

Start by loading a smaller subset of the SceneParse150 dataset from the 🤗 Datasets library. This’ll give you a chance to experiment and make sure everything works before spending more time training on the full dataset.
  Copied >>> from datasets import load_dataset
>>> ds = load_dataset("scene_parse_150", split="train[:50]")
Split the dataset’s train split into a train and test set with the train_test_split method:
  Copied >>> ds = ds.train_test_split(test_size=0.2)
>>> train_ds = ds["train"]
>>> test_ds = ds["test"]
Then take a look at an example:
  Copied >>> train_ds[0]
{'image': <PIL.JpegImagePlugin.JpegImageFile image mode=RGB size=512x683 at 0x7F9B0C201F90>,
 'annotation': <PIL.PngImagePlugin.PngImageFile image mode=L size=512x683 at 0x7F9B0C201DD0>,
 'scene_category': 368}
image: a PIL image of the scene. annotation: a PIL image of


Load SceneParse150 dataset

the segmentation map, which is also the model’s target. scene_category: a category id that describes the image scene like “kitchen” or “office”. In this guide, you’ll only need image and annotation, both of which are PIL images.
You’ll also want to create a dictionary that maps a label id to a label class which will be useful when you set up the model later. Download the mappings from the Hub and create the id2label and label2id dictionaries:
  Copied >>> import json
>>> from huggingface_hub import cached_download, hf_hub_url
>>> repo_id = "huggingface/label-files"
>>> filename = "ade20k-id2label.json"
>>> id2label = json.load(open(cached_download(hf_hub_url(repo_id, filename, repo_type="dataset")), "r"))
>>> id2label = {int(k): v for k, v in id2label.items()}
>>> label2id = {v: k for k, v in id2label.items()}
>>> num_labels = len(id2label)


Preprocess

The next step is to load a SegFormer image processor to prepare the images and annotations for the model. Some datasets, like this one, use the zero-index as the background class. However, the background class isn’t actually included in the 150 classes, so you’ll need to set reduce_labels=True to subtract one from all the labels. The zero-index is replaced by 255 so it’s ignored by SegFormer’s loss function:
  Copied >>> from transformers import AutoImageProcessor
>>> checkpoint = "nvidia/mit-b0"
>>> image_processor = AutoImageProcessor.from_pretrained(checkpoint, reduce_labels=True)
  Pytorch  Hide Pytorch content  It is common to apply some data augmentations to an image dataset to make a model more robust against overfitting. In this guide, you’ll use the ColorJitter function from torchvision to randomly change the color properties of an image, but you can also use any


Preprocess

 image library you like.   Copied >>> from torchvision.transforms import ColorJitter
>>> jitter = ColorJitter(brightness=0.25, contrast=0.25, saturation=0.25, hue=0.1) Now create two preprocessing functions to prepare the images and annotations for the model. These functions convert the images into pixel_values and annotations to labels. For the training set, jitter is applied before providing the images to the image processor. For the test set, the image processor crops and normalizes the images, and only crops the labels because no data augmentation is applied during testing.   Copied >>> def train_transforms(example_batch):
...     images = [jitter(x) for x in example_batch["image"]]
...     labels = [x for x in example_batch["annotation"]]
...     inputs = image_processor(images, labels)
...     return inputs
>>> def val_transforms(example_batch):
...     images = [x


Preprocess

for x in example_batch["image"]]
...     labels = [x for x in example_batch["annotation"]]
...     inputs = image_processor(images, labels)
...     return inputs To apply the jitter over the entire dataset, use the 🤗 Datasets set_transform function. The transform is applied on the fly which is faster and consumes less disk space:   Copied >>> train_ds.set_transform(train_transforms)
>>> test_ds.set_transform(val_transforms)
  TensorFlow  Hide TensorFlow content  It is common to apply some data augmentations to an image dataset to make a model more robust against overfitting.
In this guide, you’ll use tf.image to randomly change the color properties of an image, but you can also use any image
library you like.
Define two separate transformation functions: training data transformations that include image augmentation validation data transformations that only transpose the


Preprocess

images, since computer vision models in 🤗 Transformers expect channels-first layout   Copied >>> import tensorflow as tf
>>> def aug_transforms(image):
...     image = tf.keras.utils.img_to_array(image)
...     image = tf.image.random_brightness(image, 0.25)
...     image = tf.image.random_contrast(image, 0.5, 2.0)
...     image = tf.image.random_saturation(image, 0.75, 1.25)
...     image = tf.image.random_hue(image, 0.1)
...     image = tf.transpose(image, (2, 0, 1))
...     return image
>>> def transforms(image):
...     image = tf.keras.utils.img_to_array(image)
...     image = tf.transpose(image, (2, 0, 1))
...     return image Next, create two preprocessing functions to prepare batches of images and annotations for the model. These functions apply
the image transformations and use the earlier loaded image_processor to convert the images into pixel_values and
annotat


Preprocess

ions to labels. ImageProcessor also takes care of resizing and normalizing the images.   Copied >>> def train_transforms(example_batch):
...     images = [aug_transforms(x.convert("RGB")) for x in example_batch["image"]]
...     labels = [x for x in example_batch["annotation"]]
...     inputs = image_processor(images, labels)
...     return inputs
>>> def val_transforms(example_batch):
...     images = [transforms(x.convert("RGB")) for x in example_batch["image"]]
...     labels = [x for x in example_batch["annotation"]]
...     inputs = image_processor(images, labels)
...     return inputs To apply the preprocessing transformations over the entire dataset, use the 🤗 Datasets set_transform function.
The transform is applied on the fly which is faster and consumes less disk space:   Copied >>> train_ds.set_transform(train_transforms)
>>> test_ds.set_transform(val_transform


Evaluate

Including a metric during training is often helpful for evaluating your model’s performance. You can quickly load an evaluation method with the 🤗 Evaluate library. For this task, load the mean Intersection over Union (IoU) metric (see the 🤗 Evaluate quick tour to learn more about how to load and compute a metric):
  Copied >>> import evaluate
>>> metric = evaluate.load("mean_iou")
Then create a function to compute the metrics. Your predictions need to be converted to
logits first, and then reshaped to match the size of the labels before you can call compute:
  Pytorch  Hide Pytorch content    Copied >>> import numpy as np
>>> import torch
>>> from torch import nn
>>> def compute_metrics(eval_pred):
...     with torch.no_grad():
...         logits, labels = eval_pred
...         logits_tensor = torch.from_numpy(logits)
...         logits_tensor = nn.functional.interp


Evaluate

olate(
...             logits_tensor,
...             size=labels.shape[-2:],
...             mode="bilinear",
...             align_corners=False,
...         ).argmax(dim=1)
...         pred_labels = logits_tensor.detach().cpu().numpy()
...         metrics = metric.compute(
...             predictions=pred_labels,
...             references=labels,
...             num_labels=num_labels,
...             ignore_index=255,
...             reduce_labels=False,
...         )
...         for key, value in metrics.items():
...             if type(value) is np.ndarray:
...                 metrics[key] = value.tolist()
...         return metrics
  TensorFlow  Hide TensorFlow content    Copied >>> def compute_metrics(eval_pred):
...     logits, labels = eval_pred
...     logits = tf.transpose(logits, perm=[0, 2, 3, 1])
...     logits_resized = tf.image.resize(
...


Evaluate

logits,
...         size=tf.shape(labels)[1:],
...         method="bilinear",
...     )
...     pred_labels = tf.argmax(logits_resized, axis=-1)
...     metrics = metric.compute(
...         predictions=pred_labels,
...         references=labels,
...         num_labels=num_labels,
...         ignore_index=-1,
...         reduce_labels=image_processor.do_reduce_labels,
...     )
...     per_category_accuracy = metrics.pop("per_category_accuracy").tolist()
...     per_category_iou = metrics.pop("per_category_iou").tolist()
...     metrics.update({f"accuracy_{id2label[i]}": v for i, v in enumerate(per_category_accuracy)})
...     metrics.update({f"iou_{id2label[i]}": v for i, v in enumerate(per_category_iou)})
...     return {"val_" + k: v for k, v in metrics.items()}
Your compute_metrics function is ready to go now, and you’ll return to it when you setup your trainin


Train

Pytorch  Hide Pytorch content  If you aren’t familiar with finetuning a model with the Trainer, take a look at the basic tutorial here! You’re ready to start training your model now! Load SegFormer with AutoModelForSemanticSegmentation, and pass the model the mapping between label ids and label classes:   Copied >>> from transformers import AutoModelForSemanticSegmentation, TrainingArguments, Trainer
>>> model = AutoModelForSemanticSegmentation.from_pretrained(checkpoint, id2label=id2label, label2id=label2id) At this point, only three steps remain: Define your training hyperparameters in TrainingArguments. It is important you don’t remove unused columns because this’ll drop the image column. Without the image column, you can’t create pixel_values. Set remove_unused_columns=False to prevent this behavior! The only other required parameter is output_dir which specifies


Train

where to save your model. You’ll push this model to the Hub by setting push_to_hub=True (you need to be signed in to Hugging Face to upload your model). At the end of each epoch, the Trainer will evaluate the IoU metric and save the training checkpoint. Pass the training arguments to Trainer along with the model, dataset, tokenizer, data collator, and compute_metrics function. Call train() to finetune your model.   Copied >>> training_args = TrainingArguments(
...     output_dir="segformer-b0-scene-parse-150",
...     learning_rate=6e-5,
...     num_train_epochs=50,
...     per_device_train_batch_size=2,
...     per_device_eval_batch_size=2,
...     save_total_limit=3,
...     evaluation_strategy="steps",
...     save_strategy="steps",
...     save_steps=20,
...     eval_steps=20,
...     logging_steps=1,
...     eval_accumulation_steps=5,
...     remove_unused_column


Train

s=False,
...     push_to_hub=True,
... )
>>> trainer = Trainer(
...     model=model,
...     args=training_args,
...     train_dataset=train_ds,
...     eval_dataset=test_ds,
...     compute_metrics=compute_metrics,
... )
>>> trainer.train() Once training is completed, share your model to the Hub with the push_to_hub() method so everyone can use your model:   Copied >>> trainer.push_to_hub()
  TensorFlow  Hide TensorFlow content  If you are unfamiliar with fine-tuning a model with Keras, check out the basic tutorial first! To fine-tune a model in TensorFlow, follow these steps: Define the training hyperparameters, and set up an optimizer and a learning rate schedule. Instantiate a pretrained model. Convert a 🤗 Dataset to a tf.data.Dataset. Compile your model. Add callbacks to calculate metrics and upload your model to 🤗 Hub Use the fit() method to run the training. S


Train

tart by defining the hyperparameters, optimizer and learning rate schedule:   Copied >>> from transformers import create_optimizer
>>> batch_size = 2
>>> num_epochs = 50
>>> num_train_steps = len(train_ds) * num_epochs
>>> learning_rate = 6e-5
>>> weight_decay_rate = 0.01
>>> optimizer, lr_schedule = create_optimizer(
...     init_lr=learning_rate,
...     num_train_steps=num_train_steps,
...     weight_decay_rate=weight_decay_rate,
...     num_warmup_steps=0,
... ) Then, load SegFormer with TFAutoModelForSemanticSegmentation along with the label mappings, and compile it with the
optimizer. Note that Transformers models all have a default task-relevant loss function, so you don’t need to specify one unless you want to:   Copied >>> from transformers import TFAutoModelForSemanticSegmentation
>>> model = TFAutoModelForSemanticSegmentation.from_pretrained(
...     checkp


Train

oint,
...     id2label=id2label,
...     label2id=label2id,
... )
>>> model.compile(optimizer=optimizer)  # No loss argument! Convert your datasets to the tf.data.Dataset format using the to_tf_dataset and the DefaultDataCollator:   Copied >>> from transformers import DefaultDataCollator
>>> data_collator = DefaultDataCollator(return_tensors="tf")
>>> tf_train_dataset = train_ds.to_tf_dataset(
...     columns=["pixel_values", "label"],
...     shuffle=True,
...     batch_size=batch_size,
...     collate_fn=data_collator,
... )
>>> tf_eval_dataset = test_ds.to_tf_dataset(
...     columns=["pixel_values", "label"],
...     shuffle=True,
...     batch_size=batch_size,
...     collate_fn=data_collator,
... ) To compute the accuracy from the predictions and push your model to the 🤗 Hub, use Keras callbacks.
Pass your compute_metrics function to KerasMetricCallback,
and use


Train

 the PushToHubCallback to upload the model:   Copied >>> from transformers.keras_callbacks import KerasMetricCallback, PushToHubCallback
>>> metric_callback = KerasMetricCallback(
...     metric_fn=compute_metrics, eval_dataset=tf_eval_dataset, batch_size=batch_size, label_cols=["labels"]
... )
>>> push_to_hub_callback = PushToHubCallback(output_dir="scene_segmentation", tokenizer=image_processor)
>>> callbacks = [metric_callback, push_to_hub_callback] Finally, you are ready to train your model! Call fit() with your training and validation datasets, the number of epochs,
and your callbacks to fine-tune the model:   Copied >>> model.fit(
...     tf_train_dataset,
...     validation_data=tf_eval_dataset,
...     callbacks=callbacks,
...     epochs=num_epochs,
... ) Congratulations! You have fine-tuned your model and shared it on the 🤗 Hub. You can now use it for inferen


Inference

Great, now that you’ve finetuned a model, you can use it for inference!
Load an image for inference:
  Copied >>> image = ds[0]["image"]
>>> image
  Pytorch  Hide Pytorch content  The simplest way to try out your finetuned model for inference is to use it in a pipeline(). Instantiate a pipeline for image segmentation with your model, and pass your image to it:   Copied >>> from transformers import pipeline
>>> segmenter = pipeline("image-segmentation", model="my_awesome_seg_model")
>>> segmenter(image)
[{'score': None,
  'label': 'wall',
  'mask': <PIL.Image.Image image mode=L size=640x427 at 0x7FD5B2062690>},
 {'score': None,
  'label': 'sky',
  'mask': <PIL.Image.Image image mode=L size=640x427 at 0x7FD5B2062A50>},
 {'score': None,
  'label': 'floor',
  'mask': <PIL.Image.Image image mode=


Inference

L size=640x427 at 0x7FD5B2062B50>},
 {'score': None,
  'label': 'ceiling',
  'mask': <PIL.Image.Image image mode=L size=640x427 at 0x7FD5B2062A10>},
 {'score': None,
  'label': 'bed ',
  'mask': <PIL.Image.Image image mode=L size=640x427 at 0x7FD5B2062E90>},
 {'score': None,
  'label': 'windowpane',
  'mask': <PIL.Image.Image image mode=L size=640x427 at 0x7FD5B2062390>},
 {'score': None,
  'label': 'cabinet',
  'mask': <PIL.Image.Image image mode=L size=640x427 at 0x7FD5B2062550>},
 {'score': None,
  'label': 'chair',
  'mask': <PIL.Image.Image image mode=L size=640x427 at 0x7FD5B2062D90>},
 {'score': None,
  'label': 'armchair',
  'mask': <PIL.Image.Image image mode=L size=640x427 at 0x7FD5B2062E10>}] You can also manually replicate the results of the pipeline if you’d like. Process the im


Inference

age with an image processor and place the pixel_values on a GPU:   Copied >>> device = torch.device("cuda" if torch.cuda.is_available() else "cpu")  # use GPU if available, otherwise use a CPU
>>> encoding = image_processor(image, return_tensors="pt")
>>> pixel_values = encoding.pixel_values.to(device) Pass your input to the model and return the logits:   Copied >>> outputs = model(pixel_values=pixel_values)
>>> logits = outputs.logits.cpu() Next, rescale the logits to the original image size:   Copied >>> upsampled_logits = nn.functional.interpolate(
...     logits,
...     size=image.size[::-1],
...     mode="bilinear",
...     align_corners=False,
... )
>>> pred_seg = upsampled_logits.argmax(dim=1)[0]
  TensorFlow  Hide TensorFlow content  Load an image processor to preprocess the image


Inference

and return the input as TensorFlow tensors:   Copied >>> from transformers import AutoImageProcessor
>>> image_processor = AutoImageProcessor.from_pretrained("MariaK/scene_segmentation")
>>> inputs = image_processor(image, return_tensors="tf") Pass your input to the model and return the logits:   Copied >>> from transformers import TFAutoModelForSemanticSegmentation
>>> model = TFAutoModelForSemanticSegmentation.from_pretrained("MariaK/scene_segmentation")
>>> logits = model(**inputs).logits Next, rescale the logits to the original image size and apply argmax on the class dimension:   Copied >>> logits = tf.transpose(logits, [0, 2, 3, 1])
>>> upsampled_logits = tf.image.resize(
...     logits,
...     # We reverse the shape of `image` because `image.size` returns width and height.
...     im


Inference

age.size[::-1],
... )
>>> pred_seg = tf.math.argmax(upsampled_logits, axis=-1)[0]
To visualize the results, load the dataset color palette as ade_palette() that maps each class to their RGB values. Then you can combine and plot your image and the predicted segmentation map:
  Copied >>> import matplotlib.pyplot as plt
>>> import numpy as np
>>> color_seg = np.zeros((pred_seg.shape[0], pred_seg.shape[1], 3), dtype=np.uint8)
>>> palette = np.array(ade_palette())
>>> for label, color in enumerate(palette):
...     color_seg[pred_seg == label, :] = color
>>> color_seg = color_seg[..., ::-1]  # convert to BGR
>>> img = np.array(image) * 0.5 + color_seg * 0.5  # plot the image with the segmentation map
>>> img = img.astype(np.uint8)
>>> plt.figure(figsize=(15, 10))
>>> plt.imshow(img)
>>> plt.sho


Image captioning

Image captioning is the task of predicting a caption for a given image. Common real world applications of it include
aiding visually impaired people that can help them navigate through different situations. Therefore, image captioning
helps to improve content accessibility for people by describing images to them.
This guide will show you how to:
Fine-tune an image captioning model. Use the fine-tuned model for inference.
Before you begin, make sure you have all the necessary libraries installed:
  Copied pip install transformers datasets evaluate -q
pip install jiwer -q
We encourage you to log in to your Hugging Face account so you can upload and share your model with the community. When prompted, enter your token to log in:
  Copied from huggingface_hub import notebook_login
notebook_login()


Load the Pokémon BLIP captions dataset

Use the 🤗 Dataset library to load a dataset that consists of {image-caption} pairs. To create your own image captioning dataset
in PyTorch, you can follow this notebook.
  Copied from datasets import load_dataset
ds = load_dataset("lambdalabs/pokemon-blip-captions")
ds
  Copied DatasetDict({
    train: Dataset({
        features: ['image', 'text'],
        num_rows: 833
    })
})
The dataset has two features, image and text.
Many image captioning datasets contain multiple captions per image. In those cases, a common strategy is to randomly sample a caption amongst the available ones during training.
Split the dataset’s train split into a train and test set with the [~datasets.Dataset.train_test_split] method:
  Copied ds =


Load the Pokémon BLIP captions dataset

ds["train"].train_test_split(test_size=0.1)
train_ds = ds["train"]
test_ds = ds["test"]
Let’s visualize a couple of samples from the training set.
  Copied from textwrap import wrap
import matplotlib.pyplot as plt
import numpy as np
def plot_images(images, captions):
    plt.figure(figsize=(20, 20))
    for i in range(len(images)):
        ax = plt.subplot(1, len(images), i + 1)
        caption = captions[i]
        caption = "\n".join(wrap(caption, 12))
        plt.title(caption)
        plt.imshow(images[i])
        plt.axis("off")
sample_images_to_visualize = [np.array(train_ds[i]["image"]) for i in range(5)]
sample_captions = [train_ds[i]["text"] for i in range(5)]
plot_images(sample_images_to_visualize, sample_captions


Preprocess the dataset

Since the dataset has two modalities (image and text), the pre-processing pipeline will preprocess images and the captions.
To do so, load the processor class associated with the model you are about to fine-tune.
  Copied from transformers import AutoProcessor
checkpoint = "microsoft/git-base"
processor = AutoProcessor.from_pretrained(checkpoint)
The processor will internally pre-process the image (which includes resizing, and pixel scaling) and tokenize the caption.
  Copied def transforms(example_batch):
    images = [x for x in example_batch["image"]]
    captions = [x for x in example_batch["text"]]
    inputs = processor(images=images, text=captions, padding="max_length")
    inputs.update({"labels": inputs["input_ids"]})
    return inputs
train_ds.set_transform(transforms)
test_ds.set_transform(transforms)
With the dataset ready, you can now set up the model for fine-tuning.


Load a base model

Load the “microsoft/git-base” into a AutoModelForCausalLM object.
  Copied from transformers import AutoModelForCausalLM
model = AutoModelForCausalLM.from_pretrained(checkpoint)


Evaluate

Image captioning models are typically evaluated with the Rouge Score or Word Error Rate. For this guide, you will use the Word Error Rate (WER).
We use the 🤗 Evaluate library to do so. For potential limitations and other gotchas of the WER, refer to this guide.
  Copied from evaluate import load
import torch
wer = load("wer")
def compute_metrics(eval_pred):
    logits, labels = eval_pred
    predicted = logits.argmax(-1)
    decoded_labels = processor.batch_decode(labels, skip_special_tokens=True)
    decoded_predictions = processor.batch_decode(predicted, skip_special_tokens=True)
    wer_score = wer.compute(predictions=decoded_predictions, references=decoded_labels)
    return {"wer_score": wer_score}


Train!

Now, you are ready to start fine-tuning the model. You will use the 🤗 Trainer for this.
First, define the training arguments using TrainingArguments.
  Copied from transformers import TrainingArguments, Trainer
model_name = checkpoint.split("/")[1]
training_args = TrainingArguments(
    output_dir=f"{model_name}-pokemon",
    learning_rate=5e-5,
    num_train_epochs=50,
    fp16=True,
    per_device_train_batch_size=32,
    per_device_eval_batch_size=32,
    gradient_accumulation_steps=2,
    save_total_limit=3,
    evaluation_strategy="steps",
    eval_steps=50,
    save_strategy="steps",
    save_steps=50,
    logging_steps=50,
    rem


Train!

ove_unused_columns=False,
    push_to_hub=True,
    label_names=["labels"],
    load_best_model_at_end=True,
)
Then pass them along with the datasets and the model to 🤗 Trainer.
  Copied trainer = Trainer(
    model=model,
    args=training_args,
    train_dataset=train_ds,
    eval_dataset=test_ds,
    compute_metrics=compute_metrics,
)
To start training, simply call train() on the Trainer object.
  Copied trainer.train()
You should see the training loss drop smoothly as training progresses.
Once training is completed, share your model to the Hub with the push_to_hub() method so everyone can use your model:
  Copied trainer.push_to_hub(


Inference

Take a sample image from test_ds to test the model.
  Copied from PIL import Image
import requests
url = "https://huggingface.co/datasets/sayakpaul/sample-datasets/resolve/main/pokemon.png"
image = Image.open(requests.get(url, stream=True).raw)
image
  Copied device = "cuda" if torch.cuda.is_available() else "cpu"
inputs = processor(images=image, return_tensors="pt").to(device)
pixel_values = inputs.pixel_values
Call generate and decode the predictions.
  Copied generated_ids = model.generate(pixel_values=pixel_values, max_length=50)
generated_caption = processor.batch_decode(generated_ids, skip_special_tokens=True)[0]
print(generated_caption)
  Copied a drawing of a pink and blue pokemon
Looks like the fine-tuned model generated a pretty good caption!


Document Question Answering

Document Question Answering, also referred to as Document Visual Question Answering, is a task that involves providing
answers to questions posed about document images. The input to models supporting this task is typically a combination of an image and
a question, and the output is an answer expressed in natural language. These models utilize multiple modalities, including
text, the positions of words (bounding boxes), and the image itself.
This guide illustrates how to:
Fine-tune LayoutLMv2 on the DocVQA dataset. Use your fine-tuned model for inference.
The task illustrated in this tutorial is supported by the following model architectures: LayoutLM, LayoutLMv2, LayoutLMv3
LayoutLMv2 solves the document question-answering task by adding a question-answering head on top of the final hidden
states of the tokens, to predict the positions of the start and end tokens of the
answer. In other words, the problem is treated as extractive question


Document Question Answering

answering: given the context, extract which piece
of information answers the question. The context comes from the output of an OCR engine, here it is Google’s Tesseract.
Before you begin, make sure you have all the necessary libraries installed. LayoutLMv2 depends on detectron2, torchvision and tesseract.
  Copied pip install -q transformers datasets
  Copied pip install 'git+https://github.com/facebookresearch/detectron2.git'
pip install torchvision
  Copied sudo apt install tesseract-ocr
pip install -q pytesseract
Once you have installed all of the dependencies, restart your runtime.
We encourage you to share your model with the community. Log in to your Hugging Face account to upload it to the 🤗 Hub.
When prompted, enter your token to log in:
  Copied >>> from huggingface_hub import notebook_login
>>> notebook_login()
Let’s define some global variables.
  Copied >>> model_checkpoint = "microsoft/layoutlmv2-base-uncased"
>>> batch_size =