Update: timit_asr - make the dataset streamable

The TIMIT ASR dataset had two issues that was preventing it from being streamable:

1. it was missing a call to `open` before `pd.read_csv`
2. it was using `os.path.dirname` which is not supported for streaming

I made the dataset streamable by using `open` to load the CSV, and by adding the support for `os.path.dirname` in dataset scripts to stream data

You can now do
```python
from datasets import load_dataset

timit_asr = load_dataset("timit_asr", streaming=True)
print(next(iter(timit_asr["train"])))
```
prints:

```json
{"file": "zip://data/TRAIN/DR4/MMDM0/SI681.WAV::https://data.deepai.org/timit.zip",
"phonetic_detail": {"start": [0, 1960, 2466, 3480, 4000, 5960, 7480, 7880, 9400, 9960, 10680, 13480, 15680, 15880, 16920, 18297, 18882, 19480, 21723, 22516, 24040, 25190, 27080, 28160, 28560, 30120, 31832, 33240, 34640, 35968, 37720],
"utterance": ["h#", "w", "ix", "dcl", "s", "ah", "tcl", "ch", "ix", "n", "ae", "kcl", "t", "ix", "v", "r", "ix", "f", "y", "ux", "zh", "el", "bcl", "b", "iy", "y", "ux", "s", "f", "el", "h#"],
"stop": [1960, 2466, 3480, 4000, 5960, 7480, 7880, 9400, 9960, 10680, 13480, 15680, 15880, 16920, 18297, 18882, 19480, 21723, 22516, 24040, 25190, 27080, 28160, 28560, 30120, 31832, 33240, 34640, 35968, 37720, 39920]},
"sentence_type": "SI", "id": "SI681",
"speaker_id": "MMDM0",
"dialect_region": "DR4",
"text": "Would such an act of refusal be useful?",
"word_detail": {
    "start": [1960, 4000, 9400, 10680, 15880, 18297, 27080, 30120],
    "utterance": ["would", "such", "an", "act", "of", "refusal", "be", "useful"],
    "stop": [4000, 9400, 10680, 15880, 18297, 27080, 30120, 37720]
}}
```

cc @patrickvonplaten @vrindaprabhu


Fix IndexError by ignoring empty RecordBatch

We need to ignore the empty record batches for the interpolation search to work correctly when querying arrow tables

Close #2833

cc @SaulLu


IndexError when accessing first element of a Dataset if first RecordBatch is empty

The computation of the offsets of the underlying Table of a Dataset has some issues if the first RecordBatch is empty.

```python
from datasets import Dataset
import pyarrow as pa

pa_table = pa.Table.from_pydict({"a": [1]})
pa_table2 = pa.Table.from_pydict({"a": []}, schema=pa_table.schema)
ds_table = pa.concat_tables([pa_table2, pa_table])

dataset = Dataset(ds_table)

print([len(b) for b in dataset.data._batches])
# [0, 1]

print(dataset.data._offsets)
# [0 0 1] (should be [0, 1])

dataset[0]
```
raises
```python
---------------------------------------------------------------------------
IndexError                                Traceback (most recent call last)
/usr/local/lib/python3.7/dist-packages/datasets/table.py in _interpolation_search(arr, x)
     90         else:
     91             i, j = i, k
---> 92     raise IndexError(f"Invalid query '{x}' for size {arr[-1] if len(arr) else 'none'}.")
     93
     94

IndexError: Invalid query '0' for size 1.
```

This can be fixed by ignoring empty batches when computing `table._batches` and `table._offsets`

cc @SaulLu


Logging levels not taken into account

bug

Something isn't working

## Describe the bug

The `logging` module isn't working as intended relative to the levels to set.

## Steps to reproduce the bug

```python
from datasets import logging

logging.set_verbosity_debug()
logger = logging.get_logger()

logger.error("ERROR")
logger.warning("WARNING")
logger.info("INFO")
logger.debug("DEBUG"
```

## Expected results

I expect all logs to be output since I'm putting a `debug` level.

## Actual results

Only the two first logs are output.

## Environment info

- `datasets` version: 1.11.0
- Platform: Linux-5.13.9-arch1-1-x86_64-with-glibc2.33
- Python version: 3.9.6
- PyArrow version: 5.0.0

## To go further

This logging issue appears in `datasets` but not in `transformers`. It happens because there is no handler defined for the logger. When no handler is defined, the `logging` library will output a one-off error to stderr, using a `StderrHandler` with level `WARNING`.

`transformers` sets a default `StreamHandler` [here](https://github.com/huggingface/transformers/blob/5c6eca71a983bae2589eed01e5c04fcf88ba5690/src/transformers/utils/logging.py#L86)


ArrowInvalid when mapping dataset with missing values

Something isn't working

## Describe the bug
I encountered an `ArrowInvalid` when mapping dataset with missing values.
Here are the files for a minimal example. The exception is only thrown when the first line in the csv has a missing value (if you move the last line to the top it isn't thrown).
[data_small.csv](https://github.com/huggingface/datasets/files/7037838/data_small.csv)
[data.csv](https://github.com/huggingface/datasets/files/7037842/data.csv)

## Steps to reproduce the bug
```python
from datasets import load_dataset

datasets = load_dataset("csv", data_files=['data_small.csv'])

datasets = datasets.map(lambda e: {'labels': e['match']},
                        remove_columns=['id'])
```

## Expected results
No error

## Actual results
```
File "pyarrow/error.pxi", line 84, in pyarrow.lib.check_status
pyarrow.lib.ArrowInvalid: Invalid null value
```

## Environment info
- `datasets` version: 1.5.0
- Platform: Linux-5.11.0-25-generic-x86_64-with-glibc2.29
- Python version: 3.8.10
- PyTorch version (GPU?): 1.7.1+cpu (False)
- Tensorflow version (GPU?): 2.4.1 (False)
- Using GPU in script?: no
- Using distributed or parallel set-up in script?: no


Add imagefolder dataset

A generic imagefolder dataset inspired by `torchvision.datasets.ImageFolder`.

Resolves #2508

---

Example Usage:

[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/gist/nateraw/954fa8cba4ff806f6147a782fa9efd1a/imagefolder-official-example.ipynb)


Optimize streaming from TAR archives

enhancement

New feature or request

Hi ! As you know TAR has some constraints for data streaming. While it is optimized for buffering, the files in the TAR archive **need to be streamed in order**. It means that we can't choose which file to stream from, and this notation is to be avoided for TAR archives:
```
tar://books_large_p1.txt::https://storage.googleapis.com/huggingface-nlp/datasets/bookcorpus/bookcorpus.tar.bz2
```
Instead, I suggest we implement `iter_archive` for the `StreamingDownloadManager`.
The regular `DownloadManager` already has it.

Then we will have to update the json/txt/csv/etc. loaders to make them use `iter_archive` on TAR archives.

That's also what Tensorflow Datasets is doing in this case.
See this [dataset](https://github.com/tensorflow/datasets/blob/93895059c80a9e05805e8f32a2e310f66a23fc98/tensorflow_datasets/image_classification/flowers.py) for example.

Therefore instead of doing
```python
uncompressed = dl_manager.extract(tar_archive)
filename = "books_large_p1.txt"
with open(os.path.join(uncompressed, filename)) as f:
    for line in f:
        ...
```
we'll do
```python
for filename, f in dl_manager.iter_archive(tar_archive):
    for line in f:
        ...
```


Add code-mixed Kannada Hope speech dataset

## Adding a Dataset
- **Name:** *KanHope*
- **Description:** *A code-mixed English-Kannada dataset for Hope speech detection*
- **Paper:** *https://arxiv.org/abs/2108.04616*
- **Data:** *https://github.com/adeepH/KanHope/tree/main/dataset*
- **Motivation:** *The dataset is amongst the very few resources available for code-mixed low-resourced Dravidian languages of India*


add a text classification dataset


Add a Text Classification dataset: KanHope

dataset request

Requesting to add a new dataset

## Adding a Dataset
- **Name:** *KanHope*
- **Description:** *A code-mixed English-Kannada dataset for Hope speech detection*
- **Paper:** *https://arxiv.org/abs/2108.04616* (I am the author of the paper}
- **Author:** *[AdeepH](https://github.com/adeepH)*
- **Data:** *https://github.com/adeepH/KanHope/tree/main/dataset*
- **Motivation:** *The dataset is amongst the very few resources available for code-mixed Dravidian languages*

- I tried following the steps as per the instructions. However, could not resolve an error. Any help would be appreciated.

- The dataset card and the scripts for the dataset *https://github.com/adeepH/datasets/tree/multilingual-hope-speech/datasets/mhs_eval*

```
Using custom data configuration default
Downloading and preparing dataset bn_hate_speech/default (download: Unknown size, generated: Unknown size, post-processed: Unknown size, total: Unknown size) to /root/.cache/huggingface/datasets/bn_hate_speech/default/0.0.0/5f417ddc89777278abd29988f909f39495f0ec802090f7d8fa63b5bffb121762...
---------------------------------------------------------------------------
KeyError                                  Traceback (most recent call last)
<ipython-input-114-4a9cdb519e4c> in <module>()
      1 from datasets import load_dataset
      2
----> 3 data = load_dataset('/content/bn')

9 frames
/usr/local/lib/python3.7/dist-packages/datasets/load.py in load_dataset(path, name, data_dir, data_files, split, cache_dir, features, download_config, download_mode, ignore_verifications, keep_in_memory, save_infos, script_version, use_auth_token, task, streaming, **config_kwargs)
    850         ignore_verifications=ignore_verifications,
    851         try_from_hf_gcs=try_from_hf_gcs,
--> 852         use_auth_token=use_auth_token,
    853     )
    854

/usr/local/lib/python3.7/dist-packages/datasets/builder.py in download_and_prepare(self, download_config, download_mode, ignore_verifications, try_from_hf_gcs, dl_manager, base_path, use_auth_token, **download_and_prepare_kwargs)
    614                     if not downloaded_from_gcs:
    615                         self._download_and_prepare(
--> 616                             dl_manager=dl_manager, verify_infos=verify_infos, **download_and_prepare_kwargs
    617                         )
    618                     # Sync info

/usr/local/lib/python3.7/dist-packages/datasets/builder.py in _download_and_prepare(self, dl_manager, verify_infos, **prepare_split_kwargs)
    691             try:
    692                 # Prepare split will record examples associated to the split
--> 693                 self._prepare_split(split_generator, **prepare_split_kwargs)
    694             except OSError as e:
    695                 raise OSError(

/usr/local/lib/python3.7/dist-packages/datasets/builder.py in _prepare_split(self, split_generator)
   1107                     disable=bool(logging.get_verbosity() == logging.NOTSET),
   1108                 ):
-> 1109                     example = self.info.features.encode_example(record)
   1110                     writer.write(example, key)
   1111             finally:

/usr/local/lib/python3.7/dist-packages/datasets/features.py in encode_example(self, example)
   1015         """
   1016         example = cast_to_python_objects(example)
-> 1017         return encode_nested_example(self, example)
   1018
   1019     def encode_batch(self, batch):

/usr/local/lib/python3.7/dist-packages/datasets/features.py in encode_nested_example(schema, obj)
    863     if isinstance(schema, dict):
    864         return {
--> 865             k: encode_nested_example(sub_schema, sub_obj) for k, (sub_schema, sub_obj) in utils.zip_dict(schema, obj)
    866         }
    867     elif isinstance(schema, (list, tuple)):

/usr/local/lib/python3.7/dist-packages/datasets/features.py in <dictcomp>(.0)
    863     if isinstance(schema, dict):
    864         return {
--> 865             k: encode_nested_example(sub_schema, sub_obj) for k, (sub_schema, sub_obj) in utils.zip_dict(schema, obj)
    866         }
    867     elif isinstance(schema, (list, tuple)):

/usr/local/lib/python3.7/dist-packages/datasets/features.py in encode_nested_example(schema, obj)
    890     # ClassLabel will convert from string to int, TranslationVariableLanguages does some checks
    891     elif isinstance(schema, (ClassLabel, TranslationVariableLanguages, Value, _ArrayXD)):
--> 892         return schema.encode_example(obj)
    893     # Other object should be directly convertible to a native Arrow type (like Translation and Translation)
    894     return obj

/usr/local/lib/python3.7/dist-packages/datasets/features.py in encode_example(self, example_data)
    665         # If a string is given, convert to associated integer
    666         if isinstance(example_data, str):
--> 667             example_data = self.str2int(example_data)
    668
    669         # Allowing -1 to mean no label.

/usr/local/lib/python3.7/dist-packages/datasets/features.py in str2int(self, values)
    623                 if value not in self._str2int:
    624                     value = str(value).strip()
--> 625                 output.append(self._str2int[str(value)])
    626             else:
    627                 # No names provided, try to integerize

KeyError: ' '
```


The datasets.map function does not load cached dataset after moving python script

bug

Something isn't working

## Describe the bug
The datasets.map function caches the processed data to a certain directory. When the map function is called another time with totally the same parameters, the cached data are supposed to be reloaded instead of re-processing. However, it doesn't reuse cached data sometimes. I use the common data processing in different tasks, the datasets are processed again, the only difference is that I run them in different files.

## Steps to reproduce the bug
Just run the following codes in different .py files.
```python
if __name__ == '__main__':
    from datasets import load_dataset
    from transformers import AutoTokenizer
    raw_datasets = load_dataset("wikitext", "wikitext-2-raw-v1")

    tokenizer = AutoTokenizer.from_pretrained("bert-base-uncased")

    def tokenize_function(examples):
        return tokenizer(examples["text"], padding="max_length", truncation=True)

    tokenized_datasets = raw_datasets.map(tokenize_function, batched=True)
```

## Expected results
The map function should reload data in the second or any later runs.

## Actual results
The processing happens in each run.

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 1.8.0
- Platform: linux
- Python version: 3.7.6
- PyArrow version: 3.0.0

This is the first time I report a bug. If there is any problem or confusing description, please let me know 😄.


Fix defaults in cache_dir docstring in load.py

Fix defaults in the `cache_dir` docstring.


HF_DATASETS_CACHE variable in Windows

I can't seem to use a custom Cache directory in Windows. I have tried:

set HF_DATASETS_CACHE = "C:\Datasets"
set HF_DATASETS_CACHE = "C:/Datasets"
set HF_DATASETS_CACHE = "C:\\Datasets"
set HF_DATASETS_CACHE = "r'C:\Datasets'"
set HF_DATASETS_CACHE = "\Datasets"
set HF_DATASETS_CACHE = "/Datasets"

In each instance I get the "[WinError 123] The filename, directory name, or volume label syntax is incorrect" error when attempting to load a dataset


Add url prefix convention for many compression formats

## Intro

When doing dataset streaming, the uncompression of compressed files is done on the fly using `fsspec`.

In particular, the download manager method `download_and_extract` doesn't return a path to the local download and extracted file, but instead a chained URL so that the uncompression can be done when the file is opened. A few examples of chained URLS:
- `gz://file.txt::https://foo.bar/file.txt.gz`
- `bz2://file.txt::https://foo.bar/file.txt.bz2`
- `zip://::https://foo.bar/archive.zip`
- `tar://::https://foo.bar/archive.tar.gz` (the TAR uncompression includes gz, bz2 etc. uncompression in `fsspec`)

This syntax is highly inspired by the `fsspec` URL chaining syntax from https://filesystem-spec.readthedocs.io/en/latest/features.html#url-chaining

This url prefixing allows `open` to know what kind of uncompression to do in a dataset script when doing
```python
def _generate_examples(self, urlpath):
    with open(urlpath) as f:
        ....
```

## What it changes

This changes the previous behavior from https://github.com/huggingface/datasets/pull/2786 , in which `open` was trying to infer the compression automatically. Infering the compression made it impossible to know whether the user wanted `open` to return compressed data (as the default behavior of the buitin open), or the uncompressed data. By adding uncompression prefixes to the URL, `open` know directly if it has to uncompress or not, and also which protocol to use.

## Additional notes

This PR should close https://github.com/huggingface/datasets/issues/2813

It should also close this PR https://github.com/huggingface/datasets/pull/2811 since the oscar dataset script won't try to uncompress twice anymore

Note that I had to temporarily remove the support for passing tar and zip files to `data_files` for streaming to make it work, since it makes it ambiguous whether a zip file passed as `data_files` should be uncompressed or not. IMO we can make it work again by changing the syntax to make the glob explicit:
```python
load_dataset("json", data_files="zip://*.jsonl::https://foo.bar/archive.zip")
```
This is the exact same convention as fsspec and it removes all ambiguities

cc @albertvillanova @lewtun


Cannot load linnaeus dataset

Something isn't working

## Describe the bug
The [linnaeus](https://huggingface.co/datasets/linnaeus) dataset cannot be loaded. To reproduce:
```
from datasets import load_dataset

datasets = load_dataset("linnaeus")
```
This results in:
```
Downloading and preparing dataset linnaeus/linnaeus (download: 17.36 MiB, generated: 8.74 MiB, post-processed: Unknown size, total: 26.10 MiB) to /root/.cache/huggingface/datasets/linnaeus/linnaeus/1.0.0/2ff05dbc256108233262f596e09e322dbc3db067202de14286913607cd9cb704...
---------------------------------------------------------------------------
ConnectionError                           Traceback (most recent call last)
<ipython-input-4-7ef3a88f6276> in <module>()
      1 from datasets import load_dataset
      2
----> 3 datasets = load_dataset("linnaeus")

11 frames
/usr/local/lib/python3.7/dist-packages/datasets/utils/file_utils.py in get_from_cache(url, cache_dir, force_download, proxies, etag_timeout, resume_download, user_agent, local_files_only, use_etag, max_retries, use_auth_token)
    603             raise FileNotFoundError("Couldn't find file at {}".format(url))
    604         _raise_if_offline_mode_is_enabled(f"Tried to reach {url}")
--> 605         raise ConnectionError("Couldn't reach {}".format(url))
    606
    607     # Try a second time

ConnectionError: Couldn't reach https://drive.google.com/u/0/uc?id=1OletxmPYNkz2ltOr9pyT0b0iBtUWxslh&export=download/
```


Downloading “reddit” dataset keeps timing out.

Something isn't working

## Describe the bug
A clear and concise description of what the bug is.
Everytime I try and download the reddit dataset it times out before finishing and I have to try again.

There is some timeout error that I will post once it happens again.

## Steps to reproduce the bug
```python
from datasets import load_dataset

dataset = load_dataset("reddit", ignore_verifications=True, cache_dir="/Volumes/My Passport for Mac/og-chat-data")
```

## Expected results
A clear and concise description of the expected results.

I would expect the download to finish, or at least provide a parameter to extend the read timeout window.

## Actual results
Specify the actual results or traceback.

Shown below in error message.

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version:  1.11.0
- Platform: macOS
- Python version: 3.9.6 (conda env)
- PyArrow version: N/A


Added XL-Sum dataset

Added XL-Sum dataset published in ACL-IJCNLP 2021. (https://aclanthology.org/2021.findings-acl.413/). The default timeout values in `src/datasets/utils/file_utls.py` were increased to enable downloading from the original google drive links.


cannot load data from my loacal path

Something isn't working

## Describe the bug
I just want to directly load data from my local path,but find a bug.And I compare it with pandas to provide my local path is real.

here is my code
```python3
# print my local path
print(config.train_path)
# read data and print data length
tarin=pd.read_csv(config.train_path)
print(len(tarin))

# loading data by load_dataset
data = load_dataset('csv',data_files=config.train_path)

print(len(data))
```

## Steps to reproduce the bug
```python
C:\Users\wie\Documents\项目\文本分类\data\train.csv
7613
Traceback (most recent call last):
  File "c:/Users/wie/Documents/项目/文本分类/lib/DataPrecess.py", line 17, in <module>
    data = load_dataset('csv',data_files=config.train_path)
  File "C:\Users\wie\Miniconda3\lib\site-packages\datasets\load.py", line 830, in load_dataset
    **config_kwargs,
  File "C:\Users\wie\Miniconda3\lib\site-packages\datasets\load.py", line 710, in load_dataset_builder
    **config_kwargs,
  File "C:\Users\wie\Miniconda3\lib\site-packages\datasets\builder.py", line 271, in __init__
    **config_kwargs,
  File "C:\Users\wie\Miniconda3\lib\site-packages\datasets\builder.py", line 386, in _create_builder_config
    config_kwargs, custom_features=custom_features, use_auth_token=self.use_auth_token
  File "C:\Users\wie\Miniconda3\lib\site-packages\datasets\builder.py", line 156, in create_config_id
    raise ValueError("Please provide a valid `data_files` in `DatasetBuilder`")
ValueError: Please provide a valid `data_files` in `DatasetBuilder`
```

## Expected results
A clear and concise description of the expected results.

## Actual results
Specify the actual results or traceback.

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version:  1.11.0
- Platform: win10
- Python version: 3.7.9
- PyArrow version: 5.0.0


Rename The Pile subsets

After discussing with @yjernite we think it's better to have the subsets of The Pile explicitly have "the_pile" in their names.

I'm doing the changes for the subsets that @richarddwang added:
- [x] books3 -> the_pile_books3 https://github.com/huggingface/datasets/pull/2801
- [x] stack_exchange -> the_pile_stack_exchange https://github.com/huggingface/datasets/pull/2803
- [x] openwebtext2 -> the_pile_openwebtext2 https://github.com/huggingface/datasets/pull/2802

For consistency we should also rename `bookcorpusopen` to `the_pile_bookcorpus` IMO, but let me know what you think.
(we can just add a deprecation message to `bookcorpusopen` for now and add `the_pile_bookcorpus`)


Add Mostly Basic Python Problems Dataset

dataset request

Requesting to add a new dataset

## Adding a Dataset
- **Name:** Mostly Basic Python Problems Dataset
- **Description:** The benchmark consists of around 1,000 crowd-sourced Python programming problems, designed to be solvable by entry level programmers, covering programming fundamentals, standard library functionality, and so on. Each problem consists of a task description, code solution and 3 automated test cases.
- **Paper:** *link to the dataset paper if available*
- **Data:** https://github.com/google-research/google-research/tree/master/mbpp
- **Motivation:** Simple, small dataset related to coding problems.

Instructions to add a new dataset can be found [here](https://github.com/huggingface/datasets/blob/master/ADD_NEW_DATASET.md).


Tiny typo fixes of "fo" -> "of"

Noticed a few of these when reading docs- feel free to ignore the PR and just fix on some main contributor branch if more helpful. Thanks for the great library! :)


Bump tqdm version

The recently released tqdm 4.62.1 includes a fix for PermissionError on Windows (submitted by me in https://github.com/tqdm/tqdm/pull/1207), which means we can remove expensive `gc.collect` calls by bumping tqdm to that version. This PR does exactly that and, additionally, fixes a `disable_tqdm` definition that would previously, if used, raise a PermissionError on Windows.


Remove compression from xopen

generic discussion

Generic discussion on the library

We implemented support for streaming with 2 requirements:
- transparent use for the end user: just needs to pass the parameter `streaming=True`
- no additional work for the contributors: previous loading scripts should also work in streaming mode with no (or minor) changes; and new loading scripts should not involve additional code to support streaming

In order to fulfill these requirements, streaming implementation patched some Python functions:
- the `open(urlpath)` function was patched with `fsspec.open(urlpath)`
- the `os.path.join(urlpath, *others)` function was patched in order to add to `urlpath` hops (`::`) and extractor protocols (`zip://`), which are required by `fsspec.open`

Recently, we implemented support for streaming all archive+compression formats: zip, tar, gz, bz2, lz4, xz, zst; tar.gz, tar.bz2,...
Under the hood, the implementation:
- passes an additional parameter `compression` to `fsspec.open`, so that it performs the decompression on the fly: `fsspec.open(urlpath, compression=...)`

Some concerns have been raised about passing the parameter `compression` to `fsspec.open`:
- https://github.com/huggingface/datasets/pull/2786#discussion_r689550254
- #2811

The main argument is that if `open` decompresses the file and afterwards we call `gzip.open` on it, that will raise an error in `oscar` dataset:
```python
gzip.open(open(urlpath
```
While this is true:
- it is not natural/usual to call `open` inside `gzip.open` (never seen this before)
- indeed, this was recently (2 months ago) coded that way in `datasets` in order to allow streaming support (with previous implementation of streaming)

In this particular case, there is a natural fix solution: #2811:
- Revert the `open` inside the `gzip.open` (change done 2 months ago): `gzip.open(open(urlpath` => `gzip.open(urlpath`
- Patch `gzip.open(urlpath` with `fsspec.open(urlpath, compression="gzip"`

Are there other issues apart from this?

Note that there is an issue just because the open inside of the gzip.open. There is no issue in the other cases where datasets loading scripts use just
- `gzip.open`
- `open` (after having called dl_manager.download_and_extract)

TODO:
- [ ] Is this really an issue? Please enumerate the `datasets` loading scripts where this is problematic.
  - For the moment, there are only 3 datasets where we have an `open` inside a `gzip.open`:
    - oscar (since 23 June), mc4 (since 2 July) and c4 (since 2 July)
  - In the 3 datasets, the only reason to put an open inside a gzip.open was indeed to force supporting streaming
- [ ] If this is indeed an issue, which are the possible alternatives? Pros/cons?


arXiv Dataset verification problem

bug

Something isn't working

dataset bug

A bug in a dataset script provided in the library

## Describe the bug
`dataset_infos.json` for `arxiv_dataset` contains a fixed number of training examples, however the data (downloaded from an external source) is updated every week with additional examples.
Therefore, loading the dataset without `ignore_verifications=True` results in a verification error.


Fix stream oscar

Previously, an additional `open` was added to oscar to make it stream-compatible: 587bbb94e891b22863b312b99696e32708c379f4.

This was argued that might be problematic: https://github.com/huggingface/datasets/pull/2786#discussion_r690045921

This PR:
- removes that additional `open`
- patches `gzip.open` with `xopen` + `compression="gzip"`


Add WIT Dataset

Adds Google's [WIT](https://github.com/google-research-datasets/wit) dataset.


Add Beans Dataset

Adds the [beans](https://github.com/AI-Lab-Makerere/ibean/) image classification dataset.


Enable streaming for Wikipedia corpora

enhancement

New feature or request

**Is your feature request related to a problem? Please describe.**
Several of the [Wikipedia corpora](https://huggingface.co/datasets?search=wiki) on the Hub involve quite large files that would be a good candidate for streaming. Currently it is not possible to stream these corpora:

```python
from datasets import load_dataset

# Throws ValueError: Builder wikipedia is not streamable.
wiki_dataset_streamed = load_dataset("wikipedia", "20200501.en", split="train", streaming=True)
```

Given that these corpora are derived from Wikipedia dumps in XML format which are then processed with Apache Beam, I am not sure whether streaming is possible in principle. The goal of this issue is to discuss whether this feature even makes sense :)

**Describe the solution you'd like**
It would be nice to be able to stream Wikipedia corpora from the Hub with something like

```python
from datasets import load_dataset

wiki_dataset_streamed = load_dataset("wikipedia", "20200501.en", split="train", streaming=True)
```


Add cats_vs_dogs dataset

Adds Microsoft's [Cats vs. Dogs](https://www.microsoft.com/en-us/download/details.aspx?id=54765) dataset.


Fix streaming tar files from canonical datasets

Previous PR #2800 implemented support to stream remote tar files when passing the parameter `data_files`: they required a glob string `"*"`.

However, this glob string creates an error when streaming canonical datasets (with a `join` after the `open`).

This PR fixes this issue and allows streaming tar files both from:
- canonical datasets scripts and
- data files.

This PR also adds support for compressed tar files: `.tar.gz`, `.tar.bz2`,...


Fix streaming zip files from canonical datasets

Previous PR #2798 fixed streaming remote zip files when passing the parameter `data_files`.

However, that broke streaming zip files used in canonical `datasets` scripts, which normally have a subsequent `join()` (patched with `xjoin()`) after the `StreamingDownloadManager.download_and_extract()` is called.

This PR fixes this issue and allows streaming zip files both from:
- canonical datasets scripts and
- data files.


Add Food-101

Adds image classification dataset [Food-101](https://data.vision.ee.ethz.ch/cvl/datasets_extra/food-101/).


add stack exchange

stack exchange is part of EleutherAI/The Pile, but AFAIK, The Pile dataset blend all sub datasets together thus we are not able to use just one of its sub dataset from The Pile data. So I create an independent dataset using The Pile preliminary components.

I also change default `timeout` to 100 seconds instead of 10 seconds, otherwise I keep getting read time out when downloading source data of stack exchange and cc100 dataset.

When I was creating dataset card. I found there is room for creating / editing dataset card. I've made it an issue. #2797

Also I am wondering whether the import of The Pile dataset is actively undertaken (because I may need it recently)? #1675


add openwebtext2

openwebtext2 is part of EleutherAI/The Pile, but AFAIK, The Pile dataset blend all sub datasets together thus we are not able to use just one of its sub dataset from The Pile data. So I create an independent dataset using The Pile preliminary components.

When I was creating dataset card. I found there is room for creating / editing dataset card. I've made it an issue. #2797

Also I am wondering whether the import of The Pile dataset is actively undertaken (because I may need it recently)? #1675


add books3

books3 is part of EleutherAI/The Pile, but AFAIK, The Pile dataset blend all sub datasets together thus we are not able to use just one of its sub dataset from The Pile data. So I create an independent dataset using The Pile preliminary components.

When I was creating dataset card. I found there is room for creating / editing dataset card. I've made it an issue. #2797

Also I am wondering whether the import of The Pile dataset is actively undertaken (because I may need it recently)? #1675


Support streaming tar files

This PR adds support to stream tar files by using the `fsspec` tar protocol.

It also uses the custom `readline` implemented in PR #2786.

The corresponding test is implemented in PR #2786.