[![PyPI version](https://badge.fury.io/py/cloud-volume.svg)](https://badge.fury.io/py/cloud-volume) [![SfN 2018 Poster](https://img.shields.io/badge/poster-SfN%202018-blue.svg)](https://drive.google.com/open?id=1RKtaAGV2f7F13opnkQfbp6YBqmoD3fZi) [![codecov](https://img.shields.io/badge/codecov-link-%23d819a6)](https://codecov.io/gh/seung-lab/cloud-volume) [![DOI](https://zenodo.org/badge/98333149.svg)](https://zenodo.org/badge/latestdoi/98333149)

# CloudVolume: IO for Neuroglancer Datasets

```python
from cloudvolume import CloudVolume

vol = CloudVolume('gs://mylab/mouse/image', parallel=True, progress=True)
image = vol[:,:,:] # Download a whole image stack into a numpy array from the cloud
vol[:,:,:] = image # Upload an entire image stack from a numpy array to the cloud

label = 1
mesh = vol.mesh.get(label)
skel = vol.skeleton.get(label)
```

CloudVolume is a serverless Python client for random access reading and writing of [Neuroglancer](https://github.com/google/neuroglancer/) volumes in "[Precomputed](https://github.com/google/neuroglancer/tree/master/src/neuroglancer/datasource/precomputed)" format, a set of representations for arbitrarily large volumetric images, meshes, and skeletons. CloudVolume is typically paired with [Igneous](https://github.com/seung-lab/igneous), a Kubernetes compatible system for generating image hierarchies, meshes, skeletons, and other dependency free jobs that can be applied to petavoxel scale images.

Precomputed volumes are typically stored on [AWS S3](https://aws.amazon.com/s3/), [Google Storage](https://cloud.google.com/storage/), or locally. CloudVolume can read and write to these object storage providers given a service account token with appropriate permissions. However, these volumes can be stored on any service, including an ordinary webserver or local filesystem, that supports key-value access.

The combination of [Neuroglancer](https://github.com/google/neuroglancer/), [Igneous](https://github.com/seung-lab/igneous), and CloudVolume comprises a system for visualizing, processing, and sharing (via browser viewable URLs) petascale datasets within and between laboratories. A typical example usage would be to visualize raw electron microscope scans of mouse, fish, or fly brains up to a cubic millimeter in physical dimension. Neuroglancer and Igneous would enable you to visualize each step of the process of montaging the image, fine tuning alignment vector fields, creating segmentation layers, ROI masks, or performing other types of analysis. CloudVolume enables you to read from and write to each of these layers. Recently, we have introduced the ability to interact with the graph server ("PyChunkGraph") that backs proofreading automated segmentations via the `graphene://` format.

You can find a collection of CloudVolume accessible and Neuroglancer viewable datasets at https://neurodata.io/project/ocp/, an open data project by some of our collaborators.

## Highlights

- Random access to petavoxel Neuroglancer images, meshes, and skeletons.
- Nearly all output is immediately visualizable using Neuroglancer.\*
- Reads graph server backed proofreading volumes (via `graphene://`).
- Serverless (except `graphene://`) and multi-cloud.

### Detailed Highlights

- Multi-threaded, supports multi-process and green threads.
- Memory optimized, supports shared memory.
- Lossless connectomics relevant codecs ([`compressed_segmentation`](https://github.com/seung-lab/compressedseg), [`compresso`](https://github.com/seung-lab/compresso), [`crackle`](https://github.com/seung-lab/crackle) (BETA), [`fpzip`](https://github.com/seung-lab/fpzip/), [`zfpc`](https://github.com/seung-lab/zfpc), [`png`](https://en.wikipedia.org/wiki/Portable_Network_Graphics), and [`brotli`](https://en.wikipedia.org/wiki/Brotli))
- Understands image hierarchies & anisotropic pixel resolutions.
- Accomodates downloading missing tiles (`fill_missing=True`).
- Accomodates uploading compressed black tiles to erasure coded file systems (`delete_black_uploads=True`).
- Growing support for the Neuroglancer [sharded format](https://github.com/google/neuroglancer/tree/master/src/neuroglancer/datasource/precomputed) which dramatically condenses the number of files required to represent petascale datasets, similar to [Cloud Optimized GeoTIFF](https://www.cogeo.org/), which can result in [dramatic cost savings](https://github.com/seung-lab/kimimaro/wiki/The-Economics:-Skeletons-for-the-People).
- Reads Precomputed meshes and skeletons.
- Includes viewers for small images, meshes, and skeletons.
- Only 3 dimensions + RBG channels currently supported for images.
- No data versioning.

## Setup

Cloud-volume is regularly tested on Ubuntu with 3.7, 3.8, 3.9 and 3.10. We officially support Linux and Mac OS. Windows is community supported. After installation, you'll also need to set up your cloud credentials if you're planning on writing files or reading from a private dataset. Once you're finished setting up, you can try [reading from a public dataset](https://github.com/seung-lab/cloud-volume/wiki/Reading-Public-Data-Examples).

#### `pip` Binary Installation

```bash
pip install cloud-volume # standard installation
```

CloudVolume depends on several PyPI packages which are Cython bindings for C++. We have provided compiled binaries for many platforms and python versions, however if you are on an unsupported system, pip will attempt to install from source. In that case, follow the instructions below.

**Windows Note:** If you get errors related to a missing C++ compiler, this blog post might help you: https://www.scivision.dev/python-windows-visual-c-14-required/

#### Optional Dependencies

| Tag             | Description                             | Dependencies          |
|-----------------|-----------------------------------------|-----------------------|
| boss            | `boss://` format support                | intern                |
| test            | Supports testing                        | pytest                |
| mesh_viewer     | `mesh.viewer()` GUI                     | vtk                   |
| skeleton_viewer | `skeleton.viewer()` GUI                 | matplotlib            |
| all_viewers     | All viewers now and in the future.      | vtk, matplotlib       |
| dask            | Supports converting to/from dask arrays | dask\[array\]         |

Example:

```bash
pip install cloud-volume[boss,test,all_viewers]
```

#### `pip` Source Installation

*C++ compiler required.*

```bash
sudo apt-get install g++ python3-dev # python-dev if you're on python2
pip install numpy
pip install cloud-volume
```

Due to packaging problems endemic to Python, Cython packages that depend on numpy require numpy header files be installed before attempting to install the package you want. The numpy headers are not recognized unless numpy is installed in a seperate process that runs first. There are hacks for this issue, but I haven't gotten them to work. If you think binaries should be available for your platform, please let us know by opening an issue.

#### Manual Installation

This can be desirable if you want to hack on CloudVolume itself.

```bash
git clone git@github.com:seung-lab/cloud-volume.git
cd cloud-volume

# With virtualenvwrapper
mkvirtualenv cv
workon cv
# With only virtualenv
virtualenv venv
source venv/bin/activate

sudo apt-get install g++ python3-dev # python-dev if you're on python2
pip install numpy # additional step needed for accelerated compressed_segmentation and fpzip
pip install -e . # without optional dependencies
pip install -e .[all_viewers] # with e.g. the all_viewers optional dependency
```

### Credentials

By default, CloudVolume's configuration and cache files are stored in `$HOME/.cloudvolume` (or in `$HOME/.cloudfiles` since we use CloudFiles for the backend). You can configure where CloudVolume looks for these files with the environment variable `$CLOUD_VOLUME_DIR`.

Credentials are stored in `$CLOUD_VOLUME_DIR/secrets`. You'll need credentials only for the services you'll use. If you plan to use the local filesystem, you won't need any. For Google Storage ([setup instructions here](https://github.com/seung-lab/cloud-volume/wiki/Setting-up-Google-Cloud-Storage)), default account credentials will be used if available and no service account is provided.

If neither of those two conditions apply, you need a service account credential. If you have your credentials handy, you can provide them like so as a dict, JSON string, or a bare token if the service will accept that.

```python
cv = CloudVolume(..., secrets=...)
```

However, it may be simpler to save your credential to disk so you don't have to always provide it. `google-secret.json` is a service account credential for Google Storage, `aws-secret.json` is a service account for S3, etc. You can support multiple projects at once by prefixing the bucket you are planning to access to the credential filename. `google-secret.json` will be your defaut service account, but if you also want to also access bucket ABC, you can provide `ABC-google-secret.json` and you'll have simultaneous access to your ordinary buckets and ABC. The secondary credentials are accessed on the basis of the bucket name, not the project name.

```bash
mkdir -p ~/.cloudvolume/secrets/
mv aws-secret.json ~/.cloudvolume/secrets/ # needed for Amazon
mv google-secret.json ~/.cloudvolume/secrets/ # needed for Google
mv boss-secret.json ~/.cloudvolume/secrets/ # needed for the BOSS
mv matrix-secret.json ~/.cloudvolume/secrets/ # needed for Matrix
mv tigerdata-secret.json ~/.cloudvolume/secrets/ # needed for Tigerdata
```

#### `aws-secret.json` and `matrix-secret.json`

Create an [IAM user service account](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_users.html) that can read, write, and delete objects from at least one bucket.

```json
{
	"AWS_ACCESS_KEY_ID": "$MY_AWS_ACCESS_KEY_ID",
	"AWS_SECRET_ACCESS_KEY": "$MY_SECRET_ACCESS_TOKEN"
}
```

#### `google-secret.json`

You can create the `google-secret.json` file [here](https://console.cloud.google.com/iam-admin/serviceaccounts). You don't need to manually fill in JSON by hand, the below example is provided to show you what the end result should look like. You should be able to read, write, and delete objects from at least one bucket.

```json
{
  "type": "service_account",
  "project_id": "$YOUR_GOOGLE_PROJECT_ID",
  "private_key_id": "...",
  "private_key": "...",
  "client_email": "...",
  "client_id": "...",
  "auth_uri": "https://accounts.google.com/o/oauth2/auth",
  "token_uri": "https://accounts.google.com/o/oauth2/token",
  "auth_provider_x509_cert_url": "https://www.googleapis.com/oauth2/v1/certs",
  "client_x509_cert_url": ""
}
```

#### `cave-secret.json`

*Note: used to be called chunkedgraph-secret.json. This is still supported but deprecated.*

If you have a token from Graphene/Chunkedgraph server, create the `cave-secret.json` file as shown in the example below. You may also pass the token to `CloudVolume(..., secrets=token)`.

```json
{
  "token": "<your_token>"
}
```

Note that to take advantage of multiple credential files, prepend the fully qualified domain name (FQDN) of the server instead of the bucket for GCS and S3. For example, `sudomain.domain.com-cave-secret.json`.

## Usage

CloudVolume supports reading and writing to Neuroglancer data layers on Amazon S3, Google Storage, The BOSS, and the local file system.

Supported URLs are of the forms:

`$FORMAT://$PROTOCOL://$BUCKET/$DATASET/$LAYER`

The format or protocol fields may be omitted where required. In the case of the precomputed format, the format specifier is optional.

| Format      | Protocols                                    | Default | Example                                |
|-------------|----------------------------------------------|---------|----------------------------------------|
| precomputed | gs, s3, http, https, file, matrix, tigerdata | Yes     | gs://mybucket/dataset/layer            |
| graphene    | gs, s3, http, https, file, matrix, tigerdata |         | graphene://gs://mybucket/dataset/layer |
| boss        | N/A                                          |         | boss://collection/experiment/channel   |
| n5          | gs, s3, http, https, file, matrix, tigerdata |         | n5://gs://mybucket/dataset/layer       |

### Supported Formats

* precomputed: Neuroglancer's native format. ([specification](https://github.com/google/neuroglancer/tree/master/src/neuroglancer/datasource/precomputed))
* graphene: Precomputed based format used by the PyChunkGraph server.
* boss: The BOSS (https://docs.theboss.io/docs)
* n5: Not HDF5 (https://github.com/saalfeldlab/n5) Read-only support. Supports raw, gzip, bz2, and xz but not lz4 compression. mode 0 datasets only.

### Supported Protocols

* gs:   Google Storage
* s3:   Amazon S3
* http(s): (read-only) Ordinary Web Servers
* file: Local File System (absolute path)
* matrix: Princeton Internal System (run in large part by Seung Lab)
* tigerdata: Princeton Internal System (run by Princeton OIT)

CloudVolume also supports [alternative s3 aliases](https://github.com/seung-lab/cloud-files#alias-for-alternative-s3-endpoints) via CloudFiles.

### `info` Files - New Dataset

Neuroglancer relies on an [`info`](https://github.com/google/neuroglancer/tree/master/src/neuroglancer/datasource/precomputed#info-json-file-specification) file located at the root of a dataset layer to tell it how to compute file locations and interpret the data in each file. CloudVolume piggy-backs on this functionality.

In the below example, assume you are creating a new segmentation volume from a 3d numpy array "rawdata". Note Precomputed stores data in Fortran (column major, aka CZYX) order. You should do a small test to see if the image is written transposed. You can fix this by uploading `rawdata.T`. A more detailed example for uploading a local volume [is located here](https://github.com/seung-lab/cloud-volume/wiki/Example-Single-Machine-Dataset-Upload).

```python3
from cloudvolume import CloudVolume

info = CloudVolume.create_new_info(
    num_channels    = 1,
    layer_type      = 'segmentation',
    data_type       = 'uint64', # Channel images might be 'uint8'
    # raw, png, jpeg, compressed_segmentation, fpzip, kempressed, zfpc, compresso, crackle
    encoding        = 'raw',
    resolution      = [4, 4, 40], # Voxel scaling, units are in nanometers
    voxel_offset    = [0, 0, 0], # x,y,z offset in voxels from the origin
    mesh            = 'mesh',
    # Pick a convenient size for your underlying chunk representation
    # Powers of two are recommended, doesn't need to cover image exactly
    chunk_size      = [ 512, 512, 16 ], # units are voxels
    volume_size     = [ 250000, 250000, 25000 ], # e.g. a cubic millimeter dataset
)
vol = CloudVolume(cfg.path, info=info)
vol.commit_info()
vol[cfg.x: cfg.x + cfg.length, cfg.y:cfg.y + cfg.length, cfg.z: cfg.z + cfg.length] = rawdata[:,:,:]
```
| Encoding                | Image Type                 | Lossless | Neuroglancer Viewable | Description                                                                              |
|-------------------------|----------------------------|----------|-------------|------------------------------------------------------------------------------------------|
| raw                     | Any                        | Y        | Y           | Serialized numpy arrays.                                                                 |
| png                     | Image                      | Y        | Y           | Multiple slices stiched into a single PNG.                                               |
| jpeg                    | Image                      | N        | Y           | Multiple slices stiched into a single JPEG.                                              |
| compressed_segmentation | Segmentation               | Y        | Y           | Renumbered numpy arrays to reduce data width. Also used by Neuroglancer internally.      |
| compresso               | Segmentation               | Y        | Y           | Lossless high compression algorithm for connectomics segmentation.                       |
| crackle                 | Segmentation               | Y        | Y*           | Lossless high compression algorithm for connectomics segmentation.                       |
| fpzip                   | Floating Point             | Y        | Y*           | Takes advantage of IEEE 754 structure + L1 Lorenzo predictor to get higher compression.  |
| kempressed              | Anisotropic Z Floating Point | N**      | Y*           | Adds manipulations on top of fpzip to achieve higher compression.                        |
| zfpc                    | Alignment Vector Fields    | N***     | Y*          | zfp stream container.                        |

\* Not integrated into official Neuroglancer yet, but available on a [fork](https://allcodecs-dot-neuromancer-seung-import.appspot.com/) which can be seen on Github [here](https://github.com/william-silversmith/neuroglancer/tree/wms_combined_codecs).
\*\* Lossless if your data can handle adding and then subtracting 2.
\*\*\* Lossless by default, but you probably want to use the lossy mode.

Note on `compressed_segmentation`: To use, make sure `compressed_segmentation_block_size` is specified (usually `[8,8,8]`. This field will appear in the `info` file in the relevant scale.

Note on `zfpc`: To configure, use the fields `zfpc_rate`, `zfpc_precision`, `zfpc_tolerance`, `zfpc_correlated_dims` in the relevant scale of the `info` file.

### Examples

```python
# Basic Examples
vol = CloudVolume('gs://mybucket/retina/image')
vol = CloudVolume('gs://mybucket/retina/image', secrets=token, dict or json)
vol = CloudVolume('gs://bucket/dataset/channel', mip=0, bounded=True, fill_missing=False)
vol = CloudVolume('gs://bucket/dataset/channel', mip=[ 8, 8, 40 ], bounded=True, fill_missing=False) # set mip at this resolution
vol = CloudVolume('gs://bucket/datasset/channel', info=info) # New info file from scratch
image = vol[:,:,:] # Download the entire image stack into a numpy array
image = vol.download(bbox, mip=2, renumber=True) # download w/ smaller dtype
image = vol.download(bbox, mip=2, label=777) # download binary image for label
uniq = vol.unique(bbox, mip=0) # efficient extraction of unique labels
listing = vol.exists( np.s_[0:64, 0:128, 0:64] ) # get a report on which chunks actually exist
exists = vol.image.has_data(mip=0) # boolean check to see if any data is there
listing = vol.delete( np.s_[0:64, 0:128, 0:64] ) # delete this region (bbox must be chunk aligned)
vol[64:128, 64:128, 64:128] = image # Write a 64^3 image to the volume
img = vol.download_point( (x,y,z), size=256, mip=3 ) # download region around (mip 0) x,y,z at mip 3
pts = vol.scattered_points([ (x1,y1,z1), (x2,y2,z2) ]) # download voxel labels located at indicated points
# download image files without decompressing or rendering them. Good for caching!
files = vol.download_files(bbox, mip, decompress=False)

# creates an anonymous in-memory CloudVolume that
# will self-clean when the reference count drops to zero.
# Store compressed images in memory for quick access!
mem_vol = vol.image.memory_cutout(bbox, mip=1, encoding="compresso")

# Server
vol.viewer() # launches neuroglancer compatible web server on http://localhost:1337

# Microviewer (outdated, see https://github.com/seung-lab/microviewer/)
img = vol[64:1028, 64:1028, 64:128]
img.viewer() # launches web viewer on http://localhost:8080

# Meshes
vol.mesh.save(12345) # save 12345 as ./12345.ply on disk
vol.mesh.save([12345, 12346, 12347]) # merge three segments into one file
vol.mesh.save(12345, file_format='obj') # 'ply' and 'obj' are both supported
vol.mesh.get(12345) # return the mesh as vertices and faces instead of writing to disk
vol.mesh.get([ 12345, 12346 ]) # return these two segids fused into a single mesh
vol.mesh.get([ 12345, 12346 ], fuse=False) # return { 12345: mesh, 12346: mesh }
vol.mesh.put(meshes) # works for unsharded legacy only
vol.mesh.delete(segids) # works for unsharded meshes only

mesh.viewer() # Opens GUI. Requires vtk.

# Skeletons
skel = vol.skeleton.get(12345)
vol.skeleton.upload_raw(segid, skel.vertices, skel.edges, skel.radii, skel.vertex_types)
vol.skeleton.upload(skel)

# specified in nm, only available for datasets with a generated index
skels = vol.skeleton.get_by_bbox( Bbox( (0,0,0), (500, 500, 500) ) )
vol.skeleton.spatial_index # None if not available

skel.empty() # boolean

bytes = skel.encode() # encode to Precomputed format (bytes)
skel = Skeleton.decode(bytes) # decode from PrecomputedFormat

skel = skel.crop(slices or bbox) # eliminate vertices and edges outside bbox
skel = skel.consolidate() # eliminate duplicate vertices and edges
skel3 = skel.merge(skel2) # merge two skeletons into one
skel = skel.clone() # create copy
skel = Skeleton.from_swc(swcstr) # decode an SWC file
skel_str = skel.to_swc() # convert to SWC file in string representation
skel.viewer() # Opens GUI. Requires matplotlib

skel.cable_length() # sum of all edge lengths
skel = skel.downsample(2) # reduce size of skeleton by factor of 2

skel1 == skel2 # check if contents of internal arrays match
Skeleton.equivalent(skel1, skel2) # ...even if there are differences like differently numbered edges

# Parallel Operation
vol = CloudVolume('gs://mybucket/retina/image', parallel=True) # Use all cores
vol.parallel = 4 # e.g. any number > 1, use this many cores
data = vol[:] # uses shared memory to coordinate processes under the hood

# Shared Memory Output (can be used by other processes)
vol = CloudVolume(...)
# data backed by a shared memory buffer
# location is optional (defaults to vol.shared_memory_id)
data = vol.download_to_shared_memory(np.s_[:], location='some-example')
vol.unlink_shared_memory() # delete the shared memory associated with this cloudvolume
vol.shared_memory_id # get/set the default shared memory location for this instance

# Shared Memory Upload
vol = CloudVolume(...)
vol.upload_from_shared_memory('my-shared-memory-id', # do not prefix with /dev/shm
    bbox=Bbox( (0,0,0), (10000, 7500, 64) ))

# Download or Upload directly with Files
# The files must be in Precomputed raw format.
vol.download_to_file('/path/to/file', bbox=Bbox(...), mip=0) # bbox is the download region
vol.upload_from_file('/path/to/file', bbox=Bbox(...), mip=0) # bbox is the region it represents

# Transfer w/o Excess Memory Allocation
vol = CloudVolume(...)
# single core, send all of vol to destination, no painting memory
# you can also transcode the image encoding and compression type
vol.transfer_to('gs://bucket/dataset/layer', vol.bounds)

# Caching, default located at $HOME/.cloudvolume/cache/$PROTOCOL/$BUCKET/$DATASET/$LAYER/$RESOLUTION
# You can also set the cache location using
# cache=str or with environment variable CLOUD_VOLUME_CACHE_DIR
vol = CloudVolume('gs://mybucket/retina/image', cache=True) # Basic Example
image = vol[0:10,0:10,0:10] # Download partial image and cache
vol[0:10,0:10,0:10] = image # Upload partial image and cache

# Resizing and clearing the LRU in-memory cache
vol = CloudVolume(..., lru_bytes=num_bytes) # >= 0, 0 means disabled
vol.image.lru.resize(num_bytes) # same
vol.image.lru.clear()
len(vol.image.lru) # number of items in lru
vol.image.lru.nbytes # size in bytes (not counting LRU structures, nor recursive)
vol.image.lru.items() # etc, also functions as a dict

# Evaluating the on-disk Cache
vol.cache.list() # list files in cache at this mip level
vol.cache.list(mip=1) # list files in cache at mip 1
vol.cache.list_meshes()
vol.cache.list_skeletons()
vol.cache.num_files() # number of files at this mip level
vol.cache.num_bytes(all_mips=True) # Return num files for each mip level in a list
vol.cache.num_bytes() # number of bytes taken up by files, size on disk can be bigger
vol.cache.num_bytes(all_mips=True) # Return num bytes for each mip level in a list

vol.cache.enabled = True/False # Turn the cache on/off
vol.cache.path = Str # set the cache location
vol.cache.compress = None/True/False # None: Link to cloud setting, Boolean: Force cache to compressed (True) or uncompressed (False)

# Deleting Cache
vol.cache.flush() # Delete local cache for this layer at this mip level
vol.cache.flush(preserve=Bbox(...)) # Same, but preserve cache in a region of space
vol.cache.flush_region(region=Bbox(...), mips=[...]) # Delete the cached files in this region at these mip levels (default all mips)
vol.cache.flush_info()
vol.cache.flush_provenance()

# Using Green Threads
import gevent.monkey
gevent.monkey.patch_all(thread=False)

cv = CloudVolume(..., green_threads=True)
img = cv[...] # now green threads will be used

# Dask Interface (requires dask installation)
arr = cv.to_dask()
arr = cloudvolume.dask.from_cloudvolume(cloudpath) # same as to_dask
res = cloudvolume.dask.to_cloudvolume(arr, cloudpath, compute=bool, return_store=bool)
```

### CloudVolume Constructor

```python3
CloudVolume(
    cloudpath:str, mip:int=0, bounded:bool=True,
    autocrop:bool=False, fill_missing:bool=False, cache:CacheType=False,
    compress_cache:CompressType=None, cdn_cache:bool=True,
    progress:bool=INTERACTIVE, info:dict=None, provenance:dict=None,
    compress:CompressType=None, compress_level:Optional[int]=None,
    non_aligned_writes:bool=False, parallel:ParallelType=1, delete_black_uploads:bool=False,
    background_color:int=0, green_threads:bool=False, use_https:bool=False,
    max_redirects:int=10, mesh_dir:Optional[str]=None, skel_dir:Optional[str]=None,
    agglomerate:bool=False, secrets:SecretsType=None,
    spatial_index_db:Optional[str]=None, lru_bytes:int = 0,
    cache_locking:bool = True
)
```

*      agglomerate: (bool, graphene only) sets the default mode for downloading
        images to agglomerated (True) vs watershed (False).
*      autocrop: (bool) If the specified retrieval bounding box exceeds the
          volume bounds, process only the area contained inside the volume.
          This can be useful way to ensure that you are staying inside the
          bounds when `bounded=False`.
*      background_color: (number) Specifies what the "background value" of the
        volume is (traditionally 0). This is mainly for changing the behavior
        of delete_black_uploads.
*      bounded: (bool) If a region outside of volume bounds is accessed:
          True: Throw an error
          False: Allow accessing the region. If no files are present, an error
              will still be thrown. Consider combining this option with
              `fill_missing=True`. However, this can be dangrous as it allows
              missing files and potentially network errors to be intepreted as
              zeros.
*      cache: (bool or str) Store downs and uploads in a cache on disk
            and preferentially read from it before redownloading.
          - falsey value: no caching will occur.
          - True: cache will be located in a standard location.
          - non-empty string: cache is located at this file path

          After initialization, you can adjust this setting via:
          `cv.cache.enabled = ...` which accepts the same values.

          Note: This cache is totally separate from the LRU controlled by
          lru_bytes.
*      cache_locking: (bool) The local cache will use file locks via fasteners to prevent issues with multi-process cache access. If this is not a concern, performance can be slightly improved by setting this to False. This uses CloudFiles' locking mechanism.
*      cdn_cache: (int, bool, or str) Sets Cache-Control HTTP header on uploaded
        image files. Most cloud providers perform some kind of caching. As of
        this writing, Google defaults to 3600 seconds. Most of the time you'll
        want to go with the default.
        - int: number of seconds for cache to be considered fresh (max-age)
        - bool: True: max-age=3600, False: no-cache
        - str: set the header manually
*      compress: (bool, str, None) pick which compression method to use.
*          None: (default) gzip for raw arrays and no additional compression
            for compressed_segmentation and fpzip.
          bool:
            True=gzip,
            False=no compression, Overrides defaults
          str:
            'gzip': Extension so that we can add additional methods in the future
                    like lz4 or zstd.
            'br': Brotli compression, better compression rate than gzip
            '': no compression (same as False).
*      compress_level: (int, None) level for compression. Higher number results
          in better compression but takes longer.
        Defaults to 9 for gzip (ranges from 0 to 9).
        Defaults to 5 for brotli (ranges from 0 to 11).
*      compress_cache: (None or bool) If not None, override default compression
          behavior for the cache.
*      delete_black_uploads: (bool) If True, on uploading an entirely black chunk,
          issue a DELETE request instead of a PUT. This can be useful for avoiding storing
          tiny files in the region around an ROI. Some storage systems using erasure coding
          don't do well with tiny file sizes.
*      fill_missing: (bool) If a chunk file is unable to be fetched:
          True: Use a block of zeros
          False: Throw an error
*      green_threads: (bool) Use green threads instead of preemptive threads. This
        can result in higher download performance for some compression types. Preemptive
        threads seem to reduce performance on multi-core machines that aren't densely
        loaded as the CPython threads are assigned to multiple cores and the thrashing
        + GIL reduces performance. You'll need to add the following code to the top
        of your program to use green threads:

            import gevent.monkey
            gevent.monkey.patch_all(threads=False)
*      lru_bytes: (int) number of bytes used to cache recently used image
        tiles in memory. This is an in-memory cache and is completely separate from
        the `cache` parameter that handles disk IO. Tiles are stripped over only their
        second stage compression.
*      info: (dict) In lieu of fetching a neuroglancer info file, use this one.
          This is useful when creating new datasets and for repeatedly initializing
          a new cloudvolume instance.
*      max_redirects: (int) if > 0, allow up to this many redirects via info file 'redirect'
          data fields. If <= 0, allow no redirections and access the current info file directly
          without raising an error.
*      mesh_dir: (str) if not None, override the info['mesh'] key before pulling the
        mesh info file.
*      mip: (int or iterable) Which level of downsampling to read and write from.
          0 is the highest resolution. You can also specify the voxel resolution
          like mip=[6,6,30] which will search for the appropriate mip level.
*      non_aligned_writes: (bool) Enable non-aligned writes. Not multiprocessing
          safe without careful design. When not enabled, a
          cloudvolume.exceptions.AlignmentError is thrown for non-aligned writes.

          https://github.com/seung-lab/cloud-volume/wiki/Advanced-Topic:-Non-Aligned-Writes

      parallel (int: 1, bool): Number of extra processes to launch, 1 means only
          use the main process. If parallel is True use the number of CPUs
          returned by multiprocessing.cpu_count(). When parallel > 1, shared
          memory (Linux) or emulated shared memory via files (other platforms)
          is used by the underlying download.
*      progress: (bool) Show progress bars.
          Defaults to True in interactive python, False in script execution mode.
*      provenance: (string, dict) In lieu of fetching a provenance
          file, use this one.
*      secrets: (dict) provide per-instance authorization tokens. If not provided,
        defaults to looking in .cloudvolume/secrets for necessary tokens.
*      skel_dir: (str) if not None, override the info['skeletons'] key before
        pulling the skeleton info file.
*      spatial_index_db: (str) A path to an sqlite3 or mysql database that follows
        the following uri schema. sqlite is assumed if no scheme is present in
        the uri.
          [sqlite://]filename.db
          mysql://<username>:<password>@<host>:<port>/<db_name>

        Igneous generated datasets include a JSON based spatial
        database that tiles the dataset. This can be fast enough up to about 100 TVx
        datasets. Above that, a proper database is required for efficient queries.
        We provide multiple SQL database types that the index can be hosted on.
*      use_https: (bool) maps gs:// and s3:// to their respective https paths. The
        https paths hit a cached, read-only version of the data and may be faster.

### CloudVolume Methods

Better documentation coming later, but for now, here's a summary of the most useful method calls. Use help(cloudvolume.CloudVolume.$method) for more info.

* create_new_info (class method) - Helper function for creating info files for creating new data layers.
* refresh_info - Repull the info file.
* refresh_provenance - Repull the provenance file.
* bbox_to_mip - Covert a bounding box or slice from one mip level to another.
* slices_from_global_coords - *deprecated, why not use bbox_to_mip?* Find the CloudVolume slice from MIP 0 coordinates if you're on a different MIP. Often used in combination with neuroglancer.
* reset_scales - Delete mips other than 0 in the info file. Does not autocommit.
* add_scale - Generate a new mip level in the info property. Does not autocommit.
* commit_info - Push the current info property into the cloud as a JSON file.
* commit_provenance - Push the current provenance property into the cloud as a JSON file.
* image - Access image operations directly.
  * download - Download bounding boxes from a given mip level.
  * upload - Upload images to bounding boxes at a given mip level.
  * transfer_to - Transfer data without painting a container array to avoid out of memory errors.
  * exists - Check which chunk files exist in a given bounding box.
  * delete - Delete chunks in a given bounding box at a given mip level.
* mesh - Access mesh operations
	* get - Download an object. Can merge multiple segmentids
	* save - Download an object and save it in `.obj` format. You can combine equivialences into a single object too.
* skeleton - Access Skeletons
  * get - Download an object.
  * upload - Save a skeleton object to the cloud.
* cache - Access cache operations
	* enabled - Boolean switch to enable/disable cache. If true, on reading, check local disk cache before downloading, and save downloaded chunks to cache. When writing, write to the cloud then save the chunks you wrote to cache. If false, bypass cache completely. The cache is located at `$HOME/.cloudvolume/cache`.
	* path - Property that shows the current filesystem path to the cache
	* list - List files in cache
	* num_files - Number of files in cache at this mip level , use all_mips=True to get them all
	* num_bytes - Return the number of bytes in cache at this mip level, all_mips=True to get them all
	* flush - Delete the cache at this mip level, preserve=Bbox/slice to save a spatial region
	* flush_region - Delete a spatial region at this mip level
* exists - Generate a report on which chunks within a bounding box exist.
* delete - Delete the chunks within this bounding box.
* transfer_to - Transfer data from a bounding box to another data storage location. Does not allocate memory and transfers in blocks, so can transfer large volumes of data. May be less efficient than a dedicated tool like `gsutil` or `aws s3`.
* unlink_shared_memory - Delete shared memory associated with this instance (`vol.shared_memory_id`)
* generate_shared_memory_location - Create a new unique shared memory identifier string. No side effects.
* download_to_shared_memory - Instead of using ordinary numpy memory allocations, download to shared memory.
    Be careful, shared memory is like a file and doesn't disappear unless explicitly unlinked. (`vol.unlink_shared_memory()`)
* upload_from_shared_memory - Upload from a given shared memory block without making a copy.
* download_point - Download the region around this mip 0 coordinate at a given mip level.

### CloudVolume Properties

Accessed as `vol.$PROPERTY` like `vol.mip`. Parens next to each property mean (data type:default, writability). (r) means read only, (w) means write only, (rw) means read/write.

* mip (uint:0, rw) - Read from and write to this mip level (0 is highest res). Each additional increment in the number is typically a 2x reduction in resolution.
* bounded (bool:True, rw) - If a region outside of volume bounds is accessed throw an error if True or Fill the region with black (useful for e.g. marching cubes's 1px boundary) if False.
* autocrop (bool:False, rw) - If bounded is False and this option is True, automatically crop requested uploads and downloads to the volume boundary.
* fill_missing (bool:False, rw) - If a file inside volume bounds is unable to be fetched use a block of zeros if True, else throw an error.
* delete_black_uploads (bool:False, rw) - If True, issue a DELETE http request instead of a PUT when an individual uploaded chunk is all zeros.
* info (dict, rw) - Python dict representation of Neuroglancer info JSON file. You must call `vol.commit_info()` to save your changes to storage.
* provenance (dict-like, rw) - Data layer provenance file representation. You must call `vol.commit_provenance()` to save your changes to storage.
* available_mips (list of ints, r) - Query which mip levels are defined for reading and writing.
* dataset_name (str, rw) - Which dataset (e.g. test_v0, snemi3d_v0) on S3, GS, or FS you're reading and writing to. Known as an "experiment" in BOSS terminology. Writing to this property triggers an info refresh.
* layer (str, rw) - Which data layer (e.g. image, segmentation) on S3, GS, or FS you're reading and writing to. Known as a "channel" in BOSS terminology. Writing to this property triggers an info refresh.
* base_cloudpath (str, r) - The cloud path to the dataset e.g. s3://bucket/dataset/
* layer_cloudpath (str, r) - The cloud path to the data layer e.g. gs://bucket/dataset/image
* info_cloudpath (str, r) - Generate the cloud path to this data layer's info file.
* scales (dict, r) - Shortcut to the 'scales' property of the info object
* scale (dict, rw)* - Shortcut to the working scale of the current mip level
* shape (Vec4, r)* - Like numpy.ndarray.shape for the entire data layer.
* volume_size (Vec3, r)* - Like shape, but omits channel (x,y,z only).
* num_channels (int, r) - The number of channels, the last element of shape.
* layer_type (str, r) - The neuroglancer info type, 'image' or 'segmentation'.
* dtype (str, r) - The info data_type of the volume, e.g. uint8, uint32, etc. Similar to numpy.ndarray.dtype.
* encoding (str, r) - The neuroglancer info encoding. e.g. 'raw', 'jpeg', 'npz'
* resolution (Vec3, r)* - The 3D physical resolution of a voxel in nanometers at the working mip level.
* downsample_ratio (Vec3, r) - Ratio of the current resolution to the highest resolution mip available.
* chunk_size (Vec3, r)* - Size of the underlying chunks that constitute the volume in storage. e.g. Vec(64, 64, 64)
* key (str, r)* - The 'directory' we're accessing the current working mip level from within the data layer. e.g. '6_6_30'
* bounds (Bbox, r)* - A Bbox object that represents the bounds of the entire volume.
* shared_memory_id (str, rw) - Shared memory location used for parallel operation or for output.

\* These properties can also be accessed with a function named like `vol.mip_$PROPERTY($MIP)`. By default they return the current mip level assigned to the CloudVolume, but any mip level can be accessed via the corresponding `mip_` function. Example: `vol.mip_resolution(2)` would return the resolution of mip 2.

### VolumeCutout Functions

When you download an image using CloudVolume it gives you a `VolumeCutout`. These are `numpy.ndarray` subclasses that support a few extra properties to help make book keeping easier. The major advantage is `save_images()` which can help you view your dataset as PNG slices.

* `dataset_name` - The dataset this image came from.
* `layer` - Which layer it came from.
* `mip` - Which mip it came from
* `layer_type` - "image" or "segmentation"
* `bounds` - The bounding box of the cutout
* `num_channels` - Alias for `vol.shape[3]`
* `save_images()` - Save Z slice PNGs of the current image to `./saved_images` for manual inspection
* `viewer()` - Start a local web server (http://localhost:8080) that can view small volumes interactively. This was recently changed from `view` as `view` is a useful numpy method.

### Viewing a Precomputed Volume on Disk

If you have Precomputed volume onto local disk and would like to point neuroglancer to it:

```python
vol = CloudVolume(...)
vol.viewer()
```

You can then point any version of neuroglancer at it using `precomputed://http://localhost:1337/NAME_OF_LAYER`.

### Microviewer

CloudVolume includes a built-in dependency free viewer for 3D volumetric datasets smaller than about 2GB uncompressed. It supports bool, uint8, uint16, uint32, float32, and float64 numpy data types for both images and segmentation and can render a composite overlay of image and segmentation.

You can launch a viewer using the `.viewer()` method of a VolumeCutout object or by using the `view(...)` or `hyperview(...)` functions that come with the cloudvolume module. This launches a web server on `http://localhost:8080`. You can read more [on the wiki](https://github.com/seung-lab/cloud-volume/wiki/%CE%BCViewer).

```python3
from cloudvolume import CloudVolume, view, hyperview

channel_vol = CloudVolume(...)
seg_vol = CloudVolume(...)
img = vol[...]
seg = vol[...]

img.viewer() # works on VolumeCutouts
seg.viewer() # segmentation type derived from info
view(img) # alternative for arbitrary numpy arrays
view(seg, segmentation=True)
hyperview(img, seg) # img and seg shape must match

>>> Viewer server listening to http://localhost:8080
```

There are also seperate viewers for skeleton and mesh objects that can be invoked by calling `.viewer()` on either object. However, skeletons depend on `matplotlib` and meshes depend on `vtk` and OpenGL to function.

```bash
pip install vtk matplotlib
```

## Python 2.7 End of Life

Python 2.7 is no longer supported by CloudVolume. Updated versions of `pip` will download the last supported release 1.21.1. You can read more on the policy page: https://github.com/seung-lab/cloud-volume/wiki/Policy#python-27-end-of-life

## Related Projects

1. [Igneous](https://github.com/seung-lab/igneous): Computational pipeline for visualizing neuroglancer volumes.
2. [CloudVolume.jl](https://github.com/seung-lab/CloudVolume.jl): CloudVolume in Julia
3. [fpzip](https://github.com/seung-lab/fpzip): A Python Package for the C++ code by Lindstrom et al.
4. [compressed_segmentation](https://github.com/seung-lab/compressedseg): A Python Package wrapping the code for the compressed_segmentation format developed by Jeremy Maitin-Shepard and Stephen Plaza.
5. [Kimimaro](https://github.com/seung-lab/kimimaro): High performance skeletonization of densely labeled 3D volumes.
6. [compresso](https://github.com/seung-lab/compresso): High lossless compression of connectomics segmentation. Algorithm by and code derived from Matejek et al.
7. [zfpc](https://github.com/seung-lab/zfpc): Optimized zfp multi-stream container for alignment vector fields (and similar floating point data).
8. [crackle](https://github.com/seung-lab/crackle): Lossless high compression of connectomics segmentation. (BETA)

## Acknowledgments

Thank you to everyone that has contributed past or current to CloudVolume or the ecosystem it serves. We love you!

Jeremy Maitin-Shepard created [Neuroglancer](https://github.com/google/neuroglancer) and defined the Precomputed format. Yann Leprince provided a [pure Python codec](https://github.com/HumanBrainProject/neuroglancer-scripts) for the compressed_segmentation format. Jeremy Maitin-Shepard and Stephen Plaza created C++ code defining the compression and decompression (respectively) protocol for [compressed_segmentation](https://github.com/janelia-flyem/compressedseg). Peter Lindstrom et al. created [the fpzip algorithm](https://computation.llnl.gov/projects/floating-point-compression), and contributed a C++ implementation and advice. Nico Kemnitz adapted our data to fpzip using the "Kempression" protocol (we named it, not him). Dan Bumbarger contributed code and information helpful for getting CloudVolume working on Windows. Fredrik Kihlander's [pure python implementation](https://github.com/wc-duck/pymmh3) of murmurhash3 and [Austin Appleby](https://github.com/aappleby/smhasher) developed murmurhash3 which is necessary for the sharded format. Ben Falk advocated for and did the bulk of the work on brotli compression. Some of the ideas in CloudVolume are based on work by Jingpeng Wu in [BigArrays.jl](https://github.com/seung-lab/BigArrays.jl).  Sven Dorkenwald, Manuel Castro, and Akhilesh Halageri contributed advice and code towards implementing the graphene interface. Oluwaseun Ogedengbe contributed documentation for the sharded format. Eric Perlman wrote the reader for Neuroglancer Multi-LOD meshes. Ignacio Tartavull and William Silversmith wrote the initial version of CloudVolume.


Large Image
===========

|build-status| |codecov-io| |license-badge| |doi-badge| |pypi-badge|

.. |build-status| image:: https://img.shields.io/circleci/build/github/girder/large_image.svg
    :target: https://circleci.com/gh/girder/large_image
    :alt: Build Status

.. |license-badge| image:: https://img.shields.io/badge/license-Apache%202-blue.svg
    :target: https://raw.githubusercontent.com/girder/large_image/master/LICENSE
    :alt: License

.. |codecov-io| image:: https://img.shields.io/codecov/c/github/girder/large_image.svg
   :target: https://codecov.io/github/girder/large_image?branch=master
   :alt: codecov.io

.. |doi-badge| image:: https://img.shields.io/badge/DOI-10.5281%2Fzenodo.4723355-blue.svg
   :target: https://zenodo.org/badge/latestdoi/45569214

.. |pypi-badge| image:: https://img.shields.io/pypi/v/large-image.svg?logo=python&logoColor=white
   :target: https://pypi.org/project/large-image/

*Python modules to work with large, multiresolution images.*

Large Image is developed and maintained by the Data & Analytics group at `Kitware, Inc. <https://kitware.com>`_ for processing large geospatial and medical images. This provides the backbone for several of our image analysis platforms including `Resonant GeoData <https://github.com/ResonantGeoData/ResonantGeoData>`_, `HistomicsUI <https://github.com/DigitalSlideArchive/HistomicsUI>`_, and `the Digital Slide Archive <https://digitalslidearchive.github.io/digital_slide_archive/>`_.

Highlights
----------

- Tile serving made easy
- Supports a wide variety of geospatial and medical image formats
- Convert to tiled Cloud Optimized (Geo)Tiffs (also known as pyramidal tiffs)
- Python methods for retiling or accessing regions of images efficiently
- Options for restyling tiles, such as dynamically applying color and band transform

Installation
------------

In addition to installing the ``large-image`` package, you'll need at least one tile source (a ``large-image-source-xxx`` package).   You can install everything from the main project with one of these commands:

Pip
~~~

Install common tile sources on linux, OSX, or Windows::

    pip install large-image[common]

Install all tile sources on linux::

    pip install large-image[all] --find-links https://girder.github.io/large_image_wheels

Install all tile sources and all Girder plugins on linux::

    pip install large-image[all] girder-large-image-annotation[tasks] --find-links https://girder.github.io/large_image_wheels

Conda
~~~~~

Conda makes dependency management a bit easier if not on Linux. Some of the source modules are available on conda-forge. You can install the following::

    conda install -c conda-forge large-image-source-gdal
    conda install -c conda-forge large-image-source-tiff
    conda install -c conda-forge large-image-converter

Docker Image
~~~~~~~~~~~~

Included in this repository’s packages is a pre-built Docker image that has all
of the dependencies to read any supported image format.

This is particularly useful if you do not want to install some of the heavier
dependencies like GDAL on your system or want a dedicated and isolated
environment for working with large images.

To use, pull the image and run it by mounting a local volume where the
imagery is stored::

    docker pull ghcr.io/girder/large_image:latest
    docker run -v /path/to/images:/opt/images ghcr.io/girder/large_image:latest

Modules
-------

Large Image consists of several Python modules designed to work together.  These include:

- ``large-image``: The core module.

  You can specify extras_require of the name of any tile source included with this repository.  For instance, you can do ``pip install large-image[tiff]``.  There are additional extras_require options:

  - ``sources``: all of the tile sources in the repository, a specific source name (e.g., ``tiff``)

  - ``memcached``: use memcached for tile caching

  - ``converter``: include the converter module

  - ``colormaps``: use matplotlib for named color palettes used in styles

  - ``tiledoutput``: support for emitting large regions as tiled tiffs

  - ``performance``: include optional modules that can improve performance

  - ``common``: all of the tile sources and above packages that will install directly from pypi without other external libraries on linux, OSX, and Windows.

  - ``all``: for all of the above

- ``large-image-converter``: A utility for using pyvips and other libraries to convert images into pyramidal tiff files that can be read efficiently by large_image.
  You can specify extras_require of ``jp2k`` to include modules to allow output to JPEG2000 compression, ``sources`` to include all sources, ``stats`` to include modules to allow computing compression noise statistics, ``geospatial`` to include support for converting geospatial sources, or ``all`` for all of the optional extras_require.

- Tile sources:

  - ``large-image-source-bioformats``: A tile source for reading any file handled by the Java Bioformats library.

  - ``large-image-source-deepzoom``: A tile source for reading Deepzoom tiles.

  - ``large-image-source-dicom``: A tile source for reading DICOM WSI images.

  - ``large-image-source-gdal``: A tile source for reading geotiff files via GDAL.  This handles source data with more complex transforms than the mapnik tile source.

  - ``large-image-source-mapnik``: A tile source for reading geotiff and netcdf files via Mapnik and GDAL.  This handles more vector issues than the gdal tile source.

  - ``large-image-source-multi``: A tile source for compositing other tile sources into a single multi-frame source.

  - ``large-image-source-nd2``: A tile source for reading nd2 (NIS Element) images.

  - ``large-image-source-ometiff``: A tile source using the tiff library that can handle some multi-frame OMETiff files.

  - ``large-image-source-openjpeg``: A tile source using the Glymur library to read jp2 (JPEG 2000) files.

  - ``large-image-source-openslide``: A tile source using the OpenSlide library.  This works with svs, ndpi, Mirax, tiff, vms, and other file formats.

  - ``large-image-source-pil``: A tile source for small images via the Python Imaging Library (Pillow).

  - ``large-image-source-tiff``: A tile source for reading pyramidal tiff files in common compression formats.

  - ``large-image-source-tifffile``: A tile source using the tifffile library that can handle a wide variety of tiff-like files.

  - ``large-image-source-vips``: A tile source for reading any files handled by libvips.  This also can be used for writing tiled images from numpy arrays.

  - ``large-image-source-zarr``: A tile source using the zarr library that can handle OME-Zarr (OME-NGFF) files as well as some other zarr files.

  - ``large-image-source-test``: A tile source that generates test tiles, including a simple fractal pattern.  Useful for testing extreme zoom levels.

  - ``large-image-source-dummy``: A tile source that does nothing.

  Most tile sources can be used with girder-large-image.  You can specific an extras_require of ``girder`` to include ``girder-large-image`` with the source.

- As a Girder plugin:

  - ``girder-large-image``: Large Image as a Girder_ 3.x plugin.
    You can specify extras_require of ``tasks`` to install a Girder Worker task that can convert otherwise unreadable images to pyramidal tiff files.

  - ``girder-large-image-annotation``: Annotations for large images as a Girder_ 3.x plugin.

  - ``large-image-tasks``: A utility for running the converter via Girder Worker.
    You can specify an extras_require of ``girder`` to include modules needed to work with the Girder remote worker or ``worker`` to include modules needed on the remote side of the Girder remote worker.  If neither is specified, some conversion tasks can be run using Girder local jobs.

Developer Installation
----------------------

To install all packages from source, clone the repository::

    git clone https://github.com/girder/large_image.git
    cd large_image

Install all packages and dependencies::

    pip install -e . -r requirements-dev.txt

If you aren't developing with Girder 3, you can skip installing those components.  Use ``requirements-dev-core.txt`` instead of ``requirements-dev.txt``::

    pip install -e . -r requirements-dev-core.txt

Tile source prerequisites
=========================

Many tile sources have complex prerequisites.  These can be installed directly using your system's package manager or from some prebuilt Python wheels for Linux.  The prebuilt wheels are not official packages, but they can be used by instructing pip to use them by preference::

    pip install -e . -r requirements-dev.txt --find-links https://girder.github.io/large_image_wheels

.. _Girder: https://github.com/girder/girder


# seq2science
[![bioconda-badge](https://img.shields.io/badge/install%20with-bioconda-brightgreen.svg?style=flat)](http://bioconda.github.io/recipes/seq2science/README.html)
[![anaconda-version](https://anaconda.org/bioconda/seq2science/badges/version.svg)](https://anaconda.org/bioconda/seq2science/badges/version.svg)
[![nr downloads](https://img.shields.io/conda/dn/bioconda/seq2science.svg?style=flat)](http://bioconda.github.io/recipes/seq2science/README.html)
[![star this repo](https://img.shields.io/github/stars/vanheeringen-lab/seq2science?style=flat&color=brightgreen)](https://github.com/vanheeringen-lab/seq2science/stargazers)
[![Test Status](http://ocimum.science.ru.nl/jenkins/buildStatus/icon?job=seq2science%2Fmaster&subject=tests)](http://ocimum.science.ru.nl/jenkins/job/seq2science/job/master/lastBuild/display/redirect/)
[![docs](https://github.com/vanheeringen-lab/seq2science/workflows/docs/badge.svg)](https://vanheeringen-lab.github.io/seq2science)
[![DOI](https://img.shields.io/badge/DOI-10.7717%2Fpeerj.16380-%2302A7FC)](https://doi.org/10.7717/peerj.16380)
[![DOI](https://zenodo.org/badge/196379320.svg)](https://zenodo.org/badge/latestdoi/196379320)

Seq2science is the attempt of the *van heeringen lab* to generate a collection of generic pipelines/workflows which can be used by complete beginners to bioinformatics and experienced bioinformaticians alike. Please take a look at our [docs](https://vanheeringen-lab.github.io/seq2science/) for help with installation, how to run it, and best practices.

![alt text](https://vanheeringen-lab.github.io/seq2science/_static/seq2science.png "seq2science")

Our supported workflows:
* [Downloading of fastq](https://vanheeringen-lab.github.io/seq2science/content/workflows/download_fastq.html)
* [Alignment](https://vanheeringen-lab.github.io/seq2science/content/workflows/alignment.html)
* [ATAC-seq](https://vanheeringen-lab.github.io/seq2science/content/workflows/atac_seq.html)
* [RNA-seq](https://vanheeringen-lab.github.io/seq2science/content/workflows/rna_seq.html)
* [ChIP-seq](https://vanheeringen-lab.github.io/seq2science/content/workflows/chip_seq.html)
* [scATAC-seq](https://vanheeringen-lab.github.io/seq2science/content/workflows/scatac_seq.html)
* [scRNA-seq](https://vanheeringen-lab.github.io/seq2science/content/workflows/scrna_seq.html)

## New users
Information for new users, such as how to install and configure can be found in our [getting started](https://vanheeringen-lab.github.io/seq2science/content/gettingstarted.html) page, and our [Frequently Asked Questions (FAQ)](https://vanheeringen-lab.github.io/seq2science/content/faq.html) section.

## Feedback
Anything not working as expected? Please reach out to us via our [issues](https://github.com/vanheeringen-lab/seq2science/issues) page and we'll try to help you as soon as possible!

## Cite us
van der Sande M, Frölich S, Schäfers T, Smits JGA, Snabel RR, Rinzema S, van Heeringen SJ. 2023. Seq2science: an end-to-end workflow for functional genomics analysis. PeerJ 11:e16380 https://doi.org/10.7717/peerj.16380


# CliMetLab

[![PyPI version fury.io](https://badge.fury.io/py/climetlab.svg)](https://pypi.python.org/pypi/climetlab/)
[![PyPI pyversions](https://img.shields.io/pypi/pyversions/climetlab.svg)](https://pypi.python.org/pypi/climetlab/)
[![Build](https://github.com/ecmwf/climetlab/actions/workflows/long-test.yml/badge.svg)](https://github.com/ecmwf/climetlab/actions/workflows/long-test.yml)

**CliMetLab** is a Python package aiming at simplifying access to climate and meteorological datasets, allowing users to focus on science instead of
technical issues such as data access and data formats. It is mostly intended to be used in Jupyter notebooks, and be interoperable with all popular
data analytic packages, such as Numpy, Pandas, Xarray, SciPy, Matplotlib, etc. as well as machine learning frameworks, such as Tensorflow, Keras or PyTorch.

The documentation can be found at <https://climetlab.readthedocs.io/>.

### Plugins

See https://climetlab.readthedocs.io/en/latest/guide/pluginlist.html.

### License
[Apache License 2.0](LICENSE) In applying this licence, ECMWF does not waive the privileges and immunities
granted to it by virtue of its status as an intergovernmental organisation nor does it submit to any jurisdiction.


![](https://github.com/VUIIS/dax/blob/master/docs/images/dax_logo.png)
Distributed Automation for XNAT
===

# Install DAX

Install it with:

~~~~~~~~
pip install dax
~~~~~~~~

Use --upgrade to overwrite previous installation

To install current main branch:

~~~~~~~~
pip install git+https://github.com/VUIIS/dax.git --upgrade
~~~~~~~~

or a specific branch:

~~~~~~~~
pip install git+https://github.com/VUIIS/dax.git@branch_name --upgrade
~~~~~~~~

# Docs
[![Documentation Status](https://readthedocs.org/projects/dax/badge/?version=latest)](http://dax.readthedocs.org/en/latest/?badge=latest)

<!-- Disabled since masijenkins is no longer in use for this purpose
# Build
[![Build Status](http://masijenkins.vuse.vanderbilt.edu:8080/buildStatus/icon?job=Build_DAX)](http://masijenkins.vuse.vanderbilt.edu:8080/job/Build_DAX/)
-->

<!-- Disabled since spiders are now repos on github
# Spiders
All of our piplines are available on NITRC. [Come join our team!](https://www.nitrc.org/projects/masimatlab)
-->


# Welcome to PSYDAC

[![devel_tests](https://github.com/pyccel/psydac/actions/workflows/continuous-integration.yml/badge.svg)](https://github.com/pyccel/psydac/actions/workflows/continuous-integration.yml) [![docs](https://github.com/pyccel/psydac/actions/workflows/documentation.yml/badge.svg)](https://github.com/pyccel/psydac/actions/workflows/documentation.yml)

**PSYDAC** is a Python 3 Library for isogeometric analysis.

## Table of contents

-   [Requirements](#requirements)
-   [Python setup and project download](#python-setup-and-project-download)
-   [Installing the library](#installing-the-library)
-   [Optional PETSc installation](#optional-petsc-installation)
-   [Uninstall](#uninstall)
-   [Running tests](#running-tests)
-   [Speeding up Psydac's core](#speeding-up-psydacs-core)
-   [User Documentation](#user-documentation)
-   [Code Documentation](#code-documentation)

## Requirements

Psydac requires a certain number of components to be installed on the machine:

-   Fortran and C compilers with OpenMP support
-   OpenMP library
-   BLAS and LAPACK libraries
-   MPI library
-   HDF5 library with MPI support

The installations instructions depend on the operating system and on the packaging manager used.

### Linux Debian-Ubuntu-Mint

To install all requirements on a Linux Ubuntu operating system, just use APT, the Advanced Packaging Tool:
```sh
sudo apt update
sudo apt install python3 python3-dev python3-pip
sudo apt install gcc gfortran
sudo apt install libblas-dev liblapack-dev
sudo apt install libopenmpi-dev openmpi-bin
sudo apt install libomp-dev libomp5
sudo apt install libhdf5-openmpi-dev
```

### macOS

To install all the requirements on a macOS operating system we recommend using [Homebrew](https://brew.sh/):

```eh
brew update
brew install gcc
brew install openblas
brew install lapack
brew install open-mpi
brew install libomp
brew install hdf5-mpi
```

### Other operating systems

Please see the [instructions for the pyccel library](https://github.com/pyccel/pyccel#Requirements) for further details.

## Python setup and project download

We recommend creating a clean Python virtual environment using [venv](https://packaging.python.org/en/latest/guides/installing-using-pip-and-virtual-environments/#creating-a-virtual-environment):
```sh
python3 -m venv <ENV-PATH>
```
where `<ENV-PATH>` is the location to create the virtual environment.
(A new directory will be created at the required location.)

In order to activate the environment from a new terminal session just run the command
```sh
source <ENV-PATH>/bin/activate
```

One can clone the Psydac repository at any location `<ROOT-PATH>` in the filesystem which does not require administrator privileges, using either
```sh
git clone https://github.com/pyccel/psydac.git
```
or
```sh
git clone git@github.com:pyccel/psydac.git
```
The latter command requires a GitHub account.

## Installing the library

Psydac depends on several Python packages, which should be installed in the newly created virtual environment.
These dependencies can be installed from the cloned directory `<ROOT-PATH>/psydac` with the following steps.

First, set an environment variable with the path to the parallel HDF5 library.
This path can be obtained with a command which depends on your system.

-   **Ubuntu/Debian**:
    ```sh
    export HDF5_DIR=$(dpkg -L libhdf5-openmpi-dev | grep "libhdf5.so" | xargs dirname)
    ```

-   **macOS**:
    ```sh
    export HDF5_DIR=$(brew list hdf5-mpi | grep "libhdf5.dylib" | xargs dirname | xargs dirname)
    ```

Next, install the Python dependencies using `pip`:
```sh
export CC="mpicc"
export HDF5_MPI="ON"

python3 -m pip install --upgrade pip
python3 -m pip install -r requirements.txt
python3 -m pip install -r requirements_extra.txt --no-build-isolation
```

At this point the Psydac library may be installed in **standard mode**, which copies the relevant files to the correct locations of the virtual environment, or in **development mode**, which only installs symbolic links to the Psydac directory. The latter mode allows one to effect the behavior of Psydac by modifying the source files.

-   **Standard mode**:
    ```bash
    python3 -m pip install .
    ```

-   **Development mode**:
    ```bash
    python3 -m pip install --editable .
    ```

## Optional PETSc installation

Although Psydac provides several iterative linear solvers which work with our native matrices and vectors, it is often useful to access a dedicated library like [PETSc](https://petsc.org). To this end, our matrices and vectors have the method `topetsc()`, which converts them to the corresponding `petsc4py` objects.
(`petsc4py` is a Python package which provides Python bindings to PETSc.) After solving the linear system with a PETSc solver, the function `petsc_to_psydac` allows converting the solution vector back to the Psydac format.

In order to use these additional feature, PETSc and petsc4py must be installed as follows.
First, we download the latest release of PETSc from its [official Git repository](https://gitlab.com/petsc/petsc):
```sh
git clone --depth 1 --branch v3.20.5 https://gitlab.com/petsc/petsc.git
```
Next, we specify a configuration for complex numbers, and install PETSc in a local directory:
```sh
cd petsc

export PETSC_DIR=$(pwd)
export PETSC_ARCH=petsc-cmplx

./configure --with-scalar-type=complex --with-fortran-bindings=0 --have-numpy=1

make all check

cd -
```
Finally, we install the Python package `petsc4py` which is included in the `PETSc` source distribution:
```sh
python3 -m pip install wheel Cython numpy
python3 -m pip install petsc/src/binding/petsc4py
```

## Uninstall

-   **Whichever the install mode**:
    ```bash
    python3 -m pip uninstall psydac
    ```
-   **If PETSc was installed**:
    ```bash
    python3 -m pip uninstall petsc4py
    ```

The non-Python dependencies can be uninstalled manually using the package manager.
In the case of PETSc, it is sufficient to remove the cloned source directory given that the installation has been performed locally.

## Running tests

Let `<PSYDAC-PATH>` be the installation directory of Psydac.
In order to run all serial and parallel tests which do not use PETSc, just type:
```bash
export PSYDAC_MESH_DIR=<PSYDAC-PATH>/mesh/
python3 -m pytest --pyargs psydac -m "not parallel and not petsc"
python3 <PSYDAC-PATH>/mpi_tester.py --pyargs psydac -m "parallel and not petsc"
```

If PETSc and petsc4py were installed, some additional tests can be run:
```bash
python3 -m pytest --pyargs psydac -m "not parallel and petsc"
python3 <PSYDAC-PATH>/mpi_tester.py --pyargs psydac -m "parallel and petsc"
```

## Speeding up **Psydac**'s core

Many of Psydac's low-level Python functions can be translated to a compiled language using the [Pyccel](https://github.com/pyccel/pyccel) transpiler. Currently, all of those functions are collected in modules which follow the name pattern `[module]_kernels.py`.

The classical installation translates all kernel files to Fortran without user intervention. This does not happen in the case of an editable install, but the command `psydac-accelerate` is made available to the user instead. This command applies Pyccel to all the kernel files in the source directory, and the C language may be selected instead of Fortran (which is the default).

-   **Only in development mode**:
    ```bash
    python3 /path/to/psydac/psydac_accelerate.py [--language LANGUAGE] [--openmp]
    ```

## User documentation

-   [Output formats](./output.md)
-   [Notebook examples](./examples/notebooks/)
-   [Other examples](./examples/)

## Code documentation

Find our latest code documentation [here](https://pyccel.github.io/psydac/).

## Mesh Generation

After installation, a command `psydac-mesh` will be available.

### Example of usage

```bash
psydac-mesh -n='16,16' -d='3,3' square mesh.h5
```


Mission Pinball Framework - Media Controller (mpf-mc)
=====================================================

<img align="right" height="146" src="https://missionpinball.org/images/mpfmc-logo.png"/>

This package is for the "Media Controller" for the Mission Pinball Framework (MPF).

The architecture of MPF breaks the pinball software into two pieces. The core engine which controls the pinball machine
and runs the game logic is in the MPF package. Separate from that is the Media Controller which controls the display and
sound, including the DMD, Color DMD, RGB LED DMD, and/or onscreen or LCD display. (Note that you need a media controller
to drive a DMD and sound even if you don't have an on-screen LCD window. Don't worry--you can run this headless and/or
in a console-only environment for those cases.)

This package (mpf-mc) is MPF's "in box" media controller which is based on Kivy and Python. It leverages OpenGL and the
GPU of the computer it's running on. There are other media controller projects (at various levels of completeness) you can use with MPF built on Unity, Godot, Rust, and others.

MPF-MC runs on Windows, Mac, Linux, and Raspberry Pi. It can run on the same machine as the core MPF engine, or it can be a
separate machine. It runs as a separate process from MPF, so it works well on a multi-core computer.

The MPF project homepage is here : https://missionpinball.org

[![Coverage Status](https://coveralls.io/repos/missionpinball/mpf-mc/badge.svg?branch=dev&service=github)](https://coveralls.io/github/missionpinball/mpf-mc?branch=dev)
[![Test & Build Status](https://github.com/missionpinball/mpf-mc/actions/workflows/build_wheels.yml/badge.svg)](https://github.com/missionpinball/mpf-mc/actions/workflows/build_wheels.yml)

Documentation
-------------

https://missionpinball.org

Support
-------

MPF is open source and has no official support. Some MPF users follow the MPF-users Google group: https://groups.google.com/forum/#!forum/mpf-users. Individual hardware providers may provide additional support for users of their hardware.

Contributing
------------

MPF is a passion project created and maintained by volunteers. If you're a Python coder, documentation writer, or pinball maker, feel free to make a change and submit a pull request. For more information about contributing see the [Contributing Code](https://missionpinball.org/about/contributing_to_mpf)
and [Contributing Documentation](https://missionpinball.org/about/help) pages.

License
-------

MPF and related projects are released under the MIT License. Refer to the LICENSE file for details. Docs are released under Creative Commons CC BY 4.0.


Pipenv: Python Development Workflow for Humans
==============================================

[![image](https://img.shields.io/pypi/v/pipenv.svg)](https://python.org/pypi/pipenv)
[![image](https://img.shields.io/pypi/l/pipenv.svg)](https://python.org/pypi/pipenv)
[![CI](https://github.com/pypa/pipenv/actions/workflows/ci.yaml/badge.svg)](https://github.com/pypa/pipenv/actions/workflows/ci.yaml)
[![image](https://img.shields.io/pypi/pyversions/pipenv.svg)](https://python.org/pypi/pipenv)

------------------------------------------------------------------------

**Pipenv** is a Python virtualenv management tool that supports a multitude of systems and nicely bridges the gaps between pip, python (using system python, pyenv or asdf) and virtualenv.
*Linux, macOS, and Windows are all first-class citizens in pipenv.*

Pipenv automatically creates and manages a virtualenv for your projects, as well as adds/removes packages from your `Pipfile` as you install/uninstall packages. It also generates a project `Pipfile.lock`, which is used to produce deterministic builds.

Pipenv is primarily meant to provide users and developers of applications with an easy method to arrive at a consistent working project environment.

The problems that Pipenv seeks to solve are multi-faceted:

- You no longer need to use `pip` and `virtualenv` separately: they work together.
- Managing a `requirements.txt` file with package hashes can be problematic.  Pipenv uses `Pipfile` and `Pipfile.lock` to separate abstract dependency declarations from the last tested combination.
- Hashes are documented in the lock file which are verified during install. Security considerations are put first.
- Strongly encourage the use of the latest versions of dependencies to minimize security risks [arising from outdated components](https://www.owasp.org/index.php/Top_10-2017_A9-Using_Components_with_Known_Vulnerabilities).
- Gives you insight into your dependency graph (e.g. `$ pipenv graph`).
- Streamline development workflow by supporting local customizations with `.env` files.

Table Of Contents
------------------

- [Pipenv](#pipenv-python-development-workflow-for-humans)

- [Installation](#installation)

- [Features](#features)

- [Basic Concepts](#basic-concepts)

- [Other Commands](#other-commands)

- [Shell Completion](#shell-completion)

- [Usage](#usage)

    - [Usage Examples](#usage-examples)

    - [Commands](#commands)

    - [Locate the Project](#locate-the-project)

    - [Locate the virtualenv](#locate-the-virtualenv)

    - [Locate the Python Interpreter](#locate-the-python-interpreter)

    - [Install Packages](#install-packages)

    - [Installing from git](#installing-from-git)

    - [Install a dev dependency](#install-a-dev-dependency)

    - [Show a dependency graph](#show-a-dependency-graph)

    - [Generate a lockfile](#generate-a-lockfile)

    - [Install all dev dependencies](#install-all-dev-dependencies)

    - [Uninstall everything](#uninstall-everything)

    - [Use the shell](#use-the-shell)

- [Documentation](#documentation)

Installation
------------

**Pipenv can be installed with Python 3.7 and above.**

For most users, we recommend installing Pipenv using `pip`:

    pip install --user pipenv

Or, if you\'re using Fedora:

    sudo dnf install pipenv

Or, if you\'re using FreeBSD:

    pkg install py39-pipenv

Or, if you\'re using Gentoo:

    sudo emerge pipenv

Or, if you\'re using Void Linux:

    sudo xbps-install -S python3-pipenv

Alternatively, some users prefer to use [Pipx](https://pypi.org/p/pipx):

    pipx install pipenv

Or, some users prefer to use Python pip module

    python -m pip install pipenv

Refer to the [documentation](https://pipenv.pypa.io/en/latest/#install-pipenv-today) for latest instructions.

✨🍰✨

Features
----------

-   Enables truly *deterministic builds*, while easily specifying *only
    what you want*.
-   Generates and checks file hashes for locked dependencies.
-   Automatically install required Pythons, if `pyenv` or `asdf` is available.
-   Automatically finds your project home, recursively, by looking for a
    `Pipfile`.
-   Automatically generates a `Pipfile`, if one doesn\'t exist.
-   Automatically creates a virtualenv in a standard location.
-   Automatically adds/removes packages to a `Pipfile` when they are installed/uninstalled.
-   Automatically loads `.env` files, if they exist.

For command reference, see [Commands](https://pipenv.pypa.io/en/latest/commands/).

### Basic Concepts

-   A virtualenv will automatically be created, when one doesn\'t exist.
-   When no parameters are passed to `install`, all packages
    `[packages]` specified will be installed.
-   Otherwise, whatever virtualenv defaults to will be the default.

### Shell Completion

To enable completion in fish, add this to your configuration `~/.config/fish/completions/pipenv.fish`:

    eval (env _PIPENV_COMPLETE=fish_source pipenv)

There is also a [fish plugin](https://github.com/fisherman/pipenv), which will automatically
activate your subshells for you!

Alternatively, with zsh, add this to your configuration `~/.zshrc`:

    eval "$(_PIPENV_COMPLETE=zsh_source pipenv)"

Alternatively, with bash, add this to your configuration `~/.bashrc` or `~/.bash_profile`:

    eval "$(_PIPENV_COMPLETE=bash_source pipenv)"

Magic shell completions are now enabled!

Usage
-------

    $ pipenv --help
    Usage: pipenv [OPTIONS] COMMAND [ARGS]...

    Options:
      --where                         Output project home information.
      --venv                          Output virtualenv information.
      --py                            Output Python interpreter information.
      --envs                          Output Environment Variable options.
      --rm                            Remove the virtualenv.
      --bare                          Minimal output.
      --man                           Display manpage.
      --support                       Output diagnostic information for use in
                                      GitHub issues.
      --site-packages / --no-site-packages
                                      Enable site-packages for the virtualenv.
                                      [env var: PIPENV_SITE_PACKAGES]
      --python TEXT                   Specify which version of Python virtualenv
                                      should use.
      --clear                         Clears caches (pipenv, pip).  [env var:
                                      PIPENV_CLEAR]
      -q, --quiet                     Quiet mode.
      -v, --verbose                   Verbose mode.
      --pypi-mirror TEXT              Specify a PyPI mirror.
      --version                       Show the version and exit.
      -h, --help                      Show this message and exit.

   ### Usage Examples:

      Create a new project using Python 3.7, specifically:
      $ pipenv --python 3.7

      Remove project virtualenv (inferred from current directory):
      $ pipenv --rm

      Install all dependencies for a project (including dev):
      $ pipenv install --dev

      Create a lockfile containing pre-releases:
      $ pipenv lock --pre

      Show a graph of your installed dependencies:
      $ pipenv graph

      Check your installed dependencies for security vulnerabilities:
      $ pipenv check

      Install a local setup.py into your virtual environment/Pipfile:
      $ pipenv install -e .

      Use a lower-level pip command:
      $ pipenv run pip freeze

   ### Commands:

      check         Checks for PyUp Safety security vulnerabilities and against
                    PEP 508 markers provided in Pipfile.
      clean         Uninstalls all packages not specified in Pipfile.lock.
      graph         Displays currently-installed dependency graph information.
      install       Installs provided packages and adds them to Pipfile, or (if no
                    packages are given), installs all packages from Pipfile.
      lock          Generates Pipfile.lock.
      open          View a given module in your editor.
      requirements  Generate a requirements.txt from Pipfile.lock.
      run           Spawns a command installed into the virtualenv.
      scripts       Lists scripts in current environment config.
      shell         Spawns a shell within the virtualenv.
      sync          Installs all packages specified in Pipfile.lock.
      uninstall     Uninstalls a provided package and removes it from Pipfile.
      update        Runs lock, then sync.
      upgrade       Update the lock of the specified dependency / sub-dependency,
                    but does not actually install the packages.
      verify        Verify the hash in Pipfile.lock is up-to-date.

### Locate the project:

    $ pipenv --where
    /Users/kennethreitz/Library/Mobile Documents/com~apple~CloudDocs/repos/kr/pipenv/test

### Locate the virtualenv:

    $ pipenv --venv
    /Users/kennethreitz/.local/share/virtualenvs/test-Skyy4vre

### Locate the Python interpreter:

    $ pipenv --py
    /Users/kennethreitz/.local/share/virtualenvs/test-Skyy4vre/bin/python

### Install packages:

    $ pipenv install
    Creating a virtualenv for this project...
    ...
    No package provided, installing all dependencies.
    Virtualenv location: /Users/kennethreitz/.local/share/virtualenvs/test-EJkjoYts
    Installing dependencies from Pipfile.lock...
    ...

    To activate this project's virtualenv, run the following:
    $ pipenv shell

### Installing from git:

You can install packages with pipenv from git and other version control systems using URLs formatted according to the following rule:

    <vcs_type>+<scheme>://<location>/<user_or_organization>/<repository>@<branch_or_tag>#<package_name>

The only optional section is the `@<branch_or_tag>` section.  When using git over SSH, you may use the shorthand vcs and scheme alias `git+git@<location>:<user_or_organization>/<repository>@<branch_or_tag>#<package_name>`. Note that this is translated to `git+ssh://git@<location>` when parsed.

Valid values for `<vcs_type>` include `git`, `bzr`, `svn`, and `hg`.  Valid values for `<scheme>` include `http,`, `https`, `ssh`, and `file`.  In specific cases you also have access to other schemes: `svn` may be combined with `svn` as a scheme, and `bzr` can be combined with `sftp` and `lp`.

Note that it is **strongly recommended** that you install any version-controlled dependencies in editable mode, using `pipenv install -e`, in order to ensure that dependency resolution can be performed with an up to date copy of the repository each time it is performed, and that it includes all known dependencies.

Below is an example usage which installs the git repository located at `https://github.com/requests/requests.git` from tag `v2.19.1` as package name `requests`:

    $ pipenv install -e git+https://github.com/requests/requests.git@v2.19#egg=requests
    Creating a Pipfile for this project...
    Installing -e git+https://github.com/requests/requests.git@v2.19.1#egg=requests...
    [...snipped...]
    Adding -e git+https://github.com/requests/requests.git@v2.19.1#egg=requests to Pipfile's [packages]...
    [...]

You can read more about [pip's implementation of vcs support here](https://pip.pypa.io/en/stable/topics/vcs-support/).

### Install a dev dependency:

    $ pipenv install pytest --dev
    Installing pytest...
    ...
    Adding pytest to Pipfile's [dev-packages]...

### Show a dependency graph:

    $ pipenv graph
    requests==2.18.4
      - certifi [required: >=2017.4.17, installed: 2017.7.27.1]
      - chardet [required: >=3.0.2,<3.1.0, installed: 3.0.4]
      - idna [required: >=2.5,<2.7, installed: 2.6]
      - urllib3 [required: <1.23,>=1.21.1, installed: 1.22]

### Generate a lockfile:

    $ pipenv lock
    Assuring all dependencies from Pipfile are installed...
    Locking [dev-packages] dependencies...
    Locking [packages] dependencies...
    Note: your project now has only default [packages] installed.
    To install [dev-packages], run: $ pipenv install --dev

### Install all dev dependencies:

    $ pipenv install --dev
    Pipfile found at /Users/kennethreitz/repos/kr/pip2/test/Pipfile. Considering this to be the project home.
    Pipfile.lock out of date, updating...
    Assuring all dependencies from Pipfile are installed...
    Locking [dev-packages] dependencies...
    Locking [packages] dependencies...

### Uninstall everything:

    $ pipenv uninstall --all
    No package provided, un-installing all dependencies.
    Found 25 installed package(s), purging...
    ...
    Environment now purged and fresh!

### Use the shell:

    $ pipenv shell
    Loading .env environment variables...
    Launching subshell in virtual environment. Type 'exit' or 'Ctrl+D' to return.
    $ ▯

### PURPOSE AND ADVANTAGES OF PIPENV

To understand the problems that Pipenv solves, it's useful to show how Python package management has evolved.

Take yourself back to the first Python iteration. We had Python, but there was no clean way to install packages.

Then came Easy Install, a package that installs other Python packages with relative ease. But it came with a catch: it wasn't easy to uninstall packages that were no longer needed.

Enter pip, which most Python users are familiar with. pip lets us install and uninstall packages. We could specify versions, run pip freeze > requirements.txt to output a list of installed packages to a text file, and use that same text file to install everything an app needed with pip install -r requirements.txt.

But pip didn't include a way to isolate packages from each other. We might work on apps that use different versions of the same libraries, so we needed a way to enable that.

Pipenv aims to solve several problems.
First, the problem of needing the pip library for package installation, plus a library for creating a virtual environment, plus a library for managing virtual environments, plus all the commands associated with those libraries. That's a lot to manage. Pipenv ships with package management and virtual environment support, so you can use one tool to install, uninstall, track, and document your dependencies and to create, use, and organize your virtual environments. When you start a project with it, Pipenv will automatically create a virtual environment for that project if you aren't already using one.

Pipenv accomplishes this dependency management by abandoning the requirements.txt norm and trading it for a new document called a Pipfile. When you install a library with Pipenv, a Pipfile for your project is automatically updated with the details of that installation, including version information and possibly the Git repository location, file path, and other information.

Second, Pipenv wants to make it easier to manage complex interdependencies.

Using Pipenv, which gives you Pipfile, lets you avoid these problems by managing dependencies for different environments for you. This command will install the main project dependencies:

 pipenv install

Adding the --dev tag will install the dev/testing requirements:

 pipenv install --dev
To generate a Pipfile.lock file, run:

pipenv lock

You can also run Python scripts with Pipenv. To run a top-level Python script called hello.py, run:

pipenv run python hello.py

And you will see your expected result in the console.

To start a shell, run:

pipenv shell

If you would like to convert a project that currently uses a requirements.txt file to use Pipenv, install Pipenv and run:

pipenv install requirements.txt

This will create a Pipfile and install the specified requirements.

Documentation
---------------

Documentation resides over at [pipenv.pypa.io](https://pipenv.pypa.io/en/latest/).

## Star History

  <picture>
    <source media="(prefers-color-scheme: dark)" srcset="https://api.star-history.com/svg?repos=pypa/pipenv&type=Date&theme=dark" />
    <source media="(prefers-color-scheme: light)" srcset="https://api.star-history.com/svg?repos=pypa/pipenv&type=Date" />

  </picture>