# Epitran

A library and tool for transliterating orthographic text as IPA (International Phonetic Alphabet).

## Usage

The Python modules ```epitran``` and ```epitran.vector``` can be used to easily write more sophisticated Python programs for deploying the **Epitran** mapping tables, preprocessors, and postprocessors. This is documented below.

If you wish to use Epitran to convert **English** to IPA, **you must install the Flite (including `lex_lookup`) as detailed below.**

## Using the `epitran` Module

### The Epitran class

The most general functionality in the `epitran` module is encapsulated in the very simple `Epitran` class:

Epitran(code, preproc=True, postproc=True, ligatures=False, cedict_file=None).

Its constructor takes one argument, `code`, the ISO 639-3 code of the language to be transliterated plus a hyphen plus a four letter code for the script (e.g. 'Latn' for Latin script, 'Cyrl' for Cyrillic script, and 'Arab' for a Perso-Arabic script). It also takes optional keyword arguments:
* `preproc` and `postproc` enable pre- and post-processors. These are enabled by default.
* `ligatures` enables non-standard IPA ligatures like "ʤ" and "ʨ".
* `cedict_file` gives the path to the [CC-CEDict](https://cc-cedict.org/wiki/) dictionary file (relevant only when working with Mandarin Chinese and which, because of licensing restrictions cannot be distributed with Epitran).
* `tones` allows IPA tones (˩˨˧˦˥) to be included and is needed for tonal languages like Vietnamese and Hokkien. By default, this value is false and will remove IPA tones from the transcription.
* For more options, type ```help(epitran.Epitran.__init__)``` into a Python terminal session

```python
>>> import epitran
>>> epi = epitran.Epitran('uig-Arab')  # Uyghur in Perso-Arabic script
```

It is now possible to use the Epitran class for English and Mandarin Chinese (Simplified and Traditional) G2P as well as the other langugages that use Epitran's "classic" model. For Chinese, it is necessary to point the constructor to a copy of the [CC-CEDict](https://cc-cedict.org/wiki/) dictionary:

```python
>>> import epitran
>>> epi = epitran.Epitran('cmn-Hans', cedict_file='cedict_1_0_ts_utf-8_mdbg.txt')
```

The most useful public method of the Epitran class is `transliterate`:

Epitran.**transliterate**(text, normpunc=False, ligatures=False). Convert `text` (in Unicode-encoded orthography of the language specified in the constructor) to IPA, which is returned. `normpunc` enables punctuation normalization and `ligatures` enables non-standard IPA ligatures like "ʤ" and "ʨ". Usage is illustrated below (Python 2):

```python
>>> epi.transliterate(u'Düğün')
u'dy\u0270yn'
>>> print(epi.transliterate(u'Düğün'))
dyɰyn
```

Epitran.**word_to_tuples**(word, normpunc=False):
Takes a `word` (a Unicode string) in a supported orthography as input and returns a list of tuples with each tuple corresponding to an IPA segment of the word. The tuples have the following structure:
```
(
    character_category :: String,
    is_upper :: Integer,
    orthographic_form :: Unicode String,
    phonetic_form :: Unicode String,
    segments :: List<Tuples>
)
```

Note that **word_to_tuples** is not implemented for all language-script pairs.

The codes for `character_category` are from the initial characters of the two character sequences listed in the "General Category" codes found in [Chapter 4 of the Unicode Standard](http://www.unicode.org/versions/Unicode8.0.0/ch04.pdf#G134153). For example, "L" corresponds to letters and "P" corresponds to production marks. The above data structure is likely to change in subsequent versions of the library. The structure of ```segments``` is as follows:
```
(
    segment :: Unicode String,
    vector :: List<Integer>
)
```

Here is an example of an interaction with ```word_to_tuples``` (Python 2):

```python
>>> import epitran
>>> epi = epitran.Epitran('tur-Latn')
>>> epi.word_to_tuples(u'Düğün')
[(u'L', 1, u'D', u'd', [(u'd', [-1, -1, 1, -1, -1, -1, -1, -1, 1, -1, -1, 1, 1, -1, -1, -1, -1, -1, -1, 0, -1])]), (u'L', 0, u'u\u0308', u'y', [(u'y', [1, 1, -1, 1, -1, -1, -1, 0, 1, -1, -1, -1, -1, -1, 1, 1, -1, -1, 1, 1, -1])]), (u'L', 0, u'g\u0306', u'\u0270', [(u'\u0270', [-1, 1, -1, 1, 0, -1, -1, 0, 1, -1, -1, 0, -1, 0, -1, 1, -1, 0, -1, 1, -1])]), (u'L', 0, u'u\u0308', u'y', [(u'y', [1, 1, -1, 1, -1, -1, -1, 0, 1, -1, -1, -1, -1, -1, 1, 1, -1, -1, 1, 1, -1])]), (u'L', 0, u'n', u'n', [(u'n', [-1, 1, 1, -1, -1, -1, 1, -1, 1, -1, -1, 1, 1, -1, -1, -1, -1, -1, -1, 0, -1])])]
```

### The Backoff class

Sometimes, when parsing text in more than one script, it is useful to employ a graceful backoff. If one language mode does not work, it can be useful to fall back to another, and so on. This functionality is provided by the Backoff class:

Backoff(lang_script_codes, cedict_file=None)

Note that the Backoff class does not currently support parameterized preprocessor and postprocessor application and does not support non-standard ligatures. It also does not support punctuation normalization. `lang_script_codes` is a list of codes like `eng-Latn` or `hin-Deva`. For example, if one was transcribing a Hindi text with many English loanwords and some stray characters of Simplified Chinese, one might use the following code (Python 3):

```python
from epitran.backoff import Backoff
>>> backoff = Backoff(['hin-Deva', 'eng-Latn', 'cmn-Hans'], cedict_file=‘cedict_1_0_ts_utf-8_mdbg.txt')
>>> backoff.transliterate('हिन्दी')
'ɦindiː'
>>> backoff.transliterate('English')
'ɪŋɡlɪʃ'
>>> backoff.transliterate('中文')
'ʈ͡ʂoŋwən'
```

Backoff works on a token-by-token basis: tokens that contain mixed scripts will be returned as the empty string, since they cannot be fully converted by any of the modes.

The Backoff class has the following public methods:
* **transliterate**: returns a unicode string of IPA phonemes
* **trans_list**: returns a list of IPA unicode strings, each of which is a
  phoneme
* **xsampa_list**: returns a list of X-SAMPA (ASCII) strings, each of which is
  phoneme

Consider the following example (Python 3):

```python
>>> backoff.transliterate('हिन्दी')
'ɦindiː'
>>> backoff.trans_list('हिन्दी')
['ɦ', 'i', 'n', 'd', 'iː']
>>> backoff.xsampa_list('हिन्दी')
['h\\', 'i', 'n', 'd', 'i:']
```

### DictFirst

The `DictFirst` class provides a simple alternative to the `Backoff` class. It
requires a dictionary of words known to be of Language A, one word per line in a
UTF-8 encoded text file. It accepts three arguments: the language-script code
for Language A, that for Language B, and a path to the dictionary file. It has one public method, `transliteration`, which works like `Epitran.transliterate` except that it returns the transliteration for Language A if the input token is in the dictionary; otherwise, it returns the Language B transliteration of the token:

```python
>>> import dictfirst
>>> df = dictfirst.DictFirst('tpi-Latn', 'eng-Latn', '../sample-dict.txt')
>>> df.transliterate('pela')
'pela'
>>> df.transliterate('pelo')
'pɛlow'
```

### Preprocessors, postprocessors, and their pitfalls

In order to build a maintainable orthography to phoneme mapper, it is sometimes necessary to employ preprocessors that make contextual substitutions of symbols before text is passed to a orthography-to-IPA mapping system that preserves relationships between input and output characters. This is particularly true of languages with a poor sound-symbols correspondence (like French and English). Languages like French are particularly good targets for this approach because the pronunciation of a given string of letters is highly predictable even though the individual symbols often do not map neatly into sounds. (Sound-symbol correspondence is so poor in English that effective English G2P systems rely heavily on pronouncing dictionaries.)

Preprocessing the inputs words to allow for straightforward grapheme-to-phoneme mappings (as is done in the current version of ```epitran``` for some languages) is advantageous because the restricted regular expression language used to write the preprocessing rules is more powerful than the language for the mapping rules and allows the equivalent of many mapping rules to be written with a single rule. Without them, providing ```epitran``` support for languages like French and German would not be practical. However, they do present some problems. Specifically, when using a language with a preprocessor, one **must** be aware that the input word will not always be identical to the concatenation of the orthographic strings (```orthographic_form```) output by ```Epitran.word_to_tuples```. Instead, the output of ```word_to_tuple``` will reflect the output of the preprocessor, which may delete, insert, and change letters in order to allow direct orthography-to-phoneme mapping at the next step. The same is true of other methods that rely on ```Epitran.word_to_tuple``` such as ```VectorsWithIPASpace.word_to_segs``` from the ```epitran.vector``` module.

For information on writing new pre- and post-processors, see the section on "[Extending Epitran with map files, preprocessors and postprocessors](#extending-epitran)", below.

## Using the ```epitran.vector``` Module

The ```epitran.vector``` module is also very simple. It contains one class, ```VectorsWithIPASpace```, including one method of interest, ```word_to_segs```:

The constructor for ```VectorsWithIPASpace``` takes two arguments:
- `code`: the language-script code for the language to be processed.
- `spaces`: the codes for the punctuation/symbol/IPA space in which the characters/segments from the data are expected to reside. The available spaces are listed [below](#language-support).

Its principle method is ```word_to_segs```:

VectorWithIPASpace.**word_to_segs**(word, normpunc=False). `word` is a Unicode string. If the keyword argument *normpunc* is set to True, punctuation discovered in `word` is normalized to ASCII equivalents.

A typical interaction with the ```VectorsWithIPASpace``` object via the ```word_to_segs``` method is illustrated here (Python 2):

```python
>>> import epitran.vector
>>> vwis = epitran.vector.VectorsWithIPASpace('uzb-Latn', ['uzb-Latn'])
>>> vwis.word_to_segs(u'darë')
[(u'L', 0, u'd', u'd\u032a', u'40', [-1, -1, 1, -1, -1, -1, -1, -1, 1, -1, -1, 1, 1, 1, -1, -1, -1, -1, -1, 0, -1]), (u'L', 0, u'a', u'a', u'37', [1, 1, -1, 1, -1, -1, -1, 0, 1, -1, -1, -1, -1, -1, -1, -1, 1, 1, -1, 1, -1]), (u'L', 0, u'r', u'r', u'54', [-1, 1, 1, 1, 0, -1, -1, -1, 1, -1, -1, 1, 1, -1, -1, 0, 0, 0, -1, 0, -1]), (u'L', 0, u'e\u0308', u'ja', u'46', [-1, 1, -1, 1, -1, -1, -1, 0, 1, -1, -1, -1, -1, 0, -1, 1, -1, -1, -1, 0, -1]), (u'L', 0, u'e\u0308', u'ja', u'37', [1, 1, -1, 1, -1, -1, -1, 0, 1, -1, -1, -1, -1, -1, -1, -1, 1, 1, -1, 1, -1])]
```

(It is important to note that, though the word that serves as input--*darë*--has four letters, the output contains five tuples because the last letter in *darë* actually corresponds to two IPA segments, /j/ and /a/.) The returned data structure is a list of tuples, each with the following structure:

```
(
    character_category :: String,
    is_upper :: Integer,
    orthographic_form :: Unicode String,
    phonetic_form :: Unicode String,
    in_ipa_punc_space :: Integer,
    phonological_feature_vector :: List<Integer>
)
```

A few notes are in order regarding this data structure:
- ```character_category``` is defined as part of the Unicode standard ([Chapter 4](http://www.unicode.org/versions/Unicode8.0.0/ch04.pdf#G134153)). It consists of a single, uppercase letter from the set {'L', 'M', 'N', 'P', 'S', 'Z', 'C'}.. The most frequent of these are 'L' (letter), 'N' (number), 'P' (punctuation), and 'Z' (separator [including separating white space]).
- ```is_upper``` consists only of integers from the set {0, 1}, with 0 indicating lowercase and 1 indicating uppercase.
- The integer in ```in_ipa_punc_space``` is an index to a list of known characters/segments such that, barring degenerate cases, each character or segment is assignmed a unique and globally consistant number. In cases where a character is encountered which is not in the known space, this field has the value -1.
- The length of the list ```phonological_feature_vector``` should be constant for any instantiation of the class (it is based on the number of features defined in panphon) but is--in principle--variable. The integers in this list are drawn from the set {-1, 0, 1}, with -1 corresponding to '-', 0 corresponding to '0', and 1 corresponding to '+'. For characters with no IPA equivalent, all values in the list are 0.

## Language Support

### Transliteration Language/Script Pairs

| Code        | Language (Script)       |
|-------------|-------------------------|
| aar-Latn    | Afar                    |
| aii-Syrc    | Assyrian Neo-Aramaic    |
| amh-Ethi    | Amharic                 |
| amh-Ethi-pp | Amharic (more phonetic) |
| amh-Ethi-red| Amharic (reduced)       |
| ara-Arab    | Literary Arabic         |
| ava-Cyrl    | Avaric                  |
| aze-Cyrl    | Azerbaijani (Cyrillic)  |
| aze-Latn    | Azerbaijani (Latin)     |
| ben-Beng    | Bengali                 |
| ben-Beng-red| Bengali (reduced)       |
| bxk-Latn    | Bukusu                  |
| cat-Latn    | Catalan                 |
| ceb-Latn    | Cebuano                 |
| ces-Latn    | Czech                   |
| cjy-Latn    | Jin (Wiktionary)        |
| cmn-Hans    | Mandarin (Simplified)\* |
| cmn-Hant    | Mandarin (Traditional)\*|
| cmn-Latn    | Mandarin (Pinyin)\*     |
| ckb-Arab    | Sorani                  |
| csb-Latn    | Kashubian               |
| deu-Latn    | German                  |
| deu-Latn-np | German†                 |
| deu-Latn-nar| German (more phonetic)  |
| eng-Latn    | English‡                |
| epo-Latn    | Esperanto               |
| fas-Arab    | Farsi (Perso-Arabic)    |
| fra-Latn    | French                  |
| fra-Latn-np | French†                 |
| fra-Latn-p  | French (more phonetic)  |
| ful-Latn    | Fulah                   |
| gan-Latn    | Gan (Wiktionary)        |
| got-Latn    | Gothic                  |
| hak-Latn    | Hakka (pha̍k-fa-sṳ)      |
| hau-Latn    | Hausa                   |
| hin-Deva    | Hindi                   |
| hmn-Latn    | Hmong                   |
| hrv-Latn    | Croatian                |
| hsn-Latn    | Xiang (Wiktionary)      |
| hun-Latn    | Hungarian               |
| ilo-Latn    | Ilocano                 |
| ind-Latn    | Indonesian              |
| ita-Latn    | Italian                 |
| jam-Latn    | Jamaican                |
| jav-Latn    | Javanese                |
| kaz-Cyrl    | Kazakh (Cyrillic)       |
| kaz-Cyrl-bab| Kazakh (Cyrillic—Babel) |
| kaz-Latn    | Kazakh (Latin)          |
| kbd-Cyrl    | Kabardian               |
| khm-Khmr    | Khmer                   |
| kin-Latn    | Kinyarwanda             |
| kir-Arab    | Kyrgyz (Perso-Arabic)   |
| kir-Cyrl    | Kyrgyz (Cyrillic)       |
| kir-Latn    | Kyrgyz (Latin)          |
| kmr-Latn    | Kurmanji                |
| kmr-Latn-red| Kurmanji (reduced)      |
| lao-Laoo    | Lao                     |
| lij-Latn    | Ligurian                |
| lsm-Latn    | Saamia                  |
| ltc-Latn-bax | Middle Chinese (Baxter and Sagart 2014) |
| mal-Mlym    | Malayalam               |
| mar-Deva    | Marathi                 |
| mlt-Latn    | Maltese                 |
| mon-Cyrl-bab| Mongolian (Cyrillic)    |
| mri-Latn    | Maori                   |
| msa-Latn    | Malay                   |
| mya-Mymr    | Burmese                 |
| nan-Latn    | Hokkien (pe̍h-oē-jī)     |
| nan-Latn-tl | Hokkien (Tâi-lô)        |
| nld-Latn    | Dutch                   |
| nya-Latn    | Chichewa                |
| ood-Lat-alv | Tohono O'odham          |
| ood-Latn-sax| Tohono O'odham          |
| ori-Orya    | Odia                    |
| orm-Latn    | Oromo                   |
| pan-Guru    | Punjabi (Eastern)       |
| pol-Latn    | Polish                  |
| por-Latn    | Portuguese              |
| quy-Latn    | Ayacucho Quechua / Quechua Chanka       |
| ron-Latn    | Romanian                |
| run-Latn    | Rundi                   |
| rus-Cyrl    | Russian                 |
| sag-Latn    | Sango                   |
| sin-Sinh    | Sinhala                 |
| sna-Latn    | Shona                   |
| som-Latn    | Somali                  |
| spa-Latn    | Spanish                 |
| spa-Latn-eu | Spanish (Iberian)       |
| sqi-Latn    | Albanian                |
| srp-Latn    | Serbian                 |
| swa-Latn    | Swahili                 |
| swa-Latn-red| Swahili (reduced)       |
| swe-Latn    | Swedish                 |
| tam-Taml    | Tamil                   |
| tam-Taml-red| Tamil (reduced)         |
| tel-Telu    | Telugu                  |
| tgk-Cyrl    | Tajik                   |
| tgl-Latn    | Tagalog                 |
| tgl-Latn-red| Tagalog (reduced)       |
| tha-Thai    | Thai                    |
| tir-Ethi    | Tigrinya                |
| tir-Ethi-pp | Tigrinya (more phonemic)|
| tir-Ethi-red| Tigrinya (reduced)      |
| tpi-Latn    | Tok Pisin               |
| tuk-Cyrl    | Turkmen (Cyrillic)      |
| tuk-Latn    | Turkmen (Latin)         |
| tur-Latn    | Turkish (Latin)         |
| tur-Latn-bab| Turkish (Latin—Babel)   |
| tur-Latn-red| Turkish (reduced)       |
| ukr-Cyrl    | Ukranian                |
| urd-Arab    | Urdu                    |
| uig-Arab    | Uyghur (Perso-Arabic)   |
| uzb-Cyrl    | Uzbek (Cyrillic)        |
| uzb-Latn    | Uzbek (Latin)           |
| vie-Latn    | Vietnamese              |
| wuu-Latn    | Shanghainese Wu (Wiktionary) |
| xho-Latn    | Xhosa                   |
| yor-Latn    | Yoruba                  |
| yue-Latn    | Cantonese               |
| zha-Latn    | Zhuang                  |
| zul-Latn    | Zulu                    |

\*Chinese G2P requires the freely available [CC-CEDict](https://cc-cedict.org/wiki/) dictionary.

†These language preprocessors and maps naively assume a phonemic orthography.

‡English G2P requires the installation of the freely available [CMU Flite](http://tts.speech.cs.cmu.edu/awb/flite-2.0.5-current.tar.bz2) speech synthesis system.

### Languages with limited support due to highly ambiguous orthographies

Some the languages listed above should be approached with caution. It is not possible to provide highly accurate support for these language-script pairs due to the high degree of ambiguity inherent in the orthographies. Eventually, we plan to support these languages with a different back end based on WFSTs or neural methods.

| Code        | Language (Script)       |
|-------------|-------------------------|
| ara-Arab    | Arabic                  |
| cat-Latn    | Catalan                 |
| ckb-Arab    | Sorani                  |
| fas-Arab    | Farsi (Perso-Arabic)    |
| fra-Latn    | French                  |
| fra-Latn-np | French†                 |
| mya-Mymr    | Burmese                 |
| por-Latn    | Portuguese              |

### Language "Spaces"

| Code            | Language | Note                                 |
|-----------------|----------|--------------------------------------|
| amh-Ethi        | Amharic  |                                      |
| deu-Latn        | German   |                                      |
| eng-Latn        | English  |                                      |
| nld-Latn        | Dutch    |                                      |
| spa-Latn        | Spanish  |                                      |
| tur-Latn        | Turkish  | Based on data with suffixes attached |
| tur-Latn-nosuf  | Turkish  | Based on data with suffixes removed  |
| uzb-Latn-suf    | Uzbek    | Based on data with suffixes attached |

Note that major languages, including **French**, are missing from this table due to a lack of appropriate text data.

## Installation of Flite (for English G2P)

For use with most languages, Epitran requires no special installation steps. It can be installed as an ordinarary python package, either with `pip` or by running `python setup.py install` in the root of the source directory. However, English G2P in Epitran relies on CMU Flite, a speech synthesis package by Alan Black and other speech researchers at Carnegie Mellon University. For the current version of Epitran, you should follow the installation instructions for `lex_lookup`, which is used as the default G2P interface for Epitran.

### `t2p`

**Not recommended** This interface to Flite is now deprecated; Use `lex_lookup`.

### `lex_lookup`

**Recommended**

`t2p` does not behave as expected on letter sequences that are highly infrequent in English. In such cases, `t2p` gives the pronunciation of the English letters of the name, rather than an attempt at the pronunciation of the name. There is a different binary included in the most recent (pre-release) versions of Flite that behaves better in this regard, but takes some extra effort to install. To install, you need to obtain at least version [2.0.5](http://tts.speech.cs.cmu.edu/awb/flite-2.0.5-current.tar.bz2) of Flite. We recommend that you obtain the source from GitHub (https://github.com/festvox/flite). Untar and compile the source, following the steps below, adjusting where appropriate for your system:

```bash
$ tar xjf flite-2.0.5-current.tar.bz2
$ cd flite-2.0.5-current
```

or

```bash
$ git clone git@github.com:festvox/flite.git
$ cd flite/
```

then

```bash
$ ./configure && make
$ sudo make install
$ cd testsuite
$ make lex_lookup
$ sudo cp lex_lookup /usr/local/bin
```

When installing on MacOS and other systems that use a BSD version of `cp`, some modification to a Makefile must be made in order to install flite-2.0.5 (between steps 3 and 4). Edit `main/Makefile` and change both instances of `cp -pd` to `cp -pR`. Then resume the steps above at step 4.

### Usage

To use `lex_lookup`, simply instantiate Epitran as usual, but with the `code` set to 'eng-Latn':

```python
>>> import epitran
>>> epi = epitran.Epitran('eng-Latn')
>>> print epi.transliterate(u'Berkeley')
bɹ̩kli
```

## Extending Epitran with map files, preprocessors and postprocessors

Language support in Epitran is provided through map files, which define mappings between orthographic and phonetic units, preprocessors that run before the map is applied, and postprocessors that run after the map is applied. Maps are defined in UTF8-encoded, comma-delimited value (CSV) files. The files are each named `<iso639>-<iso15924>.csv` where `<iso639>` is the (three letter, all lowercase) ISO 639-3 code for the language and `<iso15924>` is the (four letter, capitalized) ISO 15924 code for the script. These files reside in the `data` directory of the Epitran installation under the `map`, `pre`, and `post` subdirectories, respectively. The pre- and post-processor files are text files whose format is described belown. They follow the same naming conventions except that they have the file extensions `.txt`.

### Map files (mapping tables)

The map files are simple, two-column files where the first column contains the orthgraphic characters/sequences and the second column contains the phonetic characters/sequences. The two columns are separated by a comma; each row is terminated by a newline. For many languages (most languages with unambiguous, phonemically adequate orthographies) just this easy-to-produce mapping file is adequate to produce a serviceable G2P system.

The first row is a header and is discarded. For consistency, it should contain the fields "Orth" and "Phon". The following rows by consist of fields of any length, separated by a comma. The same phonetic form (the second field) may occur any number of times but an orthographic form may only occur once. Where one orthograrphic form is a prefix of another form, the longer form has priority in mapping. In other words, matching between orthographic units and orthographic strings is greedy. Mapping works by finding the longest prefix of the orthographic form and adding the corresponding phonetic string to the end of the phonetic form, then removing the prefix from the orthographic form and continuing, in the same manner, until the orthographic form is consumed. If no non-empty prefix of the orthographic form is present in the mapping table, the first character in the orthographic form is removed and appended to the phonetic form. The normal sequence then resumes. This means that non-phonetic characters may end up in the "phonetic" form, which we judge to be better than losing information through an inadequate mapping table.

### Preprocesssors and postprocessors

For language-script pairs with more complicated orthographies, it is sometimes necessary to manipulate the orthographic form prior to mapping or to manipulate the phonetic form after mapping. This is done, in Epitran, with grammars of context-sensitive string rewrite rules. In truth, these rules would be more than adequate to solve the mapping problem as well but in practical terms, it is usually easier to let easy-to-understand and easy-to-maintain mapping files carry most of the weight of conversion and reserve the more powerful context sensitive grammar formalism for pre- and post-processing.

The preprocessor and postprocessor files have the same format. They consist of a sequence of lines, each consisting of one of four types:

1. Symbol definitions
2. Context-sensitive rewrite rules
3. Comments
4. Blank lines

#### Symbol definitions

Lines like the following

```
::vowels:: = a|e|i|o|u
```

define symbols that can be reused in writing rules. Symbols must consist of a prefix of two colons, a sequence of one or more lowercase letters and underscores, and a suffix of two colons. The are separated from their definitions by the equals sign (optionally set off with white space). The definition consists of a substring from a regular expression.

Symbols must be defined before they are referenced.

#### Rewrite rules

Context-sensitive rewrite rules in Epitran are written in a format familiar to phonologists but transparent to computer scientists. They can be schematized as

```
a -> b / X _ Y
```

which can be rewitten as

```
XaY → XbY
```

The arrow `->` can be read as "is rewritten as" and the slash `/` can be read as "in the context". The underscore indicates the position of the symbol(s) being rewritten. Another special symbol is the octothorp `#`, which indicates the beginning or end of a (word length) string (a word boundary). Consider the following rule:

```
e -> ə / _ #
```

This rule can be read as "/e/ is rewritten as /ə/ in the context at the end of the word." A final special symbol is zero `0`, which represents the empty string. It is used in rules that insert or delete segments. Consider the following rule that deletes /ə/ between /k/ and /l/:

```
ə　-> 0 / k _ l
```

All rules must include the arrow operator, the slash operator, and the underscore. A rule that applies in a context-free fashion can be written in the following way:

```
ch -> x / _
```

The implementation of context-sensitive rules in Epitran pre- and post-processors uses regular expression replacement. Specifically, it employs the `regex` package, a drop-in replacement for `re`. Because of this, regular expression notation can be used in writing rules:

```
c -> s / _ [ie]
```

or

```
c -> s / _ (i|e)
```

For a complete guide to `regex` regular expressions, see the documentation for [`re`](https://docs.python.org/2/library/re.html) and for [`regex`](https://pypi.python.org/pypi/regex), specifically.

Fragments of regular expressions can be assigned to symbols and reused throughout a file. For example, symbol for the disjunction of vowels in a language can be used in a rule that changes /u/ into /w/ before vowels:

```
::vowels:: = a|e|i|o|u
...
u -> w / _ (::vowels::)
```

There is a special construct for handling cases of metathesis (where "AB" is replaced with "BA"). For example, the rule:

```
(?P<sw1>[เแโไใไ])(?P<sw2>.) -> 0 / _
```

Will "swap" the positions of any character in "เแโไใไ" and any following character. Left of the arrow, there should be two groups (surrounded by parentheses) with the names `sw1` and `sw2` (a name for a group is specified by `?P<name>` appearing immediately after the open parenthesis for a group). The substrings matched by the two groups, `sw1` and `sw2` will be "swapped" or metathesized. The item immediately right of the arrow is ignored, but the context is not.

To move IPA tones to the end of the word, first ensure that tones=True in the instantiated Epitran object and use the following rule:
```
(?P<sw1>[˩˨˧˦˥]+)(?P<sw2>\w+) -> 0 / _\b
```

The rules apply in order, so earlier rules may "feed" and "bleed" later rules. Therefore, their sequence is *very important* and can be leveraged in order to achieve valuable results.

#### Comments and blank lines

Comments and blank lines (lines consisting only of white space) are allowed to make your code more readable. Any line in which the first non-whitespace character is a percent sign `%` is interpreted as comment. The rest of the line is ignored when the file is interpreted. Blank lines are also ignored.

### A strategy for adding language support

Epitran uses a mapping-and-repairs approach to G2P. It is expected that there is a mapping between graphemes and phonemes that can do most of the work of converting orthographic representations to phonological representations. In phonemically adequate orthogrphies, this mapping can do *all* of the work. This mapping should be completed first. For many languages, a basis for this mapping table already exists on [Wikipedia](http://www.wikipedia.org) and [Omniglot](http://www.omniglot.com) (though the Omniglot tables are typically not machine readable).

On the other hand, many writing systems deviate from the phonemically adequate idea. It is here that pre- and post-processors must be introduced. For example, in Swedish, the letter `<a>` receives a different pronunciation before two consonants (/ɐ/) than elsewhere (/ɑː/). It makes sense to add a preprocessor rule that rewrites `<a>` as /ɐ/ before two consonants (and similar rules for the other vowels, since they are affected by the same condition). Preprocessor rules should generally be employed whenever the orthographic representation must be adjusted (by contextual changes, deletions, etc.) prior to the mapping step.

One common use for postprocessors is to eliminate characters that are needed by the preprocessors or maps, but which should not appear in the output. A classic example of this is the virama used in Indic scripts. In these scripts, in order to write a consonant *not followed* by a vowel, one uses the form of the consonant symbol with particular inherent vowel followed by a virama (which has various names in different Indic languages). An easy way of handling this is to allow the mapping to translate the consonant into an IPA consonant + an inherent vowel (which, for a given language, will always be the same), then use the postprocessor to delete the vowel + virama sequence (wherever it occurs).

In fact, any situation where a character that is introduced by the map needs to be subsequently deleted is a good use-case for postprocessors. Another example from Indian languages includes so-called schwa deletion. Some vowels implied by a direct mapping between the orthography and the phonology are not actually pronounced; these vowels can generally be predicted. In most languages, they occur in the context after a vowel+consonant sequence and before a consonant+vowel sequence. In other words, the rule looks like the following:

```
ə -> 0 / (::vowel::)(::consonant::) _ (::consonant::)(::vowel::)
```

Perhaps the best way to learn how to structure language support for a new language is to consult the existing languages in Epitran. The French preprocessor `fra-Latn.txt` and the Thai postprocessor `tha-Thai.txt` illustrate many of the use-cases for these rules.

# Citing Epitran

If you use Epitran in published work, or in other research, please use the following citation:

David R. Mortensen, Siddharth Dalmia, and Patrick Littell. 2018. Epitran: Precision G2P for many languages. In *Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018)*, Paris, France. European Language Resources Association (ELRA).

```
@InProceedings{Mortensen-et-al:2018,
  author = {Mortensen, David R.  and Dalmia, Siddharth and Littell, Patrick},
  title = {Epitran: Precision {G2P} for Many Languages},
  booktitle = {Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018)},
  year = {2018},
  month = {May},
  date = {7--12},
  location = {Miyazaki, Japan},
  editor = {Nicoletta Calzolari (Conference chair) and Khalid Choukri and Christopher Cieri and Thierry Declerck and Sara Goggi and Koiti Hasida and Hitoshi Isahara and Bente Maegaard and Joseph Mariani and H\'el\`ene Mazo and Asuncion Moreno and Jan Odijk and Stelios Piperidis and Takenobu Tokunaga},
  publisher = {European Language Resources Association (ELRA)},
  address = {Paris, France},
  isbn = {979-10-95546-00-9},
  language = {english}
  }
  ```


================
django-reversion
================

|PyPI latest| |PyPI Version| |PyPI License| |Github actions| |Docs|

**django-reversion** is an extension to the Django web framework that provides
version control for model instances.

Requirements
============

- Python 3.7 or later
- Django 3.2 or later

Features
========

-  Roll back to any point in a model instance's history.
-  Recover deleted model instances.
-  Simple admin integration.

Documentation
=============

Check out the latest ``django-reversion`` documentation at `Getting Started <http://django-reversion.readthedocs.io/>`_

Issue tracking and source code can be found at the
`main project website <http://github.com/etianen/django-reversion>`_.

You can keep up to date with the latest announcements by joining the
`django-reversion discussion group <http://groups.google.com/group/django-reversion>`_.

Upgrading
=========

Please check the `Changelog <https://github.com/etianen/django-reversion/blob/master/CHANGELOG.rst>`_ before upgrading
your installation of django-reversion.

Contributing
============

Bug reports, bug fixes, and new features are always welcome. Please raise issues on the
`django-reversion project site <http://github.com/etianen/django-reversion>`_, and submit
pull requests for any new code.

1. Fork the `repository <http://github.com/etianen/django-reversion>`_ on GitHub.
2. Make a branch off of master and commit your changes to it.
3. Install requirements.

.. code:: bash

    $ pip install django psycopg2 mysqlclient -e .

4. Run the tests

.. code:: bash

    $ tests/manage.py test tests

5. Create a Pull Request with your contribution

Contributors
============

The django-reversion project was developed by `Dave Hall <http://www.etianen.com/>`_ and contributed
to by `many other people <https://github.com/etianen/django-reversion/graphs/contributors>`_.

.. |Docs| image:: https://readthedocs.org/projects/django-reversion/badge/?version=latest
   :target: http://django-reversion.readthedocs.org/en/latest/?badge=latest
.. |PyPI Version| image:: https://img.shields.io/pypi/pyversions/django-reversion.svg?maxAge=60
   :target: https://pypi.python.org/pypi/django-reversion
.. |PyPI License| image:: https://img.shields.io/pypi/l/django-reversion.svg?maxAge=120
   :target: https://github.com/rhenter/django-reversion/blob/master/LICENSE
.. |PyPI latest| image:: https://img.shields.io/pypi/v/django-reversion.svg?maxAge=120
   :target: https://pypi.python.org/pypi/django-reversion
.. |Github actions| image:: https://github.com/etianen/django-reversion/workflows/Python%20package/badge.svg
   :target: https://github.com/etianen/django-reversion


[PyBuilder — an easy-to-use build automation tool for Python](https://pybuilder.io)
=========

[![Follow PyBuilder on Twitter](https://img.shields.io/twitter/follow/pybuilder_?label=Follow%20PyBuilder&style=social)](https://twitter.com/intent/follow?screen_name=pybuilder_)
[![Gitter](https://img.shields.io/gitter/room/pybuilder/pybuilder?logo=gitter)](https://app.gitter.im/#/room/#pybuilder_pybuilder:gitter.im)
[![Build Status](https://img.shields.io/github/actions/workflow/status/pybuilder/pybuilder/pybuilder.yml?branch=master)](https://github.com/pybuilder/pybuilder/actions/workflows/pybuilder.yml)
[![Coverage Status](https://img.shields.io/coveralls/github/pybuilder/pybuilder/master?logo=coveralls)](https://coveralls.io/r/pybuilder/pybuilder?branch=master)

[![PyBuilder Version](https://img.shields.io/pypi/v/pybuilder?logo=pypi)](https://pypi.org/project/pybuilder/)
[![PyBuilder Python Versions](https://img.shields.io/pypi/pyversions/pybuilder?logo=pypi)](https://pypi.org/project/pybuilder/)
[![PyBuilder Downloads Per Day](https://img.shields.io/pypi/dd/pybuilder?logo=pypi)](https://pypi.org/project/pybuilder/)
[![PyBuilder Downloads Per Week](https://img.shields.io/pypi/dw/pybuilder?logo=pypi)](https://pypi.org/project/pybuilder/)
[![PyBuilder Downloads Per Month](https://img.shields.io/pypi/dm/pybuilder?logo=pypi)](https://pypi.org/project/pybuilder/)

PyBuilder is a software build tool written in 100% pure Python, mainly
targeting Python applications.

PyBuilder is based on the concept of dependency based programming, but it also
comes with a powerful plugin mechanism, allowing the construction of build life
cycles similar to those known from other famous (Java) build tools.

PyBuilder is running on the following versions of Python 3.8, 3.9, 3.10, 3.11, 3.12 and PyPy 3.8 and 3.9.

See the [GitHub Actions Workflow](https://github.com/pybuilder/pybuilder/actions/workflows/pybuilder.yml) for version specific output.

## Installing

PyBuilder is available using pip:

    $ pip install pybuilder

For development builds use:

    $ pip install --pre pybuilder

See the [PyPI](https://pypi.org/project/pybuilder/) for more information.

## Getting started

PyBuilder emphasizes simplicity. If you want to build a pure Python project and
use the recommended directory layout, all you have to do is create a file
build.py with the following content:

```python
from pybuilder.core import use_plugin

use_plugin("python.core")
use_plugin("python.unittest")
use_plugin("python.coverage")
use_plugin("python.distutils")

default_task = "publish"
```

See the [PyBuilder homepage](https://pybuilder.io) for more details and
a list of plugins.

## Release Notes

The release notes can be found [here](https://pybuilder.io/release-notes/).
There will also be a git tag with each release. Please note that we do not currently promote tags to GitHub "releases".

## Development
See [Developing PyBuilder](https://pybuilder.io/documentation/developing-pybuilder)


[![Package version](https://badge.fury.io/py/sentinelhub.svg)](https://pypi.org/project/sentinelhub/)
[![Conda version](https://img.shields.io/conda/vn/conda-forge/sentinelhub.svg)](https://anaconda.org/conda-forge/sentinelhub)
[![Supported Python versions](https://img.shields.io/pypi/pyversions/sentinelhub.svg?style=flat-square)](https://pypi.org/project/sentinelhub/)
[![Build Status](https://github.com/sentinel-hub/sentinelhub-py/actions/workflows/ci_action.yml/badge.svg?branch=master)](https://github.com/sentinel-hub/sentinelhub-py/actions)
[![Docs status](https://readthedocs.org/projects/sentinelhub-py/badge/?version=latest)](https://sentinelhub-py.readthedocs.io/en/latest/)
[![Overall downloads](https://pepy.tech/badge/sentinelhub)](https://pepy.tech/project/sentinelhub)
[![Last month downloads](https://pepy.tech/badge/sentinelhub/month)](https://pepy.tech/project/sentinelhub)
[![](https://img.shields.io/pypi/l/sentinelhub.svg)](https://github.com/sentinel-hub/sentinelhub-py/blob/master/LICENSE.md)
[![Code coverage](https://codecov.io/gh/sentinel-hub/sentinelhub-py/branch/master/graph/badge.svg)](https://codecov.io/gh/sentinel-hub/sentinelhub-py)

## Introduction

The **sentinelhub** Python package is the official Python interface for [Sentinel Hub services](https://www.sentinel-hub.com/). It supports most of the services described in the [Sentinel Hub documentation](https://docs.sentinel-hub.com/api/latest/) and any type of [satellite data collections](https://docs.sentinel-hub.com/api/latest/data/), including Sentinel, Landsat, MODIS, DEM, and custom collections produced by users.

The package also provides a collection of basic tools and utilities for working with geospatial and satellite data. It builds on top of well known packages such as `numpy`, `shapely`, `pyproj`, etc. It is also a core dependency of [`eo-learn`](https://github.com/sentinel-hub/eo-learn) Python package for creating geospatial data-processing workflows.

The main package resources are [GitHub repository](https://github.com/sentinel-hub/sentinelhub-py), [documentation page](https://sentinelhub-py.readthedocs.io/en/latest/), and [Sentinel Hub forum](https://forum.sentinel-hub.com/).

## Installation

The package requires a Python version >= 3.8. The package is available at the PyPI package index and can be installed with

```
$ pip install sentinelhub
```

or with an extension tag for additional functionalities

```
$ pip install sentinelhub[AWS]  # extra dependencies for interacting with Amazon Web Services
```

Alternatively, the package can be installed with Conda from `conda-forge` channel

```
$ conda install -c conda-forge sentinelhub
```

To install the package manually, clone the repository and run

```
$ pip install .
```

Before installing `sentinelhub` on **Windows** it is recommended to install `shapely` package from
[Unofficial Windows wheels repository](https://www.lfd.uci.edu/~gohlke/pythonlibs/)

Once installed the package can be configured according to [configuration instructions](https://sentinelhub-py.readthedocs.io/en/latest/configure.html) in documentation.

## Content

A high-level overview of the main functionalities:

- Sentinel Hub services
  * [Process API](https://docs.sentinel-hub.com/api/latest/api/process/),
  * [Catalog API](https://docs.sentinel-hub.com/api/latest/api/catalog/),
  * [Batch Processing API](https://docs.sentinel-hub.com/api/latest/api/batch/),
  * [BYOC API](https://docs.sentinel-hub.com/api/latest/api/byoc/),
  * [Statistical API](https://docs.sentinel-hub.com/api/latest/api/statistical/),
  * [OGC services (WMS/WCS/WFS)](https://docs.sentinel-hub.com/api/latest/api/ogc/),
  * [FIS](https://www.sentinel-hub.com/develop/api/ogc/fis-request/),
  * authentication and rate-limit handling,

- geospatial utilities
  * interface for geospatial objects and transformations,
  * large area splitting,
  * data collection objects,
  * IO tools,

- [Geopedia](https://portal.geopedia.world/) WMS and REST API.

## Documentation

For more information on the package and to access the documentation, visit [readthedocs](https://sentinelhub-py.readthedocs.io/).

## Examples

The package has a collection of Jupyter notebooks with examples. They are available in the [examples folder](https://github.com/sentinel-hub/sentinelhub-py/tree/master/examples) on GitHub and converted into documentation under [Examples section](https://sentinelhub-py.readthedocs.io/en/latest/examples.html).

Additionally, some examples are explained in Sentinel Hub webinar videos:

- [Process API in Python](https://www.youtube.com/watch?v=sX3w3Wd3FBw&list=PL46vEE2ks3tn8NGesSFllgJW5MSYRi4od&index=10&t=2220s)
- [OGC API in Python](https://www.youtube.com/watch?v=CBIlTOl2po4&list=PL46vEE2ks3tn8NGesSFllgJW5MSYRi4od&index=4&t=1766s)

## Blog posts

The package played a key role in many projects and use cases described at [Sentinel Hub blog](https://medium.com/sentinel-hub). The following blog posts are about the package itself:

 * [Upgrading the sentinelhub Python package](https://medium.com/sentinel-hub/upgrading-the-sentinelhub-python-package-2665f9c10df)
 * [Release of sentinelhub Python Package 2.0](https://medium.com/sentinel-hub/release-of-sentinelhub-python-package-2-0-a3d47709f8fd)

## Questions and Issues

Feel free to ask questions about the package and its use cases at [Sentinel Hub forum](https://forum.sentinel-hub.com/) or raise an issue on [GitHub](https://github.com/sentinel-hub/sentinelhub-py/issues).

You are welcome to send your feedback to the package authors, Sentinel Hub research team, through any of [Sentinel Hub communication channels](https://sentinel-hub.com/develop/communication-channels).

## License

See [LICENSE](https://github.com/sentinel-hub/sentinelhub-py/blob/master/LICENSE.md).


=====
PyXRF
=====

.. image:: https://img.shields.io/pypi/v/pyxrf.svg
        :target: https://pypi.python.org/pypi/pyxrf

.. image:: https://img.shields.io/conda/vn/conda-forge/pyxrf?color=green
        :target: https://anaconda.org/conda-forge/pyxrf

.. image:: https://img.shields.io/codecov/c/github/NSLS-II/pyxrf
        :target: https://codecov.io/gh/NSLS-II/pyxrf

.. image:: https://img.shields.io/github/commits-since/NSLS-II/PyXRF/latest
        :target: https://github.com/NSLS-II/PyXRF

.. image:: https://img.shields.io/pypi/dm/pyxrf?label=PyPI%20downloads
        :target: https://pypi.python.org/pypi/pyxrf

.. image:: https://img.shields.io/conda/dn/conda-forge/pyxrf?label=Conda-Forge%20downloads
        :target: https://anaconda.org/conda-forge/pyxrf

PyXRF is a python-based sophisticated fluorescence analysis package for
fitting and visualizing X-ray fluorescence data.

`PyXRF documentation <http://nsls-ii.github.io/PyXRF>`_

`Report an issue with PyXRF <https://github.com/NSLS-II/pyxrf/issues/new>`_

Installation instructions
-------------------------

PyXRF can be installed Linux, MacOS and Windows systems. Please refer
the `installation instructions <http://nsls-ii.github.io/PyXRF/installation.html>`_
in the `User's Manual <http://nsls-ii.github.io/PyXRF>`_.

Youtube tutorial
----------------

The Youtube tutorial for PyXRF can be found
`here <https://www.youtube.com/watch?v=2nFLccehaHI>`_.

Notes
-----

The core fitting functions are a part of the
`scikit-beam <https://github.com/scikit-beam/scikit-beam>`_
data analysis library for x-ray data analysis.


========================
HistomicsTK |codecov-io|
========================

.. |codecov-io| image:: https://codecov.io/github/DigitalSlideArchive/HistomicsTK/coverage.svg?branch=master
    :target: https://codecov.io/github/DigitalSlideArchive/HistomicsTK?branch=master
    :alt: codecov.io

.. |gitter| image:: https://badges.gitter.im/DigitalSlideArchive/HistomicsTK.svg
   :target: https://gitter.im/DigitalSlideArchive/HistomicsTK?utm_source=badge&utm_medium=badge&utm_campaign=pr-badge&utm_content=badge
   :alt: Join the chat at https://gitter.im/DigitalSlideArchive/HistomicsTK

`HistomicsTK`_ is a Python package for the analysis of digital pathology images. It can function as a stand-alone library, or as a Digital Slide Archive plugin that allows users to invoke image analysis jobs through HistomicsUI. The functionality offered by HistomicsTK can be extended using `slicer cli web <https://github.com/girder/slicer_cli_web>`__ which allows developers to integrate their image analysis algorithms into DSA for dissemination through HistomicsUI.

Whole-slide imaging captures the histologic details of tissues in large multiresolution images. Improvements in imaging technology, decreases in storage costs, and regulatory approval of digital pathology for primary diagnosis have resulted in an explosion of whole-slide imaging data. Digitization enables the application of computational image analysis and machine learning algorithms to characterize the contents of these images, and to understand the relationships between histology, clinical outcomes, and molecular data from genomic platforms. Compared to the related areas of radiology and genomics, open-source tools for the management, visualization, and analysis of digital pathology has lagged. To address this we have developed HistomicsTK in coordination with the `Digital Slide Archive`_ (DSA), a platform for managing and sharing digital pathology images in a centralized web-accessible server, and `HistomicsUI`_, a specialized user interface for annotation and markup of whole-slide images and for running image analysis tools and for scalable visualizing of dense outputs from image analysis algorithms. HistomicsTK aims to serve the needs of both pathologists/biologists interested in using state-of-the-art algorithms to analyze their data, and algorithm researchers interested in developing new/improved algorithms and disseminate them for wider use by the community.

HistomicsTK can be used in two ways:

- **As a pure Python package**: enables application of image analysis algorithms to data independent of the `Digital Slide Archive`_ (DSA). HistomicsTK provides a collection of fundamental algorithms for tasks such as color normalization, color deconvolution, nuclei segmentation, and feature extraction. Read more about these capabilities here:  `api-docs <https://digitalslidearchive.github.io/HistomicsTK/api-docs.html>`__ and `examples <https://digitalslidearchive.github.io/HistomicsTK/examples.html>`__ for more information.

  **Installation instructions on Linux:**

  *To install HistomicsTK using PyPI*::

  $ python -m pip install histomicstk --find-links https://girder.github.io/large_image_wheels

  *To install HistomicsTK from source*::

  $ git clone https://github.com/DigitalSlideArchive/HistomicsTK/
  $ cd HistomicsTK/
  $ python -m pip install setuptools-scm "Cython>=0.25.2" "scikit-build>=0.8.1" "cmake>=0.6.0" "numpy>=1.12.1"
  $ python -m pip install -e .

  HistomicsTK uses the `large_image`_ library to read content from whole-slide and microscopy image formats. Depending on your exact system, installing the necessary libraries to support these formats can be complex.  There are some non-official prebuilt libraries available for Linux that can be included as part of the installation by specifying ``pip install histomicstk --find-links https://girder.github.io/large_image_wheels``. Note that if you previously installed HistomicsTK or large_image without these, you may need to add ``--force-reinstall --no-cache-dir`` to the ``pip install`` command to force it to use the find-links option.

  The system version of various libraries are used if the ``--find-links`` option is not specified.  You will need to use your package manager to install appropriate libraries (on Ubuntu, for instance, you'll need ``libopenslide-dev`` and ``libtiff-dev``).

  **To install from source on Windows:**

  1- Run the following::

  $ pip install large-image
  $ pip install cmake
  $ git clone https://github.com/DigitalSlideArchive/HistomicsTK/
  $ cd HistomicsTK/
  $ python -m pip install setuptools-scm "Cython>=0.25.2" "scikit-build>=0.8.1" "cmake>=0.6.0" "numpy>=1.12.1"

  2- Run ``pip install libtiff``

  3- Run ``pip install large-image-source-tiff`` to install typical tile sources.  You may need other sources, which would require other libraries.

  4- Install Visual Studio 15 2017 `Community Version <https://my.visualstudio.com/Downloads?q=visual%20studio%202017&wt.mc_id=o~msft~vscom~older-downloads>`_

  5- Install C++ build tools. Under Tools > Get Tools and Features ... > Desktop Development with C++, ensure that the first 8 boxes are checked.

  6- Run this::

  $ python -m pip install -e .
  $ pip install girder-client

  **To install from source on OSX:**

  Note: This needs to be confirmed and expanded by an OSX user.  There are probably assumptions made about available libraries.

  Use homebrew to install libtiff and openslide or other libraries depending on your desired tile sources.

  Run::

  $ python -m pip install histomicstk large-image-source-tiff large-image-source-openslide

- **As a image-processing task library for HistomicsUI and the Digital Slide Archive**: This allows end users to apply containerized analysis modules/pipelines over the web. See the `Digital Slide Archive`_ for installation instructions.

Refer to `our website`_ for more information.

Previous Versions
-----------------

The HistomicsTK repository used to contain almost all of the Digital Slide Archive and HistomicsUI, and now container primarily code for image analysis algorithms and processing of annotation data.  The deployment and installation code and instructions for DSA have moved to the `Digital Slide Archive`_ repository.  The user interface and annotation functionality has moved to the `HistomicsUI`_ repository.

The deployment and UI code will eventually be removed from the master branch of this repository; any new development on those topics should be done in those locations.

Funding
-------

This work is funded by the NIH grant U24-CA194362-01_.

See Also
---------

**DSA/HistomicsTK project website:**
`Demos <https://digitalslidearchive.github.io/digital_slide_archive/demos-examples/>`_ |
`Success stories <https://digitalslidearchive.github.io/digital_slide_archive/success-stories/>`_

**Source repositories:** `Digital Slide Archive`_ | `HistomicsUI`_ | `large_image`_ | `slicer_cli_web`_

**Discussion:** `GitHub Discussion`_ | `Discourse forum`_

.. Links for everything above (not rendered):
.. _HistomicsTK: https://digitalslidearchive.github.io/digital_slide_archive/
.. _Digital Slide Archive: http://github.com/DigitalSlideArchive/digital_slide_archive
.. _HistomicsUI: http://github.com/DigitalSlideArchive/HistomicsUI
.. _large_image: https://github.com/girder/large_image
.. _our website: https://digitalslidearchive.github.io/digital_slide_archive/
.. _slicer execution model: https://www.slicer.org/slicerWiki/index.php/Slicer3:Execution_Model_Documentation
.. _slicer_cli_web: https://github.com/girder/slicer_cli_web
.. _Docker: https://www.docker.com/
.. _Kitware: http://www.kitware.com/
.. _U24-CA194362-01: http://grantome.com/grant/NIH/U24-CA194362-01
.. _GitHub Discussion: https://github.com/DigitalSlideArchive/digital_slide_archive/discussions
.. _Discourse forum: https://discourse.girder.org/c/histomicstk
.. _Gitter Chatroom: https://gitter.im/DigitalSlideArchive/HistomicsTK?utm_source=badge&utm_medium=badge&utm_campaign=pr-badge&utm_content=badge


Aerospike Python Client
=======================
|Build| |Release| |Wheel| |Downloads| |License|

.. |Build| image:: https://travis-ci.org/aerospike/aerospike-client-python.svg?branch=master
.. |Release| image:: https://img.shields.io/pypi/v/aerospike.svg
.. |Wheel| image:: https://img.shields.io/pypi/wheel/aerospike.svg
.. |Downloads| image:: https://img.shields.io/pypi/dm/aerospike.svg
.. |License| image:: https://img.shields.io/pypi/l/aerospike.svg

Compatibility
-------------

The Python client for Aerospike works with Python 3.8 - 3.12 and supports the following OS'es:

* macOS 11 and 12
* CentOS 7 Linux
* RHEL 8 and 9
* Amazon Linux 2023
* Debian 11 and 12
* Ubuntu 20.04 and 22.04
* Windows (x64)

The client is also verified to run on these operating systems, but we do not officially support them (i.e we don't distribute wheels or prioritize fixing bugs for these OSes):

* Alpine Linux

**NOTE:** Aerospike Python client 5.0.0 and up MUST be used with Aerospike server 4.9 or later.
If you see the error "-10, ‘Failed to connect’", please make sure you are using server 4.9 or later.

Install
-------

::

    pip install aerospike

In most cases ``pip`` will install a precompiled binary (wheel) matching your OS
and version of Python. If a matching wheel isn't found it, or the
``--install-option`` argument is provided, pip will build the Python client
from source.

Please see the `build instructions <https://github.com/aerospike/aerospike-client-python/blob/master/BUILD.md>`__
for more.

Troubleshooting
~~~~~~~~~~~~~~~

::

    # client >=3.8.0 will attempt a manylinux wheel installation for Linux distros
    # to force a pip install from source:
    pip install aerospike --no-binary :all:

    # to troubleshoot pip versions >= 6.0 you can
    pip install --no-cache-dir aerospike

If you run into trouble installing the client on a supported OS, you may be
using an outdated ``pip``.
Versions of ``pip`` older than 7.0.0 should be upgraded, as well as versions of
``setuptools`` older than 18.0.0.

Troubleshooting macOS
~~~~~~~~~~~~~~~~~~~~~

In some versions of macOS, Python 2.7 is installed as ``python`` with
``pip`` as its associated package manager, and Python 3 is installed as ``python3``
with ``pip3`` as the associated package manager. Make sure to use the ones that
map to Python 3, such as ``pip3 install aerospike``.

Attempting to install the client with pip for the system default Python may cause permissions issues when copying necessary files. In order to avoid
those issues the client can be installed for the current user only with the command: ``pip install --user aerospike``

::

    # to trouleshoot installation on macOS try
    pip install --no-cache-dir --user aerospike

Build
-----

For instructions on manually building the Python client, please refer to
`BUILD.md <https://github.com/aerospike/aerospike-client-python/blob/master/BUILD.md>`__.

Documentation
-------------

Documentation is hosted at `aerospike-python-client.readthedocs.io <https://aerospike-python-client.readthedocs.io/>`__
and at `aerospike.com/apidocs/python <http://www.aerospike.com/apidocs/python/>`__.

Examples
--------

Example applications are provided in the `examples directory of the GitHub repository <https://github.com/aerospike/aerospike-client-python/tree/master/examples/client>`__

For examples, to run the ``kvs.py``:

::

    python examples/client/kvs.py

Benchmarks
----------

To run the benchmarks the python module 'tabulate' need to be installed. In order to display heap information the module `guppy` must be installed.
Note that `guppy` is only available for Python2. If `guppy` is not installed the benchmarks will still be runnable.
Benchmark applications are provided in the `benchmarks directory of the GitHub repository <https://github.com/aerospike/aerospike-client-python/tree/master/benchmarks>`__

By default the benchmarks will try to connect to a server located at 127.0.0.1:3000 , instructions on changing that setting and other command line flags may be displayed by appending the `--help` argument to the benchmark script. For example:
::

    python benchmarks/keygen.py --help

License
-------

The Aerospike Python Client is made availabled under the terms of the
Apache License, Version 2, as stated in the file ``LICENSE``.

Individual files may be made available under their own specific license,
all compatible with Apache License, Version 2. Please see individual
files for details.


ASDF standard 1.6.0
===================
![CI](https://github.com/asdf-format/asdf-standard/actions/workflows/ci.yml/badge.svg)
![Downstream](https://github.com/asdf-format/asdf-standard/actions/workflows/downstream.yml/badge.svg)
[![pre-commit](https://img.shields.io/badge/pre--commit-enabled-brightgreen?logo=pre-commit&logoColor=white)](https://github.com/pre-commit/pre-commit)
[![Code style: black](https://img.shields.io/badge/code%20style-black-000000.svg)](https://github.com/psf/black)
[![Imports: isort](https://img.shields.io/badge/%20imports-isort-%231674b1?style=flat&labelColor=ef8336)](https://pycqa.github.io/isort/)
[![DOI](https://zenodo.org/badge/18113139.svg)](https://zenodo.org/badge/latestdoi/18113139)

This document describes the Advanced Scientific Data Format (ASDF),
pronounced *AZ*-diff.  You can read the full specification at [the online documentation](http://asdf-standard.readthedocs.io/).

> This is the **A**dvanced **S**cientific **D**ata **F**ormat - if you are looking for the **A**daptable **S**eismic **D**ata **F**ormat, go here: http://seismic-data.org/

## Contributing
Please open a new [issue](https://github.com/spacetelescope/asdf-standard/issue) or new [pull request](https://github.com/spacetelescope/asdf-standard/pulls) for bugs, feedback, or new features you would like to see. If there is an issue you would like to work on, please leave a comment and we will be happy to assist. New contributions and contributors are very welcome!

There are two mailing lists for ASDF:

* [asdf-users](https://groups.google.com/forum/#!forum/asdf-users)
* [asdf-developers](https://groups.google.com/forum/#!forum/asdf-developers)

## Getting More Information

* [FAQ page](https://github.com/spacetelescope/asdf-standard/wiki/FAQ) on the wiki.
* [ASDF: A new data format for astronomy](https://doi.org/10.1016/j.ascom.2015.06.004), published in *Astronomy and Computing*:

> Greenfield, P., Droettboom, M., & Bray, E. (2015). ASDF: A new data format for astronomy. *Astronomy and Computing*, 12: 240-251.
> doi:[10.1016/j.ascom.2015.06.004](https://doi.org/10.1016/j.ascom.2015.06.004)


# Python library to parse, validate and create SPDX documents

CI status (Linux, macOS and Windows): [![Install and Test][1]][2]

[1]: https://github.com/spdx/tools-python/actions/workflows/install_and_test.yml/badge.svg

[2]: https://github.com/spdx/tools-python/actions/workflows/install_and_test.yml

# Breaking changes v0.7 -> v0.8

Please be aware that the upcoming 0.8 release has undergone a significant refactoring in preparation for the upcoming
SPDX v3.0 release, leading to breaking changes in the API.
Please refer to the [migration guide](https://github.com/spdx/tools-python/wiki/How-to-migrate-from-0.7-to-0.8)
to update your existing code.

The main features of v0.8 are:
- full validation of SPDX documents against the v2.2 and v2.3 specification
- support for SPDX's RDF format with all v2.3 features
- experimental support for the upcoming SPDX v3 specification. Note, however, that support is neither complete nor
  stable at this point, as the spec is still evolving. SPDX3-related code is contained in a separate subpackage "spdx3"
  and its use is optional. We do not recommend using it in production code yet.

# Information

This library implements SPDX parsers, convertors, validators and handlers in Python.

- Home: https://github.com/spdx/tools-python
- Issues: https://github.com/spdx/tools-python/issues
- PyPI: https://pypi.python.org/pypi/spdx-tools
- Browse the API: https://spdx.github.io/tools-python

Important updates regarding this library are shared via the SPDX tech mailing list: https://lists.spdx.org/g/Spdx-tech.

# License

[Apache-2.0](LICENSE)

# Features

* API to create and manipulate SPDX v2.2 and v2.3 documents
* Parse, convert, create and validate SPDX files
* supported formats: Tag/Value, RDF, JSON, YAML, XML
* visualize the structure of a SPDX document by creating an `AGraph`. Note: This is an optional feature and requires
additional installation of optional dependencies

## Experimental support for SPDX 3.0
* Create v3.0 elements and payloads
* Convert v2.2/v2.3 documents to v3.0
* Serialize to JSON-LD

See [Quickstart to SPDX 3.0](#quickstart-to-spdx-30) below.
The implementation is based on the descriptive markdown files in the repository https://github.com/spdx/spdx-3-model (latest commit: a5372a3c145dbdfc1381fc1f791c68889aafc7ff).

# Installation

As always you should work in a virtualenv (venv). You can install a local clone
of this repo with `yourenv/bin/pip install .` or install it from PyPI
(check for the [newest release](https://pypi.org/project/spdx-tools/#history) and install it like
`yourenv/bin/pip install spdx-tools==0.8.0a2`). Note that on Windows it would be `Scripts`
instead of `bin`.

# How to use

## Command-line usage

1. **PARSING/VALIDATING** (for parsing any format):

* Use `pyspdxtools -i <filename>` where `<filename>` is the location of the file. The input format is inferred automatically from the file ending.

* If you are using a source distribution, try running:
  `pyspdxtools -i tests/data/SPDXJSONExample-v2.3.spdx.json`

2. **CONVERTING** (for converting one format to another):

* Use `pyspdxtools -i <input_file> -o <output_file>` where `<input_file>` is the location of the file to be converted
  and `<output_file>` is the location of the output file. The input and output formats are inferred automatically from the file endings.

* If you are using a source distribution, try running:
  `pyspdxtools -i tests/data/SPDXJSONExample-v2.3.spdx.json -o output.tag`

* If you want to skip the validation process, provide the `--novalidation` flag, like so:
  `pyspdxtools -i tests/data/SPDXJSONExample-v2.3.spdx.json -o output.tag --novalidation`
  (use this with caution: note that undetected invalid documents may lead to unexpected behavior of the tool)

* For help use `pyspdxtools --help`

3. **GRAPH GENERATION** (optional feature)

* This feature generates a graph representing all elements in the SPDX document and their connections based on the provided
  relationships. The graph can be rendered to a picture. Below is an example for the file `tests/data/SPDXJSONExample-v2.3.spdx.json`:
![SPDXJSONExample-v2.3.spdx.png](assets/SPDXJSONExample-v2.3.spdx.png)
* Make sure you install the optional dependencies `networkx` and `pygraphviz`. To do so run `pip install ".[graph_generation]"`.
* Use `pyspdxtools -i <input_file> --graph -o <output_file>` where `<output_file>` is an output file name with valid format for `pygraphviz` (check
  the documentation [here](https://pygraphviz.github.io/documentation/stable/reference/agraph.html#pygraphviz.AGraph.draw)).
* If you are using a source distribution, try running
  `pyspdxtools -i tests/data/SPDXJSONExample-v2.3.spdx.json --graph -o SPDXJSONExample-v2.3.spdx.png` to generate
  a png with an overview of the structure of the example file.

## Library usage
1. **DATA MODEL**
  * The `spdx_tools.spdx.model` package constitutes the internal SPDX v2.3 data model (v2.2 is simply a subset of this). All relevant classes for SPDX document creation are exposed in the `__init__.py` found [here](src%2Fspdx_tools%2Fspdx%2Fmodel%2F__init__.py).
  * SPDX objects are implemented via `@dataclass_with_properties`, a custom extension of `@dataclass`.
    * Each class starts with a list of its properties and their possible types. When no default value is provided, the property is mandatory and must be set during initialization.
    * Using the type hints, type checking is enforced when initializing a new instance or setting/getting a property on an instance
      (wrong types will raise `ConstructorTypeError` or `TypeError`, respectively). This makes it easy to catch invalid properties early and only construct valid documents.
    * Note: in-place manipulations like `list.append(item)` will circumvent the type checking (a `TypeError` will still be raised when reading `list` again). We recommend using `list = list + [item]` instead.
  * The main entry point of an SPDX document is the `Document` class from the [document.py](src%2Fspdx_tools%2Fspdx%2Fmodel%2Fdocument.py) module, which links to all other classes.
  * For license handling, the [license_expression](https://github.com/nexB/license-expression) library is used.
  * Note on `documentDescribes` and `hasFiles`: These fields will be converted to relationships in the internal data model. As they are deprecated, these fields will not be written in the output.
2. **PARSING**
  * Use `parse_file(file_name)` from the `parse_anything.py` module to parse an arbitrary file with one of the supported file endings.
  * Successful parsing will return a `Document` instance. Unsuccessful parsing will raise `SPDXParsingError` with a list of all encountered problems.
3. **VALIDATING**
  * Use `validate_full_spdx_document(document)` to validate an instance of the `Document` class.
  * This will return a list of `ValidationMessage` objects, each consisting of a String describing the invalidity and a `ValidationContext` to pinpoint the source of the validation error.
  * Validation depends on the SPDX version of the document. Note that only versions `SPDX-2.2` and `SPDX-2.3` are supported by this tool.
4. **WRITING**
  * Use `write_file(document, file_name)` from the `write_anything.py` module to write a `Document` instance to the specified file.
    The serialization format is determined from the filename ending.
  * Validation is performed per default prior to the writing process, which is cancelled if the document is invalid. You can skip the validation via `write_file(document, file_name, validate=False)`.
    Caution: Only valid documents can be serialized reliably; serialization of invalid documents is not supported.

## Example
Here are some examples of possible use cases to quickly get you started with the spdx-tools.
If you want more examples, like how to create an SPDX document from scratch, have a look [at the examples folder](examples).
```python
import logging

from license_expression import get_spdx_licensing

from spdx_tools.spdx.model import (Checksum, ChecksumAlgorithm, File,
                                   FileType, Relationship, RelationshipType)
from spdx_tools.spdx.parser.parse_anything import parse_file
from spdx_tools.spdx.validation.document_validator import validate_full_spdx_document
from spdx_tools.spdx.writer.write_anything import write_file

# read in an SPDX document from a file
document = parse_file("spdx_document.json")

# change the document's name
document.creation_info.name = "new document name"

# define a file and a DESCRIBES relationship between the file and the document
checksum = Checksum(ChecksumAlgorithm.SHA1, "71c4025dd9897b364f3ebbb42c484ff43d00791c")

file = File(name="./fileName.py", spdx_id="SPDXRef-File", checksums=[checksum],
            file_types=[FileType.TEXT],
            license_concluded=get_spdx_licensing().parse("MIT and GPL-2.0"),
            license_comment="licenseComment", copyright_text="copyrightText")

relationship = Relationship("SPDXRef-DOCUMENT", RelationshipType.DESCRIBES, "SPDXRef-File")

# add the file and the relationship to the document
# (note that we do not use "document.files.append(file)" as that would circumvent the type checking)
document.files = document.files + [file]
document.relationships = document.relationships + [relationship]

# validate the edited document and log the validation messages
# (depending on your use case, you might also want to utilize the validation_message.context)
validation_messages = validate_full_spdx_document(document)
for validation_message in validation_messages:
    logging.warning(validation_message.validation_message)

# if there are no validation messages, the document is valid
# and we can safely serialize it without validating again
if not validation_messages:
    write_file(document, "new_spdx_document.rdf", validate=False)
```

# Quickstart to SPDX 3.0
In contrast to SPDX v2, all elements are now subclasses of the central `Element` class.
This includes packages, files, snippets, relationships, annotations, but also SBOMs, SpdxDocuments, and more.
For serialization purposes, all Elements that are to be serialized into the same file are collected in a `Payload`.
This is just a dictionary that maps each Element's SpdxId to itself.
Use the `write_payload()` functions to serialize a payload.
There currently are two options:
* The `spdx_tools.spdx3.writer.json_ld.json_ld_writer` module generates a JSON-LD file of the payload.
* The `spdx_tools.spdx3.writer.console.payload_writer` module prints a debug output to console. Note that this is not an official part of the SPDX specification and will probably be dropped as soon as a better standard emerges.

You can convert an SPDX v2 document to v3 via the `spdx_tools.spdx3.bump_from_spdx2.spdx_document` module.
The `bump_spdx_document()` function will return a payload containing an `SpdxDocument` Element and one Element for each package, file, snippet, relationship, or annotation contained in the v2 document.

# Dependencies

* PyYAML: https://pypi.org/project/PyYAML/ for handling YAML.
* xmltodict: https://pypi.org/project/xmltodict/ for handling XML.
* rdflib: https://pypi.python.org/pypi/rdflib/ for handling RDF.
* ply: https://pypi.org/project/ply/ for handling tag-value.
* click: https://pypi.org/project/click/ for creating the CLI interface.
* beartype: https://pypi.org/project/beartype/ for type checking.
* uritools: https://pypi.org/project/uritools/ for validation of URIs.
* license-expression: https://pypi.org/project/license-expression/ for handling SPDX license expressions.

# Support

* Submit issues, questions or feedback at https://github.com/spdx/tools-python/issues
* Join the chat at https://gitter.im/spdx-org/Lobby
* Join the discussion on https://lists.spdx.org/g/spdx-tech and
  https://spdx.dev/participate/tech/

# Contributing

Contributions are very welcome! See [CONTRIBUTING.md](./CONTRIBUTING.md) for instructions on how to contribute to the
codebase.

# History

This is the result of an initial GSoC contribution by @[ah450](https://github.com/ah450)
(or https://github.com/a-h-i) and is maintained by a community of SPDX adopters and enthusiasts.
In order to prepare for the release of SPDX v3.0, the repository has undergone a major refactoring during the time from 11/2022 to 07/2023.