# Requirements:
- Running from Windows binary:
  - minimum Windows 10 with latest updates installed.

- Running from source code:
  - Python 3.8.0+ (https://www.python.org/)
  - Additional library listed in requirements.txt
  - IDE Environment: see https://github.com/Nandaka/PixivUtil2/wiki/IDE-Enviroment-(Windows)

- Dependent software
  - FFmpeg (https://www.ffmpeg.org/) - used for converting ugoira to video.
  - [VC++ Redistributable](https://visualstudio.microsoft.com/downloads/#microsoft-visual-c-redistributable-for-visual-studio-2019) - Needed for pyexiv2 to write XMP metadata in Windows (if enabled).

# Capabilities:
- Download by member_id
- Download by image_id
- Download by tags
- Download from list (list.txt)
- Download from bookmarked artists (/bookmark.php?type=user)
  including private/hidden bookmarks.
- Download from bookmarked images (/bookmark.php)
  including private/hidden bookmarks.
- Download from tags list (tags.txt)
- Download new illustrations from bookmarked artist (/bookmark_new_illust.php)
- Download by Title/Caption
- Download by Tag and Member Id
- Download Member Bookmark (/bookmark.php?id=)
- Download by Group Id
- Download from supported artists (FANBOX)
- Download by artist/creator id (FANBOX)
- Download by post id (FANBOX)
- Download from followed artists (FANBOX)
- Re-encoding of all ugoira present in folder
- Batch Download from batch_job.json (experimental)
  See https://github.com/Nandaka/PixivUtil2/wiki/Using-Batch-Job-(Experimental)
- Manage database:
  - Show all member
  - Show all downloaded images
  - Export list (member_id only)
  - Export list (detailed)
  - Export local database (image_id)
  - Show member by last downloaded date
  - Show image by image_id
  - Show member by member_id
  - Show image by member_id
  - Delete member by member_id
  - Delete image by image_id
  - Delete member and image (cascade deletion)
  - Blacklist image by image_id
  - Show all deleted member
  - Export FANBOX post list
  - Delete FANBOX download history by member_id
  - Delete FANBOX download history by post_id
  - Delete Sketch download history by member_id
  - Delete Sketch download history by post_id
  - Clean Up Database (remove db entry if downloaded file is missing)
- Export user bookmark (member_id) to a text files.

# Docker

```sh
$ docker build -t pixivutil2 .
$ docker run -it --rm \
  -v $(pwd):/workdir \
  -w /workdir \
  pixivutil2 \
  /bin/bash -c "python PixivUtil2.py"
```

# WARNING
Overusage can lead to Pixiv blocking your IP for a few hours.

# FAQs

## A. Usage
```
Q1. How to paste Japanese tags to the console window?
    - Click the top-left icon -> select Edit -> Paste (Cannot use Ctrl-V), if
      it show up as question mark -> Change the Language for non-Unicode
      program to Japanese (google it).
    - or use online url encoder (http://meyerweb.com/eric/tools/dencoder/)
      and paste the encoded tag back to the console.
    - or paste it to tags.txt and select download by tags list. Separate each
      tags with space, and separate with new line for new query.

Q2. My password doesn't show up in the console!
    - This is normal. The program still reads it.
    - or you can put in the config.ini if not sure.

Q3. I cannot login to Pixiv!
    - Check your password.
    - Try to login to the Pixiv Website.
    - Try to use the config.ini on the [Authentication] section.
    - Check your date and time setting (e.g.: https://www.timeanddate.com/)
    - Disable Daylight Saving Time and try again.
    - Copy your session values from browser:
      1. Open Firefox.
      2. Go to Pixiv website and login, remember to enable [Remember Me]
          check box.
      3. Press F12 to open Developer Tools, and select the Storage tab.
      4. Click the Cookies and select for the pixiv.net.
      5. Look for Cookie named = PHPSESSID.
      6. Copy the content value. https://imgur.com/a/BppHOoQ
      7. Open config.ini, go to [Authentication] section, paste the value
         to cookie. https://imgur.com/VB2g3qn

Q4. PixivUtil working from local terminal on Linux box but not working when I
    used SSH with PuTTY!
    - export LANG=en_US.UTF-8. PuTTY does not set locales right, when they are
      not set, python does not know what to write (Thanks to nho!)
    - ... and export PYTHONIOENCODING=utf-8, so it can create DB and populate
      it properly (Thanks to Mailia!)

Q5. How to delete member id from Database?
    - Open the application and choose Manage Database (d) then select delete
      Member by Member Id.
    - Open the database (db.sqlite) directly using sqlite browser and use sql
      command to delete it.
    - If you are downloading using Download from List.txt (3), you can create
      ignore_list.txt to skip the member id.

Q6. The app doesn't download all the images! (I want to download SFW images too).
    - Pixiv only allow to search up to 1000 pages if you don't have Pixiv
      Premium.
    - Check your pixiv website settings (refer to https://goo.gl/gQi09v),
      then delete the cookie value in config.ini and retry.
    - Check the value of r18mode in config.ini. Setting it to True will only
      download R-18 images.

Q7. The apps show square/question mark texts in the console output!
    - This is because your Windows is not set to Japanese for the Regional Settings
      in control panel.
    - Since 20161114+ version, you need to set the console font properties to
      use font with unicode support (e.g. Arial Unicode, MS Gothic).

Q8. Where to get FFmpeg software? How to enable `createwebm`?
    - Download the stable version of FFmpeg from https://www.ffmpeg.org/download.html.
    - For Windows:
      - Extract the archive to a folder.
      - Open the extracted folder and open to the `/bin` folder.
      - Copy the application `ffmpeg.exe` to your PixivUtil2 folder.
    - For Linux:
      - Install the package using your favorite package manager.

Q9. The downloaded images are corrupted, how to redownload it again?
    - You can delete the download history in databases by manually delete the image id
      from databases (enter d, followed by 10).
    - Or, you can set alwaysCheckFileSize = True and verifyimage = True in config.ini
      and retry the download.

Q10. I got this error またはメールアドレス、パスワードが正しいかチェックしてください。
    - Use your email address for the username, or check your password in config.ini

Q11. Older windows support (e.g. Win7)?
    - You can try to run from source code with the latest supported python 3.x.
      See the instruction here: https://github.com/Nandaka/PixivUtil2/wiki/IDE-Enviroment-(Windows)

```
## B.Bugs/Source Code/Supports
```
Q1. Where I can report bugs?
    - Please report any bug to https://github.com/Nandaka/PixivUtil2/issues.

Q2. Where I can support/donate to you?
    - You can send it to my PayPal account (nchek2000[at]gmail[dot]com).
    - or visit https://bit.ly/PixivUtilDonation.

Q3. I want to use/modify the source code!
    - Feel free to use/modify the source code as long you give credit to me
      and make the modificated source code open.
    - if you want to add feature/bug fix, you can do fork the repository in
      https://github.com/Nandaka/PixivUtil2 and issue Pull Requests.

Q4. I got ValueError: invalid literal for int() with base 10: '<something>'
    - Please modify _html.py from mechanize library, search for
      'def unescape_charref(data, encoding):' and replace with patch in
      https://pastebin.com/5bT5HFkb.

Q5. I got '<library_name> module no found error'
    - Download the library from the source (see links from the Requirements
      section) and copy the file into your Lib\site-packages directory.
    - Or use pip install (google on how to use).
```
## C.Log Messages
```
Q1: HTTPError: HTTP Error 404: Not Found
    - This is because the file doesn't exist in the pixiv server, usually
       because there is no big images version for the manga mode (currently the
       apps will try to download the big version first then try the normal size
       if failed, this is only for the manga mode and it is normal).

Q2: Error at process_image(): (<type 'exceptions.WindowsError'>, WindowsError
    (32, 'Prosessi ei voi kayttaa tiedostoa, koska se on toisen prosessin
    kaytossa')
    - The file is being used by another process (google translate). Either you
      ran multiple instace of Pixiv downloader from the same folder, or there
      are other processes locking the file/db.sqllite (usually from antivirus
      or some sync/backup application).

Q3: Error at process_image(): (<type 'exceptions.AttributeError'>,
    AttributeError ("'NoneType' object has no attribute 'find'",)
    - Usually this is because of failed login (cookie not valid). Try to change
      your password to simple one for testing, or copy the cookie from browser:
      1. Open Firefox/Chrome.
      2. Login to your Pixiv.
      3. On Pixiv page, press F12 and choose the Storage tab (Firefox), or
         Right click on the leftmost address bar/the (i) icon (Chrome)
      5. Click the View Cookies button.
      6. Look for Cookie named = PHPSESSID.
      7. Copy the content value.
      8. Open config.ini, go to [Authentication] section, paste the value to
         cookie.
    - Or because Pixiv has changed the layout code, so the Pixiv
      downloader cannot parse the page correctly. Please tell me by posting a
      comment if this happens and include the details, such as the member/image
      id, dump html, and log file (check on the application folder).

Q4: URLError: <urlopen error [Errno 11004] getaddrinfo failed>
    - Update version to > pixivutil20221029.
    - This is because the Pixiv downloader cannot resolve the address to
      download the images, please try to restart the network connection or do
      ipconfig /flushdns to refresh the dns cache (windows).

Q5: Error at download_image(): (<class 'socket.timeout'>, timeout('timed out',)
    - This is because the Pixiv downloader didn't receive any reply for
      specified time in config.ini from Pixiv. Please retry the download again
      later.

Q6: httperror_seek_wrapper: HTTP Error 403: request disallowed by robots.txt
    - Set userobots = False in config.ini
```

# Command Line Option
Please refer run with `--help` for latest information.
```
  -h, --help            show this help message and exit
  -s STARTACTION, --startaction=STARTACTION
                        Action you want to load your program with:
                        1 - Download by member_id
                            (required: list of member_ids separated by space
                             optional: --include_sketch to also download Pixiv Sketch)
                        2 - Download by image_id
                            (required: followed by image_ids separated by space)
                        3 - Download by tags
                            (required: tags
                             optional: --use_wildcard_tag, --sp=START_PAGE, and --ep=END_PAGE, --start_date, --end_date)
                        4 - Download from list
                            (required: -f LIST_FILE and followed with optional tag)
                        5 - Download from user bookmark
                            (optional: -p BOOKMARK_FLAG [y/n/o] for private bookmark, --sp=START_PAGE, and --ep=END_PAGE)
                        6 - Download from image bookmark
                            (required: -p BOOKMARK_FLAG [y/n/o] for private bookmark
                             optional: --sp=START_PAGE, and --ep=END_PAGE, and followed with tag)
                        7 - Download from tags list
                            (required: -f LIST_FILE,
                             optional: --sp=START_PAGE, and --ep=END_PAGE, --start_date, --end_date)
                        8 - Download new illust from bookmark
                            (optional: --sp=START_PAGE, and --ep=END_PAGE)
                        9 - Download by Title/Caption
                            (required: title/caption
                             optional: --sp=START_PAGE, and --ep=END_PAGE, --start_date, --end_date)
                        10 - Download by Tag and Member Id
                            (required: member_id, followed by tags
                             optional: --sp=START_PAGE, and --ep=END_PAGE)
                        11 - Download Member's Bookmarked Images
                            (required: followed by member_ids separated by space)
                        12 - Download by Group ID
                            (required: Group ID, limit, and process external[y/n])
                        13 - Download by Manga Series ID
                            (required: Manga Series ID separated by space
                            optional: --sp=START_PAGE, and --ep=END_PAGE)
                        f1 - Download from supported artists (FANBOX)
                            (optional: End Page)
                        f2 - Download by artist/creator id (FANBOX)
                            (required: artist(digits only)/creator ids separated by space,
                             optional: end page)
                        f3 - Download by post id (FANBOX)
                            (required: post ids, separated with space)
                        f4 - Download from followed artists (FANBOX)
                            (optional: End Page)
                        f5 - Download from custom artist list (FANBOX)
                            (optional: End page, path to list)
                        b - Batch Download from batch_job.json (experimental)
                            (optional: --bf=BATCH_FILE)
                        l - Export local database image_id/post_id
                            (required: --up=USE_PIXIV, and --uf=USE_FANBOX, and --us=USE_SKETCH)
                        e - Export online bookmark
                            (required: -p BOOKMARK_FLAG [y/n/o] for private bookmark,
                             optional: --ef=EXPORT_FILENAME)
                        m - Export online user bookmark
                            (required: member_id, optional: --ef=EXPORT_FILENAME)
                        d - Manage database
  -x, --exitwhendone    Exit programm when done.
                        (only useful when DB-Manager)
  -i, --irfanview       start IrfanView after downloading images using
                        downloaded_on_%date%.txt
  -n NUMBEROFPAGES, --numberofpages=NUMBEROFPAGES
                        temporarily overwrites numberOfPage set in config.ini
  -c [PATH], --config [PATH] provide different config.ini
```

# Error Codes
- 100  = Not Logged in.
- 1001 = User ID not exist/deleted.
- 1002 = User Account is Suspended.
- 1003 = Unknown Member Error.
- 1004 = No image found.
- 1005 = Cannot login.
- 2001 = Unknown Error in Image Page.
- 2002 = Not in MyPick List, Need Permission.
- 2003 = Public works can not be viewed by the appropriate level.
- 2004 = Image not found/already deleted.
- 2005 = Image is disabled for under 18, check your setting page (R-18/R-18G).
- 2006 = Unknown Image Error.
- 9000 = Download Failed.
- 9001 = Download Failed: Harddisk related.
- 9002 = Download Failed: Network related.
- 9005 = Server Error.

# config.ini
## [Authentication]
- username

  Your pixiv username. Needed for OAuth. Please make sure the combination of username and password is valid in case of OAuth error. If you get error 103, please try changing username from pixiv ID to email address or the other way around.
- password

  Your pixiv password, in clear text! Needed for OAuth. Please make sure the combination of username and password is valid in case of OAuth error.
- cookie

  Your cookies for pixiv login, will be automatically updated in the login. See https://github.com/Nandaka/PixivUtil2/issues/814#issuecomment-711182644 for details.
- cookieFanbox

  Cookie for fanbox.cc, normally no need to fill in.
- refresh_token

  Used for OAuth refresh token to avoid relogin too many time. Automatically generated upon succesful OAuth login.

## [Pixiv]
- numberofpage

  Number of page to be processed, put `0` to process all pages.

- r18mode

  Only list images tagged R18, for member, member's bookmark, and search by tag. Set to `True` to enable.

- r18Type

  Allow filtering for R-18 type (R-18 or R-18G)
  Set `r18Type` with value `0` = both R18 and R-18G, `1` = only R18, or `2` = only R18G

- dateformat

  Pixiv DateTime format, leave blank to use default format (YYYY-MM-DD).
  Refer to http://strftime.org/ for syntax. Quick Reference:
  - %d = Day, %m = Month, %Y = Year (4 digit)
  - %H = Hour (24h), %M = Minute, %S = Seconds

- autoAddMember

  Automatically save member id to db for all download.

- aiDisplayFewer

  if true, filter out AI-generated images from downloading.

## [FANBOX]
- filenameFormatFanboxContent

  Similar to filename format, but for files inside FANBOX posts.
- filenameFormatFanboxCover

  Similar to filename format, but for FANBOX post cover images
- filenameFormatFanboxInfo

  Similar to filename format, but for info dumps.
- writeHtml

  A switch to decide whether to write FANBOX posts into HTMLs or not.
  - If set to `True`, article type posts will for sure be written into HTMLs, while non-article type posts are controlled with `minTextLengthForNonArticle` and `minImageCountForNonArticle`.
  - If set to `False`, no post will be written into HTMLs.
  - `filenameFormatFanboxInfo` will be used for filename.
  - For HTML format, please refer to 'HTML Format' section
- minTextLengthForNonArticle

  Works with `minImageCountForNonArticle`.
  When 'writeHtml' is True, a non-article post should contain text longer than this value to be written into HTML.
- minImageCountForNonArticle

  Works with `minTextLengthForNonArticle`.
  When `writeHtml` is True, a non-article post should contain at least this many files/images to be written into HTML.
- useAbsolutePathsInHtml

  Set to `True` to use absolute paths in HTMLs.
  Set to `False` to use relative paths.
- downloadCoverWhenRestricted

  Set to `True` to download FANBOX post cover images even if they are restricted.
- checkDBProcessHistory
  Each FANBOX post has a updated_date value, which will be recorded/updated in database after it is processed.
  - When this is `True`, the values in database would be checked when processing each post. If record is no earlier than the newly retrieved date, which means that the post has not been processed at all or changed since last time, the post would be skipped.
  - When this is `False`, posts will be processed anyways.
- listPathFanbox

  The list file for fanbox creators. One creator per line.
  Doesn't support custom path.

## [Network]
- useproxy

  Set `True` to use proxy server, or `False` to disable it.
- proxyaddress

  Proxy server address, use this format:
  - `http://<username>:<password>@<proxy_server>:<port>` or
  - `socks5://<username>:<password>@<proxy_server>:<port>` or
  - `socks4://<username>:<password>@<proxy_server>:<port>`
- useragent

  Browser user agent to spoof. You can check it from https://www.whatismybrowser.com/detect/what-is-my-user-agent
- userobots

  Download robots.txt for mechanize.
- timeout

  Time to wait before giving up the connection, in seconds.
- retry

  Number of retries.
- retrywait

  Waiting time for each retry, in seconds.
- downloadDelay

  Set random delay up to n seconds for each image post.
  Set to 0 to disable.
- checkNewVersion

  Set to `True` to check new releases in github.
- notifyBetaVersion

  Set to `False` to ignore beta releases.
- openNewVersion

  Set to `False` to disable opening new releases in browser.
- enableSSLVerification

  Enable SSL verication, only set to `False` if you always encounter SSL Error (this disable the security)

## [Debug]
- logLevel

  Set log level, valid values are CRITICAL, ERROR, WARNING, INFO, DEBUG, and NOTSET
- enableDump

  Enable HTML Dump. Set to False to disable.
- skipDumpFilter

  Skip HTML Dump based on error code (using regex format).
  E.g.: 1.*|2.* => skip all HTML dump for error code 1xxx/2xxx.
- dumpMediumPage

  Dump all medium page for debugging. Set to True to enable.
- dumpTagSearchPage

  Dump tags search page for debugging.
- debughttp

  Print http header, useful for debuggin. Set 'False' to disable.

## [IrfanView]
- IrfanViewPath

  Set directory where IrfanView is installed (needed to start IrfanView)
- startIrfanView

  Set to `True` to start IrfanView with downloaded images when exiting pixivUtil
  - This will create download-lists
  - Be sure to set IrfanView to load Unicode-Plugin on startup when there are unicode-named files!
- startIrfanSlide

  Set to `True` to start IrfanView-Slideshow with downloaded images when exiting pixivUtil.
  - This will create download-lists
  - Be sure to set IrfanView to load Unicode-Plugin on startup when there are unicode-named files!
  - Slideshow-options will be same as you have set in IrfanView before!
- createDownloadLists

  Set to `True` to automatically create download-lists.

## [Settings]
- downloadlistdirectory

  list.txt path, also used for download-lists needed for `createDownloadLists` and IrfanView-Handling
  If leaved blank it will create download-lists in pixivUtil-directory.
- uselist

  Set to `True` to parse list.txt.
  This will update the DB content from the list.txt (member_id and custom folder).
- processfromdb

  Set `True` to use the member_id from the DB.
- rootdirectory

  Your root directory for saving the images.
- downloadavatar

  Set to `True` to download the member avatar as 'folder.jpg'
- usesuppresstags

  Remove the suppressed tags from %tags% meta for filename.
  The list is taken from suppress_tags.txt, each tags is separated by new line.
- tagsLimit

  Number of tags to be used for %tags% meta in filename.
  Use -1 to use all tags.
- writeImageJSON

  Set to `True` to export the compact image information to JSON file.
  The filename is following `filename(Manga)Infoformat` + .json.
  If you want the original info from source, use with `writeRawJSON`.
- writeimageinfo

  Set to `True` to export the compact image information to text file.
  The filename is following `filename(Manga)Infoformat` + .txt.
  If you want the original info from source, use with `writeRawJSON`.
- writeRawJSON

  Set to `True` to export the original JSON untouched of the image for `writeImageJSON`.
- RawJSONFilter

  Enter the JSON keys which you want to filter out for `writeRawJSON`. Keys are seperated by a comma.
- includeSeriesJSON

  Set to `True` to export the series information to JSON. Non-series artwork doesn't have this info.
  The filename is following `filenameSeriesJSON` + .json.
- writeImageXMP

  Set to `True` to export the image information to a .XMP sidecar file, this does not add XMP metadata to the image header.
- writeImageXMPPerImage

  Set to `True` to export the image information to a .XMP sidecar file, one per image in the album. The data contained within the file is the same but some software requires matching file names to detect the metadata. If set to `True`, then `writeImageXMP` is ignored.
  Additionally, enabling this option will create a .XMP sidecar for every ugoira encoding enabled, and allow you to customise the name of each file using `%image_ext%`. For example, if you enable `createWebp` and `createGif`, then set your `filenameInfoFormat` to something like `%urlFilename%.%image_ext%`, then you will end up with `<image ID>.gif.xmp` and `<image ID>.webp.xmp` files created.
- verifyimage

  Check if downloaded files are valid image or zip. Set the value to `True` to enable.
- writeUrlInDescription

  Write all url found in the image description to a text file at the root directory. Set to `True` to enable. The list will be saved to to the application folder as url_list_<timestamp>.txt
- stripHTMLTagsFromCaption

  Remove all HTML tags and their contents from the image caption/description when writing metadata to files. The contents of any links will be lost, so consider enabling writeUrlInDescription to retain them.
- urlBlacklistRegex

  Used to filter out the url in the description using regular expression.
- dbPath

  Use different database.
- setLastModified

  Set last modified timestamp based on pixiv upload timestamp to the file.
- useLocalTimezone

  Use local timezone in the .txt file of `writeimageinfo` and .XMP file of `writeImageXMP`.
- defaultSketchOption

  Skip the "Include Pixiv Sketch" prompt when downloading by `member_id` option by using a default option. Set the value to `y` to always include sketches or `n` to exclude sketches from the download.

## [DownloadControl]
- minFileSize

  Skip if file size is less than minFileSize, set `0` to disable.
- maxFileSize

  Skip if file size is more than minFileSize, set `0` to disable.
- checkLastModified

  If the last-modified timestamp of the local files is the same with the uploaded date of the artwork, it'll log "match" and skip to process the current image_id.
  Require `setlastmodified = True` in config.ini to work properly
- alwaysCheckFileSize

  Actually, it'll always check the file size. But if `this` is false, if the `overwrite` is also false and this file is recorded in db, it'll skip to process the current image_id.
  This will override the image_id checking from db (always fetch the image page to check the remote size).
- overwrite

  If is true, when found file size different, it'll just delete the file (unless the backupOldFile is true), then start to re-download the image.
- backupOldFile

  Set to True to backup old file if the file size is different.
  Old filename will be renamed to filename.unix-time.extension.
- daylastupdated

  Only process member_id which were processed at least x days since the last check.
- checkUpdatedLimit

  Jump to the next member id if already see n-number of previously downloaded images.
  `alwaysCheckFileSize` must be set to False.
- useblacklisttags

  Skip image if containing blacklisted tags.
  The list is taken from `blacklist_tags.txt`, each tags is separated by new line.
- useblacklisttitles

  Skip image if the title contains a blacklisted character sequence.
  The list is taken from `blacklist_titles.txt`, each sequence is separated by new line.
- useblacklisttitlesregex

  Make the title blacklist check interpret each sequence as a regular expression.
- dateDiff

  Process only new images within the given date difference.
  Set `0` to disable. Skip to next member id if in 'Download by Member', stop processing if in 'Download New Illust' mode.
- enableInfiniteLoop

  Enable infinite loop for download by tags.
  Only applicable for download in descending order (newest first).
- useBlacklistMembers

  Skip image by member id based on `blacklist_members.txt` in the same folder of the application.
- downloadResized

  Download the medium size, rather than the original size.
- skipUnknownSize

  Skip downloading if the remote size is not known when `alwaysCheckFileSize` is set to True.

- enablePostProcessing

  If true, it enabled post processing cmd for every downloaded files. Default: False.

- postProcessingCmd

  command to execute. add %filename% to pass the downloaded filename.
  **NO ERROR HANDLING AT ALL, use on your own risk.**

- extensionFilter

  Provide a | seperated list of acceptable file extensions to download. Eg. jpg|png|gif|ugoira

- downloadBuffer

  Download buffer before it write to disk in kiloByte, default is 512kB.
  You can change it based on your download speed. Mainly useful for smoother progress bar.
  Usually no need to change this value.

## [FFmpeg]
- ffmpeg

  ffmpeg executable path.
- ffmpegcodec

  Codec to be used for encoding, default is using `libvpx-vp9`.
- ffmpegExt

  The file extension (container format) to use for encoding. default: `webm`.
- ffmpegparam

  Parameter to be used to encode webm, default: `-lossless 0 -crf 15 -b 0 -vsync 0`.
- mkvcodec

  Codec to be used for encoding mkv, default is using `copy`.
- mkvparam

  Parameter to be used to encode mkv, default: ` `.
- webpcodec

  Codec to be used for encoding webm, default is using `libwebp`.
- webpparam

  Parameter to be used to encode webm, default: `-lossless 0 -compression_level 5 -quality 100 -loop 0 -vsync 0`.

## [Ugoira]
- writeugoirainfo

  If set to `True`, it will write the info of ugoira frames to a `filename(Manga)Infoformat`+.zip.js file. `writeImageJSON` contains this info as well.
- createugoira

  If set to `True`, it will create .ugoira file.
  This is Pixiv own format for animated images. You can use Honeyview to see the animation.
- createmkv

  Set to True to create mkv file (video format). The default settings is lossless(no encoding), it will pack the images in the container. Very large file size.
  Required `createUgoira = True` and ffmpeg executeable.
- createwebm

  Set to True to create webm file (video format). The default encoding settings is lossy encoding but high quality with smallest file size.
  Required `createUgoira = True` and ffmpeg executeable.
- createwebp

  Set to True to create webp file (image format). The default encoding settings is lossy encoding but high quality with smaller file size.
  Required `createUgoira = True` and ffmpeg executeable.
- creategif

  Set to True to convert ugoira file to gif. The default encoding settings is lossy encoding but moderate quality with smaller file size.
  Required `createUgoira = True` and ffmpeg executeable.
- createapng

  Set to True to convert ugoira file to animated png. The default encoding settings is lossless encoding but very large file size.
  Required `createUgoira = True` and ffmpeg executeable.
- deleteugoira

  Set to True to delete the created .ugoira after conversion.
- deleteZipFile

  If set to `True`, it will delete the orignal .zip (i.e. the actual image) file.
  Only active if `createUgoira = True`.

## [Filename]
- filenameformat

  The format for the filename, reserved/illegal character will be replaced with underscore '_', repeated space will be trimmed to single space. The filename (+full path) will be trimmed to the first 250 character (Windows limitation).
  Refer to Filename Format Syntax for available format.
- filenamemangaformat

  Similar to filename format, but for manga pages.
- filenameinfoformat

  Similar to filename format, but for info dumps.
- filenameSeriesJSON

  Similar to filename format, but for series JSON dumps.
- avatarNameFormat

  Similar to filename format, but for the avatar image.
  Not all formats are available.
- backgroundNameFormat

  Similar to filename format, but for the background image.
  Not all formats are available.
- tagsseparator

  Separator for each tag in filename, put %space% for space and %ideo_space% for ideographic space ("　").
- createmangadir

  Create a directory if the imageMode is manga. The directory is created by splitting the image_id by '_pxx' pattern.
  This setting is depends on %urlFilename% format.
- usetagsasdir

  Append the query tags in tagslist.txt to the root directory as save folder.
- urlDumpFilename

  Define the dump filename, use python strftime() format.
  Default value is 'url_list_%Y%m%d'
- filenameFormatSketch

  Similar to filename format, but for Pixiv Sketch.
- customBadChars

  For sanitizing filenames with custom rules. Supports regular expressions.
  For detailed syntax, please refer to 'Bad chars' section.
- customCleanUpRe

  TODO.

# Filename Format Syntax
Available for filenameFormat, filenameMangaFormat, avatarNameFormat, filenameInfoFormat,
filenameFormatFanboxCover, filenameFormatFanboxContent and filenameFormatFanboxInfo:
```
-> %member_token%
   Member token, might change.
-> %member_id%
   Member id, in number.
-> %artist%
   Artist name, might change too.
-> %urlFilename%
   The actual filename stored in server without the file extensions.
-> %date%
   Current date in YYYYMMMDD format.
-> %date_fmt{format}%
   Current date using custom format.
   Use Python string format notation, refer: https://goo.gl/3UiMAb
   e.g. %date_fmt{%Y-%m-%d}%
-> %image_ext%
   The image's file extension (jpg, png, etc.), the "." is not included.
   The correct file extension is already appended to the end of all files.
   This is available if you want to add more, or want to add the image's file extension to info files etc.
```
Available for filenameFormat and filenameMangaFormat:
```
-> %image_id%
   Image id, in number. (Post id for FANBOX and sketches)
-> %title%
   Image title, usually in japanese character.
-> %tags%
   Image tags, usually in japanese character. (not implemented for FANBOX yet)
-> %works_date%
   Works date, complete with time.
-> %works_date_only%
   Only the works date.
-> %works_date_fmt{<format>}%
   works date using custom format.
   Use Python string format notation, refer: https://goo.gl/3UiMAb
   e.g. %works_date_fmt{%Y-%m-%d}%
-> %works_res%
   Image resolution, will be containing the page count if manga.
-> %works_tools%
   Tools used for the image.
-> %R-18%
   Append R-18/R-18 based on image tag, can be used for creating directory
   by appending directory separator, e.g.: %R-18%\%image_id%.
-> %page_big%
   for manga mode, add big in the filename.
-> %page_index%
   for manga mode, add page number with 0-index. It will auto-pad with 0 based on the total count.
-> %page_number%
   for manga mode, add page number with 1-index. It will auto-pad with 0 based on the total count.
-> %bookmark%
   for bookmark mode, add 'Bookmarks' string.
-> %original_member_id%
   for bookmark mode, put original member id.
-> %original_member_token%
   for bookmark mode, put original member token.
-> %original_artist%
   for bookmark mode, put original artist name.
-> %searchTags%
   for download by tags and bookmarked images, put searched tags.
-> %bookmark_count%
   Bookmark count, will have overhead except on download by tags.
-> %image_response_count%
   Image respose count, will have overhead except on download by tags.
-> %manga_series_order%
   the order in the manga series.
-> %manga_series_id%
   original manga series id.
-> %manga_series_title%
   original manga series title, different from work title.
-> %AI%
   Add 'AI' for AI-generated images (aiType==2).
```
Specific for PixivSketch (option 1 if PixivSketch included, s1, and s2 ):
```
-> %sketch_member_id%
   Pixiv Sketch artist id, might be different from Pixiv's artist id.
```
Specific for Fanbox:
```
-> %fanbox_name%
   Fanbox name, might be different from Pixiv's artist name.
   Useful if the artist is suspended from Pixiv and there is no record in the DB to avoid interuption.
```
# list.txt Format
- This file should be build in the following way, white space will be trimmed,
  see example:
```
member_id1 directory1
member_id2 directory2
  ...
#comment - lines starting with # will be ignored
```
- member_id = in number only
- directory = path to download-directory for member_id
  - %root%\directory will save directory in rootFolder specified in config.ini
    \directory will save the folder in the root of your PixivUtil-drive
  - C:\directory will save the folder in drive C: (change to any other
    drive as you wish)
  - .\directory will save the folder in same directory as PixivUtil2.exe
  - directory-path can end with \ or not

- Examples for list:
```
### START EXAMPLE LIST####
# this is a comment line, lines starting with # will be ignored
# here is the first member:
123456
# you can see, the line has only the member id
# usually I use it the following way:
#
# username (so I can recognize it ;) )
123456
#
# next 2 lines contain a special folder for this member
123456 .\test
123456 ".\test"
# now all images from member no. 123456 will be safed in directory "test" in the
# same directory as PixivUtil2
# as you can see you can use it with "" or without ;)
#
# next will be stored at the same partition as PixivUtil, but the directory is
# located in root-part of it
123456 \test
123456 "\test"
# this will lead to "C:\test" when pixivUtil is located on "C:\"
#
# next line uses complete path to store the files
123456 F:\new Folder\test
123456 "F:\new Folder\test"
# this will set the folder everywhere on your partitions
#
123456 %root%\special folder
123456 "%root%\special folder"
# this will set the download location to "special folder" in your rootDirectory
# given in config
http://www.pixiv.net/member.php?id=123456
http://www.pixiv.net/member_illust.php?id=123456
# also support url format.
### END EXAMPLE LIST####
```

# tags.txt Format
- This file will be used as source for Download from tags list (7)
- Separate tags with space, ensure to set Use Wildcard to 'y'.
- Each line will be treated as one search.
- Save the files with UTF-8 encoding.

# suppress_tags.txt Format
- This file is used for suppressing the tags from being used in %tags%.
- If matches, the tags will be removed from filename.
- Each line is one tag only.
- Save the files with UTF-8 encoding

# blacklist_tags.txt Format
- This file is used for tag blacklist checking for downloading image.
- If matches, the image will be skipped.
- Each line is one tag only.
- Save the files with UTF-8 encoding

# blacklist_members.txt Format
- similar to list.txt, but without custom folder.

# HTML Format
- A simple default format will be used when no 'template.html' is provided.
- Urls originally in the post will be overwritten with local paths.
- Currently available syntaxes are:
```
-> %coverImage%
   A 'div' tag with its 'class' set to 'cover', and a child 'img' tag with
   the url to the cover image as its 'src' attribute.
-> %coverImageUrl%
   Simply the url to the cover image in clear text.
-> %artistName%
   Same as %artist% in 'Filename Format Syntax' in clear text.
-> %imageTitle%"
   Title of the post in clear text.
-> "%worksDate%"
   Published date of the post in clear text.
-> %body_text(article)%
   This works for article type posts only.
   A 'div' tag with its 'class' set to 'article', and the post's content,
   which is already formatted HTML if the post is article, as its inner text.
-> %images(non-article)%
   This works for none-article type posts only.
   A 'div' tag with its 'class' set to 'non-article images', and 'a' tags
   of all files in the post as its children tokens.
   For each 'a' tag, its 'href' would be url to the file, and the inner text
   would be an 'img' tag with its 'src' set to the url to the file if the
   file's extension is 'jpg', 'jpeg', 'png' or 'bmp'. Otherwise the inner text
   would simply be the url to the file.
-> %text(non-article)%
   This works for none-article type posts only.
   A 'div' tag with its 'class' set to 'non-article text' and all paragraphs
   of text put in 'p' tags as its children tokens.
```
- If there is a 'div' tag with 'main' in its 'class' in the template, 'article' or
  'non-article' would be appended to its 'class' depending on the type of the post.

# Bad chars
- Originally for removing single bad chars for use between different OSs.
- Now also supports strings and regular expressions.
- The value set in option `customBadChars` would be parsed from left to right.
- Currently available syntaxes are:
```
-> %replace<default>(your_default_replace_with)%
   Use this syntax to define default value to replace with.
   If this syntax gets used multiple times in the option value, the first value would be used.
   If this value is not set, "_" would be used.
-> %pattern<you_group_name>(your_pattern)%
-> %replace<you_group_name>(your_replace_with)%
   Use these two syntaxes to set groups of rules. Supports regular expression.
   You should not use "default" as group names, otherwise the first replace would
   be parsed as default value to replace with, while the others would be ignored.
   Groups with no "pattern" would be ignored.
   Groups with no "replace" use default value.
   If multiple "pattern"s or "replace"s share the same group name, the last value set
   would be used.
```
- Chars/string not wrapped with syntaxes above would be considered single chars
  to be replaced with global replacement char/string, "_" if unset.
- When configuration file gets written to file, `customBadChars` would be
  replaced with parsed valid value. Single chars would be placed first, followed by
  `%replace<default>(your_default_replace_with)%`, and each group.
- Examples:
```
# If you just want to replace some single chars with "_"
\@[]
# If you want to replace them with "@":
\@[]%replace<default>(@)%
# If you want to replace certain words:
# This example would first replace all "maze" with "labyrinth",
# then all "labyrinth" with "nevermind"
%pattern<1>(maze)%%replace<1>(labyrinth)%%pattern<2>(labyrinth)%%replace<2>(nevermind)%
# If you want to replace characters within certain unicode range,
# then remove all continuous "_"s with a single "_":
%pattern<unicode>([\U0001d400-\U0001ffff])%%pattern<1>(_+)%%replace<1>(_)%
```

# Development
PixivUtil2 posesses robust test suite. To run it, one needs pytest suite:
```
pip install --user pytest

pytest -v ./test_*
```

# Credits/Contributor
- Nandaka (Main Developer) - https://nandaka.devnull.zone
- Yavos (Contributor)
- Joe (Contributor)
- Hamuko <hamuko@burakku.com>
- Kwang Ketcham <prototype27+github@prototype27.com>
- woky <nechtom@gmail.com>
- a.evseev <nmbr213@gmail.com>
- pixtrix
- Abram Wiebe <awiebe@burningthumb.com>
- Masaki Takano
- hi117
- Wildfoot
- J.Gocke
- Magnus Boman
- Abdulah Jasim
- Yifei Fu
- nixxquality
- DukeValentine
- NHOrus
- whinette
- yzaoui
- Kieri Suizahn
- amatuerCoder
- Alex
- wmjdgla
- fireattack
- Jared Shields
- DenDen047
- Baa

** If I forget someone, please send me a pull request with the commit/merge id.

# License Agreement
See LICENSE.

[![Run on Repl.it](https://repl.it/badge/github/Nandaka/PixivUtil2)](https://repl.it/github/Nandaka/PixivUtil2)


  River is a Python library for online machine learning. It aims to be the most user-friendly library for doing machine learning on streaming data. River is the result of a merger between creme and scikit-multiflow.

## ⚡ Quickstart

As a quick example, we'll train a logistic regression to classify the [website phishing dataset](http://archive.ics.uci.edu/ml/datasets/Website+Phishing). Here's a look at the first observation in the dataset.

```python
>>> from pprint import pprint
>>> from river import datasets

>>> dataset = datasets.Phishing()

>>> for x, y in dataset:
...     pprint(x)
...     print(y)
...     break
{'age_of_domain': 1,
 'anchor_from_other_domain': 0.0,
 'empty_server_form_handler': 0.0,
 'https': 0.0,
 'ip_in_url': 1,
 'is_popular': 0.5,
 'long_url': 1.0,
 'popup_window': 0.0,
 'request_from_other_domain': 0.0}
True

```

Now let's run the model on the dataset in a streaming fashion. We sequentially interleave predictions and model updates. Meanwhile, we update a performance metric to see how well the model is doing.

```python
>>> from river import compose
>>> from river import linear_model
>>> from river import metrics
>>> from river import preprocessing

>>> model = compose.Pipeline(
...     preprocessing.StandardScaler(),
...     linear_model.LogisticRegression()
... )

>>> metric = metrics.Accuracy()

>>> for x, y in dataset:
...     y_pred = model.predict_one(x)      # make a prediction
...     metric.update(y, y_pred)  # update the metric
...     model.learn_one(x, y)              # make the model learn

>>> metric
Accuracy: 89.28%

```

Of course, this is just a contrived example. We welcome you to check the [introduction](https://riverml.xyz/dev/introduction/installation/) section of the documentation for a more thorough tutorial.

## 🛠 Installation

River is intended to work with **Python 3.8 and above**. Installation can be done with `pip`:

```sh
pip install river
```

There are [wheels available](https://pypi.org/project/river/#files) for Linux, MacOS, and Windows. This means you most probably won't have to build River from source.

You can install the latest development version from GitHub as so:

```sh
pip install git+https://github.com/online-ml/river --upgrade
pip install git+ssh://git@github.com/online-ml/river.git --upgrade  # using SSH
```

This method requires having Cython and Rust installed on your machine.

## 🔮 Features

River provides online implementations of the following family of algorithms:

- Linear models, with a wide array of optimizers
- Decision trees and random forests
- (Approximate) nearest neighbors
- Anomaly detection
- Drift detection
- Recommender systems
- Time series forecasting
- Bandits
- Factorization machines
- Imbalanced learning
- Clustering
- Bagging/boosting/stacking
- Active learning

River also provides other online utilities:

- Feature extraction and selection
- Online statistics and metrics
- Preprocessing
- Built-in datasets
- Progressive model validation
- Model pipelines

Check out [the API](https://riverml.xyz/latest/api/overview/) for a comprehensive overview

## 🤔 Should I be using River?

You should ask yourself if you need online machine learning. The answer is likely no. Most of the time batch learning does the job just fine. An online approach might fit the bill if:

- You want a model that can learn from new data without having to revisit past data.
- You want a model which is robust to [concept drift](https://www.wikiwand.com/en/Concept_drift).
- You want to develop your model in a way that is closer to what occurs in a production context, which is usually event-based.

Some specificities of River are that:

- It focuses on clarity and user experience, more so than performance.
- It's very fast at processing one sample at a time. Try it, you'll see.
- It plays nicely with the rest of Python's ecosystem.

## 🔗 Useful links

- [Documentation](https://riverml.xyz)
- [Package releases](https://pypi.org/project/river/#history)
- [awesome-online-machine-learning](https://github.com/online-ml/awesome-online-machine-learning)
- [2022 presentation at GAIA](https://www.youtube.com/watch?v=nzFTmJnIakk&list=PLIU25-FciwNaz5PqWPiHmPCMOFYoEsJ8c&index=5)
- [Online Clustering: Algorithms, Evaluation, Metrics, Applications and Benchmarking](https://dl.acm.org/doi/10.1145/3534678.3542600) from [KDD'22](https://kdd.org/kdd2022/).

## 👐 Contributing

Feel free to contribute in any way you like, we're always open to new ideas and approaches.

- [Open a discussion](https://github.com/online-ml/river/discussions/new) if you have any question or enquiry whatsoever. It's more useful to ask your question in public rather than sending us a private email. It's also encouraged to open a discussion before contributing, so that everyone is aligned and unnecessary work is avoided.
- Feel welcome to [open an issue](https://github.com/online-ml/river/issues/new/choose) if you think you've spotted a bug or a performance issue.
- Our [roadmap](https://github.com/orgs/online-ml/projects/3?query=is%3Aopen+sort%3Aupdated-desc) is public. Feel free to work on anything that catches your eye, or to make suggestions.

Please check out the [contribution guidelines](https://github.com/online-ml/river/blob/main/CONTRIBUTING.md) if you want to bring modifications to the code base.

## 🤝 Affiliations

## 💬 Citation

If River has been useful to you, and you would like to cite it in a scientific publication, please refer to the [paper](https://www.jmlr.org/papers/volume22/20-1380/20-1380.pdf) published at JMLR:

```bibtex
@article{montiel2021river,
  title={River: machine learning for streaming data in Python},
  author={Montiel, Jacob and Halford, Max and Mastelini, Saulo Martiello
          and Bolmier, Geoffrey and Sourty, Raphael and Vaysse, Robin and Zouitine, Adil
          and Gomes, Heitor Murilo and Read, Jesse and Abdessalem, Talel and others},
  year={2021}
}
```

## 📝 License

River is free and open-source software licensed under the [3-clause BSD license](https://github.com/online-ml/river/blob/main/LICENSE).


简体中文 | [English](README_EN.md)

**飞桨高性能图像分割开发套件，端到端完成从训练到部署的全流程图像分割应用。**

[![License](https://img.shields.io/badge/license-Apache%202-blue.svg)](LICENSE)
[![Version](https://img.shields.io/github/release/PaddlePaddle/PaddleSeg.svg)](https://github.com/PaddlePaddle/PaddleSeg/releases)
![python version](https://img.shields.io/badge/python-3.6+-orange.svg)
![support os](https://img.shields.io/badge/os-linux%2C%20win%2C%20mac-yellow.svg)
![stars](https://img.shields.io/github/stars/PaddlePaddle/PaddleSeg?color=ccf)

##  最新动态

* [2023-10-29] :fire:  PaddleSeg 2.9版本发布！详细发版信息请参考[Release Note](https://github.com/PaddlePaddle/PaddleSeg/releases)。
  * 增加对多标签分割[Multi-label segmentation](./configs/multilabelseg)，提供数据转换代码及结果可视化，实现对一系列语义分割模型的多标签分割支持。
  * 发布轻量视觉大模型[MobileSAM](./contrib/SegmentAnything)，实现更快速的SAM推理。
  * 支持量化蒸馏训练压缩功能[Quant Aware Distillation Training Compression](./deploy/slim/act)，对PP-LiteSeg、PP-MobileSeg、OCRNet、SegFormer-B0增加量化训练压缩功能，提升推理速度。

* [2023-09-01] :fire::fire: 飞桨AI套件 [PaddleX](https://aistudio.baidu.com/intro/paddlex) 发布全新版本，围绕飞桨精选模型提供了一站式、全流程、高效率的开发平台，希望助力AI技术快速落地、使人人成为AI Developer！欢迎大家使用和共建。
* [2022-04-11] PaddleSeg 2.8版本发布视觉大模型[Segment Anything Model](./contrib/SegmentAnything)、轻量级语义分割SOTA模型[PP-MobileSeg](./configs/pp_mobileseg)、工业质检全流程解决方案[QualityInspector v0.5](./contrib/QualityInspector)、通用的全景分割解决方案[PanopticSeg v0.5](./contrib/PanopticSeg)。
* [2022-11-30] PaddleSeg 2.7版本发布实时人像抠图模型[PP-MattingV2](./Matting/)、3D医疗影像分割方案[MedicalSegV2](./contrib/MedicalSeg/)、轻量级语义分割模型[RTFormer](./configs/rtformer/)。
* [2022-07-20] PaddleSeg 2.6版本发布实时人像分割SOTA方案[PP-HumanSegV2](./contrib/PP-HumanSeg)、高性能智能标注工具[EISeg v1.0](./EISeg)正式版、ImageNet分割伪标签数据预训练方法PSSL，开源PP-MattingV1代码和预训练模型。
* [2022-04-20] PaddleSeg 2.5版本发布超轻量级语义分割模型[PP-LiteSeg](./configs/pp_liteseg)，高精度抠图模型PP-MattingV1，3D医疗影像开发套件MedicalSegV1，交互式分割工具EISeg v0.5。
* [2022-01-20] PaddleSeg 2.4版本发布交互式分割工具EISeg v0.4，超轻量级人像分割方案PP-HumanSegV1，以及大规模视频会议数据集[PP-HumanSeg14K](./contrib/PP-HumanSeg/paper.md#pp-humanseg14k-a-large-scale-teleconferencing-video-dataset)。

##  简介

**PaddleSeg**是基于飞桨PaddlePaddle的端到端图像分割套件，内置**45+模型算法**及**140+预训练模型**，支持**配置化驱动**和**API调用**开发方式，打通数据标注、模型开发、训练、压缩、部署的**全流程**，提供**语义分割、交互式分割、Matting、全景分割**四大分割能力，助力算法在医疗、工业、遥感、娱乐等场景落地应用。

##  特性

* **高精度**：跟踪学术界的前沿分割技术，结合高精度训练的骨干网络，提供45+主流分割网络、150+的高质量预训练模型，效果优于其他开源实现。

* **高性能**：使用多进程异步I/O、多卡并行训练、评估等加速策略，结合飞桨核心框架的显存优化功能，大幅度减少分割模型的训练开销，让开发者更低成本、更高效地完成图像分割训练。

* **模块化**：源于模块化设计思想，解耦数据准备、分割模型、骨干网络、损失函数等不同组件，开发者可以基于实际应用场景出发，组装多样化的配置，满足不同性能和精度的要求。

* **全流程**：打通数据标注、模型开发、模型训练、模型压缩、模型部署全流程，经过业务落地的验证，让开发者完成一站式开发工作。

##  技术交流

- 飞桨低代码开发工具（PaddleX）—— 面向国内外主流AI硬件的飞桨精选模型一站式开发工具。包含如下核心优势：
  - 【产业高精度模型库】：覆盖10个主流AI任务 40+精选模型，丰富齐全。
  - 【特色模型产线】：提供融合大小模型的特色模型产线，精度更高，效果更好。
  - 【低代码开发模式】：图形化界面支持统一开发范式，便捷高效。
  - 【私有化部署多硬件支持】：适配国内外主流AI硬件，支持本地纯离线使用，满足企业安全保密需要。

- PaddleX官网地址：https://aistudio.baidu.com/intro/paddlex

- PaddleX官方交流频道：https://aistudio.baidu.com/community/channel/610

##  产品矩阵

        模型

        组件

        特色案例

        语义分割模型

PP-LiteSeg

PP-MobileSeg

DeepLabV3P

OCRNet

MobileSeg

ANN

Att U-Net

BiSeNetV1

BiSeNetV2

CCNet

DANet

DDRNet

DecoupledSeg

DeepLabV3

DMNet

DNLNet

EMANet

ENCNet

ENet

ESPNetV1

ESPNetV2

FastFCN

Fast-SCNN

GCNet

GINet

GloRe

GSCNN

HarDNet

HRNet-FCN

HRNet-Contrast

ISANet

PFPNNet

PointRend

PotraitNet

PP-HumanSeg-Lite

PSPNet

PSSL

SegFormer

SegMenter

SegNet

SETR

SFNet

STDCSeg

U2Net

UNet

UNet++

UNet3+

UperNet

RTFormer

UHRNet

TopFormer

MscaleOCRNet-PSA

CAE

MaskFormer

ViT-Adapter

HRFormer

LPSNet

SegNeXt

K-Net

        交互式分割模型

EISeg

RITM

EdgeFlow

        图像抠图模型

PP-MattingV2

PP-MattingV1

DIM

MODNet

PP-HumanMatting

RVM

        全景分割

Mask2Former

Panoptic-DeepLab

        骨干网络

HRNet

ResNet

STDCNet

MobileNetV2

MobileNetV3

ShuffleNetV2

GhostNet

LiteHRNet

XCeption

VIT

MixVIT

Swin Transformer

TopTransformer

HRTransformer

MSCAN

        损失函数

Binary CE Loss

Bootstrapped CE Loss

Cross Entropy Loss

Relax Boundary Loss

Detail Aggregate Loss

Dice Loss

Edge Attention Loss

Focal Loss

MultiClassFocal Loss

GSCNN Dual Task Loss

KL Loss

L1 Loss

Lovasz Loss

MSE Loss

OHEM CE Loss

Pixel Contrast CE Loss

Point CE Loss

RMI Loss

Connectivity Loss

        评估指标

mIoU

Accuracy

Kappa

Dice

AUC_ROC

        支持数据集

ADE20K

Cityscapes

COCO Stuff

Pascal VOC

EG1800

Pascal Context

SUPERVISELY

OPTIC DISC SEG

CHASE_DB1

HRF

DRIVE

STARE

PP-HumanSeg14K

PSSL

        数据增强

Flipping

Resize

ResizeByLong

ResizeByShort

LimitLong

ResizeRangeScaling

ResizeStepScaling

Normalize

Padding

PaddingByAspectRatio

RandomPaddingCrop

RandomCenterCrop

ScalePadding

RandomNoise

RandomBlur

RandomRotation

RandomScaleAspect

RandomDistort

RandomAffine

        分割一切模型

SegmentAnything

        模型选型工具

PaddleSMRT

        人像分割模型

PP-HumanSegV1

PP-HumanSegV2

        3D医疗分割模型

VNet

UNETR

nnFormer

nnUNet-D

TransUNet

SwinUNet

        Cityscapes打榜模型

HMSA

        CVPR冠军模型

MLA Transformer

        领域自适应

PixMatch

##  产业级分割模型库

高精度语义分割模型

#### 高精度模型，分割mIoU高、推理算量大，适合部署在服务器端GPU和Jetson等设备。

| 模型名称  | 骨干网络   | Cityscapes精度mIoU(%) |  V100 TRT推理速度(FPS)  |  配置文件 |
|:-------- |:--------:|:---------------------:|:---------------------:|:--------:|
| FCN            | HRNet_W18        | 78.97                 | 24.43     | [yml](./configs/fcn/)         |
| FCN            | HRNet_W48        | 80.70                 | 10.16     | [yml](./configs/fcn/)         |
| DeepLabV3      | ResNet50_OS8     | 79.90                 | 4.56      | [yml](./configs/deeplabv3/)   |
| DeepLabV3      | ResNet101_OS8    | 80.85                 | 3.2       | [yml](./configs/deeplabv3/)   |
| DeepLabV3      | ResNet50_OS8     | 80.36                 | 6.58      | [yml](./configs/deeplabv3p/)  |
| DeepLabV3      | ResNet101_OS8    | 81.10                 | *3.94*    | [yml](./configs/deeplabv3p/)  |
| OCRNet :star2:    | HRNet_w18        | 80.67                 | 13.26     | [yml](./configs/ocrnet/)      |
| OCRNet         | HRNet_w48        | 82.15                 | 6.17      | [yml](./configs/ocrnet/)      |
| CCNet          | ResNet101_OS8    | 80.95                 | 3.24      | [yml](./configs/ccnet/)       |

测试条件：
* V100上测速条件：针对Nvidia GPU V100，使用PaddleInference预测库的Python API，开启TensorRT加速，数据类型是FP32，输入图像维度是1x3x1024x2048。

轻量级语义分割模型

#### 轻量级模型，分割mIoU中等、推理算量中等，可以部署在服务器端GPU、服务器端X86 CPU和移动端ARM CPU。

| 模型名称  | 骨干网络   | Cityscapes精度mIoU(%) |  V100 TRT推理速度(FPS) | 骁龙855推理速度(FPS) |  配置文件 |
|:-------- |:--------:|:---------------------:|:---------------------:|:-----------------:|:--------:|
| PP-LiteSeg :star2:    | STDC1         | 77.04               | 69.82           | 17.22      | [yml](./configs/pp_liteseg/)  |
| PP-LiteSeg :star2:      | STDC2         | 79.04               | 54.53           | 11.75       | [yml](./configs/pp_liteseg/)  |
| BiSeNetV1           | -             | 75.19               | 14.67           | 1.53      |[yml](./configs/bisenetv1/)  |
| BiSeNetV2           | -             | 73.19               | 61.83           | 13.67       |[yml](./configs/bisenet/)  |
| STDCSeg             | STDC1         | 74.74               | 62.24           | 14.51       |[yml](./configs/stdcseg/)  |
| STDCSeg             | STDC2         | 77.60               | 51.15           | 10.95       |[yml](./configs/stdcseg/)  |
| DDRNet_23           | -             | 79.85               | 42.64           | 7.68      |[yml](./configs/ddrnet/)  |
| HarDNet             | -             | 79.03               | 30.3            | 5.44      |[yml](./configs/hardnet/)  |
| SFNet               | ResNet18_OS8  |  78.72              | *10.72*         |   -         | [yml](./configs/sfnet/)  |

测试条件：
* V100上测速条件：针对Nvidia GPU V100，使用PaddleInference预测库的Python API，开启TensorRT加速，数据类型是FP32，输入图像维度是1x3x1024x2048。
* 骁龙855上测速条件：针对小米9手机，使用PaddleLite预测库的CPP API，ARMV8编译，单线程，输入图像维度是1x3x256x256。

超轻量级语义分割模型

#### 超轻量级模型，分割mIoU一般、推理算量低，适合部署在服务器端X86 CPU和移动端ARM CPU。

| 模型名称 | 骨干网络 | ADE20K精度mIoU(%) | 骁龙855推理延时(ms) | 参数量(M) | 配置文件 |
|:--------:|:--------:|:--------:|:--------:|:--------:|:--------:|
|TopFormer-Base|TopTransformer-Base|38.28|480.6|5.13|[config](./configs/topformer/)
|PP-MobileSeg-Base|StrideFormer-Base|41.57|265.5|5.62|[config](./configs/pp_mobileseg/)|
|TopFormer-Tiny|TopTransformer-Tiny|32.46|490.3|1.41|[config](./configs/topformer/)
|PP-MobileSeg-Tiny|StrideFormer-Tiny|36.39|215.3|1.61|[config](./configs/pp_mobileseg/)|

测试条件:
* 针对小米9手机，使用[PaddleLite](https://github.com/PaddlePaddle/Paddle-Lite/blob/develop/README_en.md)预测库的CPP API，ARMV8编译，单线程，输入图像维度是1x3x512x512。测试模型在带有最后一个argmax算子的条件下进行测试。

| 模型名称  | 骨干网络   | Cityscapes精度mIoU(%) |  V100 TRT推理速度(FPS)  | 骁龙855推理速度(FPS)|  配置文件 |
|:-------- |:--------:|:---------------------:|:---------------------:|:-----------------:|:--------:|
| MobileSeg      | MobileNetV2              | 73.94                 | 67.57          | 27.01   | [yml](./configs/mobileseg/)  |
| MobileSeg :star2:  | MobileNetV3              | 73.47                 | 67.39          | 32.90   | [yml](./configs/mobileseg/)  |
| MobileSeg      | Lite_HRNet_18            | 70.75                 | *10.5*         | 13.05   | [yml](./configs/mobileseg/)  |
| MobileSeg      | ShuffleNetV2_x1_0        | 69.46                 | *37.09*        | 39.61  | [yml](./configs/mobileseg/)  |
| MobileSeg      | GhostNet_x1_0            | 71.88                 | *35.58*        | 38.74  | [yml](./configs/mobileseg/)  |

测试条件：
* V100上测速条件：针对Nvidia GPU V100，使用PaddleInference预测库的Python API，开启TensorRT加速，数据类型是FP32，输入图像维度是1x3x1024x2048。
* 骁龙855上测速条件：针对小米9手机，使用PaddleLite预测库的CPP API，ARMV8编译，单线程，输入图像维度是1x3x256x256。

##  使用教程

**入门教程**

* [安装说明](./docs/install_cn.md)
* [快速体验](./docs/quick_start_cn.md)
* [20分钟快速上手PaddleSeg](./docs/whole_process_cn.md)
* [模型库](./docs/model_zoo_overview_cn.md)

**基础教程**

* 准备数据集
   * [准备公开数据集](./docs/data/pre_data_cn.md)
   * [准备自定义数据集](./docs/data/marker/marker_cn.md)
   * [EISeg 数据标注](./EISeg)
* [准备配置文件](./docs/config/pre_config_cn.md)
* [模型训练](./docs/train/train_cn.md)
* [模型评估](./docs/evaluation/evaluate_cn.md)
* [模型预测](./docs/predict/predict_cn.md)

* 模型导出
    * [导出预测模型](./docs/model_export_cn.md)
    * [导出ONNX模型](./docs/model_export_onnx_cn.md)

* 模型部署
    * [FastDeploy部署](./deploy/fastdeploy)
    * [Paddle Inference部署(Python)](./docs/deployment/inference/python_inference_cn.md)
    * [Paddle Inference部署(C++)](./docs/deployment/inference/cpp_inference_cn.md)
    * [Paddle Lite部署](./docs/deployment/lite/lite_cn.md)
    * [Paddle Serving部署](./docs/deployment/serving/serving_cn.md)
    * [Paddle JS部署](./docs/deployment/web/web_cn.md)
    * [推理Benchmark](./docs/deployment/inference/infer_benchmark_cn.md)

**进阶教程**

* [训练技巧](./docs/train/train_tricks_cn.md)

* 模型压缩
    * [量化](./docs/deployment/slim/quant/quant_cn.md)
    * [蒸馏](./docs/deployment/slim/distill/distill_cn.md)
    * [裁剪](./docs/deployment/slim/prune/prune_cn.md)
    * [自动压缩](./docs/deployment/slim/act/readme.md)

* [常见问题汇总](./docs/faq/faq/faq_cn.md)

**欢迎贡献**

* [API文档](./docs/apis/README_CN.md)
*  二次开发教程
    * [配置文件详解](./docs/design/use/use_cn.md)
    * [如何创造自己的模型](./docs/design/create/add_new_model_cn.md)
*  模型贡献
    * [提交PR说明](./docs/pr/pr/pr_cn.md)
    * [模型PR规范](./docs/pr/pr/style_cn.md)

##  特色能力

- [交互式分割](./EISeg)
- [图像抠图](./Matting)
- [人像分割](./contrib/PP-HumanSeg)
- [3D医疗分割](./contrib/MedicalSeg)
- [Cityscapes打榜模型](./contrib/CityscapesSOTA)
- [全景分割](./contrib/PanopticDeepLab)
- [CVPR冠军模型](./contrib/AutoNUE)
- [领域自适应](./contrib/DomainAdaptation)

##  产业实践范例

* [使用PP-HumanSegV2进行人像分割](https://aistudio.baidu.com/aistudio/projectdetail/4504982?contributionType=1)
* [使用PP-HumanSegV1进行人像分割](https://aistudio.baidu.com/aistudio/projectdetail/2189481?channelType=0&channel=0)
* [使用PP-LiteSeg进行遥感道路分割](https://aistudio.baidu.com/aistudio/projectdetail/3873145?contributionType=1)
* [PaddleSeg实战之人脸部件分割与变妆](https://aistudio.baidu.com/aistudio/projectdetail/5326422)
* [PaddleSeg实战之小数据集3D椎骨分割](https://aistudio.baidu.com/aistudio/projectdetail/3878920)
* [PaddleSeg实战之车道线图像分割](https://aistudio.baidu.com/aistudio/projectdetail/1752986?channelType=0&channel=0)
* [PaddleSeg动态图API使用教程](https://aistudio.baidu.com/aistudio/projectdetail/1339458?channelType=0&channel=0)
* [10分钟上手PaddleSeg](https://aistudio.baidu.com/aistudio/projectdetail/1672610?channelType=0&channel=0)
* [车路协同：交互式分割技术在智慧建图中的应用和实践](https://aistudio.baidu.com/aistudio/projectdetail/5089472)
* [基于PaddleSeg的美甲预览机](https://aistudio.baidu.com/aistudio/projectdetail/5156312)
* [基于PaddleSeg的钢筋长度超限监控](https://aistudio.baidu.com/aistudio/projectdetail/5633532)

更多范例项目可参考：[『图像分割经典项目集』用PaddleSeg能做什么？](https://aistudio.baidu.com/aistudio/projectdetail/5436669)

## 许可证书
本项目的发布受Apache 2.0 license许可认证。

## 社区贡献

- 非常感谢[jm12138](https://github.com/jm12138)贡献U2-Net模型。
- 非常感谢[zjhellofss](https://github.com/zjhellofss)（傅莘莘）贡献Attention U-Net模型，和Dice loss损失函数。
- 非常感谢[liuguoyu666](https://github.com/liguoyu666)贡献U-Net++模型。
- 非常感谢[yazheng0307](https://github.com/yazheng0307) (刘正)贡献快速开始教程文档。
- 非常感谢[CuberrChen](https://github.com/CuberrChen)贡献STDC (rethink BiSeNet) PointRend，和 Detail Aggregate损失函数。
- 非常感谢[stuartchen1949](https://github.com/stuartchen1949)贡献 SegNet。
- 非常感谢[justld](https://github.com/justld)(郎督)贡献 UPerNet, DDRNet, CCNet, ESPNetV2, DMNet, ENCNet, HRNet_W48_Contrast, BiSeNetV1, FastFCN, SECrossEntropyLoss 和PixelContrastCrossEntropyLoss。
- 非常感谢[Herman-Hu-saber](https://github.com/Herman-Hu-saber)(胡慧明)参与贡献 ESPNetV2。
- 非常感谢[zhangjin12138](https://github.com/zhangjin12138)贡献数据增强方法 RandomCenterCrop。
- 非常感谢[simuler](https://github.com/simuler) 贡献 ESPNetV1。
- 非常感谢[ETTR123](https://github.com/ETTR123)(张恺) 贡献 ENet，PFPNNet。

##  学术引用

如果我们的项目在学术上帮助到你，请考虑以下引用：

```latex
@misc{liu2021paddleseg,
      title={PaddleSeg: A High-Efficient Development Toolkit for Image Segmentation},
      author={Yi Liu and Lutao Chu and Guowei Chen and Zewu Wu and Zeyu Chen and Baohua Lai and Yuying Hao},
      year={2021},
      eprint={2101.06175},
      archivePrefix={arXiv},
      primaryClass={cs.CV}
}

@misc{paddleseg2019,
    title={PaddleSeg, End-to-end image segmentation kit based on PaddlePaddle},
    author={PaddlePaddle Authors},
    howpublished = {\url{https://github.com/PaddlePaddle/PaddleSeg}},
    year={2019}
}
```


# Plyer

Plyer is a platform-independent Python API for accessing hardware features
of various platforms (Android, iOS, macOS, Linux and Windows).

Plyer is managed by the [Kivy Team](https://kivy.org/about.html). It is suitable for
use with Kivy apps, but can be used independently.

[![Backers on Open Collective](https://opencollective.com/kivy/backers/badge.svg)](#backers)
[![Sponsors on Open Collective](https://opencollective.com/kivy/sponsors/badge.svg)](#sponsors)
[![GitHub contributors](https://img.shields.io/github/contributors-anon/kivy/plyer)](https://github.com/kivy/plyer/graphs/contributors)
[![Contributor Covenant](https://img.shields.io/badge/Contributor%20Covenant-2.1-4baaaa.svg)](code_of_conduct.md)

![PyPI - Version](https://img.shields.io/pypi/v/plyer)
![PyPI - Python Version](https://img.shields.io/pypi/pyversions/plyer)

[![coverage](https://coveralls.io/repos/kivy/plyer/badge.svg?branch=master)](https://coveralls.io/r/kivy/plyer?branch=master)
![Continuous Integration with Ubuntu](https://github.com/kivy/plyer/workflows/Continuous%20Integration%20with%20Ubuntu/badge.svg)
![Continuous Integration with OSX](https://github.com/kivy/plyer/workflows/Continuous%20Integration%20with%20OSX/badge.svg)
![Continuous Integration with Windows](https://github.com/kivy/plyer/workflows/Continuous%20Integration%20with%20Windows/badge.svg)
![Deploy to PyPI](https://github.com/kivy/plyer/workflows/Deploy%20to%20PyPI/badge.svg)

## How plyer works?

Plyer tries not to reinvent the wheel, and will call external libraries to
implement the API in the easiest way, depending on the current platform.

- On Android ([python-for-android](https://python-for-android.readthedocs.io/)), [PyJNIus](https://pypi.org/project/pyjnius/) is used.
- On iOS ([kivy-ios](https://pypi.org/project/kivy-ios/)) and macOS,
  [pyobjus](https://pypi.org/project/pyobjus/) is used.
- On Windows, macOS and Linux, other commonly found libraries and programs
  are used.

## Supported APIs

| Platform                       | Android | iOS | Windows | macOS | Linux |
| ------------------------------ |:-------:|:---:|:-------:|:-----:|:-----:|
| Accelerometer                  | ✔       |  ✔ |         | ✔     |   ✔   |
| Audio recording                | ✔       |     |    ✔    | ✔     |       |
| Barometer                      | ✔       |  ✔  |         |       |       |
| Battery                        | ✔       |  ✔  |    ✔    | ✔     |   ✔   |
| Bluetooth                      | ✔       |     |         | ✔     |       |
| Brightness                     | ✔       |  ✔  |         |       |   ✔   |
| Call                           | ✔       |  ✔  |         |       |       |
| Camera (taking picture)        | ✔       |  ✔  |         |       |       |
| Compass                        | ✔       |  ✔  |         |       |       |
| CPU count                      |         |     |    ✔    | ✔     |   ✔   |
| Devicename                     | ✔       |     |    ✔    | ✔     |   ✔   |
| Email (open mail client)       | ✔       |  ✔  |    ✔    | ✔     |   ✔   |
| Flash                          | ✔       |  ✔  |         |       |       |
| GPS                            | ✔       |  ✔  |         |       |       |
| Gravity                        | ✔       |  ✔  |         |       |       |
| Gyroscope                      | ✔       |  ✔  |         |       |       |
| Humidity                       | ✔       |     |         |       |       |
| IR Blaster                     | ✔       |     |         |       |       |
| Keystore                       | ✔       |  ✔  |    ✔    | ✔     |   ✔   |
| Light                          | ✔       |     |         |       |       |
| Maps                           |         |  ✔  |         | ✔     |       |
| Native file chooser            | ✔       |  ✔  |    ✔    | ✔     |   ✔   |
| Notifications                  | ✔       |     |    ✔    | ✔     |   ✔   |
| Orientation                    | ✔       |     |         |       |   ✔   |
| Proximity                      | ✔       |     |         |       |       |
| Screenshot                     |         |     |    ✔    | ✔     |   ✔   |
| SMS (send messages)            | ✔       |  ✔  |         | ✔     |       |
| Spatial Orientation            | ✔       |  ✔  |         |       |       |
| Speech to text                 | ✔       |     |         |       |       |
| Storage Path                   | ✔       |  ✔  |    ✔    | ✔     |   ✔   |
| Temperature                    | ✔       |     |         |       |       |
| Text to speech                 | ✔       |  ✔  |    ✔    | ✔     |   ✔   |
| Unique ID                      | ✔       |  ✔  |    ✔    | ✔     |   ✔   |
| Vibrator                       | ✔       |  ✔  |         |       |       |
| Wifi                           |         |     |    ✔    | ✔     |   ✔   |

## Documentation

Full documentation, including details about the API, is available
[online](https://plyer.readthedocs.io/en/latest/). If you are not using the
latest version of Plyer, earlier versions of the documentations are linked
from there.

## Installation

To use on desktop: `pip install plyer`
To use in python-for-android and Kivy for iOS, add `plyer` to your requirements
if needed.

## License

Plyer is [MIT licensed](LICENSE), actively developed by a great
community and is supported by many projects managed by the
[Kivy Organization](https://www.kivy.org/about.html).

## Support

Are you having trouble using Plyer or any of its related projects in the Kivy
ecosystem?
Is there an error you don’t understand? Are you trying to figure out how to use
it? We have volunteers who can help!

The best channels to contact us for support are listed in the latest
[Contact Us](https://github.com/kivy/plyer/blob/master/CONTACT.md) document.

## Contributing

Plyer is part of the [Kivy](https://kivy.org) ecosystem - a large group of
products used by many thousands of developers for free, but it
is built entirely by the contributions of volunteers. We welcome (and rely on)
users who want to give back to the community by contributing to the project.

Contributions can come in many forms. See the latest
[Contribution Guidelines](https://github.com/kivy/plyer/blob/master/CONTRIBUTING.md)
for how you can help us.

## Code of Conduct

In the interest of fostering an open and welcoming community, we as
contributors and maintainers need to ensure participation in our project and
our sister projects is a harassment-free and positive experience for everyone.
It is vital that all interaction is conducted in a manner conveying respect,
open-mindedness and gratitude.

Please consult the [latest Code of Conduct](https://github.com/kivy/plyer/blob/master/CODE_OF_CONDUCT.md).

## Contributors

This project exists thanks to
[all the people who contribute](https://github.com/kivy/plyer/graphs/contributors).
[[Become a contributor](CONTRIBUTING.md)].

## Backers

Thank you to [all of our backers](https://opencollective.com/kivy)!
🙏 [[Become a backer](https://opencollective.com/kivy#backer)]

## Sponsors

Special thanks to
[all of our sponsors, past and present](https://opencollective.com/kivy).
Support this project by
[[becoming a sponsor](https://opencollective.com/kivy#sponsor)].

Here are our top current sponsors. Please click through to see their websites,
and support them as they support us.


_Bad software is everywhere, and we're tired of it. Sentry is on a mission to help developers write better software faster, so we can get back to enjoying technology. If you want to join us [**Check out our open positions**](https://sentry.io/careers/)_

# Official Sentry SDK for Python

[![Build Status](https://github.com/getsentry/sentry-python/actions/workflows/ci.yml/badge.svg)](https://github.com/getsentry/sentry-python/actions/workflows/ci.yml)
[![PyPi page link -- version](https://img.shields.io/pypi/v/sentry-sdk.svg)](https://pypi.python.org/pypi/sentry-sdk)
[![Discord](https://img.shields.io/discord/621778831602221064)](https://discord.gg/cWnMQeA)

This is the official Python SDK for [Sentry](http://sentry.io/)

## Getting Started

### Install

```bash
pip install --upgrade sentry-sdk
```

### Configuration

```python
import sentry_sdk

sentry_sdk.init(
    "https://12927b5f211046b575ee51fd8b1ac34f@o1.ingest.sentry.io/1",

    # Set traces_sample_rate to 1.0 to capture 100%
    # of transactions for performance monitoring.
    traces_sample_rate=1.0,
)
```

### Usage

```python
from sentry_sdk import capture_message
capture_message("Hello World")  # Will create an event in Sentry.

raise ValueError()  # Will also create an event in Sentry.
```

- To learn more about how to use the SDK [refer to our docs](https://docs.sentry.io/platforms/python/).
- Are you coming from `raven-python`? [Use this migration guide](https://docs.sentry.io/platforms/python/migration/).
- To learn about internals use the [API Reference](https://getsentry.github.io/sentry-python/).

## Integrations

(If you want to create a new integration, have a look at the [Adding a new integration checklist](https://github.com/getsentry/sentry-python/blob/master/CONTRIBUTING.md#adding-a-new-integration).)

See [the documentation](https://docs.sentry.io/platforms/python/integrations/) for an up-to-date list of libraries and frameworks we support. Here are some examples:

- [Django](https://docs.sentry.io/platforms/python/integrations/django/)
- [Flask](https://docs.sentry.io/platforms/python/integrations/flask/)
- [FastAPI](https://docs.sentry.io/platforms/python/integrations/fastapi/)
- [AIOHTTP](https://docs.sentry.io/platforms/python/integrations/aiohttp/)
- [SQLAlchemy](https://docs.sentry.io/platforms/python/integrations/sqlalchemy/)
- [asyncpg](https://docs.sentry.io/platforms/python/integrations/asyncpg/)
- [Redis](https://docs.sentry.io/platforms/python/integrations/redis/)
- [Celery](https://docs.sentry.io/platforms/python/integrations/celery/)
- [Apache Airflow](https://docs.sentry.io/platforms/python/integrations/airflow/)
- [Apache Spark](https://docs.sentry.io/platforms/python/integrations/pyspark/)
- [asyncio](https://docs.sentry.io/platforms/python/integrations/asyncio/)
- [Graphene](https://docs.sentry.io/platforms/python/integrations/graphene/)
- [Logging](https://docs.sentry.io/platforms/python/integrations/logging/)
- [Loguru](https://docs.sentry.io/platforms/python/integrations/loguru/)
- [HTTPX](https://docs.sentry.io/platforms/python/integrations/httpx/)
- [AWS Lambda](https://docs.sentry.io/platforms/python/integrations/aws-lambda/)
- [Google Cloud Functions](https://docs.sentry.io/platforms/python/integrations/gcp-functions/)

## Migrating

### Migrating From `1.x` to `2.x`

If you're on SDK version 1.x, we highly recommend updating to the 2.x major. To make the process easier we've prepared a [migration guide](https://docs.sentry.io/platforms/python/migration/1.x-to-2.x) with the most common changes as well as a [detailed changelog](MIGRATION_GUIDE.md).

### Migrating From `raven-python`

The old `raven-python` client has entered maintenance mode and was moved [here](https://github.com/getsentry/raven-python).

If you're using `raven-python`, we recommend you to migrate to this new SDK. You can find the benefits of migrating and how to do it in our [migration guide](https://docs.sentry.io/platforms/python/migration/raven-to-sentry-sdk/).

## Contributing to the SDK

Please refer to [CONTRIBUTING.md](CONTRIBUTING.md).

## Getting Help/Support

If you need help setting up or configuring the Python SDK (or anything else in the Sentry universe) please head over to the [Sentry Community on Discord](https://discord.com/invite/Ww9hbqr). There is a ton of great people in our Discord community ready to help you!

## Resources

- [![Documentation](https://img.shields.io/badge/documentation-sentry.io-green.svg)](https://docs.sentry.io/quickstart/)
- [![Forum](https://img.shields.io/badge/forum-sentry-green.svg)](https://forum.sentry.io/c/sdks)
- [![Discord](https://img.shields.io/discord/621778831602221064)](https://discord.gg/Ww9hbqr)
- [![Stack Overflow](https://img.shields.io/badge/stack%20overflow-sentry-green.svg)](http://stackoverflow.com/questions/tagged/sentry)
- [![Twitter Follow](https://img.shields.io/twitter/follow/getsentry?label=getsentry&style=social)](https://twitter.com/intent/follow?screen_name=getsentry)

## License

Licensed under the MIT license, see [`LICENSE`](LICENSE)

### Thanks to all the people who contributed!


# New way for visual programming!

We are working on new way for visual programming. We developed desktop application called [MLJAR Studio](https://mljar.com).
It is a notebook based development environment with interactive code recipes and managed Python environment. All running locally on your machine. We are waiting for your feedback.

---

# MLJAR Automated Machine Learning for Humans

[![Tests status](https://github.com/mljar/mljar-supervised/actions/workflows/run-tests.yml/badge.svg)](https://github.com/mljar/mljar-supervised/actions/workflows/run-tests.yml)
[![PyPI version](https://badge.fury.io/py/mljar-supervised.svg)](https://badge.fury.io/py/mljar-supervised)
[![Anaconda-Server Badge](https://anaconda.org/conda-forge/mljar-supervised/badges/version.svg)](https://anaconda.org/conda-forge/mljar-supervised)
[![PyPI pyversions](https://img.shields.io/pypi/pyversions/mljar-supervised.svg)](https://pypi.python.org/pypi/mljar-supervised/)

[![Anaconda-Server Badge](https://anaconda.org/conda-forge/mljar-supervised/badges/platforms.svg)](https://anaconda.org/conda-forge/mljar-supervised)
[![Anaconda-Server Badge](https://anaconda.org/conda-forge/mljar-supervised/badges/license.svg)](https://anaconda.org/conda-forge/mljar-supervised)
[![Downloads](https://pepy.tech/badge/mljar-supervised)](https://pepy.tech/project/mljar-supervised)

---

**Documentation**: https://supervised.mljar.com/

**Source Code**: https://github.com/mljar/mljar-supervised

**Looking for commercial support**: Please contact us by [email](https://mljar.com/contact/) for details

---

## Table of Contents

 - [Automated Machine Learning](https://github.com/mljar/mljar-supervised#automated-machine-learning)
 - [What's good in it?](https://github.com/mljar/mljar-supervised#whats-good-in-it)
 - [AutoML Web App with GUI](https://github.com/mljar/mljar-supervised#automl-web-app-with-user-interface)
 - [Automatic Documentation](https://github.com/mljar/mljar-supervised#automatic-documentation)
 - [Available Modes](https://github.com/mljar/mljar-supervised#available-modes)
 - [Fairness Aware Training](https://github.com/mljar/mljar-supervised#fairness-aware-training)
 - [Examples](https://github.com/mljar/mljar-supervised#examples)
 - [FAQ](https://github.com/mljar/mljar-supervised#faq)
 - [Documentation](https://github.com/mljar/mljar-supervised#documentation)
 - [Installation](https://github.com/mljar/mljar-supervised#installation)
 - [Demo](https://github.com/mljar/mljar-supervised#demo)
 - [Contributing](https://github.com/mljar/mljar-supervised#contributing)
 - [Cite](https://github.com/mljar/mljar-supervised#cite)
 - [License](https://github.com/mljar/mljar-supervised#license)
 - [Commercial support](https://github.com/mljar/mljar-supervised#commercial-support)
 - [MLJAR](https://github.com/mljar/mljar-supervised#mljar)

# Automated Machine Learning

The `mljar-supervised` is an Automated Machine Learning Python package that works with tabular data. It is designed to save time for a data scientist. It abstracts the common way to preprocess the data, construct the machine learning models, and perform hyper-parameters tuning to find the best model :trophy:. It is no black box, as you can see exactly how the ML pipeline is constructed (with a detailed Markdown report for each ML model).

The `mljar-supervised` will help you with:
 - explaining and understanding your data (Automatic Exploratory Data Analysis),
 - trying many different machine learning models (Algorithm Selection and Hyper-Parameters tuning),
 - creating Markdown reports from analysis with details about all models (Automatic-Documentation),
 - saving, re-running, and loading the analysis and ML models.

It has four built-in modes of work:
 - `Explain` mode, which is ideal for explaining and understanding the data, with many data explanations, like decision trees visualization, linear models coefficients display, permutation importance, and SHAP explanations of data,
 - `Perform` for building ML pipelines to use in production,
 - `Compete` mode that trains highly-tuned ML models with ensembling and stacking, with the purpose to use in ML competitions.
 - `Optuna` mode can be used to search for highly-tuned ML models should be used when the performance is the most important, and computation time is not limited (it is available from version `0.10.0`)

Of course, you can further customize the details of each `mode` to meet the requirements.

## What's good in it?

- It uses many algorithms: `Baseline`, `Linear`, `Random Forest`, `Extra Trees`, `LightGBM`, `Xgboost`, `CatBoost`, `Neural Networks`, and `Nearest Neighbors`.
- It can compute Ensemble based on a greedy algorithm from [Caruana paper](http://www.cs.cornell.edu/~alexn/papers/shotgun.icml04.revised.rev2.pdf).
- It can stack models to build a level 2 ensemble (available in `Compete` mode or after setting the `stack_models` parameter).
- It can do features preprocessing, like missing values imputation and converting categoricals. What is more, it can also handle target values preprocessing.
- It can do advanced features engineering, like [Golden Features](https://supervised.mljar.com/features/golden_features/), [Features Selection](https://supervised.mljar.com/features/features_selection/), Text and Time Transformations.
- It can tune hyper-parameters with a `not-so-random-search` algorithm (random-search over a defined set of values) and hill climbing to fine-tune final models.
- It can compute the `Baseline` for your data so that you will know if you need Machine Learning or not!
- It has extensive explanations. This package is training simple `Decision Trees` with `max_depth <= 5`, so you can easily visualize them with amazing [dtreeviz](https://github.com/parrt/dtreeviz) to better understand your data.
- The `mljar-supervised` uses simple linear regression and includes its coefficients in the summary report, so you can check which features are used the most in the linear model.
- It cares about the explainability of models: for every algorithm, the feature importance is computed based on permutation. Additionally, for every algorithm, the SHAP explanations are computed: feature importance, dependence plots, and decision plots (explanations can be switched off with the `explain_level` parameter).
- There is automatic documentation for every ML experiment run with AutoML. The `mljar-supervised` creates markdown reports from AutoML training full of ML details, metrics, and charts.

# AutoML Web App with User Interface

We created a Web App with GUI, so you don't need to write any code 🐍. Just upload your data. Please check the Web App at [github.com/mljar/automl-app](https://github.com/mljar/automl-app). You can run this Web App locally on your computer, so your data is safe and secure :cat:

# Automatic Documentation

## The AutoML Report

The report from running AutoML will contain the table with information about each model score and the time needed to train the model. There is a link for each model, which you can click to see the model's details. The performance of all ML models is presented as scatter and box plots so you can visually inspect which algorithms perform the best :trophy:.

![AutoML leaderboard](https://github.com/mljar/mljar-examples/blob/master/media/automl_summary.gif)

## The `Decision Tree` Report

The example for `Decision Tree` summary with trees visualization. For classification tasks, additional metrics are provided:
- confusion matrix
- threshold (optimized in the case of binary classification task)
- F1 score
- Accuracy
- Precision, Recall, MCC

![Decision Tree summary](https://github.com/mljar/mljar-examples/blob/master/media/decision_tree_summary.gif)

## The `LightGBM` Report

The example for `LightGBM` summary:

![Decision Tree summary](https://github.com/mljar/mljar-examples/blob/master/media/lightgbm_summary.gif)

## Available Modes

In the [docs](https://supervised.mljar.com/features/modes/) you can find details about AutoML modes that are presented in the table.

### Explain

```py
automl = AutoML(mode="Explain")
```

It is aimed to be used when the user wants to explain and understand the data.
 - It is using 75%/25% train/test split.
 - It uses: `Baseline`, `Linear`, `Decision Tree`, `Random Forest`, `Xgboost`, `Neural Network' algorithms, and ensemble.
 - It has full explanations: learning curves, importance plots, and SHAP plots.

### Perform

```py
automl = AutoML(mode="Perform")
```

It should be used when the user wants to train a model that will be used in real-life use cases.
 - It uses a 5-fold CV.
 - It uses: `Linear`, `Random Forest`, `LightGBM`, `Xgboost`, `CatBoost`, and `Neural Network`. It uses ensembling.
 - It has learning curves and importance plots in reports.

### Compete

```py
automl = AutoML(mode="Compete")
```

It should be used for machine learning competitions.
 - It adapts the validation strategy depending on dataset size and `total_time_limit`. It can be: a train/test split (80/20), 5-fold CV or 10-fold CV.
 - It is using: `Linear`, `Decision Tree`, `Random Forest`, `Extra Trees`, `LightGBM`, `Xgboost`, `CatBoost`, `Neural Network`, and `Nearest Neighbors`. It uses ensemble and **stacking**.
 - It has only learning curves in the reports.

### Optuna

```py
automl = AutoML(mode="Optuna", optuna_time_budget=3600)
```

It should be used when the performance is the most important and time is not limited.
- It uses a 10-fold CV
- It uses: `Random Forest`, `Extra Trees`, `LightGBM`, `Xgboost`, and `CatBoost`. Those algorithms are tuned by `Optuna` framework for `optuna_time_budget` seconds, each. Algorithms are tuned with original data, without advanced feature engineering.
- It uses advanced feature engineering, stacking and ensembling. The hyperparameters found for original data are reused with those steps.
- It produces learning curves in the reports.

## How to save and load AutoML?

All models in the AutoML are saved and loaded automatically. No need to call `save()` or `load()`.

### Example:

#### Train AutoML

```python
automl = AutoML(results_path="AutoML_classifier")
automl.fit(X, y)
```

You will have all models saved in the `AutoML_classifier` directory. Each model will have a separate directory with the `README.md` file with all details from the training.

#### Compute predictions
```python
automl = AutoML(results_path="AutoML_classifier")
automl.predict(X)
```

The  AutoML automatically loads models from the `results_path` directory. If you will call `fit()` on already trained AutoML then you will get a warning message that AutoML is already fitted.

### Why do you automatically save all models?

All models are automatically saved to be able to restore the training after interruption. For example, you are training AutoML for 48 hours, and after 47 hours, there is some unexpected interruption. In MLJAR AutoML you just call the same training code after the interruption and AutoML reloads already trained models and finishes the training.

## Supported evaluation metrics (`eval_metric` argument in `AutoML()`)

- for binary classification: `logloss`, `auc`, `f1`, `average_precision`, `accuracy`- default is `logloss`
- for multiclass classification: `logloss`, `f1`, `accuracy` - default is `logloss`
- for regression: `rmse`, `mse`, `mae`, `r2`, `mape`, `spearman`, `pearson` - default is `rmse`

If you don't find the `eval_metric` that you need, please add a new issue. We will add it.

## Fairness Aware Training

Starting from version `1.0.0` AutoML can optimize the Machine Learning pipeline with sensitive features. There are the following fairness related arguments in the AutoML constructor:
 - `fairness_metric` - metric which will be used to decide if the model is fair,
 - `fairness_threshold` - threshold used in decision about model fairness,
 - `privileged_groups` - privileged groups used in fairness metrics computation,
 - `underprivileged_groups` - underprivileged groups used in fairness metrics computation.

The `fit()` method accepts `sensitive_features`. When sensitive features are passed to AutoML, the best model will be selected among fair models only. In the AutoML reports, additional information about fairness metrics will be added. The MLJAR AutoML supports two methods for bias mitigation:
 - Sample Weighting - assigns weights to samples to treat samples equally,
 - Smart Grid Search - similar to Sample Weighting, where different weights are checked to optimize fairness metric.

The fair ML building can be used with all algorithms, including `Ensemble` and `Stacked Ensemble`. We support three Machine Learning tasks:
 - binary classification,
 - mutliclass classification,
 - regression.

Example code:

```python
from sklearn.model_selection import train_test_split
from sklearn.datasets import fetch_openml
from supervised.automl import AutoML

data = fetch_openml(data_id=1590, as_frame=True)
X = data.data
y = (data.target == ">50K") * 1
sensitive_features = X[["sex"]]

X_train, X_test, y_train, y_test, S_train, S_test = train_test_split(
    X, y, sensitive_features, stratify=y, test_size=0.75, random_state=42
)

automl = AutoML(
    algorithms=[
        "Xgboost"
    ],
    train_ensemble=False,
    fairness_metric="demographic_parity_ratio",
    fairness_threshold=0.8,
    privileged_groups = [{"sex": "Male"}],
    underprivileged_groups = [{"sex": "Female"}],
)

automl.fit(X_train, y_train, sensitive_features=S_train)
```

You can read more about fairness aware AutoML training in our article https://mljar.com/blog/fairness-machine-learning/

![Fairness aware AutoML](https://raw.githubusercontent.com/mljar/visual-identity/main/automl/fairness-automl.gif)

# Examples

## :point_right: Binary Classification Example

There is a simple interface available with `fit` and `predict` methods.

```python
import pandas as pd
from sklearn.model_selection import train_test_split
from supervised.automl import AutoML

df = pd.read_csv(
    "https://raw.githubusercontent.com/pplonski/datasets-for-start/master/adult/data.csv",
    skipinitialspace=True,
)
X_train, X_test, y_train, y_test = train_test_split(
    df[df.columns[:-1]], df["income"], test_size=0.25
)

automl = AutoML()
automl.fit(X_train, y_train)

predictions = automl.predict(X_test)
```

AutoML `fit` will print:
```py
Create directory AutoML_1
AutoML task to be solved: binary_classification
AutoML will use algorithms: ['Baseline', 'Linear', 'Decision Tree', 'Random Forest', 'Xgboost', 'Neural Network']
AutoML will optimize for metric: logloss
1_Baseline final logloss 0.5519845471086654 time 0.08 seconds
2_DecisionTree final logloss 0.3655910192804364 time 10.28 seconds
3_Linear final logloss 0.38139916864708445 time 3.19 seconds
4_Default_RandomForest final logloss 0.2975204390214936 time 79.19 seconds
5_Default_Xgboost final logloss 0.2731086827200411 time 5.17 seconds
6_Default_NeuralNetwork final logloss 0.319812276905242 time 21.19 seconds
Ensemble final logloss 0.2731086821194617 time 1.43 seconds
```

- the AutoML results in [Markdown report](https://github.com/mljar/mljar-examples/tree/master/Income_classification/AutoML_1#automl-leaderboard)
- the Xgboost [Markdown report](https://github.com/mljar/mljar-examples/blob/master/Income_classification/AutoML_1/5_Default_Xgboost/README.md), please take a look at amazing dependence plots produced by SHAP package :sparkling_heart:
- the Decision Tree [Markdown report](https://github.com/mljar/mljar-examples/blob/master/Income_classification/AutoML_1/2_DecisionTree/README.md), please take a look at beautiful tree visualization :sparkles:
- the Logistic Regression [Markdown report](https://github.com/mljar/mljar-examples/blob/master/Income_classification/AutoML_1/3_Linear/README.md), please take a look at coefficients table, and you can compare the SHAP plots between (Xgboost, Decision Tree and Logistic Regression) :coffee:

## :point_right: Multi-Class Classification Example

The example code for classification of the optical recognition of handwritten digits dataset. Running this code in less than 30 minutes will result in test accuracy ~98%.

```python
import pandas as pd
# scikit learn utilites
from sklearn.datasets import load_digits
from sklearn.metrics import accuracy_score
from sklearn.model_selection import train_test_split
# mljar-supervised package
from supervised.automl import AutoML

# load the data
digits = load_digits()
X_train, X_test, y_train, y_test = train_test_split(
    pd.DataFrame(digits.data), digits.target, stratify=digits.target, test_size=0.25,
    random_state=123
)

# train models with AutoML
automl = AutoML(mode="Perform")
automl.fit(X_train, y_train)

# compute the accuracy on test data
predictions = automl.predict_all(X_test)
print(predictions.head())
print("Test accuracy:", accuracy_score(y_test, predictions["label"].astype(int)))
```

## :point_right: Regression Example

Regression example on `California Housing` house prices data.

```python
import numpy as np
import pandas as pd
from sklearn.datasets import fetch_california_housing
from sklearn.model_selection import train_test_split
from sklearn.metrics import mean_squared_error
from supervised.automl import AutoML # mljar-supervised

# Load the data
housing = fetch_california_housing()
X_train, X_test, y_train, y_test = train_test_split(
    pd.DataFrame(housing.data, columns=housing.feature_names),
    housing.target,
    test_size=0.25,
    random_state=123,
)

# train models with AutoML
automl = AutoML(mode="Explain")
automl.fit(X_train, y_train)

# compute the MSE on test data
predictions = automl.predict(X_test)
print("Test MSE:", mean_squared_error(y_test, predictions))
```

## :point_right: More Examples

- [**Income classification**](https://github.com/mljar/mljar-examples/tree/master/Income_classification) - it is a binary classification task on census data
- [**Iris classification**](https://github.com/mljar/mljar-examples/tree/master/Iris_classification) - it is a multiclass classification on Iris flowers data
- [**House price regression**](https://github.com/mljar/mljar-examples/tree/master/House_price_regression) - it is a regression task on Boston houses data

# FAQ

What method is used for hyperparameters optimization?
  - For modes: `Explain`, `Perform`, and `Compete` there is used a random search method combined with hill climbing. In this approach, all checked models are saved and used for building Ensemble.
  - For mode: `Optuna` the Optuna framework is used. It uses using TPE sampler for tuning. Models checked during the Optuna hyperparameters search are not saved, only the best model is saved (the final model from tuning). You can check the details about checked hyperparameters from optuna by checking study files in the `optuna` directory in your AutoML `results_path`.

How to save and load AutoML?

The save and load of AutoML models is automatic. All models created during AutoML training are saved in the directory set in `results_path` (argument of `AutoML()` constructor). If there is no `results_path` set, then the directory is created based on following name convention: `AutoML_{number}` the `number` will be number from 1 to 1000 (depends which directory name will be free).

Example save and load:

```python
automl = AutoML(results_path='AutoML_1')
automl.fit(X, y)
```

The all models from AutoML are saved in `AutoML_1` directory.

To load models:

```python
automl = AutoML(results_path='AutoML_1')
automl.predict(X)
```

How to set ML task (select between classification or regression)?

The MLJAR AutoML can work with:
- binary classification
- multi-class classification
- regression

The ML task detection is automatic based on target values. There can be situation if you want to manually force AutoML to select the ML task, then you need to set `ml_task` parameter. It can be set to `'binary_classification'`, `'multiclass_classification'`, `'regression'`.

Example:
```python
automl = AutoML(ml_task='regression')
automl.fit(X, y)
```
In the above example the regression model will be fitted.

How to reuse Optuna hyperparameters?

  You can reuse Optuna hyperparameters that were found in other AutoML training. You need to pass them in `optuna_init_params` argument. All hyperparameters found during Optuna tuning are saved in the `optuna/optuna.json` file (inside `results_path` directory).

 Example:

 ```python
 optuna_init = json.loads(open('previous_AutoML_training/optuna/optuna.json').read())

 automl = AutoML(
     mode='Optuna',
     optuna_init_params=optuna_init
 )
 automl.fit(X, y)
 ```

 When reusing Optuna hyperparameters the Optuna tuning is simply skipped. The model will be trained with hyperparameters set in `optuna_init_params`. Right now there is no option to continue Optuna tuning with seed parameters.

How to know the order of classes for binary or multiclass problem when using predict_proba?

To get predicted probabilites with information about class label please use the `predict_all()` method. It returns the pandas DataFrame with class names in the columns. The order of predicted columns is the same in the `predict_proba()` and `predict_all()` methods. The `predict_all()` method will additionaly have the column with the predicted class label.

# Documentation

For details please check [mljar-supervised docs](https://supervised.mljar.com).

# Installation

From PyPi repository:

```
pip install mljar-supervised
```

To install this package with conda run:
```
conda install -c conda-forge mljar-supervised
```

From source code:

```
git clone https://github.com/mljar/mljar-supervised.git
cd mljar-supervised
python setup.py install
```

Installation for development
```
git clone https://github.com/mljar/mljar-supervised.git
virtualenv venv --python=python3.6
source venv/bin/activate
pip install -r requirements.txt
pip install -r requirements_dev.txt
```

Running in the docker:
```
FROM python:3.7-slim-buster
RUN apt-get update && apt-get -y update
RUN apt-get install -y build-essential python3-pip python3-dev
RUN pip3 -q install pip --upgrade
RUN pip3 install mljar-supervised jupyter
CMD ["jupyter", "notebook", "--port=8888", "--no-browser", "--ip=0.0.0.0", "--allow-root"]
```

Install from GitHub with pip:
```
pip install -q -U git+https://github.com/mljar/mljar-supervised.git@master
```
# Demo

In the below demo GIF you will see:
- MLJAR AutoML trained in Jupyter Notebook on the Titanic dataset
- overview of created files
- a showcase of selected plots created during AutoML training
- algorithm comparison report along with their plots
- example of README file and CSV file with results

![](https://github.com/mljar/mljar-examples/raw/master/media/mljar_files.gif)

# Contributing

To get started take a look at our [Contribution Guide](https://supervised.mljar.com/contributing/) for information about our process and where you can fit in!

### Contributors

# Cite

Would you like to cite MLJAR? Great! :)

You can cite MLJAR as follows:

```
@misc{mljar,
  author    = {Aleksandra P\l{}o\'{n}ska and Piotr P\l{}o\'{n}ski},
  year      = {2021},
  publisher = {MLJAR},
  address   = {\L{}apy, Poland},
  title     = {MLJAR: State-of-the-art Automated Machine Learning Framework for Tabular Data.  Version 0.10.3},
  url       = {https://github.com/mljar/mljar-supervised}
}
```

Would love to hear from you about how have you used MLJAR AutoML in your project.
Please feel free to let us know at
![image](https://user-images.githubusercontent.com/6959032/118103228-f5ea9a00-b3d9-11eb-87ed-8cfb1f873f91.png)

# License

The `mljar-supervised` is provided with [MIT license](https://github.com/mljar/mljar-supervised/blob/master/LICENSE).

# Commercial support

Looking for commercial support? Do you need new feature implementation? Please contact us by [email](https://mljar.com/contact/) for details.

# MLJAR

The `mljar-supervised` is an open-source project created by [MLJAR](https://mljar.com). We care about ease of use in Machine Learning.
The [mljar.com](https://mljar.com) provides a beautiful and simple user interface for building machine learning models.