petl - Extract, Transform and Load
===================================================

``petl`` is a general purpose Python package for extracting, transforming and
loading tables of data.

.. image:: docs/petl-architecture.png
    :align: center
    :alt: petl usage possibilities

Resources
---------

- Documentation: http://petl.readthedocs.org/
- PyPI: http://pypi.python.org/pypi/petl
- Conda: https://anaconda.org/conda-forge/petl
- Discussion: http://groups.google.com/group/python-etl

DevOps Status
-------------

|downloads| |monthly|

|ci| |pypi| |conda|

|coveralls| |readthedocs| |zenodo|

.. |downloads|    image:: https://static.pepy.tech/badge/petl
    :target: https://pepy.tech/project/petl
    :alt: Downloads

.. |monthly|    image:: https://static.pepy.tech/badge/petl/month
    :target: https://pepy.tech/project/petl
    :alt: Downloads/Month

.. |ci|    image:: https://github.com/petl-developers/petl/actions/workflows/test-changes.yml/badge.svg
    :target: https://github.com/petl-developers/petl/actions/workflows/test-changes.yml
    :alt: Continuous Integration build status

.. |pypi|    image:: https://github.com/petl-developers/petl/actions/workflows/publish-release.yml/badge.svg
    :target: https://github.com/petl-developers/petl/actions/workflows/publish-release.yml
    :alt: PyPI release status

.. |conda|    image:: https://github.com/conda-forge/petl-feedstock/actions/workflows/automerge.yml/badge.svg
    :target: https://github.com/conda-forge/petl-feedstock/actions/workflows/automerge.yml
    :alt: Conda Forge release status

.. |readthedocs|    image:: https://readthedocs.org/projects/petl/badge/?version=stable
    :target: http://petl.readthedocs.io/en/stable/?badge=stable
    :alt: readthedocs.org release status

.. |coveralls|    image:: https://coveralls.io/repos/github/petl-developers/petl/badge.svg?branch=master
    :target: https://coveralls.io/github/petl-developers/petl?branch=master
    :alt: Coveralls release status

.. |zenodo|    image:: https://zenodo.org/badge/2233194.svg
   :target: https://zenodo.org/badge/latestdoi/2233194


PaddleSlim

PaddleSlim是一个专注于深度学习模型压缩的工具库，提供**低比特量化、知识蒸馏、稀疏化和模型结构搜索**等模型压缩策略，帮助开发者快速实现模型的小型化。

## 产品动态

- 🔥 2022.01.18: 发布[YOLOv8自动化压缩示例](https://github.com/PaddlePaddle/PaddleSlim/tree/develop/example/auto_compression/detection)，量化预测加速2.5倍。

- **【直播分享】2022-12-13 20:30 《自动化压缩技术详解及ViT模型实战》，微信扫码报名**

2022.08.16：自动化压缩功能升级

  - 支持直接加载ONNX模型和Paddle模型导出至ONNX
  - 发布量化分析工具，发布[YOLO系列离线量化工具](example/post_training_quantization/pytorch_yolo_series)
  - 更新[YOLO-Series自动化压缩模型库](example/auto_compression/pytorch_yolo_series)

  | 模型  | Base mAPval
0.5:0.95 | ACT量化mAPval
0.5:0.95  | 模型体积压缩比 | 预测时延FP32
  | 预测时延INT8
 | 预测加速比 |
  | :-------- |:-------- |:--------: | :--------: | :---------------------: | :----------------: | :----------------: |
  | PPYOLOE-s | 43.1 | 42.6  | 3.9倍  | 6.51ms  | 2.12ms  | 3.1倍 |
  | YOLOv5s | 37.4   | 36.9  | 3.8倍  | 5.95ms  |  1.87ms | 3.2倍 |
  | YOLOv6s | 42.4   | 41.3 | 3.9倍 |  9.06ms  |   1.83ms   | 5.0倍   |
  | YOLOv7 |  51.1   | 50.9 | 3.9倍 |  26.84ms  |   4.55ms   |  5.9倍  |
  | YOLOv7-Tiny | 37.3   | 37.0 | 3.9倍 | 5.06ms  |   1.68ms   |  3.0倍  |

历史更新

- **2022.07.01: 发布[v2.3.0版本](https://github.com/PaddlePaddle/PaddleSlim/releases/tag/v2.3.0)**

  - 发布[自动化压缩功能](example/auto_compression)
    - 支持代码无感知压缩：开发者只需提供推理模型文件和数据，既可进行离线量化（PTQ）、量化训练（QAT）、稀疏训练等压缩任务。
    - 支持自动策略选择，根据任务特点和部署环境特性：自动搜索合适的离线量化方法,自动搜索最佳的压缩策略组合方式。
    - 发布[自然语言处理](example/auto_compression/nlp)、[图像语义分割](example/auto_compression/semantic_segmentation)、[图像目标检测](example/auto_compression/detection)三个方向的自动化压缩示例。
    - 发布`X2Paddle`模型自动化压缩方案:[YOLOv5](example/auto_compression/pytorch_yolo_series)、[YOLOv6](example/auto_compression/pytorch_yolo_series)、[YOLOv7](example/auto_compression/pytorch_yolo_series)、[HuggingFace](example/auto_compression/pytorch_huggingface)、[MobileNet](example/auto_compression/tensorflow_mobilenet)。
  - 升级量化功能
    - 统一量化模型格式；离线量化支持while op；修复BERT大模型量化训练过慢的问题。
    - 新增7种[离线量化方法](docs/zh_cn/tutorials/quant/Quantization_intro.md), 包括HIST, AVG, EMD, Bias Correction, AdaRound等。
  - 支持半结构化稀疏训练
  - 新增延时预估工具
    - 支持对稀疏化模型、低比特量化模型的性能预估；支持预估指定模型在特定部署环境下 (ARM CPU + Paddle Lite) 的推理性能；提供 SD625、SD710、RK3288 芯片 + Paddle Lite 的预估接口。
    - 提供部署环境自动扩展工具，可以自动增加在更多 ARM CPU 设备上的预估工具。

- **2021.11.15: 发布v2.2.0版本**

  - 支持动态图离线量化功能.

- **2021.5.20: 发布V2.1.0版本**

  - 扩展离线量化方法
  - 新增非结构化稀疏
  - 增强剪枝功能
  - 修复OFA功能若干bug

更多信息请参考：[release note](https://github.com/PaddlePaddle/PaddleSlim/releases)

## 基础压缩功能概览

PaddleSlim支持以下功能，也支持自定义量化、裁剪等功能。

  Quantization
  Pruning
  NAS
  Distilling

QAT

PACT

PTQ Static

PTQ Dynamic

Embedding Quant

SensitivityPruner

FPGMFilterPruner

L1NormFilterPruner

**L2NormFilterPruner

*SlimFilterPruner

*OptSlimFilterPruner

*Simulate Anneal based NAS

*Reinforcement Learning based NAS

**DARTS

**PC-DARTS

**Once-for-All

*Hardware-aware Search

*FSP

*DML

*DK

注：
- *表示仅支持静态图，**表示仅支持动态图
- 敏感度裁剪指的是通过各个层的敏感度分析来确定各个卷积层的剪裁率，需要和其他裁剪方法配合使用。

PaddleSlim在典型视觉和自然语言处理任务上做了模型压缩，并且测试了Nvidia GPU、ARM等设备上的加速情况，这里展示部分模型的压缩效果，详细方案可以参考下面CV和NLP模型压缩方案:

表1: 部分场景模型压缩加速情况

注意事项

- YOLOv3: 在移动端SD855上加速3.55倍。
- PP-OCR: 体积由8.9M减少到2.9M, 在SD855上加速1.27倍。
- BERT: 模型参数由110M减少到80M，精度提升的情况下，Tesla T4 GPU FP16计算加速1.47倍。

### 不同压缩方法效果

自动压缩效果

表3: 自动压缩效果

离线量化效果对比

表2: 多种离线量化方法效果对比

## 安装

安装发布版本：
```bash
pip install paddleslim
```

安装develop版本：
```bash
git clone https://github.com/PaddlePaddle/PaddleSlim.git & cd PaddleSlim
python setup.py install
```

- 验证安装：安装完成后您可以使用 python 或 python3 进入 python 解释器，输入import paddleslim, 没有报错则说明安装成功。
- 版本对齐：

|  PaddleSlim   | PaddlePaddle   | PaddleLite    |
| :-----------: | :------------: | :------------:|
| 2.0.0         | 2.0            |       2.8     |
| 2.1.0         | 2.1.0          |       2.8     |
| 2.1.1         | 2.1.1          |       >=2.8   |
| 2.3.0         | 2.3.0          |       >=2.11  |
| 2.4.0         | 2.4.0          |       >=2.11  |
| develop       | develop        |       >=2.11  |

## 文档教程

### 模型压缩技术
- [模型量化技术](docs/zh_cn/tutorials/quant/Quantization_intro.md)

### 快速开始

- 🔥 [自动压缩](example/auto_compression)
- [量化训练](docs/zh_cn/quick_start/static/quant_aware_tutorial.md)
- [离线量化](docs/zh_cn/quick_start/static/quant_post_static_tutorial.md)
- [结构化剪枝](docs/zh_cn/quick_start/static/pruning_tutorial.md)
- [蒸馏](docs/zh_cn/quick_start/static/distillation_tutorial.md)
- [NAS](docs/zh_cn/quick_start/static/nas_tutorial.md)
- [量化分析工具](https://github.com/PaddlePaddle/PaddleSlim/blob/develop/docs/zh_cn/tutorials/quant/AnalysisPTQ.md)

### 更多教程

进阶教程详细介绍了每一步的流程，帮助您把相应方法迁移到您自己的模型上。

- 通道剪裁
  - [四种剪裁策略效果对比与应用方法](docs/zh_cn/tutorials/pruning/overview.md)
    - [L1NormFilterPruner](docs/zh_cn/tutorials/pruning/overview.md#l1normfilterpruner)
    - [FPGMFilterPruner](docs/zh_cn/tutorials/pruning/overview.md#fpgmfilterpruner)
    - [SlimFilterFilterPruner](docs/zh_cn/tutorials/pruning/overview.md#slimfilterpruner)
    - [OptSlimFilterPruner](docs/zh_cn/tutorials/pruning/overview.md#optslimfilterpruner)
  - 自定义剪裁策略：[动态图](docs/zh_cn/tutorials/pruning/dygraph/self_defined_filter_pruning.md)

- 低比特量化
  - [三种量化方法介绍与应用](docs/zh_cn/tutorials/quant/overview.md)
    - [量化训练](docs/zh_cn/quick_start/static/quant_aware_tutorial.md)
    - [离线量化](docs/zh_cn/tutorials/quant/static/quant_post_tutorial.md) | [离线量化方法解析](docs/zh_cn/tutorials/quant/post_training_quantization.md)
    - [embedding量化](docs/zh_cn/tutorials/quant/static/embedding_quant_tutorial.md)

- NAS
  - [四种NAS策略介绍和应用](docs/zh_cn/tutorials/nas/overview.md)
    - [Once-For-All](docs/zh_cn/tutorials/nas/dygraph/nas_ofa.md)
    - [SANAS](docs/zh_cn/tutorials/nas/static/sanas_darts_space.md)
    - [RLNAS](https://github.com/PaddlePaddle/PaddleSlim/tree/release/2.0.0/demo/nas#rlnas%E7%BD%91%E7%BB%9C%E7%BB%93%E6%9E%84%E6%90%9C%E7%B4%A2%E7%A4%BA%E4%BE%8B)
    - [DARTS](demo/darts/README.md)

- 蒸馏
  - [知识蒸馏示例](demo/distillation)

### 推理部署
- [Benchmark](https://github.com/PaddlePaddle/PaddleSlim/blob/develop/docs/zh_cn/QUANT_BENCHMARK.md)
- 量化部署支持的预测库：
    - Paddle Inference：[GPU量化部署](demo/quant/deploy/TensorRT/README.md)、Intel CPU量化部署。
    - Paddle Lite：[量化部署](docs/zh_cn/deploy/deploy_cls_model_on_mobile_device.md)
    - ONNX

## CV模型压缩
### 多场景效果展示

本系列教程均基于Paddle官方的模型套件中模型进行压缩，若您不是模型套件用户，更推荐使用快速教程和进阶教程。

- 检测模型压缩
  - 压缩方案
    - [PPDetection-YOLOv3 压缩方案](docs/zh_cn/cv/detection/static/yolov3_slim.md)

  - 方法应用-静态图
    - [蒸馏](docs/zh_cn/cv/detection/static/paddledetection_slim_distillation_tutorial.md)
    - [量化训练](docs/zh_cn/cv/detection/static/paddledetection_slim_quantization_tutorial.md)
    - [模型结构搜索](docs/zh_cn/cv/detection/static/paddledetection_slim_nas_tutorial.md)
    - [剪枝](docs/zh_cn/cv/detection/static/paddledetection_slim_pruing_tutorial.md)
    - [剪枝与蒸馏的结合使用](docs/zh_cn/cv/detection/static/paddledetection_slim_prune_dist_tutorial.md)
    - [卷积层敏感度分析](docs/zh_cn/cv/detection/static/paddledetection_slim_sensitivy_tutorial.md)

  - 方法应用-动态图
    - [剪枝](https://github.com/PaddlePaddle/PaddleDetection/tree/release/2.0-rc/dygraph/configs/slim#%E5%89%AA%E8%A3%81)
    - [量化训练](https://github.com/PaddlePaddle/PaddleDetection/tree/release/2.0-rc/dygraph/configs/slim#%E9%87%8F%E5%8C%96)

- 分割模型压缩

  - 压缩方案

  - 方法应用-静态图
    - [蒸馏](https://github.com/PaddlePaddle/PaddleSeg/tree/release/v0.8.0/slim/distillation)
    - [量化训练](https://github.com/PaddlePaddle/PaddleSeg/tree/release/v0.8.0/slim/quantization)
    - [模型结构搜索](https://github.com/PaddlePaddle/PaddleSeg/tree/release/v0.8.0/slim/nas)
    - [剪枝](https://github.com/PaddlePaddle/PaddleSeg/tree/release/v0.8.0/slim/prune)

  - 方法应用-动态图
    - [剪枝](https://github.com/PaddlePaddle/PaddleSeg/tree/develop/slim#%E6%A8%A1%E5%9E%8B%E8%A3%81%E5%89%AA)
    - [量化训练](https://github.com/PaddlePaddle/PaddleSeg/tree/develop/slim#%E6%A8%A1%E5%9E%8B%E9%87%8F%E5%8C%96)

- OCR模型压缩

  - 压缩方案
    - [3.5M模型压缩方案](docs/zh_cn/cv/ocr/static/3.5M_slim.md)

  - 方法应用-静态图
    - [量化训练](https://github.com/PaddlePaddle/PaddleOCR/tree/release/1.1/deploy/slim/quantization)
    - [剪枝](https://github.com/PaddlePaddle/PaddleOCR/tree/release/1.1/deploy/slim/prune)

  - 方法应用-动态图
    - [剪枝](https://github.com/PaddlePaddle/PaddleOCR/tree/develop/deploy/slim/prune)
    - [量化训练](https://github.com/PaddlePaddle/PaddleOCR/tree/develop/deploy/slim/quantization)

### NLP模型压缩

- [PaddleNLP-BERT](docs/zh_cn/nlp/paddlenlp_slim_ofa_tutorial.md)
- [ERNIE-ERNIE](docs/zh_cn/nlp/ernie_slim_ofa_tutorial.md)

### API文档

- [动态图](docs/zh_cn/api_cn/dygraph)
- [静态图](docs/zh_cn/api_cn/static)

### [FAQ](docs/zh_cn/FAQ/quantization_FAQ.md)

#### 1. 量化训练或者离线量化后的模型体积为什么没有变小？
答：这是因为量化后保存的参数是虽然是int8范围，但是类型是float。这是因为Paddle训练前向默认的Kernel不支持INT8 Kernel实现，只有Paddle Inference TensorRT的推理才支持量化推理加速。为了方便量化后验证量化精度，使用Paddle训练前向能加载此模型，默认保存的Float32类型权重，体积没有发生变换。

#### 2. macOS + Python3.9环境或者Windows环境下, 安装出错, "command 'swig' failed"

答: 请参考https://github.com/PaddlePaddle/PaddleSlim/issues/1258

## 许可证书

本项目的发布受[Apache 2.0 license](https://github.com/PaddlePaddle/PaddleSlim/blob/develop/LICENSE)许可认证。

## 贡献代码

我们非常欢迎你可以为PaddleSlim提供代码，也十分感谢你的反馈。

##  技术交流

- 如果你发现任何PaddleSlim存在的问题或者是建议, 欢迎通过[GitHub Issues](https://github.com/PaddlePaddle/PaddleSlim/issues)给我们提issues。

- 欢迎加入PaddleSlim 微信技术交流群


# OSF

The code for [https://osf.io](https://osf.io).

- `master` ![Build Status](https://github.com/CenterForOpenScience/osf.io/workflows/osf.io/badge.svg?branch=master)
- `develop` ![Build Status](https://github.com/CenterForOpenScience/osf.io/workflows/osf.io/badge.svg?branch=develop)
- Versioning Scheme:  [![CalVer Scheme](https://img.shields.io/badge/calver-YY.MINOR.MICRO-22bfda.svg)](http://calver.org)
- Issues: https://github.com/CenterForOpenScience/osf.io/issues?state=open
- COS Development Docs: http://cosdev.readthedocs.org/

## Running the OSF For Development

To run the OSF for local development, see [README-docker-compose.md](https://github.com/CenterForOpenScience/osf.io/blob/develop/README-docker-compose.md).

Optional, but recommended: To set up pre-commit hooks (will run
formatters and linters on staged files):

```
pip install pre-commit

pre-commit install --allow-missing-config
```

## More Resources

The [COS Development Docs](http://cosdev.readthedocs.org/) provide detailed information about all aspects of OSF development.
This includes style guides, process docs, troubleshooting, and more.


.. image:: https://img.shields.io/github/stars/hhursev/recipe-scrapers?style=social
    :target: https://github.com/hhursev/recipe-scrapers/
    :alt: Github
.. image:: https://img.shields.io/pypi/v/recipe-scrapers.svg?
    :target: https://pypi.org/project/recipe-scrapers/
    :alt: Version
.. image:: https://img.shields.io/pypi/pyversions/recipe-scrapers
    :target: https://pypi.org/project/recipe-scrapers/
    :alt: PyPI - Python Version
.. image:: https://pepy.tech/badge/recipe-scrapers
    :target: https://pepy.tech/project/recipe-scrapers
    :alt: Downloads
.. image:: https://github.com/hhursev/recipe-scrapers/workflows/unittests/badge.svg?branch=main
    :target: https://github.com/hhursev/recipe-scrapers/actions/
    :alt: GitHub Actions Unittests
.. image:: https://coveralls.io/repos/hhursev/recipe-scraper/badge.svg?branch=main&service=github
    :target: https://coveralls.io/github/hhursev/recipe-scraper?branch=main
    :alt: Coveralls
.. image:: https://img.shields.io/github/license/hhursev/recipe-scrapers?
    :target: https://github.com/hhursev/recipe-scrapers/blob/main/LICENSE
    :alt: License
.. image:: https://app.codacy.com/project/badge/Grade/3ee8da77aaa3475a8085ca22287dea89
    :target: https://app.codacy.com/gh/hhursev/recipe-scrapers/dashboard
    :alt: Codacy Badge

------

A simple web scraping tool for recipe sites.

.. code:: shell

    pip install recipe-scrapers

then:

.. code:: python

    from recipe_scrapers import scrape_me

    scraper = scrape_me('https://www.allrecipes.com/recipe/158968/spinach-and-feta-turkey-burgers/')

    # Q: What if the recipe site I want to extract information from is not listed below?
    # A: You can give it a try with the wild_mode option! If there is Schema/Recipe available it will work just fine.
    scraper = scrape_me('https://www.feastingathome.com/tomato-risotto/', wild_mode=True)

    scraper.host()
    scraper.title()
    scraper.total_time()
    scraper.image()
    scraper.ingredients()
    scraper.ingredient_groups()
    scraper.instructions()
    scraper.instructions_list()
    scraper.yields()
    scraper.to_json()
    scraper.links()
    scraper.nutrients()  # not always available
    scraper.canonical_url()  # not always available
    scraper.equipment()  # not always available
    scraper.cooking_method()  # not always available
    scraper.keywords()  # not always available
    scraper.dietary_restrictions() # not always available

You also have an option to scrape html-like content

.. code:: python

    import requests
    from recipe_scrapers import scrape_html

    url = "https://www.allrecipes.com/recipe/158968/spinach-and-feta-turkey-burgers/"
    html = requests.get(url).content

    scraper = scrape_html(html=html, org_url=url)

    scraper.title()
    scraper.total_time()
    # etc...

Notes:

- ``scraper.links()`` returns a list of dictionaries containing all of the <a> tag attributes. The attribute names are the dictionary keys.

Some Python HTTP clients that you can use to retrieve HTML include `requests <https://pypi.org/project/requests/>`_ and `httpx <https://pypi.org/project/httpx/>`_.  Please refer to their documentation to find out what options (timeout configuration, proxy support, etc) are available.

Scrapers available for:
-----------------------

- `https://101cookbooks.com/ <https://101cookbooks.com/>`_
- `https://15gram.be <https://15gram.be>`_
- `https://www.750g.com <https://www.750g.com>`_
- `https://aberlehome.com/ <https://aberlehome.com>`_
- `https://abuelascounter.com/ <https://abuelascounter.com>`_
- `https://www.acouplecooks.com <https://acouplecooks.com/>`_
- `https://addapinch.com/ <https://addapinch.com/>`_
- `http://www.afghankitchenrecipes.com/ <http://www.afghankitchenrecipes.com/>`_
- `https://aflavorjournal.com/ <https://aflavorjournal.com/>`_
- `https://ah.nl/ <https://ah.nl/>`_
- `https://akispetretzikis.com/ <https://akispetretzikis.com/>`_
- `https://aldi.com.au/ <https://aldi.com.au/>`_
- `https://alexandracooks.com/ <https://alexandracooks.com/>`_
- `https://alittlebityummy.com/ <https://alittlebityummy.com/>`_
- `https://allrecipes.com/ <https://allrecipes.com/>`_
- `https://allthehealthythings.com/ <https://allthehealthythings.com/>`_
- `https://alltommat.se/ <https://alltommat.se/>`_
- `https://altonbrown.com/ <https://altonbrown.com/>`_
- `https://amazingribs.com/ <https://amazingribs.com/>`_
- `https://ambitiouskitchen.com/ <https://ambitiouskitchen.com>`_
- `https://archanaskitchen.com/ <https://archanaskitchen.com/>`_
- `https://www.argiro.gr/ <https://www.argiro.gr/>`_
- `https://www.arla.se/ <https://www.arla.se/>`_
- `https://www.atelierdeschefs.fr/ <https://www.atelierdeschefs.fr/>`_
- `https://averiecooks.com/ <https://www.averiecooks.com/>`_
- `https://www.bakels.com.au/ <https://www.bakels.com.au/>`_
- `https://baking-sense.com/ <https://baking-sense.com/>`_
- `https://bakingmischief.com/ <https://bakingmischief.com/>`_
- `https://barefeetinthekitchen.com/ <https://barefeetinthekitchen.com/>`_
- `https://barefootcontessa.com/ <https://barefootcontessa.com>`_
- `https://bbc.com/ <https://bbc.com/food/recipes>`_
    - `.co.uk <https://bbc.co.uk/food/recipes>`_
- `https://bbcgoodfood.com/ <https://bbcgoodfood.com>`_
- `https://bestrecipes.com.au/ <https://bestrecipes.com.au>`_
- `https://bettybossi.ch/ <https://bettybossi.ch>`_
- `https://bettycrocker.com/ <https://bettycrocker.com>`_
- `https://biancazapatka.com/ <https://biancazapatka.com>`_
- `https://bigoven.com/ <https://bigoven.com>`_
- `https://blueapron.com/ <https://blueapron.com>`_
- `https://bluejeanchef.com/ <https://bluejeanchef.com/>`_
- `https://www.bodybuilding.com/ <https://www.bodybuilding.com/>`_
- `https://bonappetit.com/ <https://bonappetit.com>`_
- `https://bongeats.com/ <https://bongeats.com/>`_
- `https://bowlofdelicious.com/ <https://bowlofdelicious.com/>`_
- `https://breadtopia.com/ <https://breadtopia.com/>`_
- `https://briceletbaklava.ch/ <https://briceletbaklava.ch/>`_
- `https://budgetbytes.com/ <https://budgetbytes.com>`_
- `https://cafedelites.com/ <https://cafedelites.com/>`_
- `https://carlsbadcravings.com/ <https://carlsbadcravings.com/>`_
- `https://castironketo.net/ <https://castironketo.net/>`_
- `https://cdkitchen.com/ <https://cdkitchen.com/>`_
- `https://chefkoch.de/ <https://chefkoch.de>`_
- `https://www.chefnini.com/ <https://www.chefnini.com/>`_
- `https://chefsavvy.com/ <https://chefsavvy.com/>`_
- `https://claudia.abril.com.br/ <https://claudia.abril.com.br>`_
- `https://closetcooking.com/ <https://closetcooking.com>`_
- `https://comidinhasdochef.com/ <https://comidinhasdochef.com/>`_
- `https://cook-talk.com/ <https://cook-talk.com/>`_
- `https://cookeatshare.com/ <https://cookeatshare.com/>`_
- `https://cookieandkate.com/ <https://cookieandkate.com/>`_
- `https://cooking.nytimes.com/ <https://cooking.nytimes.com>`_
- `https://cookingcircle.com/ <https://cookingcircle.com/>`_
- `https://cookinglight.com/ <https://cookinglight.com/>`_
- `https://cookpad.com/ <https://cookpad.com/>`_
- `https://www.coop.se/ <https://www.coop.se/>`_
- `https://copykat.com/ <https://copykat.com>`_
- `https://www.costco.com/ <https://www.costco.com>`_
- `https://countryliving.com/ <https://countryliving.com>`_
- `https://creativecanning.com/ <https://creativecanning.com>`_
- `https://cucchiaio.it/ <https://cucchiaio.it>`_
- `https://cuisineaz.com/ <https://cuisineaz.com>`_
- `https://cybercook.com.br/ <https://cybercook.com.br/>`_
- `https://damndelicious.net/ <https://damndelicious.net/>`_
- `https://www.davidlebovitz.com/ <https://www.davidlebovitz.com/>`_
- `https://delish.com/ <https://delish.com>`_
- `https://dinneratthezoo.com/ <https://dinneratthezoo.com>`_
- `https://dinnerthendessert.com/ <https://dinnerthendessert.com/>`_
- `https://dish.co.nz/ <https://dish.co.nz>`_
- `https://domesticate-me.com/ <https://domesticate-me.com/>`_
- `https://downshiftology.com/ <https://downshiftology.com/>`_
- `https://www.dr.dk/ <https://www.dr.dk/>`_
- `https://www.eatingbirdfood.com/ <https://www.eatingbirdfood.com>`_
- `https://www.eatingwell.com/ <https://www.eatingwell.com>`_
- `https://www.eatliverun.com/ <https://www.eatliverun.com/>`_
- `https://eatsmarter.com/ <https://eatsmarter.com/>`_
    - `.de <https://eatsmarter.de/>`_
- `https://eattolerant.de/ <https://eattolerant.de/>`_
- `https://www.eatwell101.com <https://www.eatwell101.com>`_
- `https://eatwhattonight.com/ <https://eatwhattonight.com/>`_
- `https://elavegan.com/ <https://elavegan.com/>`_
- `https://emmikochteinfach.de/ <https://emmikochteinfach.de/>`_
- `https://en.wikibooks.org/ <https://en.wikibooks.org>`_
- `https://epicurious.com/ <https://epicurious.com>`_
- `https://www.errenskitchen.com/ <https://www.errenskitchen.com/>`_
- `https://ethanchlebowski.com/ <https://ethanchlebowski.com>`_
- `https://www.evolvingtable.com/ <https://www.evolvingtable.com/>`_
- `https://www.familyfoodonthetable.com/ <https://www.familyfoodonthetable.com/>`_
- `https://www.farmhouseonboone.com/ <https://www.farmhouseonboone.com/>`_
- `https://www.fattoincasadabenedetta.it/ <https://www.fattoincasadabenedetta.it/>`_
- `https://felix.kitchen <https://felix.kitchen>`_
- `https://fifteenspatulas.com/ <https://www.fifteenspatulas.com/>`_
- `https://finedininglovers.com/ <https://www.finedininglovers.com>`_
- `https://fitmencook.com/ <https://www.fitmencook.com>`_
- `https://fitslowcookerqueen.com <https://fitslowcookerqueen.com/>`_
- `https://food.com/ <https://www.food.com>`_
- `https://food52.com/ <https://www.food52.com>`_
- `https://foodandwine.com/ <https://www.foodandwine.com>`_
- `https://foodfidelity.com/ <https://foodfidelity.com>`_
- `https://foodnetwork.co.uk/ <https://www.foodnetwork.co.uk>`_
    - `.com <https://www.foodnetwork.com>`_
- `https://foodrepublic.com/ <https://foodrepublic.com>`_
- `https://www.forksoverknives.com/ <https://www.forksoverknives.com/>`_
- `https://forktospoon.com/ <https://forktospoon.com/>`_
- `https://franzoesischkochen.de/ <https://franzoesischkochen.de/>`_
- `https://www.gesund-aktiv.com/ <https://www.gesund-aktiv.com>`_
- `https://gimmesomeoven.com/ <https://www.gimmesomeoven.com/>`_
- `https://godt.no/ <https://godt.no/>`_
- `https://gonnawantseconds.com/ <https://gonnawantseconds.com>`_
- `https://goodfooddiscoveries.com/ <https://goodfooddiscoveries.com/>`_
- `https://goodhousekeeping.com/ <https://www.goodhousekeeping.com/>`_
- `https://gourmettraveller.com.au/ <https://gourmettraveller.com.au>`_
- `https://gousto.co.uk/ <https://gousto.co.uk>`_
- `https://www.grandfrais.com/ <https://www.grandfrais.com>`_
- `https://greatbritishchefs.com/ <https://greatbritishchefs.com>`_
- `https://grimgrains.com/ <https://grimgrains.com>`_
- `http://www.grouprecipes.com/ <http://www.grouprecipes.com/>`_
- `https://halfbakedharvest.com/ <https://www.halfbakedharvest.com/>`_
- `https://handletheheat.com/ <https://handletheheat.com/>`_
- `https://www.hassanchef.com/ <https://www.hassanchef.com/>`_
- `https://headbangerskitchen.com/ <https://www.headbangerskitchen.com/>`_
- `https://healthyeating.nhlbi.nih.gov/ <https://healthyeating.nhlbi.nih.gov>`_
- `https://heatherchristo.com/ <https://heatherchristo.com/>`_
- `https://www.heb.com/ <https://www.heb.com/recipe/landing>`_
- `https://hellofresh.com/ <https://hellofresh.com>`_
    - `.at <https://www.hellofresh.at/>`_ `.be <https://www.hellofresh.be/>`_, `.ca <https://www.hellofresh.ca/>`_, `.ch <https://www.hellofresh.ch/>`_, `.co.nz <https://www.hellofresh.co.nz/>`_, `.co.uk <https://hellofresh.co.uk>`_, `.com.au <https://www.hellofresh.com.au/>`_, `.de <https://www.hellofresh.de/>`_, `.dk <https://www.hellofresh.dk/>`_, `.es <https://www.hellofresh.es/>`_, `.fr <https://www.hellofresh.fr/>`_, `.ie <https://www.hellofresh.ie/>`_, `.it <https://www.hellofresh.it/>`_, `.lu <https://www.hellofresh.lu/>`_, `.nl <https://www.hellofresh.nl/>`_, `.no <https://www.hellofresh.no/>`_, `.se <https://www.hellofresh.se/>`_
- `https://www.hersheyland.com/ <https://www.hersheyland.com/>`_
- `https://www.homechef.com/ <https://www.homechef.com/>`_
- `https://hostthetoast.com/ <https://hostthetoast.com/>`_
- `https://www.ica.se/ <https://www.ica.se/>`_
- `https://www.im-worthy.com/ <https://www.im-worthy.com>`_
- `https://inbloombakery.com/ <https://inbloombakery.com/>`_
- `https://indianhealthyrecipes.com <https://www.indianhealthyrecipes.com>`_
- `https://www.innit.com/ <https://www.innit.com/>`_
- `https://insanelygoodrecipes.com <https://insanelygoodrecipes.com/>`_
- `https://inspiralized.com/ <https://inspiralized.com>`_
- `https://izzycooking.com/ <https://izzycooking.com/>`_
- `https://jamieoliver.com/ <https://jamieoliver.com>`_
- `https://jimcooksfoodgood.com/ <https://jimcooksfoodgood.com/>`_
- `https://www.jocooks.com/ <https://www.jocooks.com>`_
- `https://joshuaweissman.com/ <https://joshuaweissman.com/>`_
- `https://joyfoodsunshine.com/ <https://joyfoodsunshine.com>`_
- `https://joythebaker.com/ <https://joythebaker.com>`_
- `https://juliegoodwin.com.au/ <https://juliegoodwin.com.au>`_
- `https://justataste.com/ <https://justataste.com>`_
- `https://justbento.com/ <https://justbento.com>`_
- `https://www.justonecookbook.com/ <https://www.justonecookbook.com>`_
- `https://kennymcgovern.com/ <https://kennymcgovern.com>`_
- `https://keukenliefde.nl/ <https://keukenliefde.nl>`_
- `https://www.kingarthurbaking.com <https://www.kingarthurbaking.com>`_
- `https://kitchenaid.com.au/ <https://kitchenaid.com.au/blogs/kitchenthusiast/tagged/blog-category-recipes>`_
- `https://www.kitchensanctuary.com/ <https://www.kitchensanctuary.com>`_
- `https://www.kitchenstories.com/ <https://www.kitchenstories.com>`_
- `https://kochbar.de/ <https://kochbar.de>`_
- `https://kochbucher.com/ <https://kochbucher.com/>`_
- `http://koket.se/ <http://koket.se>`_
- `https://kristineskitchenblog.com/ <https://kristineskitchenblog.com>`_
- `https://kuchnia-domowa.pl/ <https://www.kuchnia-domowa.pl/>`_
- `https://kuchynalidla.sk/ <https://www.kuchynalidla.sk/>`_
- `https://www.kwestiasmaku.com/ <https://www.kwestiasmaku.com/>`_
- `https://www.latelierderoxane.com <https://www.latelierderoxane.com/blog/recettes/>`_
- `https://leanandgreenrecipes.net <https://leanandgreenrecipes.net>`_
- `https://www.lecker.de <https://www.lecker.de/rezepte>`_
- `https://lecremedelacrumb.com/ <https://lecremedelacrumb.com/>`_
- `https://lekkerensimpel.com <https://lekkerensimpel.com>`_
- `https://leukerecepten.nl/ <https://www.leukerecepten.nl>`_
- `https://lifestyleofafoodie.com <https://lifestyleofafoodie.com>`_
- `https://littlespicejar.com/ <https://littlespicejar.com>`_
- `http://livelytable.com/ <http://livelytable.com/>`_
- `https://lovingitvegan.com/ <https://lovingitvegan.com/>`_
- `https://www.maangchi.com <https://www.maangchi.com>`_
- `https://madensverden.dk/ <https://madensverden.dk/>`_
- `https://www.madewithlau.com/ <https://www.madewithlau.com/>`_
- `https://madsvin.com/ <https://madsvin.com/>`_
- `https://marleyspoon.com/ <https://marleyspoon.com/>`_
    - `.at <https://marleyspoon.at/>`_, `.be <https://marleyspoon.be/>`_, `.com.au <https://marleyspoon.com.au/>`_, `.de <https://marleyspoon.de/>`_, `.nl <https://marleyspoon.nl/>`_, `.se <https://marleyspoon.se/>`_
- `https://marmiton.org/ <https://marmiton.org/>`_
- `https://www.marthastewart.com/ <https://www.marthastewart.com/>`_
- `https://matprat.no/ <https://matprat.no/>`_
- `https://www.mccormick.com/ <https://www.mccormick.com/>`_
- `https://meljoulwan.com/ <https://meljoulwan.com/>`_
- `https://www.melskitchencafe.com/ <https://www.melskitchencafe.com/>`_
- `http://mindmegette.hu/ <http://mindmegette.hu/>`_
- `https://minimalistbaker.com/ <https://minimalistbaker.com/>`_
- `https://ministryofcurry.com/ <https://ministryofcurry.com/>`_
- `https://misya.info/ <https://misya.info>`_
- `https://www.mob.co.uk/ <https://www.mob.co.uk/>`_
- `https://mobile.kptncook.com/ <https://mobile.kptncook.com/>`_
- `https://mobkitchen.co.uk/ <https://mobkitchen.co.uk/>`_
- `https://www.modernhoney.com/ <https://www.modernhoney.com/>`_
- `https://www.momontimeout.com/ <https://www.momontimeout.com/>`_
- `https://momswithcrockpots.com/ <https://momswithcrockpots.com>`_
- `https://monsieur-cuisine.com/ <https://monsieur-cuisine.com>`_
- `http://motherthyme.com/ <http://motherthyme.com/>`_
- `https://www.moulinex.fr/ <https://www.moulinex.fr/>`_
- `https://www.mundodereceitasbimby.com.pt/ <https://www.mundodereceitasbimby.com.pt/>`_
- `https://mybakingaddiction.com/ <https://mybakingaddiction.com>`_
- `https://myjewishlearning.com/ <https://myjewishlearning.com>`_
- `https://mykitchen101.com/ <https://mykitchen101.com>`_
- `https://mykitchen101en.com/ <https://mykitchen101en.com>`_
- `https://mykoreankitchen.com/ <https://mykoreankitchen.com>`_
- `https://www.myplate.gov/ <https://www.myplate.gov/>`_
- `https://myrecipes.com/ <https://myrecipes.com>`_
- `https://www.nhs.uk/healthier-families/ <https://www.nhs.uk/healthier-families/>`_
- `https://nibbledish.com/ <https://nibbledish.com>`_
- `https://norecipes.com/ <https://norecipes.com/>`_
- `https://www.notenoughcinnamon.com/ <https://www.notenoughcinnamon.com/>`_
- `https://nourishedbynutrition.com/ <https://nourishedbynutrition.com/>`_
- `https://www.nrk.no/ <https://www.nrk.no/>`_
- `https://www.number-2-pencil.com/ <https://www.number-2-pencil.com/>`_
- `https://nutritionbynathalie.com/blog <https://nutritionbynathalie.com/blog>`_
- `https://nutritionfacts.org/ <https://nutritionfacts.org/>`_
- `https://ohsheglows.com/ <https://ohsheglows.com>`_
- `https://omnivorescookbook.com <https://omnivorescookbook.com>`_
- `https://www.onceuponachef.com <https://www.onceuponachef.com>`_
- `https://onesweetappetite.com/ <https://onesweetappetite.com>`_
- `https://owen-han.com/ <https://owen-han.com>`_
- `https://www.paleorunningmomma.com/ <https://www.paleorunningmomma.com>`_
- `https://www.panelinha.com.br/ <https://www.panelinha.com.br>`_
- `https://paninihappy.com/ <https://paninihappy.com>`_
- `https://www.persnicketyplates.com/ <https://www.persnicketyplates.com/>`_
- `https://www.pickuplimes.com/ <https://www.pickuplimes.com/>`_
- `https://pinchofyum.com/ <https://pinchofyum.com/>`_
- `https://www.pingodoce.pt/ <https://www.pingodoce.pt>`_
- `https://pinkowlkitchen.com/ <https://pinkowlkitchen.com/>`_
- `https://www.platingpixels.com/ <https://www.platingpixels.com/>`_
- `https://plowingthroughlife.com/ <https://plowingthroughlife.com/>`_
- `https://popsugar.com/ <https://popsugar.com>`_
- `https://potatorolls.com/ <https://potatorolls.com/>`_
- `https://practicalselfreliance.com/ <https://practicalselfreliance.com>`_
- `https://pressureluckcooking.com/ <https://pressureluckcooking.com/>`_
- `https://www.primaledgehealth.com/ <https://www.primaledgehealth.com/>`_
- `https://www.projectgezond.nl/ <https://www.projectgezond.nl/>`_
- `https://przepisy.pl/ <https://przepisy.pl>`_
- `https://purelypope.com/ <https://purelypope.com>`_
- `https://purplecarrot.com/ <https://purplecarrot.com>`_
- `https://rachlmansfield.com/ <https://rachlmansfield.com>`_
- `https://rainbowplantlife.com/ <https://rainbowplantlife.com/>`_
- `https://realfood.tesco.com/ <https://realfood.tesco.com>`_
- `https://realsimple.com/ <https://www.realsimple.com>`_
- `https://receitas.globo.com/ <https://www.receitas.globo.com/>`_
- `https://receitas.ig.com.br/ <https://receitas.ig.com.br>`_
- `https://www.receitasnestle.com.br <https://www.receitasnestle.com.br>`_
- `https://recept.se/ <https://recept.se/>`_
- `https://www.recipegirl.com/ <https://www.recipegirl.com/>`_
- `https://reciperunner.com/ <https://www.reciperunner.com>`_
- `https://recipes.farmhousedelivery.com/ <https://recipes.farmhousedelivery.com/>`_
- `https://recipes.timesofindia.com/ <https://recipes.timesofindia.com/>`_
- `https://recipetineats.com/ <https://www.recipetineats.com/>`_
- `https://redhousespice.com/ <https://redhousespice.com/>`_
- `https://reishunger.de/ <https://www.reishunger.de/>`_
- `https://rezeptwelt.de/ <https://rezeptwelt.de>`_
- `https://ricetta.it/ <https://ricetta.it>`_
- `https://ricette.giallozafferano.it/ <https://ricette.giallozafferano.it>`_
- `https://www.ricetteperbimby.it/ <https://www.ricetteperbimby.it/>`_
- `https://rosannapansino.com <https://rosannapansino.com>`_
- `https://rutgerbakt.nl/ <https://rutgerbakt.nl/>`_
- `https://www.saboresajinomoto.com.br/ <https://www.saboresajinomoto.com.br/>`_
- `https://sallys-blog.de <https://sallys-blog.de/>`_
- `https://sallysbakingaddiction.com <https://sallysbakingaddiction.com/>`_
- `https://saltpepperskillet.com/ <https://saltpepperskillet.com/>`_
- `https://www.saveur.com/ <https://www.saveur.com/>`_
- `https://www.savorynothings.com/ <https://www.savorynothings.com/>`_
- `https://seriouseats.com/ <https://seriouseats.com>`_
- `https://sharing.kptncook.com/ <https://sharing.kptncook.com/>`_
- `https://simple-veganista.com/ <https://simple-veganista.com/>`_
- `https://simply-cookit.com/ <https://simply-cookit.com>`_
- `https://simplyquinoa.com/ <https://simplyquinoa.com>`_
- `https://simplyrecipes.com/ <https://simplyrecipes.com>`_
- `https://simplywhisked.com/ <https://simplywhisked.com>`_
- `https://skinnytaste.com/ <https://www.skinnytaste.com>`_
- `https://smulweb.nl/ <https://smulweb.nl>`_
- `https://sobors.hu/ <https://sobors.hu>`_
- `https://www.southerncastiron.com/ <https://www.southerncastiron.com>`_
- `https://southernliving.com/ <https://southernliving.com/>`_
- `https://spendwithpennies.com/ <https://spendwithpennies.com/>`_
- `https://www.springlane.de <https://www.springlane.de>`_
- `https://www.staysnatched.com/ <https://www.staysnatched.com/>`_
- `https://steamykitchen.com/ <https://steamykitchen.com>`_
- `https://streetkitchen.hu/ <https://streetkitchen.hu>`_
- `https://www.strongrfastr.com <https://www.strongrfastr.com>`_
- `https://sunbasket.com/ <https://sunbasket.com>`_
- `https://sundpaabudget.dk/ <https://sundpaabudget.dk>`_
- `https://www.sunset.com/ <https://www.sunset.com/>`_
- `https://sweetcsdesigns.com/ <https://www.sweetcsdesigns.com/>`_
- `https://sweetpeasandsaffron.com/ <https://sweetpeasandsaffron.com/>`_
- `https://www.taste.com.au/ <https://www.taste.com.au/>`_
- `https://www.tasteatlas.com/ <https://www.tasteatlas.com/>`_
- `https://tasteofhome.com <https://tasteofhome.com>`_
- `https://tastesbetterfromscratch.com <https://tastesbetterfromscratch.com>`_
- `https://tastesoflizzyt.com <https://tastesoflizzyt.com>`_
- `https://tasty.co <https://tasty.co>`_
- `https://tastykitchen.com/ <https://tastykitchen.com>`_
- `https://theclevercarrot.com/ <https://theclevercarrot.com>`_
- `https://www.thecookierookie.com/ <https://www.thecookierookie.com/>`_
- `https://thecookingguy.com/ <https://thecookingguy.com>`_
- `https://theexpertguides.com/ <https://theexpertguides.com>`_
- `https://thehappyfoodie.co.uk/ <https://thehappyfoodie.co.uk>`_
- `https://thekitchencommunity.org/ <https://thekitchencommunity.org/>`_
- `https://www.thekitchenmagpie.com/ <https://www.thekitchenmagpie.com>`_
- `https://thekitchn.com/ <https://thekitchn.com/>`_
- `https://www.themagicalslowcooker.com/ <https://www.themagicalslowcooker.com/>`_
- `https://themodernproper.com/ <https://themodernproper.com/>`_
- `https://www.thepalatablelife.com <https://www.thepalatablelife.com/>`_
- `https://thepioneerwoman.com/ <https://thepioneerwoman.com>`_
- `https://therecipecritic.com/ <https://therecipecritic.com>`_
- `https://thesaltymarshmallow.com/ <https://thesaltymarshmallow.com/>`_
- `https://thespruceeats.com/ <https://thespruceeats.com/>`_
- `https://thevintagemixer.com/ <https://thevintagemixer.com>`_
- `https://thewoksoflife.com/ <https://thewoksoflife.com/>`_
- `https://thinlicious.com/ <https://thinlicious.com/>`_
- `https://tidymom.net <https://tidymom.net>`_
- `https://tine.no/ <https://tine.no>`_
- `https://tofoo.co.uk <https://tofoo.co.uk>`_
- `https://tudogostoso.com.br/ <https://www.tudogostoso.com.br/>`_
- `https://twopeasandtheirpod.com/ <http://twopeasandtheirpod.com>`_
- `https://uitpaulineskeuken.nl/ <https://uitpaulineskeuken.nl>`_
- `https://unsophisticook.com/ <https://unsophisticook.com/>`_
- `https://usapears.org/ <https://usapears.org>`_
- `https://www.valdemarsro.dk/ <https://www.valdemarsro.dk/>`_
- `https://vanillaandbean.com/ <https://vanillaandbean.com>`_
- `https://www.vegetarbloggen.no/ <https://www.vegetarbloggen.no/>`_
- `https://vegolosi.it/ <https://vegolosi.it>`_
- `https://vegrecipesofindia.com/ <https://www.vegrecipesofindia.com/>`_
- `https://www.waitrose.com/ <https://www.waitrose.com/>`_
- `https://watchwhatueat.com/ <https://watchwhatueat.com/>`_
- `https://wearenotmartha.com/ <https://wearenotmartha.com/>`_
- `https://www.weightwatchers.com/ <https://www.weightwatchers.com/>`_ (*)
- `https://www.wellplated.com/ <https://www.wellplated.com/>`_
- `https://whatsgabycooking.com/ <https://whatsgabycooking.com>`_
- `https://whole30.com/ <https://whole30.com/>`_
- `https://www.wholefoodsmarket.com/ <https://www.wholefoodsmarket.com/>`_
    - `.co.uk <https://www.wholefoodsmarket.co.uk/>`_
- `https://www.williams-sonoma.com/ <https://www.williams-sonoma.com/>`_
- `https://womensweeklyfood.com.au/ <https://womensweeklyfood.com.au/>`_
- `https://woolworths.com.au/shop/recipes <https://www.woolworths.com.au/shop/recipes/>`_
- `https://woop.co.nz/ <https://woop.co.nz/>`_
- `https://yemek.com/ <https://yemek.com>`_
- `https://yummly.com/ <https://yummly.com>`_ (*)
- `https://www.zaubertopf.de <https://www.zaubertopf.de>`_
- `https://zeit.de/ (wochenmarkt) <https://www.zeit.de/zeit-magazin/wochenmarkt/index>`_
- `https://zenbelly.com/ <https://zenbelly.com>`_

(*) offline saved files only

Contribute
----------

If you spot a design change (or something else) that makes the scraper unable to work for a given site - please fire an issue asap.

If you are programmer PRs with fixes are warmly welcomed and acknowledged with a virtual beer. You can find documentation on how to develop scrapers `here <https://github.com/hhursev/recipe-scrapers/blob/main/docs/README.md>`__.

If you want a scraper for a new site added
------------------------------------------

- Open an `Issue <https://github.com/hhursev/recipe-scraper/issues/new>`_ providing us the site name, as well as a recipe link from it.
- You are a developer and want to code the scraper on your own:

  - If `Schema is available <#faq>`_ on the site - `you can go like this. <https://github.com/hhursev/recipe-scrapers/pull/176>`_
  - Otherwise, scrape the HTML - `like this <https://github.com/hhursev/recipe-scrapers/commit/ffee963d04>`_
  - Generating a new scraper class:

    .. code:: shell

        python generate.py <ClassName> <URL>

    - **ClassName**: The name of the new scraper class.
    - **URL**: The URL of an example recipe from the target site. The content will be stored in ``test_data`` to be used with the test class.

    You can find a more detailed guide `here <https://github.com/hhursev/recipe-scrapers/blob/main/docs/how-to-develop-scraper.md>`__.

For Devs / Contribute
---------------------

Assuming you have ``>=python3.8`` installed, navigate to the directory where you want this project to live in and drop these lines

.. code:: shell

    git clone git@github.com:hhursev/recipe-scrapers.git &&
    cd recipe-scrapers &&
    python -m venv .venv &&
    source .venv/bin/activate &&
    python -m pip install --upgrade pip &&
    pip install -r requirements-dev.txt &&
    pip install pre-commit &&
    pre-commit install &&
    python -m unittest

In case you want to run a single unittest for a newly developed scraper

.. code:: shell

    python -m unittest -k <test_file_name>

FAQ
---
- **How do I know if a website has a Recipe Schema?** Run in python shell:

.. code:: python

    from recipe_scrapers import scrape_me
    scraper = scrape_me('<url of a recipe from the site>', wild_mode=True)
    # if no error is raised - there's schema available:
    scraper.title()
    scraper.instructions()  # etc.

Netiquette
----------

If you're using this library to collect large numbers of recipes from the web, please use the software responsibly and try to avoid creating high volumes of network traffic.

Python's standard library provides a ``robots.txt`` `parser <https://docs.python.org/3/library/urllib.robotparser.html>`_ that may be helpful to automatically follow common instructions specified by websites for web crawlers.

Another parser option -- particularly if you find that many web requests from ``urllib.robotparser`` are blocked -- is the `robotexclusionrulesparser <https://pypi.org/project/robotexclusionrulesparser/>`_ library.

Special thanks to:
------------------

All the `contributors that helped improving <https://github.com/hhursev/recipe-scrapers/graphs/contributors>`_  the package. You are awesome!

.. image:: https://contrib.rocks/image?repo=hhursev/recipe-scrapers
   :target: https://github.com/hhursev/recipe-scrapers/graphs/contributors

Extra:
------
| You want to gather recipes data?
| You have an idea you want to implement?
| Check out `our "Share a project" wall <https://github.com/hhursev/recipe-scrapers/issues/9>`_ - it may save you time and spark ideas!


# PyPSA - Python for Power System Analysis

[![PyPI version](https://img.shields.io/pypi/v/pypsa.svg)](https://pypi.python.org/pypi/pypsa)
[![Conda version](https://img.shields.io/conda/vn/conda-forge/pypsa.svg)](https://anaconda.org/conda-forge/pypsa)
[![Tests](https://github.com/PyPSA/PyPSA/actions/workflows/test.yml/badge.svg)](https://github.com/PyPSA/PyPSA/actions/workflows/test.yml)
[![Code coverage](https://codecov.io/gh/PyPSA/PyPSA/branch/master/graph/badge.svg?token=kCpwJiV6Jr)](https://codecov.io/gh/PyPSA/PyPSA)
[![Documentation Status](https://readthedocs.org/projects/pypsa/badge/?version=latest)](https://pypsa.readthedocs.io/en/latest/?badge=latest)
[![License](https://img.shields.io/pypi/l/pypsa.svg)](LICENSE.txt)
[![Zenodo](https://zenodo.org/badge/DOI/10.5281/zenodo.3946412.svg)](https://doi.org/10.5281/zenodo.3946412)
[![Examples of use](https://mybinder.org/badge_logo.svg)](https://mybinder.org/v2/gh/PyPSA/PyPSA/master?filepath=examples%2Fnotebooks)
[![pre-commit.ci status](https://results.pre-commit.ci/badge/github/PyPSA/PyPSA/master.svg)](https://results.pre-commit.ci/latest/github/PyPSA/PyPSA/master)
[![Code style: black](https://img.shields.io/badge/code%20style-black-000000.svg)](https://github.com/psf/black)
[![Discord](https://img.shields.io/discord/911692131440148490?logo=discord)](https://discord.gg/AnuJBk23FU)
[![Contributor Covenant](https://img.shields.io/badge/Contributor%20Covenant-2.1-4baaaa.svg)](CODE_OF_CONDUCT.md)
[![Stack Exchange questions](https://img.shields.io/stackexchange/stackoverflow/t/pypsa)](https://stackoverflow.com/questions/tagged/pypsa)

PyPSA stands for "Python for Power System Analysis". It is pronounced
"pipes-ah".

PyPSA is an open source toolbox for simulating and optimising modern power and
energy systems that include features such as conventional generators with unit
commitment, variable wind and solar generation, storage units, coupling to other
energy sectors, and mixed alternating and direct current networks. PyPSA is
designed to scale well with large networks and long time series.

This project is maintained by the [Department of Digital Transformation in
Energy Systems](https://tub-ensys.github.io) at the [Technical University of
Berlin](https://www.tu.berlin). Previous versions were developed by the Energy
System Modelling group at the [Institute for Automation and Applied
Informatics](https://www.iai.kit.edu/english/index.php) at the [Karlsruhe
Institute of Technology](http://www.kit.edu/english/index.php) funded by the
[Helmholtz Association](https://www.helmholtz.de/en/), and by the [Renewable
Energy
Group](https://fias.uni-frankfurt.de/physics/schramm/renewable-energy-system-and-network-analysis/)
at [FIAS](https://fias.uni-frankfurt.de/) to carry out simulations for the
[CoNDyNet project](https://fias.institute/en/projects/condynet/), financed by the [German Federal
Ministry for Education and Research (BMBF)](https://www.bmbf.de/en/index.html)
as part of the [Stromnetze Research
Initiative](http://forschung-stromnetze.info/projekte/grundlagen-und-konzepte-fuer-effiziente-dezentrale-stromnetze/).

## Functionality

PyPSA can calculate:

-   static power flow (using both the full non-linear network equations and the
    linearised network equations)
-   linear optimal power flow (least-cost optimisation of power plant and
    storage dispatch within network constraints, using the linear network
    equations, over several snapshots)
-   security-constrained linear optimal power flow
-   total electricity/energy system least-cost investment optimisation (using
    linear network equations, over several snapshots and investment periods
    simultaneously for optimisation of generation and storage dispatch and
    investment in the capacities of generation, storage, transmission and other
    infrastructure)

It has models for:

-   meshed multiply-connected AC and DC networks, with controllable converters
    between AC and DC networks
-   standard types for lines and transformers following the implementation in
    [pandapower](https://www.pandapower.org/)
-   conventional dispatchable generators and links with unit commitment
-   generators with time-varying power availability, such as wind and solar
    generators
-   storage units with efficiency losses
-   simple hydroelectricity with inflow and spillage
-   coupling with other energy carriers (e.g. resistive Power-to-Heat (P2H),
    Power-to-Gas (P2G), battery electric vehicles (BEVs), Fischer-Tropsch,
    direct air capture (DAC))
-   basic components out of which more complicated assets can be built, such as
    Combined Heat and Power (CHP) units and heat pumps.

## Documentation

[Documentation](https://pypsa.readthedocs.io/en/latest/index.html)

[Quick start](https://pypsa.readthedocs.io/en/latest/quick_start.html)

[Examples](https://pypsa.readthedocs.io/en/latest/examples-basic.html)

[Known users of
PyPSA](https://pypsa.readthedocs.io/en/latest/users.html)

## Installation

pip:

```pip install pypsa```

conda/mamba:

```conda install -c conda-forge pypsa```

Additionally, install a solver.

## Usage

```py
import pypsa

# create a new network
n = pypsa.Network()
n.add("Bus", "mybus")
n.add("Load", "myload", bus="mybus", p_set=100)
n.add("Generator", "mygen", bus="mybus", p_nom=100, marginal_cost=20)

# load an example network
n = pypsa.examples.ac_dc_meshed()

# run the optimisation
n.optimize()

# plot results
n.generators_t.p.plot()
n.plot()

# get statistics
n.statistics()
n.statistics.energy_balance()
```

There are [more extensive
examples](https://pypsa.readthedocs.io/en/latest/examples-basic.html) available
as [Jupyter notebooks](https://jupyter.org/). They are also described in the
[doc/examples.rst](doc/examples.rst) and are available as Python scripts in
[examples/](examples/).

## Screenshots

[PyPSA-Eur](https://github.com/PyPSA/pypsa-eur) optimising capacities of
generation, storage and transmission lines (9% line volume expansion allowed)
for a 95% reduction in CO2 emissions in Europe compared to 1990 levels

![image](doc/img/elec_s_256_lv1.09_Co2L-3H.png)

[SciGRID model](https://power.scigrid.de/) simulating the German power system
for 2015.

![image](doc/img/stacked-gen_and_storage-scigrid.png)

![image](doc/img/lmp_and_line-loading.png)

## Dependencies

PyPSA is written and tested to be compatible with Python 3.7 and above.
The last release supporting Python 2.7 was PyPSA 0.15.0.

It leans heavily on the following Python packages:

-   [pandas](http://pandas.pydata.org/) for storing data about
    components and time series
-   [numpy](http://www.numpy.org/) and [scipy](http://scipy.org/) for
    calculations, such as linear algebra and sparse matrix calculations
-   [networkx](https://networkx.github.io/) for some network
    calculations
-   [matplotlib](https://matplotlib.org/) for static plotting
-   [linpy](https://github.com/PyPSA/linopy) for preparing optimisation problems
    (currently only linear and mixed integer linear optimisation)
-   [cartopy](https://scitools.org.uk/cartopy) for plotting the
    baselayer map
-   [pytest](http://pytest.org/) for unit testing
-   [logging](https://docs.python.org/3/library/logging.html) for
    managing messages

The optimisation uses interface libraries like `linopy` which are independent of
the preferred solver. You can use e.g. one of the free solvers
[HiGHS](https://highs.dev/), [GLPK](https://www.gnu.org/software/glpk/) and
[CLP/CBC](https://github.com/coin-or/Cbc/) or the commercial solver
[Gurobi](http://www.gurobi.com/) for which free academic licenses are available.

## Documentation

Please check the [documentation](https://pypsa.readthedocs.io).

## Contributing and Support

We strongly welcome anyone interested in contributing to this project. If you have any ideas, suggestions or encounter problems, feel invited to file issues or make pull requests on GitHub.

-   In case of code-related **questions**, please post on [stack overflow](https://stackoverflow.com/questions/tagged/pypsa).
-   For non-programming related and more general questions please refer to the [mailing list](https://groups.google.com/group/pypsa).
-   To **discuss** with other PyPSA users, organise projects, share news, and get in touch with the community you can use the [discord server](https://discord.gg/AnuJBk23FU).
-   For **bugs and feature requests**, please use the [PyPSA Github Issues page](https://github.com/PyPSA/PyPSA/issues).
-   For **troubleshooting**, please check the [troubleshooting](https://pypsa.readthedocs.io/en/latest/troubleshooting.html) in the documentation.

Detailed guidelines can be found in the [Contributing](https://pypsa.readthedocs.io/en/latest/contributing.html) section of our documentation.

## Code of Conduct

Please respect our [code of conduct](CODE_OF_CONDUCT.md).

## Citing PyPSA

If you use PyPSA for your research, we would appreciate it if you would
cite the following paper:

-   T. Brown, J. Hörsch, D. Schlachtberger, [PyPSA: Python for Power
    System Analysis](https://arxiv.org/abs/1707.09913), 2018, [Journal
    of Open Research
    Software](https://openresearchsoftware.metajnl.com/), 6(1),
    [arXiv:1707.09913](https://arxiv.org/abs/1707.09913),
    [DOI:10.5334/jors.188](https://doi.org/10.5334/jors.188)

Please use the following BibTeX:

    @article{PyPSA,
       author = {T. Brown and J. H\"orsch and D. Schlachtberger},
       title = {{PyPSA: Python for Power System Analysis}},
       journal = {Journal of Open Research Software},
       volume = {6},
       issue = {1},
       number = {4},
       year = {2018},
       eprint = {1707.09913},
       url = {https://doi.org/10.5334/jors.188},
       doi = {10.5334/jors.188}
    }

If you want to cite a specific PyPSA version, each release of PyPSA is
stored on [Zenodo](https://zenodo.org/) with a release-specific DOI. The
release-specific DOIs can be found linked from the overall PyPSA Zenodo
DOI for Version 0.17.1 and onwards:

[![image](https://zenodo.org/badge/DOI/10.5281/zenodo.3946412.svg)](https://doi.org/10.5281/zenodo.3946412)

or from the overall PyPSA Zenodo DOI for Versions up to 0.17.0:

[![image](https://zenodo.org/badge/DOI/10.5281/zenodo.786605.svg)](https://doi.org/10.5281/zenodo.786605)

# Licence

Copyright 2015-2024 [PyPSA
Developers](https://pypsa.readthedocs.io/en/latest/developers.html)

PyPSA is licensed under the open source [MIT
License](https://github.com/PyPSA/PyPSA/blob/master/LICENSE.txt).


# X2Paddle

[![PyPI - X2Paddle Version](https://img.shields.io/pypi/v/x2paddle.svg?label=pip&logo=PyPI&logoColor=white)](https://pypi.org/project/x2paddle/)
[![PyPI Status](https://pepy.tech/badge/x2paddle/month)](https://pepy.tech/project/x2paddle)
[![License](https://img.shields.io/badge/license-Apache%202-blue.svg)](LICENSE)
[![Version](https://img.shields.io/github/release/PaddlePaddle/X2Paddle.svg)](https://github.com/PaddlePaddle/X2Paddle/releases)
![python version](https://img.shields.io/badge/python-3.5+-orange.svg)

## 简介

X2Paddle是飞桨生态下的模型转换工具，致力于帮助其它深度学习框架用户快速迁移至飞桨框架。目前支持**推理模型的框架转换**与**PyTorch训练代码迁移**，我们还提供了详细的不同框架间API对比文档，降低开发者将模型迁移到飞桨的时间成本。

## 特性

- **支持主流深度学习框架**

  - 目前已经支持Caffe/TensorFlow/ONNX/PyTorch四大框架的预测模型的转换，PyTorch训练项目的转换，涵盖了目前市面主流深度学习框架

- **支持的模型丰富**

  - 在主流的CV和NLP模型上支持大部分模型转换，目前X2Paddle支持130+ PyTorch OP，90+ ONNX OP，90+ TensorFlow OP 以及 30+ Caffe OP，详见 ***[支持列表](./docs/inference_model_convertor/op_list.md)***

- **简洁易用**

  - 一条命令行或者一个API即可完成模型转换

## 能力

- **预测模型转换**

  - 支持Caffe/TensorFlow/ONNX/PyTorch的模型一键转为飞桨的预测模型，并使用PaddleInference/PaddleLite进行CPU/GPU/Arm等设备的部署

- **PyTorch训练项目转换**

  - 支持PyTorch项目Python代码（包括训练、预测）一键转为基于飞桨框架的项目代码，帮助开发者快速迁移项目，并可享受[AIStudio平台](https://aistudio.baidu.com/)对于飞桨框架提供的海量免费计算资源[**【新功能，试一下！】**](/docs/pytorch_project_convertor/README.md)

- **API对应文档**

  - 详细的API文档对比分析，帮助开发者快速从PyTorch框架的使用迁移至飞桨框架的使用，大大降低学习成本 [**【新内容，了解一下！】**](docs/pytorch_project_convertor/API_docs/README.md)

## 安装

### 环境依赖
- python >= 3.5
- paddlepaddle >= 2.2.2
- tensorflow == 1.14 (如需转换TensorFlow模型)
- onnx >= 1.6.0 (如需转换ONNX模型)
- torch >= 1.5.0 (如需转换PyTorch模型)
- paddlelite >= 2.9.0 (如需一键转换成Paddle-Lite支持格式,推荐最新版本)

### pip安装(推荐）

如需使用稳定版本，可通过pip方式安装X2Paddle：
```
pip install x2paddle
```

### 源码安装

如需体验最新功能，可使用源码安装方式：
```
git clone https://github.com/PaddlePaddle/X2Paddle.git
cd X2Paddle
git checkout develop
python setup.py install
```

## 快速开始

### 功能一：推理模型转换

#### PyTorch模型转换
``` python
from x2paddle.convert import pytorch2paddle
pytorch2paddle(module=torch_module,
               save_dir="./pd_model",
               jit_type="trace",
               input_examples=[torch_input])
# module (torch.nn.Module): PyTorch的Module。
# save_dir (str): 转换后模型的保存路径。
# jit_type (str): 转换方式。默认为"trace"。
# input_examples (list[torch.tensor]): torch.nn.Module的输入示例，list的长度必须与输入的长度一致。默认为None。
```
```script```模式以及更多细节可参考[PyTorch模型转换文档](./docs/inference_model_convertor/pytorch2paddle.md)。

#### TensorFlow模型转换
```shell
x2paddle --framework=tensorflow --model=tf_model.pb --save_dir=pd_model
```

#### ONNX模型转换
```shell
x2paddle --framework=onnx --model=onnx_model.onnx --save_dir=pd_model
```

#### Caffe模型转换
```shell
x2paddle --framework=caffe --prototxt=deploy.prototxt --weight=deploy.caffemodel --save_dir=pd_model
```

#### 转换参数说明

| 参数                 | 作用                                                         |
| -------------------- | ------------------------------------------------------------ |
| --framework          | 源模型类型 (tensorflow、caffe、onnx)                         |
| --prototxt           | 当framework为caffe时，该参数指定caffe模型的proto文件路径     |
| --weight             | 当framework为caffe时，该参数指定caffe模型的参数文件路径      |
| --save_dir           | 指定转换后的模型保存目录路径                                 |
| --model              | 当framework为tensorflow/onnx时，该参数指定tensorflow的pb模型文件或onnx模型路径 |
| --input_shape_dict   | **[可选]** For ONNX, 定义ONNX模型输入大小 |
| --caffe_proto        | **[可选]** 由caffe.proto编译成caffe_pb2.py文件的存放路径，当存在自定义Layer时使用，默认为None |
| --define_input_shape | **[可选]** For TensorFlow, 当指定该参数时，强制用户输入每个Placeholder的shape，见[文档Q2](./docs/inference_model_convertor/FAQ.md) |
| --enable_code_optim  | **[可选]** For PyTorch, 是否对生成代码进行优化，默认为False |
| --to_lite            | **[可选]** 是否使用opt工具转成Paddle-Lite支持格式，默认为False |
| --lite_valid_places  | **[可选]** 指定转换类型，可以同时指定多个backend(以逗号分隔)，opt将会自动选择最佳方式，默认为arm |
| --lite_model_type    | **[可选]** 指定模型转化类型，目前支持两种类型：protobuf和naive_buffer，默认为naive_buffer |
| --disable_feedback   | **[可选]** 是否关闭X2Paddle使用反馈；X2Paddle默认会统计用户在进行模型转换时的成功率，以及转换框架来源等信息，以便于帮忙X2Paddle根据用户需求进行迭代，不会上传用户的模型文件。如若不想参与反馈，可指定此参数为False即可 |

#### X2Paddle API
目前X2Paddle提供API方式转换模型，可参考[X2PaddleAPI](docs/inference_model_convertor/x2paddle_api.md)

#### 一键转换Paddle-Lite支持格式
可参考[使用X2paddle导出Padde-Lite支持格式](docs/inference_model_convertor/convert2lite_api.md)

### 功能二：PyTorch模型训练迁移

项目转换包括3个步骤

1. 项目代码预处理
2. 代码/预训练模型一键转换
3. 转换后代码后处理

详见[PyTorch训练项目转换文档](./docs/pytorch_project_convertor/README.md)。

### 使用VisualDL进行模型转换

飞桨可视化工具VisualDL已经将模型转换工具部署在官网提供服务，可以点击[服务链接](https://www.paddlepaddle.org.cn/paddle/visualdl/modelconverter/)进行在线的ONNX2Paddle模型转换。

![ONNX2Paddle](https://user-images.githubusercontent.com/22424850/226797893-ef697887-a056-445f-933e-f1bbc7c7df76.gif)

## 使用教程

1. [TensorFlow预测模型转换教程](./docs/inference_model_convertor/demo/tensorflow2paddle.ipynb)
2. [MMDetection模型转换指南](./docs/inference_model_convertor/toolkits/MMDetection2paddle.md)
3. [PyTorch预测模型转换教程](./docs/inference_model_convertor/demo/pytorch2paddle.ipynb)
4. [PyTorch训练项目转换教程](./docs/pytorch_project_convertor/demo/README.md)

## :hugs:贡献代码:hugs:

我们非常欢迎您为X2Paddle贡献代码或者提供使用建议。如果您可以修复某个issue或者增加一个新功能，欢迎给我们提交Pull Requests，如果有PyTorch训练项目转换需求欢迎随时提issue~


Description
===========

.. image:: https://github.com/mittagessen/kraken/actions/workflows/test.yml/badge.svg
    :target: https://github.com/mittagessen/kraken/actions/workflows/test.yml

kraken is a turn-key OCR system optimized for historical and non-Latin script
material.

kraken's main features are:

  - Fully trainable layout analysis, reading order, and character recognition
  - `Right-to-Left <https://en.wikipedia.org/wiki/Right-to-left>`_, `BiDi
    <https://en.wikipedia.org/wiki/Bi-directional_text>`_, and Top-to-Bottom
    script support
  - `ALTO <https://www.loc.gov/standards/alto/>`_, PageXML, abbyyXML, and hOCR
    output
  - Word bounding boxes and character cuts
  - Multi-script recognition support
  - `Public repository <https://zenodo.org/communities/ocr_models>`_ of model files
  - Variable recognition network architecture

Installation
============

kraken only runs on **Linux or Mac OS X**. Windows is not supported.

The latest stable releases can be installed either from `PyPi <https://pypi.org>`_:

::

  $ pip install kraken

or through `conda <https://anaconda.org>`_:

::

  $ conda install -c conda-forge -c mittagessen kraken

If you want direct PDF and multi-image TIFF/JPEG2000 support it is necessary to
install the `pdf` extras package for PyPi:

::

  $ pip install kraken[pdf]

or install `pyvips` manually with pip:

::

  $ pip install pyvips

Conda environment files are provided for the seamless installation of the main
branch as well:

::

  $ git clone https://github.com/mittagessen/kraken.git
  $ cd kraken
  $ conda env create -f environment.yml

or:

::

  $ git clone https://github.com/mittagessen/kraken.git
  $ cd kraken
  $ conda env create -f environment_cuda.yml

for CUDA acceleration with the appropriate hardware.

Finally you'll have to scrounge up a model to do the actual recognition of
characters. To download the default model for printed French text and place it
in the kraken directory for the current user:

::

  $ kraken get 10.5281/zenodo.10592716

A list of libre models available in the central repository can be retrieved by
running:

::

  $ kraken list

Quickstart
==========

Recognizing text on an image using the default parameters including the
prerequisite steps of binarization and page segmentation:

::

  $ kraken -i image.tif image.txt binarize segment ocr

To binarize a single image using the nlbin algorithm:

::

  $ kraken -i image.tif bw.png binarize

To segment an image (binarized or not) with the new baseline segmenter:

::

  $ kraken -i image.tif lines.json segment -bl

To segment and OCR an image using the default model(s):

::

  $ kraken -i image.tif image.txt segment -bl ocr -m catmus-print-fondue-large.mlmodel

All subcommands and options are documented. Use the ``help`` option to get more
information.

Documentation
=============

Have a look at the `docs <https://kraken.re>`_.

Related Software
================

These days kraken is quite closely linked to the `eScriptorium
<https://gitlab.com/scripta/escriptorium/>`_ project developed in the same eScripta research
group. eScriptorium provides a user-friendly interface for annotating data,
training models, and inference (but also much more). There is a `gitter channel
<https://gitter.im/escripta/escriptorium>`_ that is mostly intended for
coordinating technical development but is also a spot to find people with
experience on applying kraken on a wide variety of material.

Funding
=======

kraken is developed at the `École Pratique des Hautes Études <https://www.ephe.psl.eu>`_, `Université PSL <https://www.psl.eu>`_.

.. container:: twocol

   .. container::

        .. image:: https://raw.githubusercontent.com/mittagessen/kraken/main/docs/_static/normal-reproduction-low-resolution.jpg
          :width: 100
          :alt: Co-financed by the European Union

   .. container::

        This project was partially funded through the RESILIENCE project, funded from
        the European Union’s Horizon 2020 Framework Programme for Research and
        Innovation.

.. container:: twocol

   .. container::

      .. image:: https://projet.biblissima.fr/sites/default/files/2021-11/biblissima-baseline-sombre-ia.png
         :width: 400
         :alt: Received funding from the Programme d’investissements d’Avenir

   .. container::

        Ce travail a bénéficié d’une aide de l’État gérée par l’Agence Nationale de la
        Recherche au titre du Programme d’Investissements d’Avenir portant la référence
        ANR-21-ESRE-0005 (Biblissima+).


# Froide

[![Froide CI](https://github.com/okfde/froide/workflows/Froide%20CI/badge.svg)](https://github.com/okfde/froide/actions?query=workflow%3A%22Froide+CI%22)

Froide is a Freedom Of Information Portal using Django 3.2 on Python 3.8+.

It is used by the German and the Austrian FOI site, but it is fully
internationalized and written in English.

## Development on Froide

After clone, create a Python 3.8+ virtual environment and install dependencies:

```
python3 -m venv froide-env
source froide-env/bin/activate

# Install dev dependencies
pip install -r requirements-test.txt

# Install git pre-commit hook
pre-commit install
```

### Start services

You can run your own Postgres+PostGIS database and Elasticsearch service or run them with Docker.

You need [docker](https://www.docker.com/community-edition) and [docker-compose](https://docs.docker.com/compose/). Make sure Docker is running and use the following command:

```
docker-compose up
```

This will start Postgres and Elasticsearch and listen on port 5432 and 9200 respectively. You can adjust the port mapping in the `docker-compose.yml`.

### Setup database and search index, start server

If you need to adjust settings, you can copy the `froide/local_settings.py.example` to `froide/local_settings.py` and edit it. More steps:

```
# To initialise the database:
python manage.py migrate --skip-checks
# Create a superuser
python manage.py createsuperuser
# Create and populate search index
python manage.py search_index --create
python manage.py search_index --populate
# Run the Django development server
python manage.py runserver
```

### Run tests

Make sure the services are running.

```
# Run all tests
make test
# Run only unit/integration tests
make testci
# Run only end-to-end tests
make testui
```

### Development tooling

For Python code, we use flake8 following black code style. JavaScript, Vue and SCSS files are formatted and linted with ESLint and Prettier.

Make sure to have pre-commit hooks registered (`pre-commit install`). For VSCode, the [Python](https://code.visualstudio.com/docs/python/linting), [ESLint](https://marketplace.visualstudio.com/items?itemName=dbaeumer.vscode-eslint) and [Vetur](https://marketplace.visualstudio.com/items?itemName=octref.vetur) extensions are helpful, with these workspace settings recommended:

```json
{
  "eslint.format.enable": true,
  "eslint.packageManager": "yarn",
  "vetur.format.defaultFormatter.css": "prettier",
  "vetur.format.defaultFormatter.html": "prettier",
  "vetur.format.defaultFormatter.js": "prettier-eslint"
}
```

### Upgrade dependencies

```
# with pip-tools
pip-compile -U requirements.in
pip-compile -U requirements-test.in
```

## Docs

[Read the documentation](http://froide.readthedocs.org/en/latest/) including a [Getting Started Guide](http://froide.readthedocs.org/en/latest/gettingstarted/).

Froide is supported by the [Open Knowledge Foundation Germany](http://www.okfn.de/) and [Open Knowledge Foundation International](http://okfn.org/).

## License

Froide is licensed under the MIT License.


# kcli repository

[![CI](https://github.com/karmab/kcli/actions/workflows/ci.yml/badge.svg)](https://github.com/karmab/kcli/actions/workflows/ci.yml)
[![Copr](https://copr.fedorainfracloud.org/coprs/karmab/kcli/package/kcli/status_image/last_build.png)](https://copr.fedorainfracloud.org/coprs/karmab/kcli/package/kcli)

![Screenshot](kcli.png)

## Demo!

![](kcli.gif)

# Description

This tool is meant to interact with existing virtualization providers (libvirt, KubeVirt, oVirt, OpenStack, VMware vSphere, GCP and AWS) and to easily deploy and customize VMs from cloud images.

You can also interact with those VMs (list, info, ssh, start, stop, delete, console, serialconsole, add/delete disk, add/delete nic, ...).

Furthermore, you can deploy VMs using predefined profiles, several at once using plan files or entire products for which plans were already created for you.

Refer to the [documentation](https://kcli.readthedocs.io) for more information

## Quick Start

```
curl -s https://raw.githubusercontent.com/karmab/kcli/main/install.sh | bash
```

Deploy your first vm with:

```
kcli create vm -i centos8stream myvm
kcli list vm
# wait 5-10 seconds for vm to grab an ip
kcli ssh myvm
kcli delete vm
```

##  What you can do

- Interact with all the virtualization providers using a single tool
- Declare all your objects (VM, containers, networks, ansible playbooks, ...) in a single yaml plan file with a simple syntax
- Customize a plan deployment using parameters and Jinja templating
- Adjust VMs from a plan (memory, CPU, flavor, disks and NICs) to match what's defined in the plans
- Inject all configuration with cloud-init/Ignition or the equivalent in cloud providers
- Use profiles to launch VMs with same hardware characteristics
- Launch a plan from an URL
- Share your plan or use existing ones from Github repo as products
- Use the existing plans to deploy Kubernetes, OpenShift, OpenStack, oVirt, kubevirt, ...
- Use the existing cloud images for each distribution
- Easily share private keys between your VMs
- Handle dns entries for the VMs
- Automatically subscribe your RHEL VMs
- Get a push button notification when a vm has finished its deployment
- Alternatively use web UI to do the same

## Contributors

See [contributors on GitHub](https://github.com/karmab/kcli/graphs/contributors)

## Copyright

Copyright 2017-2019 Karim Boumedhel

Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.

## Problems?

Open an issue or join us on slack

- [k8s slack instance](https://slack.k8s.io)
- [kcli slack channel](https://app.slack.com/client/T09NY5SBT/CU76B52JE)

Mc Fly!!!

karmab


|tox-pytest| |tox-checks| |appveyor| |coveralls| |codecov|

|scrutinizer| |codacy| |codeclimate|

|wheel| |packaging| |supported-versions|

|docs| |zenodo|

|version| |commits-since| |chat|

------------------------------

.. |tox-pytest| image:: https://github.com/oemof/oemof-solph/workflows/tox%20pytests/badge.svg?branch=dev
     :target: https://github.com/oemof/oemof-solph/actions?query=workflow%3A%22tox+checks%22

.. |tox-checks| image:: https://github.com/oemof/oemof-solph/workflows/tox%20checks/badge.svg?branch=dev
     :target: https://github.com/oemof/oemof-solph/actions?query=workflow%3A%22tox+checks%22

.. |packaging| image:: https://github.com/oemof/oemof-solph/workflows/packaging/badge.svg?branch=dev
     :target: https://github.com/oemof/oemof-solph/actions?query=workflow%3Apackaging

.. |docs| image:: https://readthedocs.org/projects/oemof-solph/badge/?style=flat
    :target: https://readthedocs.org/projects/oemof-solph
    :alt: Documentation Status

.. |appveyor| image:: https://ci.appveyor.com/api/projects/status/github/oemof/oemof-solph?branch=dev&svg=true
    :alt: AppVeyor Build Status
    :target: https://ci.appveyor.com/project/oemof-developer/oemof-solph

.. |coveralls| image:: https://coveralls.io/repos/oemof/oemof-solph/badge.svg?branch=dev&service=github
    :alt: Coverage Status
    :target: https://coveralls.io/github/oemof/oemof-solph

.. |codecov| image:: https://codecov.io/gh/oemof/oemof-solph/branch/dev/graphs/badge.svg?branch=dev
    :alt: Coverage Status
    :target: https://codecov.io/gh/oemof/oemof-solph

.. |codacy| image:: https://api.codacy.com/project/badge/Grade/a6e5cb2dd2694c73895e142e4cf680d5
    :target: https://app.codacy.com/gh/oemof/oemof-solph/dashboard
    :alt: Codacy Code Quality Status

.. |codeclimate| image:: https://codeclimate.com/github/oemof/oemof-solph/badges/gpa.svg
   :target: https://codeclimate.com/github/oemof/oemof-solph
   :alt: CodeClimate Quality Status

.. |version| image:: https://img.shields.io/pypi/v/oemof.solph.svg
    :alt: PyPI Package latest release
    :target: https://pypi.org/project/oemof.solph

.. |wheel| image:: https://img.shields.io/pypi/wheel/oemof.solph.svg
    :alt: PyPI Wheel
    :target: https://pypi.org/project/oemof.solph

.. |supported-versions| image:: https://img.shields.io/pypi/pyversions/oemof.solph.svg
    :alt: Supported versions
    :target: https://pypi.org/project/oemof.solph

.. |supported-implementations| image:: https://img.shields.io/pypi/implementation/oemof.solph.svg
    :alt: Supported implementations
    :target: https://pypi.org/project/oemof.solph

.. |commits-since| image:: https://img.shields.io/github/commits-since/oemof/oemof-solph/v0.5.2/dev
    :alt: Commits since latest release
    :target: https://github.com/oemof/oemof-solph/compare/v0.5.2...dev

.. |zenodo| image:: https://zenodo.org/badge/DOI/10.5281/zenodo.596235.svg
    :alt: Zenodo DOI
    :target: https://doi.org/10.5281/zenodo.596235

.. |scrutinizer| image:: https://img.shields.io/scrutinizer/quality/g/oemof/oemof-solph/dev.svg
    :alt: Scrutinizer Status
    :target: https://scrutinizer-ci.com/g/oemof/oemof-solph/

.. |chat| image:: https://img.shields.io/badge/chat-oemof:matrix.org-%238ADCF7
     :alt: matrix-chat
     :target: https://matrix.to/#/#oemof:matrix.org

.. figure:: https://raw.githubusercontent.com/oemof/oemof-solph/492e3f5a0dda7065be30d33a37b0625027847518/docs/_logo/logo_oemof_solph_FULL.svg
    :align: center

------------------------------

===========
oemof.solph
===========

**A model generator for energy system modelling and optimisation (LP/MILP)**

.. contents::
    :depth: 2
    :local:
    :backlinks: top

Introduction
============

The oemof.solph package is part of the
`Open energy modelling framework (oemof) <https://github.com/oemof/oemof>`_.
This is an organisational framework to bundle tools for energy (system) modelling.
oemof-solph is a model generator for energy system modelling and optimisation.

The package ``oemof.solph`` is very often called just ``oemof``.
This is because installing the ``oemof`` meta package was once the best way to get ``oemof.solph``.
Notice that you should prefeably install ``oemof.solph`` instead of ``oemof``
if you want to use ``solph``.

Everybody is welcome to use and/or develop oemof.solph.
Read our `contribution <https://oemof.readthedocs.io/en/latest/contributing.html>`_ section.

Contribution is already possible on a low level by simply fixing typos in
oemof's documentation or rephrasing sections which are unclear.
If you want to support us that way please fork the oemof-solph repository to your own
GitHub account and make changes as described in the `github guidelines <https://docs.github.com/en/get-started/quickstart/hello-world>`_

If you have questions regarding the use of oemof including oemof.solph you can visit the openmod forum (`tag oemof <https://forum.openmod-initiative.org/tags/c/qa/oemof>`_ or `tag oemof-solph <https://forum.openmod-initiative.org/tags/c/qa/oemof-solph>`_) and open a new thread if your questions hasn't been already answered.

Keep in touch! - You can become a watcher at our `github site <https://github.com/oemof/oemof>`_,
but this will bring you quite a few mails and might be more interesting for developers.
If you just want to get the latest news, like when is the next oemof meeting,
you can follow our news-blog at `oemof.org <https://oemof.org/>`_.

Documentation
=============
The `oemof.solph documentation <https://oemof-solph.readthedocs.io/>`_ is powered by readthedocs. Use the `project site <https://readthedocs.org/projects/oemof>`_ of oemof.solph to choose the version of the documentation. Go to the `download page <https://readthedocs.org/projects/oemof/downloads/>`_ to download different versions and formats (pdf, html, epub) of the documentation.

.. _installation_label:

Installation
============

If you have a working Python installation, use pypi to install the latest version of oemof.solph.
Python >= 3.8 is recommended. Lower versions may work but are not tested.

We highly recommend to use virtual environments.
Please refer to the documentation of your Python distribution (e.g. Anaconda,
Micromamba, or the version of Python that came with your Linux installation)
to learn how to set up and use virtual environments.

::

    (venv) pip install oemof.solph

If you want to use the latest features, you might want to install the **developer version**. The developer version is not recommended for productive use::

    (venv) pip install https://github.com/oemof/oemof-solph/archive/dev.zip

For running an oemof-solph optimisation model, you need to install a solver.
Following you will find guidelines for the installation process for different operating systems.

.. _windows_solver_label:
.. _linux_solver_label:

Installing a solver
-------------------

There are several solvers that can work with oemof, both open source and commercial.
Two open source solvers are widely used (CBC and GLPK), but oemof suggests CBC (Coin-or branch and cut).
It may be useful to compare results of different solvers to see which performs best.
Other commercial solvers, like Gurobi or Cplex, are also options.
Have a look at the `pyomo docs <https://pyomo.readthedocs.io/en/stable/solving_pyomo_models.html#supported-solvers>`_ to learn about which solvers are supported.

Check the solver installation by executing the test_installation example below (see section Installation Test).

**Linux**

To install the solvers have a look at the package repository of your Linux distribution or search for precompiled packages. GLPK and CBC ares available at Debian, Feodora, Ubuntu and others.

**Windows**

 1. Download `CBC <https://github.com/coin-or/Cbc/releases>`_
 2. Download `GLPK (64/32 bit) <https://sourceforge.net/projects/winglpk/>`_
 3. Unpack CBC/GLPK to any folder (e.g. C:/Users/Somebody/my_programs)
 4. Add the path of the executable files of both solvers to the PATH variable using `this tutorial <https://www.computerhope.com/issues/ch000549.htm>`_
 5. Restart Windows

Check the solver installation by executing the test_installation example (see the `Installation test` section).

**Mac OSX**

Please follow the installation instructions on the respective homepages for details.

CBC-solver: https://projects.coin-or.org/Cbc

GLPK-solver: http://arnab-deka.com/posts/2010/02/installing-glpk-on-a-mac/

If you install the CBC solver via brew (highly recommended), it should work without additional configuration.

**conda**

Provided you are using a Linux or MacOS, the CBC-solver can also be installed in a `conda` environment. Please note, that it is highly recommended to `use pip after conda <https://www.anaconda.com/blog/using-pip-in-a-conda-environment>`_, so:

.. code:: console

    (venv) conda install -c conda-forge coincbc
    (venv) pip install oemof.solph

.. _check_installation_label:

Installation test
-----------------

Test the installation and the installed solver by running the installation test
in your virtual environment:

.. code:: console

  (venv) oemof_installation_test

If the installation was successful, you will receive something like this:

.. code:: console

    *********
    Solver installed with oemof:
    glpk: working
    cplex: not working
    cbc: working
    gurobi: not working
    *********
    oemof.solph successfully installed.

as an output.

Contributing
============

A warm welcome to all who want to join the developers and contribute to
oemof.solph.

Information on the details and how to approach us can be found
`in the oemof documentation <https://oemof.readthedocs.io/en/latest/contributing.html>`_ .

Citing
======

For explicitly citing solph, you might want to refer to
`DOI:10.1016/j.simpa.2020.100028 <https://doi.org/10.1016/j.simpa.2020.100028>`_,
which gives an overview over the capabilities of solph.
The core ideas of oemof as a whole are described in
`DOI:10.1016/j.esr.2018.07.001 <https://doi.org/10.1016/j.esr.2018.07.001>`_
(preprint at `arXiv:1808.0807 <https://arxiv.org/abs/1808.08070v1>`_).
To allow citing specific versions, we use the zenodo project to get a DOI for each version.

Example Applications
====================

The combination of specific modules (often including other packages) is called an
application (app). For example, it can depict a concrete energy system model.
You can find a large variety of helpful examples in the documentation.
The examples show the optimisation of different energy systems and are supposed
to help new users to understand the framework's structure.

You are welcome to contribute your own examples via a `pull request <https://github.com/oemof/oemof-solph/pulls>`_
or by e-mailing us (see `here <https://oemof.org/contact/>`_ for contact information).

License
=======

Copyright (c) oemof developer group

Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.


[![GitHub Discussions](https://img.shields.io/github/discussions/calliope-project/calliope)](https://github.com/calliope-project/calliope/discussions)
[![Main branch build status](https://github.com/calliope-project/calliope/actions/workflows/commit-ci.yml/badge.svg?branch=main)](https://github.com/calliope-project/calliope/actions/workflows/commit-ci.yml)
[![Documentation build status](https://img.shields.io/readthedocs/calliope.svg?version=latest)](https://readthedocs.org/projects/calliope/builds/)
[![Test coverage](https://codecov.io/gh/calliope-project/calliope/graph/badge.svg?token=UM542yaYrh)](https://codecov.io/gh/calliope-project/calliope)
[![PyPI version](https://img.shields.io/pypi/v/calliope.svg)](https://pypi.python.org/pypi/calliope)
[![Anaconda.org/conda-forge version](https://img.shields.io/conda/vn/conda-forge/calliope.svg?label=conda)](https://anaconda.org/conda-forge/calliope)
[![JOSS DOI](https://img.shields.io/badge/JOSS-10.21105/joss.00825-green.svg)](https://doi.org/10.21105/joss.00825)

---

<img src="https://raw.githubusercontent.com/calliope-project/calliope/main/docs/img/logo.png" width="364">

*A multi-scale energy systems modelling framework* | [www.callio.pe](http://www.callio.pe/)

---

## Contents

- [Contents](#contents)
- [About](#about)
- [Quick start](#quick-start)
- [Documentation](#documentation)
- [Contributing](#contributing)
- [What's new](#whats-new)
- [Citing Calliope](#citing-calliope)
- [License](#license)

---

## About

Calliope is a framework to develop energy system models, with a focus on flexibility, high spatial and temporal resolution, the ability to execute many runs based on the same base model, and a clear separation of framework (code) and model (data). Its primary focus is on planning energy systems at scales ranging from urban districts to entire continents. In an optional operational it can also test a pre-defined system under different operational conditions.

A Calliope model consists of a collection of text files (in YAML and CSV formats) that fully define a model, with details on technologies, locations, resource potentials, etc. Calliope takes these files, constructs an optimization problem, solves it, and reports back results. Results can be saved to CSV or NetCDF files for further processing, or analysed directly in Python through Python's extensive scientific data processing capabilities provided by libraries like [Pandas](http://pandas.pydata.org/) and [xarray](https://docs.xarray.dev/en/stable/).

Calliope comes with several built-in analysis and visualisation tools. Having some knowledge of the Python programming language helps when running Calliope and using these tools, but is not a prerequisite.

## Quick start

Calliope can run on Windows, macOS and Linux. Installing it is quickest with the `mamba` package manager by running a single command: `mamba create -c conda-forge -n calliope calliope`.

See the documentation for more [information on installing](https://calliope.readthedocs.io/en/stable/user/installation.html).

Several easy to understand example models are [included with Calliope](https://github.com/calliope-project/calliope/tree/main/src/calliope/example_models) and accessible through the `calliope.examples` submodule.

The [tutorials in the documentation run through these examples](https://calliope.readthedocs.io/en/stable/user/tutorials.html). A good place to start is to look at these tutorials to get a feel for how Calliope works, and then to read the "Introduction", "Building a model", "Running a model", and "Analysing a model" sections in the online documentation.

More fully-featured examples that have been used in peer-reviewed scientific publications are available in our [model gallery](https://www.callio.pe/research/#models).

## Documentation

Documentation is available on [Read the Docs](https://calliope.readthedocs.io/en/stable/).

## Contributing

See our documentation for more on how to [contribute to Calliope](http://calliope.readthedocs.io/en/latest/contributing/).

## What's new

See changes made in recent versions in the [changelog](https://github.com/calliope-project/calliope/blob/main/CHANGELOG.md).

## Citing Calliope

If you use Calliope for academic work please cite:

Stefan Pfenninger and Bryn Pickering (2018). Calliope: a multi-scale energy systems modelling framework. *Journal of Open Source Software*, 3(29), 825. [doi: 10.21105/joss.00825](https://doi.org/10.21105/joss.00825)

## License

Copyright since 2013 Calliope contributors listed in AUTHORS

Licensed under the Apache License, Version 2.0 (the "License"); you
may not use this file except in compliance with the License. You may
obtain a copy of the License at

<http://www.apache.org/licenses/LICENSE-2.0>

Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.


# Rucio - Scientific Data Management

Rucio is a software framework that provides functionality to organize, manage, and access large volumes of scientific data using customisable policies.
The data can be spread across globally distributed locations and across heterogeneous data centers, uniting different storage and network technologies as a single federated entity.
Rucio offers advanced features such as distributed data recovery or adaptive replication, and is highly scalable, modular, and extensible.
Rucio has been originally developed to meet the requirements of the high-energy physics experiment ATLAS, and is continuously extended to support LHC experiments and other diverse scientific communities.

## Documentation

General information, API/REST description and guides can be found in our [documentation](https://rucio.cern.ch/documentation) or on our [webpage](https://rucio.cern.ch).

## Try it out

We provide a [dockerized environment](https://github.com/rucio/rucio/tree/master/etc/docker/dev) which serves both as a demo environment and a development environment.
It includes all the necessary preconfigured components for multiple storage and transfers developments.

## Developers

For information on how to contribute to Rucio, please refer and follow our [CONTRIBUTING](https://rucio.cern.ch/documentation/contributing) guidelines. We strongly recommend to use the [dockerized environment](https://github.com/rucio/rucio/tree/master/etc/docker/dev) for development.

## Operators

To learn how to deploy and configure Rucio, consult the [documentation](https://rucio.cern.ch/documentation) available online.

## Getting Support

If you are looking for support, please contact us via one of our [official channels](https://rucio.cern.ch/documentation/contact_us/).


![AutoGOAL Logo](https://autogoal.github.io/autogoal-banner.png)

---

[](https://pypi.org/project/autogoal/) [](https://pypi.org/project/autogoal/) [](https://autogoal.github.io/contributing) [](https://github.com/autogoal/autogoal/stargazers) [](https://twitter.com/auto_goal)

[](https://codecov.io/gh/autogoal/autogoal/)
[](https://hub.docker.com/r/autogoal/autogoal)
[](https://hub.docker.com/r/autogoal/autogoal)

[![Build and Push to Docker (main branch)](https://github.com/autogoal/autogoal/actions/workflows/docker.yml/badge.svg)](https://github.com/autogoal/autogoal/actions/workflows/docker.yml)

[![Release to PyPI](https://github.com/autogoal/autogoal/actions/workflows/release.yml/badge.svg)](https://github.com/autogoal/autogoal/actions/workflows/release.yml)

---

AutoGOAL is a Python library for automatically finding the best way to solve a given task.
It has been designed mainly for _Automated Machine Learning_ (aka [AutoML](https://www.automl.org))
but it can be used in any scenario where you have several possible ways to solve a given task.

Technically speaking, AutoGOAL is a framework for program synthesis, i.e., finding the best program to solve
a given problem, provided that the user can describe the space of all possible programs.
AutoGOAL provides a set of low-level components to define different spaces and efficiently search in them.
In the specific context of machine learning, AutoGOAL also provides high-level components that can be used as a black-box in almost any type of problem and dataset format.

## ⭐ Quickstart

AutoGOAL is first and foremost a framework for Automated Machine Learning.
As such, it comes pre-packaged with hundreds of low-level machine learning
algorithms that can be automatically assembled into pipelines for different problems.

The core of this functionality lies in the [`AutoML`](https://autogoal.github.io/api/autogoal.ml#automl) class.

To illustrate the simplicity of its use we will load a dataset and run an automatic classifier in it.
The following code will run for approximately 5 minutes on a classic dataset.

```python
from autogoal.datasets import cars
from autogoal.kb import (MatrixContinuousDense,
                         Supervised,
                         VectorCategorical)
from autogoal.ml import AutoML

# Load dataset
X, y = cars.load()

# Instantiate AutoML and define input/output types
automl = AutoML(
    input=(MatrixContinuousDense,
           Supervised[VectorCategorical]),
    output=VectorCategorical
)

# Run the pipeline search process
automl.fit(X, y)

# Report the best pipeline
print(automl.best_pipeline_)
print(automl.best_score_)
```

Sensible defaults are defined for each of the many parameters of `AutoML`.
Make sure to [read the documentation](https://autogoal.github.io/guide/) for more information.

## ⚙ Installation

The easiest way to get AutoGOAL up and running with all the dependencies is to pull the development Docker image, which is somewhat big:

    docker pull autogoal/autogoal

Instructions for setting up Docker are available [here](https://www.docker.com/get-started).

Once you have the development image downloaded, you can fire up a console and use AutoGOAL interactively.

![](https://autogoal.github.io/shell.svg)

If you prefer to not use Docker, or you don't want all the dependencies, you can also install AutoGOAL directly with pip:

    pip install autogoal

This will install the core library but you won't be able to use any of the underlying machine learning algorithms until you install the corresponding optional dependencies. You can install them all with:

    pip install autogoal[contrib]

To fine-pick which dependencies you want, read the [dependencies section](https://autogoal.github.io/dependencies/).

> ⚠ **NOTE**: By installing through `pip` you will get the latest release version of AutoGOAL, while by installing through Docker, you will get the latest development version.
>
> The development version is mostly up-to-date with the `main` branch, hence it will probably contain more features, but also more bugs, than the release version.

## 💻 CLI

You can use AutoGOAL directly from the CLI. To see options just type:

    autogoal

Using the CLI you can train and use AutoML models, download datasets and inspect the contrib libraries without writing a single line of code.

![](https://autogoal.github.io/shell/autogoal_cli.svg)

Read more in the [CLI documentation](https://autogoal.github.io/cli).

## 🤩 Demo

An online demo app is available at [autogoal.github.io/demo](https://autogoal.github.io/demo).
This app showcases the main features of AutoGOAL in interactive case studies.

To run the demo locally, simply type:

    docker run -p 8501:8501 autogoal/autogoal

And navigate to [localhost:8501](http://localhost:8501).

## ⚖ API stability

We make a conscious effort to maintain a consistent public API across versions, but the private API can change at any time.
In general, everything you can import from `autogoal` without underscores is considered public.

For example:

```python
# "clean" imports are part of the public API
from autogoal import optimize
from autogoal.ml import AutoML
from autogoal.contrib.sklearn import find_classes

# public members of public types as well
automl = AutoML
automl.fit(...)

# underscored imports are part of the private API
from autogoal.ml._automl import ...
from autogoal.contrib.sklearn._generated import ...

# as well as private members of any type
automl._input_type(...)

```

These are our consistency rules:

- Major breaking changes are introduced between major version updates, e.g., `x.0` and `y.0`. These can be additions, removals, or modifications of any kind in any part of the API.

- Between minor version updates, e.g., `1.x` and `1.y`, you can expect to find new functionality, but anything you can use from the   public API will still be there with a consistent semantic (save for bugfixes).

- Between micro version updates, e.g., `1.3.x` and `1.3.y`, the public API is frozen even for additions.

- The private API can be changed at all times.

⚠ While AutoGOAL is on public beta (versions `0.x`) the public API is considered unstable and thus everything can change. However, we try to keep breaking changes to a minimum.

## 📚 Documentation

This documentation is available online at [autogoal.github.io](https://autogoal.github.io). Check the following sections:

- [**User Guide**](https://autogoal.github.io/guide/): Step-by-step showcase of everything you need to know to use AutoGOAL.
- [**Examples**](https://autogoal.github.io/examples/): The best way to learn how to use AutoGOAL by practice.
- [**API**](https://autogoal.github.io/api/autogoal): Details about the public API for AutoGOAL.

The HTML version can be deployed offline by downloading the [AutoGOAL Docker image](https://hub.docker.com/autogoal/autogoal) and running:

    docker run -p 8000:8000 autogoal/autogoal mkdocs serve -a 0.0.0.0:8000

And navigating to [localhost:8000](http://localhost:8000).

## 📃 Publications

If you use AutoGOAL in academic research, please cite the following paper:

```bibtex
@article{estevez2020general,
  title={General-purpose hierarchical optimisation of machine learning pipelines with grammatical evolution},
  author={Est{\'e}vez-Velarde, Suilan and Guti{\'e}rrez, Yoan and Almeida-Cruz, Yudivi{\'a}n and Montoyo, Andr{\'e}s},
  journal={Information Sciences},
  year={2020},
  publisher={Elsevier},
  doi={10.1016/j.ins.2020.07.035}
}
```

The technologies and theoretical results leading up to AutoGOAL have been presented at different venues:

- [Optimizing Natural Language Processing Pipelines: Opinion Mining Case Study](https://link.springer.com/chapter/10.1007/978-3-030-33904-3_15) marks the inception of the idea of using evolutionary optimization with a probabilistic search space for pipeline optimization.

- [AutoML Strategy Based on Grammatical Evolution: A Case Study about Knowledge Discovery from Text](https://www.aclweb.org/anthology/P19-1428/) applied probabilistic grammatical evolution with a custom-made grammar in the context of entity recognition in medical text.

- [General-purpose Hierarchical Optimisation of Machine Learning Pipelines with Grammatical Evolution](https://doi.org/10.1016/j.ins.2020.07.035) presents a more uniform framework with different grammars in different problems, from tabular datasets to natural language processing.

- [Solving Heterogeneous AutoML Problems with AutoGOAL](https://www.automl.org/wp-content/uploads/2020/07/AutoML_2020_paper_20.pdf) is the first actual description of AutoGOAL as a framework, unifying the ideas presented in the previous papers.

## 🤝 Contribution

Code is licensed under MIT. Read the details in the [collaboration section](https://autogoal.github.io/contributing).

This project follows the [all-contributors](https://allcontributors.org) specification. Any contribution will be given credit, from fixing typos, to reporting bugs, to implementing new core functionalities.

Here are all the current contributions.

> **🙏 Thanks!**

Suilan Estevez-Velarde
💻 ⚠ 🤔 📖

Alejandro Piad
💻 ⚠ 📖

Yudivián Almeida Cruz
🤔 📖

ygutierrez
🤔 📖

Ernesto Luis Estevanell Valladares
💻 ⚠

Alexander Gonzalez
💻 ⚠

Anshu Trivedi
💻

Alex Coto
📖

Guillermo Blanco
🐛 💻 📖

yacth
🐛 💻

Brandon Fergerson
🐛

Aditya Nikhil
🐛

lucas-FP
🐛 💻

Leynier Gutiérrez González
📖

Ender Minyard
📖