Simple Finite Element Analysis in Python

![Build Status](https://github.com/JWock82/PyNite/actions/workflows/build-and-test.yml/badge.svg)
![PyPI - Downloads](https://img.shields.io/pypi/dm/PyNiteFEA)

![GitHub last commit](https://img.shields.io/github/last-commit/JWock82/PyNite)
![GitHub](https://img.shields.io/github/license/JWock82/PyNite)
[![Documentation Status](https://readthedocs.org/projects/pynite/badge/?version=latest)](https://pynite.readthedocs.io/en/latest/?badge=latest)

An easy to use elastic 3D structural engineering finite element analysis library for Python.

# Installation
The easiest way to install Pynite is with pip: `pip install PyniteFEA`

For a more detailed discussion on installation options and dependencies see https://pynite.readthedocs.io/en/latest/installation.html

# Current Capabilities
* 3D static analysis of elastic structures.
* P-Δ analysis of frame type structures.
* Member point loads, linearly varying distributed loads, and nodal loads are supported.
* Classify loads by load case and create load combinations from load cases.
* Produces shear, moment, and deflection results and diagrams for each member.
* Automatic handling of internal nodes on frame members (physical members).
* Tension-only and compression-only elements.
* Spring elements: two-way, tension-only, and compression-only.
* Spring supports: two-way and one-way.
* Quadrilateral plate elements (MITC4 formulation).
* Rectangular plate elements (12-term polynomial formulation).
* Basic meshing algorithms for some common shapes and for openings in rectangular walls.
* Reports support reactions.
* Rendering of model geometry, supports, load cases, load combinations, and deformed shapes.
* Generates PDF reports for models and model results.

# Project Objectives
As I've gotten into the structural engineering profession, I've found there's a need for an easy to use open-source finite element package. I hope to help fill that need by prioritizing the following:

1. Accuracy: There are no guarantees PyNite is error free, but accuracy and correctness are a priority. When bugs or errors are identified, top priority will be given to eliminate them. PyNite's code is frequently reviewed, and its output is tested against a suite of textbook problems with known solutions using continuous integration (CI) anytime a change to the code base is made. If you do happen to find an error, please report it as an issue.

2. Simplicity: There are other finite element alternatives out there with many more capabilities, but they are often lacking in documentation, written in difficult languages, or require extensive knowledge of finite element theory and/or element formulations to use. PyNite is not intended to be the most technically advanced solver out there. Rather, the goal is to provide a robust yet simple general purpose package.

4. Improvement: PyNite is getting better at what it does. Each new feature provides leverage to build upon previous features in more elaborate ways. Key to improvement is (a) maintaining the core features that other features rely on, and (b) adding new features that are solid stepping stones to other features. Improvement most often happens by getting the small and simple things right incrementally, rather than with sweeping overhauls all at once.

5. Collaboration: If you see a way to improve Pynite, you are encouraged to contribute. There are many simple ways to contribute that don't take much effort. Issue reports and pull requests can be very helpful. One easy way to contribute is to add to or improve the library of simple example problems in the `Examples` folder. Please keep them relatively simple. Most users learn Pynite from following these simple examples. Another way to help Pynite without having to know too much about its internal workings is to help with the documentation files in the `docs\source` folder of this repository. These files help new users learn Pynite. If you are able to make a bigger commitment, and would like to become a regular contributor to the project, please reach out about becoming a collaborator.

# Support
Whether you just need help getting started with PyNite, or are looking to build something more complex, there are a few resources available:
* The examples in the "Examples" folder in this repository cover a variety of simple problems. The comments in the examples provide additional guidance on how PyNite works.
* Documentation is a work in progress and can be found on readthedocs here: https://pynite.readthedocs.io/en/latest/index.html.
* If you're looking for more direct guidance on using PyNite, or for help coding a project, I am available on a private consulting basis. You can reach out to me directly at Building.Code@outlook.com to discuss options.

# Example Projects
Here's a list of projects that use PyNite:

* Building Code (https://building-code.herokuapp.com/) - This one is my personal side project.
* Standard Solver (https://www.standardsolver.com/)
* Civils.ai (https://civils.ai/1/free-3D-finite-element-structural-analysis)
* Phaenotyp (https://github.com/bewegende-Architektur/Phaenotyp) (https://youtu.be/shloSw9HjVI)

# What's New?
v0.0.95
* Bug fix for rendering negative point loads via `Pyvista`. They were being rendered as positive loads. The analysis was not impacted by this bug.

v0.0.94
* Added rendering via `Pyvista`. This greatly simplified the rendering code and provided a fresh look to the rendereings. Renderings in jupyter are now interactive. Global axes are also now shown in rendereings. To use `Pyvista` instead of `VTK`, use the new `Rendering` library rather than the old `Visualization` library. Rendering via `VTK` directly is still available.
* Bug fix for member self-weight. The program was throwing exceptions instead of calculating member self-weight. Added a unit test to help prevent this issue from occuring again as code changes.
* Refactored `material` to be `material_name` in the code. The prior naming convention caused confusion which led to the self-weight bug.

v0.0.93
* Fixed phantom reactions showing up at unsupported nodes. If there was a support defined at a node, the program was summing reactions for all directions at the node, rather than just the supported directions. This caused the program to report "extra" reaction directions at any supported node (if the user queried them). Element forces/stresses were not affected as this was a post-processing reaction summing issue. Reactions for supported directions were summed correctly, except in the case of nodes with both spring supports and other supports. Only unsupported directions, and nodes with both spring supports and other supports, were showing phantom reactions. This bug also caused statics checks to fail from time to time.
* Reorganized physical member code to match member code more consistently.

v0.0.92
* Added member self-weight calculations via `FEModel3D.add_member_self_weight()`. This only applies to members. This feature does not calculate self-weight for plate and quad elements.

v0.0.89-0.0.91
* Migrating visualizaton code from VTK to PyVista. PyVista greatly simplifies the rendering code, and simplifies adding new features to the renderings. This feature is only partially complete and partially functional.

v0.0.88
* Reorganized physical member code to match member code more consistently.

v0.0.87
* Fine-tuned P-$\delta$ effects. P-$\delta$ effects are now included in member internal slope and deflection calculations.


Workload Automation
+++++++++++++++++++

Workload Automation (WA) is a framework for executing workloads and collecting
measurements on Android and Linux devices. WA includes automation for nearly 40
workloads and supports some common instrumentation (ftrace, hwmon) along with a
number of output formats.

WA is designed primarily as a developer tool/framework to facilitate data driven
development by providing a method of collecting measurements from a device in a
repeatable way.

WA is highly extensible. Most of the concrete functionality is implemented via
plug-ins, and it is easy to write new plug-ins to support new device types,
workloads, instruments or output processing.

Requirements
============

- Python 3.5+
- Linux (should work on other Unixes, but untested)
- Latest Android SDK (ANDROID_HOME must be set) for Android devices, or
- SSH for Linux devices

Installation
============

To install::

        git clone git@github.com:ARM-software/workload-automation.git workload-automation
        sudo -H python setup [install|develop]

Note: A `requirements.txt` is included however this is designed to be used as a
reference for known working versions rather than as part of a standard
installation.

Please refer to the `installation section <http://workload-automation.readthedocs.io/en/latest/user_information.html#install>`_
in the documentation for more details.

Basic Usage
===========

Please see the `Quickstart <http://workload-automation.readthedocs.io/en/latest/user_information.html#user-guide>`_
section of the documentation.

Documentation
=============

You can view pre-built HTML documentation `here <http://workload-automation.readthedocs.io/en/latest/>`_.

Documentation in reStructuredText format may be found under ``doc/source``. To
compile it into cross-linked HTML, make sure you have `Sphinx
<http://sphinx-doc.org/install.html>`_ installed, and then ::

        cd doc
        make html

License
=======

Workload Automation is distributed under `Apache v2.0 License
<http://www.apache.org/licenses/LICENSE-2.0>`_. Workload automation includes
binaries distributed under different licenses (see LICENSE files in specific
directories).

Feedback, Contributions and Support
===================================

- Please use the GitHub Issue Tracker associated with this repository for
  feedback.
- ARM licensees may contact ARM directly via their partner managers.
- We welcome code contributions via GitHub Pull requests. Please see
  "Contributing Code" section of the documentation for details.


Qiita (canonically pronounced *cheetah*)
========================================

|Build Status| |Coverage Status|

Advances in sequencing, proteomics, transcriptomics, metabolomics, and others are giving
us new insights into the microbial world and dramatically improving our ability
to understand their community composition and function at high resolution.
These new technologies are generating vast amounts of data, even from a single
study or sample, leading to challenges in storage, representation, analysis,
and integration of the disparate data types. Qiita was designed to allow users
address these new challenges by keeping track of multiple studies with multiple
'omics data. Additionally, Qiita is capable of supporting multiple analytical
pipelines through a 3rd-party plugin system, allowing the user to have a single
entry point for all their analyses. Qiita's main site provides database and
compute resources to the global community, alleviating the technical burdens,
such as familiarity with the command line or access to compute power, that are
typically limiting for researchers studying microbial ecology.

Qiita is currently in production/stable status. We are very open to community
contributions and feedback. If you're interested in contributing to Qiita,
see `CONTRIBUTING.md <https://github.com/qiita-spots/qiita/blob/master/CONTRIBUTING.md>`__.
If you'd like to report bugs or request features, you can do that in the
`Qiita issue tracker <https://github.com/qiita-spots/qiita/issues>`__.

To install and configure your own Qiita server, see
`INSTALL.md <https://github.com/qiita-spots/qiita/blob/master/INSTALL.md>`__. However, Qiita is not designed to be used locally but rather on a server, we therefore advise against installing your own version on a personal computer. Nevertheless, it can run just fine on a laptop or small computer for development and educational purposes. For example, for every single PR and release, we install Qiita from scratch as GitHub Actions, you can follow `these steps <https://github.com/qiita-spots/qiita/actions>`__.

For more specific details about Qiita's philosophy and design visit `the Qiita main site tutorial <https://qiita.microbio.me/static/doc/html/qiita-philosophy/index.html>`__.

Current features
----------------

* Full study management: Create, delete, update samples in the sample and
  multiple preparation information files.
* Upload files via direct drag & drop from the web interface or via scp
  from any server that allows these connections.
* Study privacy management: Sandboxed -> Private -> Public.
* Easy long-term sequence data deposition to the European Nucleotide Archive (ENA),
  part of the European Bioinformatics Institute (EBI) for private and public
  studies.
* Raw data processing for `Target Gene, Metagenomic, Metabolomic, Genome Isolates and BIOM files <https://qiita.ucsd.edu/static/doc/html/processingdata/index.html#processing-recommendations>`__. NOTE: BIOM files can be added as new preparation files for downstream analyses; however, this cannot be made public in the system.
* Basic downstream analyses using QIIME 2. Note that Qiita produces qza/qzv in the analytical steps but you can also convert `non QIIME 2 artifacts <https://qiita.ucsd.edu/static/doc/html/faq.html#how-to-convert-qiita-files-to-qiime2-artifacts>`__.
* Bulk download of `studies and artifacts <https://qiita.ucsd.edu/static/doc/html/downloading.html>`__.
* Basic study search in the study listing page.
* Complex metadata search via redbiom.

For more detailed information visit the `Qiita tutorial <https://cmi-workshop.readthedocs.io/en/latest/>`__
and the `Qiita help <https://qiita.ucsd.edu/static/doc/html/index.html>`__.

Accepted raw files
------------------

* Multiplexed SFF
* Multiplexed FASTQ: forward, reverse (optional), and barcodes
* Per sample FASTQ: forward and reverse (optional)
* Multiplexed FASTA/qual files
* Per sample FASTA, only for "Full Length Operon"

.. |Build Status| image:: https://github.com/qiita-spots/qiita/actions/workflows/qiita-ci.yml/badge.svg
   :target: https://github.com/qiita-spots/qiita/actions/workflows/qiita-ci.yml
.. |Coverage Status| image:: https://coveralls.io/repos/github/qiita-spots/qiita/badge.svg?branch=dev
   :target: https://coveralls.io/github/qiita-spots/qiita?branch=master


# ormar

### Overview

The `ormar` package is an async mini ORM for Python, with support for **Postgres,
MySQL**, and **SQLite**.

The main benefits of using `ormar` are:

*  getting an **async ORM that can be used with async frameworks** (fastapi, starlette etc.)
*  getting just **one model to maintain** - you don't have to maintain pydantic and other orm models (sqlalchemy, peewee, gino etc.)

The goal was to create a simple ORM that can be **used directly (as request and response models) with [`fastapi`][fastapi]** that bases it's data validation on pydantic.

Ormar - apart from the obvious "ORM" in name - gets its name from _ormar_ in Swedish which means _snakes_, and _ormar_ in Croatian which means _cabinet_.

And what's a better name for python ORM than snakes cabinet :)

**If you like ormar remember to star the repository in [github](https://github.com/collerek/ormar)!**

The bigger community we build, the easier it will be to catch bugs and attract contributors ;)

### Documentation

Check out the [documentation][documentation] for details.

**Note that for brevity most of the documentation snippets omit the creation of the database
and scheduling the execution of functions for asynchronous run.**

If you want more real life examples than in the documentation you can see the [tests][tests] folder,
since they actually have to create and connect to a database in most of the tests.

Yet remember that those are - well - tests and not all solutions are suitable to be used in real life applications.

### Part of the `fastapi` ecosystem

As part of the fastapi ecosystem `ormar` is supported in libraries that somehow work with databases.

As of now `ormar` is supported by:

*  [`fastapi-crudrouter`](https://github.com/awtkns/fastapi-crudrouter)
*  [`fastapi-pagination`](https://github.com/uriyyo/fastapi-pagination)

If you maintain or use a different library and would like it to support `ormar` let us know how we can help.

### Dependencies

Ormar is built with:

* [`sqlalchemy core`][sqlalchemy-core] for query building.
* [`databases`][databases] for cross-database async support.
* [`pydantic`][pydantic] for data validation.
* `typing_extensions` for python 3.6 - 3.7

### License

`ormar` is built as open-sorce software and will remain completely free (MIT license).

As I write open-source code to solve everyday problems in my work or to promote and build strong python
community you can say thank you and buy me a coffee or sponsor me with a monthly amount to help ensure my work remains free and maintained.

Sponsor - Github Sponsors

### Migrating from `sqlalchemy` and existing databases

If you currently use `sqlalchemy` and would like to switch to `ormar` check out the auto-translation
tool that can help you with translating existing sqlalchemy orm models so you do not have to do it manually.

**Beta** versions available at github: [`sqlalchemy-to-ormar`](https://github.com/collerek/sqlalchemy-to-ormar)
or simply `pip install sqlalchemy-to-ormar`

`sqlalchemy-to-ormar` can be used in pair with `sqlacodegen` to auto-map/ generate `ormar` models from existing database, even if you don't use `sqlalchemy` for your project.

### Migrations & Database creation

Because ormar is built on SQLAlchemy core, you can use [`alembic`][alembic] to provide
database migrations (and you really should for production code).

For tests and basic applications the `sqlalchemy` is more than enough:
```python
# note this is just a partial snippet full working example below
# 1. Imports
import sqlalchemy
import databases

# 2. Initialization
DATABASE_URL = "sqlite:///db.sqlite"
database = databases.Database(DATABASE_URL)
metadata = sqlalchemy.MetaData()

# Define models here

# 3. Database creation and tables creation
engine = sqlalchemy.create_engine(DATABASE_URL)
metadata.create_all(engine)
```

For a sample configuration of alembic and more information regarding migrations and
database creation visit [migrations][migrations] documentation section.

### Package versions
**ormar is still under development:**
We recommend pinning any dependencies (with i.e. `ormar~=0.9.1`)

`ormar` also follows the release numeration that breaking changes bump the major number,
while other changes and fixes bump minor number, so with the latter you should be safe to
update, yet always read the [releases][releases] docs before.
`example: (0.5.2 -> 0.6.0 - breaking, 0.5.2 -> 0.5.3 - non breaking)`.

### Asynchronous Python

Note that `ormar` is an asynchronous ORM, which means that you have to `await` the calls to
the methods, that are scheduled for execution in an event loop. Python has a builtin module
[`asyncio`][asyncio] that allows you to do just that.

Note that most "normal" python interpreters do not allow execution of `await`
outside of a function (because you actually schedule this function for delayed execution
and don't get the result immediately).

In a modern web framework (like `fastapi`), the framework will handle this for you, but if
you plan to do this on your own you need to perform this manually like described in the
quick start below.

### Quick Start

Note that you can find the same script in examples folder on github.

```python
from typing import Optional

import databases
import pydantic

import ormar
import sqlalchemy

DATABASE_URL = "sqlite:///db.sqlite"
base_ormar_config = ormar.OrmarConfig(
    database=databases.Database(DATABASE_URL),
    metadata=sqlalchemy.MetaData(),
    engine=sqlalchemy.create_engine(DATABASE_URL),
)

# Note that all type hints are optional
# below is a perfectly valid model declaration
# class Author(ormar.Model):
#     ormar_config = base_ormar_config.copy(tablename="authors")
#
#     id = ormar.Integer(primary_key=True) # <= notice no field types
#     name = ormar.String(max_length=100)

class Author(ormar.Model):
    ormar_config = base_ormar_config.copy(tablename="authors")

    id: int = ormar.Integer(primary_key=True)
    name: str = ormar.String(max_length=100)

class Book(ormar.Model):
    ormar_config = base_ormar_config.copy(tablename="books")

    id: int = ormar.Integer(primary_key=True)
    author: Optional[Author] = ormar.ForeignKey(Author)
    title: str = ormar.String(max_length=100)
    year: int = ormar.Integer(nullable=True)

# create the database
# note that in production you should use migrations
# note that this is not required if you connect to existing database
# just to be sure we clear the db before
base_ormar_config.metadata.drop_all(base_ormar_config.engine)
base_ormar_config.metadata.create_all(base_ormar_config.engine)

# all functions below are divided into functionality categories
# note how all functions are defined with async - hence can use await AND needs to
# be awaited on their own
async def create():
    # Create some records to work with through QuerySet.create method.
    # Note that queryset is exposed on each Model's class as objects
    tolkien = await Author.objects.create(name="J.R.R. Tolkien")
    await Book.objects.create(author=tolkien, title="The Hobbit", year=1937)
    await Book.objects.create(author=tolkien, title="The Lord of the Rings", year=1955)
    await Book.objects.create(author=tolkien, title="The Silmarillion", year=1977)

    # alternative creation of object divided into 2 steps
    sapkowski = Author(name="Andrzej Sapkowski")
    # do some stuff
    await sapkowski.save()

    # or save() after initialization
    await Book(author=sapkowski, title="The Witcher", year=1990).save()
    await Book(author=sapkowski, title="The Tower of Fools", year=2002).save()

    # to read more about inserting data into the database
    # visit: https://collerek.github.io/ormar/queries/create/

async def read():
    # Fetch an instance, without loading a foreign key relationship on it.
    # Django style
    book = await Book.objects.get(title="The Hobbit")
    # or python style
    book = await Book.objects.get(Book.title == "The Hobbit")
    book2 = await Book.objects.first()

    # first() fetch the instance with lower primary key value
    assert book == book2

    # you can access all fields on loaded model
    assert book.title == "The Hobbit"
    assert book.year == 1937

    # when no condition is passed to get()
    # it behaves as last() based on primary key column
    book3 = await Book.objects.get()
    assert book3.title == "The Tower of Fools"

    # When you have a relation, ormar always defines a related model for you
    # even when all you loaded is a foreign key value like in this example
    assert isinstance(book.author, Author)
    # primary key is populated from foreign key stored in books table
    assert book.author.pk == 1
    # since the related model was not loaded all other fields are None
    assert book.author.name is None

    # Load the relationship from the database when you already have the related model
    # alternatively see joins section below
    await book.author.load()
    assert book.author.name == "J.R.R. Tolkien"

    # get all rows for given model
    authors = await Author.objects.all()
    assert len(authors) == 2

    # to read more about reading data from the database
    # visit: https://collerek.github.io/ormar/queries/read/

async def update():
    # read existing row from db
    tolkien = await Author.objects.get(name="J.R.R. Tolkien")
    assert tolkien.name == "J.R.R. Tolkien"
    tolkien_id = tolkien.id

    # change the selected property
    tolkien.name = "John Ronald Reuel Tolkien"
    # call update on a model instance
    await tolkien.update()

    # confirm that object was updated
    tolkien = await Author.objects.get(name="John Ronald Reuel Tolkien")
    assert tolkien.name == "John Ronald Reuel Tolkien"
    assert tolkien.id == tolkien_id

    # alternatively update data without loading
    await Author.objects.filter(name__contains="Tolkien").update(name="J.R.R. Tolkien")

    # to read more about updating data in the database
    # visit: https://collerek.github.io/ormar/queries/update/

async def delete():
    silmarillion = await Book.objects.get(year=1977)
    # call delete() on instance
    await silmarillion.delete()

    # alternatively delete without loading
    await Book.objects.delete(title="The Tower of Fools")

    # note that when there is no record ormar raises NoMatch exception
    try:
        await Book.objects.get(year=1977)
    except ormar.NoMatch:
        print("No book from 1977!")

    # to read more about deleting data from the database
    # visit: https://collerek.github.io/ormar/queries/delete/

    # note that despite the fact that record no longer exists in database
    # the object above is still accessible and you can use it (and i.e. save()) again.
    tolkien = silmarillion.author
    await Book.objects.create(author=tolkien, title="The Silmarillion", year=1977)

async def joins():
    # Tho join two models use select_related

    # Django style
    book = await Book.objects.select_related("author").get(title="The Hobbit")
    # Python style
    book = await Book.objects.select_related(Book.author).get(
        Book.title == "The Hobbit"
    )

    # now the author is already prefetched
    assert book.author.name == "J.R.R. Tolkien"

    # By default you also get a second side of the relation
    # constructed as lowercase source model name +'s' (books in this case)
    # you can also provide custom name with parameter related_name

    # Django style
    author = await Author.objects.select_related("books").all(name="J.R.R. Tolkien")
    # Python style
    author = await Author.objects.select_related(Author.books).all(
        Author.name == "J.R.R. Tolkien"
    )
    assert len(author[0].books) == 3

    # for reverse and many to many relations you can also prefetch_related
    # that executes a separate query for each of related models

    # Django style
    author = await Author.objects.prefetch_related("books").get(name="J.R.R. Tolkien")
    # Python style
    author = await Author.objects.prefetch_related(Author.books).get(
        Author.name == "J.R.R. Tolkien"
    )
    assert len(author.books) == 3

    # to read more about relations
    # visit: https://collerek.github.io/ormar/relations/

    # to read more about joins and subqueries
    # visit: https://collerek.github.io/ormar/queries/joins-and-subqueries/

async def filter_and_sort():
    # to filter the query you can use filter() or pass key-value pars to
    # get(), all() etc.
    # to use special methods or access related model fields use double
    # underscore like to filter by the name of the author use author__name
    # Django style
    books = await Book.objects.all(author__name="J.R.R. Tolkien")
    # python style
    books = await Book.objects.all(Book.author.name == "J.R.R. Tolkien")
    assert len(books) == 3

    # filter can accept special methods also separated with double underscore
    # to issue sql query ` where authors.name like "%tolkien%"` that is not
    # case sensitive (hence small t in Tolkien)
    # Django style
    books = await Book.objects.filter(author__name__icontains="tolkien").all()
    # python style
    books = await Book.objects.filter(Book.author.name.icontains("tolkien")).all()
    assert len(books) == 3

    # to sort use order_by() function of queryset
    # to sort decreasing use hyphen before the field name
    # same as with filter you can use double underscores to access related fields
    # Django style
    books = (
        await Book.objects.filter(author__name__icontains="tolkien")
        .order_by("-year")
        .all()
    )
    # python style
    books = (
        await Book.objects.filter(Book.author.name.icontains("tolkien"))
        .order_by(Book.year.desc())
        .all()
    )
    assert len(books) == 3
    assert books[0].title == "The Silmarillion"
    assert books[2].title == "The Hobbit"

    # to read more about filtering and ordering
    # visit: https://collerek.github.io/ormar/queries/filter-and-sort/

async def subset_of_columns():
    # to exclude some columns from loading when querying the database
    # you can use fields() method
    hobbit = await Book.objects.fields(["title"]).get(title="The Hobbit")
    # note that fields not included in fields are empty (set to None)
    assert hobbit.year is None
    assert hobbit.author is None

    # selected field is there
    assert hobbit.title == "The Hobbit"

    # alternatively you can provide columns you want to exclude
    hobbit = await Book.objects.exclude_fields(["year"]).get(title="The Hobbit")
    # year is still not set
    assert hobbit.year is None
    # but author is back
    assert hobbit.author is not None

    # also you cannot exclude primary key column - it's always there
    # even if you EXPLICITLY exclude it it will be there

    # note that each model have a shortcut for primary_key column which is pk
    # and you can filter/access/set the values by this alias like below
    assert hobbit.pk is not None

    # note that you cannot exclude fields that are not nullable
    # (required) in model definition
    try:
        await Book.objects.exclude_fields(["title"]).get(title="The Hobbit")
    except pydantic.ValidationError:
        print("Cannot exclude non nullable field title")

    # to read more about selecting subset of columns
    # visit: https://collerek.github.io/ormar/queries/select-columns/

async def pagination():
    # to limit number of returned rows use limit()
    books = await Book.objects.limit(1).all()
    assert len(books) == 1
    assert books[0].title == "The Hobbit"

    # to offset number of returned rows use offset()
    books = await Book.objects.limit(1).offset(1).all()
    assert len(books) == 1
    assert books[0].title == "The Lord of the Rings"

    # alternatively use paginate that combines both
    books = await Book.objects.paginate(page=2, page_size=2).all()
    assert len(books) == 2
    # note that we removed one book of Sapkowski in delete()
    # and recreated The Silmarillion - by default when no order_by is set
    # ordering sorts by primary_key column
    assert books[0].title == "The Witcher"
    assert books[1].title == "The Silmarillion"

    # to read more about pagination and number of rows
    # visit: https://collerek.github.io/ormar/queries/pagination-and-rows-number/

async def aggregations():
    # count:
    assert 2 == await Author.objects.count()

    # exists
    assert await Book.objects.filter(title="The Hobbit").exists()

    # maximum
    assert 1990 == await Book.objects.max(columns=["year"])

    # minimum
    assert 1937 == await Book.objects.min(columns=["year"])

    # average
    assert 1964.75 == await Book.objects.avg(columns=["year"])

    # sum
    assert 7859 == await Book.objects.sum(columns=["year"])

    # to read more about aggregated functions
    # visit: https://collerek.github.io/ormar/queries/aggregations/

async def raw_data():
    # extract raw data in a form of dicts or tuples
    # note that this skips the validation(!) as models are
    # not created from parsed data

    # get list of objects as dicts
    assert await Book.objects.values() == [
        {"id": 1, "author": 1, "title": "The Hobbit", "year": 1937},
        {"id": 2, "author": 1, "title": "The Lord of the Rings", "year": 1955},
        {"id": 4, "author": 2, "title": "The Witcher", "year": 1990},
        {"id": 5, "author": 1, "title": "The Silmarillion", "year": 1977},
    ]

    # get list of objects as tuples
    assert await Book.objects.values_list() == [
        (1, 1, "The Hobbit", 1937),
        (2, 1, "The Lord of the Rings", 1955),
        (4, 2, "The Witcher", 1990),
        (5, 1, "The Silmarillion", 1977),
    ]

    # filter data - note how you always get a list
    assert await Book.objects.filter(title="The Hobbit").values() == [
        {"id": 1, "author": 1, "title": "The Hobbit", "year": 1937}
    ]

    # select only wanted fields
    assert await Book.objects.filter(title="The Hobbit").values(["id", "title"]) == [
        {"id": 1, "title": "The Hobbit"}
    ]

    # if you select only one column you could flatten it with values_list
    assert await Book.objects.values_list("title", flatten=True) == [
        "The Hobbit",
        "The Lord of the Rings",
        "The Witcher",
        "The Silmarillion",
    ]

    # to read more about extracting raw values
    # visit: https://collerek.github.io/ormar/queries/aggregations/

async def with_connect(function):
    # note that for any other backend than sqlite you actually need to
    # connect to the database to perform db operations
    async with base_ormar_config.database:
        await function()

    # note that if you use framework like `fastapi` you shouldn't connect
    # in your endpoints but have a global connection pool
    # check https://collerek.github.io/ormar/fastapi/ and section with db connection

# gather and execute all functions
# note - normally import should be at the beginning of the file
import asyncio

# note that normally you use gather() function to run several functions
# concurrently but we actually modify the data and we rely on the order of functions
for func in [
    create,
    read,
    update,
    delete,
    joins,
    filter_and_sort,
    subset_of_columns,
    pagination,
    aggregations,
    raw_data,
]:
    print(f"Executing: {func.__name__}")
    asyncio.run(with_connect(func))

# drop the database tables
base_ormar_config.metadata.drop_all(base_ormar_config.engine)
```

## Ormar Specification

### QuerySet methods

*  `create(**kwargs): -> Model`
*  `get(*args, **kwargs): -> Model`
*  `get_or_none(*args, **kwargs): -> Optional[Model]`
*  `get_or_create(_defaults: Optional[Dict[str, Any]] = None, *args, **kwargs) -> Tuple[Model, bool]`
*  `first(*args, **kwargs): -> Model`
*  `update(each: bool = False, **kwargs) -> int`
*  `update_or_create(**kwargs) -> Model`
*  `bulk_create(objects: List[Model]) -> None`
*  `bulk_update(objects: List[Model], columns: List[str] = None) -> None`
*  `delete(*args, each: bool = False, **kwargs) -> int`
*  `all(*args, **kwargs) -> List[Optional[Model]]`
*  `iterate(*args, **kwargs) -> AsyncGenerator[Model]`
*  `filter(*args, **kwargs) -> QuerySet`
*  `exclude(*args, **kwargs) -> QuerySet`
*  `select_related(related: Union[List, str]) -> QuerySet`
*  `prefetch_related(related: Union[List, str]) -> QuerySet`
*  `limit(limit_count: int) -> QuerySet`
*  `offset(offset: int) -> QuerySet`
*  `count(distinct: bool = True) -> int`
*  `exists() -> bool`
*  `max(columns: List[str]) -> Any`
*  `min(columns: List[str]) -> Any`
*  `avg(columns: List[str]) -> Any`
*  `sum(columns: List[str]) -> Any`
*  `fields(columns: Union[List, str, set, dict]) -> QuerySet`
*  `exclude_fields(columns: Union[List, str, set, dict]) -> QuerySet`
*  `order_by(columns:Union[List, str]) -> QuerySet`
*  `values(fields: Union[List, str, Set, Dict])`
*  `values_list(fields: Union[List, str, Set, Dict])`

#### Relation types

*  One to many  - with `ForeignKey(to: Model)`
*  Many to many - with `ManyToMany(to: Model, Optional[through]: Model)`

#### Model fields types

Available Model Fields (with required args - optional ones in docs):

* `String(max_length)`
* `Text()`
* `Boolean()`
* `Integer()`
* `Float()`
* `Date()`
* `Time()`
* `DateTime()`
* `JSON()`
* `BigInteger()`
* `SmallInteger()`
* `Decimal(scale, precision)`
* `UUID()`
* `LargeBinary(max_length)`
* `Enum(enum_class)`
* `Enum` like Field - by passing `choices` to any other Field type
* `EncryptedString` - by passing `encrypt_secret` and `encrypt_backend`
* `ForeignKey(to)`
* `ManyToMany(to)`

### Available fields options
The following keyword arguments are supported on all field types.

* `primary_key: bool`
* `nullable: bool`
* `default: Any`
* `server_default: Any`
* `index: bool`
* `unique: bool`
* `choices: typing.Sequence`
* `name: str`

All fields are required unless one of the following is set:

* `nullable` - Creates a nullable column. Sets the default to `False`. Read the fields common parameters for details.
* `sql_nullable` - Used to set different setting for pydantic and the database. Sets the default to `nullable` value. Read the fields common parameters for details.
* `default` - Set a default value for the field. **Not available for relation fields**
* `server_default` - Set a default value for the field on server side (like sqlalchemy's `func.now()`). **Not available for relation fields**
* `primary key` with `autoincrement` - When a column is set to primary key and autoincrement is set on this column.
  Autoincrement is set by default on int primary keys.

### Available signals

Signals allow to trigger your function for a given event on a given Model.

* `pre_save`
* `post_save`
* `pre_update`
* `post_update`
* `pre_delete`
* `post_delete`
* `pre_relation_add`
* `post_relation_add`
* `pre_relation_remove`
* `post_relation_remove`
* `post_bulk_update`

[sqlalchemy-core]: https://docs.sqlalchemy.org/en/latest/core/
[databases]: https://github.com/encode/databases
[pydantic]: https://pydantic-docs.helpmanual.io/
[encode/orm]: https://github.com/encode/orm/
[alembic]: https://alembic.sqlalchemy.org/en/latest/
[fastapi]: https://fastapi.tiangolo.com/
[documentation]: https://collerek.github.io/ormar/
[migrations]: https://collerek.github.io/ormar/models/migrations/
[asyncio]: https://docs.python.org/3/library/asyncio.html
[releases]: https://collerek.github.io/ormar/releases/
[tests]: https://github.com/collerek/ormar/tree/master/tests


===============================================================
xclim: Climate services library |logo| |logo-dark| |logo-light|
===============================================================

+----------------------------+-----------------------------------------------------+
| Versions                   | |pypi| |conda| |versions|                           |
+----------------------------+-----------------------------------------------------+
| Documentation and Support  | |docs| |discussions|                                |
+----------------------------+-----------------------------------------------------+
| Open Source                | |license| |fair| |ossf| |zenodo| |pyOpenSci| |joss| |
+----------------------------+-----------------------------------------------------+
| Coding Standards           | |black| |ruff| |pre-commit| |security| |fossa|      |
+----------------------------+-----------------------------------------------------+
| Development Status         | |status| |build| |coveralls|                        |
+----------------------------+-----------------------------------------------------+

`xclim` is an operational Python library for climate services, providing numerous climate-related indicator tools
with an extensible framework for constructing custom climate indicators, statistical downscaling and bias
adjustment of climate model simulations, as well as climate model ensemble analysis tools.

`xclim` is built using `xarray`_ and can seamlessly benefit from the parallelization handling provided by `dask`_.
Its objective is to make it as simple as possible for users to perform typical climate services data treatment workflows.
Leveraging xarray and dask, users can easily bias-adjust climate simulations over large spatial domains or compute indices from large climate datasets.

For example, the following would compute monthly mean temperature from daily mean temperature:

.. code-block:: python

    import xclim
    import xarray as xr

    ds = xr.open_dataset(filename)
    tg = xclim.atmos.tg_mean(ds.tas, freq="MS")

For applications where metadata and missing values are important to get right, xclim provides a class for each index
that validates inputs, checks for missing values, converts units and assigns metadata attributes to the output.
This also provides a mechanism for users to customize the indices to their own specifications and preferences.
`xclim` currently provides over 150 indices related to mean, minimum and maximum daily temperature, daily precipitation,
streamflow and sea ice concentration, numerous bias-adjustment algorithms, as well as a dedicated module for ensemble analysis.

.. _xarray: https://docs.xarray.dev/
.. _dask: https://docs.dask.org/

Quick Install
-------------
`xclim` can be installed from PyPI:

.. code-block:: shell

    $ pip install xclim

or from Anaconda (conda-forge):

.. code-block:: shell

    $ conda install -c conda-forge xclim

Documentation
-------------
The official documentation is at https://xclim.readthedocs.io/

How to make the most of xclim: `Basic Usage Examples`_ and `In-Depth Examples`_.

.. _Basic Usage Examples: https://xclim.readthedocs.io/en/stable/notebooks/usage.html
.. _In-Depth Examples: https://xclim.readthedocs.io/en/stable/notebooks/index.html

Conventions
-----------
In order to provide a coherent interface, `xclim` tries to follow different sets of conventions. In particular, input data should follow the `CF conventions`_ whenever possible for variable attributes. Variable names are usually the ones used in `CMIP6`_, when they exist.

However, xclim will *always* assume the temporal coordinate is named "time". If your data uses another name (for example: "T"), you can rename the variable with:

.. code-block:: python

    ds = ds.rename(T="time")

.. _CF Conventions: http://cfconventions.org/
.. _CMIP6: https://clipc-services.ceda.ac.uk/dreq/mipVars.html

Contributing to xclim
---------------------
`xclim` is in active development and is being used in production by climate services specialists around the world.

* If you're interested in participating in the development of `xclim` by suggesting new features, new indices or report bugs, please leave us a message on the `issue tracker`_.
    * If you have a support/usage question or would like to translate `xclim` to a new language, be sure to check out the existing |discussions| first!

* If you would like to contribute code or documentation (which is greatly appreciated!), check out the `Contributing Guidelines`_ before you begin!

.. _issue tracker: https://github.com/Ouranosinc/xclim/issues
.. _Contributing Guidelines: https://github.com/Ouranosinc/xclim/blob/main/CONTRIBUTING.rst

How to cite this library
------------------------
If you wish to cite `xclim` in a research publication, we kindly ask that you refer to our article published in The Journal of Open Source Software (`JOSS`_): https://doi.org/10.21105/joss.05415

To cite a specific version of `xclim`, the bibliographical reference information can be found through `Zenodo`_

.. _JOSS: https://joss.theoj.org/
.. _Zenodo: https://doi.org/10.5281/zenodo.2795043

License
-------
This is free software: you can redistribute it and/or modify it under the terms of the `Apache License 2.0`_. A copy of this license is provided in the code repository (`LICENSE`_).

.. _Apache License 2.0: https://opensource.org/license/apache-2-0/
.. _LICENSE: https://github.com/Ouranosinc/xclim/blob/main/LICENSE

Credits
-------
`xclim` development is funded through Ouranos_, Environment and Climate Change Canada (ECCC_), the `Fonds vert`_ and the Fonds d'électrification et de changements climatiques (FECC_), the Canadian Foundation for Innovation (CFI_), and the Fonds de recherche du Québec (FRQ_).

This package was created with Cookiecutter_ and the `audreyfeldroy/cookiecutter-pypackage`_ project template.

.. _audreyfeldroy/cookiecutter-pypackage: https://github.com/audreyfeldroy/cookiecutter-pypackage/
.. _CFI: https://www.innovation.ca/
.. _Cookiecutter: https://github.com/cookiecutter/cookiecutter/
.. _ECCC: https://www.canada.ca/en/environment-climate-change.html
.. _FECC: https://www.environnement.gouv.qc.ca/ministere/fonds-electrification-changements-climatiques/index.htm
.. _Fonds vert: https://www.environnement.gouv.qc.ca/ministere/fonds-vert/index.htm
.. _FRQ: https://frq.gouv.qc.ca/
.. _Ouranos: https://www.ouranos.ca/

.. |pypi| image:: https://img.shields.io/pypi/v/xclim.svg
        :target: https://pypi.python.org/pypi/xclim
        :alt: Python Package Index Build

.. |conda| image:: https://img.shields.io/conda/vn/conda-forge/xclim.svg
        :target: https://anaconda.org/conda-forge/xclim
        :alt: Conda-forge Build Version

.. |discussions| image:: https://img.shields.io/badge/GitHub-Discussions-blue
        :target: https://github.com/Ouranosinc/xclim/discussions
        :alt: Static Badge

.. |gitter| image:: https://badges.gitter.im/Ouranosinc/xclim.svg
        :target: https://gitter.im/Ouranosinc/xclim?utm_source=badge&utm_medium=badge&utm_campaign=pr-badge
        :alt: Gitter Chat

.. |build| image:: https://github.com/Ouranosinc/xclim/actions/workflows/main.yml/badge.svg
        :target: https://github.com/Ouranosinc/xclim/actions/workflows/main.yml
        :alt: Build Status

.. |coveralls| image:: https://coveralls.io/repos/github/Ouranosinc/xclim/badge.svg
        :target: https://coveralls.io/github/Ouranosinc/xclim
        :alt: Coveralls

.. |docs| image:: https://readthedocs.org/projects/xclim/badge
        :target: https://xclim.readthedocs.io/en/latest
        :alt: Documentation Status

.. |zenodo| image:: https://zenodo.org/badge/142608764.svg
        :target: https://zenodo.org/badge/latestdoi/142608764
        :alt: DOI

.. |pyOpenSci| image:: https://tinyurl.com/y22nb8up
        :target: https://github.com/pyOpenSci/software-review/issues/73
        :alt: pyOpenSci

.. |joss| image:: https://joss.theoj.org/papers/10.21105/joss.05415/status.svg
        :target: https://doi.org/10.21105/joss.05415
        :alt: JOSS

.. |license| image:: https://img.shields.io/github/license/Ouranosinc/xclim.svg
        :target: https://github.com/Ouranosinc/xclim/blob/main/LICENSE
        :alt: License

.. |security| image:: https://bestpractices.coreinfrastructure.org/projects/6041/badge
        :target: https://bestpractices.coreinfrastructure.org/projects/6041
        :alt: Open Source Security Foundation

.. |fair| image:: https://img.shields.io/badge/fair--software.eu-%E2%97%8F%20%20%E2%97%8F%20%20%E2%97%8F%20%20%E2%97%8F%20%20%E2%97%8B-yellow
        :target: https://fair-software.eu
        :alt: FAIR Software Compliance

.. |ossf| image:: https://api.securityscorecards.dev/projects/github.com/Ouranosinc/xclim/badge
        :target: https://securityscorecards.dev/viewer/?uri=github.com/Ouranosinc/xclim
        :alt: OpenSSF Scorecard

.. |fossa| image:: https://app.fossa.com/api/projects/git%2Bgithub.com%2FOuranosinc%2Fxclim.svg?type=shield
        :target: https://app.fossa.com/projects/git%2Bgithub.com%2FOuranosinc%2Fxclim?ref=badge_shield
        :alt: FOSSA

.. |black| image:: https://img.shields.io/badge/code%20style-black-000000.svg
        :target: https://github.com/psf/black
        :alt: Python Black

.. |logo| image:: https://raw.githubusercontent.com/Ouranosinc/xclim/main/docs/logos/xclim-logo-small-light.png
        :target: https://github.com/Ouranosinc/xclim
        :alt: Xclim
        :class: xclim-logo-small no-theme

.. |logo-light| image:: https://raw.githubusercontent.com/Ouranosinc/xclim/main/docs/logos/empty.png
        :target: https://github.com/Ouranosinc/xclim
        :alt:
        :class: xclim-logo-small only-light-inline

.. |logo-dark| image:: https://raw.githubusercontent.com/Ouranosinc/xclim/main/docs/logos/empty.png
        :target: https://github.com/Ouranosinc/xclim
        :alt:
        :class: xclim-logo-small only-dark-inline

.. |pre-commit| image:: https://results.pre-commit.ci/badge/github/Ouranosinc/xclim/main.svg
        :target: https://results.pre-commit.ci/latest/github/Ouranosinc/xclim/main
        :alt: pre-commit.ci status

.. |ruff| image:: https://img.shields.io/endpoint?url=https://raw.githubusercontent.com/astral-sh/ruff/main/assets/badge/v2.json
    :target: https://github.com/astral-sh/ruff
    :alt: Ruff

.. |status| image:: https://www.repostatus.org/badges/latest/active.svg
        :target: https://www.repostatus.org/#active
        :alt: Project Status: Active – The project has reached a stable, usable state and is being actively developed.

.. |versions| image:: https://img.shields.io/pypi/pyversions/xclim.svg
        :target: https://pypi.python.org/pypi/xclim
        :alt: Supported Python Versions


# OCR-D/core

> Python modules implementing [OCR-D specs](https://github.com/OCR-D/spec) and related tools

[![image](https://img.shields.io/pypi/v/ocrd.svg)](https://pypi.org/project/ocrd/)
[![Docker Image CI](https://github.com/OCR-D/core/actions/workflows/docker-image.yml/badge.svg)](https://github.com/OCR-D/core/actions/workflows/docker-image.yml)
[![Unit Test CI](https://github.com/OCR-D/core/actions/workflows/unit-test.yml/badge.svg)](https://github.com/OCR-D/core/actions/workflows/unit-test.yml)
[![image](https://codecov.io/gh/OCR-D/core/branch/master/graph/badge.svg)](https://codecov.io/gh/OCR-D/core)
[![image](https://scrutinizer-ci.com/g/OCR-D/core/badges/build.png?b=master)](https://scrutinizer-ci.com/g/OCR-D/core)
[![image](https://scrutinizer-ci.com/g/OCR-D/core/badges/quality-score.png?b=master)](https://scrutinizer-ci.com/g/OCR-D/core)

[![Gitter chat](https://badges.gitter.im/gitterHQ/gitter.png)](https://gitter.im/OCR-D/Lobby)

<!-- BEGIN-MARKDOWN-TOC -->
* [Introduction](#introduction)
* [Installation](#installation)
* [Command line tools](#command-line-tools)
	* [`ocrd` CLI](#ocrd-cli)
	* [`ocrd-dummy` CLI](#ocrd-dummy-cli)
* [Configuration](#configuration)
* [Packages](#packages)
	* [ocrd_utils](#ocrd_utils)
	* [ocrd_models](#ocrd_models)
	* [ocrd_modelfactory](#ocrd_modelfactory)
	* [ocrd_validators](#ocrd_validators)
	* [ocrd_network](#ocrd_network)
	* [ocrd](#ocrd)
* [bash library](#bash-library)
* [Testing](#testing)
* [See Also](#see-also)

<!-- END-MARKDOWN-TOC -->

## Introduction

This repository contains the python packages that form the base for tools within the
[OCR-D ecosphere](https://github.com/topics/ocr-d).

All packages are also published to [PyPI](https://pypi.org/search/?q=ocrd).

## Installation

**NOTE** Unless you want to contribute to OCR-D/core, we recommend installation
as part of [ocrd_all](https://github.com/OCR-D/ocrd_all) which installs a
complete stack of OCR-D-related software.

The easiest way to install is via `pip`:

```sh
pip install ocrd

# or just the functionality you need, e.g.

pip install ocrd_modelfactory
```

All Python software released by [OCR-D](https://github.com/OCR-D) requires Python 3.8 or higher.

**NOTE** Some OCR-D-Tools (or even test cases) _might_ reveal an unintended behavior if you have specific environment modifications, like:
* using a custom build of [ImageMagick](https://github.com/ImageMagick/ImageMagick), whose format delegates are different from what OCR-D supposes
* custom Python logging configurations in your personal account

## Command line tools

**NOTE:** All OCR-D CLI tools support a `--help` flag which shows usage and
supported flags, options and arguments.

### `ocrd` CLI

* [CLI usage](https://ocr-d.de/core/api/ocrd/ocrd.cli.html)
* [Introduction to `ocrd workspace`](https://github.com/OCR-D/ocrd-website/wiki/Intro-ocrd-workspace-CLI)
* [OCR-D user guide](https://ocr-d.de/en/use)

### `ocrd-dummy` CLI

A minimal [OCR-D processor](https://ocr-d.de/en/user_guide#using-the-ocr-d-processors) that copies from `-I/-input-file-grp` to `-O/-output-file-grp`

## Configuration

Almost all behaviour of the OCR-D/core software is configured via CLI options and flags, which can be listed with the `--help` flag that all CLI support.

Some parts of the software are configured via environment variables:

* `OCRD_METS_CACHING`: If set to `true`, access to the METS file is cached, speeding in-memory search and modification.
* `OCRD_PROFILE`: This variable configures the built-in CPU and memory profiling. If empty, no profiling is done. Otherwise expected to contain any of the following tokens:
  * `CPU`: Enable CPU profiling of processor runs
  * `RSS`: Enable RSS memory profiling
  * `PSS`: Enable proportionate memory profiling
* `OCRD_PROFILE_FILE`: If set, then the CPU profile is written to this file for later peruse with a analysis tools like [snakeviz](https://jiffyclub.github.io/snakeviz/)

* `PATH`: Search path for processor executables (affects `ocrd process` and `ocrd resmgr`).
* `HOME`: Directory to look for `ocrd_logging.conf`, fallback for unset XDG variables (see below).

* `XDG_CONFIG_HOME`: Directory to look for `./ocrd/resources.yml` (i.e. `ocrd resmgr` user database) – defaults to `$HOME/.config`.
* `XDG_DATA_HOME`: Directory to look for `./ocrd-resources/*` (i.e. `ocrd resmgr` data location) – defaults to `$HOME/.local/share`.

* `OCRD_DOWNLOAD_RETRIES`: Number of times to retry failed attempts for downloads of workspace files.
* `OCRD_DOWNLOAD_TIMEOUT`: Timeout in seconds for connecting or reading (comma-separated) when downloading.

* `OCRD_METS_CACHING`: Whether to enable in-memory storage of OcrdMets data structures for speedup during processing or workspace operations.

* `OCRD_MAX_PROCESSOR_CACHE`: Maximum number of processor instances (for each set of parameters) to be kept in memory (including loaded models) for processing workers or processor servers.

* `OCRD_NETWORK_SERVER_ADDR_PROCESSING`: Default address of Processing Server to connect to (for `ocrd network client processing`).
* `OCRD_NETWORK_SERVER_ADDR_WORKFLOW`: Default address of Workflow Server to connect to (for `ocrd network client workflow`).
* `OCRD_NETWORK_SERVER_ADDR_WORKSPACE`: Default address of Workspace Server to connect to (for `ocrd network client workspace`).
* `OCRD_NETWORK_RABBITMQ_CLIENT_CONNECT_ATTEMPTS`: Number of attempts for a worker to create its queue. Helpful if the rabbitmq-server needs time to be fully started.

## Packages

### ocrd_utils

Contains utilities and constants, e.g. for logging, path normalization, coordinate calculation etc.

See [README for `ocrd_utils`](./README_ocrd_utils.md) for further information.

### ocrd_models

Contains file format wrappers for PAGE-XML, METS, EXIF metadata etc.

See [README for `ocrd_models`](./README_ocrd_models.md) for further information.

### ocrd_modelfactory

Code to instantiate [models](#ocrd-models) from existing data.

See [README for `ocrd_modelfactory`](./README_ocrd_modelfactory.md) for further information.

### ocrd_validators

Schemas and routines for validating BagIt, `ocrd-tool.json`, workspaces, METS, page, CLI parameters etc.

See [README for `ocrd_validators`](./README_ocrd_validators.md) for further information.

### ocrd_network

Components related to OCR-D Web API

See [README for `ocrd_network`](./README_ocrd_network.md) for further information.

### ocrd

Depends on all of the above, also contains decorators and classes for creating OCR-D processors and CLIs.

Also contains the command line tool `ocrd`.

See [README for `ocrd`](./README_ocrd.md) for further information.

## bash library

Builds a bash script that can be sourced by other bash scripts to create OCRD-compliant CLI.

See [README for `bashlib`](./README_bashlib.md) for further information.

## Testing

Download assets (`make assets`)

Test with local files: `make test`

- Test with remote assets:
  - `make test OCRD_BASEURL='https://github.com/OCR-D/assets/raw/master/data/'`

## See Also

  - [OCR-D Specifications](https://https://ocr-d.de/en/spec/) ([Repo](https://github.com/ocr-d/spec))
  - [OCR-D core API documentation](https://ocr-d.de/core) (built here via `make docs`)
  - [OCR-D Website](https://ocr-d.de) ([Repo](https://github.com/ocr-d/ocrd-website))


# Radeon ProRender Blender Addon

### Build Requirements

4.1+
====
- Blender 4.1+
- Python 3.11 (Blender 4.1 uses 3.11) x64(for Core) - all code, addon and misc tested with python3
- python-cffi
  - `py -3.11 -m pip install cffi`
  - `py -3.11 -m pip install numpy`
- Visual Studio 2015 SP3 / 2017 / 2019 with SDK 8.1 and 2015.3 v140 toolset installed
- If you are using Visual studio 2019 you would need to install the Windows SDK 8.1 manually from Microsoft website https://developer.microsoft.com/en-us/windows/downloads/sdk-archive/
- cmake 3.x. Make sure it's added to the PATH environment variable

Note that the .sln provided is for easy editing and searching of files on Windows.  The blender code builds on the command line rather than in the solution file.  Visual Studio does provided support for debugging Python when you attach to the running Blender process with loaded addon.

### ThirdParty libraries

There is ThirdParty repository included to the project as a submodule. Please update submodules:

Plugin includes 4 submodules:
RadeonProRender SDK:
git@github.com:Radeon-Pro/RadeonProRenderSDK.git

Shared components
Image Processing Library:
git@github.com:Radeon-Pro/RadeonProImageProcessingSDK.git

ThirdParty components and miscellaneous tools
git@github.com:Radeon-Pro/RadeonProRenderThirdPartyComponents.git

All of them are included via SSH protocol. You will need to create and install SSH keys https://help.github.com/en/github/authenticating-to-github/connecting-to-github-with-ssh

Once SSH keys are installed update/checkout submodules for active branch

`git submodule update --init -f --recursive`

## Developing

### Coding Conventions

Aim is to conform to [pep8](https://www.python.org/dev/peps/pep-0008/).
At minimum it's 4 spaces for indentation, sources are utf-8, there's .gitconfig in the root of the project - please set you editor to use it(for most simplicity). E.g. PyCharm(recommended!) default setting are fine and seems that it also picks up .editorconfig automatically also, Tortoise Merge has a checkbox 'Enable EditorConfig', for Visual Studio there's [EditorConfig extension](https://visualstudiogallery.msdn.microsoft.com/c8bccfe2-650c-4b42-bc5c-845e21f96328)

Git - we try to avoid merge commits, easiest way to do it:

`git config [--global] merge.ff only` # this one rejects merges that would result in merge commit

`git config [--global] pull.rebase true` # converts pull to do, essentially, fetch&rebase

Also, make more meaningful commits(one commit per feature) the easy way:

`git merge <branch> --squash` # this will create a single change set from multiple commits coming from <branch>

### Recommended software

- PyCharm Community Edition - very recommended, possible to enable intellisense(limited) for Blender code and for RPR Core
- Visual Studio - has a very nice python extension, possible to enable intellisense for Blender and for RPR Core, provides remote debugging in Blender

## Build

run `build.py` to build.

## Run Addon while developing it(without real installation)

- make sure you have no installed addon for Blender version you want to use; remove installed version if needed.
- set environment variable BLENDER_EXE to blender.exe you want to use via the command line or system environment settings.
- run run_blender_with_rpr.cmd

Example:

`set BLENDER_EXE="C:\Program Files\Blender Foundation\Blender 2.93\blender.exe" && run_blender_with_rpr.cmd`

### Debugging

#### log

Using python's 'logging' module underneath, rprblender.utils.logging has functions similar to logging. It also includes callable class Log which provides simplified interface to do logging.
Example:
    from rprblender.utils import logging
    log = logging.Log(tag='export.mesh')

    log("sync", mesh, obj)

e.g. `logging.debug(*argv, tag)` where argv is what is printed(same as with print) and tag is string suffix for logger name, for filtering
so that `logging.limit_log(name, level_show_always)` will allow to filter out what doesn't start with `name`(expect levels equal or above `level_show_always`)

 configdev.py(loaded very early) can be used to include code like `limit_log` to configure your session

    from .utils import logging
    logging.limit_log('default', logging.DEBUG)

    from . import config
    config.pyrpr_log_calls = True #  log all Core function calls to console, can be VERY useful

- Visual Studio has really nice(and working) mixed(python and C stack) debugging - recommended!
- Blender debug - it's easiest to [build Blender](https://wiki.blender.org/index.php/Dev:Doc/Building_Blender/Windows/msvc/CMake) in Release or RelWithDebInfo(and add ``#pragma optimize( "", off )``)
- Debug in PyCharm - `import pydevd; pydevd.settrace('localhost', port=52128, stdoutToServer=True, stderrToServer=True, suspend=False)`

## Making a new release

- run `build_installer.py <build_folder>`. Where `build_folder` is some separate location - it will clone needed repos(if not already), reset then to needed branch and build installer. Byt default it builds windows installer on master.
- tag the commit in the build folder's ProRenderBlenderPlugin `git tag builds/x.y.zz`
- push the tag `git push --tags`
- increase version in `src/rprblender/__init__.py`

## PyCharm

### Blender api intellisense support

Get [pycharm-blender](https://github.com/mutantbob/pycharm-blender). See instructions on the github page or, in short,
run `pypredef_gen.py` from Blender itself or using command line, e.g. `blender --python pypredef_gen.py`,
add "pypredef" folder path that this script creates to you PyCharm Interpreter paths, find paths settings under `File | Settings(or Default Settings) | Project Interpreter`

Increase max file size for Pycharm intellisence(bpy.py generated is huge), go to `Help | Edit Custom VM Options` and add the following line:

  -Didea.max.intellisense.filesize=5000000

Restart PyCharm

## Visual Studio

### Create and configure RPRBlender python project

Install python extension in Visual Studio

Create new project from existing python code: Menu -> File -> New -> Project -> Python tab -> From Existing Python Code

Add following Search Paths to project:
  - rprblender\support
  - <path to Blender 2.93>\2.93\scripts\modules   # path where to Blender's modules
  - <path to "PyCharm->Blender api intellisense support">

### Configure VS remote debugger to Blender

#### Install Blender-VScode-Debugger addon

Get Blender-VScode-Debugger plugin for Blender from https://github.com/Barbarbarbarian/Blender-VScode-Debugger

Install plugin Blender_VScode_Debugger.py in Blender: Menu -> File -> User Preferences -> Click "Install Add-on from file" -> Select Blender_VScode_Debugger.py -> Click "Install Add-on from file". New adddon "Development: Debugger for Visual Code" should be appeared

Enable "Development: Debugger for Visual Code" addon. Select "Path to PTVSD module": C:\Program Files (x86)\Microsoft Visual Studio 14.0\Common7\IDE\Extensions\Microsoft\Python Tools for Visual Studio\2.2\

Click "Save User Settings"

#### Attach VS remote debugger to Blender

In Blender: press <Space> -> in appeared dialog type "debug" -> select "Connect to Visual Studio Code Debugger"

In VS: Menu -> Debug -> Attach To Process -> select Transport "Python remote (ptvsd)" -> type in Qualifier "my_secret@localhost:3000" -> click Refresh: process Blender.exe "tcp://localhost:3000" should be appeared -> select process -> click Attach

Remote debugging connection established.

### Versioning

The version number should be updated when a new plugin is released.  This is done by editing the version field
of the bl_info structure in the src/rprblender/__init__.py file. Currently a build script will update the build
number when checkins happen to the master branch.  So it is only necessary to update the major or minor number
when required.


silx toolkit
============

.. |silxView| image:: http://www.silx.org/doc/silx/img/silx-view-v1-0.gif
   :height: 480px

The purpose of the *silx* project is to provide a collection of Python packages to support the
development of data assessment, reduction and analysis applications at synchrotron
radiation facilities.
*silx* aims to provide reading/writing tools for different file formats, data reduction routines
and a set of Qt widgets to browse and visualise data.

The current version features:

* Support of `HDF5 <https://www.hdfgroup.org/HDF5/>`_,
  `SPEC <https://certif.com/spec.html>`_ and
  `FabIO <http://www.silx.org/doc/fabio/dev/getting_started.html#list-of-file-formats-that-fabio-can-read-and-write>`_
  images file formats.
* OpenCL-based data processing: image alignment (SIFT),
  image processing (median filter, histogram),
  filtered backprojection for tomography,
  convolution
* Data reduction: histogramming, fitting, median filter
* A set of Qt widgets, including:

  * 1D and 2D visualization widgets with a set of associated tools using multiple backends (matplotlib or OpenGL)
  * OpenGL-based widgets to visualize data in 3D (scalar field with isosurface and cut plane, scatter plot)
  * a unified browser for HDF5, SPEC and image file formats supporting inspection and
    visualization of n-dimensional datasets.

* a set of applications:

  * a unified viewer (*silx view filename*) for HDF5, SPEC and image file formats

    |silxView|

  * a unified converter to HDF5 format (*silx convert filename*)

Installation
------------

To install silx (and all its dependencies), run:

.. code-block:: bash

    pip install silx[full]

To install silx with a minimal set of dependencies, run:

.. code-block:: bash

    pip install silx

Or using Anaconda on Linux and MacOS:

.. code-block:: bash

    conda install silx -c conda-forge

Unofficial packages for different distributions are available:

- Unofficial Debian10 and Ubuntu20.04 packages are available at http://www.silx.org/pub/linux-repo/
- CentOS 7 rpm packages are provided by Max IV at: http://pubrepo.maxiv.lu.se/rpm/el7/x86_64/
- Fedora 23 rpm packages are provided by Max IV at http://pubrepo.maxiv.lu.se/rpm/fc23/x86_64/
- Arch Linux (AUR) packages are also available: https://aur.archlinux.org/packages/python-silx

`Detailed installation instructions <http://www.silx.org/doc/silx/latest/install.html>`_
are available in the documentation.

Documentation
-------------

The documentation of `latest release <http://www.silx.org/doc/silx/latest/>`_ and
the documentation of `the nightly build <http://www.silx.org/doc/silx/dev>`_ are
available at http://www.silx.org/doc/silx/

Testing
-------

*silx* features a comprehensive test-suite used in continuous integration for
all major operating systems:

- Github Actions CI status: |Github Actions Status|
- Appveyor CI status: |Appveyor Status|

Please refer to the `documentation on testing <http://www.silx.org/doc/silx/latest/install.html#testing>`_
for details.

Examples
--------

Some examples of sample code using silx are provided with the
`silx documentation <http://www.silx.org/doc/silx/latest/sample_code/index.html>`_.

License
-------

The source code of *silx* is licensed under the MIT license.
See the `LICENSE <https://github.com/silx-kit/silx/blob/master/LICENSE>`_ and
`copyright <https://github.com/silx-kit/silx/blob/master/copyright>`_ files for details.

Citation
--------

*silx* releases can be cited via their DOI on Zenodo: |zenodo DOI|

.. |Github Actions Status| image:: https://github.com/silx-kit/silx/workflows/CI/badge.svg
   :target: https://github.com/silx-kit/silx/actions
.. |Appveyor Status| image:: https://ci.appveyor.com/api/projects/status/qgox9ei0wxwfagrb/branch/master?svg=true
   :target: https://ci.appveyor.com/project/ESRF/silx?branch=master
.. |zenodo DOI| image:: https://zenodo.org/badge/DOI/10.5281/zenodo.591709.svg
   :target: https://doi.org/10.5281/zenodo.591709


# CAIRIS

[![Coverage Status](https://coveralls.io/repos/github/cairis-platform/cairis/badge.svg?branch=master)](https://coveralls.io/github/cairis-platform/cairis?branch=master)
[![DOI](https://zenodo.org/badge/3790944.svg)](https://zenodo.org/badge/latestdoi/3790944)
[![GitHub issues](https://img.shields.io/github/issues/cairis-platform/cairis.svg?style=flat-square)](https://github.com/cairis-platform/cairis/issues?utf8=✓&q=is%3Aissue+is%3Aopen)
[![GitHub watchers](https://img.shields.io/github/watchers/cairis-platform/cairis.svg?style=flat-square)](https://github.com/cairis-platform/cairis/watchers)
[![GitHub stars](https://img.shields.io/github/stars/cairis-platform/cairis.svg?style=flat-square)](https://github.com/cairis-platform/cairis/stargazers)
[![GitHub forks](https://img.shields.io/github/forks/cairis-platform/cairis.svg?style=flat-square)](https://github.com/cairis-platform/cairis/network)
[![Docker Pulls](https://img.shields.io/docker/pulls/shamalfaily/cairis.svg?style=flat-square)](https://hub.docker.com/r/shamalfaily/cairis/)
[![Requirements Status](https://requires.io/github/cairis-platform/cairis/requirements.svg?branch=master)](https://requires.io/github/cairis-platform/cairis/requirements/?branch=master)
[![Documentation Status](https://readthedocs.org/projects/cairis/badge/?version=latest)](http://cairis.readthedocs.io/en/latest/?badge=latest)
[![License](https://img.shields.io/github/license/cairis-platform/cairis.svg)](https://github.com/cairis-platform/cairis/blob/master/LICENSE)
[![CII Best Practices](https://bestpractices.coreinfrastructure.org/projects/1411/badge)](https://bestpractices.coreinfrastructure.org/projects/1411)
[![FOSSA Status](https://app.fossa.com/api/projects/git%2Bgithub.com%2Fcairis-platform%2Fcairis.svg?type=shield)](https://app.fossa.com/projects/git%2Bgithub.com%2Fcairis-platform%2Fcairis?ref=badge_shield)

CAIRIS (Computer Aided Integration of Requirements and Information Security) is a platform for modelling secure and usable systems. CAIRIS was built from the ground-up to support usability, requirements, and risk analysis.

See [the CAIRIS website](https://cairis.org) for more information, and details on how to get started.

## License
[![FOSSA Status](https://app.fossa.com/api/projects/git%2Bgithub.com%2Fcairis-platform%2Fcairis.svg?type=large)](https://app.fossa.com/projects/git%2Bgithub.com%2Fcairis-platform%2Fcairis?ref=badge_large)


# MeerK40t
MeerK40t (pronounced MeerKat) is a built-from-the-ground-up MIT licensed open-source laser cutting software.

## Primary Goals
* Provide users with high quality laser control software.
* Provide developers with a highly extensible platform to help further their own ideas, and provide novel work to the laser community at large.

## Running
MeerK40t is written in Python and precompiled versions are [available for download](https://github.com/meerk40t/meerk40t/releases) for Windows, Mac OSX, Linux and Raspberry Pi (more infos below). Do note this sometimes will give false postitives for various viruses (especially since Meerk40t isn't signed).

### Source

Alternatively you can run MeerK40t directly from Python. `pip install meerk40t[all]` with python installed will usually be sufficient. Then merely run `meerk40t` at the command line.

See [Install: Source wiki page](https://github.com/meerk40t/meerk40t/wiki/Install:-Source)

### GUI
![meerk40t8](https://user-images.githubusercontent.com/3302478/196283699-745d0616-5e74-49b3-ba95-f4902061584b.png)

The wxMeerK40t is the GUI and is written in wxPython. We use AUI to allow to have a very highly configurable UI. We can easily add panes and tools and there are quite a few available already.

### Command Line Interface

Meerk40t has an advanced internal console system allowing access to most parts of the code with various commands. It also provides a command line interface which should allow you to automate any processes.

## Drivers

Meerk40t provides a variety of drivers with an extensible framework to provide support for other new laser devices. The code was written with the myriad of possibilities for different software in mind. For example, it may be essential that GRBL be able to reset an alarm or notify the user of particular error codes. The configuration for GRBL is not the same for the configuration of other laser control drivers. With this in mind, Meerk40t can radically change how and when it works

### Supported devices
*   Lihuiyu M2/M3-Nano
*   GRBL
*   Ezcad2-compatible JCZ controllers galvo lasers
*   Moshiboard
*   NewlyDraw System 8.1 Lasers
*   Ruida-Emulation (Middleman between Lightburn and K40)

### Lihuiyu M2-Nano/M3-Nano
For the Lihuiyu (stock driver), Meerk40t supports both the windows and libusb connection methods, making it compatible with Whisperer and with the original Chinese software. So MeerK40t can usually run alongside these other pieces of software interchangeably.

### GRBL

GRBL is itself open source and the various interfaces with the board should be quite well understood.

### EZCAD2-Compatible Galvo LMC

Meerk40t supports controlling galvo as well as gantry lasers with open source support.

### Moshiboard

The support for old moshiboards makes meerk40t the only known opensource software that controls moshiboards.

### System 8.1 Lasers (NewlyDraw)

HPGL-modified laser systems produced under many different company names

## Support
The primary source for help and documentation is the [MeerK40t Wiki - please click here](https://github.com/meerk40t/meerk40t/wiki).

If you have a bug, feature request, or other issue raise it here. These are likely to be resolved. Squeaky wheels get the grease.
https://github.com/meerk40t/meerk40t/issues

If you need additional support, please research/ask on:

*   [Facebook](https://www.facebook.com/groups/716000085655097/)
*   [Maker Forums](https://forum.makerforums.info/t/about-the-meerk40t-category/79660)
*   [YouTube - David Olsen's channel](https://www.youtube.com/channel/UCsAUV23O2FyKxC0HN7nkAQQ)
*   [YouTube - Some instruction videos](https://www.youtube.com/channel/UCMN9gGvpacxZINPZCSOecaQ)
*   [Discord](https://discord.gg/vkDD3HdQq6)

## Assisting the Project

Open source projects live and die with their support. There are a lots of ways to help the project. There are also a lot of ways the project should help you.
*   Code
*   Provide Translations in other languages.
*   Design ( Good design instincts, smooth out the rough edges)
*   Compile/Testers
*   Beta testers
*   Make helpful support content
*   Make guides ("How to set up cameras?", etc.)
*   Bounce ideas around

## Download
You can find and download all current and historical versions in the [Releases](https://github.com/meerk40t/meerk40t/releases) section.

Currently, primary branches:
* 0.9 - Active - New features and some underlying architectural changes. Try the latest released version: [0.9.4](https://github.com/meerk40t/meerk40t/releases#latest)
* 0.8 - Maintenance - may receive critical bugfixes but no more new features, latest version: [0.8.12](https://github.com/meerk40t/meerk40t/releases/tag/0.8.12000) (Oct 17, 2023)
* 0.7 - Discontinued - K40 support only (including ruidacontrol emulator for 3rd party lasersoftware integration), latest version [0.7.10](https://github.com/meerk40t/meerk40t/releases/tag/0.7.10000) (June 13, 2023)
* 0.6 - Discontinued - K40 support only, latest version: [0.6.24](https://github.com/meerk40t/meerk40t/releases/tag/0.6.24) (Oct 11, 2021)

Just download one of the files for Windows, Mac OSX, Linux and Raspberry Pi.

## Lightburn integration
Meerk40t allows to act as an intermediary between your K40 laser and software that supports Ruida-controlled laser equipment - [Lightburn](https://lightburnsoftware.com/) is a relevant example of such a software product. You just need to issue the command ``ruidacontrol`` in MeerK40ts console window, and you will then be able to add an emulated Ruida Laser inside Lightburn™. Laser jobs that are created inside Lightburn™ and sent to this laser will be picked up by MeerK40t and sent to your K40. See some more detailed instructions in this [video](https://www.youtube.com/watch?v=LUUfLf5Agu0). Please note `ruidacontrol` will require the DSP version of Lightburn™. (Present in all versions since 0.7)

With 0.9 another way of interacting with Lightburn was introduced, which will work as well with the standard version of LB: You just need to issue the command ``grblcontrol`` in MeerK40ts console window, and you will then be able to add an emulated remote GBRL-LPC laser inside Lightburn or any TCP GRBL control software.


[![ktools version](https://img.shields.io/github/tag/Oasislmf/ktools?label=ktools)](https://github.com/OasisLMF/ktools/releases)
[![PyPI version](https://badge.fury.io/py/oasislmf.svg)](https://badge.fury.io/py/oasislmf)
[![FM Testing Tool](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/OasisLMF/OasisLMF/blob/main/fm_testing_tool/FmTesting.ipynb)

[![Oasislmf Testing](https://github.com/OasisLMF/OasisLMF/actions/workflows/unittest.yml/badge.svg?branch=main&event=push)](https://github.com/OasisLMF/OasisLMF/actions/workflows/unittest.yml)
[![PiWind output check](https://github.com/OasisLMF/OasisLMF/actions/workflows/piwind-test.yml/badge.svg?branch=main&event=push)](https://github.com/OasisLMF/OasisLMF/actions/workflows/piwind-test.yml)
[![PiWind MDK](https://github.com/OasisLMF/OasisLMF/actions/workflows/piwind-mdk.yml/badge.svg?branch=main&event=push)](https://github.com/OasisLMF/OasisLMF/actions/workflows/piwind-mdk.yml)

# OasisLMF
The `oasislmf` Python package, loosely called the *model development kit (MDK)* or the *MDK package*, provides a command line toolkit for developing, testing and running Oasis models end-to-end locally, or remotely via the Oasis API. It can generate ground-up losses (GUL), direct/insured losses (IL) and reinsurance losses (RIL). It can also generate deterministic losses at all these levels.

## Versioning and Updates

### Current Stable Versions

* `1.15.x` [stable/1.15.x](https://github.com/OasisLMF/OasisLMF/tree/stable/1.15.x) From Feb 2021
* `1.23.x` [stable/1.23.x](https://github.com/OasisLMF/OasisLMF/tree/stable/1.23.x) From Dec 2021
* `1.26.x` [stable/1.26.x](https://github.com/OasisLMF/OasisLMF/tree/stable/1.26.x) From Jun 2022
* `1.27.x` [stable/1.27.x](https://github.com/OasisLMF/OasisLMF/tree/stable/1.27.x) From Jan 2023
* `1.28.x` (Up comming) July 2023

### Release Schedule
**Until end of 2023**
Until the year 2023, we will be following a six-month release cycle for our stable versions. During each six-month period, we will release a new stable version with added features. These updates will adhere to the Semantic Versioning (semver) format and will increment the minor version number.
That version of oaisislmf is then 'frozen' into a branch matching the new version number, so on release 1.28.0 the code base is copied to a branch `stable/1.28.x` where backported features and fixes are applied.

**After 2023**
Starting from 2023, we will transition to a yearly release cycle for our stable versions. Each year, we will release a new stable version with additional features.

### Monthly Updates
Every month, we will provide updates to the latest stable version. These updates will include new compatible features and bug fixes, ensuring that our software remains up-to-date and reliable.

During the monthly update, if any bug fixes are required, they will also be applied to the older stable versions. This approach guarantees that all stable versions receive necessary bug fixes, while maintaining a consistent output numbers for that stable version.

## Features

For running models locally the CLI provides a `model` subcommand with the following options:

* `model generate-exposure-pre-analysis`: generate new Exposure input using user custom code (ex: geo-coding, exposure enhancement, or dis-aggregation...)
* `model generate-keys`: generates Oasis keys files from model lookups; these are essentially line items of (location ID, peril ID, coverage type ID, area peril ID, vulnerability ID) where peril ID and coverage type ID span the full set of perils and coverage types that the model supports; if the lookup is for a complex/custom model the keys file will have the same format except that area peril ID and vulnerability ID are replaced by a model data JSON string
* `model generate-oasis-files`: generates the Oasis input CSV files for losses (GUL, GUL + IL, or GUL + IL + RIL); it requires the provision of source exposure and optionally source accounts and reinsurance info. and scope files (in OED format), as well as assets for instantiating model lookups and generating keys files
* `model generate-losses`: generates losses (GUL, or GUL + IL, or GUL + IL + RIL) from a set of pre-existing Oasis files
* `model run`: runs the model from start to finish by generating losses (GUL, or GUL + IL, or GUL + IL + RIL) from the source exposure, and optionally source accounts and reinsurance info. and scope files (in OED or RMS format), as well as assets related to lookup instantiation and keys file generation

The optional `--summarise-exposure` flag can be issued with `model generate-oasis-files` and `model run` to generate a summary of Total Insured Values (TIVs) grouped by coverage type and peril. This produces the `exposure_summary_report.json` file.

For remote model execution the `api` subcommand provides the following main subcommand:

* `api run`: runs the model remotely (same as `model run`) but via the Oasis API

For generating deterministic losses an `exposure run` subcommand is available:

* `exposure run`: generates deterministic losses (GUL, or GUL + IL, or GUL + IL + RIL)

The reusable libraries are organised into several sub-packages, the most relevant of which from a model developer or user's perspective are:

* `api_client`
* `model_preparation`
* `model_execution`
* `utils`

## Minimum Python Requirements

Starting from 1st January 2019, Pandas will no longer be supporting Python 2. As Pandas is a key dependency of the MDK we are **dropping Python 2 (2.7) support** as of this release (1.3.4). The last version which still supports Python 2.7 is version `1.3.3` (published 12/03/2019).

Also for this release (and all future releases) a **minimum of Python 3.8 is required**.

## Installation

The latest released version of the package, or a specific package version, can be installed using `pip`:

    pip install oasislmf[==<version string>]

Alternatively you can install the latest development version using:

    pip install git+{https,ssh}://git@github.com/OasisLMF/OasisLMF

You can also install from a specific branch `<branch name>` using:

    pip install [-v] git+{https,ssh}://git@github.com/OasisLMF/OasisLMF.git@<branch name>#egg=oasislmf

## Enable Bash completion

Bash completion is a functionality which bash helps users type their commands by presenting possible options when users press the tab key while typing a command.

Once oasislmf is installed you'll need to be activate the feature by sourcing a bash file. (only needs to be run once)

### Local

    oasislmf admin enable-bash-complete

### Global

    echo 'complete -C completer_oasislmf oasislmf' | sudo tee /usr/share/bash-completion/completions/oasislmf

## Dependencies

### System

The package provides a built-in lookup framework (`oasislmf.model_preparation.lookup.OasisLookup`) which uses the Rtree Python package, which in turn requires the `libspatialindex` spatial indexing C library.

https://libspatialindex.github.io/index.html

Linux users can install the development version of `libspatialindex` from the command line using `apt`.

    [sudo] apt install -y libspatialindex-dev

and OS X users can do the same via `brew`.

    brew install spatialindex

The PiWind demonstration model uses the built-in lookup framework, therefore running PiWind or any model which uses the built-in lookup, requires that you install `libspatialindex`.

#### GNU/Linux

For GNU/Linux the following is a specific list of required system libraries

 * **Debian**: g++ compiler build-essential, libtool, zlib1g-dev autoconf on debian distros

     sudo apt install g++ build-essential libtool zlib1g-dev autoconf

 * **Red Hat**: 'Development Tools' and zlib-devel

### Python

Package Python dependencies are controlled by `pip-tools`. To install the development dependencies first, install `pip-tools` using:

    pip install pip-tools

and run:

    pip-sync

To add new dependencies to the development requirements add the package name to `requirements.in` or
to add a new dependency to the installed package add the package name to `requirements-package.in`.
Version specifiers can be supplied to the packages but these should be kept as loose as possible so that
all packages can be easily updated and there will be fewer conflict when installing.

After adding packages to either `*.in` file:

    pip-compile && pip-sync

should be ran ensuring the development dependencies are kept up to date.

### ods_tools
OasisLMF uses the ods_tools package to read exposure files and the setting files
The version compatible with each OasisLMF is manage in the requirement files.
below is the summary:

- OasisLMF 1.23.x or before => no ods_tools
- OasisLMF 1.26.x => use ods_tools 2.3.2
- OasisLMF 1.27.0 => use ods_tools 3.0.0 or later
- OasisLMF 1.27.1 => use ods_tools 3.0.0 or later
- OasisLMF 1.27.2 => use ods_tools 3.0.4 or later

### pandas
Pandas has released its major version number 2 breaking some of the compatibility with the 1st version
Therefore, for all version of OasisLMF <= 1.27.2, the latest supported version for pandas is 1.5.3
Support for pandas 2, starts from version 1.27.3

## Testing

To test the code style run:

    flake8

To test against all supported python versions run:

    tox

To test against your currently installed version of python run:

    py.test

To run the full test suite run:

    ./runtests.sh

## Publishing

Before publishing the latest version of the package make you sure increment the `__version__` value in `oasislmf/__init__.py`, and commit the change. You'll also need to install the `twine` Python package which `setuptools` uses for publishing packages on PyPI. If publishing wheels then you'll also need to install the `wheel` Python package.

### Using the `publish` subcommand in `setup.py`

The distribution format can be either a source distribution or a platform-specific wheel. To publish the source distribution package run:

    python setup.py publish --sdist

or to publish the platform specific wheel run:

    python setup.py publish --wheel

### Creating a bdist for another platform

To create a distribution for a non-host platform use the `--plat-name` flag:

     python setup.py bdist_wheel --plat-name Linux_x86_64

     or

     python setup.py bdist_wheel --plat-name Darwin_x86_64

### Manually publishing, with a GPG signature

The first step is to create the distribution package with the desired format: for the source distribution run:

    python setup.py sdist

which will create a `.tar.gz` file in the `dist` subfolder, or for the platform specific wheel run:

    python setup.py bdist_wheel

which will create `.whl` file in the `dist` subfolder. To attach a GPG signature using your default private key you can then run:

    gpg --detach-sign -a dist/<package file name>.{tar.gz,whl}

This will create `.asc` signature file named `<package file name>.{tar.gz,whl}.asc` in `dist`. You can just publish the package with the signature using:

    twine upload dist/<package file name>.{tar.gz,whl} dist/<package file name>.{tar.gz,whl}.asc

## Documentation
* Issues
* Releases
* General Oasis documentation
* Model Development Kit (MDK)
* Modules

## License
The code in this project is licensed under BSD 3-clause license.


# MPP-Solar Device Python Package #

__BREAKING CHANGES__
- minimum python supported 3.10 for version >=0.16.0
- command separator changed to `#`
.

Python package with reference library of commands (and responses)
designed to get information from inverters and other solar inverters and power devices

Currently has support for:
- MPP-Solar and similar inverters, e.g.
  - PIP-4048MS
  - IPS-4000WM
  - Voltronic Axpert MKS 5KVA Off-grid Inverter-Chargers
  - LV5048
- JK BMS
- Victron VE Direct Devices:
  - tested on SmartShunt 500A
- Daly BMS

## Install ##
- `pip install mppsolar` (minimal install), or
- `pip install mppsolar[api]` (install server stuff), or
- `pip install mppsolar[ble]` (for Bluetooth support aka jkbms), or
- `pip install mppsolar[mongo]` (for MongoDB output), or
- `pip install mppsolar[pgsql]` (for PostgreSQL output), or
- `pip install mppsolar[push]` (for Prometheus PushGateway output), or
- `pip install mppsolar[systemd]` (for `--daemon` on Linux), or
- `docker pull jblance/mppsolar:latest` (docker install)

## Documentation ##
[See the wiki for documentation](https://github.com/jblance/mpp-solar/wiki)

## Support ##
If you want to tip me for this work, you can now buy me a coffee

[![buymeacoffee1](https://user-images.githubusercontent.com/1266998/225745276-54d6a4d4-a1ed-44f9-a1f2-e99eb1aa2812.png)](https://www.buymeacoffee.com/jblance)


![Build Status](https://travis-ci.org/flav-io/flavio.svg?branch=master) [![Coverage Status](https://coveralls.io/repos/github/flav-io/flavio/badge.svg)](https://coveralls.io/github/flav-io/flavio) [![DOI](https://zenodo.org/badge/22356/flav-io/flavio.svg)](https://zenodo.org/badge/latestdoi/22356/flav-io/flavio)

# flavio – flavour phenomenology in the Standard model and beyond

flavio is a Python package to compute observables in flavour physics, electroweak precision tests, Higgs physics,
and other precision tests of the Standard Model, both in
the Standard Model and in the presence of new physics encoded in Wilson
coefficients of dimension-6 operators.

## Project web site

Further information and the package documentation can be found at:

http://flav-io.github.io

This web site is built with Jekyll and hosted on Github pages. You can check
out its source (or submit pull requests) at https://github.com/flav-io/flav-io.github.io.

## Installation

See http://flav-io.github.io/docs/installation.html for instructions.

## Bugs and feature request

Please submit questions, bugs and feature request using
[Github's issue system](https://github.com/flav-io/flavio/issues) rather than
by E-mail. Thanks!

## Contributors

Maintainer:

- Peter Stangl (@peterstangl)

Original author:

- David M. Straub (@DavidMStraub)

Contributors (in alphabetical order):

- Jason Aebischer
- Jorge Alda Gallo
- Frederik Beaujean
- Adam Falkowski
- Tobias Felkl
- Ece Gürler
- Matěj Hudec
- Matthew Kirk
- Jonathan Kriewald
- Girish Kumar
- Jacky Kumar
- Christoph Langenbruch
- Christoph Niehoff
- Markus Prim
- Albert Puig
- Méril Reboud
- Stefanie Reichert
- Niladri Sahoo
- Jakub Šalko
- Michael Schmidt
- Aleks Smolkovič
- Peter Stangl
- Olcyr Sumensari
- Felicia Volle
- Zeren Simon Wang

## License

flavio is released under the MIT license.

## Citation

If you use flavio in a scientific publication, please cite:

> D. Straub, "flavio: a Python package for flavour and precision
phenomenology in the Standard Model and beyond", [arXiv:1810.08132](https://arxiv.org/abs/1810.08132)

This paper contains many references to results in the literature that went
into the code. Please also consider citing the relevant original literature,
where the real work was done.

The flavio repository itself is also citable via a DOI provided by Zenodo:

[![DOI](https://zenodo.org/badge/22356/flav-io/flavio.svg)](https://zenodo.org/badge/latestdoi/22356/flav-io/flavio)


# Streaming Integrator

[![Jenkins Build Status](https://wso2.org/jenkins/view/wso2-dependencies/job/products/job/streaming-integrator/badge/icon)](https://wso2.org/jenkins/view/wso2-dependencies/job/products/job/streaming-integrator/)
  [![GitHub Release](https://img.shields.io/github/release-pre/wso2/streaming-integrator.svg)](https://github.com/wso2/streaming-integrator/releases/)
  [![GitHub Release Date](https://img.shields.io/github/release-date-pre/wso2/streaming-integrator.svg)](https://github.com/wso2/streaming-integrator/releases)
  [![GitHub Open Issues](https://img.shields.io/github/issues-raw/wso2/streaming-integrator.svg)](https://github.com/wso2/streaming-integrator/commits/master)
  [![GitHub Last Commit](https://img.shields.io/github/last-commit/wso2/streaming-integrator.svg)](https://github.com/wso2/streaming-integrator/commits/master)
  [![License](https://img.shields.io/badge/License-Apache%202.0-blue.svg)](https://opensource.org/licenses/Apache-2.0)
  [](https://wso2-ei.slack.com/messages/streamingintegrator/)

## Overview

WSO2 Streaming Integrator (SI) is a streaming data processing server that allows you to integrate streaming data and take action based on streaming data.

WSO2 SI can be effectively used for:
- Realtime ETL with files, DBs, SaaS apps, HTTP endpoints, etc.
- Working with streaming messaging systems such as Kafka and NATS.
- Streaming data integration
- Executing complex integrations based on streaming data with [WSO2 Micro Integrator](https://github.com/wso2/micro-integrator).

WSO2 SI is powered by [Siddhi.io](https://siddhi.io/), a well-known cloud native open source stream processing engine. Siddhi lets users write complex stream processing logic using a SQL-like language known as [SiddhiQL](https://siddhi.io/en/v5.0/docs/). You can aggregate, transform, enrich, analyze, cleanse and correlate streams of data on the fly using Siddhi queries and constructs.

WSO2 SI lets you connect to any data source with any destination regardless of different protocols and data formats that are used by different endpoints. The SI store API provides the capability to fetch stored and aggregated data kept in-memory and in DBs via a REST API on demand using ad-hoc queries.

[SI tooling](https://github.com/wso2/streaming-integrator-tooling) provides a web-based IDE that allows you to build Siddhi applications with a drag-and-drop graphical editor, or a streaming SQL code editor. Its capability to simulate data streams and to debug Siddhi queries facilitates you to test your Siddhi applications. Created Siddhi applications can be directly deployed in a VM via the IDE, exported as a Docker image, or deployed as K8s artifacts that can be used with the Siddhi K8s Operator.

SI has native support for Kubernetes with a [K8s Operator](https://siddhi.io/en/v5.1/docs/siddhi-as-a-kubernetes-microservice/) designed to provide a convenient way of deploying SI on K8s. SI has a very simple deployment architecture, and you can achieve high availability with zero data loss with two nodes of SI.

Integration flows deployed in [WSO2 Micro Integrator (MI)](https://github.com/wso2/micro-integrator) can be invoked directly by SI in a seamless manner using low latency RPC. This allows you to build robust data processing and integration pipelines by combining powerful streaming and integration capabilities.

![Streaming Integrator/ Workflow](docs/images/streaming-integrator.png)

## Download

WSO2 Streaming Integrator is currently in the development stage. Therefore, you can download the source of WSO2 Streaming Integrator and build it. For detailed instructions, see the [Building from the Source section](#building-from-the-source).

## Building from the Source

To build WSO2 Streaming Integrator from source, follow the steps below.

1. Clone or download the source code from this repository.
2. Run `mvn clean install` from the root directory of the repository.
3. The generated Streaming Integrator distribution can be found at `streaming-integrator/modules/distribution/target/-streaming-integrator-<version>.zip`

When building with dependencies, you need to build in the following order.
1. [siddhi](https://github.com/siddhi-io/siddhi)
2. [carbon-analytics-common](https://github.com/wso2/carbon-analytics-common)
3. [carbon-analytics](https://github.com/wso2/carbon-analytics)
4. [carbon-dashboards](https://github.com/wso2/carbon-dashboards)
5. [carbon-ui-server](https://github.com/wso2/carbon-ui-server)

## Getting Started

To started with the Streaming Integrator follow the guides given below.

* [Quick Start Guide](https://ei.docs.wso2.com/en/latest/streaming-integrator/quick-start-guide/quick-start-guide/): Step by step guide to get your first Siddhi application running in less than 5 minutes.

* [Streaming Integrator 101](https://ei.docs.wso2.com/en/latest/streaming-integrator/quick-start-guide/getting-started/getting-started-guide-overview/): A 30 minute guide to explore the end to end development lifecycle of the Streaming Integrator.

## Deploy in Docker

WSO2 Streaming Integrator has a Docker distribution so that it can be deployed in any container-orchestration system.
The Docker image can be built from the source, or downloaded directly from Docker Hub.

### Build the Docker Image

Before building the docker image, install Docker on the host machine.

To build the Docker image from the source, issue the following command from the root directory.

`mvn clean install -Ddocker.skip=false`

### Get the Image from Docker Hub

To get the Docker image from Docker Hub, issue the following command.

```bash
docker pull wso2/streaming-integrator
```

## Deploy in Kubernetes

WSO2 Streaming Integrator can be deployed in a Kubernetes cluster using Siddhi Operator.

* [Siddhi operator](https://github.com/siddhi-io/siddhi-operator) enables the deployment of Siddhi applications directly in your Kubernetes cluster using a Kubernetes Custom Resource.
For more details see [Installing Streaming Integrator Using Kubernetes](https://docs.wso2.com/display/INSTALL/Installing+Enterprise+Integrator+Using+Kubernetes).

## Support

We are committed to ensuring that your enterprise middleware deployment is completely supported from evaluation to production. Our unique approach ensures that all support leverages our open development methodology and is provided by the very same engineers who build the technology.

For more details and to take advantage of this unique opportunity please visit our [support site](http://wso2.com/support).

## Reporting Issues

We encourage you to report issues, documentation faults and feature requests regarding WSO2 Streaming integrator through the [WSO2 SI Issue Tracker](https://github.com/wso2/streaming-integrator/issues).

When reporting security issues, please report them to [security@wso2.com](mailto:security@wso2.com), and make sure that you adhere to [WSO2 Security Vulnerability Reporting Guidelines](https://docs.wso2.com/display/Security/WSO2+Security+Vulnerability+Reporting+Guidelines).