PyPika - Python Query Builder
=============================

.. _intro_start:

|BuildStatus|  |CoverageStatus|  |Codacy|  |Docs|  |PyPi|  |License|

Abstract
--------

What is |Brand|?

|Brand| is a Python API for building SQL queries. The motivation behind |Brand| is to provide a simple interface for
building SQL queries without limiting the flexibility of handwritten SQL. Designed with data analysis in mind, |Brand|
leverages the builder design pattern to construct queries to avoid messy string formatting and concatenation. It is also
easily extended to take full advantage of specific features of SQL database vendors.

What are the design goals for |Brand|?
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

|Brand| is a fast, expressive and flexible way to replace handwritten SQL (or even ORM for the courageous souls amongst you).
Validation of SQL correctness is not an explicit goal of |Brand|. With such a large number of
SQL database vendors providing a robust validation of input data is difficult. Instead you are encouraged to check inputs you provide to |Brand| or appropriately handle errors raised from
your SQL database - just as you would have if you were writing SQL yourself.

.. _intro_end:

Read the docs: http://pypika.readthedocs.io/en/latest/

Installation
------------

.. _installation_start:

|Brand| supports python ``3.6+``.  It may also work on pypy, cython, and jython, but is not being tested for these versions.

To install |Brand| run the following command:

.. code-block:: bash

    pip install pypika

.. _installation_end:

Tutorial
--------

.. _tutorial_start:

The main classes in pypika are ``pypika.Query``, ``pypika.Table``, and ``pypika.Field``.

.. code-block:: python

    from pypika import Query, Table, Field

Selecting Data
^^^^^^^^^^^^^^

The entry point for building queries is ``pypika.Query``.  In order to select columns from a table, the table must
first be added to the query.  For simple queries with only one table, tables and columns can be references using
strings.  For more sophisticated queries a ``pypika.Table`` must be used.

.. code-block:: python

    q = Query.from_('customers').select('id', 'fname', 'lname', 'phone')

To convert the query into raw SQL, it can be cast to a string.

.. code-block:: python

    str(q)

Alternatively, you can use the `Query.get_sql()` function:

.. code-block:: python

    q.get_sql()

Tables, Columns, Schemas, and Databases
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

In simple queries like the above example, columns in the "from" table can be referenced by passing string names into
the ``select`` query builder function. In more complex examples, the ``pypika.Table`` class should be used. Columns can be
referenced as attributes on instances of ``pypika.Table``.

.. code-block:: python

    from pypika import Table, Query

    customers = Table('customers')
    q = Query.from_(customers).select(customers.id, customers.fname, customers.lname, customers.phone)

Both of the above examples result in the following SQL:

.. code-block:: sql

    SELECT id,fname,lname,phone FROM customers

An alias for the table can be given using the ``.as_`` function on ``pypika.Table``

.. code-block:: sql

    customers = Table('x_view_customers').as_('customers')
    q = Query.from_(customers).select(customers.id, customers.phone)

.. code-block:: sql

    SELECT id,phone FROM x_view_customers customers

A schema can also be specified. Tables can be referenced as attributes on the schema.

.. code-block:: sql

    from pypika import Table, Query, Schema

    views = Schema('views')
    q = Query.from_(views.customers).select(customers.id, customers.phone)

.. code-block:: sql

    SELECT id,phone FROM views.customers

Also references to databases can be used. Schemas can be referenced as attributes on the database.

.. code-block:: sql

    from pypika import Table, Query, Database

    my_db = Database('my_db')
    q = Query.from_(my_db.analytics.customers).select(customers.id, customers.phone)

.. code-block:: sql

    SELECT id,phone FROM my_db.analytics.customers

Results can be ordered by using the following syntax:

.. code-block:: python

    from pypika import Order
    Query.from_('customers').select('id', 'fname', 'lname', 'phone').orderby('id', order=Order.desc)

This results in the following SQL:

.. code-block:: sql

    SELECT "id","fname","lname","phone" FROM "customers" ORDER BY "id" DESC

Arithmetic
""""""""""

Arithmetic expressions can also be constructed using pypika.  Operators such as `+`, `-`, `*`, and `/` are implemented
by ``pypika.Field`` which can be used simply with a ``pypika.Table`` or directly.

.. code-block:: python

    from pypika import Field

    q = Query.from_('account').select(
        Field('revenue') - Field('cost')
    )

.. code-block:: sql

    SELECT revenue-cost FROM accounts

Using ``pypika.Table``

.. code-block:: python

    accounts = Table('accounts')
    q = Query.from_(accounts).select(
        accounts.revenue - accounts.cost
    )

.. code-block:: sql

    SELECT revenue-cost FROM accounts

An alias can also be used for fields and expressions.

.. code-block:: sql

    q = Query.from_(accounts).select(
        (accounts.revenue - accounts.cost).as_('profit')
    )

.. code-block:: sql

    SELECT revenue-cost profit FROM accounts

More arithmetic examples

.. code-block:: python

    table = Table('table')
    q = Query.from_(table).select(
        table.foo + table.bar,
        table.foo - table.bar,
        table.foo * table.bar,
        table.foo / table.bar,
        (table.foo+table.bar) / table.fiz,
    )

.. code-block:: sql

    SELECT foo+bar,foo-bar,foo*bar,foo/bar,(foo+bar)/fiz FROM table

Filtering
"""""""""

Queries can be filtered with ``pypika.Criterion`` by using equality or inequality operators

.. code-block:: python

    customers = Table('customers')
    q = Query.from_(customers).select(
        customers.id, customers.fname, customers.lname, customers.phone
    ).where(
        customers.lname == 'Mustermann'
    )

.. code-block:: sql

    SELECT id,fname,lname,phone FROM customers WHERE lname='Mustermann'

Query methods such as select, where, groupby, and orderby can be called multiple times.  Multiple calls to the where
method will add additional conditions as

.. code-block:: python

    customers = Table('customers')
    q = Query.from_(customers).select(
        customers.id, customers.fname, customers.lname, customers.phone
    ).where(
        customers.fname == 'Max'
    ).where(
        customers.lname == 'Mustermann'
    )

.. code-block:: sql

    SELECT id,fname,lname,phone FROM customers WHERE fname='Max' AND lname='Mustermann'

Filters such as IN and BETWEEN are also supported

.. code-block:: python

    customers = Table('customers')
    q = Query.from_(customers).select(
        customers.id,customers.fname
    ).where(
        customers.age[18:65] & customers.status.isin(['new', 'active'])
    )

.. code-block:: sql

    SELECT id,fname FROM customers WHERE age BETWEEN 18 AND 65 AND status IN ('new','active')

Filtering with complex criteria can be created using boolean symbols ``&``, ``|``, and ``^``.

AND

.. code-block:: python

    customers = Table('customers')
    q = Query.from_(customers).select(
        customers.id, customers.fname, customers.lname, customers.phone
    ).where(
        (customers.age >= 18) & (customers.lname == 'Mustermann')
    )

.. code-block:: sql

    SELECT id,fname,lname,phone FROM customers WHERE age>=18 AND lname='Mustermann'

OR

.. code-block:: python

    customers = Table('customers')
    q = Query.from_(customers).select(
        customers.id, customers.fname, customers.lname, customers.phone
    ).where(
        (customers.age >= 18) | (customers.lname == 'Mustermann')
    )

.. code-block:: sql

    SELECT id,fname,lname,phone FROM customers WHERE age>=18 OR lname='Mustermann'

XOR

.. code-block:: python

 customers = Table('customers')
 q = Query.from_(customers).select(
     customers.id, customers.fname, customers.lname, customers.phone
 ).where(
     (customers.age >= 18) ^ customers.is_registered
 )

.. code-block:: sql

    SELECT id,fname,lname,phone FROM customers WHERE age>=18 XOR is_registered

Convenience Methods
"""""""""""""""""""

In the `Criterion` class, there are the static methods `any` and `all` that allow building chains AND and OR expressions with a list of terms.

.. code-block:: python

    from pypika import Criterion

    customers = Table('customers')
    q = Query.from_(customers).select(
        customers.id,
        customers.fname
    ).where(
        Criterion.all([
            customers.is_registered,
            customers.age >= 18,
            customers.lname == "Jones",
        ])
    )

.. code-block:: sql

    SELECT id,fname FROM customers WHERE is_registered AND age>=18 AND lname = "Jones"

Grouping and Aggregating
""""""""""""""""""""""""

Grouping allows for aggregated results and works similar to ``SELECT`` clauses.

.. code-block:: python

    from pypika import functions as fn

    customers = Table('customers')
    q = Query \
        .from_(customers) \
        .where(customers.age >= 18) \
        .groupby(customers.id) \
        .select(customers.id, fn.Sum(customers.revenue))

.. code-block:: sql

    SELECT id,SUM("revenue") FROM "customers" WHERE "age">=18 GROUP BY "id"

After adding a ``GROUP BY`` clause to a query, the ``HAVING`` clause becomes available.  The method
``Query.having()`` takes a ``Criterion`` parameter similar to the method ``Query.where()``.

.. code-block:: python

    from pypika import functions as fn

    payments = Table('payments')
    q = Query \
        .from_(payments) \
        .where(payments.transacted[date(2015, 1, 1):date(2016, 1, 1)]) \
        .groupby(payments.customer_id) \
        .having(fn.Sum(payments.total) >= 1000) \
        .select(payments.customer_id, fn.Sum(payments.total))

.. code-block:: sql

    SELECT customer_id,SUM(total) FROM payments
    WHERE transacted BETWEEN '2015-01-01' AND '2016-01-01'
    GROUP BY customer_id HAVING SUM(total)>=1000

Joining Tables and Subqueries
"""""""""""""""""""""""""""""

Tables and subqueries can be joined to any query using the ``Query.join()`` method.  Joins can be performed with either
a ``USING`` or ``ON`` clauses.  The ``USING`` clause can be used when both tables/subqueries contain the same field and
the ``ON`` clause can be used with a criterion. To perform a join, ``...join()`` can be chained but then must be
followed immediately by ``...on(<criterion>)`` or ``...using(*field)``.

Join Types
~~~~~~~~~~

All join types are supported by |Brand|.

.. code-block:: python

    Query \
        .from_(base_table)
        ...
        .join(join_table, JoinType.left)
        ...

.. code-block:: python

    Query \
        .from_(base_table)
        ...
        .left_join(join_table) \
        .left_outer_join(join_table) \
        .right_join(join_table) \
        .right_outer_join(join_table) \
        .inner_join(join_table) \
        .outer_join(join_table) \
        .full_outer_join(join_table) \
        .cross_join(join_table) \
        .hash_join(join_table) \
        ...

See the list of join types here ``pypika.enums.JoinTypes``

Example of a join using `ON`
~~~~~~~~~~~~~~~~~~~~~~~~~~~~

.. code-block:: python

    history, customers = Tables('history', 'customers')
    q = Query \
        .from_(history) \
        .join(customers) \
        .on(history.customer_id == customers.id) \
        .select(history.star) \
        .where(customers.id == 5)

.. code-block:: sql

    SELECT "history".* FROM "history" JOIN "customers" ON "history"."customer_id"="customers"."id" WHERE "customers"."id"=5

As a shortcut, the ``Query.join().on_field()`` function is provided for joining the (first) table in the ``FROM`` clause
with the joined table when the field name(s) are the same in both tables.

Example of a join using `ON`
~~~~~~~~~~~~~~~~~~~~~~~~~~~~

.. code-block:: python

    history, customers = Tables('history', 'customers')
    q = Query \
        .from_(history) \
        .join(customers) \
        .on_field('customer_id', 'group') \
        .select(history.star) \
        .where(customers.group == 'A')

.. code-block:: sql

    SELECT "history".* FROM "history" JOIN "customers" ON "history"."customer_id"="customers"."customer_id" AND "history"."group"="customers"."group" WHERE "customers"."group"='A'

Example of a join using `USING`
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

.. code-block:: python

    history, customers = Tables('history', 'customers')
    q = Query \
        .from_(history) \
        .join(customers) \
        .using('customer_id') \
        .select(history.star) \
        .where(customers.id == 5)

.. code-block:: sql

    SELECT "history".* FROM "history" JOIN "customers" USING "customer_id" WHERE "customers"."id"=5

Example of a correlated subquery in the `SELECT`
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

.. code-block:: python

    history, customers = Tables('history', 'customers')
    last_purchase_at = Query.from_(history).select(
        history.purchase_at
    ).where(history.customer_id==customers.customer_id).orderby(
        history.purchase_at, order=Order.desc
    ).limit(1)
    q = Query.from_(customers).select(
        customers.id, last_purchase_at.as_('last_purchase_at')
    )

.. code-block:: sql

    SELECT
      "id",
      (SELECT "history"."purchase_at"
       FROM "history"
       WHERE "history"."customer_id" = "customers"."customer_id"
       ORDER BY "history"."purchase_at" DESC
       LIMIT 1) "last_purchase_at"
    FROM "customers"

Unions
""""""

Both ``UNION`` and ``UNION ALL`` are supported. ``UNION DISTINCT`` is synonymous with "UNION`` so |Brand| does not
provide a separate function for it.  Unions require that queries have the same number of ``SELECT`` clauses so
trying to cast a unioned query to string will throw a ``SetOperationException`` if the column sizes are mismatched.

To create a union query, use either the ``Query.union()`` method or `+` operator with two query instances. For a
union all, use ``Query.union_all()`` or the `*` operator.

.. code-block:: python

    provider_a, provider_b = Tables('provider_a', 'provider_b')
    q = Query.from_(provider_a).select(
        provider_a.created_time, provider_a.foo, provider_a.bar
    ) + Query.from_(provider_b).select(
        provider_b.created_time, provider_b.fiz, provider_b.buz
    )

.. code-block:: sql

    SELECT "created_time","foo","bar" FROM "provider_a" UNION SELECT "created_time","fiz","buz" FROM "provider_b"

Intersect
"""""""""

``INTERSECT`` is supported. Intersects require that queries have the same number of ``SELECT`` clauses so
trying to cast a intersected query to string will throw a ``SetOperationException`` if the column sizes are mismatched.

To create a intersect query, use the ``Query.intersect()`` method.

.. code-block:: python

    provider_a, provider_b = Tables('provider_a', 'provider_b')
    q = Query.from_(provider_a).select(
        provider_a.created_time, provider_a.foo, provider_a.bar
    )
    r = Query.from_(provider_b).select(
        provider_b.created_time, provider_b.fiz, provider_b.buz
    )
    intersected_query = q.intersect(r)

.. code-block:: sql

    SELECT "created_time","foo","bar" FROM "provider_a" INTERSECT SELECT "created_time","fiz","buz" FROM "provider_b"

Minus
"""""

``MINUS`` is supported. Minus require that queries have the same number of ``SELECT`` clauses so
trying to cast a minus query to string will throw a ``SetOperationException`` if the column sizes are mismatched.

To create a minus query, use either the ``Query.minus()`` method or `-` operator with two query instances.

.. code-block:: python

    provider_a, provider_b = Tables('provider_a', 'provider_b')
    q = Query.from_(provider_a).select(
        provider_a.created_time, provider_a.foo, provider_a.bar
    )
    r = Query.from_(provider_b).select(
        provider_b.created_time, provider_b.fiz, provider_b.buz
    )
    minus_query = q.minus(r)

    (or)

    minus_query = Query.from_(provider_a).select(
        provider_a.created_time, provider_a.foo, provider_a.bar
    ) - Query.from_(provider_b).select(
        provider_b.created_time, provider_b.fiz, provider_b.buz
    )

.. code-block:: sql

    SELECT "created_time","foo","bar" FROM "provider_a" MINUS SELECT "created_time","fiz","buz" FROM "provider_b"

EXCEPT
""""""

``EXCEPT`` is supported. Minus require that queries have the same number of ``SELECT`` clauses so
trying to cast a except query to string will throw a ``SetOperationException`` if the column sizes are mismatched.

To create a except query, use the ``Query.except_of()`` method.

.. code-block:: python

    provider_a, provider_b = Tables('provider_a', 'provider_b')
    q = Query.from_(provider_a).select(
        provider_a.created_time, provider_a.foo, provider_a.bar
    )
    r = Query.from_(provider_b).select(
        provider_b.created_time, provider_b.fiz, provider_b.buz
    )
    minus_query = q.except_of(r)

.. code-block:: sql

    SELECT "created_time","foo","bar" FROM "provider_a" EXCEPT SELECT "created_time","fiz","buz" FROM "provider_b"

Date, Time, and Intervals
"""""""""""""""""""""""""

Using ``pypika.Interval``, queries can be constructed with date arithmetic.  Any combination of intervals can be
used except for weeks and quarters, which must be used separately and will ignore any other values if selected.

.. code-block:: python

    from pypika import functions as fn

    fruits = Tables('fruits')
    q = Query.from_(fruits) \
        .select(fruits.id, fruits.name) \
        .where(fruits.harvest_date + Interval(months=1) < fn.Now())

.. code-block:: sql

    SELECT id,name FROM fruits WHERE harvest_date+INTERVAL 1 MONTH<NOW()

Tuples
""""""

Tuples are supported through the class ``pypika.Tuple`` but also through the native python tuple wherever possible.
Tuples can be used with ``pypika.Criterion`` in **WHERE** clauses for pairwise comparisons.

.. code-block:: python

    from pypika import Query, Tuple

    q = Query.from_(self.table_abc) \
        .select(self.table_abc.foo, self.table_abc.bar) \
        .where(Tuple(self.table_abc.foo, self.table_abc.bar) == Tuple(1, 2))

.. code-block:: sql

    SELECT "foo","bar" FROM "abc" WHERE ("foo","bar")=(1,2)

Using ``pypika.Tuple`` on both sides of the comparison is redundant and |Brand| supports native python tuples.

.. code-block:: python

    from pypika import Query, Tuple

    q = Query.from_(self.table_abc) \
        .select(self.table_abc.foo, self.table_abc.bar) \
        .where(Tuple(self.table_abc.foo, self.table_abc.bar) == (1, 2))

.. code-block:: sql

    SELECT "foo","bar" FROM "abc" WHERE ("foo","bar")=(1,2)

Tuples can be used in **IN** clauses.

.. code-block:: python

    Query.from_(self.table_abc) \
            .select(self.table_abc.foo, self.table_abc.bar) \
            .where(Tuple(self.table_abc.foo, self.table_abc.bar).isin([(1, 1), (2, 2), (3, 3)]))

.. code-block:: sql

    SELECT "foo","bar" FROM "abc" WHERE ("foo","bar") IN ((1,1),(2,2),(3,3))

Strings Functions
"""""""""""""""""

There are several string operations and function wrappers included in |Brand|.  Function wrappers can be found in the
``pypika.functions`` package.  In addition, `LIKE` and `REGEX` queries are supported as well.

.. code-block:: python

    from pypika import functions as fn

    customers = Tables('customers')
    q = Query.from_(customers).select(
        customers.id,
        customers.fname,
        customers.lname,
    ).where(
        customers.lname.like('Mc%')
    )

.. code-block:: sql

    SELECT id,fname,lname FROM customers WHERE lname LIKE 'Mc%'

.. code-block:: python

    from pypika import functions as fn

    customers = Tables('customers')
    q = Query.from_(customers).select(
        customers.id,
        customers.fname,
        customers.lname,
    ).where(
        customers.lname.regex(r'^[abc][a-zA-Z]+&')
    )

.. code-block:: sql

    SELECT id,fname,lname FROM customers WHERE lname REGEX '^[abc][a-zA-Z]+&';

.. code-block:: python

    from pypika import functions as fn

    customers = Tables('customers')
    q = Query.from_(customers).select(
        customers.id,
        fn.Concat(customers.fname, ' ', customers.lname).as_('full_name'),
    )

.. code-block:: sql

    SELECT id,CONCAT(fname, ' ', lname) full_name FROM customers

Custom Functions
""""""""""""""""

Custom Functions allows us to use any function on queries, as some functions are not covered by PyPika as default, we can appeal
to Custom functions.

.. code-block:: python

    from pypika import CustomFunction

    customers = Tables('customers')
    DateDiff = CustomFunction('DATE_DIFF', ['interval', 'start_date', 'end_date'])

    q = Query.from_(customers).select(
        customers.id,
        customers.fname,
        customers.lname,
        DateDiff('day', customers.created_date, customers.updated_date)
    )

.. code-block:: sql

    SELECT id,fname,lname,DATE_DIFF('day',created_date,updated_date) FROM customers

Case Statements
"""""""""""""""

Case statements allow fow a number of conditions to be checked sequentially and return a value for the first condition
met or otherwise a default value.  The Case object can be used to chain conditions together along with their output
using the ``when`` method and to set the default value using ``else_``.

.. code-block:: python

    from pypika import Case, functions as fn

    customers = Tables('customers')
    q = Query.from_(customers).select(
        customers.id,
        Case()
           .when(customers.fname == "Tom", "It was Tom")
           .when(customers.fname == "John", "It was John")
           .else_("It was someone else.").as_('who_was_it')
    )

.. code-block:: sql

    SELECT "id",CASE WHEN "fname"='Tom' THEN 'It was Tom' WHEN "fname"='John' THEN 'It was John' ELSE 'It was someone else.' END "who_was_it" FROM "customers"

With Clause
"""""""""""""""

With clause allows give a sub-query block a name, which can be referenced in several places within the main SQL query.
The SQL WITH clause is basically a drop-in replacement to the normal sub-query.

.. code-block:: python

    from pypika import Table, AliasedQuery, Query

    customers = Table('customers')

    sub_query = (Query
                .from_(customers)
                .select('*'))

    test_query = (Query
                .with_(sub_query, "an_alias")
                .from_(AliasedQuery("an_alias"))
                .select('*'))

You can use as much as `.with_()` as you want.

.. code-block:: sql

    WITH an_alias AS (SELECT * FROM "customers") SELECT * FROM an_alias

Inserting Data
^^^^^^^^^^^^^^

Data can be inserted into tables either by providing the values in the query or by selecting them through another query.

By default, data can be inserted by providing values for all columns in the order that they are defined in the table.

Insert with values
""""""""""""""""""

.. code-block:: python

    customers = Table('customers')

    q = Query.into(customers).insert(1, 'Jane', 'Doe', 'jane@example.com')

.. code-block:: sql

    INSERT INTO customers VALUES (1,'Jane','Doe','jane@example.com')

.. code-block:: python

    customers =  Table('customers')

    q = customers.insert(1, 'Jane', 'Doe', 'jane@example.com')

.. code-block:: sql

    INSERT INTO customers VALUES (1,'Jane','Doe','jane@example.com')

Multiple rows of data can be inserted either by chaining the ``insert`` function or passing multiple tuples as args.

.. code-block:: python

    customers = Table('customers')

    q = Query.into(customers).insert(1, 'Jane', 'Doe', 'jane@example.com').insert(2, 'John', 'Doe', 'john@example.com')

.. code-block:: python

    customers = Table('customers')

    q = Query.into(customers).insert((1, 'Jane', 'Doe', 'jane@example.com'),
                                     (2, 'John', 'Doe', 'john@example.com'))

Insert with constraint violation handling
"""""""""""""""""""""""""""""""""""""""""

MySQL
~~~~~

.. code-block:: python

    customers = Table('customers')

    q = MySQLQuery.into(customers) \
        .insert(1, 'Jane', 'Doe', 'jane@example.com') \
        .on_duplicate_key_ignore())

.. code-block:: sql

    INSERT INTO `customers` VALUES (1,'Jane','Doe','jane@example.com') ON DUPLICATE KEY IGNORE

.. code-block:: python

    customers = Table('customers')

    q = MySQLQuery.into(customers) \
        .insert(1, 'Jane', 'Doe', 'jane@example.com') \
        .on_duplicate_key_update(customers.email, Values(customers.email))

.. code-block:: sql

    INSERT INTO `customers` VALUES (1,'Jane','Doe','jane@example.com') ON DUPLICATE KEY UPDATE `email`=VALUES(`email`)

``.on_duplicate_key_update`` works similar to ``.set`` for updating rows, additionally it provides the ``Values``
wrapper to update to the value specified in the ``INSERT`` clause.

PostgreSQL
~~~~~~~~~~

.. code-block:: python

    customers = Table('customers')

    q = PostgreSQLQuery.into(customers) \
        .insert(1, 'Jane', 'Doe', 'jane@example.com') \
        .on_conflict(customers.email) \
        .do_nothing()

.. code-block:: sql

    INSERT INTO "customers" VALUES (1,'Jane','Doe','jane@example.com') ON CONFLICT ("email") DO NOTHING

.. code-block:: python

    customers = Table('customers')

    q = PostgreSQLQuery.into(customers) \
        .insert(1, 'Jane', 'Doe', 'jane@example.com') \
        .on_conflict(customers.email) \
        .do_update(customers.email, 'bob@example.com')

.. code-block:: sql

    INSERT INTO "customers" VALUES (1,'Jane','Doe','jane@example.com') ON CONFLICT ("email") DO UPDATE SET "email"='bob@example.com'

Insert from a SELECT Sub-query
""""""""""""""""""""""""""""""

.. code-block:: sql

    INSERT INTO "customers" VALUES (1,'Jane','Doe','jane@example.com'),(2,'John','Doe','john@example.com')

To specify the columns and the order, use the ``columns`` function.

.. code-block:: python

    customers = Table('customers')

    q = Query.into(customers).columns('id', 'fname', 'lname').insert(1, 'Jane', 'Doe')

.. code-block:: sql

    INSERT INTO customers (id,fname,lname) VALUES (1,'Jane','Doe','jane@example.com')

Inserting data with a query works the same as querying data with the additional call to the ``into`` method in the
builder chain.

.. code-block:: python

    customers, customers_backup = Tables('customers', 'customers_backup')

    q = Query.into(customers_backup).from_(customers).select('*')

.. code-block:: sql

    INSERT INTO customers_backup SELECT * FROM customers

.. code-block:: python

    customers, customers_backup = Tables('customers', 'customers_backup')

    q = Query.into(customers_backup).columns('id', 'fname', 'lname')
        .from_(customers).select(customers.id, customers.fname, customers.lname)

.. code-block:: sql

    INSERT INTO customers_backup SELECT "id", "fname", "lname" FROM customers

The syntax for joining tables is the same as when selecting data

.. code-block:: python

    customers, orders, orders_backup = Tables('customers', 'orders', 'orders_backup')

    q = Query.into(orders_backup).columns('id', 'address', 'customer_fname', 'customer_lname')
        .from_(customers)
        .join(orders).on(orders.customer_id == customers.id)
        .select(orders.id, customers.fname, customers.lname)

.. code-block:: sql

   INSERT INTO "orders_backup" ("id","address","customer_fname","customer_lname")
   SELECT "orders"."id","customers"."fname","customers"."lname" FROM "customers"
   JOIN "orders" ON "orders"."customer_id"="customers"."id"

Updating Data
^^^^^^^^^^^^^^
PyPika allows update queries to be constructed with or without where clauses.

.. code-block:: python

    customers = Table('customers')

    Query.update(customers).set(customers.last_login, '2017-01-01 10:00:00')

    Query.update(customers).set(customers.lname, 'smith').where(customers.id == 10)

.. code-block:: sql

    UPDATE "customers" SET "last_login"='2017-01-01 10:00:00'

    UPDATE "customers" SET "lname"='smith' WHERE "id"=10

The syntax for joining tables is the same as when selecting data

.. code-block:: python

    customers, profiles = Tables('customers', 'profiles')

    Query.update(customers)
         .join(profiles).on(profiles.customer_id == customers.id)
         .set(customers.lname, profiles.lname)

.. code-block:: sql

   UPDATE "customers"
   JOIN "profiles" ON "profiles"."customer_id"="customers"."id"
   SET "customers"."lname"="profiles"."lname"

Using ``pypika.Table`` alias to perform the update

.. code-block:: python

    customers = Table('customers')

    customers.update()
            .set(customers.lname, 'smith')
            .where(customers.id == 10)

.. code-block:: sql

    UPDATE "customers" SET "lname"='smith' WHERE "id"=10

Using ``limit`` for performing update

.. code-block:: python

    customers = Table('customers')

    customers.update()
            .set(customers.lname, 'smith')
            .limit(2)

.. code-block:: sql

    UPDATE "customers" SET "lname"='smith' LIMIT 2

Parametrized Queries
^^^^^^^^^^^^^^^^^^^^

PyPika allows you to use ``Parameter(str)`` term as a placeholder for parametrized queries.

.. code-block:: python

    customers = Table('customers')

    q = Query.into(customers).columns('id', 'fname', 'lname')
        .insert(Parameter(':1'), Parameter(':2'), Parameter(':3'))

.. code-block:: sql

    INSERT INTO customers (id,fname,lname) VALUES (:1,:2,:3)

This allows you to build prepared statements, and/or avoid SQL-injection related risks.

Due to the mix of syntax for parameters, depending on connector/driver, it is required that you specify the
parameter token explicitly or use one of the specialized Parameter types per [PEP-0249](https://www.python.org/dev/peps/pep-0249/#paramstyle):
``QmarkParameter()``, ``NumericParameter(int)``,  ``NamedParameter(str)``, ``FormatParameter()``, ``PyformatParameter(str)``

An example of some common SQL parameter styles used in Python drivers are:

PostgreSQL:
    ``$number`` OR ``%s`` + ``:name`` (depending on driver)
MySQL:
    ``%s``
SQLite:
    ``?``
Vertica:
    ``:name``
Oracle:
    ``:number`` + ``:name``
MSSQL:
    ``%(name)s`` OR ``:name`` + ``:number`` (depending on driver)

You can find out what parameter style is needed for DBAPI compliant drivers here: https://www.python.org/dev/peps/pep-0249/#paramstyle or in the DB driver documentation.

Temporal support
^^^^^^^^^^^^^^^^

Temporal criteria can be added to the tables.

Select
""""""

Here is a select using system time.

.. code-block:: python

    t = Table("abc")
    q = Query.from_(t.for_(SYSTEM_TIME.as_of('2020-01-01'))).select("*")

This produces:

.. code-block:: sql

    SELECT * FROM "abc" FOR SYSTEM_TIME AS OF '2020-01-01'

You can also use between.

.. code-block:: python

    t = Table("abc")
    q = Query.from_(
        t.for_(SYSTEM_TIME.between('2020-01-01', '2020-02-01'))
    ).select("*")

This produces:

.. code-block:: sql

    SELECT * FROM "abc" FOR SYSTEM_TIME BETWEEN '2020-01-01' AND '2020-02-01'

You can also use a period range.

.. code-block:: python

    t = Table("abc")
    q = Query.from_(
        t.for_(SYSTEM_TIME.from_to('2020-01-01', '2020-02-01'))
    ).select("*")

This produces:

.. code-block:: sql

    SELECT * FROM "abc" FOR SYSTEM_TIME FROM '2020-01-01' TO '2020-02-01'

Finally you can select for all times:

.. code-block:: python

    t = Table("abc")
    q = Query.from_(t.for_(SYSTEM_TIME.all_())).select("*")

This produces:

.. code-block:: sql

    SELECT * FROM "abc" FOR SYSTEM_TIME ALL

A user defined period can also be used in the following manner.

.. code-block:: python

    t = Table("abc")
    q = Query.from_(
        t.for_(t.valid_period.between('2020-01-01', '2020-02-01'))
    ).select("*")

This produces:

.. code-block:: sql

    SELECT * FROM "abc" FOR "valid_period" BETWEEN '2020-01-01' AND '2020-02-01'

Joins
"""""

With joins, when the table object is used when specifying columns, it is
important to use the table from which the temporal constraint was generated.
This is because `Table("abc")` is not the same table as `Table("abc").for_(...)`.
The following example demonstrates this.

.. code-block:: python

    t0 = Table("abc").for_(SYSTEM_TIME.as_of('2020-01-01'))
    t1 = Table("efg").for_(SYSTEM_TIME.as_of('2020-01-01'))
    query = (
        Query.from_(t0)
        .join(t1)
        .on(t0.foo == t1.bar)
        .select("*")
    )

This produces:

.. code-block:: sql

    SELECT * FROM "abc" FOR SYSTEM_TIME AS OF '2020-01-01'
    JOIN "efg" FOR SYSTEM_TIME AS OF '2020-01-01'
    ON "abc"."foo"="efg"."bar"

Update & Deletes
""""""""""""""""

An update can be written as follows:

.. code-block:: python

    t = Table("abc")
    q = Query.update(
        t.for_portion(
            SYSTEM_TIME.from_to('2020-01-01', '2020-02-01')
        )
    ).set("foo", "bar")

This produces:

.. code-block:: sql

    UPDATE "abc"
    FOR PORTION OF SYSTEM_TIME FROM '2020-01-01' TO '2020-02-01'
    SET "foo"='bar'

Here is a delete:

.. code-block:: python

    t = Table("abc")
    q = Query.from_(
        t.for_portion(t.valid_period.from_to('2020-01-01', '2020-02-01'))
    ).delete()

This produces:

.. code-block:: sql

    DELETE FROM "abc"
    FOR PORTION OF "valid_period" FROM '2020-01-01' TO '2020-02-01'

Creating Tables
^^^^^^^^^^^^^^^

The entry point for creating tables is ``pypika.Query.create_table``, which is used with the class ``pypika.Column``.
As with selecting data, first the table should be specified. This can be either a
string or a `pypika.Table`. Then the columns, and constraints. Here's an example
that demonstrates much of the functionality.

.. code-block:: python

    stmt = Query \
        .create_table("person") \
        .columns(
            Column("id", "INT", nullable=False),
            Column("first_name", "VARCHAR(100)", nullable=False),
            Column("last_name", "VARCHAR(100)", nullable=False),
            Column("phone_number", "VARCHAR(20)", nullable=True),
            Column("status", "VARCHAR(20)", nullable=False, default=ValueWrapper("NEW")),
            Column("date_of_birth", "DATETIME")) \
        .unique("last_name", "first_name") \
        .primary_key("id")

This produces:

.. code-block:: sql

    CREATE TABLE "person" (
        "id" INT NOT NULL,
        "first_name" VARCHAR(100) NOT NULL,
        "last_name" VARCHAR(100) NOT NULL,
        "phone_number" VARCHAR(20) NULL,
        "status" VARCHAR(20) NOT NULL DEFAULT 'NEW',
        "date_of_birth" DATETIME,
        UNIQUE ("last_name","first_name"),
        PRIMARY KEY ("id")
    )

There is also support for creating a table from a query.

.. code-block:: python

    stmt = Query.create_table("names").as_select(
        Query.from_("person").select("last_name", "first_name")
    )

This produces:

.. code-block:: sql

        CREATE TABLE "names" AS (SELECT "last_name","first_name" FROM "person")

Managing Table Indices
^^^^^^^^^^^^^^^^^^^^^^

Create Indices
""""""""""""""""

The entry point for creating indices is ``pypika.Query.create_index``.
An index name (as ``str``) or a ``pypika.terms.Index`` a table (as ``str`` or ``pypika.Table``) and
columns (as ``pypika.Column``) must be specified.

.. code-block:: python

    my_index = Index("my_index")
    person = Table("person")
    stmt = Query \
        .create_index(my_index) \
        .on(person) \
        .columns(person.first_name, person.last_name)

This produces:

.. code-block:: sql

    CREATE INDEX my_index
    ON person (first_name, last_name)

It is also possible to create a unique index

.. code-block:: python

    my_index = Index("my_index")
    person = Table("person")
    stmt = Query \
        .create_index(my_index) \
        .on(person) \
        .columns(person.first_name, person.last_name) \
        .unique()

This produces:

.. code-block:: sql

        CREATE UNIQUE INDEX my_index
        ON person (first_name, last_name)

It is also possible to create an index if it does not exist

.. code-block:: python

    my_index = Index("my_index")
    person = Table("person")
    stmt = Query \
        .create_index(my_index) \
        .on(person) \
        .columns(person.first_name, person.last_name) \
        .if_not_exists()

This produces:

.. code-block:: sql

        CREATE INDEX IF NOT EXISTS my_index
        ON person (first_name, last_name)

Drop Indices
""""""""""""""""

Then entry point for dropping indices is ``pypika.Query.drop_index``.
It takes either ``str`` or ``pypika.terms.Index`` as an argument.

.. code-block:: python

    my_index = Index("my_index")
    stmt = Query.drop_index(my_index)

This produces:

.. code-block:: sql

    DROP INDEX my_index

It is also possible to drop an index if it exists

.. code-block:: python

    my_index = Index("my_index")
    stmt = Query.drop_index(my_index).if_exists()

This produces:

.. code-block:: sql

    DROP INDEX IF EXISTS my_index

Chaining Functions
^^^^^^^^^^^^^^^^^^

The ``QueryBuilder.pipe`` method gives a more readable alternative while chaining functions.

.. code-block:: python

    # This
    (
        query
        .pipe(func1, *args)
        .pipe(func2, **kwargs)
        .pipe(func3)
    )

    # Is equivalent to this
    func3(func2(func1(query, *args), **kwargs))

Or for a more concrete example:

.. code-block:: python

    from pypika import Field, Query, functions as fn
    from pypika.queries import QueryBuilder

    def filter_days(query: QueryBuilder, col, num_days: int) -> QueryBuilder:
        if isinstance(col, str):
            col = Field(col)

        return query.where(col > fn.Now() - num_days)

    def count_groups(query: QueryBuilder, *groups) -> QueryBuilder:
        return query.groupby(*groups).select(*groups, fn.Count("*").as_("n_rows"))

    base_query = Query.from_("table")

    query = (
        base_query
        .pipe(filter_days, "date", num_days=7)
        .pipe(count_groups, "col1", "col2")
    )

This produces:

.. code-block:: sql

    SELECT "col1","col2",COUNT(*) n_rows
    FROM "table"
    WHERE "date">NOW()-7
    GROUP BY "col1","col2"

.. _tutorial_end:

.. _contributing_start:

Contributing
------------

We welcome community contributions to |Brand|. Please see the `contributing guide <6_contributing.html>`_ to more info.

.. _contributing_end:

.. _license_start:

License
-------

Copyright 2020 KAYAK Germany, GmbH

Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.

Crafted with ♥ in Berlin.

.. _license_end:

.. _appendix_start:

.. |Brand| replace:: *PyPika*

.. _appendix_end:

.. _available_badges_start:

.. |BuildStatus| image:: https://github.com/kayak/pypika/workflows/Unit%20Tests/badge.svg
   :target: https://github.com/kayak/pypika/actions
.. |CoverageStatus| image:: https://coveralls.io/repos/kayak/pypika/badge.svg?branch=master
   :target: https://coveralls.io/github/kayak/pypika?branch=master
.. |Codacy| image:: https://api.codacy.com/project/badge/Grade/6d7e44e5628b4839a23da0bd82eaafcf
   :target: https://www.codacy.com/app/twheys/pypika
.. |Docs| image:: https://readthedocs.org/projects/pypika/badge/?version=latest
   :target: http://pypika.readthedocs.io/en/latest/
.. |PyPi| image:: https://img.shields.io/pypi/v/pypika.svg?style=flat
   :target: https://pypi.python.org/pypi/pypika
.. |License| image:: https://img.shields.io/hexpm/l/plug.svg?maxAge=2592000
   :target: http://www.apache.org/licenses/LICENSE-2.0

.. _available_badges_end:


django-axes
===========

.. image:: https://jazzband.co/static/img/badge.svg
   :target: https://jazzband.co/
   :alt: Jazzband

.. image:: https://img.shields.io/github/stars/jazzband/django-axes.svg?label=Stars&style=socialcA
   :target: https://github.com/jazzband/django-axes
   :alt: GitHub

.. image:: https://img.shields.io/pypi/v/django-axes.svg
   :target: https://pypi.org/project/django-axes/
   :alt: PyPI release

.. image:: https://img.shields.io/pypi/pyversions/django-axes.svg
   :target: https://pypi.org/project/django-axes/
   :alt: Supported Python versions

.. image:: https://img.shields.io/pypi/djversions/django-axes.svg
   :target: https://pypi.org/project/django-axes/
   :alt: Supported Django versions

.. image:: https://img.shields.io/readthedocs/django-axes.svg
   :target: https://django-axes.readthedocs.io/
   :alt: Documentation

.. image:: https://github.com/jazzband/django-axes/workflows/Test/badge.svg
   :target: https://github.com/jazzband/django-axes/actions
   :alt: GitHub Actions

.. image:: https://codecov.io/gh/jazzband/django-axes/branch/master/graph/badge.svg
   :target: https://codecov.io/gh/jazzband/django-axes
   :alt: Coverage

Axes is a Django plugin for keeping track of suspicious
login attempts for your Django based website
and implementing simple brute-force attack blocking.

The name is sort of a geeky pun, since it can be interpreted as:

* ``access``, as in monitoring access attempts, or
* ``axes``, as in tools you can use to hack (generally on wood).

Functionality
-------------

Axes records login attempts to your Django powered site and prevents attackers
from attempting further logins to your site when they exceed the configured attempt limit.

Axes can track the attempts and persist them in the database indefinitely,
or alternatively use a fast and DDoS resistant cache implementation.

Axes can be configured to monitor login attempts by
IP address, username, user agent, or their combinations.

Axes supports cool off periods, IP address whitelisting and blacklisting,
user account whitelisting, and other features for Django access management.

Documentation
-------------

For more information on installation and configuration see the documentation at:

https://django-axes.readthedocs.io/

Issues
------

If you have questions or have trouble using the app please file a bug report at:

https://github.com/jazzband/django-axes/issues

Contributing
------------

See `CONTRIBUTING <CONTRIBUTING.rst>`__.


    Cartopy is a Python package designed to make drawing maps for
    data analysis and visualisation easy.

# Table of contents

- [Overview](#overview)
- [Get in touch](#get-in-touch)
- [License and copyright](#license-and-copyright)

## Overview

Cartopy is a Python package designed to make drawing maps for data
analysis and visualisation easy.

It features:

- object oriented projection definitions
- point, line, polygon and image transformations between projections
- integration to expose advanced mapping in Matplotlib with a simple and
  intuitive interface
- powerful vector data handling by integrating shapefile reading with Shapely
  capabilities

Documentation can be found at <https://scitools.org.uk/cartopy/docs/latest/>.

## Get in touch

- Ask usage questions on
  [StackOverflow](https://stackoverflow.com/questions/tagged/cartopy).
- For less well defined questions, ideas, general discussion or announcements of
  related projects use the
  [Cartopy category on Matplotlib's Discourse](https://discourse.matplotlib.org/c/3rdparty/cartopy/19).
- Report bugs, suggest features or view the source code on
  [GitHub](https://github.com/SciTools/cartopy).
- To chat with developers and other users you can use the
  [Gitter Chat](https://gitter.im/SciTools/cartopy).

## Credits, copyright and license

Cartopy is developed collaboratively under the SciTools umberella.

A full list of codecontributors ("Cartopy contributors") can be found at
https://github.com/SciTools/cartopy/graphs/contributors.

Code is just one of many ways of positively contributing to Cartopy, please see
our [contributing guide](.github/CONTRIBUTING.md) for more details on how
you can get involved.

Cartopy is released under the 3-Clause BSD license with a shared copyright model.
See [LICENSE](LICENSE) for full terms.

The [Met Office](https://metoffice.gov.uk) has made a significant
contribution to the development, maintenance and support of this library.
All Met Office contributions are copyright on behalf of the British Crown.


![image](https://github.com/ultrabug/py3status/blob/master/logo/logotype-horizontal.png)

# py3status

![CI](https://github.com/ultrabug/py3status/workflows/CI/badge.svg)

**py3status** is an extensible i3status wrapper written in Python.

Using py3status, you can take control of your i3bar easily by:

-   using one of the available
    [modules](https://ultrabug.github.io/py3status/user-guide/modules/)
    shipped with py3status
-   grouping multiple modules and automatically or manually cycle their
    display
-   writing your own modules and have their output displayed on your bar
-   handling click events on your i3bar and play with them in no time
-   seeing your clock tick every second whatever your i3status interval

**No extra configuration file needed**, just install & enjoy!

## About

You will love py3status if you're using [i3wm](https://i3wm.org)
(or [sway](https://swaywm.org)) and are frustrated by the i3status
[limitations](https://faq.i3wm.org/question/459/external-scriptsprograms-in-i3status-without-loosing-colors/) on your i3bar such as:

-   you cannot hack into it easily
-   you want more than the built-in modules and their limited configuration
-   you cannot pipe the result of one of more scripts or commands in your bar easily

## Philosophy

-   **no added configuration file, use the standard i3status.conf**
-   **rely on i3status**' strengths and its **existing configuration** as much as possible
-   **be extensible**, it must be easy for users to add their own
    stuff/output by writing a simple python class which will be loaded
    and executed dynamically
-   **easily allow interactivity** with the i3bar
-   add some **built-in enhancement/transformation** of basic i3status modules output

We apply the [Zen of py3status](https://ultrabug.github.io/py3status/dev-guide/contributing/#zen-of-py3status) to improve this project and encourage everyone to read it!

## Documentation

Up-to-date [documentation](https://ultrabug.github.io/py3status/):

-   [Installation](https://ultrabug.github.io/py3status/user-guide/installation/)
-   [Getting started](https://ultrabug.github.io/py3status/getting-started/)
-   [Using Modules](https://ultrabug.github.io/py3status/user-guide/configuration/#loading-and-ordering-py3status-modules)
-   [Custom click events](https://ultrabug.github.io/py3status/user-guide/configuration/#custom-click-events)
-   [The py3-cmd command line](https://ultrabug.github.io/py3status/user-guide/remote-control/)
-   [Writing custom modules](https://ultrabug.github.io/py3status/dev-guide/writing-modules/)
-   [Contributing](https://ultrabug.github.io/py3status/dev-guide/contributing/)

Get help or share your ideas on IRC:

-   channel **\#py3status** on **OFTC**

## Installation

See the up to date and complete [installation instructions](https://ultrabug.github.io/py3status/user-guide/installation/) for your favorite distribution.

## Usage

In your i3 config file, simply switch from *i3status* to *py3status* in
your *status_command*:

    status_command py3status

Usually you have your own i3status configuration, just point to it:

    status_command py3status -c ~/.i3/i3status.conf

## Options

You can see the help of py3status by issuing \`py3status -h\`:

    usage: py3status [-h] [-b] [-c FILE] [-d] [-g] [-i PATH] [-l FILE] [-s]
                     [-t INT] [-m] [-u PATH] [-v] [--wm WINDOW_MANAGER]

    The agile, python-powered, i3status wrapper

    optional arguments:
      -h, --help            show this help message and exit
      -b, --dbus-notify     send notifications via dbus instead of i3-nagbar
                            (default: False)
      -c, --config FILE     load config (default: /home/alexys/.i3/i3status.conf)
      -d, --debug           enable debug logging in syslog and --log-file
                            (default: False)
      -i, --include PATH    append additional user-defined module paths (default:
                            None)
      -l, --log-file FILE   enable logging to FILE (default: None)
      -s, --standalone      run py3status without i3status (default: False)
      -t, --timeout INT     default module cache timeout in seconds (default: 60)
      -m, --disable-click-events
                            disable all click events (default: False)
      -u, --i3status PATH   specify i3status path (default: /usr/bin/i3status)
      -v, --version         show py3status version and exit (default: False)
      --wm WINDOW_MANAGER   specify window manager i3 or sway (default: i3)

## Available modules

[All modules](https://ultrabug.github.io/py3status/user-guide/modules/)
shipped with py3status are [configurable directly from your current i3status.conf](https://ultrabug.github.io/py3status/user-guide/configuration/#loading-and-ordering-py3status-modules)!

[Check them out](https://ultrabug.github.io/py3status/user-guide/modules/)
to see all the configuration options.

## Control from CLI

Just like i3status, you can force an update of your i3bar by sending a
SIGUSR1 signal to py3status. Note that this will also send a SIGUSR1
signal to i3status:

    killall -USR1 py3status

To refresh individual modules, the
[py3-cmd](https://ultrabug.github.io/py3status/user-guide/remote-control/)
utility can be used, e.g:

    py3-cmd refresh wifi


        Drop a star to support Aim ⭐

          Join Aim discord community

    An easy-to-use & supercharged open-source experiment tracker

  Aim logs your training runs and any AI Metadata, enables a beautiful UI to compare, observe them and an API to query them programmatically.

  [![Discord Server](https://dcbadge.vercel.app/api/server/zXq2NfVdtF?compact=true&style=flat)](https://community.aimstack.io/)
  [![Twitter Follow](https://img.shields.io/twitter/follow/aimstackio?style=social)](https://twitter.com/aimstackio)
  [![Medium](https://img.shields.io/badge/Medium-12100E?style=flat&logo=medium&logoColor=white)](https://medium.com/aimstack)

  [![Platform Support](https://img.shields.io/badge/platform-Linux%20%7C%20macOS-blue)]()
  [![PyPI - Python Version](https://img.shields.io/badge/python-%3E%3D%203.7-blue)](https://pypi.org/project/aim/)
  [![PyPI Package](https://img.shields.io/pypi/v/aim?color=yellow)](https://pypi.org/project/aim/)
  [![License](https://img.shields.io/badge/License-Apache%202.0-orange.svg)](https://opensource.org/licenses/Apache-2.0)
  [![PyPI Downloads](https://img.shields.io/pypi/dw/aim?color=green)](https://pypi.org/project/aim/)
  [![Issues](https://img.shields.io/github/issues/aimhubio/aim)](http://github.com/aimhubio/aim/issues)

  SEAMLESSLY INTEGRATES WITH:

  TRUSTED BY ML TEAMS FROM:

  AimStack offers enterprise support that's beyond core Aim. Contact via hello@aimstack.io e-mail.

---

  About •
  Demos •
  Ecosystem •
  Quick Start •
  Examples •
  Documentation •
  Community •
  Blog

---

# ℹ About

Aim is an open-source, self-hosted ML experiment tracking tool designed to handle 10,000s of training runs.

Aim provides a performant and beautiful UI for exploring and comparing training runs.
Additionally, its SDK enables programmatic access to tracked metadata — perfect for automations and Jupyter Notebook analysis.

  Aim's mission is to democratize AI dev tools 🎯

        Log Metadata Across Your ML Pipeline 💾
        Visualize & Compare Metadata via UI 📊

ML experiments and any metadata tracking

Integration with popular ML frameworks

Easy migration from other experiment trackers

Metadata visualization via Aim Explorers

Grouping and aggregation

Querying using Python expressions

        Run ML Trainings Effectively ⚡
        Organize Your Experiments 🗂

System info and resource usage tracking

Real-time alerting on training progress

Logging and configurable notifications

Detailed run information for easy debugging

Centralized dashboard for holistic view

Runs grouping with tags and experiments

# 🎬 Demos

Check out live Aim demos NOW to see it in action.

| [Machine translation experiments](https://play.aimstack.io/nmt/metrics?grouping=O-JTdCJTIyY29sb3IlMjI6JTVCJTIycnVuLnBhcmFtcy5ocGFyYW1zLm1heF9rJTIyJTVELCUyMnN0cm9rZSUyMjolNUIlNUQsJTIyY2hhcnQlMjI6JTVCJTIybmFtZSUyMiwlMjJjb250ZXh0LnN1YnNldCUyMiU1RCwlMjJyZXZlcnNlTW9kZSUyMjolN0IlMjJjb2xvciUyMjpmYWxzZSwlMjJzdHJva2UlMjI6ZmFsc2UsJTIyY2hhcnQlMjI6ZmFsc2UlN0QsJTIyaXNBcHBsaWVkJTIyOiU3QiUyMmNvbG9yJTIyOnRydWUsJTIyc3Ryb2tlJTIyOnRydWUsJTIyY2hhcnQlMjI6dHJ1ZSU3RCwlMjJwZXJzaXN0ZW5jZSUyMjolN0IlMjJjb2xvciUyMjpmYWxzZSwlMjJzdHJva2UlMjI6ZmFsc2UlN0QsJTIyc2VlZCUyMjolN0IlMjJjb2xvciUyMjoxMCwlMjJzdHJva2UlMjI6MTAlN0QsJTIycGFsZXR0ZUluZGV4JTIyOjAlN0Q&chart=O-JTdCJTIyaGlnaGxpZ2h0TW9kZSUyMjoyLCUyMmlnbm9yZU91dGxpZXJzJTIyOnRydWUsJTIyem9vbSUyMjolN0IlMjJhY3RpdmUlMjI6ZmFsc2UsJTIybW9kZSUyMjowLCUyMmhpc3RvcnklMjI6JTVCJTVEJTdELCUyMmF4ZXNTY2FsZVR5cGUlMjI6JTdCJTIyeEF4aXMlMjI6JTIybGluZWFyJTIyLCUyMnlBeGlzJTIyOiUyMmxpbmVhciUyMiU3RCwlMjJheGVzU2NhbGVSYW5nZSUyMjolN0IlMjJ5QXhpcyUyMjolN0IlN0QsJTIyeEF4aXMlMjI6JTdCJTdEJTdELCUyMnNtb290aGluZyUyMjolN0IlMjJhbGdvcml0aG0lMjI6JTIyRVhQT05FTlRJQUxfTU9WSU5HX0FWRVJBR0UlMjIsJTIyZmFjdG9yJTIyOjAsJTIyY3VydmVJbnRlcnBvbGF0aW9uJTIyOiUyMmN1cnZlTGluZWFyJTIyLCUyMmlzQXBwbGllZCUyMjpmYWxzZSU3RCwlMjJhbGlnbm1lbnRDb25maWclMjI6JTdCJTIybWV0cmljJTIyOiUyMiUyMiwlMjJ0eXBlJTIyOiUyMnN0ZXAlMjIlN0QsJTIyZGVuc2l0eVR5cGUlMjI6NTAwLCUyMmFnZ3JlZ2F0aW9uQ29uZmlnJTIyOiU3QiUyMm1ldGhvZHMlMjI6JTdCJTIyYXJlYSUyMjoxLCUyMmxpbmUlMjI6MCU3RCwlMjJpc0FwcGxpZWQlMjI6dHJ1ZSwlMjJpc0VuYWJsZWQlMjI6dHJ1ZSU3RCwlMjJ0b29sdGlwJTIyOiU3QiUyMmFwcGVhcmFuY2UlMjI6JTIyYXV0byUyMiwlMjJkaXNwbGF5JTIyOnRydWUsJTIyc2VsZWN0ZWRGaWVsZHMlMjI6JTVCJTVELCUyMnNlbGVjdGVkUGFyYW1zJTIyOiU1QiU1RCU3RCwlMjJsZWdlbmRzJTIyOiU3QiUyMmRpc3BsYXklMjI6dHJ1ZSwlMjJtb2RlJTIyOiUyMnBpbm5lZCUyMiU3RCwlMjJmb2N1c2VkU3RhdGUlMjI6JTdCJTIyYWN0aXZlJTIyOnRydWUsJTIya2V5JTIyOiUyMk8tSlRkQ0pUSXljblZ1U0dGemFDVXlNam9sTWpKa1lUTmtNV1UzSlRJeUxDVXlNbTFsZEhKcFkwNWhiV1VsTWpJNkpUSXlZbVZ6ZEY5c2IzTnpKVEl5TENVeU1uUnlZV05sUTI5dWRHVjRkQ1V5TWpvbE4wSWxNakp6ZFdKelpYUWxNakk2SlRJeWRtRnNKVEl5SlRkRUpUZEUlMjIsJTIyeFZhbHVlJTIyOjIxLCUyMnlWYWx1ZSUyMjozLjQ3Mzk5OTk3NzEsJTIyY2hhcnRJbmRleCUyMjowLCUyMnZpc0lkJTIyOiUyMjAlMjIlN0QlN0Q&select=O-JTdCJTIyb3B0aW9ucyUyMjolNUIlN0IlMjJsYWJlbCUyMjolMjJiZXN0X2xvc3MlMjIsJTIyZ3JvdXAlMjI6JTIyYmVzdF9sb3NzJTIyLCUyMnR5cGUlMjI6JTIybWV0cmljcyUyMiwlMjJjb2xvciUyMjolMjIjN0E0Q0UwJTIyLCUyMmtleSUyMjolMjJPLUpUZENKVEl5YldWMGNtbGpUbUZ0WlNVeU1qb2xNakppWlhOMFgyeHZjM01sTWpJc0pUSXlZMjl1ZEdWNGRFNWhiV1VsTWpJNkpUSXlKVEl5SlRkRSUyMiwlMjJ2YWx1ZSUyMjolN0IlMjJvcHRpb25fbmFtZSUyMjolMjJiZXN0X2xvc3MlMjIsJTIyY29udGV4dCUyMjpudWxsJTdEJTdELCU3QiUyMmxhYmVsJTIyOiUyMmJsZXUlMjIsJTIyZ3JvdXAlMjI6JTIyYmxldSUyMiwlMjJ0eXBlJTIyOiUyMm1ldHJpY3MlMjIsJTIyY29sb3IlMjI6JTIyIzNFNzJFNyUyMiwlMjJrZXklMjI6JTIyTy1KVGRDSlRJeWJXVjBjbWxqVG1GdFpTVXlNam9sTWpKaWJHVjFKVEl5TENVeU1tTnZiblJsZUhST1lXMWxKVEl5T2lVeU1pVXlNaVUzUkElMjIsJTIydmFsdWUlMjI6JTdCJTIyb3B0aW9uX25hbWUlMjI6JTIyYmxldSUyMiwlMjJjb250ZXh0JTIyOm51bGwlN0QlN0QlNUQsJTIycXVlcnklMjI6JTIycnVuLmhwYXJhbXMubGVhcm5pbmdfcmF0ZSUyMCUzRSUyMDAuMDAwMDElMjIsJTIyYWR2YW5jZWRNb2RlJTIyOmZhbHNlLCUyMmFkdmFuY2VkUXVlcnklMjI6JTIycnVuLmhwYXJhbXMubGVhcm5pbmdfcmF0ZSUyMCUzRSUyMDAuMDAwMDElMjBhbmQlMjAoKG1ldHJpYy5uYW1lJTIwPT0lMjAlNUMlMjJibGV1JTVDJTIyKSUyMG9yJTIwKG1ldHJpYy5uYW1lJTIwPT0lMjAlNUMlMjJiZXN0X2xvc3MlNUMlMjIpJTIwb3IlMjAobWV0cmljLm5hbWUlMjA9PSUyMCU1QyUyMmJzeiU1QyUyMiUyMGFuZCUyMG1ldHJpYy5jb250ZXh0LnN1YnNldCUyMD09JTIwJTVDJTIydHJhaW4lNUMlMjIpKSUyMiU3RA) | [lightweight-GAN experiments](https://play.aimstack.io/image-generation/images?grouping=O-JTdCJTIycm93JTIyOiU1QiU1RCwlMjJyZXZlcnNlTW9kZSUyMjolN0IlMjJyb3clMjI6ZmFsc2UsJTIyZ3JvdXAlMjI6ZmFsc2UlN0QsJTIyaXNBcHBsaWVkJTIyOiU3QiUyMnJvdyUyMjp0cnVlLCUyMmdyb3VwJTIyOnRydWUlN0QsJTIyZ3JvdXAlMjI6JTVCJTIyaW5kZXglMjIsJTIyc3RlcCUyMiU1RCU3RA&select=O-JTdCJTIyb3B0aW9ucyUyMjolNUIlN0IlMjJsYWJlbCUyMjolMjJnZW5lcmF0ZWQlMjIsJTIyZ3JvdXAlMjI6JTIyZ2VuZXJhdGVkJTIyLCUyMmNvbG9yJTIyOiUyMiMzRTcyRTclMjIsJTIya2V5JTIyOiUyMk8tSlRkQ0pUSXliV1YwY21salRtRnRaU1V5TWpvbE1qSm5aVzVsY21GMFpXUWxNaklzSlRJeVkyOXVkR1Y0ZEU1aGJXVWxNakk2SlRJeUpUSXlKVGRFJTIyLCUyMnZhbHVlJTIyOiU3QiUyMm9wdGlvbl9uYW1lJTIyOiUyMmdlbmVyYXRlZCUyMiwlMjJjb250ZXh0JTIyOm51bGwlN0QlN0QlNUQsJTIycXVlcnklMjI6JTIyaW1hZ2VzLmNvbnRleHQuaW50ZXJwb2xhdGVkJTIwYW5kJTIwcnVuLmhwYXJhbXMubmFtZSUyMD09JTIwJTVDJTIybWV0ZmFjZXMlNUMlMjIlMjIsJTIyYWR2YW5jZWRNb2RlJTIyOmZhbHNlLCUyMmFkdmFuY2VkUXVlcnklMjI6JTIyaW1hZ2VzLmNvbnRleHQuaW50ZXJwb2xhdGVkJTIwYW5kJTIwcnVuLmhwYXJhbXMubmFtZSUyMD09JTIwJTVDJTIybWV0ZmFjZXMlNUMlMjIlMjIlN0Q&images=O-JTdCJTIyaW5kZXhEZW5zaXR5JTIyOjcsJTIycmVjb3JkRGVuc2l0eSUyMjoxNSwlMjJ0b29sdGlwJTIyOiU3QiUyMmFwcGVhcmFuY2UlMjI6JTIyYXV0byUyMiwlMjJkaXNwbGF5JTIyOnRydWUsJTIyc2VsZWN0ZWRGaWVsZHMlMjI6JTVCJTVELCUyMnNlbGVjdGVkUGFyYW1zJTIyOiU1QiU1RCU3RCwlMjJhZGRpdGlvbmFsUHJvcGVydGllcyUyMjolN0IlMjJhbGlnbm1lbnRUeXBlJTIyOiUyMkhlaWdodCUyMiwlMjJtZWRpYUl0ZW1TaXplJTIyOjIzLCUyMmltYWdlUmVuZGVyaW5nJTIyOiUyMnBpeGVsYXRlZCUyMiwlMjJzdGFja2luZyUyMjpmYWxzZSU3RCwlMjJmb2N1c2VkU3RhdGUlMjI6JTdCJTIyYWN0aXZlJTIyOmZhbHNlLCUyMmtleSUyMjpudWxsJTdELCUyMnNvcnRGaWVsZHMlMjI6JTVCJTVELCUyMnNvcnRGaWVsZHNEaWN0JTIyOiU3QiU3RCwlMjJpbnB1dHNWYWxpZGF0aW9ucyUyMjolN0IlMjJpbmRleERlbnNpdHklMjI6dHJ1ZSwlMjJyZWNvcmREZW5zaXR5JTIyOnRydWUlN0QsJTIyY2FsY1JhbmdlcyUyMjpmYWxzZSwlMjJzdGVwUmFuZ2UlMjI6JTVCMSw0Njk1NiU1RCwlMjJpbmRleFJhbmdlJTIyOiU1QjAsNyU1RCwlMjJyZWNvcmRTbGljZSUyMjolNUIzMTE0Miw0Njk1NyU1RCwlMjJpbmRleFNsaWNlJTIyOiU1QjAsOCU1RCwlMjJpbWFnZVByb3BlcnRpZXMlMjI6JTdCJTIyYWxpZ25tZW50VHlwZSUyMjolMjJIZWlnaHQlMjIsJTIyaW1hZ2VTaXplJTIyOjI1LCUyMmltYWdlUmVuZGVyaW5nJTIyOiUyMnBpeGVsYXRlZCUyMiU3RCU3RA)|
|:---:|:---:|
| <a href="https://play.aimstack.io/nmt/metrics?grouping=O-JTdCJTIyY29sb3IlMjI6JTVCJTIycnVuLnBhcmFtcy5ocGFyYW1zLm1heF9rJTIyJTVELCUyMnN0cm9rZSUyMjolNUIlNUQsJTIyY2hhcnQlMjI6JTVCJTIybmFtZSUyMiwlMjJjb250ZXh0LnN1YnNldCUyMiU1RCwlMjJyZXZlcnNlTW9kZSUyMjolN0IlMjJjb2xvciUyMjpmYWxzZSwlMjJzdHJva2UlMjI6ZmFsc2UsJTIyY2hhcnQlMjI6ZmFsc2UlN0QsJTIyaXNBcHBsaWVkJTIyOiU3QiUyMmNvbG9yJTIyOnRydWUsJTIyc3Ryb2tlJTIyOnRydWUsJTIyY2hhcnQlMjI6dHJ1ZSU3RCwlMjJwZXJzaXN0ZW5jZSUyMjolN0IlMjJjb2xvciUyMjpmYWxzZSwlMjJzdHJva2UlMjI6ZmFsc2UlN0QsJTIyc2VlZCUyMjolN0IlMjJjb2xvciUyMjoxMCwlMjJzdHJva2UlMjI6MTAlN0QsJTIycGFsZXR0ZUluZGV4JTIyOjAlN0Q&chart=O-JTdCJTIyaGlnaGxpZ2h0TW9kZSUyMjoyLCUyMmlnbm9yZU91dGxpZXJzJTIyOnRydWUsJTIyem9vbSUyMjolN0IlMjJhY3RpdmUlMjI6ZmFsc2UsJTIybW9kZSUyMjowLCUyMmhpc3RvcnklMjI6JTVCJTVEJTdELCUyMmF4ZXNTY2FsZVR5cGUlMjI6JTdCJTIyeEF4aXMlMjI6JTIybGluZWFyJTIyLCUyMnlBeGlzJTIyOiUyMmxpbmVhciUyMiU3RCwlMjJheGVzU2NhbGVSYW5nZSUyMjolN0IlMjJ5QXhpcyUyMjolN0IlN0QsJTIyeEF4aXMlMjI6JTdCJTdEJTdELCUyMnNtb290aGluZyUyMjolN0IlMjJhbGdvcml0aG0lMjI6JTIyRVhQT05FTlRJQUxfTU9WSU5HX0FWRVJBR0UlMjIsJTIyZmFjdG9yJTIyOjAsJTIyY3VydmVJbnRlcnBvbGF0aW9uJTIyOiUyMmN1cnZlTGluZWFyJTIyLCUyMmlzQXBwbGllZCUyMjpmYWxzZSU3RCwlMjJhbGlnbm1lbnRDb25maWclMjI6JTdCJTIybWV0cmljJTIyOiUyMiUyMiwlMjJ0eXBlJTIyOiUyMnN0ZXAlMjIlN0QsJTIyZGVuc2l0eVR5cGUlMjI6NTAwLCUyMmFnZ3JlZ2F0aW9uQ29uZmlnJTIyOiU3QiUyMm1ldGhvZHMlMjI6JTdCJTIyYXJlYSUyMjoxLCUyMmxpbmUlMjI6MCU3RCwlMjJpc0FwcGxpZWQlMjI6dHJ1ZSwlMjJpc0VuYWJsZWQlMjI6dHJ1ZSU3RCwlMjJ0b29sdGlwJTIyOiU3QiUyMmFwcGVhcmFuY2UlMjI6JTIyYXV0byUyMiwlMjJkaXNwbGF5JTIyOnRydWUsJTIyc2VsZWN0ZWRGaWVsZHMlMjI6JTVCJTVELCUyMnNlbGVjdGVkUGFyYW1zJTIyOiU1QiU1RCU3RCwlMjJsZWdlbmRzJTIyOiU3QiUyMmRpc3BsYXklMjI6dHJ1ZSwlMjJtb2RlJTIyOiUyMnBpbm5lZCUyMiU3RCwlMjJmb2N1c2VkU3RhdGUlMjI6JTdCJTIyYWN0aXZlJTIyOnRydWUsJTIya2V5JTIyOiUyMk8tSlRkQ0pUSXljblZ1U0dGemFDVXlNam9sTWpKa1lUTmtNV1UzSlRJeUxDVXlNbTFsZEhKcFkwNWhiV1VsTWpJNkpUSXlZbVZ6ZEY5c2IzTnpKVEl5TENVeU1uUnlZV05sUTI5dWRHVjRkQ1V5TWpvbE4wSWxNakp6ZFdKelpYUWxNakk2SlRJeWRtRnNKVEl5SlRkRUpUZEUlMjIsJTIyeFZhbHVlJTIyOjIxLCUyMnlWYWx1ZSUyMjozLjQ3Mzk5OTk3NzEsJTIyY2hhcnRJbmRleCUyMjowLCUyMnZpc0lkJTIyOiUyMjAlMjIlN0QlN0Q&select=O-JTdCJTIyb3B0aW9ucyUyMjolNUIlN0IlMjJsYWJlbCUyMjolMjJiZXN0X2xvc3MlMjIsJTIyZ3JvdXAlMjI6JTIyYmVzdF9sb3NzJTIyLCUyMnR5cGUlMjI6JTIybWV0cmljcyUyMiwlMjJjb2xvciUyMjolMjIjN0E0Q0UwJTIyLCUyMmtleSUyMjolMjJPLUpUZENKVEl5YldWMGNtbGpUbUZ0WlNVeU1qb2xNakppWlhOMFgyeHZjM01sTWpJc0pUSXlZMjl1ZEdWNGRFNWhiV1VsTWpJNkpUSXlKVEl5SlRkRSUyMiwlMjJ2YWx1ZSUyMjolN0IlMjJvcHRpb25fbmFtZSUyMjolMjJiZXN0X2xvc3MlMjIsJTIyY29udGV4dCUyMjpudWxsJTdEJTdELCU3QiUyMmxhYmVsJTIyOiUyMmJsZXUlMjIsJTIyZ3JvdXAlMjI6JTIyYmxldSUyMiwlMjJ0eXBlJTIyOiUyMm1ldHJpY3MlMjIsJTIyY29sb3IlMjI6JTIyIzNFNzJFNyUyMiwlMjJrZXklMjI6JTIyTy1KVGRDSlRJeWJXVjBjbWxqVG1GdFpTVXlNam9sTWpKaWJHVjFKVEl5TENVeU1tTnZiblJsZUhST1lXMWxKVEl5T2lVeU1pVXlNaVUzUkElMjIsJTIydmFsdWUlMjI6JTdCJTIyb3B0aW9uX25hbWUlMjI6JTIyYmxldSUyMiwlMjJjb250ZXh0JTIyOm51bGwlN0QlN0QlNUQsJTIycXVlcnklMjI6JTIycnVuLmhwYXJhbXMubGVhcm5pbmdfcmF0ZSUyMCUzRSUyMDAuMDAwMDElMjIsJTIyYWR2YW5jZWRNb2RlJTIyOmZhbHNlLCUyMmFkdmFuY2VkUXVlcnklMjI6JTIycnVuLmhwYXJhbXMubGVhcm5pbmdfcmF0ZSUyMCUzRSUyMDAuMDAwMDElMjBhbmQlMjAoKG1ldHJpYy5uYW1lJTIwPT0lMjAlNUMlMjJibGV1JTVDJTIyKSUyMG9yJTIwKG1ldHJpYy5uYW1lJTIwPT0lMjAlNUMlMjJiZXN0X2xvc3MlNUMlMjIpJTIwb3IlMjAobWV0cmljLm5hbWUlMjA9PSUyMCU1QyUyMmJzeiU1QyUyMiUyMGFuZCUyMG1ldHJpYy5jb250ZXh0LnN1YnNldCUyMD09JTIwJTVDJTIydHJhaW4lNUMlMjIpKSUyMiU3RA">   | <a href="https://play.aimstack.io/image-generation/images?grouping=O-JTdCJTIycm93JTIyOiU1QiU1RCwlMjJyZXZlcnNlTW9kZSUyMjolN0IlMjJyb3clMjI6ZmFsc2UsJTIyZ3JvdXAlMjI6ZmFsc2UlN0QsJTIyaXNBcHBsaWVkJTIyOiU3QiUyMnJvdyUyMjp0cnVlLCUyMmdyb3VwJTIyOnRydWUlN0QsJTIyZ3JvdXAlMjI6JTVCJTIyaW5kZXglMjIsJTIyc3RlcCUyMiU1RCU3RA&select=O-JTdCJTIyb3B0aW9ucyUyMjolNUIlN0IlMjJsYWJlbCUyMjolMjJnZW5lcmF0ZWQlMjIsJTIyZ3JvdXAlMjI6JTIyZ2VuZXJhdGVkJTIyLCUyMmNvbG9yJTIyOiUyMiMzRTcyRTclMjIsJTIya2V5JTIyOiUyMk8tSlRkQ0pUSXliV1YwY21salRtRnRaU1V5TWpvbE1qSm5aVzVsY21GMFpXUWxNaklzSlRJeVkyOXVkR1Y0ZEU1aGJXVWxNakk2SlRJeUpUSXlKVGRFJTIyLCUyMnZhbHVlJTIyOiU3QiUyMm9wdGlvbl9uYW1lJTIyOiUyMmdlbmVyYXRlZCUyMiwlMjJjb250ZXh0JTIyOm51bGwlN0QlN0QlNUQsJTIycXVlcnklMjI6JTIyaW1hZ2VzLmNvbnRleHQuaW50ZXJwb2xhdGVkJTIwYW5kJTIwcnVuLmhwYXJhbXMubmFtZSUyMD09JTIwJTVDJTIybWV0ZmFjZXMlNUMlMjIlMjIsJTIyYWR2YW5jZWRNb2RlJTIyOmZhbHNlLCUyMmFkdmFuY2VkUXVlcnklMjI6JTIyaW1hZ2VzLmNvbnRleHQuaW50ZXJwb2xhdGVkJTIwYW5kJTIwcnVuLmhwYXJhbXMubmFtZSUyMD09JTIwJTVDJTIybWV0ZmFjZXMlNUMlMjIlMjIlN0Q&images=O-JTdCJTIyaW5kZXhEZW5zaXR5JTIyOjcsJTIycmVjb3JkRGVuc2l0eSUyMjoxNSwlMjJ0b29sdGlwJTIyOiU3QiUyMmFwcGVhcmFuY2UlMjI6JTIyYXV0byUyMiwlMjJkaXNwbGF5JTIyOnRydWUsJTIyc2VsZWN0ZWRGaWVsZHMlMjI6JTVCJTVELCUyMnNlbGVjdGVkUGFyYW1zJTIyOiU1QiU1RCU3RCwlMjJhZGRpdGlvbmFsUHJvcGVydGllcyUyMjolN0IlMjJhbGlnbm1lbnRUeXBlJTIyOiUyMkhlaWdodCUyMiwlMjJtZWRpYUl0ZW1TaXplJTIyOjIzLCUyMmltYWdlUmVuZGVyaW5nJTIyOiUyMnBpeGVsYXRlZCUyMiwlMjJzdGFja2luZyUyMjpmYWxzZSU3RCwlMjJmb2N1c2VkU3RhdGUlMjI6JTdCJTIyYWN0aXZlJTIyOmZhbHNlLCUyMmtleSUyMjpudWxsJTdELCUyMnNvcnRGaWVsZHMlMjI6JTVCJTVELCUyMnNvcnRGaWVsZHNEaWN0JTIyOiU3QiU3RCwlMjJpbnB1dHNWYWxpZGF0aW9ucyUyMjolN0IlMjJpbmRleERlbnNpdHklMjI6dHJ1ZSwlMjJyZWNvcmREZW5zaXR5JTIyOnRydWUlN0QsJTIyY2FsY1JhbmdlcyUyMjpmYWxzZSwlMjJzdGVwUmFuZ2UlMjI6JTVCMSw0Njk1NiU1RCwlMjJpbmRleFJhbmdlJTIyOiU1QjAsNyU1RCwlMjJyZWNvcmRTbGljZSUyMjolNUIzMTE0Miw0Njk1NyU1RCwlMjJpbmRleFNsaWNlJTIyOiU1QjAsOCU1RCwlMjJpbWFnZVByb3BlcnRpZXMlMjI6JTdCJTIyYWxpZ25tZW50VHlwZSUyMjolMjJIZWlnaHQlMjIsJTIyaW1hZ2VTaXplJTIyOjI1LCUyMmltYWdlUmVuZGVyaW5nJTIyOiUyMnBpeGVsYXRlZCUyMiU3RCU3RA">   |
| Training logs of a neural translation model(from WMT'19 competition). | Training logs of 'lightweight' GAN, proposed in ICLR 2021. |

| [FastSpeech 2 experiments](https://play.aimstack.io/fastspeech2/runs/d9e89aa7875e44b2ba85612a/audios)| [Simple MNIST](https://play.aimstack.io/digit-recognition/runs/7f083da898624a2c98e0f363/distributions) |
|:---:|:---:|
|    |    |
| Training logs of Microsoft's "FastSpeech 2: Fast and High-Quality End-to-End Text to Speech". | Simple MNIST training logs. |

# 🌍 Ecosystem

Aim is not just an experiment tracker. It's a groundwork for an ecosystem.
Check out the two most famous Aim-based tools.

| [aimlflow](https://github.com/aimhubio/aimlflow) | [Aim-spaCy](https://github.com/aimhubio/aim-spacy) |
|:---:|:---:|
| ![aimlflow](https://user-images.githubusercontent.com/97726819/225957836-cdec88e3-4993-435a-a135-d78be3ac1635.png) | ![Aim-spaCy](https://user-images.githubusercontent.com/97726819/225957990-4edc4525-1f65-4405-b663-ce7af888bdfa.png) |
| Exploring MLflow experiments with a powerful UI | an Aim-based spaCy experiment tracker |

# 🏁 Quick start

Follow the steps below to get started with Aim.

## 1. Install Aim on your training environment

```shell
pip3 install aim
```

## 2. Integrate Aim with your code

```python
from aim import Run

# Initialize a new run
run = Run()

# Log run parameters
run["hparams"] = {
    "learning_rate": 0.001,
    "batch_size": 32,
}

# Log metrics
for i in range(10):
    run.track(i, name='loss', step=i, context={ "subset":"train" })
    run.track(i, name='acc', step=i, context={ "subset":"train" })
```

_See the full list of supported trackable objects(e.g. images, text, etc) [here](https://aimstack.readthedocs.io/en/latest/quick_start/supported_types.html)._

## 3. Run the training as usual and start Aim UI

```shell
aim up
```

## Learn more

Migrate from other tools

Aim has built-in converters to easily migrate logs from other tools.
These migrations cover the most common usage scenarios.
In case of custom and complex scenarios you can use Aim SDK to implement your own conversion script.

- [TensorBoard logs converter](https://aimstack.readthedocs.io/en/latest/quick_start/convert_data.html#show-tensorboard-logs-in-aim)
- [MLFlow logs converter](https://aimstack.readthedocs.io/en/latest/quick_start/convert_data.html#show-mlflow-logs-in-aim)
- [Weights & Biases logs converter](https://aimstack.readthedocs.io/en/latest/quick_start/convert_data.html#show-weights-and-biases-logs-in-aim)

Integrate Aim into an existing project

Aim easily integrates with a wide range of ML frameworks, providing built-in callbacks for most of them.

- [Integration with Pytorch Ignite](https://aimstack.readthedocs.io/en/latest/quick_start/integrations.html#integration-with-pytorch-ignite)
- [Integration with Pytorch Lightning](https://aimstack.readthedocs.io/en/latest/quick_start/integrations.html#integration-with-pytorch-lightning)
- [Integration with Hugging Face](https://aimstack.readthedocs.io/en/latest/quick_start/integrations.html#integration-with-hugging-face)
- [Integration with Keras & tf.Keras](https://aimstack.readthedocs.io/en/latest/quick_start/integrations.html#integration-with-keras-tf-keras)
- [Integration with Keras Tuner](https://aimstack.readthedocs.io/en/latest/quick_start/integrations.html#integration-with-keras-tuner)
- [Integration with XGboost](https://aimstack.readthedocs.io/en/latest/quick_start/integrations.html#integration-with-xgboost)
- [Integration with CatBoost](https://aimstack.readthedocs.io/en/latest/quick_start/integrations.html#integration-with-catboost)
- [Integration with LightGBM](https://aimstack.readthedocs.io/en/latest/quick_start/integrations.html#integration-with-lightgbm)
- [Integration with fastai](https://aimstack.readthedocs.io/en/latest/quick_start/integrations.html#integration-with-fastai)
- [Integration with MXNet](https://aimstack.readthedocs.io/en/latest/quick_start/integrations.html#integration-with-mxnet)
- [Integration with Optuna](https://aimstack.readthedocs.io/en/latest/quick_start/integrations.html#integration-with-optuna)
- [Integration with PaddlePaddle](https://aimstack.readthedocs.io/en/latest/quick_start/integrations.html#integration-with-paddlepaddle)
- [Integration with Stable-Baselines3](https://aimstack.readthedocs.io/en/latest/quick_start/integrations.html#integration-with-stable-baselines3)
- [Integration with Acme](https://aimstack.readthedocs.io/en/latest/quick_start/integrations.html#integration-with-acme)
- [Integration with Prophet](https://aimstack.readthedocs.io/en/latest/quick_start/integrations.html#integration-with-prophet)

Query runs programmatically via SDK

Aim Python SDK empowers you to query and access any piece of tracked metadata with ease.

```python
from aim import Repo

my_repo = Repo('/path/to/aim/repo')

query = "metric.name == 'loss'" # Example query

# Get collection of metrics
for run_metrics_collection in my_repo.query_metrics(query).iter_runs():
    for metric in run_metrics_collection:
        # Get run params
        params = metric.run[...]
        # Get metric values
        steps, metric_values = metric.values.sparse_numpy()
```

Set up a centralized tracking server

Aim remote tracking server allows running experiments in a multi-host environment and collect tracked data in a centralized location.

See the docs on how to [set up the remote server](https://aimstack.readthedocs.io/en/latest/using/remote_tracking.html).

Deploy Aim on kubernetes

- The official Aim docker image: https://hub.docker.com/r/aimstack/aim
- A guide on how to deploy Aim on kubernetes: https://aimstack.readthedocs.io/en/latest/using/k8s_deployment.html

Read the full documentation on [aimstack.readthedocs.io](https://aimstack.readthedocs.io) 📖

# 🆚 Comparisons to familiar tools

  TensorBoard vs Aim

**Training run comparison**

Order of magnitude faster training run comparison with Aim
- The tracked params are first class citizens at Aim. You can search, group, aggregate via params - deeply explore all the tracked data (metrics, params, images) on the UI.
- With tensorboard the users are forced to record those parameters in the training run name to be able to search and compare. This causes a super-tedius comparison experience and usability issues on the UI when there are many experiments and params. **TensorBoard doesn't have features to group, aggregate the metrics**

**Scalability**

- Aim is built to handle 1000s of training runs - both on the backend and on the UI.
- TensorBoard becomes really slow and hard to use when a few hundred training runs are queried / compared.

**Beloved TB visualizations to be added on Aim**

- Embedding projector.
- Neural network visualization.

  MLflow vs Aim

MLFlow is an end-to-end ML Lifecycle tool.
Aim is focused on training tracking.
The main differences of Aim and MLflow are around the UI scalability and run comparison features.

Aim and MLflow are a perfect match - check out the [aimlflow](https://github.com/aimhubio/aimlflow) - the tool that enables Aim superpowers on Mlflow.

**Run comparison**

- Aim treats tracked parameters as first-class citizens. Users can query runs, metrics, images and filter using the params.
- MLFlow does have a search by tracked config, but there are no grouping, aggregation, subplotting by hyparparams and other comparison features available.

**UI Scalability**

- Aim UI can handle several thousands of metrics at the same time smoothly with 1000s of steps. It may get shaky when you explore 1000s of metrics with 10000s of steps each. But we are constantly optimizing!
- MLflow UI becomes slow to use when there are a few hundreds of runs.

  Weights and Biases vs Aim

Hosted vs self-hosted

- Weights and Biases is a hosted closed-source MLOps platform.
- Aim is self-hosted, free and open-source experiment tracking tool.

# 🛣 Roadmap

## Detailed milestones

The [Aim product roadmap](https://github.com/orgs/aimhubio/projects/3) :sparkle:

- The `Backlog` contains the issues we are going to choose from and prioritize weekly
- The issues are mainly prioritized by the highly-requested features

## High-level roadmap

The high-level features we are going to work on the next few months:

**In progress**

- [ ] Aim SDK low-level interface
- [ ] Dashboards – customizable layouts with embedded explorers
- [ ] Ergonomic UI kit
- [ ] Text Explorer

  Next-up

**Aim UI**

- Runs management
    - Runs explorer – query and visualize runs data(images, audio, distributions, ...) in a central dashboard
- Explorers
    - Distributions Explorer

**SDK and Storage**

- Scalability
    - Smooth UI and SDK experience with over 10.000 runs
- Runs management
    - CLI commands
        - Reporting - runs summary and run details in a CLI compatible format
        - Manipulations – copy, move, delete runs, params and sequences
- Cloud storage support – store runs blob(e.g. images) data on the cloud
- Artifact storage – store files, model checkpoints, and beyond

**Integrations**

- ML Frameworks:
    - Shortlist: scikit-learn
- Resource management tools
    - Shortlist: Kubeflow, Slurm
- Workflow orchestration tools

  Done

  - [x] Live updates (Shipped: _Oct 18 2021_)
  - [x] Images tracking and visualization (Start: _Oct 18 2021_, Shipped: _Nov 19 2021_)
  - [x] Distributions tracking and visualization (Start: _Nov 10 2021_, Shipped: _Dec 3 2021_)
  - [x] Jupyter integration (Start: _Nov 18 2021_, Shipped: _Dec 3 2021_)
  - [x] Audio tracking and visualization (Start: _Dec 6 2021_, Shipped: _Dec 17 2021_)
  - [x] Transcripts tracking and visualization (Start: _Dec 6 2021_, Shipped: _Dec 17 2021_)
  - [x] Plotly integration (Start: _Dec 1 2021_, Shipped: _Dec 17 2021_)
  - [x] Colab integration (Start: _Nov 18 2021_, Shipped: _Dec 17 2021_)
  - [x] Centralized tracking server (Start: _Oct 18 2021_, Shipped: _Jan 22 2022_)
  - [x] Tensorboard adaptor - visualize TensorBoard logs with Aim (Start: _Dec 17 2021_, Shipped: _Feb 3 2022_)
  - [x] Track git info, env vars, CLI arguments, dependencies (Start: _Jan 17 2022_, Shipped: _Feb 3 2022_)
  - [x] MLFlow adaptor (visualize MLflow logs with Aim) (Start: _Feb 14 2022_, Shipped: _Feb 22 2022_)
  - [x] Activeloop Hub integration (Start: _Feb 14 2022_, Shipped: _Feb 22 2022_)
  - [x] PyTorch-Ignite integration (Start: _Feb 14 2022_, Shipped: _Feb 22 2022_)
  - [x] Run summary and overview info(system params, CLI args, git info, ...) (Start: _Feb 14 2022_, Shipped: _Mar 9 2022_)
  - [x] Add DVC related metadata into aim run (Start: _Mar 7 2022_, Shipped: _Mar 26 2022_)
  - [x] Ability to attach notes to Run from UI (Start: _Mar 7 2022_, Shipped: _Apr 29 2022_)
  - [x] Fairseq integration (Start: _Mar 27 2022_, Shipped: _Mar 29 2022_)
  - [x] LightGBM integration (Start: _Apr 14 2022_, Shipped: _May 17 2022_)
  - [x] CatBoost integration (Start: _Apr 20 2022_, Shipped: _May 17 2022_)
  - [x] Run execution details(display stdout/stderr logs) (Start: _Apr 25 2022_, Shipped: _May 17 2022_)
  - [x] Long sequences(up to 5M of steps) support (Start: _Apr 25 2022_, Shipped: _Jun 22 2022_)
  - [x] Figures Explorer (Start: _Mar 1 2022_, Shipped: _Aug 21 2022_)
  - [x] Notify on stuck runs (Start: _Jul 22 2022_, Shipped: _Aug 21 2022_)
  - [x] Integration with KerasTuner (Start: _Aug 10 2022_, Shipped: _Aug 21 2022_)
  - [x] Integration with WandB (Start: _Aug 15 2022_, Shipped: _Aug 21 2022_)
  - [x] Stable remote tracking server (Start: _Jun 15 2022_, Shipped: _Aug 21 2022_)
  - [x] Integration with fast.ai (Start: _Aug 22 2022_, Shipped: _Oct 6 2022_)
  - [x] Integration with MXNet (Start: _Sep 20 2022_, Shipped: _Oct 6 2022_)
  - [x] Project overview page (Start: _Sep 1 2022_, Shipped: _Oct 6 2022_)
  - [x] Remote tracking server scaling (Start: _Sep 11 2022_, Shipped: _Nov 26 2022_)
  - [x] Integration with PaddlePaddle (Start: _Oct 2 2022_, Shipped: _Nov 26 2022_)
  - [x] Integration with Optuna (Start: _Oct 2 2022_, Shipped: _Nov 26 2022_)
  - [x] Audios Explorer (Start: _Oct 30 2022_, Shipped: _Nov 26 2022_)
  - [x] Experiment page (Start: _Nov 9 2022_, Shipped: _Nov 26 2022_)
  - [x] HuggingFace datasets (Start: _Dec 29 2022_, _Feb 3 2023_)

# 👥 Community

## Aim README badge

Add Aim badge to your README, if you've enjoyed using Aim in your work:

[![Aim](https://img.shields.io/badge/powered%20by-Aim-%231473E6)](https://github.com/aimhubio/aim)

```
[![Aim](https://img.shields.io/badge/powered%20by-Aim-%231473E6)](https://github.com/aimhubio/aim)
```

## Cite Aim in your papers

In case you've found Aim helpful in your research journey, we'd be thrilled if you could acknowledge Aim's contribution:

```bibtex
@software{Arakelyan_Aim_2020,
  author = {Arakelyan, Gor and Soghomonyan, Gevorg and {The Aim team}},
  doi = {10.5281/zenodo.6536395},
  license = {Apache-2.0},
  month = {6},
  title = {{Aim}},
  url = {https://github.com/aimhubio/aim},
  version = {3.9.3},
  year = {2020}
}
```

## Contributing to Aim

Considering contibuting to Aim?
To get started, please take a moment to read the [CONTRIBUTING.md](https://github.com/aimhubio/aim/blob/main/CONTRIBUTING.md) guide.

Join Aim contributors by submitting your first pull request. Happy coding! 😊

Made with [contrib.rocks](https://contrib.rocks).

## More questions?

1. [Read the docs](https://aimstack.readthedocs.io/en/latest/)
2. [Open a feature request or report a bug](https://github.com/aimhubio/aim/issues)
3. [Join Discord community server](https://community.aimstack.io/)


![PyTorch Forecasting](./docs/source/_static/logo.svg)

[pypi-image]: https://badge.fury.io/py/pytorch-forecasting.svg
[pypi-url]: https://pypi.python.org/pypi/pytorch-forecasting
[conda-image]: https://img.shields.io/conda/v/conda-forge/pytorch-forecasting
[conda-url]: https://anaconda.org/conda-forge/pytorch-forecasting
[build-image]: https://github.com/jdb78/pytorch-forecasting/actions/workflows/test.yml/badge.svg?branch=master
[build-url]: https://github.com/jdb78/pytorch-forecasting/actions/workflows/test.yml?query=branch%3Amaster
[linter-image]: https://github.com/jdb78/pytorch-forecasting/actions/workflows/lint.yml/badge.svg?event=push
[linter-url]: https://github.com/jdb78/pytorch-forecasting/actions/workflows/code_quality.yml?query=branch%3Amaster
[docs-image]: https://readthedocs.org/projects/pytorch-forecasting/badge/?version=latest
[docs-url]: https://pytorch-forecasting.readthedocs.io
[coverage-image]: https://codecov.io/gh/jdb78/pytorch-forecasting/branch/master/graph/badge.svg
[coverage-url]: https://codecov.io/github/jdb78/pytorch-forecasting?branch=master

[![PyPI Version][pypi-image]][pypi-url] [![Conda Version][conda-image]][conda-url] [![Docs Status][docs-image]][docs-url] [![Linter Status][linter-image]][linter-url] [![Build Status][build-image]][build-url] [![Code Coverage][coverage-image]][coverage-url]

**[Documentation](https://pytorch-forecasting.readthedocs.io)** | **[Tutorials](https://pytorch-forecasting.readthedocs.io/en/latest/tutorials.html)** | **[Release Notes](https://pytorch-forecasting.readthedocs.io/en/latest/CHANGELOG.html)**

_PyTorch Forecasting_ is a PyTorch-based package for forecasting time series with state-of-the-art network architectures. It provides a high-level API for training networks on pandas data frames and leverages
[PyTorch Lightning](https://pytorch-lightning.readthedocs.io/) for scalable training on (multiple) GPUs, CPUs and for automatic logging.

---

Our article on [Towards Data Science](https://towardsdatascience.com/introducing-pytorch-forecasting-64de99b9ef46) introduces the package and provides background information.

PyTorch Forecasting aims to ease state-of-the-art timeseries forecasting with neural networks for real-world cases and research alike. The goal is to provide a high-level API with maximum flexibility for professionals and reasonable defaults for beginners.
Specifically, the package provides

- A timeseries dataset class which abstracts handling variable transformations, missing values,
  randomized subsampling, multiple history lengths, etc.
- A base model class which provides basic training of timeseries models along with logging in tensorboard
  and generic visualizations such actual vs predictions and dependency plots
- Multiple neural network architectures for timeseries forecasting that have been enhanced
  for real-world deployment and come with in-built interpretation capabilities
- Multi-horizon timeseries metrics
- Hyperparameter tuning with [optuna](https://optuna.readthedocs.io/)

The package is built on [pytorch-lightning](https://pytorch-lightning.readthedocs.io/) to allow training on CPUs, single and multiple GPUs out-of-the-box.

# Installation

If you are working on windows, you need to first install PyTorch with

`pip install torch -f https://download.pytorch.org/whl/torch_stable.html`.

Otherwise, you can proceed with

`pip install pytorch-forecasting`

Alternatively, you can install the package via conda

`conda install pytorch-forecasting pytorch -c pytorch>=1.7 -c conda-forge`

PyTorch Forecasting is now installed from the conda-forge channel while PyTorch is install from the pytorch channel.

To use the MQF2 loss (multivariate quantile loss), also install
`pip install pytorch-forecasting[mqf2]`

# Documentation

Visit [https://pytorch-forecasting.readthedocs.io](https://pytorch-forecasting.readthedocs.io) to read the
documentation with detailed tutorials.

# Available models

The documentation provides a [comparison of available models](https://pytorch-forecasting.readthedocs.io/en/latest/models.html).

- [Temporal Fusion Transformers for Interpretable Multi-horizon Time Series Forecasting](https://arxiv.org/pdf/1912.09363.pdf)
  which outperforms DeepAR by Amazon by 36-69% in benchmarks
- [N-BEATS: Neural basis expansion analysis for interpretable time series forecasting](http://arxiv.org/abs/1905.10437)
  which has (if used as ensemble) outperformed all other methods including ensembles of traditional statical
  methods in the M4 competition. The M4 competition is arguably the most important benchmark for univariate time series forecasting.
- [N-HiTS: Neural Hierarchical Interpolation for Time Series Forecasting](http://arxiv.org/abs/2201.12886) which supports covariates and has consistently beaten N-BEATS. It is also particularly well-suited for long-horizon forecasting.
- [DeepAR: Probabilistic forecasting with autoregressive recurrent networks](https://www.sciencedirect.com/science/article/pii/S0169207019301888)
  which is the one of the most popular forecasting algorithms and is often used as a baseline
- Simple standard networks for baselining: LSTM and GRU networks as well as a MLP on the decoder
- A baseline model that always predicts the latest known value

To implement new models or other custom components, see the [How to implement new models tutorial](https://pytorch-forecasting.readthedocs.io/en/latest/tutorials/building.html). It covers basic as well as advanced architectures.

# Usage example

Networks can be trained with the [PyTorch Lighning Trainer](https://pytorch-lightning.readthedocs.io/en/latest/common/trainer.html) on [pandas Dataframes](https://pandas.pydata.org/pandas-docs/stable/user_guide/dsintro.html#dataframe) which are first converted to a [TimeSeriesDataSet](https://pytorch-forecasting.readthedocs.io/en/latest/data.html).

```python
# imports for training
import lightning.pytorch as pl
from lightning.pytorch.loggers import TensorBoardLogger
from lightning.pytorch.callbacks import EarlyStopping, LearningRateMonitor
# import dataset, network to train and metric to optimize
from pytorch_forecasting import TimeSeriesDataSet, TemporalFusionTransformer, QuantileLoss
from lightning.pytorch.tuner import Tuner

# load data: this is pandas dataframe with at least a column for
# * the target (what you want to predict)
# * the timeseries ID (which should be a unique string to identify each timeseries)
# * the time of the observation (which should be a monotonically increasing integer)
data = ...

# define the dataset, i.e. add metadata to pandas dataframe for the model to understand it
max_encoder_length = 36
max_prediction_length = 6
training_cutoff = "YYYY-MM-DD"  # day for cutoff

training = TimeSeriesDataSet(
    data[lambda x: x.date <= training_cutoff],
    time_idx= ...,  # column name of time of observation
    target= ...,  # column name of target to predict
    group_ids=[ ... ],  # column name(s) for timeseries IDs
    max_encoder_length=max_encoder_length,  # how much history to use
    max_prediction_length=max_prediction_length,  # how far to predict into future
    # covariates static for a timeseries ID
    static_categoricals=[ ... ],
    static_reals=[ ... ],
    # covariates known and unknown in the future to inform prediction
    time_varying_known_categoricals=[ ... ],
    time_varying_known_reals=[ ... ],
    time_varying_unknown_categoricals=[ ... ],
    time_varying_unknown_reals=[ ... ],
)

# create validation dataset using the same normalization techniques as for the training dataset
validation = TimeSeriesDataSet.from_dataset(training, data, min_prediction_idx=training.index.time.max() + 1, stop_randomization=True)

# convert datasets to dataloaders for training
batch_size = 128
train_dataloader = training.to_dataloader(train=True, batch_size=batch_size, num_workers=2)
val_dataloader = validation.to_dataloader(train=False, batch_size=batch_size, num_workers=2)

# create PyTorch Lighning Trainer with early stopping
early_stop_callback = EarlyStopping(monitor="val_loss", min_delta=1e-4, patience=1, verbose=False, mode="min")
lr_logger = LearningRateMonitor()
trainer = pl.Trainer(
    max_epochs=100,
    accelerator="auto",  # run on CPU, if on multiple GPUs, use strategy="ddp"
    gradient_clip_val=0.1,
    limit_train_batches=30,  # 30 batches per epoch
    callbacks=[lr_logger, early_stop_callback],
    logger=TensorBoardLogger("lightning_logs")
)

# define network to train - the architecture is mostly inferred from the dataset, so that only a few hyperparameters have to be set by the user
tft = TemporalFusionTransformer.from_dataset(
    # dataset
    training,
    # architecture hyperparameters
    hidden_size=32,
    attention_head_size=1,
    dropout=0.1,
    hidden_continuous_size=16,
    # loss metric to optimize
    loss=QuantileLoss(),
    # logging frequency
    log_interval=2,
    # optimizer parameters
    learning_rate=0.03,
    reduce_on_plateau_patience=4
)
print(f"Number of parameters in network: {tft.size()/1e3:.1f}k")

# find the optimal learning rate
res = Tuner(trainer).lr_find(
    tft, train_dataloaders=train_dataloader, val_dataloaders=val_dataloader, early_stop_threshold=1000.0, max_lr=0.3,
)
# and plot the result - always visually confirm that the suggested learning rate makes sense
print(f"suggested learning rate: {res.suggestion()}")
fig = res.plot(show=True, suggest=True)
fig.show()

# fit the model on the data - redefine the model with the correct learning rate if necessary
trainer.fit(
    tft, train_dataloaders=train_dataloader, val_dataloaders=val_dataloader,
)
```


An open source chat-ops bot framework

---

  Quick Start •
  Documentation •
  Playground •
  Blog •
  Community

---

An open source chatbot framework written in Python. It is designed to be extendable, scalable and simple.

This framework is designed to take events from chat services and other sources and execute Python functions (skills) based on their contents. Those functions can be anything you like, from simple conversational responses to running complex tasks. The true power of this project is to act as a glue library to bring the multitude of natural language APIs, chat services and third-party APIs together.

See [our full documentation](https://docs.opsdroid.dev) to get started.

### Contributors

This project exists thanks to all the people who contribute. [[Contribute](https://docs.opsdroid.dev/en/stable/contributing/)].

## Backers

Thank you to all our backers! 🙏 [[Become a backer](https://opencollective.com/opsdroid#backer)]

## Sponsors

Support this project by becoming a sponsor. Your logo will show up here with a link to your website. [[Become a sponsor](https://opencollective.com/opsdroid#sponsor)]


aiokafka
========
.. image:: https://github.com/aio-libs/aiokafka/actions/workflows/tests.yml/badge.svg?branch=master
    :target: https://github.com/aio-libs/aiokafka/actions/workflows/tests.yml?query=branch%3Amaster
    :alt: |Build status|
.. image:: https://codecov.io/github/aio-libs/aiokafka/coverage.svg?branch=master
    :target: https://codecov.io/gh/aio-libs/aiokafka/branch/master
    :alt: |Coverage|
.. image:: https://badges.gitter.im/Join%20Chat.svg
    :target: https://gitter.im/aio-libs/Lobby
    :alt: |Chat on Gitter|

asyncio client for Kafka

AIOKafkaProducer
****************

AIOKafkaProducer is a high-level, asynchronous message producer.

Example of AIOKafkaProducer usage:

.. code-block:: python

    from aiokafka import AIOKafkaProducer
    import asyncio

    async def send_one():
        producer = AIOKafkaProducer(bootstrap_servers='localhost:9092')
        # Get cluster layout and initial topic/partition leadership information
        await producer.start()
        try:
            # Produce message
            await producer.send_and_wait("my_topic", b"Super message")
        finally:
            # Wait for all pending messages to be delivered or expire.
            await producer.stop()

    asyncio.run(send_one())

AIOKafkaConsumer
****************

AIOKafkaConsumer is a high-level, asynchronous message consumer.
It interacts with the assigned Kafka Group Coordinator node to allow multiple
consumers to load balance consumption of topics (requires kafka >= 0.9.0.0).

Example of AIOKafkaConsumer usage:

.. code-block:: python

    from aiokafka import AIOKafkaConsumer
    import asyncio

    async def consume():
        consumer = AIOKafkaConsumer(
            'my_topic', 'my_other_topic',
            bootstrap_servers='localhost:9092',
            group_id="my-group")
        # Get cluster layout and join group `my-group`
        await consumer.start()
        try:
            # Consume messages
            async for msg in consumer:
                print("consumed: ", msg.topic, msg.partition, msg.offset,
                      msg.key, msg.value, msg.timestamp)
        finally:
            # Will leave consumer group; perform autocommit if enabled.
            await consumer.stop()

    asyncio.run(consume())

Running tests
-------------

Docker is required to run tests. See https://docs.docker.com/engine/installation for installation notes. Also note, that `lz4` compression libraries for python will require `python-dev` package,
or python source header files for compilation on Linux.
NOTE: You will also need a valid java installation. It's required for the ``keytool`` utility, used to
generate ssh keys for some tests.

Setting up tests requirements (assuming you're within virtualenv on ubuntu 14.04+)::

    sudo apt-get install -y libkrb5-dev krb5-user
    make setup

Running tests with coverage::

    make cov

To run tests with a specific version of Kafka (default one is 2.8.1) use KAFKA_VERSION variable::

    make cov SCALA_VERSION=2.11 KAFKA_VERSION=0.10.2.1

Test running cheatsheat:

 * ``make test FLAGS="-l -x --ff"`` - run until 1 failure, rerun failed tests first. Great for cleaning up a lot of errors, say after a big refactor.
 * ``make test FLAGS="-k consumer"`` - run only the consumer tests.
 * ``make test FLAGS="-m 'not ssl'"`` - run tests excluding ssl.
 * ``make test FLAGS="--no-pull"`` - do not try to pull new docker image before test run.


# setuptools_scm
[![github ci](https://github.com/pypa/setuptools_scm/workflows/python%20tests+artifacts+release/badge.svg)](https://github.com/pypa/setuptools_scm/actions)
[![Documentation Status](https://readthedocs.org/projects/setuptools-scm/badge/?version=latest)](https://setuptools-scm.readthedocs.io/en/latest/?badge=latest)
[![tidelift](https://tidelift.com/badges/package/pypi/setuptools-scm) ](https://tidelift.com/subscription/pkg/pypi-setuptools-scm?utm_source=pypi-setuptools-scm&utm_medium=readme)

## about

[setuptools-scm] extracts Python package versions from `git` or
`hg` metadata instead of declaring them as the version argument
or in an SCM managed file.

Additionally, [setuptools-scm] provides setuptools
with a list of files that are managed by the SCM

(i.e. it automatically adds **all of** the SCM-managed files to the sdist).

Unwanted files must be excluded via `MANIFEST.in`.

## `pyproject.toml` usage

The preferred way to configure [setuptools-scm] is to author
settings in a `tool.setuptools_scm` section of `pyproject.toml`.

This feature requires setuptools 61 or later.
First, ensure that [setuptools-scm] is present during the project's
build step by specifying it as one of the build requirements.

```toml title="pyproject.toml"
[build-system]
requires = ["setuptools>=64", "setuptools_scm>=8"]
build-backend = "setuptools.build_meta"
```

That will be sufficient to require [setuptools-scm] for projects
that support [PEP 518] like [pip] and [build].

[pip]: https://pypi.org/project/pip
[build]: https://pypi.org/project/build
[PEP 518]: https://peps.python.org/pep-0518/

To enable version inference, you need to set the version
dynamically in the `project` section of `pyproject.toml`:

```toml title="pyproject.toml"
[project]
# version = "0.0.1"  # Remove any existing version parameter.
dynamic = ["version"]

[tool.setuptools_scm]
```

Additionally, a version file can be written by specifying:

```toml title="pyproject.toml"
[tool.setuptools_scm]
version_file = "pkg/_version.py"
```

Where `pkg` is the name of your package.

If you need to confirm which version string is being generated or debug the configuration,
you can install [setuptools-scm] directly in your working environment and run:

```console
$ python -m setuptools_scm
# To explore other options, try:
$ python -m setuptools_scm --help
```

For further configuration see the [documentation].

[setuptools-scm]: https://github.com/pypa/setuptools_scm
[documentation]: https://setuptools-scm.readthedocs.io/

## Interaction with Enterprise Distributions

Some enterprise distributions like RHEL7
ship rather old setuptools versions.

In those cases its typically possible to build by using an sdist against `setuptools_scm<2.0`.
As those old setuptools versions lack sensible types for versions,
modern [setuptools-scm] is unable to support them sensibly.

It's strongly recommended to build a wheel artifact using modern Python and setuptools,
then installing the artifact instead of trying to run against old setuptools versions.

## Code of Conduct

Everyone interacting in the [setuptools-scm] project's codebases, issue
trackers, chat rooms, and mailing lists is expected to follow the
[PSF Code of Conduct].

[PSF Code of Conduct]: https://github.com/pypa/.github/blob/main/CODE_OF_CONDUCT.md

## Security Contact

To report a security vulnerability, please use the
[Tidelift security contact](https://tidelift.com/security).
Tidelift will coordinate the fix and disclosure.