# duckdb_engine

[![Supported Python Versions](https://img.shields.io/pypi/pyversions/duckdb-engine)](https://pypi.org/project/duckdb-engine/) [![PyPI version](https://badge.fury.io/py/duckdb-engine.svg)](https://badge.fury.io/py/duckdb-engine) [![PyPI Downloads](https://img.shields.io/pypi/dm/duckdb-engine.svg)](https://pypi.org/project/duckdb-engine/) [![codecov](https://codecov.io/gh/Mause/duckdb_engine/graph/badge.svg)](https://codecov.io/gh/Mause/duckdb_engine)

Basic SQLAlchemy driver for [DuckDB](https://duckdb.org/)

<!--ts-->
* [duckdb_engine](#duckdb_engine)
   * [Installation](#installation)
   * [Usage](#usage)
   * [Usage in IPython/Jupyter](#usage-in-ipythonjupyter)
   * [Configuration](#configuration)
   * [How to register a pandas DataFrame](#how-to-register-a-pandas-dataframe)
   * [Things to keep in mind](#things-to-keep-in-mind)
      * [Auto-incrementing ID columns](#auto-incrementing-id-columns)
      * [Pandas read_sql() chunksize](#pandas-read_sql-chunksize)
      * [Unsigned integer support](#unsigned-integer-support)
   * [Alembic Integration](#alembic-integration)
   * [Preloading extensions (experimental)](#preloading-extensions-experimental)
   * [The name](#the-name)

<!-- Created by https://github.com/ekalinin/github-markdown-toc -->
<!-- Added by: me, at: Wed 20 Sep 2023 12:44:27 AWST -->

<!--te-->

## Installation
```sh
$ pip install duckdb-engine
```

DuckDB Engine also has a conda feedstock available, the instructions for the use of which are available in it's [repository](https://github.com/conda-forge/duckdb-engine-feedstock).

## Usage

Once you've installed this package, you should be able to just use it, as SQLAlchemy does a python path search

```python
from sqlalchemy import Column, Integer, Sequence, String, create_engine
from sqlalchemy.ext.declarative import declarative_base
from sqlalchemy.orm.session import Session

Base = declarative_base()

class FakeModel(Base):  # type: ignore
    __tablename__ = "fake"

    id = Column(Integer, Sequence("fakemodel_id_sequence"), primary_key=True)
    name = Column(String)

eng = create_engine("duckdb:///:memory:")
Base.metadata.create_all(eng)
session = Session(bind=eng)

session.add(FakeModel(name="Frank"))
session.commit()

frank = session.query(FakeModel).one()

assert frank.name == "Frank"
```

## Usage in IPython/Jupyter

With IPython-SQL and DuckDB-Engine you can query DuckDB natively in your notebook! Check out [DuckDB's documentation](https://duckdb.org/docs/guides/python/jupyter) or
Alex Monahan's great demo of this on [his blog](https://alex-monahan.github.io/2021/08/22/Python_and_SQL_Better_Together.html#an-example-workflow-with-duckdb).

## Configuration

You can configure DuckDB by passing `connect_args` to the create_engine function
```python
create_engine(
    'duckdb:///:memory:',
    connect_args={
        'read_only': True,
        'config': {
            'memory_limit': '500mb'
        }
    }
)
```

The supported configuration parameters are listed in the [DuckDB docs](https://duckdb.org/docs/sql/configuration)

## How to register a pandas DataFrame

```python
conn = create_engine("duckdb:///:memory:").connect()

# with SQLAlchemy 1.3
conn.execute("register", ("dataframe_name", pd.DataFrame(...)))

# with SQLAlchemy 1.4+
conn.execute(text("register(:name, :df)"), {"name": "test_df", "df": df})

conn.execute("select * from dataframe_name")
```

## Things to keep in mind
Duckdb's SQL parser is based on the PostgreSQL parser, but not all features in PostgreSQL are supported in duckdb. Because the `duckdb_engine` dialect is derived from the `postgresql` dialect, `SQLAlchemy` may try to use PostgreSQL-only features. Below are some caveats to look out for.

### Auto-incrementing ID columns
When defining an Integer column as a primary key, `SQLAlchemy` uses the `SERIAL` datatype for PostgreSQL. Duckdb does not yet support this datatype because it's a non-standard PostgreSQL legacy type, so a workaround is to use the `SQLAlchemy.Sequence()` object to auto-increment the key. For more information on sequences, you can find the [`SQLAlchemy Sequence` documentation here](https://docs.sqlalchemy.org/en/14/core/defaults.html#associating-a-sequence-as-the-server-side-default).

The following example demonstrates how to create an auto-incrementing ID column for a simple table:

```python
>>> import sqlalchemy
>>> engine = sqlalchemy.create_engine('duckdb:////path/to/duck.db')
>>> metadata = sqlalchemy.MetaData(engine)
>>> user_id_seq = sqlalchemy.Sequence('user_id_seq')
>>> users_table = sqlalchemy.Table(
...     'users',
...     metadata,
...     sqlalchemy.Column(
...         'id',
...         sqlalchemy.Integer,
...         user_id_seq,
...         server_default=user_id_seq.next_value(),
...         primary_key=True,
...     ),
... )
>>> metadata.create_all(bind=engine)
```

### Pandas `read_sql()` chunksize

**NOTE**: this is no longer an issue in versions `>=0.5.0` of `duckdb`

The `pandas.read_sql()` method can read tables from `duckdb_engine` into DataFrames, but the `sqlalchemy.engine.result.ResultProxy` trips up when `fetchmany()` is called. Therefore, for now `chunksize=None` (default) is necessary when reading duckdb tables into DataFrames. For example:

```python
>>> import pandas as pd
>>> import sqlalchemy
>>> engine = sqlalchemy.create_engine('duckdb:////path/to/duck.db')
>>> df = pd.read_sql('users', engine)                ### Works as expected
>>> df = pd.read_sql('users', engine, chunksize=25)  ### Throws an exception
```

### Unsigned integer support

Unsigned integers are supported by DuckDB, and are available in [`duckdb_engine.datatypes`](duckdb_engine/datatypes.py).

## Alembic Integration

SQLAlchemy's companion library `alembic` can optionally be used to manage database migrations.

This support can be enabling by adding an Alembic implementation class for the `duckdb` dialect.

```python
from alembic.ddl.impl import DefaultImpl

class AlembicDuckDBImpl(DefaultImpl):
    """Alembic implementation for DuckDB."""

    __dialect__ = "duckdb"
```

After loading this class with your program, Alembic will no longer raise an error when generating or applying migrations.

## Preloading extensions (experimental)

> DuckDB 0.9.0+ includes builtin support for autoinstalling and autoloading of extensions, see [the extension documentation](http://duckdb.org/docs/archive/0.9.0/extensions/overview#autoloadable-extensions) for more information.

Until the DuckDB python client allows you to natively preload extensions, I've added experimental support via a `connect_args` parameter

```python
from sqlalchemy import create_engine

create_engine(
    'duckdb:///:memory:',
    connect_args={
        'preload_extensions': ['https'],
        'config': {
            's3_region': 'ap-southeast-1'
        }
    }
)
```

## The name

Yes, I'm aware this package should be named `duckdb-driver` or something, I wasn't thinking when I named it and it's too hard to change the name now


==================
WBIA - WildBook IA
==================

|Build| |Documentation| |Downloads|

.. image:: http://i.imgur.com/TNCiEBe.png
    :alt: "(Note: the rhino and wildebeest matches may be dubious. Other species do work well though")

WBIA program for the storage and management of images and derived data for use in computer vision algorithms. It aims to compute who an animal is, what species an animal is, and where an animal is with the ultimate goal being to ask important why biological questions.

This project is the Machine Learning (ML) / computer vision component of the WildBook project: See https://github.com/WildMeOrg/.  This project is an actively maintained fork of the popular IBEIS (Image Based Ecological Information System) software suite for wildlife conservation.  The original IBEIS project is maintained by Jon Crall (@Erotemic) at https://github.com/Erotemic/ibeis.  The IBEIS toolkit originally was a wrapper around HotSpotter, which original binaries can be downloaded from: http://cs.rpi.edu/hotspotter/

Currently the system is build around a SQLite database, a web GUI, and matplotlib visualizations. Algorithms employed are: convolutional neural network detection and localization and classification, hessian-affine keypoint detection, SIFT keypoint description, LNBNN identification using approximate nearest neighbors.

Support
--------
While WBIA can be set up independently, the Wild Me team is not able to assist with independent model training. Support is limited to system setup in Docker at this time.

Requirements
------------

* Python 3.7+
* OpenCV 3.4.10
* Python dependencies listed in requirements.txt

Installation Instructions
-------------------------

.. code:: bash

    git clone https://github.com/WildMeOrg/wildbook-ia.git
    cd wildbook-ia/devops
    sudo ./build.sh wbia-base wbia-provision wbia

If your build times out, run the command again. The previous run is cached and the build will advance. Note that you cannot tag this build.

Docker
~~~~~~

The WBIA software is built and deployed as a Docker image `wildme/wbia`.  You can download and run the pre-configured instance from the command line using:

.. code:: bash

    # Install Docker - https://docs.docker.com/engine/install/
    docker pull wildme/wbia:latest
    docker run -p 84:5000 wildme/wbia:latest bash

This image is built using the multi-stage Dockerfiles in `devops/`. Once the container is up and running, go to `localhost:84` to see the WBIA landing page.

Dependencies
~~~~~~

This project depends on an array of other repositories for functionality.

First Party Toolkits (Required)

* https://github.com/WildMeOrg/wbia-utool

* https://github.com/WildMeOrg/wbia-vtool

First Party Dependencies for Third Party Libraries (Required)

* https://github.com/WildMeOrg/wbia-tpl-pyhesaff

* https://github.com/WildMeOrg/wbia-tpl-pyflann

* https://github.com/WildMeOrg/wbia-tpl-pydarknet

* https://github.com/WildMeOrg/wbia-tpl-pyrf

First Party Plug-ins (Optional)

* https://github.com/WildMeOrg/wbia-plugin-cnn

* https://github.com/WildMeOrg/wbia-plugin-flukematch

* https://github.com/WildMeOrg/wbia-plugin-deepsense

* https://github.com/WildMeOrg/wbia-plugin-finfindr

* https://github.com/WildMeOrg/wbia-plugin-curvrank

    + https://github.com/WildMeOrg/wbia-tpl-curvrank

* https://github.com/WildMeOrg/wbia-plugin-kaggle7

    + https://github.com/WildMeOrg/wbia-tpl-kaggle7

* https://github.com/WildMeOrg/wbia-plugin-2d-orientation

    + https://github.com/WildMeOrg/wbia-tpl-2d-orientation

* https://github.com/WildMeOrg/wbia-plugin-lca

    + https://github.com/WildMeOrg/wbia-tpl-lca

Deprecated Toolkits (Deprecated)
* https://github.com/WildMeOrg/wbia-deprecate-ubelt

* https://github.com/WildMeOrg/wbia-deprecate-dtool

* https://github.com/WildMeOrg/wbia-deprecate-guitool

* https://github.com/WildMeOrg/wbia-deprecate-plottool

* https://github.com/WildMeOrg/wbia-deprecate-detecttools

* https://github.com/WildMeOrg/wbia-deprecate-plugin-humpbacktl

* https://github.com/WildMeOrg/wbia-deprecate-tpl-lightnet

* https://github.com/WildMeOrg/wbia-deprecate-tpl-brambox

Plug-in Templates (Reference)

* https://github.com/WildMeOrg/wbia-plugin-template

* https://github.com/WildMeOrg/wbia-plugin-id-example

Miscellaneous (Reference)

* https://github.com/WildMeOrg/wbia-pypkg-build

* https://github.com/WildMeOrg/wbia-project-website

* https://github.com/WildMeOrg/wbia-aws-codedeploy

Citation
--------

If you use this code or its models in your research, please cite:

.. code:: text

    @inproceedings{crall2013hotspotter,
        title={Hotspotter — patterned species instance recognition},
        author={Crall, Jonathan P and Stewart, Charles V and Berger-Wolf, Tanya Y and Rubenstein, Daniel I and Sundaresan, Siva R},
        booktitle={2013 IEEE workshop on applications of computer vision (WACV)},
        pages={230--237},
        year={2013},
        organization={IEEE}
    }

    @inproceedings{parham2018animal,
        title={An animal detection pipeline for identification},
        author={Parham, Jason and Stewart, Charles and Crall, Jonathan and Rubenstein, Daniel and Holmberg, Jason and Berger-Wolf, Tanya},
        booktitle={2018 IEEE Winter Conference on Applications of Computer Vision (WACV)},
        pages={1075--1083},
        year={2018},
        organization={IEEE}
    }

    @inproceedings{berger2015ibeis,
        title={IBEIS: Image-based ecological information system: From pixels to science and conservation},
        author={Berger-Wolf, TY and Rubenstein, DI and Stewart, CV and Holmberg, J and Parham, J and Crall, J},
        booktitle={Bloomberg Data for Good Exchange Conference, New York, NY, USA},
        volume={2},
        year={2015}
    }

    @article{berger2017wildbook,
        title={Wildbook: Crowdsourcing, computer vision, and data science for conservation},
        author={Berger-Wolf, Tanya Y and Rubenstein, Daniel I and Stewart, Charles V and Holmberg, Jason A and Parham, Jason and Menon, Sreejith and Crall, Jonathan and Van Oast, Jon and Kiciman, Emre and Joppa, Lucas},
        journal={arXiv preprint arXiv:1710.08880},
        year={2017}
    }

Documentation
-------------------------

The documentation is built and available online at `wildmeorg.github.io/wildbook-ia/ <http://wildmeorg.github.io/wildbook-ia/>`_. However, if you need to build a local copy of the source, the following instructions can be used.

.. code:: bash

   # checkout the source code
   # install the project in development mode
   pip install -e .
   # build the docs
   scripts/build-docs.sh

Then open the html file at ``docs/build/html/index.html``.

Code Style and Development Guidelines
-------------------------------------

Contributing
~~~~~~~~~~~~

It's recommended that you use ``pre-commit`` to ensure linting procedures are run
on any commit you make. (See also `pre-commit.com <https://pre-commit.com/>`_)

Reference `pre-commit's installation instructions <https://pre-commit.com/#install>`_ for software installation on your OS/platform. After you have the software installed, run ``pre-commit install`` on the command line. Now every time you commit to this project's code base the linter procedures will automatically run over the changed files.  To run pre-commit on files preemtively from the command line use:

.. code:: bash

    git add .
    pre-commit run

    # or

    pre-commit run --all-files

Brunette
~~~~~~~~

Our code base has been formatted by Brunette, which is a fork and more configurable version of Black (https://black.readthedocs.io/en/stable/).

Flake8
~~~~~~

Try to conform to PEP8.  You should set up your preferred editor to use flake8 as its Python linter, but pre-commit will ensure compliance before a git commit is completed.

To run flake8 from the command line use:

.. code:: bash

    flake8

This will use the flake8 configuration within ``setup.cfg``,
which ignores several errors and stylistic considerations.
See the ``setup.cfg`` file for a full and accurate listing of stylistic codes to ignore.

PyTest
~~~~~~

Our code uses Google-style documentation tests (doctests) that uses pytest and xdoctest to enable full support.  To run the tests from the command line use:

.. code:: bash

    pytest

To run doctests with `+REQUIRES(--web-tests)` do:

.. code:: bash

    pytest --web-tests

.. |Build| image:: https://img.shields.io/github/workflow/status/WildMeOrg/wildbook-ia/Build%20and%20upload%20to%20PyPI/main
    :target: https://github.com/WildMeOrg/wildbook-ia/actions?query=branch%3Amain+workflow%3A%22Build+and+upload+to+PyPI%22
    :alt: Build and upload to PyPI

.. |Pypi| image:: https://img.shields.io/pypi/v/wildbook-ia.svg
   :target: https://pypi.python.org/pypi/wildbook-ia
   :alt: Latest PyPI version

.. |Documentation| image:: https://img.shields.io/github/workflow/status/WildMeOrg/wildbook-ia/Build%20documentation?label=documentation
    :target: https://wildmeorg.github.io/wildbook-ia/
    :alt: Documentation on GitHub Pages

.. |Downloads| image:: https://img.shields.io/pypi/dm/wildbook-ia.svg
   :target: https://pypistats.org/packages/wildbook-ia


SuperAnnotate Python SDK
===============================
|Python| |License| |Changelog|

Welcome to the SuperAnnotate Python Software Development Kit (SDK), which enables Python programmers to create software that incorporates services of the platform and effortlessly integrates SuperAnnotate into their AI process.

.. |Python| image:: https://img.shields.io/static/v1?label=python&message=3.7/3.8/3.9/3.10/3.11&color=blue&style=flat-square
    :target: https://pypi.org/project/superannotate/
    :alt: Python Versions
.. |License| image:: https://img.shields.io/static/v1?label=license&message=MIT&color=green&style=flat-square
    :target: https://github.com/superannotateai/superannotate-python-sdk/blob/master/LICENSE/
    :alt: License
.. |Changelog| image:: https://img.shields.io/static/v1?label=change&message=log&color=yellow&style=flat-square
    :target: https://github.com/superannotateai/superannotate-python-sdk/blob/master/CHANGELOG.rst
    :alt: Changelog

Resources
---------------

- API Reference and User Guide available on `Read the Docs <https://superannotate.readthedocs.io/en/stable/index.html>`__
- `Platform documentation <https://doc.superannotate.com/>`__

Authentication
---------------

.. code-block:: python

    from superannotate import SAClient
    # by environment variable SA_TOKEN
    sa_client = SAClient()
    # by token
    sa_client = SAClient(token='<team token>')
    # by config file
    # default path is ~/.superannotate/config.ini
    sa_client = SAClient(config_path='~/.superannotate/dev_config.ini')

config.ini example
------------------
.. code-block:: python

    [DEFAULT]
    SA_TOKEN = <Token>
    LOGGING_LEVEL = INFO

Using superannotate
-------------------

.. code-block:: python

    from superannotate import SAClient

    sa_client =SAClient()

    project = 'Dogs'

    sa_client.create_project(
            project_name=project,
            project_description='Test project generated via SDK',
            project_type='Vector'
        )

    sa_client.create_annotation_class(
        project=project,
        name='dog',
        color='#F9E0FA',
        class_type='tag'
    )

    sa_client.attach_items(
            project=project,
            attachments=[
                {
                    'url': 'https://drive.google.com/uc?export=download&id=1ipOrZNSTlPUkI_hnrW9aUD5yULqqq5Vl',
                    'name': 'dog.jpeg'
                }
            ]
        )

    sa_client.upload_annotations(
            project=project,
            annotations=[
                {
                    'metadata': {'name': 'dog.jpeg'},
                    'instances': [
                        {'type': 'tag', 'className': 'dog'}
                    ]
                }
            ]
        )

    sa_client.get_annotations(project=project, items=['dog.jpeg'])

Installation
------------

SuperAnnotate python SDK is available on PyPI:

.. code-block:: bash

    pip install superannotate

The package officially supports Python 3.7+ and was tested under Linux and
Windows (`Anaconda <https://www.anaconda.com/products/individual#windows>`__
) platforms.

For more detailed installation steps and package usage please have a look at the `tutorial <https://superannotate.readthedocs.io/en/stable/tutorial.sdk.html>`__

Supported Features
------------------

- search/get/create/clone/update/delete projects
- search/get/create/delete folders
- assign folders to project contributors
- upload items to a project from a local or AWS S3 folder
- attach items by URL or from an integrated storage, meanwhile keeping them secure in your cloud provider
- get integrated cloud storages
- upload annotations (also from local or AWS S3 folder)
- delete annotations
- set items annotations statuses
- get/download/export annotations from a project (also to a local or AWS S3 folder)
- invite/search team contributors or add contributors to a specific project
- search/get/copy/move items in a project
- query items using SA Query Language
- define custom metadata for items and upload custom values (query based on your custom metadata)
- upload priority scores
- get available subsets (sets of segregated items), query items in a subset or add items to a subset
- assign or anassign items to project contributors
- download an image that has been uploaded to project
- search/create/download/delete project annotation classes
- search/download models
- run predictions
- convert annotations from/to COCO format
- convert annotation from VOC, SuperVisely, LabelBox, DataLoop, VGG, VoTT, SageMaker, GoogleCloud, YOLO formats
- CLI commands for simple tasks

Questions and Issues
--------------------

For questions and issues please use this repo’s issue tracker on GitHub or contact support@superannotate.com.


# Mitogen

Please see the documentation.

![](https://i.imgur.com/eBM6LhJ.gif)

[![Total alerts](https://img.shields.io/lgtm/alerts/g/mitogen-hq/mitogen.svg?logo=lgtm&logoWidth=18)](https://lgtm.com/projects/g/mitogen-hq/mitogen/alerts/)

[![Build Status](https://dev.azure.com/mitogen-hq/mitogen/_apis/build/status/mitogen-hq.mitogen?branchName=master)](https://dev.azure.com/mitogen-hq/mitogen/_build/latest?definitionId=1&branchName=master)


|ReadTheDocs|_ |TravisCI|_

.. image:: https://badgen.net/badge/Lifecycle/Active/green
    :alt: Lifecycle Active

.. image:: https://img.shields.io/badge/code%20style-black-000000.svg
    :target: https://github.com/psf/black

.. image:: https://github.com/morganstanley/testplan/blob/main/doc/images/logo/testplan.png

a multi-testing framework
-------------------------

*..because unit tests can only go so far..*

Testplan is a `Python <http://python.org>`_ package that can start a local live
environment, setup mocks, connections to services and run tests against these.
It provides:

  * ``MultiTest`` a feature extensive functional testing system with a rich set
    of *assertions* and report rendering logic.
  * Built-in inheritable drivers to create a local live *environment*.
  * Configurable, diverse and expandable test execution mechanism including
    *parallel* execution capability.
  * Test *tagging* for flexible filtering and selective execution as well as
    generation of multiple reports (for each tag combination).
  * Integration with other unit testing frameworks (like GTest).
  * Rich, unified reports (json/PDF/XML) and soon (HTML/UI).

Basic example
=============

This is how a very basic Testplan application looks like.

.. code-block:: python

    import sys

    from testplan import test_plan
    from testplan.testing.multitest import MultiTest, testcase, testsuite

    def multiply(numA, numB):
        return numA * numB

    @testsuite
    class BasicSuite(object):

        @testcase
        def basic_multiply(self, env, result):
            result.equal(multiply(2, 3), 6, description='Passing assertion')
            result.equal(multiply(2, 2), 5, description='Failing assertion')

    @test_plan(name='Multiply')
    def main(plan):
        test = MultiTest(name='MultiplyTest',
                         suites=[BasicSuite()])
        plan.add(test)

    if __name__ == '__main__':
      sys.exit(not main())

Example execution:

.. code-block:: bash

    $ python ./test_plan.py -v
            Passing assertion - Pass
              6 == 6
            Failing assertion - Fail
              File: .../test_plan.py
              Line: 18
              4 == 5
          [basic_multiply] -> Fail
        [BasicSuite] -> Fail
      [MultiplyTest] -> Fail
    [Multiply] -> Fail

System integration testing example
==================================

Testing a server and a client communication.

.. code-block:: python

    import sys

    from testplan import test_plan
    from testplan.testing.multitest import MultiTest, testsuite, testcase
    from testplan.testing.multitest.driver.tcp import TCPServer, TCPClient
    from testplan.common.utils.context import context

    @testsuite
    class TCPTestsuite(object):
        """Testsuite for server client connection testcases."""

        def setup(self, env):
            env.server.accept_connection()

        @testcase
        def send_and_receive_msg(self, env, result):
            """Basic send and receive hello message testcase."""
            msg = env.client.cfg.name
            result.log('Client is sending his name: {}'.format(msg))
            bytes_sent = env.client.send_text(msg)

            received = env.server.receive_text(size=bytes_sent)
            result.equal(received, msg, 'Server received client name')

            response = 'Hello {}'.format(received)
            result.log('Server is responding: {}'.format(response))
            bytes_sent = env.server.send_text(response)

            received = env.client.receive_text(size=bytes_sent)
            result.equal(received, response, 'Client received response')

    @test_plan(name='TCPConnections')
    def main(plan):
        test = MultiTest(name='TCPConnectionsTest',
                         suites=[TCPTestsuite()],
                         environment=[
                             TCPServer(name='server'),
                             TCPClient(name='client',
                                       host=context('server', '{{host}}'),
                                       port=context('server', '{{port}}'))])
        plan.add(test)

    if __name__ == '__main__':
        sys.exit(not main())

Example execution:

.. code-block:: bash

    $ python ./test_plan.py -v
            Client is sending: client
            Server received - Pass
              client == client
            Server is responding: Hello client
            Client received - Pass
              Hello client == Hello client
          [send_and_receive_msg] -> Pass
        [TCPTestsuite] -> Pass
      [TCPConnectionsTest] -> Pass
    [TCPConnections] -> Pass

A persistent and human readable test evidence PDF report:

.. code-block:: bash

    $ python ./test_plan.py --pdf report.pdf
      [TCPConnectionsTest] -> Pass
    [TCPConnections] -> Pass
    PDF generated at report.pdf

.. image:: https://github.com/morganstanley/testplan/blob/main/doc/images/pdf/readme_server_client.png

Documentation
=============

For complete documentation that includes downloadable examples, visit this
`link <http://testplan.readthedocs.io>`_.

Contribution
============

A step by step guide to contribute to Testplan framework can be found
`here <http://testplan.readthedocs.io/en/latest/about.html#contributing>`_.

License
=======

License information `here <https://github.com/morganstanley/testplan/blob/main/LICENSE.md>`_.

.. |ReadTheDocs| image:: https://readthedocs.org/projects/testplan/badge/?version=latest
.. _ReadTheDocs: http://testplan.readthedocs.io/en/latest/
.. |TravisCI| image:: https://api.travis-ci.com/MorganStanley/testplan.svg?branch=main
.. _TravisCI: https://travis-ci.com/morganstanley/testplan


[![Build Status](https://github.com/openelections/openelections-data-or/actions/workflows/data_tests.yml/badge.svg?branch=master)](https://github.com/openelections/openelections-data-or/actions/workflows/data_tests.yml?query=branch%3Amaster)

# OpenElections Data Oregon

This repository contains pre-processed election results from Oregon, formatted to be ingested into the OpenElections [processing pipeline](http://docs.openelections.net/guide/). It contains mostly CSV files converted from PDF tables. Interested in contributing? We have a bunch of [easy tasks](https://github.com/openelections/openelections-data-or/labels/easy%20task) for you to tackle.

Here is what a [finished CSV file (from Ohio)](https://github.com/openelections/openelections-data-oh/blob/master/2000/20001107__oh__general__president.csv) looks like. Note that each row represents a single result for a single candidate,even if the data has multiple candidates in a single row. Also, vote totals do not contain commas or other formatting.

For extracting text from PDF tables,we recommend [Tabula](http://tabula.technology/), which can be installed and run locally on OSX,Windows or Linux.

If you're familiar with git and Github,clone this repository and get started. If not, you can still help: leave a comment on a task you'd like to work on, or just convert any of the files into CSV and send the result to openelections@gmail.com.

## Code License

See the [LICENSE](LICENSE.md) file for license rights and limitations (MIT).


# Google Cloud Platform Extensions for Jupyter and JupyterLab

This repository serves as a common repository for Google-developed Jupyter extensions.

## Disclaimer

This is not an officially supported Google product.

## Contents

### Gcloud config helper for Jupyter extensions

The `google-cloud-jupyter-config` subdirectory contains the source code for the
[google-cloud-jupyter-config](https://pypi.org/project/google-cloud-jupyter-config/)
package.

This intended to be a reusable library that other extensions can use to get configuration
information from the [gcloud command line tool](https://cloud.google.com/cli)

### GCS Contents Manager

The `jupyter-gcs-contents-manager` subdirectory contains a Jupyter Contents Manager
that reads contents from a GCS bucket.

### Kernels Mixer

The `kernels-mixer` subdirectory contains the source code for the
[kernels-mixer](https://pypi.org/project/kernels-mixer/) package.

This is an extension for jupyter_server that allows local and remote kernels to be used
simultaneously.


[![PyPi Release](https://img.shields.io/pypi/v/maestral.svg)](https://pypi.org/project/maestral/)
[![Pyversions](https://img.shields.io/pypi/pyversions/maestral.svg)](https://pypi.org/pypi/maestral/)
[![Documentation Status](https://readthedocs.org/projects/maestral/badge/?version=latest)](https://maestral.readthedocs.io/en/latest/?badge=latest)
[![codecov](https://codecov.io/gh/SamSchott/maestral/branch/master/graph/badge.svg?token=V0C7IQ1MAU)](https://codecov.io/gh/SamSchott/maestral)

# Maestral

A light-weight and open-source Dropbox client for macOS and Linux.

## About

Maestral is an open-source Dropbox client written in Python. The project's main goal is to
provide a client for platforms and file systems that are no longer directly supported by
Dropbox.

Maestral currently does not support Dropbox Paper, the management of Dropbox teams, and
the management of shared folder settings. If you need any of this functionality, please
use the Dropbox website or the official client. Maestral does support syncing
multiple Dropbox accounts and excluding local files from sync with a ".mignore" file.

The focus on "simple" file syncing does come with advantages: on macOS, the Maestral App
bundle is significantly smaller than the official Dropbox app and uses less memory. The
exact memory usage will depend on the size of your synced Dropbox folder and can be further
reduced when running Maestral without a GUI.

Maestral uses the public Dropbox API which, unlike the official client, does not support
transferring only those parts of a file which changed ("binary diff"). Maestral may
therefore use more bandwidth that the official client. However, it will avoid uploading
or downloading a file if it already exists with the same content locally or in the cloud.

## Warning

- Never sync a local folder with both the official Dropbox client and Maestral at the same
  time.
- Network drives and some external hard drives are not supported as locations for the
  Dropbox folder.

## Installation

An app bundle is provided for macOS Catalina and higher and can be downloaded from the
Releases tab. This app Bundle is also package as a Homebrew cask.

On other platforms, you can download and install Maestral as a Python package from PyPI or
as a Docker image from Docker Hub.

For more detailed information on the installation, setup and system requirements, please
check the [documentation](https://maestral.app/docs/installation).

### Homebrew

The official Maestral releases are also available as Homebrew casks. If you have
[Homebrew](https://brew.sh) on your system, you can install using:

```console
$ brew install maestral
```

### Python package using PyPI

Is is recommended to install Maestral inside a virtual environment as follows:

```console
$ python3 -m venv maestral-venv
$ source maestral-venv/bin/activate
(maestral-venv)$ python3 -m pip install --upgrade maestral
```

If you intend to use the graphical user interface, you also need to specify the GUI option
during installation or upgrade. This will install the `maestral-qt` frontend and `PyQt5`
on Linux and `maestral-cocoa` on macOS:

```console
(maestral-venv)$ python3 -m pip install --upgrade 'maestral[gui]'
```

### Docker image

A Docker image is available for x86, arm/v7 (32bit) and arm64 platforms and can be
installed with:

```colsole
$ docker pull maestraldbx/maestral
```

## Usage

Run `maestral gui` in the command line (or open the Maestral app on macOS) to start
Maestral with a graphical user interface. On its first run, Maestral will guide you
through linking and configuring your Dropbox and will then start syncing.

### Command line usage

After installation, Maestral will be available as a command line script by typing
`maestral` in the command prompt. Type `maestral --help` to get a full list of available
commands. The most important are:

- `maestral gui`: Starts the Maestral GUI. Creates a sync daemon if not already running.
- `maestral start|stop`: Starts or stops the Maestral sync daemon.
- `maestral pause|resume`: Pauses or resumes syncing.
- `maestral autostart -Y|-N`: Sets the daemon to start on log in.
- `maestral status`: Gets the current status of Maestral.
- `maestral filestatus LOCAL_PATH`: Gets the sync status of an individual file or folder.
- `maestral excluded add|remove|list`: Command group to manage excluded folders.
- `maestral ls DROPBOX_PATH`: Lists the contents of a directory on Dropbox.
- `maestral notify snooze N`: Snoozes desktop notifications for N minutes.

Maestral supports syncing multiple Dropbox accounts by running multiple instances
with different configuration files. This needs to be configured from the command
line by passing the option `--config-name` to `maestral start` or `maestral gui`.
Maestral will then select an existing config with the given name or create a new one.
For example:

```console
$ maestral start --config-name="personal"
$ maestral start --config-name="work"
```

This will start two instances of Maestral, syncing a private and a work account,
respectively. Configs will be automatically cleared when unlinking an account. You can
list all currently linked accounts with `maestral config-files`. The above setup for
example will return the following on macOS:

```console
$ maestral config-files

Config name  Account          Path
maestral     user@gmail.com   ~/Library/Application Support/maestral/maestral.ini
private      user@mycorp.org  ~/Library/Application Support/maestral/private.ini
```

By default, the Dropbox folder names will contain the capitalised config-name in braces.
In the above case, this will be "Dropbox (Personal)" and "Dropbox (Work)".

A full documentation of the CLI is available on the
[website](https://maestral.app/cli).

## Contribute

There are multiple topics that could use your help. Some of them are easy, such as adding
new CLI commands, others require more experience, such as packaging for non-macOS
platforms. Look out for issues marked with "good first issue" or "help wanted".

Relevant resources are:

- [Maestral API docs](https://maestral.readthedocs.io)
- [Dropbox API docs](https://www.dropbox.com/developers/documentation/http/documentation)
- [Dropbox Python SDK docs](https://dropbox-sdk-python.readthedocs.io/en/latest/)

[CONTRIBUTING.md](CONTRIBUTING.md) contains detailed information on the expected code
style and test format.

If you are using the macOS app bundle, please consider sponsoring the project with £1 per
month to offset the cost of an Apple Developer account to sign and notarize the bundle.

## System requirements

- macOS 10.15 Catalina or higher or Linux
- Python 3.7 or higher
- For the system tray icon on Linux:
  - [gnome-shell-extension-appindicator](https://github.com/ubuntu/gnome-shell-extension-appindicator)
    on Gnome 3.26 and higher


FabSim
======

<!---
[![Build Status](https://travis-ci.com/djgroen/FabSim3.svg?branch=master)](https://travis-ci.com/djgroen/FabSim3)
-->

[![Run Tests](https://github.com/djgroen/FabSim3/actions/workflows/Pytests.yml/badge.svg?branch=master)](https://github.com/djgroen/FabSim3/actions/workflows/Pytests.yml)
[![Docker Pulls](https://img.shields.io/docker/pulls/vecmafabsim3/fabsimdocker.svg)](https://hub.docker.com/r/vecmafabsim3/fabsimdocker/)
[![Docker Pulls](https://img.shields.io/docker/automated/vecmafabsim3/fabsimdocker.svg)](https://hub.docker.com/r/vecmafabsim3/fabsimdocker/)

[![GitHub tag (latest by date)](https://img.shields.io/github/v/tag/djgroen/FabSim3?style=flat)](https://github.com/djgroen/FabSim3/tags)
<!---
[![Language grade: Python](https://img.shields.io/lgtm/grade/python/g/djgroen/FabSim3.svg?logo=lgtm&logoWidth=18)](https://lgtm.com/projects/g/djgroen/FabSim3/context:python)
-->
[![GitHub Issues](https://img.shields.io/github/issues/djgroen/FabSim3.svg)](https://github.com/djgroen/FabSim3/issues)
[![GitHub last-commit](https://img.shields.io/github/last-commit/djgroen/FabSim3.svg)](https://github.com/djgroen/FabSim3/commits/master)

<!---
for master branch use :
[![Build Status](https://travis-ci.com/djgroen/FabSim3.svg?branch=master)](https://travis-ci.com/djgroen/FabSim3))

[![Dockerhub link](https://img.shields.io/badge/hosted-dockerhub-22b8eb.svg)](https://hub.docker.com/r/willprice/nvidia-ffmpeg/)

[![Singularity hub link](https://www.singularity-hub.org/static/img/hosted-singularity--hub-%23e32929.svg)](https://singularity-hub.org/collections/521)
-->

**For the full FabSim3 documentation, please visit https://fabsim3.readthedocs.io**

FabSim is a Python-based automation toolkit for scientific simulation and data processing workflows, licensed under the BSD 3-clause license. It aims to enable users to perform remote tasks from a local command-line, and to run applications while curating the environment variables and the input and output data in a systematic manner. To provide that curation, FabSim uses a basic data transfer functionalities such as rsync and ssh.

FabSim also contains a system for defining machine-specific configurations, including templates to execute jobs through schedulers such as PBSPro, Loadleveller and SGE. These machine-specific configurations are stored in the repository, apply to all applications run on that machine, and can be updated by any contributor who feels that a fix or improvement is required.

FabSim relies strongly on Fabric (http://www.fabfile.org, shown to work with versions 1.5.3 and 1.10.0) and PyYAML. Previous versions of FabSim (most notably FabHemeLB and FabMD) have been used to run simulation workflows on machines such as ARCHER, SuperMUC, BlueJoule, as well as local clusters and desktops.

FabSim is now publicly available at: http://www.github.com/djgroen/FabSim
The accompanying software paper can be found here: https://doi.org/10.1016/j.cpc.2016.05.020

Derivative versions of FabSim include:
* FabHemeLB (previously known as _FabricHemeLB_), which is used to automate workflows involving
the HemeLB lattice-Boltzmann simulation environment (see http://www.github.com/UCL/hemelb for
the source code of that).
* FabMD, which is used to semi-automatically coarse-grain polymer systems (part of this repository).
* FabBioMD, which is used to facilitate protein-ligand binding affinity calculations (part of this repository).
* FabFlee, which is under development and used to automate agent-based simulations of forced migration.

## Installation and usage

For instructions on how to install and test FabSim, please refer to `docs/installation.md`.

## Known issues

* The FabMUSCLE plugin does not yet work correctly with the Eagle Supercomputer.

### Citing FabSim

Please find the BibTex reference below of our FabSim software paper in _Computer Physics Communications_:

```
@article{GROEN2016375,
title = "FabSim: Facilitating computational research through automation on large-scale and distributed e-infrastructures",
journal = "Computer Physics Communications",
volume = "207",
number = "Supplement C",
pages = "375 - 385",
year = "2016",
issn = "0010-4655",
doi = "https://doi.org/10.1016/j.cpc.2016.05.020",
url = "http://www.sciencedirect.com/science/article/pii/S0010465516301448",
author = "Derek Groen and Agastya P. Bhati and James Suter and James Hetherington and Stefan J. Zasada and Peter V. Coveney",
}
```


  PennyLane is a cross-platform Python library for
  quantum computing,
  quantum machine learning,
  and
  quantum chemistry.

  Train a quantum computer the same way as a neural network.

## Key Features

- *Machine learning on quantum hardware*. Connect to quantum hardware using **PyTorch**, **TensorFlow**, **JAX**, **Keras**, or **NumPy**. Build rich and flexible hybrid quantum-classical models.

- *Just in time compilation*. Experimental support for just-in-time
  compilation. Compile your entire hybrid workflow, with support for
  advanced features such as adaptive circuits, real-time measurement
  feedback, and unbounded loops. See
  [Catalyst](https://github.com/pennylaneai/catalyst) for more details.

- *Device-independent*. Run the same quantum circuit on different quantum backends. Install
  [plugins](https://pennylane.ai/plugins.html) to access even more devices, including **Strawberry
  Fields**, **Amazon Braket**, **IBM Q**, **Google Cirq**, **Rigetti Forest**, **Qulacs**, **Pasqal**, **Honeywell**, and more.

- *Follow the gradient*. Hardware-friendly **automatic differentiation** of quantum circuits.

- *Batteries included*. Built-in tools for **quantum machine learning**, **optimization**, and
  **quantum chemistry**. Rapidly prototype using built-in quantum simulators with
  backpropagation support.

## Installation

PennyLane requires Python version 3.9 and above. Installation of PennyLane, as well as all
dependencies, can be done using pip:

```console
python -m pip install pennylane
```

## Docker support

**Docker** support exists for building using **CPU** and **GPU** (Nvidia CUDA
11.1+) images. [See a more detailed description
here](https://pennylane.readthedocs.io/en/stable/development/guide/installation.html#docker).

## Getting started

For an introduction to quantum machine learning, guides and resources are available on
PennyLane's [quantum machine learning hub](https://pennylane.ai/qml/):

* [What is quantum machine learning?](https://pennylane.ai/qml/whatisqml.html)
* [QML tutorials and demos](https://pennylane.ai/qml/demonstrations.html)
* [Frequently asked questions](https://pennylane.ai/faq.html)
* [Key concepts of QML](https://pennylane.ai/qml/glossary.html)
* [QML videos](https://pennylane.ai/qml/videos.html)

You can also check out our [documentation](https://pennylane.readthedocs.io) for [quickstart
guides](https://pennylane.readthedocs.io/en/stable/introduction/pennylane.html) to using PennyLane,
and detailed developer guides on [how to write your
own](https://pennylane.readthedocs.io/en/stable/development/plugins.html) PennyLane-compatible
quantum device.

## Tutorials and demonstrations

Take a deeper dive into quantum machine learning by exploring cutting-edge algorithms on our [demonstrations
page](https://pennylane.ai/qml/demonstrations.html).

All demonstrations are fully executable, and can be downloaded as Jupyter notebooks and Python
scripts.

If you would like to contribute your own demo, see our [demo submission
guide](https://pennylane.ai/qml/demos_submission.html).

## Videos

Seeing is believing! Check out [our videos](https://pennylane.ai/qml/videos.html) to learn about
PennyLane, quantum computing concepts, and more.

## Contributing to PennyLane

We welcome contributions—simply fork the PennyLane repository, and then make a [pull
request](https://help.github.com/articles/about-pull-requests/) containing your contribution. All
contributors to PennyLane will be listed as authors on the releases. All users who contribute
significantly to the code (new plugins, new functionality, etc.) will be listed on the PennyLane
arXiv paper.

We also encourage bug reports, suggestions for new features and enhancements, and even links to cool
projects or applications built on PennyLane.

See our [contributions
page](https://github.com/PennyLaneAI/pennylane/blob/master/.github/CONTRIBUTING.md) and our
[developer hub](https://pennylane.readthedocs.io/en/stable/development/guide.html) for more
details.

## Support

- **Source Code:** https://github.com/PennyLaneAI/pennylane
- **Issue Tracker:** https://github.com/PennyLaneAI/pennylane/issues

If you are having issues, please let us know by posting the issue on our GitHub issue tracker.

We also have a [PennyLane discussion forum](https://discuss.pennylane.ai)—come join the community
and chat with the PennyLane team.

Note that we are committed to providing a friendly, safe, and welcoming environment for all.
Please read and respect the [Code of Conduct](.github/CODE_OF_CONDUCT.md).

## Authors

PennyLane is the work of [many contributors](https://github.com/PennyLaneAI/pennylane/graphs/contributors).

If you are doing research using PennyLane, please cite [our paper](https://arxiv.org/abs/1811.04968):

> Ville Bergholm et al. *PennyLane: Automatic differentiation of hybrid quantum-classical
> computations.* 2018. arXiv:1811.04968

## License

PennyLane is **free** and **open source**, released under the Apache License, Version 2.0.


# 코딩테스트 대비 문제집 with Baekjoon

[![Auto Update README](https://github.com/tony9402/baekjoon/actions/workflows/auto_update.yml/badge.svg)](https://github.com/tony9402/baekjoon/actions/workflows/auto_update.yml)
[![Auto Update README](https://github.com/tony9402/baekjoon/actions/workflows/auto_solution_update.yml/badge.svg)](https://github.com/tony9402/baekjoon/actions/workflows/auto_solution_update.yml)
[![Auto Update README](https://github.com/tony9402/baekjoon/actions/workflows/auto_pick.yml/badge.svg)](https://github.com/tony9402/baekjoon/actions/workflows/auto_pick.yml)

[![하루 방문자 수](https://hits.seeyoufarm.com/api/count/incr/badge.svg?url=https%3A%2F%2Fgithub.com%2Ftony9402%2Fbaekjoon)](https://github.com/tony9402/baekjoon)
[![Discussions](https://img.shields.io/badge/discussions-바로가기-brightgreen)](https://github.com/tony9402/baekjoon/discussions)
[![오늘의 문제](https://img.shields.io/badge/오늘의%20문제%20%28BETA%29-바로가기-112051)](https://github.com/tony9402/baekjoon/blob/main/picked.md)
[![Contribute](https://img.shields.io/badge/집단%20지성%20프로젝트%20-바로가기-18D6A5)](https://github.com/tony9402/algorithm-solutions)
[![Email](https://img.shields.io/badge/문의하기-tony9402@naver.com-0182E0)](mailTo:tony9402@naver.com)

## 코딩테스트 문제집

백준 레포에서 확장을 하여 사이트로 만들었습니다. 필터링, 검색 기능도 있으니 깃헙으로 보는 것보다 조금 더 편할 수 있습니다.

https://algorithm.tony9402.com/

## 집단지성 프로젝트

코딩테스트 문제집 확장을 하여 해당 레포에 솔루션 업로드가 아닌 [알고리즘 솔루션](https://github.com/tony9402/algorithm-solutions)에서 PR을 올려주시면 됩니다. 현재 main 브랜치에 merge된 솔루션은 다 이전을 완료했습니다.

현재 Open된 PR 올려주신 분들도 저 레포에 다시 올려주시면 감사하겠습니다.

## 백준 코딩테스트 문제집

❈  코딩테스트을 준비하시는 분들을 위해 문제집을 만들어봤습니다. ❈

❗ 주의 ❗
 알고리즘을 처음 공부하시는 분들은 [알고리즘 설명 링크 모음](./link_for_study.md) 등을 활용하여 알고리즘 먼저 공부하시기 바랍니다.

이 레포는 꾸준히 업데이트를 할 예정이며 문제집 어느정도 완성 후 풀이를 Python, Java, C++ 3가지 언어를 올릴 예정입니다.
**(C++로 먼저 올릴 예정입니다.)**

## 최근 기업 코딩테스트 알고리즘 분류 정리

[바로가기](./CodingTest.md)

최근 기업 코딩테스트에 나왔던 알고리즘들을 정리해보았습니다.

대부분은 코딩테스트를 본 분들에게 들은거라 몇개가 누락되었거나 잘못된 부분이 있을 수도 있습니다.

## 각 알고리즘 문제집

**❗❗순번은 알고리즘 공부 순서와는 무관합니다.❗❗**

여기에서는 각 알고리즘 개념을 설명하는 것이 없습니다. [알고리즘 설명 링크 모음](./link_for_study.md)

문제 뽑은 기준 : 각 태그에 해당하는 문제(코딩 테스트에 나올 정도) 들을 최대한 많이 뽑고 반드시 풀고 넘어가면 좋은 문제를 체크해놨습니다.

### **❈ 중요❗❗ ❈**

이 레포는 **코딩테스트에 나올만한 유형**에 대한 문제를 모았습니다.
알고리즘 유형이 회사마다 다릅니다. 따라서 아래 알고리즘들을 꼭 다 안풀어도 됩니다.
지원하시는 회사에 나오는 **유형에 맞춰** 골라 푸시기 바랍니다.
(문제집에도 추천 문제도 골라 푸셔도 됩니다.)

| 순번 | Tag                          | 태그                | 문제집    | 추천 문제 수 | 총 문제 수 |  상태             |
| :--: | :--------------------------: | :-----------------: | :------:  | :---------:  | :------: |:---------------:|
| 00 | Data Structure | 자료구조 | [바로가기](./data_structure) | 16 | 25 | ![status][Doing] |
| 01 | Data Structure 2 | 자료구조 2 | [바로가기](./data_structure2) | 11 | 21 | ![status][Doing] |
| 02 | Tree | 트리 | [바로가기](./tree) | 15 | 44 | ![status][Doing] |
| 03 | Math | 수학 | [바로가기](./math) | 18 | 29 | ![status][Doing] |
| 04 | Greedy | 탐욕법 | [바로가기](./greedy) | 27 | 62 | ![status][Doing] |
| 05 | Dynamic Programming 1 | 동적계획법 1 | [바로가기](./dynamic_programming_1) | 27 | 80 | ![status][Doing] |
| 06 | Dynamic Programming 2 | 동적계획법 2 | [바로가기](./dynamic_programming_2) | 28 | 81 | ![status][Doing] |
| 07 | Two Pointer | 투 포인터 | [바로가기](./two_pointer) | 13 | 36 | ![status][Doing] |
| 08 | Implementation | 구현 | [바로가기](./implementation) | 30 | 51 | ![status][Doing] |
| 09 | Graph Traversal | 그래프 탐색 | [바로가기](./graph_traversal) | 31 | 96 | ![status][Doing] |
| 10 | Brute Force | 완전탐색 | [바로가기](./brute_force) | 33 | 94 | ![status][Doing] |
| 11 | Simulation | 시뮬레이션 | [바로가기](./simulation) | 30 | 48 | ![status][Doing] |
| 12 | Binary Search | 이분탐색 | [바로가기](./binary_search) | 20 | 66 | ![status][Doing] |
| 13 | Backtracking | 백트래킹 | [바로가기](./backtracking) | 28 | 75 | ![status][Doing] |
| 14 | Divide and conquer | 분할정복 | [바로가기](./divide_and_conquer) | 08 | 18 | ![status][Doing] |
| 15 | Prefix Sum | 누적 합 | [바로가기](./prefix_sum) | 11 | 32 | ![status][Doing] |
| 16 | String | 문자열 | [바로가기](./string) | 19 | 63 | ![status][Doing] |
| 17 | Shortest Path | 최단거리 | [바로가기](./shortest_path) | 18 | 53 | ![status][Doing] |
| 18 | Topological Sorting | 위상정렬 | [바로가기](./topological_sorting) | 05 | 14 | ![status][Doing] |
| 19 | Disjoint Set | 분리 집합 | [바로가기](./disjoint_set) | 06 | 20 | ![status][Doing] |
| 20 | Minimum Spanning Tree(MST) | 최소 스패닝 트리 | [바로가기](./minimum_spanning_tree) | 08 | 24 | ![status][Doing] |
| 21 | Trie | 트라이 | [바로가기](./trie) | 05 | 09 | ![status][Doing] |
| 22 | Dynamic Programming On Trees | 트리디피 | [바로가기](./dynamic_programming_on_trees) | 04 | 08 | ![status][Doing] |

[현재 진행정도](./status.md)

## 모의 문제집

***이 문제는 알고리즘 분류와 관련 없이 문제를 뽑아놓은 문제입니다.***

**이 부분은 [오늘의 문제](https://github.com/tony9402/baekjoon/blob/main/picked.md)로 대체합니다**

## Contributors

            tony9402

            VSFe

            kim1109123

            cola314

            nnnlog

            Baekjoon

            solved.ac

            Baekjoon

            solved.ac

            Baekjoon

            solved.ac

            Baekjoon

            solved.ac

            Baekjoon

            solved.ac

            inclue

            yeonjungin

            cpprhtn

            seastar105

            Dev-RubinJo

            Baekjoon

            solved.ac

            PRIVATE

            Baekjoon

            solved.ac

            Baekjoon

            solved.ac

            Baekjoon

            solved.ac

            shjeong92

            chj3748

            seung-00

            bn-tw2020

            kjh03160

            Baekjoon

            solved.ac

            Baekjoon

            solved.ac

            Baekjoon

            solved.ac

            Baekjoon

            solved.ac

            Baekjoon

            solved.ac

            CoodingPenguin

            bsm8734

            gkgg123

            simon-hoon

            gusdn3477

            Baekjoon

            solved.ac

            Baekjoon

            solved.ac

            Baekjoon

            solved.ac

            Baekjoon

            solved.ac

            Baekjoon

            solved.ac

            rlawngus0910

            tallua

            kyjun719

            chance0523

            lms0806

            Baekjoon

            solved.ac

            Baekjoon

            solved.ac

            Baekjoon

            solved.ac

            Baekjoon

            solved.ac

            Baekjoon

            solved.ac

            zmrdltl

            cieske

            Rain3321

            sio2whocodes

            kang-hyuck

            Baekjoon

            solved.ac

            Baekjoon

            solved.ac

            Baekjoon

            solved.ac

            Baekjoon

            solved.ac

            Baekjoon

            solved.ac

            witch-factory

            youngjun0627

            suinj8

            juheon-jeong

            Won-JoonChoi

            Baekjoon

            solved.ac

            Baekjoon

            solved.ac

            Baekjoon

            solved.ac

            Baekjoon

            solved.ac

            Baekjoon

            solved.ac

            parkgwangwook

            parkjonggyeong18

            Ryu-JM

            Cho-Hyun-Ji

            HaegyeongKim01

            Baekjoon

            solved.ac

            Baekjoon

            solved.ac

            Baekjoon

            solved.ac

            Baekjoon

            solved.ac

            Baekjoon

            solved.ac

## 업데이트 로그

뽑아야 하는 문제가 많고 이를 보기 좋게 표로 정리하는 과정은 너무 노가다성이 많기 때문에 편하게 작업할 수 있도록 업데이트 하고 있습니다.

    자세히

 - 2021.04.28
   - [오늘의 문제](https://github.com/tony9402/baekjoon/blob/main/picked.md) 뽑히는 문제 난이도 변경
     - 이전 난이도
       - 1번 브론즈 5 ~ 실버 1
       - 2번 골드 5 ~ 골드 3
       - 3번 골드 5 ~ 골드 3
       - 4번 골드 2 ~ 골드 1
     - 변경된 난이도
       - 1번 브론즈 5 ~ 실버 3
       - 2번 실버 2 ~ 골드 4
       - 3번 실버 2 ~ 골드 4
       - 4번 골드 3 ~ 골드 1
 - 2021.04.22 ~ 2021.04.23
   - 코드 리팩토링 일부 완료
     - 나머지는 시간 날때마다 하나씩 할 예정
   - 문제 뽑는 방식 변경
     - 이전 방식에서 [새로운](https://raw.githubusercontent.com/tony9402/baekjoon/main/dynamic_programming_2/list.md) 방식으로 변경
     - 이전 방식 : [추천 문제 여부],[문제 이름],[문제 번호],[난이도],[솔루션 링크]
     - 새로운 방식 : [추천 문제 여부],[문제 번호],[솔루션 링크]
   - 매일 새벽에 자동으로 문제 뽑는 스크립트 제작 및 기능 추가
     - [Solved.ac](https://solved.ac/) 기준 브론즈 5 ~ 골드 1 사이인 문제가 4문제 일정 조건 안에서 랜덤하게 뽑음
     - [오늘 문제](https://github.com/tony9402/baekjoon/blob/main/picked.md) 형식으로 업데이트.
     - [문제 뽑는 스크립트](https://github.com/tony9402/baekjoon/blob/main/scripts/pick_problem.cpp)
   - [Contributors](https://github.com/tony9402/baekjoon#contributors) 정보 만드는 표 스크립트 제작
     - Contributor 정보만 [여기](https://github.com/tony9402/baekjoon/blob/main/markdown/contributor.json)에 넣으면 자동으로 표를 생성

 - 2021.04.20
   - [tree](./tree) 태그 추가
     - 트리와 관련된 문제 뽑았습니다.
   - [Contributors](https://github.com/tony9402/baekjoon#contributors) 디자인을 변경하였습니다.
   - [문제집](https://www.acmicpc.net/workbook/by/tony9402) 전체를 업데이트 하였습니다.
   - 각 문제집에 문제 번호를 볼 수 있도록 추가하였고 문제 번호로도 백준으로 이동할 수 있도록 업데이트 했습니다.

 - 2021.04.03
   - [discussions](https://github.com/tony9402/baekjoon/discussions) 추가

 - 2021.04.02
   - [README.md](https://github.com/tony9402/baekjoon/blob/main/README.md) 자동 생성 스크립트 제작
     - 각 알고리즘 문제는 자동으로 추가되고 [status.md](https://github.com/tony9402/baekjoon/blob/main/status.md)에 자동으로 카운팅되지만 README.md에서는 안되는 현상 발견하여 해결
   - 알고리즘 문제 제목 틀린거 자동으로 찾아 수정
   - 일정 주기마다 알고리즘 난이도 갱신
   - 문제 정보를 담는 json을 만듦

 - 2021.03.31
   - [Github Actions](https://github.com/tony9402/baekjoon/actions) 도입
   - [solved](http://solved.ac)의 난이도 변경을 일정 시간마다 체크하여 자동 업데이트 (백준 문제집은 X)
   - 난이도 변경은 [로그](https://github.com/tony9402/baekjoon/blob/main/change_level.log)로 남겨놓음
   - 솔루션 파일만 폴더 형식에만 맞추면 알아서 README 업데이트(솔루션 링크 연결 등)

 - 2021.03.27
   - README.md 업데이트를 더더욱 자동화를 위해 README.md 위에 설명 부분을 header.md로 분리

 - 2021.03.15
   - 추천 문제 [solution](https://github.com/tony9402/baekjoon/tree/main/solution) (C++ 기준) 완성도 정리하는 스크립트 생성

 - 2021.01.15
   - 문제 [이렇게](https://raw.githubusercontent.com/tony9402/baekjoon/main/math/list.md) 뽑으면 자동으로 markdown 테이블을 만들어주는 스크립트 제작

## TODO

  자세히

- [x] 코드 리팩토링
- [x] Contributors 추가 스크립트화
- [ ] 문제 뽑으면 Discussions 또는 Issue에서 바로 간단한 명령어로 추가 기능(스크립트화)
- [x] 문제 추가 방식 변경

[Backtracking]: ./backtracking
[Binary Search]: ./binary_search
[Data Structure]: ./data_structure
[Data Structure2]: ./data_structure2
[Math]: ./math
[Greedy]: ./greedy
[DP1]: ./dynamic_programming_1
[DP2]: ./dynamic_programming_2
[MST]: ./minimum_spanning_tree
[Two Pointer]: ./two_pointer
[Topological Sorting]: ./topological_sorting
[Implementation]: ./implementation
[Graph Traversal]: ./graph_traversal
[Simulation]: ./simulation
[DFS]: ./dfs
[BFS]: ./bfs
[Brute Force]: ./brute_force
[Disjoint Set]: ./disjoint_set
[Trie]: ./trie
[TreeDP]: ./dynamic_programming_on_trees
[Shortest Path]: ./shortest_path
[Prefix Sum]: ./prefix_sum
[Divide and conquer]: ./divide_and_conquer
[String]: ./string
[Tree]: ./tree
[TODO]: https://img.shields.io/badge/-TODO-DFFD26
[DOING]: https://img.shields.io/badge/-DOING-31AE0F
[DONE]: https://img.shields.io/badge/-DONE-0885CC


PyAutoFit: Classy Probabilistic Programming
===========================================

.. |binder| image:: https://mybinder.org/badge_logo.svg
   :target: https://mybinder.org/v2/gh/Jammy2211/autofit_workspace/HEAD

.. |RTD| image:: https://readthedocs.org/projects/pyautofit/badge/?version=latest
    :target: https://pyautofit.readthedocs.io/en/latest/?badge=latest
    :alt: Documentation Status

.. |Tests| image:: https://github.com/rhayes777/PyAutoFit/actions/workflows/main.yml/badge.svg
   :target: https://github.com/rhayes777/PyAutoFit/actions

.. |Build| image:: https://github.com/rhayes777/PyAutoBuild/actions/workflows/release.yml/badge.svg
   :target: https://github.com/rhayes777/PyAutoBuild/actions

.. |JOSS| image:: https://joss.theoj.org/papers/10.21105/joss.02550/status.svg
   :target: https://doi.org/10.21105/joss.02550

|binder| |Tests| |Build| |RTD| |JOSS|

`Installation Guide <https://pyautofit.readthedocs.io/en/latest/installation/overview.html>`_ |
`readthedocs <https://pyautofit.readthedocs.io/en/latest/index.html>`_ |
`Introduction on Binder <https://mybinder.org/v2/gh/Jammy2211/autofit_workspace/release?filepath=notebooks/overview/overview_1_the_basics.ipynb>`_ |
`HowToFit <https://pyautofit.readthedocs.io/en/latest/howtofit/howtofit.html>`_

**PyAutoFit** is a Python based probabilistic programming language for model fitting and Bayesian inference
of large datasets.

The basic **PyAutoFit** API allows us a user to quickly compose a probabilistic model and fit it to data via a
log likelihood function, using a range of non-linear search algorithms (e.g. MCMC, nested sampling).

Users can then set up **PyAutoFit** scientific workflow, which enables streamlined modeling of small
datasets with tools to scale up to large datasets.

**PyAutoFit** supports advanced statistical methods, most
notably `a big data framework for Bayesian hierarchical analysis <https://pyautofit.readthedocs.io/en/latest/features/graphical.html>`_.

Getting Started
---------------

The following links are useful for new starters:

- `The PyAutoFit readthedocs <https://pyautofit.readthedocs.io/en/latest>`_, which includes an `installation guide <https://pyautofit.readthedocs.io/en/latest/installation/overview.html>`_ and an overview of **PyAutoFit**'s core features.

- `The introduction Jupyter Notebook on Binder <https://mybinder.org/v2/gh/Jammy2211/autofit_workspace/release?filepath=notebooks/overview/overview_1_the_basics.ipynb>`_, where you can try **PyAutoFit** in a web browser (without installation).

- `The autofit_workspace GitHub repository <https://github.com/Jammy2211/autofit_workspace>`_, which includes example scripts and the `HowToFit Jupyter notebook lectures <https://github.com/Jammy2211/autofit_workspace/tree/master/notebooks/howtofit>`_ which give new users a step-by-step introduction to **PyAutoFit**.

Support
-------

Support for installation issues, help with Fit modeling and using **PyAutoFit** is available by
`raising an issue on the GitHub issues page <https://github.com/rhayes777/PyAutoFit/issues>`_.

We also offer support on the **PyAutoFit** `Slack channel <https://pyautoFit.slack.com/>`_, where we also provide the
latest updates on **PyAutoFit**. Slack is invitation-only, so if you'd like to join send
an `email <https://github.com/Jammy2211>`_ requesting an invite.

HowToFit
--------

For users less familiar with Bayesian inference and scientific analysis you may wish to read through
the **HowToFits** lectures. These teach you the basic principles of Bayesian inference, with the
content pitched at undergraduate level and above.

A complete overview of the lectures `is provided on the HowToFit readthedocs page <https://pyautofit.readthedocs.io/en/latest/howtofit/howtofit.htmll>`_

API Overview
------------

To illustrate the **PyAutoFit** API, we use an illustrative toy model of fitting a one-dimensional Gaussian to
noisy 1D data. Here's the ``data`` (black) and the model (red) we'll fit:

.. image:: https://raw.githubusercontent.com/rhayes777/PyAutoFit/master/files/toy_model_fit.png
  :width: 400

We define our model, a 1D Gaussian by writing a Python class using the format below:

.. code-block:: python

    class Gaussian:

        def __init__(
            self,
            centre=0.0,        # <- PyAutoFit recognises these
            normalization=0.1, # <- constructor arguments are
            sigma=0.01,        # <- the Gaussian's parameters.
        ):
            self.centre = centre
            self.normalization = normalization
            self.sigma = sigma

        """
        An instance of the Gaussian class will be available during model fitting.

        This method will be used to fit the model to data and compute a likelihood.
        """

        def model_data_1d_via_xvalues_from(self, xvalues):

            transformed_xvalues = xvalues - self.centre

            return (self.normalization / (self.sigma * (2.0 * np.pi) ** 0.5)) * \
                    np.exp(-0.5 * (transformed_xvalues / self.sigma) ** 2.0)

**PyAutoFit** recognises that this Gaussian may be treated as a model component whose parameters can be fitted for via
a non-linear search like `emcee <https://github.com/dfm/emcee>`_.

To fit this Gaussian to the ``data`` we create an Analysis object, which gives **PyAutoFit** the ``data`` and a
``log_likelihood_function`` describing how to fit the ``data`` with the model:

.. code-block:: python

    class Analysis(af.Analysis):

        def __init__(self, data, noise_map):

            self.data = data
            self.noise_map = noise_map

        def log_likelihood_function(self, instance):

            """
            The 'instance' that comes into this method is an instance of the Gaussian class
            above, with the parameters set to values chosen by the non-linear search.
            """

            print("Gaussian Instance:")
            print("Centre = ", instance.centre)
            print("normalization = ", instance.normalization)
            print("Sigma = ", instance.sigma)

            """
            We fit the ``data`` with the Gaussian instance, using its
            "model_data_1d_via_xvalues_from" function to create the model data.
            """

            xvalues = np.arange(self.data.shape[0])

            model_data = instance.model_data_1d_via_xvalues_from(xvalues=xvalues)
            residual_map = self.data - model_data
            chi_squared_map = (residual_map / self.noise_map) ** 2.0
            log_likelihood = -0.5 * sum(chi_squared_map)

            return log_likelihood

We can now fit our model to the ``data`` using a non-linear search:

.. code-block:: python

    model = af.Model(Gaussian)

    analysis = Analysis(data=data, noise_map=noise_map)

    emcee = af.Emcee(nwalkers=50, nsteps=2000)

    result = emcee.fit(model=model, analysis=analysis)

The ``result`` contains information on the model-fit, for example the parameter samples, maximum log likelihood
model and marginalized probability density functions.


# PYTHON-AMAZON-SP-API

## Amazon Selling-Partner API

A wrapper to access **Amazon's Selling Partner API** with an easy-to-use interface.

##### Supports [Data Kiosk](https://developer-docs.amazon.com/sp-api/v0/docs/data-kiosk-api-v2023-11-15-use-case-guide)

---

### Version 1 Upgrade notice

Version 1 removes AWS IAM or AWS Signature Version 4 authentication.
You can now use the library without AWS credentials.
For compatibility reasons, you can still pass AWS credentials, but they are silently ignored.

### Q & A

If you have questions, please ask them in GitHub discussions

[![discussions](https://img.shields.io/badge/github-discussions-brightgreen?style=for-the-badge&logo=github)](https://github.com/saleweaver/python-amazon-sp-api/discussions)

or

[![join on slack](https://img.shields.io/badge/slack-join%20on%20slack-orange?style=for-the-badge&logo=slack)](https://join.slack.com/t/sellingpartnerapi/shared_invite/zt-zovn6tch-810j9dBPQtJsvw7lEXSuaQ)

---

### Donate

[:heart: Donate](https://github.com/sponsors/saleweaver)

### Installation
[![Badge](https://img.shields.io/pypi/v/python-amazon-sp-api?style=for-the-badge)](https://pypi.org/project/python-amazon-sp-api/)
```
pip install python-amazon-sp-api
pip install "python-amazon-sp-api[aws]" # if you want to use AWS Secret Manager Authentication.
pip install "python-amazon-sp-api[aws-caching]" # if you want to use the Cached Secrets from AWS
```

---
### Usage

```python
from sp_api.api import Orders
from sp_api.api import Reports
from sp_api.api import DataKiosk
from sp_api.api import Feeds
from sp_api.base import SellingApiException
from sp_api.base.reportTypes import ReportType
from datetime import datetime, timedelta

# DATA KIOSK API
client = DataKiosk()

res = client.create_query(query="{analytics_salesAndTraffic_2023_11_15{salesAndTrafficByAsin(startDate:\"2022-09-01\" endDate:\"2022-09-30\" aggregateBy:SKU marketplaceIds:[\"ATVPDKIKX0DER\"]){childAsin endDate marketplaceId parentAsin sales{orderedProductSales{amount currencyCode}totalOrderItems totalOrderItemsB2B}sku startDate traffic{browserPageViews browserPageViewsB2B browserPageViewsPercentage browserPageViewsPercentageB2B browserSessionPercentage unitSessionPercentageB2B unitSessionPercentage}}}}")
print(res)

# orders API
try:
    res = Orders().get_orders(CreatedAfter=(datetime.utcnow() - timedelta(days=7)).isoformat())
    print(res.payload)  # json data
except SellingApiException as ex:
    print(ex)

# report request
createReportResponse = Reports().create_report(reportType=ReportType.GET_MERCHANT_LISTINGS_ALL_DATA)

# submit feed
# feeds can be submitted like explained in Amazon's docs, or simply by calling submit_feed

Feeds().submit_feed(<feed_type>, <file_or_bytes_io>, content_type='text/tsv', **kwargs)

# PII Data

Orders(restricted_data_token='<token>').get_orders(CreatedAfter=(datetime.utcnow() - timedelta(days=7)).isoformat())

# or use the shortcut
orders = Orders().get_orders(
    RestrictedResources=['buyerInfo', 'shippingAddress'],
    LastUpdatedAfter=(datetime.utcnow() - timedelta(days=1)).isoformat()
)
```

---

### Documentation

Documentation is available [here](https://python-amazon-sp-api.readthedocs.io/en/latest/)

[![Documentation Status](https://img.shields.io/readthedocs/python-amazon-sp-api?style=for-the-badge)](https://python-amazon-sp-api.readthedocs.io/en/latest/index.html)

### New endpoints

You can create a new endpoint file by running `make_endpoint <model_json_url>`

```bash
make_endpoint https://raw.githubusercontent.com/amzn/selling-partner-api-models/main/models/listings-restrictions-api-model/listingsRestrictions_2021-08-01.json
```

This creates a ready to use client. Please consider creating a pull request with the new code.

### ADVERTISING API

You can use nearly the same client for the Amazon Advertising API. [@denisneuf](https://github.com/denisneuf) has built [Python-Amazon-Advertising-API](https://github.com/denisneuf/python-amazon-ad-api) on top of this client.
Check it out [here](https://github.com/denisneuf/python-amazon-ad-api)

### DISCLAIMER

We are not affiliated with Amazon

### LICENSE

![License](https://img.shields.io/github/license/saleweaver/python-amazon-sp-api?style=for-the-badge)

---

### Base Client

The client is pretty extensible and can be used for any other API. Check it out here:

[API Client](https://github.com/saleweaver/rapid_rest_client)

---
[![Quality gate](https://sonarcloud.io/api/project_badges/quality_gate?project=saleweaver_python-amazon-sp-api)](https://sonarcloud.io/summary/new_code?id=saleweaver_python-amazon-sp-api)

[![Maintainability Rating](https://sonarcloud.io/api/project_badges/measure?project=saleweaver_python-amazon-sp-api&metric=sqale_rating)](https://sonarcloud.io/summary/new_code?id=saleweaver_python-amazon-sp-api)
[![Reliability Rating](https://sonarcloud.io/api/project_badges/measure?project=saleweaver_python-amazon-sp-api&metric=reliability_rating)](https://sonarcloud.io/summary/new_code?id=saleweaver_python-amazon-sp-api)
[![Coverage](https://sonarcloud.io/api/project_badges/measure?project=saleweaver_python-amazon-sp-api&metric=coverage)](https://sonarcloud.io/summary/new_code?id=saleweaver_python-amazon-sp-api)

[![Bugs](https://sonarcloud.io/api/project_badges/measure?project=saleweaver_python-amazon-sp-api&metric=bugs)](https://sonarcloud.io/summary/new_code?id=saleweaver_python-amazon-sp-api)

[//]: # ([![Code Smells](https://sonarcloud.io/api/project_badges/measure?project=saleweaver_python-amazon-sp-api&metric=code_smells)](https://sonarcloud.io/summary/new_code?id=saleweaver_python-amazon-sp-api))

[//]: # ([![Technical Debt](https://sonarcloud.io/api/project_badges/measure?project=saleweaver_python-amazon-sp-api&metric=sqale_index)](https://sonarcloud.io/summary/new_code?id=saleweaver_python-amazon-sp-api))

[![Security Rating](https://sonarcloud.io/api/project_badges/measure?project=saleweaver_python-amazon-sp-api&metric=security_rating)](https://sonarcloud.io/summary/new_code?id=saleweaver_python-amazon-sp-api)
[![Lines of Code](https://sonarcloud.io/api/project_badges/measure?project=saleweaver_python-amazon-sp-api&metric=ncloc)](https://sonarcloud.io/summary/new_code?id=saleweaver_python-amazon-sp-api)
[![Duplicated Lines (%)](https://sonarcloud.io/api/project_badges/measure?project=saleweaver_python-amazon-sp-api&metric=duplicated_lines_density)](https://sonarcloud.io/summary/new_code?id=saleweaver_python-amazon-sp-api)

---
[![Downloads](https://static.pepy.tech/badge/python-amazon-sp-api)](https://pepy.tech/project/python-amazon-sp-api)
[![Downloads](https://static.pepy.tech/badge/python-amazon-sp-api/month)](https://pepy.tech/project/python-amazon-sp-api)
[![Downloads](https://static.pepy.tech/badge/python-amazon-sp-api/week)](https://pepy.tech/project/python-amazon-sp-api)


Are We Compressed Yet?
====
This repository contains the arewecompressedyet.com website source code.

Running your own local copy of the website
===
To run a local copy, you will need a copy of node and npm.

You will need node, npm, etc. For Ubuntu:
```
sudo apt install nodejs npm nodejs-legacy libicu-dev
```
You'll also need tools to build AV1 or other codecs being tested:
```
sudo apt install yasm libtool
```

First, run the `./setup.sh` script. It will create directories needed for AWCY to run.

Next, create a configuration file called `config.json`.

Here is an example config.json:

```
{ "channel": "#daalatest",
  "have_aws": false,
  "port": 3000,
  "rd_server_url": "http://localhost:4000"
}
```

You will also need a file called `secret_key` which contains the key needed to use the website.

```
echo 'fake_password_to_compile' > secret_key
```

These commands will create the configuration files and install the node.js modules that get used by
awcy. Open a node command line and run the following:

```
mkdir runs
cd www
npm install
cd ..
npm install
npm run tsc
node generate_list.js
npm run start
```

To run the server, execute the run_awcy.bat file
or run the following in your command line:
```
  node awcy_server.js
```
Now you can open localhost:3000 with your browser to see your local version of the website.

Setting up repositories
===
For the website to build codecs, you need local checkouts of every codec. While the `./setup.sh` script
automatically adds AV1, you can add your own. For example:
```
git clone https://aomedia.googlesource.com/aom av1
ln -s av1 av1-rt
```

Setting up rd_server
===
The AWCY web server manages the repositories and runs/ directory, and compiles and builds the codecs. Once this is done, it hands off the job of actually talking to all the AWS machines to rd_server.

To install rd_server, checkout the rd_tool repository in the same directory as awcy:
```
git clone https://github.com/tdaede/rd_tool.git
```

Then start the rd_server.py daemon:
```
./rd_server.py
```
The rd_server.py daemon listens on port 4000 by default.

More documentation on rd_server.py can be found in its README.

Setting up workers
===
Workers are Linux machines accessible over ssh. They need to be the same architecture as the AWCY server.

A worker needs a AWCY work root, which has a directory structure as follows:
```
dump_ciede2000/
daalatool/
slot0/
vmaf/
```

The slot* directores are created by AWCY for jobs.

daalatool must be pre-populated by a clone of the daala repo, with tools built (make tools):

```
git clone https://github.com/xiph/daala.git daalatool
cd daalatool
./autogen.sh
./configure
make
make tools
```

dump_ciede2000 must be populated by a built version of the dump_ciede2000 tool:

```
git clone https://github.com/KyleSiefring/dump_ciede2000
cd dump_ciede2000
cargo build --release
```

vmaf is a clone of the Netflix VMAF repository, also built.

```
git clone https://github.com/Netflix/vmaf.git
cd vmaf/libvmaf
meson build --buildtype release
ninja -C build
```

In addition, a copy of the test media must be accessible to the worker.

The workers should be configured on the main server with a machines.json file, which contains the host, user and port (for ssh). In addition, it contains the number of cores (which controlls the number of slots), plus the work root (containing the above mentioned directores) and the test media path. For example:

```
[
  {
    "host": "localhost",
    "user": "xiph",
    "cores": 8,
    "port": 22,
    "work_root": "/home/xiph/awcy_temp",
    "media_path": "/home/xiph/sets/"
  }
]
```

Each worker must have a unique work root - usually local storage on the worker. One worker is intended to be one physical machine or VM. For testing purposes, one machine can act as multiple workers, but the work_root must be independent. The media path is read-only and can be shared, e.g. via NFS.

The machines should be accesible via passwordless SSH authentication. If needed, the SSH_PRIVKEY_FILE environment variable can inform rd_server.py of a custom private key (.pem format) to use for logging into workers.

Run database format
===
The runs/ directory will contain all of the output files generated from a job. There is a info.json file that specifies what options were used by that particular run. Here is an example of an info.json file:

  {"codec":"daala","commit":"","run_id":"2014-09-19T22-00-08.196Z","task":"video-1-short","nick":"AWCY","task_type":"video"}

There is also an output.txt file that contains the output of the rd_tool.py script.

After each run, a cache file called list.json file is generated by the generate_list.js script. This contains all of the info.json files, as an ordered list. This should probably be replaced by a "real" database at some point.

Docker support
===

AWCY can be started as a docker container for local usage or development.

Build (server)
----

The docker image is pretty large (3.3 GB), as it includes:

 * build dependencies for all codecs to be built (gcc/g++, autotools, make, rust, ...)
 * build dependencies for the WebUI (nodejs, emscripten, ...)
 * runtime dependencies for build and analysis scripts (python, daalatool, ...)

Note: this image has not been optimized for size yet, and may be split in the future.

To build the image:

```sh
docker build --tag xiph/awcy:latest .
```

Build (worker)
----

Another dockerfile (`Dockerfile.worker`) is available to create worker nodes.

To build the worker image:

```sh
docker build --file Dockerfile.worker --tag xiph/awcy-worker:latest .
```

Run (all-in-one)
----

You can run an all-in-one container which will start:

 * awcy server
 * rd_server
 * sshd (to emulate a local rd_tool worker)

In addition, a git clone of every codecs' source is going to be done from upstream at initial startup.

Configuration is driven by a few environment variables (with their default values):

 * `CONFIG_DIR=/data/conf`: configuration files, sqlite database, generate SSH keys, set list, ...
 * `CODECS_SRC_DIR=/data/src`: where codecs' sources are going to be cloned (and compiled)
 * `RUNS_DST_DIR=/data/runs`: rd_tool jobs runs output
 * `WORK_DIR=/data/work`: rd_tool job workdir directory
 * `MEDIAS_SRC_DIR=/data/media`: directory containing media files (get them from https://media.xiph.org/sets/)
 * `LOCAL_WORKER_ENABLED=false`: if a local worker need to be started and configured
 * `LOCAL_WORKER_SLOTS=$(nproc)`: number of slots for the local worker
 * `IRC_CHANNEL=none`: IRC notifications target channel (`none` means disabled)
 * `AWCY_API_KEY=awcy_api_key`: WebUI/API key
 * `AWCY_SERVER_PORT=3000`: awcy server listening port
 * `RD_SERVER_PORT=4000`: rd_server listening port

Simple run example (all data will be contained on the docker host in `/tmp/awcy`):

```sh
docker run \
	-it \
	--rm \
	--name xiph-awcy \
	--publish 3000:3000 \
	--volume /tmp/awcy:/data \
	--volume ${HOME}/xiph-media-files:/media \
	--env MEDIAS_SRC_DIR=/media \
	--env AWCY_API_KEY=complicated_password \
	--env LOCAL_WORKER_ENABLED=true \
	--env LOCAL_WORKER_SLOTS=4 \
	xiph/awcy:latest
```

Output:

```
--2019-01-16 22:46:11--  https://people.xiph.org/~tdaede/sets/subset1-monochrome/Claudette_14_july_2003_1920Z.y4m
Resolving people.xiph.org (people.xiph.org)... 140.211.15.28, 2001:470:eb26:54::1
Connecting to people.xiph.org (people.xiph.org)|140.211.15.28|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 1485050 (1.4M) [application/octet-stream]
Saving to: ‘/media/awcy-builder-quicktest/test_frame.y4m’

/media/awcy-builder-quicktest/test_frame.y4m         100%[=====================================================================================================================>]   1.42M  1.14MB/s    in 1.2s

2019-01-16 22:46:13 (1.14 MB/s) - ‘/media/awcy-builder-quicktest/test_frame.y4m’ saved [1485050/1485050]

Cloning into '/data/src/av1'...
remote: Sending approximately 132.14 MiB ...
remote: Counting objects: 8, done
remote: Finding sources: 100% (5/5)
remote: Total 193273 (delta 156479), reused 193273 (delta 156479)
Receiving objects: 100% (193273/193273), 132.11 MiB | 2.33 MiB/s, done.
Resolving deltas: 100% (156479/156479), done.
Cloning into '/data/src/daala'...
remote: Enumerating objects: 12671, done.
remote: Total 12671 (delta 0), reused 0 (delta 0), pack-reused 12671
Receiving objects: 100% (12671/12671), 8.70 MiB | 2.37 MiB/s, done.
Resolving deltas: 100% (9442/9442), done.
Cloning into '/data/src/rav1e'...
remote: Enumerating objects: 1, done.
remote: Counting objects: 100% (1/1), done.
remote: Total 4426 (delta 0), reused 0 (delta 0), pack-reused 4425
Receiving objects: 100% (4426/4426), 2.61 MiB | 2.33 MiB/s, done.
Resolving deltas: 100% (3108/3108), done.
Cloning into '/data/src/thor'...

(...)

Generating SSH host keys
Generating public/private rsa key pair.
Your identification has been saved in /data/conf/awcy.pem.
Your public key has been saved in /data/conf/awcy.pem.pub.
The key fingerprint is:
SHA256:TJa3T7FZhByGZ3owJOtV/OXk75nyBJa8I7pq2BwNEaU xiph@4ee6bf6c8ffe
The key's randomart image is:
+---[RSA 2048]----+
|       .+o.++o.  |
|       ..++o*.  o|
|       E* o*...= |
|       * o..o=o o|
|        S ..+=  .|
|       . . o. o .|
|      + .  ..o oo|
|     . +  . ..oo.|
|      ...o.   o. |
+----[SHA256]-----+
STARTING SSHD SERVICE
STARTING AWCY SERVICE
STARTING RDTOOL SERVICE
AWCY server started! Open a browser at http://172.20.0.12:3000
AWCY server directory: /opt/app
Configuration directory: /data/conf
Codecs git repositories location: /data/src
Media samples directory: /media
Runs output directory: /data/runs
```

If you open your browser at http://localhost:3000 (or the one in the container output), you should see the AWCY WebUI.

On initial startup, the following configuration files are generated:

```
/tmp/awcy/conf/awcy.pem
/tmp/awcy/conf/awcy.pub
/tmp/awcy/conf/config.json
/tmp/awcy/conf/list.json
/tmp/awcy/conf/machines.json
/tmp/awcy/conf/secret_key
/tmp/awcy/conf/sets.json
/tmp/awcy/conf/subjective.sqlite3
```

Run (with workers)
----

You first need to start an AWCY server instance using the [all-in-one instructions](#run-all-in-one), but setting `LOCAL_WORKER_ENABLED` to false.

Next, run once instance of the worker container on each worker machine. FOr testing, you can also run multiple workers on your local machine. In a second terminal, start one or more worker instances:

```sh
docker run \
	-it \
	--detach \
	--name xiph-awcy-worker-1 \
	--volume ${HOME}/xiph-media-files:/media \
	--env SSH_PUBKEY="$(cat /tmp/awcy/conf/awcy.pub)"
	xiph/awcy-worker:latest

docker run \
	-it \
	--detach \
	--name xiph-awcy-worker-2 \
	--volume ${HOME}/xiph-media-files:/media \
	--env SSH_PUBKEY="$(cat /tmp/awcy/conf/awcy.pub)"
	xiph/awcy-worker:latest

...
```

Note that the given public key is the server-generated one.

Then stop the server instance by hitting `CTRL+C` in its terminal.

Get IP addresses of your worker nodes:

```sh
docker inspect xiph-awcy-worker-1 | jq -r '.[].NetworkSettings.IPAddress'
172.20.0.12

docker inspect xiph-awcy-worker-2 | jq -r '.[].NetworkSettings.IPAddress'
172.20.0.13
...
```

You can now edit the /tmp/awcy/conf/machines.json (paths should be left as-is):

```
cat >/tmp/awcy/conf/machines.json <<EOF
[
  {
    "host": "172.20.0.12",
    "user": "xiph",
    "cores": 8,
    "port": 22,
    "work_root": "/data/work",
    "media_path": "/media"
  },
  {
    "host": "172.20.0.13",
    "user": "xiph",
    "cores": 8,
    "port": 22,
    "work_root": "/data/work",
    "media_path": "/media"
  }
]
EOF
```

Then start the server again using the previously used command.

You should now have two (or more) workers accessible.

Samples
----

To make sure that everything is working properly, you can try a run using the `awcy-builder-quicktest` set, which runs in one minute or so.

Cleanup
----

You can stop the container by hitting `CTRL+C`.

If you have started your container using the above example, all data is contained in `/tmp/awcy`, so you can removed this directory from your host.


# Penn Clubs

[![Build and Deploy](https://github.com/pennlabs/penn-clubs/workflows/Build%20and%20Deploy/badge.svg)](https://github.com/pennlabs/penn-clubs/actions)
[![Coverage Status](https://codecov.io/gh/pennlabs/penn-clubs/branch/master/graph/badge.svg)](https://codecov.io/gh/pennlabs/penn-clubs)

Official platform for club discovery, recruitment, and events at Penn.
React/Next.js frontend and Django-based REST API.

## Installation

You will need to start both the backend and the frontend to do Penn Clubs development.

You will need to start both the backend and the frontend to develop on Penn Clubs. Clubs supports Mac and Linux/WSL development.

Questions? Check out our [extended guide](https://github.com/pennlabs/penn-clubs/wiki/Development-Guide) for FAQs.

### Backend

Running the backend requires [Python 3.11](https://www.python.org/downloads/) and [Pipenv](https://pipenv.pypa.io/en/latest/).

In production, you will need to set the following environment variables:

- `NEXT_PUBLIC_SITE_NAME` (optional, defaults to `clubs`)
- `SECRET_KEY`
- `SENTRY_URL`
- `AWS_ACCESS_KEY_ID`
- `AWS_SECRET_ACCESS_KEY`
- `AWS_STORAGE_BUCKET_NAME`
- `LABS_REDIRECT_URI`
- `LABS_CLIENT_ID` (from Platform)
- `LABS_CLIENT_SECRET` (from Platform)

To run the server, `cd` to the folder where you cloned `penn-clubs`. Then run:

- `cd backend`

Setting up `psycopg2` (this is necessary if you want to be able to modify
dependencies, you can revisit later if not)

- Mac
  - `$ brew install postgresql`
  - `$ brew install openssl`
  - `$ brew unlink openssl && brew link openssl --force`
  - `$ echo 'export PATH="/usr/local/opt/openssl@3/bin:$PATH"' >> ~/.zshrc`
  - `$ export LDFLAGS="-L/usr/local/opt/openssl@3/lib"`
  - `$ export CPPFLAGS="-I/usr/local/opt/openssl@3/include"`
- Linux/WSL
  - `$ apt-get install gcc python3-dev libpq-dev`

Now, you can run

- `$ pipenv install` to install Python dependencies. This may take a few
  minutes. Include the `--dev` argument if you are installing locally
  for development. If you skipped installing `psycopg2` earlier, you might see
  an error with locking -- this is expected!
- `$ pipenv shell`
- `$ pre-commit install`
- `$ ./manage.py migrate` OR `$ python3 manage.py migrate`
- `$ ./manage.py populate` OR `$ python3 manage.py populate` (in development,
  to populate the database with dummy data)
- `$ ./manage.py runserver` OR `$ python3 manage.py runserver`

Use `$ ./manage.py test` to run unit tests.

### Frontend

Running the frontend requires [Node.js](https://nodejs.org/en/) and
[Yarn](https://yarnpkg.com/getting-started/install).

**Please ensure you are using Node 20**. Our codebase does not support other
versions of Node (v20.11.1 is stable).

You will need to set the following environment variables on the frontend:

- `NEXT_PUBLIC_GOOGLE_API_KEY`
- `NEXT_PUBLIC_SITE_NAME` (Optional)
  - Specify `clubs` to show Penn Clubs and `fyh` to show Hub@Penn.

1. Enter the `frontend` directory with a **new terminal window**. Don't kill your backend server!
2. Install dependencies using `yarn install` in the project directory.
3. Run application using `yarn dev`.
4. Access application at [http://localhost:3000](http://localhost:3000).

Use `$ yarn test` to run Cypress tests.

### Development

Click `Login` to log in as a test user. The `./manage.py populate` command creates a test user for you with username `bfranklin` and password `test`. Go to `/api/admin` to login to this account.

#### Ticketing

To test ticketing locally, you will need to [install](https://github.com/FiloSottile/mkcert?tab=readme-ov-file#installation) `mkcert`, enter the `frontend` directory, and run the following commands:

- `$ mkcert -install`
- `$ mkcert localhost 127.0.0.1 ::1`
- `$ export DOMAIN=https://localhost:3001 NODE_TLS_REJECT_UNAUTHORIZED=0`

Then, after the frontend is running, run `yarn ssl-proxy` **in a new terminal window** and access the application at [https://localhost:3001](https://localhost:3001).