![Natlas](https://0xda.de/img/natlas-1000px.png)

![Last Commit](https://img.shields.io/github/last-commit/natlas/natlas.svg)
[![Release](https://img.shields.io/github/release/natlas/natlas.svg)](https://github.com/natlas/natlas/releases/latest)
[![Contributors](https://img.shields.io/github/contributors/natlas/natlas.svg)](https://github.com/natlas/natlas/graphs/contributors)
[![License](https://img.shields.io/badge/license-Apache%202-blue.svg?style=flat)](LICENSE)

[![Maintainability](https://api.codeclimate.com/v1/badges/321141e5cf7426874cd7/maintainability)](https://codeclimate.com/github/natlas/natlas/maintainability)
[![Total alerts](https://img.shields.io/lgtm/alerts/g/natlas/natlas.svg?logo=lgtm&logoWidth=18)](https://lgtm.com/projects/g/natlas/natlas/alerts/)
[![pre-commit](https://img.shields.io/badge/pre--commit-enabled-brightgreen?logo=pre-commit&logoColor=white)](https://github.com/pre-commit/pre-commit)
[![Code style: black](https://img.shields.io/badge/code%20style-black-000000.svg)](https://github.com/psf/black)

[![Server downloads](https://img.shields.io/docker/pulls/natlas/server?label=server%20downloads&logo=docker)](https://hub.docker.com/repository/docker/natlas/server)
[![Agent downloads](https://img.shields.io/docker/pulls/natlas/agent?label=agent%20downloads&logo=docker)](https://hub.docker.com/repository/docker/natlas/agent)

# Summary

You've got a lot of maps and they are getting pretty unruly. What do you do? You put them in a book and call it an atlas. This is like that, except it's a website and it's a collection of nmaps. Natlas' objective is to make it easy to perform continuous scanning and review collected data.

The goal of Natlas is not to identify a bunch of vulnerabilities, necessarily, but rather to identify exposure. Perhaps you want to make sure that no one is running ssh with password auth enabled. Or perhaps you want to look for any exposed nfs, smb, or rsync shares. Maybe you want to look for expiring or expired ssl certificates, or weak ssl ciphers being used. Since Natlas uses the popular [nmap](https://nmap.org) port scanner, you can easily use any default nmap scripts in your scans.

# Getting Started

To get started with your own deployment of Natlas, you're going to need a minimum of one elasticsearch node, one Natlas server and one Natlas agent. The quickest way to accomplish this is to run all three of these containers on the same server. Installation instructions for the server and the agent are linked below in their associated readmes.

The required deployment order is as follows:

1) [Elasticsearch](#Elasticsearch)
2) [Natlas Server](#Natlas-server)
3) [Natlas Agent(s)](#Natlas-agent)

**Note:** As of June 15, 2020, Natlas has moved to a docker-only deployment model.

## Elasticsearch

Because the Natlas server requires a connection to Elasticsearch in order to launch correctly, you should make sure you have an Elasticsearch node available before continuing with the next step. If you've never used Elasticsearch before, follow [Elastic's instructions](https://www.elastic.co/guide/en/elasticsearch/reference/current/docker.html) for setting up a single node cluster with docker. Make sure to pay attention to the section on persisting the elastic data.

## Natlas Server

The Natlas server controls agent orchestration, scan configuration, and provides search and browse for the data stored in Elastic.

See the [Natlas Server README](natlas-server#installation-production) for instructions on installing the Natlas server.

## Natlas Agent

The Natlas agent is what fetches work from the server and actually performs the scans.

See the [Natlas Agent README](natlas-agent#installation-production) for instructions on installing the Natlas agent.

# Contributing

Please review our [contribution guidelines](CONTRIBUTING.md) for information on how to contribute to Natlas. To get started with development, please see [Project Setup](CONTRIBUTING.md#project-setup).

# Code Of Conduct

This project strives to adhere to the code of conduct outlined in our [code of conduct](CODE_OF_CONDUCT.md). Please review the code of conduct before contributing.

# Security

Information about this project's security reporting guidelines as well as security related functionality are outlined in our [Security guidelines](SECURITY.md).

# Acknowledgements

* [Pinguino](http://www.pinguinokolb.com/) - Created Natlas logo/branding.
* [Dean Pierce](https://github.com/pierce403) - Created the initial project, nweb, that Natlas was built out of.
* [Topher Timzen](https://github.com/tophertimzen) - Testing, feedback, automation support.
* [Adam Jacques](https://github.com/ajacques) - Helping with Elasticsearch theory and just generally helping improve code quality
* [Ross Snider](https://github.com/rosswsnider) - Writing Cyclic PRNG for target selection
* [Everyone who has contributed](https://github.com/natlas/natlas/graphs/contributors)

# Disclaimer

Natlas is a platform which makes use of many other open source projects, many of which have their own licenses. Natlas does not claim ownership of any projects that it uses, and does not represent any of said projects. To the best of the Natlas Author's knowledge, the use of these tools in the Natlas platform is not violating any licenses. Natlas is a free and open source project that does not make, nor seeks to make, any revenue from the use of the other open source tools in use.

For further inquiry about licensing, please see the respective projects' licenses.

# License

```text
Copyright 2020 0xdade
Copyright 2016-2018 Intel Corporation

Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at

   http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
```


=============
flask-smorest
=============

.. image:: https://img.shields.io/pypi/v/flask-smorest.svg
    :target: https://pypi.org/project/flask-smorest/
    :alt: Latest version

.. image:: https://img.shields.io/pypi/pyversions/flask-smorest.svg
    :target: https://pypi.org/project/flask-smorest/
    :alt: Python versions

.. image:: https://img.shields.io/badge/marshmallow-3-blue.svg
    :target: https://marshmallow.readthedocs.io/en/latest/upgrading.html
    :alt: marshmallow 3 only

.. image:: https://img.shields.io/badge/OAS-2%20|%203-green.svg
    :target: https://github.com/OAI/OpenAPI-Specification
    :alt: OpenAPI Specification 2/3 compatible

.. image:: https://img.shields.io/pypi/l/flask-smorest.svg
    :target: https://flask-smorest.readthedocs.io/en/latest/license.html
    :alt: License

.. image:: https://github.com/marshmallow-code/flask-smorest/actions/workflows/build-release.yml/badge.svg
    :target: https://github.com/marshmallow-code/flask-smorest/actions?query=workflow%3Abuild
    :alt: Build status

.. image:: https://codecov.io/gh/marshmallow-code/flask-smorest/branch/master/graph/badge.svg?token=F676tOSaLF
    :target: https://codecov.io/gh/marshmallow-code/flask-smorest
    :alt: Code coverage

.. image:: https://results.pre-commit.ci/badge/github/marshmallow-code/flask-smorest/main.svg
   :target: https://results.pre-commit.ci/latest/github/marshmallow-code/flask-smorest/main
   :alt: pre-commit.ci status

.. image:: https://readthedocs.org/projects/flask-smorest/badge/
    :target: http://flask-smorest.readthedocs.io/
    :alt: Documentation

'cause everybody wants s'more
=============================

**flask-smorest** (formerly known as flask-rest-api) is a REST API framework
built upon `Flask <https://palletsprojects.com/p/flask/>`_ and
`marshmallow <https://github.com/marshmallow-code/marshmallow>`_.

- Serialization, deserialization and validation using marshmallow ``Schema``
- Explicit validation error messages returned in response
- Database-agnostic
- OpenAPI (Swagger) specification automatically generated and exposed with
  `ReDoc <https://github.com/Rebilly/ReDoc>`_,
  `Swagger UI <https://swagger.io/tools/swagger-ui/>`_ or
  `RapiDoc <https://mrin9.github.io/RapiDoc/>`_
- Pagination
- ETag

Install
=======

::

    pip install flask-smorest

Documentation
=============

Full documentation is available at http://flask-smorest.readthedocs.io/.

Support flask-smorest
======================

If you'd like to support the future of the project, please consider
contributing to marshmallow's Open Collective:

.. image:: https://opencollective.com/marshmallow/donate/button.png
    :target: https://opencollective.com/marshmallow
    :width: 200
    :alt: Donate to our collective

License
=======

MIT licensed. See the `LICENSE <https://github.com/marshmallow-code/flask-smorest/blob/master/LICENSE>`_ file for more details.


# Robotics Toolbox for Python

[![A Python Robotics Package](https://raw.githubusercontent.com/petercorke/robotics-toolbox-python/master/.github/svg/py_collection.min.svg)](https://github.com/petercorke/robotics-toolbox-python)
[![Powered by Spatial Maths](https://raw.githubusercontent.com/petercorke/spatialmath-python/master/.github/svg/sm_powered.min.svg)](https://github.com/petercorke/spatialmath-python)
[![QUT Centre for Robotics Open Source](https://github.com/qcr/qcr.github.io/raw/master/misc/badge.svg)](https://qcr.github.io)

[![PyPI version](https://badge.fury.io/py/roboticstoolbox-python.svg)](https://badge.fury.io/py/roboticstoolbox-python)
[![Anaconda version](https://anaconda.org/conda-forge/roboticstoolbox-python/badges/version.svg)](https://anaconda.org/conda-forge/roboticstoolbox-python)
![PyPI - Python Version](https://img.shields.io/pypi/pyversions/roboticstoolbox-python.svg)

[![Build Status](https://github.com/petercorke/robotics-toolbox-python/workflows/Test/badge.svg?branch=master)](https://github.com/petercorke/robotics-toolbox-python/actions?query=workflow%3ATest)
[![Coverage](https://codecov.io/gh/petercorke/robotics-toolbox-python/branch/master/graph/badge.svg)](https://codecov.io/gh/petercorke/robotics-toolbox-python)
[![PyPI - Downloads](https://img.shields.io/pypi/dw/roboticstoolbox-python)](https://pypistats.org/packages/roboticstoolbox-python)
[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)

A Python implementation of the Robotics Toolbox for MATLAB®

GitHub repository

Documentation

ICRA Paper

Wiki (examples and details)

## Contents

- [Synopsis](#1)
- [Getting going](#2)
- [Tutorials](#3)
- [Code Examples](#4)
- [Toolbox Research Applications](#5)
- [Toolbox ICRA Paper and Citation Info](#6)
- [Using the Toolbox in your Open Source Code?](#7)
- [Common Issues and Solutions](#8)

## Synopsis

This toolbox brings robotics-specific functionality to Python, and leverages
Python's advantages of portability, ubiquity and support, and the capability of
the open-source ecosystem for linear algebra (numpy, scipy), graphics
(matplotlib, three.js, WebGL), interactive development (jupyter, jupyterlab,
mybinder.org), and documentation (sphinx).

The Toolbox provides tools for representing the kinematics and dynamics of
serial-link manipulators - you can easily create your own in Denavit-Hartenberg
form, import a URDF file, or use over 30 supplied models for well-known
contemporary robots from Franka-Emika, Kinova, Universal Robotics, Rethink as
well as classical robots such as the Puma 560 and the Stanford arm.

The Toolbox contains fast implementations of kinematic operations. The forward
kinematics and the manipulator Jacobian can be computed in less than 1 microsecond
while numerical inverse kinematics can be solved in as little as 4 microseconds.

The toolbox also supports mobile robots with functions for robot motion models
(unicycle, bicycle), path planning algorithms (bug, distance transform, D\*,
PRM), kinodynamic planning (lattice, RRT), localization (EKF, particle filter),
map building (EKF) and simultaneous localization and mapping (EKF).

The Toolbox provides:

- code that is mature and provides a point of comparison for other
  implementations of the same algorithms;
- routines which are generally written in a straightforward manner which
  allows for easy understanding, perhaps at the expense of computational
  efficiency;
- source code which can be read for learning and teaching;
- backward compatability with the Robotics Toolbox for MATLAB

The Toolbox leverages the [Spatial Maths Toolbox for Python](https://github.com/petercorke/spatialmath-python) to
provide support for data types such as SO(n) and SE(n) matrices, quaternions, twists and spatial vectors.

## Getting going

You will need Python >= 3.6

### Using pip

Install a snapshot from PyPI

```shell script
pip3 install roboticstoolbox-python
```

Available options are:

- `collision` install collision checking with [pybullet](https://pybullet.org)

Put the options in a comma separated list like

```shell script
pip3 install roboticstoolbox-python[optionlist]
```

[Swift](https://github.com/jhavl/swift), a web-based visualizer, is
installed as part of Robotics Toolbox.

### From GitHub

To install the bleeding-edge version from GitHub

```shell script
git clone https://github.com/petercorke/robotics-toolbox-python.git
cd robotics-toolbox-python
pip3 install -e .
```

## Tutorials

Do you want to learn about manipulator kinematics, differential kinematics, inverse-kinematics and motion control? Have a look at our
tutorial.
This tutorial comes with two articles to cover the theory and 12 Jupyter Notebooks providing full code implementations and examples. Most of the Notebooks are also Google Colab compatible allowing them to run online.

## Code Examples

We will load a model of the Franka-Emika Panda robot defined by a URDF file

```python
import roboticstoolbox as rtb
robot = rtb.models.Panda()
print(robot)

	ERobot: panda (by Franka Emika), 7 joints (RRRRRRR), 1 gripper, geometry, collision
	┌─────┬──────────────┬───────┬─────────────┬────────────────────────────────────────────────┐
	│link │     link     │ joint │   parent    │              ETS: parent to link               │
	├─────┼──────────────┼───────┼─────────────┼────────────────────────────────────────────────┤
	│   0 │ panda_link0  │       │ BASE        │                                                │
	│   1 │ panda_link1  │     0 │ panda_link0 │ SE3(0, 0, 0.333) ⊕ Rz(q0)                      │
	│   2 │ panda_link2  │     1 │ panda_link1 │ SE3(-90°, -0°, 0°) ⊕ Rz(q1)                    │
	│   3 │ panda_link3  │     2 │ panda_link2 │ SE3(0, -0.316, 0; 90°, -0°, 0°) ⊕ Rz(q2)       │
	│   4 │ panda_link4  │     3 │ panda_link3 │ SE3(0.0825, 0, 0; 90°, -0°, 0°) ⊕ Rz(q3)       │
	│   5 │ panda_link5  │     4 │ panda_link4 │ SE3(-0.0825, 0.384, 0; -90°, -0°, 0°) ⊕ Rz(q4) │
	│   6 │ panda_link6  │     5 │ panda_link5 │ SE3(90°, -0°, 0°) ⊕ Rz(q5)                     │
	│   7 │ panda_link7  │     6 │ panda_link6 │ SE3(0.088, 0, 0; 90°, -0°, 0°) ⊕ Rz(q6)        │
	│   8 │ @panda_link8 │       │ panda_link7 │ SE3(0, 0, 0.107)                               │
	└─────┴──────────────┴───────┴─────────────┴────────────────────────────────────────────────┘

	┌─────┬─────┬────────┬─────┬───────┬─────┬───────┬──────┐
	│name │ q0  │ q1     │ q2  │ q3    │ q4  │ q5    │ q6   │
	├─────┼─────┼────────┼─────┼───────┼─────┼───────┼──────┤
	│  qr │  0° │ -17.2° │  0° │ -126° │  0° │  115° │  45° │
	│  qz │  0° │  0°    │  0° │  0°   │  0° │  0°   │  0°  │
	└─────┴─────┴────────┴─────┴───────┴─────┴───────┴──────┘
```

The symbol `@` indicates the link as an end-effector, a leaf node in the rigid-body
tree (Python prompts are not shown to make it easy to copy+paste the code, console output is indented).
We will compute the forward kinematics next

```
Te = robot.fkine(robot.qr)  # forward kinematics
print(Te)

	0.995     0         0.09983   0.484
	0        -1         0         0
	0.09983   0        -0.995     0.4126
	0         0         0         1
```

We can solve inverse kinematics very easily. We first choose an SE(3) pose
defined in terms of position and orientation (end-effector z-axis down (A=-Z) and finger
orientation parallel to y-axis (O=+Y)).

```python
from spatialmath import SE3

Tep = SE3.Trans(0.6, -0.3, 0.1) * SE3.OA([0, 1, 0], [0, 0, -1])
sol = robot.ik_LM(Tep)         # solve IK
print(sol)

	(array([ 0.20592815,  0.86609481, -0.79473206, -1.68254794,  0.74872915,
			2.21764746, -0.10255606]), 1, 114, 7, 2.890164057230228e-07)

q_pickup = sol[0]
print(robot.fkine(q_pickup))    # FK shows that desired end-effector pose was achieved

	 1         -8.913e-05  -0.0003334  0.5996
	-8.929e-05 -1          -0.0004912 -0.2998
	-0.0003334  0.0004912  -1          0.1001
	 0          0           0          1
```

We can animate a path from the ready pose `qr` configuration to this pickup configuration

```python
qt = rtb.jtraj(robot.qr, q_pickup, 50)
robot.plot(qt.q, backend='pyplot', movie='panda1.gif')
```

where we have specified the matplotlib `pyplot` backend. Blue arrows show the joint axes and the coloured frame shows the end-effector pose.

We can also plot the trajectory in the Swift simulator (a browser-based 3d-simulation environment built to work with the Toolbox)

```python
robot.plot(qt.q)
```

We can also experiment with velocity controllers in Swift. Here is a resolved-rate motion control example

```python
import swift
import roboticstoolbox as rtb
import spatialmath as sm
import numpy as np

env = swift.Swift()
env.launch(realtime=True)

panda = rtb.models.Panda()
panda.q = panda.qr

Tep = panda.fkine(panda.q) * sm.SE3.Trans(0.2, 0.2, 0.45)

arrived = False
env.add(panda)

dt = 0.05

while not arrived:

    v, arrived = rtb.p_servo(panda.fkine(panda.q), Tep, 1)
    panda.qd = np.linalg.pinv(panda.jacobe(panda.q)) @ v
    env.step(dt)

# Uncomment to stop the browser tab from closing
# env.hold()
```

### Run some examples

The [`notebooks`](https://github.com/petercorke/robotics-toolbox-python/tree/master/notebooks) folder contains some tutorial Jupyter notebooks which you can browse on GitHub. Additionally, have a look in the [`examples`](https://github.com/petercorke/robotics-toolbox-python/tree/master/roboticstoolbox/examples) folder for many ready to run examples.

## Toolbox Research Applications

The toolbox is incredibly useful for developing and prototyping algorithms for research, thanks to the exhaustive set of well documented and mature robotic functions exposed through clean and painless APIs. Additionally, the ease at which a user can visualize their algorithm supports a rapid prototyping paradigm.

### Publication List

J. Haviland, N. Sünderhauf and P. Corke, "**A Holistic Approach to Reactive Mobile Manipulation**," in _IEEE Robotics and Automation Letters_, doi: 10.1109/LRA.2022.3146554. In the video, the robot is controlled using the Robotics toolbox for Python and features a recording from the [Swift](https://github.com/jhavl/swift) Simulator.

[[Arxiv Paper](https://arxiv.org/abs/2109.04749)] [[IEEE Xplore](https://ieeexplore.ieee.org/abstract/document/9695298)] [[Project Website](https://jhavl.github.io/holistic/)] [[Video](https://youtu.be/-DXBQPeLIV4)] [[Code Example](https://github.com/petercorke/robotics-toolbox-python/blob/master/roboticstoolbox/examples/holistic_mm_non_holonomic.py)]

J. Haviland and P. Corke, "**NEO: A Novel Expeditious Optimisation Algorithm for Reactive Motion Control of Manipulators**," in _IEEE Robotics and Automation Letters_, doi: 10.1109/LRA.2021.3056060. In the video, the robot is controlled using the Robotics toolbox for Python and features a recording from the [Swift](https://github.com/jhavl/swift) Simulator.

[[Arxiv Paper](https://arxiv.org/abs/2010.08686)] [[IEEE Xplore](https://ieeexplore.ieee.org/document/9343718)] [[Project Website](https://jhavl.github.io/neo/)] [[Video](https://youtu.be/jSLPJBr8QTY)] [[Code Example](https://github.com/petercorke/robotics-toolbox-python/blob/master/roboticstoolbox/examples/neo.py)]

**A Purely-Reactive Manipulability-Maximising Motion Controller**, J. Haviland and P. Corke. In the video, the robot is controlled using the Robotics toolbox for Python.

[[Paper](https://arxiv.org/abs/2002.11901)] [[Project Website](https://jhavl.github.io/mmc/)] [[Video](https://youtu.be/Vu_rcPlaADI)] [[Code Example](https://github.com/petercorke/robotics-toolbox-python/blob/master/roboticstoolbox/examples/mmc.py)]

## Toolbox ICRA Paper and Citation Info

Check out our ICRA 2021 paper on [IEEE Xplore](https://ieeexplore.ieee.org/document/9561366) or get the PDF from [Peter's website](https://bit.ly/3ChcyNp).

If the toolbox helped you in your research, please cite

```
@inproceedings{rtb,
  title={Not your grandmother’s toolbox--the Robotics Toolbox reinvented for Python},
  author={Corke, Peter and Haviland, Jesse},
  booktitle={2021 IEEE International Conference on Robotics and Automation (ICRA)},
  pages={11357--11363},
  year={2021},
  organization={IEEE}
}
```

## Using the Toolbox in your Open Source Code?

If you are using the Toolbox in your open source code, feel free to add our badge to your readme!

For the powered by robotics toolbox badge

[![Powered by the Robotics Toolbox](https://raw.githubusercontent.com/petercorke/robotics-toolbox-python/master/.github/svg/rtb_powered.min.svg)](https://github.com/petercorke/robotics-toolbox-python)

copy the following

```
[![Powered by the Robotics Toolbox](https://raw.githubusercontent.com/petercorke/robotics-toolbox-python/master/.github/svg/rtb_powered.min.svg)](https://github.com/petercorke/robotics-toolbox-python)
```

For the powered by python robotics badge

[![Powered by Python Robotics](https://raw.githubusercontent.com/petercorke/robotics-toolbox-python/master/.github/svg/pr_powered.min.svg)](https://github.com/petercorke/robotics-toolbox-python)

copy the following

```
[![Powered by Python Robotics](https://raw.githubusercontent.com/petercorke/robotics-toolbox-python/master/.github/svg/pr_powered.min.svg)](https://github.com/petercorke/robotics-toolbox-python)
```

## Common Issues and Solutions

See the common issues with fixes [here](https://github.com/petercorke/robotics-toolbox-python/wiki/Common-Issues).


[![Build Status](https://github.com/VIDA-NYU/reprozip/workflows/Test/badge.svg)](https://github.com/VIDA-NYU/reprozip/actions)
[![Coverage Status](https://codecov.io/github/VIDA-NYU/reprozip/coverage.svg?branch=1.x)](https://codecov.io/github/VIDA-NYU/reprozip?branch=1.x)
[![Documentation Status](https://readthedocs.org/projects/reprozip/badge/?version=1.x)](https://docs.reprozip.org/en/1.x/)
[![Matrix](https://img.shields.io/badge/chat-matrix.org-blue.svg)](https://riot.im/app/#/room/#reprozip:matrix.org)
[![paper](https://img.shields.io/badge/JOSS-10.21105%2Fjoss.00107-green.svg)](http://joss.theoj.org/papers/b578b171263c73f64dfb9d040ca80fe0)
[![DOI](https://img.shields.io/badge/DOI-10.5281%2Fzenodo.10291844-green.svg)](https://doi.org/10.5281/zenodo.10291844)

ReproZip
========

[ReproZip][web] is a tool aimed at simplifying the process of creating reproducible experiments from command-line executions, a frequently-used common denominator in computational science.

It tracks operating system calls and creates a package that contains all the binaries, files and dependencies required to run a given command on the author's computational environment (packing step). A reviewer can then extract the experiment in his environment to reproduce the results (unpacking step).

Quickstart
----------

We have an [example repository](https://github.com/VIDA-NYU/reprozip-examples) with a variety of different software. Don't hesitate to check it out, and contribute your own example if use ReproZip for something new!

### Packing

Packing experiments is only available for Linux distributions. In the environment where the experiment is originally executed, first install reprozip:

    $ pip install reprozip

Then, run your experiment with reprozip. Suppose you execute your experiment by originally running the following command:

    $ ./myexperiment -my --options inputs/somefile.csv other_file_here.bin

To run it with reprozip, you just need to use the prefix *reprozip trace*:

    $ reprozip trace ./myexperiment -my --options inputs/somefile.csv other_file_here.bin

This command creates a *.reprozip-trace* directory, in which you'll find the configuration file, named *config.yml*. You can edit the command line and environment variables, and choose which files to pack.

If you are using Debian or Ubuntu, most of these files (library dependencies) are organized by package. You can add or remove files, or choose not to include a package by changing option *packfiles* from true to false. In this way, smaller packs can be created with reprozip (if space is an issue), and reprounzip can download these files from the package manager; however, note this is only available for Debian and Ubuntu for now, and also be aware that package versions might differ. Choosing which files to pack is also important to remove sensitive information and third-party software that is not open source and should not be distributed.

Once done editing the configuration file (or even if you did not change anything), run the following command to create a ReproZip package named *my_experiment*:

    $ reprozip pack my_experiment.rpz

Voilà! Now your experiment has been packed, and you can send it to your collaborators, reviewers, and researchers around the world!

Note that you can open the help message for any reprozip command by using the flag *-h*.

### Unpacking

Do you need to unpack an experiment in a Linux machine? Easy! First, install reprounzip:

    $ pip install reprounzip

Then, if you want to unpack everything in a single directory named *mydirectory* and execute the experiment from there, use the prefix *reprounzip directory*:

    $ reprounzip directory setup my_experiment.rpz mydirectory
    $ reprounzip directory run mydirectory

In case you prefer to build a chroot environment under *mychroot*, use the prefix *reprounzip chroot*:

    $ reprounzip chroot setup my_experiment.rpz mychroot
    $ reprounzip chroot run mychroot

Note that the previous options do not interfere with the original configuration of the environment, so don't worry! If you are using Debian or Ubuntu, reprounzip also has an option to install all the library dependencies directly on the machine using package managers (rather than just copying the files from the .rpz package). Be aware that this will interfere in your environment and it may update your library packages, so use it at your own risk! For this option, just use the prefix *reprounzip installpkgs*:

    $ reprounzip installpkgs my_experiment.rpz

What if you want to reproduce the experiment in Windows or Mac OS X? You can build a virtual machine with the experiment! Easy as well! First, install the plugin reprounzip-vagrant:

    $ pip install reprounzip-vagrant

Note that (i) you must install reprounzip first, and (ii) the plugin requires having [Vagrant][vagrant] installed. Then, use the prefix *reprounzip vagrant* to create and start a virtual machine under directory *mytemplate*:

    $ reprounzip vagrant setup my_experiment.rpz mytemplate

To execute the experiment, simply run:

    $ reprounzip vagrant run mytemplate

Alternatively, you may use [Docker][docker] containers to reproduce the experiment, which also works under Linux, Mac OS X, and Windows! First, install the plugin reprounzip-docker:

    $ pip install reprounzip-docker

Then, assuming that you want to create the container under directory *mytemplate*, simply use the prefix *reprounzip docker*:

    $ reprounzip docker setup my_experiment.rpz mytemplate
    $ reprounzip docker run mytemplate

Remember that you can open the help message and learn more about other available flags and options by using the flag *-h* for any reprounzip command.

Citing ReproZip
---------------

Please use the following when citing ReproZip ([BibTeX](CITATION.txt)):

    ReproZip: Computational Reproducibility With Ease
    F. Chirigati, R. Rampin, D. Shasha, and J. Freire.
    In Proceedings of the 2016 ACM SIGMOD International Conference on Management of Data (SIGMOD), pp. 2085-2088, 2016

Contribute
----------

Please subscribe to and contact the [reprozip@nyu.edu](https://groups.google.com/a/nyu.edu/g/reprozip) mailing list for questions, suggestions and discussions about using reprozip.

Bugs and feature plannings are tracked in the [GitHub issues](https://github.com/VIDA-NYU/reprozip/issues). Feel free to add an issue!

To suggest changes to this source code, feel free to raise a [GitHub pull request](https://github.com/VIDA-NYU/reprozip/pulls).  Any contributions received are assumed to be covered by the [BSD 3-Clause license](LICENSE.txt). We might ask you to sign a _Contributor License Agreement_ before accepting a larger contribution.

License
-------

* Copyright (C) 2014, New York University

Licensed under a **BSD 3-Clause license**. See the file [LICENSE.txt](LICENSE.txt) for details.

Links and References
--------------------

For more detailed information, please refer to our [website][web], as well as to our [documentation][docs].

ReproZip is currently being developed at [NYU][nyu]. The team includes:

* [Fernando Chirigati][fc]
* [Juliana Freire][jf]
* [Rémi Rampin][rr]
* [Dennis Shasha][ds]
* [Vicky Rampin][vr]

[vagrant]: https://www.vagrantup.com/
[docker]: https://www.docker.com/
[docs]: https://docs.reprozip.org/
[web]: https://www.reprozip.org/
[pz]: https://pypi.python.org/pypi/reprozip
[puz]: https://pypi.python.org/pypi/reprounzip
[puzd]: https://pypi.python.org/pypi/reprounzip-docker
[puzv]: https://pypi.python.org/pypi/reprounzip-vagrant
[fc]: http://fchirigati.com/
[jf]: https://vgc.poly.edu/~juliana/
[rr]: https://remi.rampin.org/
[ds]: http://cs.nyu.edu/shasha/
[vr]: https://vicky.rampin.org/
[nyu]: http://engineering.nyu.edu/


# datadog-serverless-functions

This repository contains our serverless functions that process streams and send data to datadog

- For AWS, [go here](./aws/README.md)
- For Azure, [go here](./azure/README.md)
- For OCI, [go here](./oci/README.md)


# ![Copier](https://github.com/copier-org/copier/raw/master/img/copier-logotype.png)

[![Copier](https://img.shields.io/endpoint?url=https://raw.githubusercontent.com/copier-org/copier/master/img/badge/badge-grayscale-inverted-border-purple.json)](https://github.com/copier-org/copier)
[![Gitpod ready-to-code](https://img.shields.io/badge/Gitpod-ready--to--code-blue?logo=gitpod)](https://gitpod.io/#https://github.com/copier-org/copier)
[![codecov](https://codecov.io/gh/copier-org/copier/branch/master/graph/badge.svg)](https://codecov.io/gh/copier-org/copier)
[![CI](https://github.com/copier-org/copier/workflows/CI/badge.svg)](https://github.com/copier-org/copier/actions?query=branch%3Amaster)
[![Checked with mypy](http://www.mypy-lang.org/static/mypy_badge.svg)](http://mypy-lang.org/)
![Python](https://img.shields.io/pypi/pyversions/copier?logo=python&logoColor=%23959DA5)
[![PyPI](https://img.shields.io/pypi/v/copier?logo=pypi&logoColor=%23959DA5)](https://pypi.org/project/copier/)
[![Code style: black](https://img.shields.io/badge/code%20style-black-000000.svg)](https://github.com/psf/black)
[![Documentation Status](https://img.shields.io/readthedocs/copier/latest?logo=readthedocs)](https://copier.readthedocs.io/en/latest)

A library and CLI app for rendering project templates.

-   Works with **local** paths and **Git URLs**.
-   Your project can include any file and Copier can dynamically replace values in any
    kind of text file.
-   It generates a beautiful output and takes care of not overwriting existing files
    unless instructed to do so.

![Sample output](https://github.com/copier-org/copier/raw/master/img/copier-output.png)

## Installation

1. Install Python 3.8 or newer.
1. Install Git 2.27 or newer.
1. To use as a CLI app: `pipx install copier`
1. To use as a library: `pip install copier` or `conda install -c conda-forge copier`

### Nix flake

To install latest Copier release with 100% reproducibility:

```shell
nix profile install 'https://flakehub.com/f/copier-org/copier/*.tar.gz'
```

## Quick start

To create a template:

```shell
📁 my_copier_template                        # your template project
├── 📄 copier.yml                            # your template configuration
├── 📁 .git/                                 # your template is a Git repository
├── 📁 {{project_name}}                      # a folder with a templated name
│   └── 📄 {{module_name}}.py.jinja          # a file with a templated name
└── 📄 {{_copier_conf.answers_file}}.jinja   # answers are recorded here
```

```yaml title="copier.yml"
# questions
project_name:
    type: str
    help: What is your project name?

module_name:
    type: str
    help: What is your Python module name?
```

```python+jinja title="{{project_name}}/{{module_name}}.py.jinja"
print("Hello from {{module_name}}!")
```

```yaml+jinja title="{{_copier_conf.answers_file}}.jinja"
# Changes here will be overwritten by Copier
{{ _copier_answers|to_nice_yaml -}}
```

To generate a project from the template:

-   On the command-line:

    ```shell
    copier copy path/to/project/template path/to/destination
    ```

-   Or in Python code, programmatically:

    ```python
    from copier import run_copy

    # Create a project from a local path
    run_copy("path/to/project/template", "path/to/destination")

    # Or from a Git URL.
    run_copy("https://github.com/copier-org/copier.git", "path/to/destination")

    # You can also use "gh:" as a shortcut of "https://github.com/"
    run_copy("gh:copier-org/copier.git", "path/to/destination")

    # Or "gl:" as a shortcut of "https://gitlab.com/"
    run_copy("gl:copier-org/copier.git", "path/to/destination")
    ```

## Basic concepts

Copier is composed of these main concepts:

1. **Templates**. They lay out how to generate the subproject.
1. **Questionnaires**. They are configured in the template. Answers are used to generate
   projects.
1. **Projects**. This is where your real program lives. But it is usually generated
   and/or updated from a template.

Copier targets these main human audiences:

1.  **Template creators**. Programmers that repeat code too much and prefer a tool to do
    it for them.

    !!! tip

         Copier doesn't replace the DRY principle... but sometimes you simply can't be
         DRY and you need a DRYing machine...

1.  **Template consumers**. Programmers that want to start a new project quickly, or
    that want to evolve it comfortably.

Non-humans should be happy also by using Copier's CLI or API, as long as their
expectations are the same as for those humans... and as long as they have feelings.

Templates have these goals:

1. **[Code scaffolding](<https://en.wikipedia.org/wiki/Scaffold_(programming)>)**. Help
   consumers have a working source code tree as quickly as possible. All templates allow
   scaffolding.
1. **Code lifecycle management**. When the template evolves, let consumers update their
   projects. Not all templates allow updating.

Copier tries to have a smooth learning curve that lets you create simple templates that
can evolve into complex ones as needed.

## Browse or tag public templates

You can browse public Copier templates on GitHub using
[the `copier-template` topic](https://github.com/topics/copier-template). Use them as
inspiration!

If you want your template to appear in that list, just add the topic to it! 🏷

## Show your support

If you're using Copier, consider adding the Copier badge to your project's `README.md`:

```md
[![Copier](https://img.shields.io/endpoint?url=https://raw.githubusercontent.com/copier-org/copier/master/img/badge/badge-grayscale-inverted-border-orange.json)](https://github.com/copier-org/copier)
```

...or `README.rst`:

```rst
.. image:: https://img.shields.io/endpoint?url=https://raw.githubusercontent.com/copier-org/copier/master/img/badge/badge-grayscale-inverted-border-orange.json
    :target: https://github.com/copier-org/copier
    :alt: Copier
```

...or, as HTML:

```html
<a href="https://github.com/copier-org/copier"><img src="https://img.shields.io/endpoint?url=https://raw.githubusercontent.com/copier-org/copier/master/img/badge/badge-grayscale-inverted-border-orange.json" alt="Copier" style="max-width:100%;"/></a>
```

### Copier badge variations

1. Badge Grayscale Border
   [![Copier](https://img.shields.io/endpoint?url=https://raw.githubusercontent.com/copier-org/copier/master/img/badge/badge-grayscale-border.json)](https://github.com/copier-org/copier)

1. Badge Grayscale Inverted Border
   [![Copier](https://img.shields.io/endpoint?url=https://raw.githubusercontent.com/copier-org/copier/master/img/badge/badge-grayscale-inverted-border.json)](https://github.com/copier-org/copier)

1. Badge Grayscale Inverted Border Orange
   [![Copier](https://img.shields.io/endpoint?url=https://raw.githubusercontent.com/copier-org/copier/master/img/badge/badge-grayscale-inverted-border-orange.json)](https://github.com/copier-org/copier)

1. Badge Grayscale Inverted Border Red
   [![Copier](https://img.shields.io/endpoint?url=https://raw.githubusercontent.com/copier-org/copier/master/img/badge/badge-grayscale-inverted-border-red.json)](https://github.com/copier-org/copier)

1. Badge Grayscale Inverted Border Teal
   [![Copier](https://img.shields.io/endpoint?url=https://raw.githubusercontent.com/copier-org/copier/master/img/badge/badge-grayscale-inverted-border-teal.json)](https://github.com/copier-org/copier)

1. Badge Grayscale Inverted Border Purple
   [![Copier](https://img.shields.io/endpoint?url=https://raw.githubusercontent.com/copier-org/copier/master/img/badge/badge-grayscale-inverted-border-purple.json)](https://github.com/copier-org/copier)

1. Badge Black
   [![Copier](https://img.shields.io/endpoint?url=https://raw.githubusercontent.com/copier-org/copier/master/img/badge/badge-black.json)](https://github.com/copier-org/copier)

## Credits

Special thanks go to [jpsca](https://github.com/jpsca) for originally creating `Copier`.
This project would not be a thing without him.

Many thanks to [pykong](https://github.com/pykong) who took over maintainership on the
project, promoted it, and laid out the bases of what the project is today.

Big thanks also go to [yajo](https://github.com/yajo) for his relentless zest for
improving `Copier` even further.

Thanks a lot, [pawamoy](https://github.com/pawamoy) for polishing very important rough
edges and improving the documentation and UX a lot.

Also special thanks to [sisp](https://github.com/sisp) for being very helpful in
polishing documentation, fixing bugs, helping the community and cleaning up the
codebase.

And thanks to all financial supporters and folks that give us a shiny star! ⭐

  <picture>
    <source media="(prefers-color-scheme: dark)" srcset="https://api.star-history.com/svg?repos=copier-org/copier&type=Date&theme=dark" />
    <source media="(prefers-color-scheme: light)" srcset="https://api.star-history.com/svg?repos=copier-org/copier&type=Date" />

  </picture>


========================
 S3 compatibility tests
========================

This is a set of unofficial Amazon AWS S3 compatibility
tests, that can be useful to people implementing software
that exposes an S3-like API. The tests use the Boto2 and Boto3 libraries.

The tests use the Tox tool. To get started, ensure you have the ``tox``
software installed; e.g. on Debian/Ubuntu::

	sudo apt-get install tox

You will need to create a configuration file with the location of the
service and two different credentials. A sample configuration file named
``s3tests.conf.SAMPLE`` has been provided in this repo. This file can be
used to run the s3 tests on a Ceph cluster started with vstart.

Once you have that file copied and edited, you can run the tests with::

	S3TEST_CONF=your.conf tox

You can specify which directory of tests to run::

	S3TEST_CONF=your.conf tox -- s3tests_boto3/functional

You can specify which file of tests to run::

	S3TEST_CONF=your.conf tox s3tests_boto3/functional/test_s3.py

You can specify which test to run::

	S3TEST_CONF=your.conf tox s3tests_boto3/functional/test_s3.py::test_bucket_list_empty

Some tests have attributes set based on their current reliability and
things like AWS not enforcing their spec stricly. You can filter tests
based on their attributes::

	S3TEST_CONF=aws.conf tox -- -m 'not fails_on_aws'

Most of the tests have both Boto3 and Boto2 versions. Tests written in
Boto2 are in the ``s3tests`` directory. Tests written in Boto3 are
located in the ``s3test_boto3`` directory.

You can run only the boto3 tests with::

	S3TEST_CONF=your.conf tox -- s3tests_boto3/functional

========================
 STS compatibility tests
========================

This section contains some basic tests for the AssumeRole, GetSessionToken and AssumeRoleWithWebIdentity API's. The test file is located under ``s3tests_boto3/functional``.

To run the STS tests, the vstart cluster should be started with the following parameter (in addition to any parameters already used with it)::

        vstart.sh -o rgw_sts_key=abcdefghijklmnop -o rgw_s3_auth_use_sts=true

Note that the ``rgw_sts_key`` can be set to anything that is 128 bits in length.
After the cluster is up the following command should be executed::

      radosgw-admin caps add --tenant=testx --uid="9876543210abcdef0123456789abcdef0123456789abcdef0123456789abcdef" --caps="roles=*"

You can run only the sts tests (all the three API's) with::

        S3TEST_CONF=your.conf tox s3tests_boto3/functional/test_sts.py

You can filter tests based on the attributes. There is a attribute named ``test_of_sts`` to run AssumeRole and GetSessionToken tests and ``webidentity_test`` to run the AssumeRoleWithWebIdentity tests. If you want to execute only ``test_of_sts`` tests you can apply that filter as below::

        S3TEST_CONF=your.conf tox -- -m test_of_sts s3tests_boto3/functional/test_sts.py

For running ``webidentity_test`` you'll need have Keycloak running.

In order to run any STS test you'll need to add "iam" section to the config file. For further reference on how your config file should look check ``s3tests.conf.SAMPLE``.

========================
 IAM policy tests
========================

This is a set of IAM policy tests.
This section covers tests for user policies such as Put, Get, List, Delete, user policies with s3 actions, conflicting user policies etc
These tests uses Boto3 libraries. Tests are written in the ``s3test_boto3`` directory.

These iam policy tests uses two users with profile name "iam" and "s3 alt" as mentioned in s3tests.conf.SAMPLE.
If Ceph cluster is started with vstart, then above two users will get created as part of vstart with same access key, secrete key etc as mentioned in s3tests.conf.SAMPLE.
Out of those two users, "iam" user is with capabilities --caps=user-policy=* and "s3 alt" user is without capabilities.
Adding above capabilities to "iam" user is also taken care by vstart (If Ceph cluster is started with vstart).

To run these tests, create configuration file with section "iam" and "s3 alt" refer s3tests.conf.SAMPLE.
Once you have that configuration file copied and edited, you can run all the tests with::

	S3TEST_CONF=your.conf tox s3tests_boto3/functional/test_iam.py

You can also specify specific test to run::

	S3TEST_CONF=your.conf tox s3tests_boto3/functional/test_iam.py::test_put_user_policy

Some tests have attributes set such as "fails_on_rgw".
You can filter tests based on their attributes::

	S3TEST_CONF=your.conf tox -- s3tests_boto3/functional/test_iam.py -m 'not fails_on_rgw'


# pygeoapi

[![DOI](https://zenodo.org/badge/121585259.svg)](https://zenodo.org/badge/latestdoi/121585259)
[![Build](https://github.com/geopython/pygeoapi/actions/workflows/main.yml/badge.svg)](https://github.com/geopython/pygeoapi/actions/workflows/main.yml)
[![Docker](https://github.com/geopython/pygeoapi/actions/workflows/containers.yml/badge.svg)](https://github.com/geopython/pygeoapi/actions/workflows/containers.yml)
[![Vulnerabilities](https://github.com/geopython/pygeoapi/actions/workflows/vulnerabilities.yml/badge.svg)](https://github.com/geopython/pygeoapi/actions/workflows/vulnerabilities.yml)

[pygeoapi](https://pygeoapi.io) is a Python server implementation of the [OGC API](https://ogcapi.ogc.org) suite of standards. The project emerged as part of the next generation OGC API efforts in 2018 and provides the capability for organizations to deploy a RESTful OGC API endpoint using OpenAPI, GeoJSON, and HTML. pygeoapi is [open source](https://opensource.org/) and released under an [MIT license](https://github.com/geopython/pygeoapi/blob/master/LICENSE.md).

Please read the docs at [https://docs.pygeoapi.io](https://docs.pygeoapi.io) for more information.


# Recipes for H2O Driverless AI

## About Driverless AI
H2O Driverless AI is Automatic Machine Learning for the Enterprise. Driverless AI automates feature engineering, model building, visualization and interpretability.
- Learn more about Driverless AI from the [H2O.ai website](https://www.h2o.ai/)
- Take the [test drive](https://www.h2o.ai/try-driverless-ai/)
- Go to the [Driverless AI community Slack channel](https://www.h2o.ai/community/driverless-ai-community/#chat) and ask your BYOR related questions in #general

## About BYOR
**BYOR** stands for **Bring Your Own Recipe** and is a key feature of Driverless AI. It allows domain scientists to solve their problems faster and with more precision.

## What are Custom Recipes?
Custom recipes are Python code snippets that can be uploaded into Driverless AI at runtime, like plugins. No need to restart Driverless AI. Custom recipes can be provided for transformers, models and scorers. During training of a supervised machine learning modeling pipeline (aka experiment), Driverless AI can then use these code snippets as building blocks, in combination with all built-in code pieces (or instead of). By providing your own custom recipes, you can gain control over the optimization choices that Driverless AI makes to best solve your machine learning problems.

## Best Practices for Recipes

### Security
* Recipes are meant to be built by people you trust and each recipe should be code-reviewed before going to production.
* Assume that a user with access to Driverless AI has access to the data inside that instance.
  * Apart from securing access to the instance via private networks, various methods of [authentication](http://docs.h2o.ai/driverless-ai/latest-stable/docs/userguide/authentication.html) are possible. Local authentication provides the most control over which users have access to Driverless AI.
  * Unless the `config.toml` setting `enable_dataset_downloading=false` is set, an authenticated user can download all imported datasets as .csv via direct APIs.
* When recipes are enabled (`enable_custom_recipes=true`, the default), be aware that:
  * The code for the recipes runs as the same native Linux user that runs the Driverless AI application.
    * Recipes have explicit access to all data passing through the transformer/model/scorer API
    * Recipes have implicit access to system resources such as disk, memory, CPUs, GPUs, network, etc.
  * An H2O-3 Java process is started in the background, for use by all recipes using H2O-3. Anyone with access to the Driverless AI instance can browse the file system, see models and data through the H2O-3 interface.

* Enable automatic detection of forbidden or dangerous code constructs in a custom recipe with `custom_recipe_security_analysis_enabled = tr
ue`. Note the following:
  * When `custom_recipe_security_analysis_enabled` is enabled, do not use modules specified in the banlist. Specify the banlist with the `cu
stom_recipe_import_banlist` config option.
    * For example: `custom_recipe_import_banlist = ["shlex", "plumbum", "pexpect", "envoy", "commands", "fabric", "subprocess", "os.system",
 "system"]` (default)
  * When `custom_recipe_security_analysis_enabled` is enabled, code is also checked for dangerous calls like `eval()`, `exec()` and other in
secure calls (regex patterns) defined in `custom_recipe_method_call_banlist`. Code is also checked for other dangerous constructs defined as
regex patterns in the `custom_recipe_dangerous_patterns` config setting.
  * Security analysis is only performed on recipes that are uploaded after the `custom_recipe_security_analysis_enabled` config option is en
abled.
  * To specify a list of modules that can be imported in custom recipes, use the `custom_recipe_import_allowlist` config option.
  * The `custom_recipe_security_analysis_enabled` config option is disabled by default.

* Best ways to control access to Driverless AI and custom recipes:
  * Control access to the Driverless AI instance
  * Use local authentication to specify exactly which users are allowed to access Driverless AI
  * Run Driverless AI in a Docker container, as a certain user, with only certain ports exposed, and only certain mount points mapped
  * To disable all recipes: Set `enable_custom_recipes=false` in the config.toml, or add the environment variable `DRIVERLESS_AI_ENABLE_CUSTOM_RECIPES=0` at startup of Driverless AI. This will disable all custom transformers, models and scorers.
  * To disable new recipes: To keep all previously uploaded recipes enabled and disable the upload of any new recipes, set `enable_custom_recipes_upload=false` or `DRIVERLESS_AI_ENABLE_CUSTOM_RECIPES_UPLOAD=0` at startup of Driverless AI.

### Safety
* Driverless AI automatically performs basic acceptance tests for all custom recipes unless disabled
* More information in the FAQ

### Performance
* Use fast and efficient data manipulation tools like `datatable`, `sklearn`, `numpy` or `pandas` instead of Python lists, for-loops etc.
* Use disk sparingly, delete temporary files as soon as possible
* Use memory sparingly, delete objects when no longer needed

## Reference Guide
* [FAQ](https://github.com/h2oai/driverlessai-recipes/blob/master/FAQ.md#faq)
* [Templates](https://github.com/h2oai/driverlessai-recipes/blob/master/FAQ.md#references)
* [Technical Architecture Diagram](https://raw.githubusercontent.com/h2oai/driverlessai-recipes/master/reference/DriverlessAI_BYOR.png)

## Sample Recipes
[Go to Recipes for Driverless 1.7.0](https://github.com/h2oai/driverlessai-recipes/tree/rel-1.7.0)
 [1.7.1](https://github.com/h2oai/driverlessai-recipes/tree/rel-1.7.1)
 [1.8.0](https://github.com/h2oai/driverlessai-recipes/tree/rel-1.8.0)
 [1.8.1](https://github.com/h2oai/driverlessai-recipes/tree/rel-1.8.1)
 [1.8.2](https://github.com/h2oai/driverlessai-recipes/tree/rel-1.8.2)
 [1.8.3](https://github.com/h2oai/driverlessai-recipes/tree/rel-1.8.3)
 [1.8.4](https://github.com/h2oai/driverlessai-recipes/tree/rel-1.8.4)
 [1.8.5](https://github.com/h2oai/driverlessai-recipes/tree/rel-1.8.5)
 [1.8.6](https://github.com/h2oai/driverlessai-recipes/tree/rel-1.8.6)
 [1.8.7](https://github.com/h2oai/driverlessai-recipes/tree/rel-1.8.7)
 [1.8.8](https://github.com/h2oai/driverlessai-recipes/tree/rel-1.8.8)
 [1.8.9](https://github.com/h2oai/driverlessai-recipes/tree/rel-1.8.9)
 [1.8.10](https://github.com/h2oai/driverlessai-recipes/tree/rel-1.8.10)
 [1.9.0](https://github.com/h2oai/driverlessai-recipes/tree/rel-1.9.0)
 [1.9.1](https://github.com/h2oai/driverlessai-recipes/tree/rel-1.9.1)
 [1.9.2](https://github.com/h2oai/driverlessai-recipes/tree/rel-1.9.2)
 [1.9.3](https://github.com/h2oai/driverlessai-recipes/tree/rel-1.9.3)
 [1.10.0](https://github.com/h2oai/driverlessai-recipes/tree/rel-1.10.0)
 [1.10.1](https://github.com/h2oai/driverlessai-recipes/tree/rel-1.10.1)
 [1.10.2](https://github.com/h2oai/driverlessai-recipes/tree/rel-1.10.2)
 [1.10.3](https://github.com/h2oai/driverlessai-recipes/tree/rel-1.10.3)
 [1.10.4](https://github.com/h2oai/driverlessai-recipes/tree/rel-1.10.4)
 [1.10.4.1](https://github.com/h2oai/driverlessai-recipes/tree/rel-1.10.4.1)
 [1.10.4.2](https://github.com/h2oai/driverlessai-recipes/tree/rel-1.10.4.2)
 [1.10.4.3](https://github.com/h2oai/driverlessai-recipes/tree/rel-1.10.4.3)
 [1.10.5](https://github.com/h2oai/driverlessai-recipes/tree/rel-1.10.5)
### Count: 277
* [AIR-GAPPED_INSTALLATIONS](./air-gapped_installations)
  * [load_custom_recipe.py](./air-gapped_installations/load_custom_recipe.py) [Load custom recipes on air-gapped installations.]
* [DATA](./data)
  * [GroupAgg.py](./data/GroupAgg.py) [Aggregation features on numeric columns across multiple categorical columns]
  * [airlines.py](./data/airlines.py) [Create airlines dataset]
  * [airlines_joined_data_flights_in_out.py](./data/airlines_joined_data_flights_in_out.py) [Create augmented airlines datasets]
  * [airlines_joined_data_flights_in_out_regression.py](./data/airlines_joined_data_flights_in_out_regression.py) [Create augmented airlines datasets for regression]
  * [airlines_multiple.py](./data/airlines_multiple.py) [Create airlines dataset]
  * [any_env.py](./data/any_env.py) [Modify dataset with arbitrary env]
  * [any_env2.py](./data/any_env2.py) [Modify dataset with arbitrary env]
  * [any_env3.py](./data/any_env3.py) [Modify dataset with arbitrary env]
  * [any_env4.py](./data/any_env4.py) [Modify dataset with arbitrary env]
  * [audio_to_image.py](./data/audio_to_image.py) [Data recipe to transform input audio to Mel spectrograms]
  * [bigquery_sharded_data_pull.py](./data/bigquery_sharded_data_pull.py) [Pull data >1Gb from GCP Bigquery using sharding functionality.Based on:    https://cloud.google.com/bigquery/docs/exporting-data#exporting_table_data    https://cloud.google.com/storage/docs/reference/libraries#using_the_client_libraryAuthor: Travis CoutureCreated: 03/18/2020Last Updated: 03/18/2020]
  * [catchallenge.py](./data/catchallenge.py) [Create cat challenge dataset]
  * [covidtracking_daily_by_states.py](./data/covidtracking_daily_by_states.py) [Upload daily Covid Tracking (https://covidtracking.com) US States   cases, hospitalization, recovery, test and death data ]
  * [create_transactional_data_or_convert_to_iid.py](./data/create_transactional_data_or_convert_to_iid.py) [Example code to generate and convert transactional data to i.i.d. data.]
  * [creditcard.py](./data/creditcard.py) [Modify credit card dataset]
  * [data_template.py](./data/data_template.py) [Custom data recipe base class]
  * [data_test.py](./data/data_test.py) [Create test dataset]
  * [data_test_multiple.py](./data/data_test_multiple.py) [Create multiple dataset test]
  * [elasticsearch_dataset.py](./data/elasticsearch_dataset.py) [Create a datasets off a scroll search query from ElasticSearch]
  * [feature_selection.py](./data/feature_selection.py) [Perform feature selection by using target perturbation technique]
  * [feature_selection_backward.py](./data/feature_selection_backward.py) [Backward Feature Selection using an sklearn estimator.]
  * [feature_selection_bidirectional.py](./data/feature_selection_bidirectional.py) [Bidirectional Feature Selection using an sklearn estimator.]
  * [feature_selection_exhaustive.py](./data/feature_selection_exhaustive.py) [Exhaustive Feature Selection using an sklearn estimator.]
  * [feature_selection_forward.py](./data/feature_selection_forward.py) [Forward Feature Selection using an sklearn estimator.]
  * [generate_random_int_columns.py](./data/generate_random_int_columns.py) [Data recipe to add one or more columns containing random integers.]
  * [ieee_data_puddle.py](./data/ieee_data_puddle.py) [Data recipe to prepare data for Kaggle IEEE-CIS Fraud Detection https://www.kaggle.com/c/ieee-fraud-detection]
  * [image_cropper.py](./data/image_cropper.py) [ Data Recipe to Crop the Cheque Image (or any Image) in a fixed dimension ]
  * [imdb_datasets.py](./data/imdb_datasets.py) [Create titles and episodes datasets from IMDB tables]
  * [kaggle_bosch.py](./data/kaggle_bosch.py) [Create Bosch competition datasets with leak]
  * [kaggle_ieee_fraud.py](./data/kaggle_ieee_fraud.py) [Data recipe to prepare data for Kaggle IEEE-CIS Fraud Detection https://www.kaggle.com/c/ieee-fraud-detection]
  * [kaggle_m5.py](./data/kaggle_m5.py) [Prepare data for m5 Kaggle Time-Series Forecast competition]
  * [keywords_data.py](./data/keywords_data.py) [Check and match a list of words from a specific string column]
  * [load_json_from_zip.py](./data/load_json_from_zip.py) [Data Recipe to load JSON datasets from a zip file. Just include this script inside the zip and upload it as a data recipe.]
  * [load_sas7bdat.py](./data/load_sas7bdat.py) [Data Recipe to load a single sas file]
  * [load_sas7bdat_from_zip.py](./data/load_sas7bdat_from_zip.py) [Data Recipe to load sas7bdat datasets from a zip file. Just include this script inside the zip and upload it as a data recipe.]
  * [load_xml_from_zip.py](./data/load_xml_from_zip.py) [Data Recipe to load XML datasets from a zip file. Just include this script inside the zip and upload it as a data recipe. ]
  * [marketbasketanalysis.py](./data/marketbasketanalysis.py) [Market basket analysis]
  * [mnist.py](./data/mnist.py) [Prep and upload the MNIST datasset]
  * [mozilla_deepspeech_wav2txt.py](./data/mozilla_deepspeech_wav2txt.py) [Speech to text using Mozilla's DeepSpeechSettings for this recipe:Assing MODEL_PATH global variable prior to usageAssign WAV_COLNAME global variable with proper column name from your dataset.This colums should contain absolute paths to .wav file which needs to be converted to text.General requirements to .wav's:1 channel (mono)16 bit16000 frequency]
  * [nytimes_covid19_cases_deaths_by_counties.py](./data/nytimes_covid19_cases_deaths_by_counties.py) [Upload daily COVID-19 cases and deaths in US by counties - NY Times github   Source:  nytimes/covid-19-data Coronavirus (Covid-19) Data in the United States   https://github.com/nytimes/covid-19-data]
  * [nytimes_covid19_cases_deaths_by_states.py](./data/nytimes_covid19_cases_deaths_by_states.py) [Upload daily COVID-19 cases and deaths in US by states from NY Times github]
  * [nytimes_covid19_cases_deaths_us.py](./data/nytimes_covid19_cases_deaths_us.py) [Upload daily COVID-19 cases and deaths in US total from NY Times github]
  * [owid_covid19_cases_deaths_by_countries.py](./data/owid_covid19_cases_deaths_by_countries.py) [Upload daily COVID-19 cases and deaths by countries   Source: Our World in Data. It is updated daily and includes data on confirmed cases, deaths, and testing.   https://ourworldindata.org/coronavirus-source-data]
  * [rowwise_aggregates_by_column_groups.py](./data/rowwise_aggregates_by_column_groups.py) [Augments dataset by computing rowwise aggregates by column groups]
  * [seattle_rain_modify.py](./data/seattle_rain_modify.py) [Transpose the Monthly Seattle Rain Inches data set for Time Series use cases]
  * [seattle_rain_upload.py](./data/seattle_rain_upload.py) [Upload Monthly Seattle Rain Inches data set from data provided by the City of Seattle]
  * [ts_fill_n_cluster.py](./data/ts_fill_n_cluster.py) [Data Recipe to fill missing values in TS data and then create new data sets from TS Clustering]
  * [two_sigma_rental.py](./data/two_sigma_rental.py) [Download and preprocess datasets from the Two Sigma Rental competition on Kaggle]
  * [video_to_image.py](./data/video_to_image.py) [Data recipe to transform input video to the images]
  * [wav2txt.py](./data/wav2txt.py) [Speech to text using Azure Cognitive ServicesSettings for this recipe:Assing AZURE_SERVICE_KEY and AZURE_SERVICE_REGION global variable prior to usageAssign WAV_COLNAME global variable with proper column name from your dataset.This colums should contain absolute paths to .wav file which needs to be converted to text.]
  * [DATABASES](./data/databases)
    * [create_dataset_from_mongodb_collection.py](./data/databases/create_dataset_from_mongodb_collection.py) [Create dataset from MonogDB]
  * [NLP](./data/nlp)
    * [sentiment_score.py](./data/nlp/sentiment_score.py) [Data recipe to get sentiment score using textblob]
    * [sentiment_score_vader.py](./data/nlp/sentiment_score_vader.py) [Data recipe to get sentiment score using vader]
    * [text_summarization.py](./data/nlp/text_summarization.py) [Data recipe to get summary of text using gensim]
    * [tokenize_chinese.py](./data/nlp/tokenize_chinese.py) [Chinese text tokenization using jieba package - https://github.com/fxsjy/jieba]
    * [tokenize_japanese.py](./data/nlp/tokenize_japanese.py) [Japanese text tokenization using janome package - https://mocobeta.github.io/janome/en/]
    * [topic_modeling.py](./data/nlp/topic_modeling.py) [Data recipe to perform topic modeling]
    * [twitter_preprocessing_recipe.py](./data/nlp/twitter_preprocessing_recipe.py) [Preprocess the tweets by normalising username, removing unnecessary punctuations, expanding the hashtags]
  * [UNSUPERVISED](./data/unsupervised)
    * [DBScanClustering.py](./data/unsupervised/DBScanClustering.py) [Data Recipe to perform DBScan Clustering.]
    * [IsolationForestRecipe.py](./data/unsupervised/IsolationForestRecipe.py) [Data Recipe to perform Isolation Forest Clustering on a dataset.]
    * [KMeansClustering.py](./data/unsupervised/KMeansClustering.py) [Data Recipe to perform KMeans Clustering on a dataset.]
    * [Kmeans_with_adjusted_mutual_info_score.py](./data/unsupervised/Kmeans_with_adjusted_mutual_info_score.py) [Kmeans Clustering with Adjusted Mutual Info Score to evaluate cluster quality.]
    * [Kmeans_with_fowlkes_mallows_score.py](./data/unsupervised/Kmeans_with_fowlkes_mallows_score.py) [Data Recipe to perform KMeans Clustering with Fowlkes Mallows Score.]
    * [Kmeans_with_homogeneity_score.py](./data/unsupervised/Kmeans_with_homogeneity_score.py) [Data Recipe to perform KMeans Clustering on a dataset with homogenity score to evaluate cluster quality.]
    * [KmodesRecipe.py](./data/unsupervised/KmodesRecipe.py) [Data Recipe to perform KModes Clustering on a dataset.]
* [EXPLAINERS](./explainers)
  * [DOC](./explainers/doc)
    * [API](./explainers/doc/api)
    * [IMAGES](./explainers/doc/images)
  * [EXPLAINERS](./explainers/explainers)
    * [ale_explainer.py](./explainers/explainers/ale_explainer.py) [Accumulated Local Effects (ALE) explainerNote:This example repurposes the Partial Dependence format render data. As such, the label"Average Prediction of {response}" is used for the y-axis instead of "ALE of {response}".]
    * [morris_sensitivity_explainer.py](./explainers/explainers/morris_sensitivity_explainer.py) [Morris Sensitivity Analysis Explainer]
    * [EXAMPLES](./explainers/explainers/examples)
      * [example_compatibility_check_explainer.py](./explainers/explainers/examples/example_compatibility_check_explainer.py) [Runs compatibility check for MLI recipe.]
      * [example_custom_explanation_explainer.py](./explainers/explainers/examples/example_custom_explanation_explainer.py) [Example of how to define custom MLI explanations.]
      * [example_dai_metadata_explainer.py](./explainers/explainers/examples/example_dai_metadata_explainer.py) [Example of how to print MLI explainer metadata.]
      * [example_eda_explainer.py](./explainers/explainers/examples/example_eda_explainer.py) [Example of MLI explainer for Exploratory Data Analysis]
      * [example_hello_world_explainer.py](./explainers/explainers/examples/example_hello_world_explainer.py) [Hello World! explainer example.]
      * [example_logging_explainer.py](./explainers/explainers/examples/example_logging_explainer.py) [Example of how to use logging on MLI explainers.]
      * [example_params_explainer.py](./explainers/explainers/examples/example_params_explainer.py) [Example of how to define MLI explainer parameters.]
      * [example_persistence_explainer.py](./explainers/explainers/examples/example_persistence_explainer.py) [Example of how to use the persistence object inside an MLI explainer to access FS directories.]
      * [example_score_explainer.py](./explainers/explainers/examples/example_score_explainer.py) [Example of how to use a model's predict method inside an MLI explainer.]
    * [TEMPLATES](./explainers/explainers/templates)
      * [template_dt_explainer.py](./explainers/explainers/templates/template_dt_explainer.py) [Decision Tree explainer which can be used to create explainer with global and local decision tree explanations.]
      * [template_featimp_explainer.py](./explainers/explainers/templates/template_featimp_explainer.py) [Feature importance explainer template which can be used create explainer with global and local feature importance explanations.]
      * [template_md_explainer.py](./explainers/explainers/templates/template_md_explainer.py) [Markdown report with raster image chart explainer template which can be used to create explainer with global report explanations.]
      * [template_md_featimp_summary_explainer.py](./explainers/explainers/templates/template_md_featimp_summary_explainer.py) [Markdown report with summary feature importance chart explainer template which can be used to create explainer with global report explanations.]
      * [template_md_vega_explainer.py](./explainers/explainers/templates/template_md_vega_explainer.py) [Markdown report with Vega chart explainer template which can be used to create explainer which creates global report explanations.]
      * [template_pd_explainer.py](./explainers/explainers/templates/template_pd_explainer.py) [PD and ICE explainer template which can be used to create example with partial dependence (global) and individual conditional explanations (local) explanations.]
      * [template_scatter_plot_explainer.py](./explainers/explainers/templates/template_scatter_plot_explainer.py) [Scatter plot explainer template which can be used to create explainer with global and local explanations.]
  * [NOTEBOOKS](./explainers/notebooks)
* [HOW_TO_WRITE_A_RECIPE](./how_to_write_a_recipe)
  * [ExampleLogTransformer.py](./how_to_write_a_recipe/ExampleLogTransformer.py) [Apply log transform to any single numeric column.]
* [INDIVIDUALS](./individuals)
  * [credit_card.py](./individuals/credit_card.py) [Custom Final Individual 0 from Experiment test_credit_card_9f03b_f55d_pawisaco ]
  * [indiv_test1.py](./individuals/indiv_test1.py) [Test individual 1]
  * [indiv_test2.py](./individuals/indiv_test2.py) [Test individual 2]
  * [individual_template.py](./individuals/individual_template.py) [Template base class for a custom individual recipe.]
  * [paribas_basemodel0.py](./individuals/paribas_basemodel0.py) [Custom Individual 0 from Experiment satocemi ]
  * [walmart_lags.py](./individuals/walmart_lags.py) [Custom Final Individual 0 from Experiment test_time_series_walmart_nolimits_3f587_2993_miduhoge ]
* [MODELS](./models)
  * [mock_gpu_test_model.py](./models/mock_gpu_test_model.py) [For GPU usage testing purposes.]
  * [mock_tf2_test.py](./models/mock_tf2_test.py) [For GPU usage testing purposes.]
  * [mock_tf_test.py](./models/mock_tf_test.py) [For GPU usage testing purposes.]
  * [model_template.py](./models/model_template.py) [Template base class for a custom model recipe.]
  * [ALGORITHMS](./models/algorithms)
    * [adaboost.py](./models/algorithms/adaboost.py) [Adaboost model from sklearn]
    * [autogluon.py](./models/algorithms/autogluon.py) [AutoGluon + RAPIDS]
    * [calibratedClassifier.py](./models/algorithms/calibratedClassifier.py) [ Calibrated Classifier Model: To calibrate predictions using Platt's scaling, Isotonic Regression or Splines] <kbd>✓ MOJO Enabled</kbd>
    * [catboost.py](./models/algorithms/catboost.py) [CatBoost gradient boosting by Yandex. Currently supports regression and binary classification.]
    * [catboost_regression_uncertanity.py](./models/algorithms/catboost_regression_uncertanity.py) [please add description]
    * [daal_trees.py](./models/algorithms/daal_trees.py) [Binary Classification and Regression for Decision Forest and Gradient Boosting based on Intel DAAL]
    * [datatable_glm.py](./models/algorithms/datatable_glm.py) [datatable Linear Model]
    * [extra_trees.py](./models/algorithms/extra_trees.py) [Extremely Randomized Trees (ExtraTrees) model from sklearn]
    * [extremeClassifier.py](./models/algorithms/extremeClassifier.py) [ Extreme Classifier Model: To speed up train of multiclass model (100s of classes) for lightGBM.    Caution: can only be used for AUC (or GINI) and accuracy metrics.    Based on: Extreme Classification in Log Memory using Count-Min Sketch: https://arxiv.org/abs/1910.13830]
    * [h2o-3-gbm-poisson.py](./models/algorithms/h2o-3-gbm-poisson.py) [H2O-3 Distributed Scalable Machine Learning Models: Poisson GBM]
    * [h2o-3-models.py](./models/algorithms/h2o-3-models.py) [H2O-3 Distributed Scalable Machine Learning Models (DL/GLM/GBM/DRF/NB/AutoML)]
    * [h2o-glm-poisson.py](./models/algorithms/h2o-glm-poisson.py) [H2O-3 Distributed Scalable Machine Learning Models: Poisson GLM]
    * [kernel_naive_bayes.py](./models/algorithms/kernel_naive_bayes.py) [Kernel Naive Bayes implementation by sklearn. For small data (< 200k rows).]
    * [knearestneighbour.py](./models/algorithms/knearestneighbour.py) [K-Nearest Neighbor implementation by sklearn. For small data (< 200k rows).]
    * [lda.py](./models/algorithms/lda.py) [Linear/Quadratic Discriminant Analysis (LDA/QDA) model from sklearn]
    * [libfm_fastfm.py](./models/algorithms/libfm_fastfm.py) [LibFM implementation of fastFM ]
    * [linear_svm.py](./models/algorithms/linear_svm.py) [Linear Support Vector Machine (SVM) implementation by sklearn. For small data.]
    * [logistic_regression.py](./models/algorithms/logistic_regression.py) [Logistic Regression based upon sklearn.]
    * [nusvm.py](./models/algorithms/nusvm.py) [Nu-SVM implementation by sklearn. For small data.]
    * [quantile_forest.py](./models/algorithms/quantile_forest.py) [Quantile Random Forest Regression model from skgarden]
    * [random_forest.py](./models/algorithms/random_forest.py) [Random Forest (RandomForest) model from sklearn]
  * [CUSTOM_LOSS](./models/custom_loss)
    * [lightgbm_quantile_regression.py](./models/custom_loss/lightgbm_quantile_regression.py) [Modified version of Driverless AI's internal LightGBM implementation with for quantile regression] <kbd>✓ MOJO Enabled</kbd>
    * [lightgbm_train_valid_score_gap.py](./models/custom_loss/lightgbm_train_valid_score_gap.py) [Modified version of Driverless AI's internal LightGBM implementation with a custom objective function (used for tree split finding).] <kbd>✓ MOJO Enabled</kbd>
    * [lightgbm_tweedie.py](./models/custom_loss/lightgbm_tweedie.py) [Modified version of Driverless AI's internal LightGBM implementation with tweedie distribution] <kbd>✓ MOJO Enabled</kbd>
    * [lightgbm_with_custom_loss.py](./models/custom_loss/lightgbm_with_custom_loss.py) [Modified version of Driverless AI's internal LightGBM implementation with a custom objective function (used for tree split finding).] <kbd>✓ MOJO Enabled</kbd>
    * [xgboost_with_custom_loss.py](./models/custom_loss/xgboost_with_custom_loss.py) [Modified version of Driverless AI's internal XGBoost implementation with a custom objective function (used for tree split finding).] <kbd>✓ MOJO Enabled</kbd>
  * [MLI](./models/mli)
    * [model_decision_tree_linear_combo.py](./models/mli/model_decision_tree_linear_combo.py) [Decision tree plus linear model]
    * [model_ebm.py](./models/mli/model_ebm.py) [Explainable Boosting Machines (EBM), implementation of GA2M with option for user-defined interaction between features. ]
    * [model_gam.py](./models/mli/model_gam.py) [Generalized Additive Model]
    * [model_skopes_rules.py](./models/mli/model_skopes_rules.py) [Skopes rules ]
    * [model_xnn.py](./models/mli/model_xnn.py) [ Explainable neural net ]
    * [FAIRNESS](./models/mli/fairness)
      * [model_fairxgb.py](./models/mli/fairness/model_fairxgb.py) [Fair XGB ]
      * [reweighing_recipe.py](./models/mli/fairness/reweighing_recipe.py) [Debiasing using reweighing]
  * [MONOTONIC](./models/monotonic)
    * [monotonic_models.py](./models/monotonic/monotonic_models.py) [LightGBM/XGBoostGBM/DecisionTree with user-given monotonicity constraints (1/-1/0) for original numeric features]
  * [NLP](./models/nlp)
    * [portuguese_bert.py](./models/nlp/portuguese_bert.py) [Custom Bert model pretrained on Portuguese.]
    * [text_binary_count_logistic.py](./models/nlp/text_binary_count_logistic.py) [Text classification model using binary count of words]
    * [text_tfidf_model.py](./models/nlp/text_tfidf_model.py) [Text classification / regression model using TFIDF]
    * [text_tfidf_model_continuous.py](./models/nlp/text_tfidf_model_continuous.py) [Text classification model using TFIDF]
  * [TIMESERIES](./models/timeseries)
    * [autoarima_parallel.py](./models/timeseries/autoarima_parallel.py) [autoarima for TimeSeries with an example of parameter mutation.]
    * [exponential_smoothing.py](./models/timeseries/exponential_smoothing.py) [Linear Model on top of Exponential Weighted Moving Average Lags for Time-Series. Provide appropriate lags and past outcomes during batch scoring for best results.]
    * [fb_prophet.py](./models/timeseries/fb_prophet.py) [Prophet by Facebook for TimeSeries with an example of parameter mutation.]
    * [fb_prophet_parallel.py](./models/timeseries/fb_prophet_parallel.py) [Prophet by Facebook for TimeSeries with an example of parameter mutation.]
    * [historic_mean.py](./models/timeseries/historic_mean.py) [Historic Mean for Time-Series problems. Predicts the mean of the target for each timegroup for regression problems.]
    * [nixtla_arimax.py](./models/timeseries/nixtla_arimax.py) [AutoARIMA for TimeSeries Forecasting Uses AutoARIMA implemented in https://github.com/Nixtla/statsforecastCurrent limitations:(1) No handling for prediction "gap"(2) Enable user to pass in seasonality length to run AutoARIMA with seasonality.  NOTE: Not passing any seasonality parameter will result in ARIMA (current default)]
    * [nixtla_ces.py](./models/timeseries/nixtla_ces.py) [AutoCES for TimeSeries Forecasting Uses AutoCES implemented in https://github.com/Nixtla/statsforecastCurrent limitations:(1) No handling for prediction "gap"(2) Enable user to pass in seasonality length to run AutoCES with seasonality.  NOTE: The seasonality will be infered based on the temporal column.]
    * [nixtla_ets.py](./models/timeseries/nixtla_ets.py) [AutoETS for TimeSeries Forecasting Uses AutoETS implemented in https://github.com/Nixtla/statsforecastCurrent limitations:(1) No handling for prediction "gap"(2) Enable user to pass in seasonality length to run AutoETS with seasonality.  NOTE: The seasonality will be infered based on the temporal column.]
    * [nixtla_theta.py](./models/timeseries/nixtla_theta.py) [AutoTheta for TimeSeries Forecasting Uses AutoTheta implemented in https://github.com/Nixtla/statsforecastCurrent limitations:(1) No handling for prediction "gap"(2) Enable user to pass in seasonality length to run AutoTheta with seasonality.  NOTE: The seasonality will be infered based on the temporal column.]
    * [ts_rnn.py](./models/timeseries/ts_rnn.py) [Recurrent Neural Networks with Convolution for Time-series problems using LSTM, GRU or Elman cells.]
  * [UNSUPERVISED](./models/unsupervised)
    * [TextKMeansIsolationForest.py](./models/unsupervised/TextKMeansIsolationForest.py) [Variety of unsupervised models that mimic internal versions but includes text handling via text embedding using custom transformer]
    * [TextSentiment.py](./models/unsupervised/TextSentiment.py) [Extract sentiment from text using pretrained models from TextBlob]
    * [aggregator.py](./models/unsupervised/aggregator.py) [Unsupervised Aggregator algorithm (by Leland Wilkinson) to segment data into user-given number of exemplars]
    * [dbscan.py](./models/unsupervised/dbscan.py) [Clustering using DBScan]
    * [isolation_forest_custom_scorer.py](./models/unsupervised/isolation_forest_custom_scorer.py) [Custom Anomaly Score for Isolation Forest]
    * [lda.py](./models/unsupervised/lda.py) [Unsupervised way to extract topic information from one text column]
    * [local_outlier_factor.py](./models/unsupervised/local_outlier_factor.py) [Outlier detection with Local Outlier Factor]
    * [pytod_lof.py](./models/unsupervised/pytod_lof.py) [Local Outlier Factor method for outlier detection, based on PyTOD library.]
    * [rapids_kmeans.py](./models/unsupervised/rapids_kmeans.py) [KMeans clustering using RAPIDS.ai]
  * [UPLIFT](./models/uplift)
    * [GLMTransformedOutcome.py](./models/uplift/GLMTransformedOutcome.py) [GLM for uplift modeling] <kbd>✓ MOJO Enabled</kbd>
    * [LightGBMTransformedOutcome.py](./models/uplift/LightGBMTransformedOutcome.py) [LightGBM for uplift modeling] <kbd>✓ MOJO Enabled</kbd>
    * [XGBoostTransformedOutcome.py](./models/uplift/XGBoostTransformedOutcome.py) [XGBoost for uplift modeling] <kbd>✓ MOJO Enabled</kbd>
* [RECIPES](./recipes)
  * [amazon.py](./recipes/amazon.py) [Recipe for Kaggle Competition: Amazon.com - Employee Access Challenge]
* [REFERENCE](./reference)
* [SCORERS](./scorers)
  * [huber_loss.py](./scorers/huber_loss.py) [Huber Loss for Regression or Binary Classification. Robust loss, combination of quadratic loss and linear loss.]
  * [scorer_template.py](./scorers/scorer_template.py) [Template base class for a custom scorer recipe.]
  * [CLASSIFICATION](./scorers/classification)
    * [f3_score.py](./scorers/classification/f3_score.py) [F3 Score]
    * [f4_score.py](./scorers/classification/f4_score.py) [F4 Score]
    * [probF.py](./scorers/classification/probF.py) [Probabilistic F Score with optimized threshold]
    * [BINARY](./scorers/classification/binary)
      * [average_mcc.py](./scorers/classification/binary/average_mcc.py) [Averaged Matthews Correlation Coefficient (averaged over several thresholds, for imbalanced problems). Example how to use Driverless AI's internal scorer.]
      * [balanced_accuracy.py](./scorers/classification/binary/balanced_accuracy.py) [balanced_accuracy_score]
      * [brier_loss.py](./scorers/classification/binary/brier_loss.py) [Brier Loss]
      * [cost.py](./scorers/classification/binary/cost.py) [Using hard-coded dollar amounts x for false positives and y for false negatives, calculate the cost of a model using: `(x * FP + y * FN) / N`]
      * [cost_access_to_data.py](./scorers/classification/binary/cost_access_to_data.py) [Same as CostBinary, but provides access to full Data]
      * [cost_smooth.py](./scorers/classification/binary/cost_smooth.py) [Using hard-coded dollar amounts x for false positives and y for false negatives, calculate the cost of a model using: `(1 - y_true) * y_pred * fp_cost + y_true * (1 - y_pred) * fn_cost`]
      * [fair_auc.py](./scorers/classification/binary/fair_auc.py) [Custom scorer for detecting and reducing bias in machine learning models.]
      * [kolmogorov_smirnov.py](./scorers/classification/binary/kolmogorov_smirnov.py) [Kolmogorov-Smirnov scorer recipe.If you need to print debug messages into DAI log, uncomment lines with logger and loggerinfo.Starting 1.10.2 - DAI handles exceptions raised by custom scorers.Default DAI behavior is to continue experiment in case of Scorer failure.To enable forcing experiment to fail, in case of scorer error, set following parameters in DAI:  - skip_scorer_failures=false (Disabled)  - skip_model_failures=false (Disabled)]
      * [logloss_with_costs.py](./scorers/classification/binary/logloss_with_costs.py) [Logloss with costs associated with each type of 4 outcomes - typically applicable to fraud use case]
      * [marketing_campaign.py](./scorers/classification/binary/marketing_campaign.py) [Computes the mean profit per outbound marketing letter, given a fraction of the population addressed, and fixed cost and reward]
      * [profit.py](./scorers/classification/binary/profit.py) [Profit Scorer for binary classification]
    * [MULTICLASS](./scorers/classification/multiclass)
      * [hamming_loss.py](./scorers/classification/multiclass/hamming_loss.py) [Hamming Loss - Misclassification Rate (1 - Accuracy)]
      * [linear_weighted_kappa.py](./scorers/classification/multiclass/linear_weighted_kappa.py) [Cohen’s Kappa with linear weights]
      * [map@k.py](./scorers/classification/multiclass/map@k.py) [Mean Average Precision @ k (MAP@k)]
      * [quadratic_weighted_kappa.py](./scorers/classification/multiclass/quadratic_weighted_kappa.py) [Qudratic Weighted Kappa]
  * [REGRESSION](./scorers/regression)
    * [WAPE_scorer.py](./scorers/regression/WAPE_scorer.py) [Weighted Absoluted Percent Error]
    * [asymmetric_mae.py](./scorers/regression/asymmetric_mae.py) [MAE with a penalty that differs for positive and negative errors]
    * [auuc.py](./scorers/regression/auuc.py) [Area under uplift curve]
    * [cosh_loss.py](./scorers/regression/cosh_loss.py) [Hyperbolic Cosine Loss]
    * [explained_variance.py](./scorers/regression/explained_variance.py) [Explained Variance. Fraction of variance that is explained by the model.]
    * [gamma_deviance.py](./scorers/regression/gamma_deviance.py) [Gamma Deviance scorer recipe.This is same as Tweedie Deviance scorer with power=2If you need to print debug messages into DAI log, uncomment lines with logger and loggerinfo.Starting 1.10.2 - DAI handles exceptions raised by custom scorers.Default DAI behavior is to continue experiment in case of Scorer failure.To enable forcing experiment to fail, in case of scorer error, set following parameters in DAI:  - skip_scorer_failures=false (Disabled)  - skip_model_failures=false (Disabled)]
    * [largest_error.py](./scorers/regression/largest_error.py) [Largest error for regression problems. Highly sensitive to outliers.]
    * [log_mae.py](./scorers/regression/log_mae.py) [Log Mean Absolute Error for regression]
    * [mean_absolute_scaled_error.py](./scorers/regression/mean_absolute_scaled_error.py) [Mean Absolute Scaled Error for time-series regression]
    * [mean_squared_log_error.py](./scorers/regression/mean_squared_log_error.py) [Mean Squared Log Error for regression]
    * [median_absolute_error.py](./scorers/regression/median_absolute_error.py) [Median Absolute Error for regression]
    * [pearson_correlation.py](./scorers/regression/pearson_correlation.py) [Pearson Correlation Coefficient for regression]
    * [poisson_deviance.py](./scorers/regression/poisson_deviance.py) [Poisson Deviance scorer recipe.]
    * [quantile_loss.py](./scorers/regression/quantile_loss.py) [Quantile Loss regression]
    * [r2_by_tgc.py](./scorers/regression/r2_by_tgc.py) [Custom R2 scorer computes R2 on each time series, then averages them out for the final score.]
    * [rmse_with_x.py](./scorers/regression/rmse_with_x.py) [Custom RMSE Scorer that also gets X (original features) - for demo/testing purposes only]
    * [top_decile.py](./scorers/regression/top_decile.py) [Median Absolute Error for predictions in the top decile]
    * [tweedie_deviance.py](./scorers/regression/tweedie_deviance.py) [Tweedie Deviance scorer recipe.User inputs can be provided through recipe_dict in config.To pass power parameterrecipe_dict = "{'power':2.0}"The default value is 1.5If you need to print debug messages into DAI log, uncomment lines with logger and loggerinfo.Starting 1.10.2 - DAI handles exceptions raised by custom scorers.Default DAI behavior is to continue experiment in case of Scorer failure.To enable forcing experiment to fail, in case of scorer error, set following parameters in DAI:  - skip_scorer_failures=false (Disabled)  - skip_model_failures=false (Disabled)]
* [TRANSFORMERS](./transformers)
  * [how_to_debug_transformer.py](./transformers/how_to_debug_transformer.py) [Example how to debug a transformer outside of Driverless AI (optional)]
  * [how_to_test_from_py_client.py](./transformers/how_to_test_from_py_client.py) [Testing a BYOR Transformer the PyClient - works on 1.7.0 & 1.7.1-17]
  * [transformer_template.py](./transformers/transformer_template.py) [Template base class for a custom transformer recipe.]
  * [ANOMALY ](./transformers/anomaly )
    * [isolation_forest.py](./transformers/anomaly /isolation_forest.py) [H2O-3 Distributed Scalable Machine Learning Transformers (IF)]
  * [AUGMENTATION](./transformers/augmentation)
    * [germany_landers_holidays.py](./transformers/augmentation/germany_landers_holidays.py) [Returns a flag for whether a date falls on a holiday for each of Germany's Bundeslaender. ]
    * [holidays_this_week.py](./transformers/augmentation/holidays_this_week.py) [Returns the amount of US holidays for a given week]
    * [ipaddress_features.py](./transformers/augmentation/ipaddress_features.py) [Parses IP addresses and networks and extracts its properties.]
    * [is_ramadan.py](./transformers/augmentation/is_ramadan.py) [Returns a flag for whether a date falls on Ramadan in Saudi Arabia]
    * [singapore_public_holidays.py](./transformers/augmentation/singapore_public_holidays.py) [Flag for whether a date falls on a public holiday in Singapore.]
    * [usairportcode_origin_dest.py](./transformers/augmentation/usairportcode_origin_dest.py) [Transformer to parse and augment US airport codes with geolocation info.]
    * [usairportcode_origin_dest_geo_features.py](./transformers/augmentation/usairportcode_origin_dest_geo_features.py) [Transformer to augment US airport codes with geolocation info.]
    * [uszipcode_features_database.py](./transformers/augmentation/uszipcode_features_database.py) [Transformer to parse and augment US zipcodes with info from zipcode database.]
    * [uszipcode_features_light.py](./transformers/augmentation/uszipcode_features_light.py) [Lightweight transformer to parse and augment US zipcodes with info from zipcode database.]
  * [DATETIME](./transformers/datetime)
    * [datetime_diff_transformer.py](./transformers/datetime/datetime_diff_transformer.py) [Difference in time between two datetime columns]
    * [datetime_encoder_transformer.py](./transformers/datetime/datetime_encoder_transformer.py) [Converts datetime column into an integer (milliseconds since 1970)]
    * [days_until_dec2020.py](./transformers/datetime/days_until_dec2020.py) [Creates new feature for any date columns, by computing the difference in days between the date value and 31st Dec 2020]
  * [EXECUTABLES](./transformers/executables)
    * [pe_data_directory_features.py](./transformers/executables/pe_data_directory_features.py) [Extract LIEF features from PE files]
    * [pe_exports_features.py](./transformers/executables/pe_exports_features.py) [Extract LIEF features from PE files]
    * [pe_general_features.py](./transformers/executables/pe_general_features.py) [Extract LIEF features from PE files]
    * [pe_header_features.py](./transformers/executables/pe_header_features.py) [Extract LIEF features from PE files]
    * [pe_imports_features.py](./transformers/executables/pe_imports_features.py) [Extract LIEF features from PE files]
    * [pe_normalized_byte_count.py](./transformers/executables/pe_normalized_byte_count.py) [Extract LIEF features from PE files]
    * [pe_section_characteristics.py](./transformers/executables/pe_section_characteristics.py) [Extract LIEF features from PE files]
    * [DATA](./transformers/executables/data)
  * [GENERIC](./transformers/generic)
    * [count_missing_values_transformer.py](./transformers/generic/count_missing_values_transformer.py) [Count of missing values per row]
    * [missing_flag_transformer.py](./transformers/generic/missing_flag_transformer.py) [Returns 1 if a value is missing, or 0 otherwise]
    * [specific_column_transformer.py](./transformers/generic/specific_column_transformer.py) [Example of a transformer that operates on the entire original frame, and hence on any column(s) desired.]
  * [GEOSPATIAL](./transformers/geospatial)
    * [geodesic.py](./transformers/geospatial/geodesic.py) [Calculates the distance in miles between two latitude/longitude points in space]
    * [myhaversine.py](./transformers/geospatial/myhaversine.py) [Computes miles between first two *_latitude and *_longitude named columns in the data set]
  * [HIERARCHICAL](./transformers/hierarchical)
    * [firstNCharCVTE.py](./transformers/hierarchical/firstNCharCVTE.py) [Target-encode high cardinality categorical text by their first few characters in the string ]
    * [log_scale_target_encoding.py](./transformers/hierarchical/log_scale_target_encoding.py) [Target-encode numbers by their logarithm]
  * [IMAGE](./transformers/image)
    * [image_ocr_transformer.py](./transformers/image/image_ocr_transformer.py) [Convert a path to an image to text using OCR based on tesseract]
    * [image_url_transformer.py](./transformers/image/image_url_transformer.py) [Convert a path to an image (JPG/JPEG/PNG) to a vector of class probabilities created by a pretrained ImageNet deeplearning model (Keras, TensorFlow).]
  * [NLP](./transformers/nlp)
    * [continuous_TextTransformer.py](./transformers/nlp/continuous_TextTransformer.py) [Creates a TF-IDF based text transformation that can be continuously updated with new data and vocabulary.] <kbd>✓ MOJO Enabled</kbd>
    * [fuzzy_text_similarity_transformers.py](./transformers/nlp/fuzzy_text_similarity_transformers.py) [Row-by-row similarity between two text columns based on FuzzyWuzzy]
    * [text_binary_count_transformer.py](./transformers/nlp/text_binary_count_transformer.py) [Explainable Text transformer that uses binary counts of words using sklearn's CountVectorizer]
    * [text_char_tfidf_count_transformers.py](./transformers/nlp/text_char_tfidf_count_transformers.py) [Character level TFIDF and Count followed by Truncated SVD on text columns]
    * [text_embedding_similarity_transformers.py](./transformers/nlp/text_embedding_similarity_transformers.py) [Row-by-row similarity between two text columns based on pretrained Deep Learning embedding space]
    * [text_lang_detect_transformer.py](./transformers/nlp/text_lang_detect_transformer.py) [Detect the language for a text value using Google's 'langdetect' package]
    * [text_meta_transformers.py](./transformers/nlp/text_meta_transformers.py) [Extract common meta features from text]
    * [text_named_entities_transformer.py](./transformers/nlp/text_named_entities_transformer.py) [Extract the counts of different named entities in the text (e.g. Person, Organization, Location)]
    * [text_pos_tagging_transformer.py](./transformers/nlp/text_pos_tagging_transformer.py) [Extract the count of nouns, verbs, adjectives and adverbs in the text]
    * [text_preprocessing_transformer.py](./transformers/nlp/text_preprocessing_transformer.py) [Preprocess the text column by stemming, lemmatization and stop word removal]
    * [text_readability_transformers.py](./transformers/nlp/text_readability_transformers.py) [    Custom Recipe to extract Readability features from the text data]
    * [text_sentiment_transformer.py](./transformers/nlp/text_sentiment_transformer.py) [Extract sentiment from text using pretrained models from TextBlob]
    * [text_similarity_transformers.py](./transformers/nlp/text_similarity_transformers.py) [Row-by-row similarity between two text columns based on common N-grams, Jaccard similarity, Dice similarity and edit distance.]
    * [text_spelling_correction_transformers.py](./transformers/nlp/text_spelling_correction_transformers.py) [Correct the spelling of text column]
    * [text_topic_modeling_transformer.py](./transformers/nlp/text_topic_modeling_transformer.py) [Extract topics from text column using LDA]
    * [text_url_summary_transformer.py](./transformers/nlp/text_url_summary_transformer.py) [Extract text from URL and summarizes it]
    * [vader_text_sentiment_transformer.py](./transformers/nlp/vader_text_sentiment_transformer.py) [Extract sentiment from text using lexicon and rule-based sentiment analysis tool called VADER]
  * [NUMERIC](./transformers/numeric)
    * [boxcox_transformer.py](./transformers/numeric/boxcox_transformer.py) [Box-Cox Transform]
    * [clusterdist_all.py](./transformers/numeric/clusterdist_all.py) [Cluster Distance for all columns]
    * [count_negative_values_transformer.py](./transformers/numeric/count_negative_values_transformer.py) [Count of negative values per row]
    * [count_positive_values_transformer.py](./transformers/numeric/count_positive_values_transformer.py) [Count of positive values per row]
    * [exp_diff_transformer.py](./transformers/numeric/exp_diff_transformer.py) [Exponentiated difference of two numbers]
    * [factor_analysis_transformer.py](./transformers/numeric/factor_analysis_transformer.py) [Factor Analysis Transformer]
    * [log_transformer.py](./transformers/numeric/log_transformer.py) [Converts numbers to their Logarithm] <kbd>✓ MOJO Enabled</kbd>
    * [ohe.py](./transformers/numeric/ohe.py) [One-Hot Encoding for categorical columns]
    * [pca_transformer.py](./transformers/numeric/pca_transformer.py) [Principal Component Analysis (PCA) Transformer]
    * [product.py](./transformers/numeric/product.py) [Products together 3 or more numeric features]
    * [random_transformer.py](./transformers/numeric/random_transformer.py) [Creates random numbers]
    * [round_transformer.py](./transformers/numeric/round_transformer.py) [Rounds numbers to 1, 2 or 3 decimals]
    * [square_root_transformer.py](./transformers/numeric/square_root_transformer.py) [Converts numbers to the square root, preserving the sign of the original numbers]
    * [sum.py](./transformers/numeric/sum.py) [Adds together 3 or more numeric features]
    * [truncated_svd_all.py](./transformers/numeric/truncated_svd_all.py) [Truncated SVD for all columns]
    * [yeojohnson_transformer.py](./transformers/numeric/yeojohnson_transformer.py) [Yeo-Johnson Power Transformer]
  * [OUTLIERS](./transformers/outliers)
    * [h2o3-dl-anomaly.py](./transformers/outliers/h2o3-dl-anomaly.py) [Anomaly score for each row based on reconstruction error of an H2O-3 deep learning autoencoder]
    * [quantile_winsorizer.py](./transformers/outliers/quantile_winsorizer.py) [Winsorizes (truncates) univariate outliers outside of a given quantile threshold]
    * [twosigma_winsorizer.py](./transformers/outliers/twosigma_winsorizer.py) [Winsorizes (truncates) univariate outliers outside of two standard deviations from the mean.]
  * [RECOMMENDATIONS](./transformers/recommendations)
    * [matrixfactorization.py](./transformers/recommendations/matrixfactorization.py) [Collaborative filtering features using various techniques of Matrix Factorization for recommendations.Recommended for large data]
  * [SIGNAL_PROCESSING](./transformers/signal_processing)
    * [signal_processing.py](./transformers/signal_processing/signal_processing.py) [This custom transformer processes signal files to create features used by DriverlessAI to solve a regression problem]
  * [SPEECH](./transformers/speech)
    * [audio_MFCC_transformer.py](./transformers/speech/audio_MFCC_transformer.py) [Extract MFCC and spectrogram features from audio files]
    * [azure_speech_to_text.py](./transformers/speech/azure_speech_to_text.py) [An example of integration with Azure Speech Recognition Service]
  * [STRING](./transformers/string)
    * [simple_grok_parser.py](./transformers/string/simple_grok_parser.py) [Extract column data using grok patterns]
    * [strlen_transformer.py](./transformers/string/strlen_transformer.py) [Returns the string length of categorical values]
    * [to_string_transformer.py](./transformers/string/to_string_transformer.py) [Converts numbers to strings]
    * [user_agent_transformer.py](./transformers/string/user_agent_transformer.py) [A best effort transformer to determine browser device characteristics from a user-agent string]
  * [SURVIVAL](./transformers/survival)
    * [dummy-pretransformer.py](./transformers/survival/dummy-pretransformer.py) [Dummy Pre-Transformer to use as a template for custom pre-transformer recipes.   This transformer consumes all features at once, adds 'pre:' to the names and passes   them down to transformer level and GA as-is.]
    * [h2o-3-coxph-pretransformer.py](./transformers/survival/h2o-3-coxph-pretransformer.py) [Pre-transformer utilizing survival analysis modeling using CoxPH (Cox proportional hazard)   using H2O-3 CoxPH function.   It adds risk score produced by CoxPH model and drops stop_column feature used for   survival modeling along with actual target as event.]
  * [TARGETENCODING](./transformers/targetencoding)
    * [ExpandingMean.py](./transformers/targetencoding/ExpandingMean.py) [CatBoost-style target encoding. See https://youtu.be/d6UMEmeXB6o?t=818 for short explanation]
    * [leaky_mean_target_encoder.py](./transformers/targetencoding/leaky_mean_target_encoder.py) [Example implementation of a out-of-fold target encoder (leaky, not recommended)]
  * [TIMESERIES](./transformers/timeseries)
    * [auto_arima_forecast.py](./transformers/timeseries/auto_arima_forecast.py) [Auto ARIMA transformer is a time series transformer that predicts target using ARIMA models.]
    * [general_time_series_transformer.py](./transformers/timeseries/general_time_series_transformer.py) [Demonstrates the API for custom time-series transformers.]
    * [parallel_auto_arima_forecast.py](./transformers/timeseries/parallel_auto_arima_forecast.py) [Parallel Auto ARIMA transformer is a time series transformer that predicts target using ARIMA models.In this implementation, Time Group Models are fitted in parallel]
    * [parallel_prophet_forecast.py](./transformers/timeseries/parallel_prophet_forecast.py) [Parallel FB Prophet transformer is a time series transformer that predicts target using FBProphet models.]
    * [parallel_prophet_forecast_using_individual_groups.py](./transformers/timeseries/parallel_prophet_forecast_using_individual_groups.py) [Parallel FB Prophet transformer is a time series transformer that predicts target using FBProphet models.This transformer fits one model for each time group column values and is significantly fasterthan the implementation available in parallel_prophet_forecast.py.]
    * [serial_prophet_forecast.py](./transformers/timeseries/serial_prophet_forecast.py) [Transformer that uses FB Prophet for time series prediction.Please see the parallel implementation for more information]
    * [time_encoder_transformer.py](./transformers/timeseries/time_encoder_transformer.py) [converts the Time Column to an ordered integer]
    * [trading_volatility.py](./transformers/timeseries/trading_volatility.py) [Calculates Historical Volatility for numeric features (makes assumptions on the data)]