# Configure Kubernetes cluster for Azure Machine Learning

As part of Azure Machine Learning (AzureML) service capabilities, AzureML Kubernetes extends AzureML service capabilities seamlessly from Azure cloud setting to any infrastructure across on-premises, multicloud, and the edge. With a simple AzureML extension deployment, you can instantly onboard your teams of ML professionals with productivity tools for full ML lifecycle, and have access to both Azure managed compute and customer managed Kubernetes anywhere. Your teams of ML professionals are flexible to build, train, and deploy models wherever and whenever business requires so and get consistent ML experience across different infrastructures.

You can easily bring AzureML capabilities to your Kubernetes cluster from cloud or on-premises by deploying AzureML extension.

- For Azure Kubernetes Service (AKS) in Azure, deploy AzureML extension to the AKS directly. For more information, see [Deploy and manage cluster extensions for Azure Kubernetes Service (AKS)](https://docs.microsoft.com/en-us/azure/aks/cluster-extensions).
- For Kubernetes clusters on-premises or from other cloud providers, you can connect the cluster with Azure Arc, then deploy AzureML extension. For more information, see [Azure Arc-enabled Kubernetes](https://docs.microsoft.com/en-us/azure/azure-arc/kubernetes/overview).

This repository is intended to serve as an information hub for customers and partners who are interested in Azure Machine Learning anywhere with Kubernetes. Use this repository for onboarding and testing instructions as well as an avenue to provide feedback, issues, enhancement requests and stay up to date as the product evolves.

## Why use Azure Machine Learning Kubernetes?

AzureML Kubernetes is customer fully configured and managed compute for machine learning. It can be used as both [training compute target](https://docs.microsoft.com/en-us/azure/machine-learning/concept-compute-target#train) and [inference compute target](https://docs.microsoft.com/en-us/azure/machine-learning/concept-compute-target#deploy). It provides the following benefits:

- Harness existing heterogeneous or homogeneous Kubernetes cluster, with CPUs or GPUs.
- Share the same Kubernetes cluster in multiple AzureML Workspaces across region.
- Use the same Kubernetes cluster for different machine learning purposes, including model training, batch scoring, and real-time inference.
- Secure network communication between the cluster and cloud via Azure Private Link and Private Endpoint.
- Isolate team projects and machine learning workloads with Kubernetes node selector and namespace.
- Target certain types of compute nodes and CPU/Memory/GPU resource allocation for training and inference workloads.
- Connect with custom data sources for machine learning workloads using Kubernetes PV and PVC.

## Prerequisites

1. A Kubernetes cluster up and running - **We recommend minimum of 4 vCPU cores and 8GB memory, around 2 vCPU cores and 3GB memory will be used by Azure Arc agent and AzureML extension components**.
1. Other than Azure Kubernetes Services (AKS) cluster in Azure, you must connect your Kubernetes cluster to Azure Arc. Please follow instructions in [connect existing Kubernetes cluster to Azure Arc](https://docs.microsoft.com/azure/azure-arc/kubernetes/quickstart-connect-cluster).
   * if you have Azure RedHat OpenShift Service (ARO) cluster or OpenShift Container Patform (OCP) cluster, follow additional prerequisite step [here](./docs/deploy-on-ocp.md) before AzureML extension deployment.
   * if you have AKS cluster in Azure, Azure Arc connection is not required.
1. Cluster running behind an outbound proxy server or firewall needs additional network configurations. Fulfill the [network requirements](./docs/network-requirements.md)
1. [Install or upgrade Azure CLI](https://docs.microsoft.com/cli/azure/install-azure-cli) to version >=2.16.0
1. Install Azure CLI extension ```k8s-extension``` (version>=1.2.2) by running ```az extension add --name k8s-extension```

## Getting started

Getting started with AzureML anywhere is easy with following steps:

1. [Deploy AzureML extension](./docs/deploy-extension.md)
1. [Attach Kubernetes cluster to AzureML workspace and create a compute target](./docs/attach-compute.md)
1. [Train image classification model with AzureML CLI v2](./docs/simple-train-cli.md)
1. [Deploy a trained image classification model](./docs/simple-flow.md)
1. [Troubleshooting guide](./docs/troubleshooting.md)

## Supported AzureML built-in features and unique features for Kubernetes

AzureML anywhere essentially brings a new compute target to Azure Machine Learning service. With this new Kubernetes compute target, you can use existing AzureML tools and service capabilities to build, train, and deploy model on Kubernetes cluster anywhere. This seamless integration supports many AzureML built-in features , and existing [AzureML examples](https://github.com/Azure/azureml-examples) can run on Kubernetes cluster with a smple change of compute target name.

In addition to many built-in AzureML features support, AzureML anywhere also supports following unique features leveraging native Kubernetes capability:

* [Run training or inference workload with custom instance types for resource efficiency](./docs/instance-type.md)
* [Train model with PV/PVC storage mount](./docs/pvc.md)
* [Assign managed identity to the compute](./docs/managed-identity.md)
* Use managed identity for your endpoint
* [Custom container registry support](https://github.com/Azure/azure-arc-kubernetes-preview/blob/master/docs/custom-registry/connect-cluster.md)
* Multiple AzureML workspaces share the same Kubernetes cluster
* [Deploy model using custom container with built-in model or entry script](./docs/inference-byoc.md). In this case, the model and the entry scripts will stay on-premises localy and not uploaded to Azure cloud.

## Supported Kubernetes distributions and locations

* Azure Kubernetes Services (AKS)
* AKS Engine
* AKS on Azure Stack HCI
* Azure RedHat OpenShift Service (ARO)
* OpenShift Container Platform (OCP)
* Google GKE
* Canonical Kubernetes Distribution
* Amazon EKS
* Kind
* K3s-Lightweight Kubernetes
* Azure location availability: East US, East US 2, South Central US, West US 2, Australia East, Southeast Asia, North Europe, UK South, West Europe, West Central US, Central US, North Central US, West US, Korea Central, France Central

# Supported Kubernetes version

Kubernetes clusters installing AzureML extension have a version support window of "N-2", that is aligned with [Azure Kubernetes Service (AKS) version support policy](https://docs.microsoft.com/en-us/azure/aks/supported-kubernetes-versions?tabs=azure-cli#kubernetes-version-support-policy), where 'N' is the latest GA minor version of Azure Kubernetes Service.

For example, if AKS introduces 1.20.a today, versions 1.20.a, 1.20.b, 1.19.c, 1.19.d, 1.18.e, and 1.18.f are supported.

If customers are running an unsupported Kubernetes version, they will be asked to upgrade when requesting support for the cluster. Clusters running unsupported Kubernetes releases are not covered by the AzureML extension support policies.

## Additional resources

### [Use AKS on Azure Stack HCI for on-premises machine learning](https://aka.ms/aks-hci-ml)

### [Frequently Asked Questions](./docs/faq.md)

<!-- ### [Release notes](./docs/release-notes.md)  -->

### [Limitations and known issues](./docs/limitations-and-known-issues.md)

<!-- ## Examples

path|status|
-|-
[deploy-safe-rollout-kubernetes-online-endpoints.sh](https://github.com/Azure/azureml-examples/blob/main/cli/deploy-safe-rollout-kubernetes-online-endpoints.sh)|[![deploy-safe-rollout-kubernetes-online-endpoints](https://github.com/Azure/azureml-examples/workflows/cli-scripts-deploy-safe-rollout-kubernetes-online-endpoints/badge.svg?branch=main)](https://github.com/Azure/azureml-examples/actions/workflows/cli-scripts-deploy-safe-rollout-kubernetes-online-endpoints.yml)
|[kubernetes-online-endpoints-safe-rollout](https://github.com/Azure/azureml-examples/blob/sdk-preview/sdk/endpoints/online/kubernetes/kubernetes-online-endpoints-safe-rollout.ipynb)|[![kubernetes-online-endpoints-safe-rollout](https://github.com/Azure/azureml-examples/actions/workflows/sdk-endpoints-online-kubernetes-kubernetes-online-endpoints-safe-rollout.yml/badge.svg?branch=main)](https://github.com/Azure/azureml-examples/actions/workflows/sdk-endpoints-online-kubernetes-kubernetes-online-endpoints-safe-rollout.yml)|
|[kubernetes-online-endpoints-simple-deployment](https://github.com/Azure/azureml-examples/blob/sdk-preview/sdk/endpoints/online/kubernetes/kubernetes-online-endpoints-simple-deployment.ipynb)|[![kubernetes-online-endpoints-simple-deployment](https://github.com/Azure/azureml-examples/actions/workflows/sdk-endpoints-online-kubernetes-kubernetes-online-endpoints-simple-deployment.yml/badge.svg?branch=main)](https://github.com/Azure/azureml-examples/actions/workflows/sdk-endpoints-online-kubernetes-kubernetes-online-endpoints-simple-deployment.yml)| -->

## Get more examples

All AzureML examples can be found in [https://github.com/Azure/azureml-examples.git](https://github.com/Azure/azureml-examples).

For any AzureML example, you only need to update the compute target name to your own Kubernetes compute target, then you are all done.
* Explore training job samples with CLI v2 - [https://github.com/Azure/azureml-examples/tree/main/cli/jobs](https://github.com/Azure/azureml-examples/tree/main/cli/jobs)
* Explore model deployment with online endpoint samples with CLI v2 - [https://github.com/Azure/azureml-examples/tree/main/cli/endpoints/online/kubernetes](https://github.com/Azure/azureml-examples/tree/main/cli/endpoints/online/kubernetes)
* Explore batch endpoint samples with CLI v2 - [https://github.com/Azure/azureml-examples/tree/main/cli/endpoints/batch](https://github.com/Azure/azureml-examples/tree/main/cli/endpoints/batch)
* Explore training job samples with SDK v2 -[https://github.com/Azure/azureml-examples/tree/main/sdk/jobs](https://github.com/Azure/azureml-examples/tree/main/sdk/jobs)
* Explore model deployment with online endpoint samples with SDK v2 -[https://github.com/Azure/azureml-examples/tree/main/sdk/endpoints/online/kubernetes](https://github.com/Azure/azureml-examples/tree/main/sdk/endpoints/online/kubernetes)

## Support

We are always looking for feedback on our current experiences and what we should work on next. If there is anything you would like us to prioritize, please feel free to suggest so via our [GitHub Issue Tracker](https://github.com/Azure/AML-Kubernetes/issues). You can submit a bug report, a feature suggestion or participate in discussions.

Or reach out to us: amlarc-pm@microsoft.com if you have any questions or feedback.

## Activities

* [Check for known issues](https://github.com/Azure/amlk8s-preview/labels/known-issue)
* [View general feedback](https://github.com/Azure/amlk8s-preview/labels/feedback)
* [Browse roadmap items](https://github.com/Azure/amlk8s-preview/labels/roadmap)
* [Open a bug, provide feedback, or suggest an improvement](https://github.com/Azure/amlk8s-preview/issues/new/choose)

## Contributing

This project welcomes contributions and suggestions.  Most contributions require you to agree to a
Contributor License Agreement (CLA) declaring that you have the right to, and actually do, grant us
the rights to use your contribution. For details, visit https://cla.opensource.microsoft.com.

When you submit a pull request, a CLA bot will automatically determine whether you need to provide
a CLA and decorate the PR appropriately (e.g., status check, comment). Simply follow the instructions
provided by the bot. You will only need to do this once across all repos using our CLA.

This project has adopted the [Microsoft Open Source Code of Conduct](https://opensource.microsoft.com/codeofconduct/).
For more information see the [Code of Conduct FAQ](https://opensource.microsoft.com/codeofconduct/faq/) or
contact [opencode@microsoft.com](mailto:opencode@microsoft.com) with any additional questions or comments.

![Impressions](https://PixelServer20190423114238.azurewebsites.net/api/impressions/CMK8s-Samples/README.png)

## Disclaimer

The lifecycle management (health, kubernetes version upgrades, security updates to nodes, scaling, etc.) of the AKS or Arc enabled Kubernetes cluster is the responsibility of the customer.

For AKS, read what is managed and what is shared responsibility [here](https://docs.microsoft.com/azure/aks/support-policies)

All preview features are available on a self-service, opt-in basis and are subject to breaking design and API changes. Previews are provided "as is" and "as available," and they're excluded from the service-level agreements and limited warranty.


**[dbt](https://www.getdbt.com/)** enables data analysts and engineers to transform their data using the same practices that software engineers use to build applications.

The **[Databricks Lakehouse](https://www.databricks.com/)** provides one simple platform to unify all your data, analytics and AI workloads.

# dbt-databricks

The `dbt-databricks` adapter contains all of the code enabling dbt to work with Databricks. This adapter is based off the amazing work done in [dbt-spark](https://github.com/dbt-labs/dbt-spark). Some key features include:

- **Easy setup**. No need to install an ODBC driver as the adapter uses pure Python APIs.
- **Open by default**. For example, it uses the the open and performant [Delta](https://delta.io/) table format by default. This has many benefits, including letting you use `MERGE` as the the default incremental materialization strategy.
- **Support for Unity Catalog**. dbt-databricks>=1.1.1 supports the 3-level namespace of Unity Catalog (catalog / schema / relations) so you can organize and secure your data the way you like.
- **Performance**. The adapter generates SQL expressions that are automatically accelerated by the native, vectorized [Photon](https://databricks.com/product/photon) execution engine.

## Choosing between dbt-databricks and dbt-spark
If you are developing a dbt project on Databricks, we recommend using `dbt-databricks` for the reasons noted above.

`dbt-spark` is an actively developed adapter which works with Databricks as well as Apache Spark anywhere it is hosted e.g. on AWS EMR.

## Getting started

### Installation

Install using pip:
```nofmt
pip install dbt-databricks
```

Upgrade to the latest version
```nofmt
pip install --upgrade dbt-databricks
```

### Profile Setup

```nofmt
your_profile_name:
  target: dev
  outputs:
    dev:
      type: databricks
      catalog: [optional catalog name, if you are using Unity Catalog, only available in dbt-databricks>=1.1.1]
      schema: [database/schema name]
      host: [your.databrickshost.com]
      http_path: [/sql/your/http/path]
      token: [dapiXXXXXXXXXXXXXXXXXXXXXXX]
```

### Quick Starts

These following quick starts will get you up and running with the `dbt-databricks` adapter:
- [Developing your first dbt project](https://github.com/databricks/dbt-databricks/blob/main/docs/local-dev.md)
- Using dbt Cloud with Databricks ([Azure](https://docs.microsoft.com/en-us/azure/databricks/integrations/prep/dbt-cloud) | [AWS](https://docs.databricks.com/integrations/prep/dbt-cloud.html))
- [Running dbt production jobs on Databricks Workflows](https://github.com/databricks/dbt-databricks/blob/main/docs/databricks-workflows.md)
- [Using Unity Catalog with dbt-databricks](https://github.com/databricks/dbt-databricks/blob/main/docs/uc.md)
- [Using GitHub Actions for dbt CI/CD on Databricks](https://github.com/databricks/dbt-databricks/blob/main/docs/github-actions.md)
- [Loading data from S3 into Delta using the databricks_copy_into macro](https://github.com/databricks/dbt-databricks/blob/main/docs/databricks-copy-into-macro-aws.md)
- [Contribute to this repository](CONTRIBUTING.MD)

### Compatibility

The `dbt-databricks` adapter has been tested:

- with Python 3.7 or above.
- against `Databricks SQL` and `Databricks runtime releases 9.1 LTS` and later.

### Tips and Tricks
## Choosing compute for a Python model
You can override the compute used for a specific Python model by setting the `http_path` property in model configuration. This can be useful if, for example, you want to run a Python model on an All Purpose cluster, while running SQL models on a SQL Warehouse. Note that this capability is only available for Python models.

```
def model(dbt, session):
    dbt.config(
      http_path="sql/protocolv1/..."
    )
```


YDB Python SDK
---
[![License](https://img.shields.io/badge/License-Apache%202.0-blue.svg)](https://github.com/ydb-platform/ydb/blob/main/LICENSE)
[![PyPI version](https://badge.fury.io/py/ydb.svg)](https://badge.fury.io/py/ydb)
[![Functional tests](https://github.com/ydb-platform/ydb-python-sdk/actions/workflows/tests.yaml/badge.svg)](https://github.com/ydb-platform/ydb-python-sdk/actions/workflows/tests.yaml)
[![Style checks](https://github.com/ydb-platform/ydb-python-sdk/actions/workflows/style.yaml/badge.svg)](https://github.com/ydb-platform/ydb-python-sdk/actions/workflows/style.yaml)

Officially supported Python client for YDB.

## Quickstart

### Prerequisites

- Python 3.8 or higher
- `pip` version 9.0.1 or higher

If necessary, upgrade your version of `pip`:

```sh
$ python -m pip install --upgrade pip
```

If you cannot upgrade `pip` due to a system-owned installation, you can
run the example in a virtualenv:

```sh
$ python -m pip install virtualenv
$ virtualenv venv
$ source venv/bin/activate
$ python -m pip install --upgrade pip
```

Install YDB python sdk:

```sh
$ python -m pip install ydb
```


# The AIPlan4EU Unified Planning Library

[![Actions Status](https://github.com/aiplan4eu/unified-planning/actions/workflows/release.yml/badge.svg)](https://github.com/aiplan4eu/unified-planning/actions)
[![Coverage Status](https://codecov.io/gh/aiplan4eu/unified-planning/branch/master/graph/badge.svg?token=GBM7HYNDRB)](https://codecov.io/gh/aiplan4eu/unified-planning)
[![Documentation Status](https://readthedocs.org/projects/unified-planning/badge/?version=latest)](https://unified-planning.readthedocs.io/en/latest/)
[![PyPI version](https://badge.fury.io/py/unified-planning.svg)](https://pypi.python.org/pypi/unified-planning)

The unified_planning library makes it easy to formulate planning problems and to invoke automated planners.

* Define problems in a *simple*, *intuitive*, and *planner independent* way
* Solve your planning problems using one of the native solvers, or by using any PDDL planner
* Dump your problems in PDDL (or  ANML) format
* Parse PDDL problem formulations
* Simplification, grounding, removal of conditional effects and many other transformations are available
* and more...

## Usage
```python
from unified_planning.shortcuts import *

x = Fluent("x")

a = InstantaneousAction("a")
a.add_precondition(Not(x))
a.add_effect(x, True)

problem = Problem("basic")
problem.add_fluent(x)
problem.add_action(a)
problem.set_initial_value(x, False)
problem.add_goal(x)

with OneshotPlanner(problem_kind=problem.kind) as planner:
    result = planner.solve(problem)
    if result.status in unified_planning.engines.results.POSITIVE_OUTCOMES:
        print(f"{planner.name} found this plan: {result.plan}")
    else:
        print("No plan found.")
```

## Documentation

The documentation is available [here](https://unified-planning.readthedocs.io/en/latest/)

## Examples

More complex examples are available as notebooks [here](https://unified-planning.readthedocs.io/en/latest/examples.html).

## Staying Updated

To stay informed about project updates, announcements, and general discussions, we encourage you to join our public mailing list: (https://groups.google.com/g/unified-planning).

You can subscribe to this list to receive notifications and engage in community discussions.

## Acknowledgments

<img src="https://www.aiplan4eu-project.eu/wp-content/uploads/2021/07/euflag.png" width="60" height="40">

This library is being developed for the AIPlan4EU H2020 project (https://aiplan4eu-project.eu) that is funded by the European Commission under grant agreement number 101016442.


.. image:: https://jazzband.co/static/img/badge.svg
   :target: https://jazzband.co/
   :alt: Jazzband

.. image:: https://img.shields.io/pypi/v/wagtailmenus.svg
   :target: https://pypi.python.org/pypi/wagtailmenus
   :alt: PyPI Download

.. image:: https://img.shields.io/pypi/v/wagtailmenus.svg
    :alt: PyPi Version
    :target: https://pypi.python.org/pypi/wagtailmenus

.. image:: https://github.com/jazzband/wagtailmenus/actions/workflows/test.yml/badge.svg
    :alt: Build Status
    :target: https://github.com/jazzband/wagtailmenus/actions/workflows/test.yml

.. image:: https://codecov.io/gh/jazzband/wagtailmenus/branch/master/graph/badge.svg
    :alt: Code coverage
    :target: https://codecov.io/gh/jazzband/wagtailmenus

.. image:: https://readthedocs.org/projects/wagtailmenus/badge/?version=latest
    :alt: Documentation Status
    :target: http://wagtailmenus.readthedocs.io/en/latest/

============
wagtailmenus
============

wagtailmenus is an extension for Torchbox's `Wagtail CMS <https://github.com/torchbox/wagtail>`_ to help you manage and render multi-level navigation and simple flat menus in a consistent, flexible way.

The current version is tested for compatibility with the following:

- Wagtail versions >= 5.2
- Django versions 3.2, 4.2 and >= 5.0
- Python versions 3.8 to 3.12

.. image:: https://raw.githubusercontent.com/jazzband/wagtailmenus/master/docs/source/_static/images/repeating-item.png

New to wagtailmenus?
====================

Start by reading our `Overview and key concepts <http://wagtailmenus.readthedocs.io/en/stable/overview.html>`_ page.
It will give you a better understand of how the app works, and help you decide whether it is a good fit for your project.

How do I use it?
================

For everything you need to get up and running with wagtailmenus, `view the official documentation <http://wagtailmenus.readthedocs.io/>`_.

Contributing
============

Instructions on how to contribute can be found `here <https://wagtailmenus.readthedocs.io/en/stable/contributing/index.html>`_.

As we are members of a `JazzBand project <https://jazzband.co/projects>`_, `wagtailmenus` contributors should adhere to the `Contributor Code of Conduct <https://jazzband.co/about/conduct>`_.


Awpy

[![Awpy Discord](https://img.shields.io/discord/868146581419999232?color=blue&label=Discord&logo=discord)](https://discord.gg/W34XjsSs2H) [![Awpy Downloads](https://static.pepy.tech/personalized-badge/awpy?period=total&units=international_system&left_color=grey&right_color=blue&left_text=Downloads)](https://pepy.tech/project/awpy) [![Build](https://github.com/pnxenopoulos/awpy/actions/workflows/build.yml/badge.svg)](https://github.com/pnxenopoulos/awpy/actions/workflows/build.yml) [![Documentation Status](https://readthedocs.org/projects/awpy/badge/?version=latest)](https://awpy.readthedocs.io/en/latest/?badge=latest) [![MIT License](https://img.shields.io/badge/license-MIT-lightgrey)](https://github.com/pnxenopoulos/awpy/blob/main/LICENSE)

**Counter-Strike 2 Demo Parsing, Analytics and Visualization in Python**

- :computer: Parse Counter-Strike demos in Python in one line.
- :skull: Access kill, damage, grenade, bomb, player trajectory data, and more
- :chart: Calculate common statistics, such as ADR, KAST, Rating and win probability
- 🎞 Visualize Counter-Strike data, including animated round gifs
- :speaker: Active [Discord](https://discord.gg/W34XjsSs2H) community

## Installation

To install Awpy, you can run

```
pip install awpy
```

> [!NOTE]
> `awpy` requires [Python](https://www.python.org/downloads/) >= 3.9. To update the library, just run `pip install --upgrade awpy`. To check your current version, run `pip freeze | grep awpy`.

> [!TIP]
> Don't worry if you get stuck, visit us [our Discord](https://discord.gg/W34XjsSs2H) for help.

## Example Code

Using Awpy is easy. Just find a demo you want to analyze and use the example below to get started. For example, take [NaVi vs Virtus.pro](https://www.hltv.org/stats/matches/mapstatsid/169189/natus-vincere-vs-virtuspro).

```python
from awpy import Demo

# Simply call `Demo(path="...")` to parse a demo
dem = Demo("natus-vincere-vs-virtus-pro-m1-overpass.dem")

# Access various dictionaries & dataframes
dem.header
dem.rounds
dem.grenades
dem.kills
dem.damages
dem.bomb
dem.smokes
dem.infernos
dem.weapon_fires
dem.ticks
```

> [!TIP]
> Want to learn more about the parser output? Visit the [parser primer](https://awpy.readthedocs.io/en/latest/parser_output.html) in our documentation!

### Help! The parser doesn't work or returns weird data

Counter-Strike demos can be a bit troublesome. It is likely you'll see increased error rates in POV demos. To help us address parsing issues, please open a bug report in our [Github issues](https://github.com/pnxenopoulos/awpy/issues) as well as in our [our Discord](https://discord.gg/3JrhKYcEKW). We're committed to increasing parsing coverage rates and appreciate any errors you may find.

## Examples and Projects

Take a look at the following Jupyter notebooks provided in our `examples/` directory. These will help you get started parsing and analyzing Counter-Strike data.

- [Parsing a CS2 demofile](https://github.com/pnxenopoulos/awpy/blob/main/examples/00_Parsing_a_CS2_Demofile.ipynb)

If you use the parser for any public analysis, we kindly ask you to link to the Awpy repository, so that others may know how you parsed, analyzed or visualized your data. If you have a paper or project that uses the parser, please let us know in Discord so we can add it to our growing list!

## Contributing

We welcome any contributions from the community, no matter the skill-level. You can visit our [issue page](https://github.com/pnxenopoulos/awpy/issues) to see what issues are still open, the [Awpy project](https://github.com/users/pnxenopoulos/projects/5) for a different view of project priorities, or you can message us on Discord. Some examples of where you can make a difference are in documentation, quality assurance, developing new features, or creating unique content with Awpy. You can see more examples of community content [here](https://awpy.readthedocs.io/en/latest/projects.html). If you are interested in contributing to Awpy, learn more [here](https://github.com/pnxenopoulos/awpy/blob/main/CONTRIBUTING.md).

> [!TIP]
> We are happy to walk through those that want to contribute, no matter your skill level. There are a diverse set of ways one can contribute to Awpy. We welcome first-time contributors.

## Structure

Awpy is structured as follows:

```
.
├── awpy
│   ├── data                      # Data directory (PLANNED)
│   ├── stats                     # Stats and analytics module
│   └── visualization             # Visualization module (PLANNED)
│   converters.py                 # Utilities for converting to readable strings
│   demo.py                       # Defines the base Demo class
│   parsers.py                    # Defines simple parsers for different events
│   utils.py                      # Utilities used across the project
├── doc                           # Documentation files
├── examples                      # Jupyter Notebooks showing example code
└── tests                         # Tests
```

## Acknowledgments

The name "Awpy" is due to [Nick Wan](https://www.twitch.tv/nickwan_datasci) -- we recommend his stream for sports data analytics enthusiasts.

Awpy was first built on the amazing work done in the [demoinfocs-golang](https://github.com/markus-wa/demoinfocs-golang) Golang library. We now rely on [demoparser2](https://github.com/LaihoE/demoparser) for parsing, which is another fantastic parsing project, built specifically for Python.


# pgeu-system <br/> Non-Profit Organization and Conference Management System

## Introduction

pgeu-system is a system for managing non-profit organizations along with an
extensive conference management system.

## Features

### Conference Administration

The conference management system includes time reports, cross conference emails,
conference series and individual conference management including registration,
waitlisting, attendee emails, a wiki, individual signups, invoicing, sessions,
scheduling, sponsorship management, news, prepaid vouchers and discount codes,
volunteer scheduling, lots of reports, and more.

### Invoice Management

The invoicing system supports creating individual invoices, and is linked into
the conference and membership parts of the system, and into the accounting
system.

### News

News posts to the main website as well as integration with social media systems
is included.

### Accounting

A structured accounting system with account classes, groups, individual accounts
and objects is built into the system and integrated with the conference management,
membership, and invoice systems.

### Membership

Simple membership tracking and membership dues are supported.

### Elections

Multi-seat elections are supported (eg: for voting in new board members).

## Contributions

Contributions to pgeu-system are certainly welcome!

Please feel free to create PRs or issues for any bugs found in the pgeu-system-
note that the specific websites which are running pgeu-system are able to 'skin'
the system, so in many cases if you find a 404 or similar on a website running
pgeu-system, that's an issue you'll need to address with that particular organization
and is not a pgeu-system bug or issue.

If you are interested in contributing to the development of pgeu-system by
working on new features, please reach out to us first and discuss your feature
idea.  Once a feature has been discussed and the general concept agreed to, a
feature issue can be opened to work on the details of the implementation.

The mailing list for discussing pgeu-system is <pgeu-system@lists.postgresql.eu>.


![Java Build Status](https://dataproctemplatesci.com/buildStatus/icon?job=dataproc-templates-build%2Fbuild-job-java&&subject=build-java)
![Java Dataproc Serverless Integration Tests Status](https://dataproctemplatesci.com/buildStatus/icon?job=dataproc-templates-build%2Fintegration-tests-java&&subject=java-serverless-integration-tests)
![Java Dataproc Cluster Integration Tests Status](https://dataproctemplatesci.com/buildStatus/icon?job=dataproc-templates-build%2Fcluster-integration-tests-java&&subject=java-cluster-integration-tests)

![Python Build Status](https://dataproctemplatesci.com/buildStatus/icon?job=dataproc-templates-build%2Fbuild-job-python&&subject=build-python)
![Python Dataproc Serverless Integration Test Status](https://dataproctemplatesci.com/buildStatus/icon?job=dataproc-templates-build%2Fintegration-tests-python&&subject=python-serverless-integration-tests)
![Python Dataproc Cluster Integration Tests Status](https://dataproctemplatesci.com/buildStatus/icon?job=dataproc-templates-build%2Fcluster-integration-tests-python&&subject=python-cluster-integration-tests)

# Dataproc Templates
Dataproc templates are designed to address various in-cloud data tasks, including data import/export/backup/restore and bulk API operations. These templates leverage the power of [Google Cloud's Dataproc](https://cloud.google.com/dataproc/), supporting both Dataproc Serverless and Dataproc clusters.

Google provides this collection of pre-implemented Dataproc templates as a reference and for easy customization. (Video [Link](https://www.youtube.com/watch?v=ZQIaY0pNlmI))

[![Open in Cloud Shell](http://gstatic.com/cloudssh/images/open-btn.svg)](https://console.cloud.google.com/cloudshell/editor)

## Dataproc Templates (Java - Spark)
Please refer to the [Dataproc Templates (Java - Spark) README](/java) for more information

* [BigQueryToGCS](java/src/main/java/com/google/cloud/dataproc/templates/bigquery#BigQuery-To-GCS)
* [CassandraToBigQuery](java/src/main/java/com/google/cloud/dataproc/templates/databases#executing-cassandra-to-bigquery-template)
* [CassandraToGCS](java/src/main/java/com/google/cloud/dataproc/templates/databases#executing-cassandra-to-gcs-template) (blogpost [link](https://medium.com/google-cloud/migrate-data-from-cassandra-to-gcs-using-java-and-dataproc-serverless-5358ef498f6b))
* [DataplexGCStoBQ](/java/src/main//java/com/google/cloud/dataproc/templates/dataplex#dataplex-gcs-to-bigquery)(blogpost [link](https://medium.com/google-cloud/using-dataproc-serverless-to-migrate-your-dataplex-gcs-data-to-bigquery-1e47bc8de74c))
* [GCSToBigQuery](/java/src/main/java/com/google/cloud/dataproc/templates/gcs#1-gcs-to-bigquery) (blogpost [link](https://medium.com/google-cloud/importing-data-from-gcs-to-bigquery-via-spark-bq-connector-using-dataproc-serverless-25e29f84888d))
* [GCSToBigTable](/java/src/main/java/com/google/cloud/dataproc/templates/gcs#1-gcs-to-bigquery) (blogpost [link](https://medium.com/google-cloud/load-data-from-gcs-to-bigtable-using-gcp-dataproc-serverless-5c43d773e615)) (Video [link](https://www.youtube.com/watch?v=JwO72i2nVPU))
* [GCSToGCS](/java/src/main/java/com/google/cloud/dataproc/templates/gcs#4-gcs-to-gcs) (blogpost [link](https://medium.com/@ankuljain/migrate-gcs-to-gcs-using-dataproc-serverless-3b7b0f6ad6b9))
* [GCSToJDBC](/java/src/main/java/com/google/cloud/dataproc/templates/gcs#3-gcs-to-jdbc) (blogpost [link](https://medium.com/google-cloud/importing-data-from-gcs-to-databases-via-jdbc-using-dataproc-serverless-7ed75eab93ba))
* [GCSToMongo](/java/src/main/java/com/google/cloud/dataproc/templates/gcs#6-gcs-to-mongo)
* [GCSToSpanner](/java/src/main/java/com/google/cloud/dataproc/templates/gcs#2-gcs-to-spanner) (blogpost [link](https://medium.com/google-cloud/fast-export-large-database-tables-using-gcp-serverless-dataproc-spark-bb32b1260268))
* [GeneralTemplate](/java/src/main/java/com/google/cloud/dataproc/templates/general)
* [HBaseToGCS](/java/src/main/java/com/google/cloud/dataproc/templates/hbase#1-hbase-to-gcs)(blogpost [link](https://medium.com/google-cloud/using-dataproc-serverless-to-migrate-your-hbase-data-to-gcs-bf1ccf4ab945))
* [HiveToBigQuery](/java/src/main/java/com/google/cloud/dataproc/templates/hive#1-hive-to-bigquery)(blogpost [link](https://medium.com/google-cloud/using-dataproc-serverless-to-migrate-your-hive-data-to-bigquery-8e2d4fcd1c24))
* [HiveToGCS](/java/src/main/java/com/google/cloud/dataproc/templates/hive#2-hive-to-gcs) (blogpost [link](https://medium.com/@nehamodgil_21070/processing-and-migrating-large-data-tables-from-hive-to-gcs-using-java-and-dataproc-serverless-b6dbbae61c5d))
* [JDBCToBigQuery](/java/src/main/java/com/google/cloud/dataproc/templates/jdbc#1-jdbc-to-bigquery) (blogpost [link](https://medium.com/@sjlva/java-fast-export-large-database-tables-using-gcp-serverless-dataproc-fe6ffffe28b5))
* [JDBCToGCS](/java/src/main/java/com/google/cloud/dataproc/templates/jdbc#2-jdbc-to-gcs) (blogpost [link](https://medium.com/google-cloud/fast-export-large-database-tables-using-gcp-serverless-dataproc-spark-bb32b1260268))
* [JDBCToJDBC](/java/src/main/java/com/google/cloud/dataproc/templates/jdbc#3-jdbc-to-jdbc)
* [JDBCToSpanner](/java/src/main/java/com/google/cloud/dataproc/templates/jdbc#4-jdbc-to-spanner)
* [KafkaToBQ](/java/src/main/java/com/google/cloud/dataproc/templates/kafka#1-kafka-to-bigquery) (blogpost [link](https://medium.com/google-cloud/export-data-from-apache-kafka-to-bigquery-using-dataproc-serverless-4a666535117c))
* [KafkaToBQDstream](/java/src/main/java/com/google/cloud/dataproc/templates/kafka#4-kafka-to-bq-via-spark-direct-stream)
* [KafkaToGCS](/java/src/main/java/com/google/cloud/dataproc/templates/kafka/README.md#2-kafka-to-gcs) (blogpost [link](https://medium.com/@pniralakeri/importing-data-from-kafka-to-gcs-using-dataproc-serverless-38e449d559f9))
* [KafkaToGCSDstream](/java/src/main/java/com/google/cloud/dataproc/templates/kafka#5-kafka-to-gcs-via-spark-direct-stream)
* [KafkaToPubSub](/java/src/main/java/com/google/cloud/dataproc/templates/kafka/README.md#3-kafka-to-pubsub)
* [MongoToGCS](/java/src/main/java/com/google/cloud/dataproc/templates/databases#executing-mongo-to-gcs-template) (blogpost [link](https://medium.com/google-cloud/migrating-data-from-mongo-to-gcs-using-java-and-dataproc-serverless-template-390500481804))
* [PubSubToBigQuery](/java/src/main/java/com/google/cloud/dataproc/templates/pubsub#1-pubsub-to-bigquery) (blogpost [link](https://medium.com/google-cloud/from-pub-sub-to-bigquery-streaming-data-in-near-real-time-b550aeff595d))
* [PubSubToBigTable](/java/src/main/java/com/google/cloud/dataproc/templates/pubsub#1-pubsub-to-bigtable) (blogpost [link](https://medium.com/google-cloud/stream-data-from-pub-sub-to-bigtable-using-dataproc-serverless-3142c1bcc22a))
* [PubSubLiteToBigTable](/java/src/main/java/com/google/cloud/dataproc/templates/pubsublite#1-pubsublite-to-bigtable) (blogpost [link](https://medium.com/google-cloud/stream-data-from-pub-sub-lite-to-bigtable-using-dataproc-serverless-2c8816f40581))
* [PubSubToGCS](/java/src/main/java/com/google/cloud/dataproc/templates/pubsub#2-pubsub-to-gcs) (blogpost [link](https://medium.com/google-cloud/stream-data-from-pub-sub-to-cloud-storage-using-dataproc-serverless-7a1e4823926e))
* [RedshiftToGCS](/java/src/main/java/com/google/cloud/dataproc/templates/databases#executing-redshift-to-gcs-template) (blogpost [Link](https://medium.com/google-cloud/exporting-data-from-redshift-to-gcs-using-gcp-dataproc-serverless-java-5b82044d78e7))
* [S3ToBigQuery](/java/src/main/java/com/google/cloud/dataproc/templates/s3#1-s3-to-bigquery) (blogpost [link](https://medium.com/google-cloud/export-data-from-aws-s3-to-bigquery-using-dataproc-serverless-6dc7a9952fc4))
* [SnowflakeToGCS](/java/src/main/java/com/google/cloud/dataproc/templates/snowflake#1-snowflake-to-gcs) (blogpost [link](https://medium.com/google-cloud/export-snowflake-query-results-to-gcs-using-dataproc-serverless-3d68f5a01ca9))
* [SpannerToGCS](/java/src/main/java/com/google/cloud/dataproc/templates/databases#executing-spanner-to-gcs-template) (blogpost [link](https://medium.com/google-cloud/cloud-spanner-export-query-results-using-dataproc-serverless-6f2f65b583a4))
* [TextToBigquery](/java/src/main/java/com/google/cloud/dataproc/templates/gcs#7-text-to-bigquery)
* [WordCount](/java/src/main/java/com/google/cloud/dataproc/templates/word/WordCount.java)

## Dataproc Templates (Python - PySpark)
Please refer to the [Dataproc Templates (Python - PySpark) README](/python) for more information
* [AzureBlobToBigQuery](/python/dataproc_templates/azure#azure-blob-to-bigquery)
* [BigQueryToGCS](/python/dataproc_templates/bigquery#bigquery-to-gcs) (blogpost [link](https://medium.com/google-cloud/moving-data-from-bigquery-to-gcs-using-gcp-dataproc-serverless-and-pyspark-f6481b86bcd1))
* [CassandraToBigquery](/python/dataproc_templates/cassandra)
* [CassandraToGCS](/python/dataproc_templates/cassandra) (blogpost [link](https://medium.com/google-cloud/export-data-from-cassandra-to-google-cloud-storage-using-dataproc-serverless-2569a00e17fe))
* [GCSToBigQuery](/python/dataproc_templates/gcs#gcs-to-bigquery) (blogpost [link](https://medium.com/@ppaglilla/getting-started-with-dataproc-serverless-pyspark-templates-e32278a6a06e))
* [GCSToBigTable](/python/dataproc_templates/gcs#gcs-to-bigtable)(blogpost [link](https://medium.com/google-cloud/pyspark-load-data-from-gcs-to-bigtable-using-gcp-dataproc-serverless-c373430fe157))
* [GCSToGCS](/python/dataproc_templates/gcs#gcs-to-gcs---sql-transformation) (blogpost [link](https://medium.com/@ankuljain/migrate-gcs-to-gcs-using-dataproc-serverless-3b7b0f6ad6b9))
* [GCSToJDBC](/python/dataproc_templates/gcs#gcs-to-jdbc) (blogpost [link](https://medium.com/google-cloud/import-data-from-gcs-to-jdbc-databases-using-dataproc-serverless-c7154b242430))
* [GCSToMongo](/python/dataproc_templates/gcs#gcs-to-mongodb) (blogpost [link](https://medium.com/google-cloud/importing-data-from-gcs-to-mongodb-using-dataproc-serverless-fed58904633a))
* [HbaseToGCS](/python/dataproc_templates/hbase#hbase-to-gcs) (blogpost [link](https://medium.com/@nehamodgil_21070/dataproc-serverless-for-exporting-large-data-files-from-hbase-to-gcs-d89057279512))
* [HiveToBigQuery](/python/dataproc_templates/hive#hive-to-bigquery) (blogpost [link](https://medium.com/google-cloud/processing-data-from-hive-to-bigquery-using-pyspark-and-dataproc-serverless-217c7cb9e4f8))
* [HiveToGCS](/python/dataproc_templates/hive#hive-to-gcs) (blogpost [link](https://medium.com/@surjitsh/processing-large-data-tables-from-hive-to-gcs-using-pyspark-and-dataproc-serverless-35d3d16daaf))
* [JDBCToBigQuery](/python/dataproc_templates/jdbc#3-jdbc-to-bigquery) (blogpost [link](https://medium.com/@sjlva/python-fast-export-large-database-tables-using-gcp-serverless-dataproc-bfe77a132485))
* [JDBCToGCS](/python/dataproc_templates/jdbc#2-jdbc-to-gcs) (blogpost [link](https://medium.com/google-cloud/importing-data-from-databases-into-gcs-via-jdbc-using-dataproc-serverless-f330cb0160f0))
* [JDBCToJDBC](/python/dataproc_templates/jdbc#1-jdbc-to-jdbc) (blogpost [link](https://medium.com/google-cloud/migrating-data-from-one-databases-into-another-via-jdbc-using-dataproc-serverless-c5336c409b18))
* [KafkaToGCS](/python/dataproc_templates/kafka/#kafka-to-gcs) (Blogpost [link](https://medium.com/google-cloud/kafka-to-gcs-using-dataproc-serverless-python-template-750ed218a5cc))
* [KafkaToBigQuery](/python/dataproc_templates/kafka/#kafka-to-bq) (Blogpost [link](https://medium.com/google-cloud/dataproc-serverless-template-to-migrate-your-data-from-kafka-to-bigquery-9b635a66b9dc))
* [MongoToGCS](/python/dataproc_templates/mongo#mongo-to-gcs) (blogpost [link](https://medium.com/google-cloud/exporting-data-from-mongodb-to-gcs-buckets-using-dataproc-serverless-64830fb15b51))
* [PubSubLiteToBigtable](/python/dataproc_templates/pubsublite#pubsublite-to-bigtable)
* [RedshiftToGCS](/python/dataproc_templates/redshift#redshift-to-gcs) (blogpost [link](https://medium.com/google-cloud/exporting-data-from-redshift-to-gcs-using-gcp-dataproc-serverless-and-pyspark-9ab78de11405))
* [S3ToBigQuery](/python/dataproc_templates/s3#amazon-s3-to-bigquery)
* [SnowflakeToGCS](/python/dataproc_templates/snowflake#1-snowflake-to-gcs) (blogpost [link](https://medium.com/@varunikagupta96/exporting-data-from-snowflake-to-gcs-using-pyspark-on-dataproc-serverless-363d3bed551b))
* [TextToBigQuery](/python/dataproc_templates/gcs#text-to-bigquery) (blogpost [link](https://medium.com/google-cloud/dataproc-serverless-pyspark-template-for-ingesting-compressed-text-files-to-bigquery-c6eab8fb6bc9))

## Dataproc Templates (Notebooks)
Please refer to the [Dataproc Templates (Notebooks) README](/notebooks) for more information

* [HiveToBigQuery](/notebooks/hive2bq#dataproc-template-to-migrate-hive-tables-to-bigquery-using-jupyter-notebooks) (blogpost [link](https://medium.com/google-cloud/hive-to-bigquery-move-data-efficiently-using-gcp-dataproc-serverless-ee30d35aaf03))
* [MsSqlToBigQuery](/notebooks/mssql2bq#jupyter-notebook-solution-for-migrating-mssql-sql-server-to-bigquery-dwh-using-dataproc-templates)(blogpost [link](https://medium.com/google-cloud/mssql-to-bigquery-migrate-efficiently-using-vertex-ai-notebook-and-gcp-dataproc-serverless-98358943568a))
* [MySQLToSpanner](/notebooks/mysql2spanner#jupyter-notebook-solution-for-migrating-mysql-database-to-cloud-spanner-using-dataproc-templates) (blogpost [link](https://medium.com/google-cloud/mysql-to-cloud-spanner-migrate-mysql-database-to-cloud-spanner-using-vertex-ai-notebooks-and-gcp-ad7d2ed8a317))
* [SQLServerToPostgres](/notebooks/mssql2postgresql#jupyter-notebook-solution-for-migrating-mssql-sql-server-to-postgres-database-using-dataproc-templates)
* [OracleToBigQuery](/notebooks/oracle2bq#jupyter-notebook-solution-for-migrating-oracle-database-to-bigquery-using-dataproc-templates)(blogpost [link](https://medium.com/@neerajshivhare/oracle-to-bigquery-migrate-oracle-to-bigquery-using-vertex-ai-notebooks-and-gcp-dataproc-9d22b73968c3))
* [OracleToPostgres](/notebooks/oracle2postgres#jupyter-notebook-solution-for-migrating-oracle-database-to-postgres-using-dataproc-templates)(blogpost [Link](https://medium.com/google-cloud/oracle-to-postgresql-migrate-oracle-to-postgresql-using-vertex-ai-notebooks-and-gcp-dataproc-9924f3f5c51f))
* [OracleToSpanner](/notebooks/oracle2spanner#jupyter-notebook-solution-for-migrating-oracle-database-to-cloud-spanner-using-dataproc-templates) (blogpost [Link](https://medium.com/@surjitsh/oracle-to-cloud-spanner-migrate-oracle-database-to-cloud-spanner-using-vertex-ai-notebooks-and-gcp-49152ce7f4e8))

## Getting Started

1) Clone this repository

        git clone https://github.com/GoogleCloudPlatform/dataproc-templates.git
2) Obtain authentication credentials

   Create local credentials by running the following command and following the
   oauth2 flow (read more about the command [here](https://cloud.google.com/sdk/gcloud/reference/auth/application-default/login).

        gcloud auth application-default login

   Or manually set the `GOOGLE_APPLICATION_CREDENTIALS` environment variable
   to point to a service account key JSON file path.

   Learn more at [Setting Up Authentication for Server to Server Production Applications](https://developers.google.com/identity/protocols/oauth2/service-account).

*Note:* Application Default Credentials is able to implicitly find the credentials as long as the application is running on Compute Engine, Kubernetes Engine, App Engine, or Cloud Functions.

3) Executing a Template

    Follow the specific guide, depending on your use case:
   - [Dataproc Templates (Java - Spark)](java)
   - [Dataproc Templates (Python - PySpark)](python)

## Flow diagram

Below flow diagram shows execution flow for Dataproc Templates:

![Dataproc templates flow diagram](dp-templates.png)

## Contributing
See the contributing [instructions](/CONTRIBUTING.md) to get started contributing.

## License
All solutions within this repository are provided under the [Apache 2.0](https://www.apache.org/licenses/LICENSE-2.0) license. Please see the [LICENSE](/LICENSE) file for more detailed terms and conditions.

## Disclaimer
This repository and its contents are not an official Google Product.

## Contact
Share your feedback, ideas, thoughts [feedback-form](https://forms.gle/XXCJeWeCJJ9fNLQS6)

Questions, issues, and comments should be directed to dataproc-templates-support-external@googlegroups.com

[gcf]: https://cloud.google.com/functions/
[gcf-bg]: https://cloud.google.com/functions/docs/writing/background
[logs-export]: https://cloud.google.com/logging/docs/export/


# MISP-STIX - Python library to handle the conversion between MISP standard and STIX

<img src="https://raw.githubusercontent.com/MISP/misp-stix/main/documentation/logos/misp-stix.png" width="125" height="125">

[![Python version](https://img.shields.io/badge/python-3.8+-blue.svg)](https://www.python.org/downloads/release/python-370/)
[![MISP-STIX version](https://badge.fury.io/gh/MISP%2Fmisp-stix.svg)](https://badge.fury.io/gh/MISP%2Fmisp-stix)
[![Github Actions](https://github.com/MISP/misp-stix/workflows/misp-stix/badge.svg)](https://github.com/MISP/misp-stix/actions?query=workflow%3Amisp-stix)
[![License](https://img.shields.io/github/license/MISP/misp-stix.svg)](#License)

MISP-STIX-converter is a Python library (>=3.8) to handle all the conversions between the [MISP standard format](https://www.misp-standard.org/) and STIX formats.

The package is available as [misp-stix](https://pypi.org/project/misp-stix/) in PyPI.

## Features

- MISP standard format conversion to STIX 1.x (1.1.1 and 1.2)
- MISP standard format conversion to STIX 2.x (2.0 and 2.1)
- Maps [MISP Objects](https://github.com/MISP/misp-objects) and [MISP galaxies](https://github.com/misp/misp-galaxy) with [respective semantically similar objects](https://github.com/MISP/misp-stix/tree/main/documentation)
- STIX to MISP standard format
- Provides an extendable library for mapping and facilitate extension

This library is used by the [MISP core software](https://github.com/MISP/MISP) to perform STIX conversion and serving as a useful tool for anyone looking for a clean way of converting between the MISP standard format and various STIX versions (1.1.1, 1.2, 2.0, 2.1).

A complete [documentation is available](./documentation/) including the mappings between the different formats.

## Install from pip

**It is strongly recommended to use a virtual environment**

If you want to know more about virtual environments, [python has you covered](https://docs.python.org/3/tutorial/venv.html)

From the current repository:
```
pip3 install misp-stix
```

Package details at PyPI: [misp-stix](https://pypi.org/project/misp-stix/)

## Install the latest version from the repository for development purposes

**Note**: poetry is required; e.g., `pip3 install poetry`

```
git clone https://github.com/MISP/misp-stix.git && cd misp-stix
git submodule update --init
poetry install
```

If you already have poetry face any issue with it while installing or updating misp-stix with it, you can try `pip3 install -U poetry` to make sure you have a version >= 1.2

### Running the tests

Tests for MISP format export as STIX 1.1.1 & 1.2:
```bash
poetry run pytest tests/test_stix1_export.py
```

Tests for MISP format export as STIX 2.0:
```bash
poetry run pytest tests/test_stix20_export.py
```

Tests for MISP format export as STIX 2.1:
```bash
poetry run pytest tests/test_stix21_export.py
```

## Usage

### Command-line Usage

```
misp_stix_converter export --version 2.1 -f tests/test_events_collection_1.json
```

#### Parameters

```bash
usage: misp_stix_converter [-h] [--debug] {export,import} ...

Convert MISP <-> STIX

options:
  -h, --help       show this help message and exit
  --debug          Show errors and warnings

Main feature:
  {export,import}
    export         Export MISP to STIX - try `misp_stix_converter export -h` for more help.
    import         Import STIX to MISP - try `misp_stix_converter import -h` for more help.
```

##### Export parameters

```bash
usage: misp_stix_converter export [-h] -f FILE [FILE ...] -v {1.1.1,1.2,2.0,2.1} [-s] [-m] [--output_dir OUTPUT_DIR] [-o OUTPUT_NAME] [--level {attribute,event}] [--format {json,xml}] [-n NAMESPACE] [-org ORG]

options:
  -h, --help            show this help message and exit
  -f FILE [FILE ...], --file FILE [FILE ...]
                        Path to the file(s) to convert.
  -v {1.1.1,1.2,2.0,2.1}, --version {1.1.1,1.2,2.0,2.1}
                        STIX specific version.
  -s, --single_output   Produce only one result file (in case of multiple input file).
  -m, --in_memory       Store result in memory (in case of multiple result files) instead of storing it in tmp files.
  --output_dir OUTPUT_DIR
                        Output path - used in the case of multiple input files when the `single_output` argument is not used.
  -o OUTPUT_NAME, --output_name OUTPUT_NAME
                        Output file name - used in the case of a single input file or when the `single_output` argument is used.

STIX 1 specific arguments:
  --level {attribute,event}
                        MISP data structure level.
  --format {json,xml}   STIX 1 format.
  -n NAMESPACE, --namespace NAMESPACE
                        Namespace to be used in the STIX 1 header.
  -org ORG              Organisation name to be used in the STIX 1 header.
```

##### Import parameters

```bash
usage: misp_stix_converter import [-h] -f FILE [FILE ...] -v {1,2} [-s] [-o OUTPUT_NAME] [--output_dir OUTPUT_DIR] [-d DISTRIBUTION] [-sg SHARING_GROUP] [--galaxies_as_tags]

options:
  -h, --help            show this help message and exit
  -f FILE [FILE ...], --file FILE [FILE ...]
                        Path to the file(s) to convert.
  -v {1,2}, --version {1,2}
                        STIX major version.
  -s, --single_output   Produce only one MISP event per STIX file(in case of multiple Report, Grouping or Incident objects).
  -o OUTPUT_NAME, --output_name OUTPUT_NAME
                        Output file name - used in the case of a single input file or when the `single_output` argument is used.
  --output_dir OUTPUT_DIR
                        Output path - used in the case of multiple input files when the `single_output` argument is not used.
  -d DISTRIBUTION, --distribution DISTRIBUTION
                        Distribution level for the imported MIPS content.
  -sg SHARING_GROUP, --sharing_group SHARING_GROUP
                        Sharing group ID when distribution is 4.
  --galaxies_as_tags    Import MISP Galaxies as tag names instead of the standard Galaxy format.
```

### In Python scripts

Given a MISP Event (with its metadata fields, attributes, objects, galaxies and tags), declared in an `event` variable in Python dict format, you can get the result of a conversion into one of the supported STIX versions:

- Convert a MISP Event in STIX1:

```python
from misp_stix_converter import MISPtoSTIX1EventsParser

parser = MISPtoSTIX1EventsParser(
    'MISP-Project', # Example of Org name
    '1.1.1' # STIX1 version (1.1.1 or 1.2)
)
parser.parse_misp_event(event)

stix_package = parser.stix_package
```

- Convert a MISP Event in STIX1 using directly its file name:

```python
from misp_stix_converter import misp_to_stix1

response = misp_to_stix1(
    filename, # file name of the file containing a MISP Event
    'xml', # return format (XML or JSON)
    '1.1.1' # STIX1 version (1.1.1 or 1.2)
)
# if everything went well, response is a dictionary where `success` = 1
```
The resulting STIX1 Package is then available in a `filename.out` file

- Convert a MISP Event in STIX2:

```python
# for STIX 2.0
from misp_stix_converter import MISPtoSTIX20Parser
# for STIX 2.1
from misp_stix_converter import MISPtoSTIX21Parser

parser20 = MISPtoSTIX20Parser()
parser20.parse_misp_event(event)

parser21 = MISPtoSTIX21Parser()
parser21.parse_misp_event(event)

# To get the list of parsed STIX objects
stix_20_objects = parser20.stix_objects
stix_21_objects = parser21.stix_objects

# To get the list of parser STIX objects within a STIX 2.0 or 2.1 Bundle
bundle20 = parser20.bundle
bundle21 = parser21.bundle
```

- Convert a MISP Event in STIX2 using directly its file name:

```python
from misp_stix_converter import misp_to_stix2

response_20 = misp_to_stix2(filename, version='2.0')
response_21 = misp_to_stix2(filename, version='2.1')
# Again response_20 & response_21 have a `success` field equal to 1 if everything went well
```
The resulting STIX2 Bundle is the available in a `filename.out` file, or you can define the output name with the `output_name` argument.

If you get some MISP collection of data, it is also possible to convert it straight into some STIX format:

```python
from misp_stix_converter import MISPtoSTIX1EventsParser, MISPtoSTIX20Parser, MISPtoSTIX21Parser

filename = _PATH_TO_YOUR_FILE_CONTAINING_MISP_FORMAT_

parser1 = MISPtoSTIX1EventsParser('MISP', '1.1.1')
parser1.parse_json_content(filename)
stix_package = parser1.stix_package

parser20 = MISPtoSTIX20Parser()
parser20.parse_json_content(filename)
stix_20_objects = parser20.stix_objects
bundle20 = parser20.bundle

parser21 = MISPtoSTIX21Parser()
parser21.parse_json_content(filename)
stix_21_objects = parser21.stix_objects
bundle21 = parser21.bundle
```

But in order to parse multiple data collections, you can also use the following helpers:

```python
from misp_stix_converter import misp_event_collection_to_stix1, misp_event_collection_to_stix2

input_filenames = [filename for filename in Path(_PATH_TO_YOUR_MISP_FILES_).glob('*.json')]

stix1_response = misp_event_collection_to_stix1(
    *input_filenames,
    output_name=output_filename, # path to the file where the results are going to be written
    return_format='xml', # STIX1 return format (XML or JSON)
    version='1.1.1' # STIX1 version (1.1.1 or 1.2)
)

stix20_response = misp_event_collection_to_stix2(
    *input_filenames,
    version='2.0' # STIX 2 version
)

stix21_response = misp_event_collection_to_stix2_1(
    *input_filenames,
    version='2.1',
    single_output=True, # For a single resulting file
    output_name=output_file_name, # path to the file where the results are going to be written
    in_memory=True # To keep results in memory before writing the full converted content at the end in the result file
)
```
Again, all the responses should have a `success` field equal to 1 and the resulting STIX1 Package and STIX 2.0 & 2.1 Bundles are available in the specific output file names.

### Samples and examples

Various examples are provided and used by the different tests scripts in the [tests](tests/) directory.
Those example files are showing the results of MISP format exported in the various supported STIX formats.

## MISP <--> STIX Mapping

A specific documentation concerning the mapping between MISP and the various supported STIX versions is also provided in the [documentation](documentation/) directory.
You can find there all the different cases illustrated with examples.

# License

misp-stix is released under a BSD 2-Clause "Simplified" License allow easy reuse with other libraries.

~~~
Copyright 2019-2023 Christian Studer
Copyright 2019-2023 CIRCL - Computer Incident Response Center Luxembourg c/o "security made in Lëtzebuerg" (SMILE) g.i.e.

Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met:

1. Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer.

2. Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution.

THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
~~~


=======
RavenPy
=======

|pypi| |conda| |license| |build| |docs| |coveralls|

A Python wrapper to setup and run the hydrologic modelling framework Raven_.

* Free software: MIT license
* Documentation: https://ravenpy.readthedocs.io

`RavenPy` is a Python wrapper for Raven_, accompanied by utility functions that facilitate model configuration, calibration, and evaluation.

Raven_ is an hydrological modeling framework that lets hydrologists build hydrological models by combining different hydrological processes together. It can also be used to emulate a variety of existing lumped and distributed models. Model structure, parameters, initial conditions and forcing files are configured in text files, which Raven parses to build and run hydrological simulations. A detailed description about modeling capability of Raven_ can be found in the `docs`_.

`RavenPy` provides a Python interface to Raven_, automating the creation of configuration files and allowing the model to be launched from Python. Results, or errors, are automatically parsed and exposed within the programming environment. This facilitates the launch of parallel simulations, multi-model prediction ensembles, sensitivity analyses and other experiments involving a large number of model runs.

Note that version 0.12 includes major changes compared to the previous 0.11 release, and breaks backward compatibility. The benefits of these changes are a much more intuitive interface for configuring and running the model.

Features
--------

* Configure, run and parse Raven outputs from Python
* Utility command to create grid weight files
* Extract physiographic information about watersheds
* Algorithms to estimate model parameters from ungauged watersheds
* Exposes outputs (flow, storage) as `xarray.DataArray` objects

Install
-------

Please see the detailed `installation docs`_.

Acknowledgements
----------------

RavenPy's development has been funded by CANARIE_ and Ouranos_ and would be not be possible without the help of Juliane Mai and James Craig.

This package was created with Cookiecutter_ and the `Ouranosinc/cookiecutter-pypackage`_ project template.

.. _CANARIE: https://www.canarie.ca
.. _Cookiecutter: https://github.com/audreyfeldroy/cookiecutter-pypackage
.. _Ouranos: https://www.ouranos.ca
.. _Ouranosinc/cookiecutter-pypackage: https://github.com/Ouranosinc/cookiecutter-pypackage
.. _Raven: http://raven.uwaterloo.ca
.. _docs: https://www.civil.uwaterloo.ca/raven/files/v3.7/RavenManual_v3.7.pdf
.. _installation docs: https://ravenpy.readthedocs.io/en/latest/installation.html
.. _raven-hydro: https://github.com/Ouranosinc/raven-hydro

.. |pypi| image:: https://img.shields.io/pypi/v/ravenpy.svg
        :target: https://pypi.python.org/pypi/ravenpy
        :alt: PyPI

.. |conda| image:: https://img.shields.io/conda/vn/conda-forge/ravenpy
        :target: https://anaconda.org/conda-forge/ravenpy
        :alt: Conda-Forge

.. |license| image:: https://img.shields.io/github/license/CSHS-CWRA/RavenPy.svg
        :target: https://github.com/CSHS-CWRA/RavenPy/blob/master/LICENSE
        :alt: License

.. |build| image:: https://github.com/CSHS-CWRA/RavenPy/actions/workflows/main.yml/badge.svg
        :target: https://github.com/CSHS-CWRA/RavenPy/actions/workflows/main.yml
        :alt: Build status

.. |docs| image:: https://readthedocs.org/projects/ravenpy/badge/?version=latest
        :target: https://ravenpy.readthedocs.io
        :alt: Documentation Status

.. |coveralls| image:: https://coveralls.io/repos/github/CSHS-CWRA/RavenPy/badge.svg?branch=master
        :target: https://coveralls.io/github/CSHS-CWRA/RavenPy?branch=master
        :alt: Coveralls


# IDF Component Manager

The IDF Component manager is a tool that downloads dependencies for any [ESP-IDF](https://www.espressif.com/en/products/sdks/esp-idf) CMake project. It makes sure that the right versions of all components required for a successful build of your project are in place. The download happens automatically during a run of CMake. It can source components either from [the ESP Component Registry](https://components.espressif.com/) or from a git repository.

**A list of components can be found at https://components.espressif.com/**

## [Official documentation](https://docs.espressif.com/projects/idf-component-manager/en/latest/)

## Disabling the Component Manager

The component manager can be explicitly disabled by setting `IDF_COMPONENT_MANAGER` environment variable to `0`.

## Using with a project

You can add `idf_component.yml` manifest files with the list of dependencies to any component in your project.

IDF Component Manager will download dependencies automatically during the project build process.

When CMake configures the project (e.g. `idf.py reconfigure`) component manager does a few things:

- Processes `idf_component.yml` manifests for every component in the project
- Creates a `dependencies.lock` file in the root of the project with a full list of dependencies
- Downloads all dependencies to the `managed_components` directory

The component manager won't try to regenerate `dependencies.lock` or download any components if manifests, lock file, and content of `managed_component` directory weren't modified since the last successful build.

## Defining dependencies in the manifest

All dependencies are defined in the manifest file.

```yaml
dependencies:
  # Required IDF version
  idf: ">=4.1"
  # For components maintained by Espressif only name can be used.
  # Same as `espressif/component`
  component:
    version: "~2.0.0"
  # Or in a shorter form
  component2: ">=1.0.0"
  # For 3rd party components :
  username/component:
    version: "~1.0.0"
    # For transient dependencies `public` flag can be set.
    public: true
  anotheruser/component: "<3.2.20"
  # For components hosted on non-default registry:
  company_user/component:
    version: "~1.0.0"
    registry_url: "https://componentregistry.company.com"
  # For components in git repository:
  test_component:
    path: test_component
    git: ssh://git@gitlab.com/user/components.git
  # For test projects during component development
  # components can be used from a local directory
  # with relative or absolute path
  some_local_component:
    path: ../../projects/component
  # For optional dependencies
  optional_component:
    version: "~1.0.0"
    rules: # will add "optional_component" only when all if clauses are True
      - if: "idf_version >=3.3,<5.0" # supports all SimpleSpec grammars (https://python-semanticversion.readthedocs.io/en/latest/reference.html#semantic_version.SimpleSpec)
      - if: "target in [esp32, esp32c3]" # supports boolean operator ==, !=, in, not in.
  # For example of the component
  namespace/component_with_example:
    version: "~1.0.0" # if there is no `override_path` field, use component from registry
    override_path: "../../" # use component in a local directory, not from registry
  namespace/no_required_component:
    version: "*"
    require: no # Download component but don't add it as a requirement
  namespace/pre_release_component:
    version: "*"
    pre_release: true # Allow downloading of pre-release versions
```

## Component metadata caching

By default, information about available versions of components not cached. If you make many requests to the registry from one machine, you can enable caching by setting `IDF_COMPONENT_API_CACHE_EXPIRATION_MINUTES` environment variable to the number of minutes to cache the data.

## External links

You can add links to the `idf_component.yml` file to the root of the manifest:

```yaml
url: "https://example.com/homepage" # URL of the component homepage
repository: "https://gitexample.com/test_project" # URL of the public repository with component source code, i.e GitHub, GitLab, etc.
documentation: "https://example.com/documentation" # URL of the component documentation
issues: "https://git.example.com/test_project/tracker" # URL of the issue tracker
discussion: "https://discord.example.com/test_project" # URL of the component discussion, i.e. Discord, Gitter, forum, etc.
```

A link should be a correct HTTP(S) URL like `https://example.com/path` except the `repository` field,
it is expected to be a valid [Git remote](https://git-scm.com/book/en/v2/Git-Basics-Working-with-Remotes) URL.

## Add examples to the component

To add examples to your component, place them in the `examples` directory inside your component.
Examples are discovered recursively in subdirectories at this path.
A directory with `CMakeLists.txt` that registers a project is considered as an example.

## Custom example paths

You can specify custom example paths for uploading them to the component registry.
For that, add `examples` field to the root of the manifest:

```yaml
examples:
  - path: ../some/path
  - path: ../some/other_path
```

## Environment variables

| Variable                                     | Default value (or example for required) | Description                                                                                |
|----------------------------------------------|-----------------------------------------|---------------------------------------------------------------------------------------------------------------|
| IDF_COMPONENT_API_TOKEN                      |                                         | API token to access the component registry                                                                    |
| IDF_COMPONENT_REGISTRY_URL                   | https://components.espressif.com/       | URL of the default component registry                                                                         |
| IDF_COMPONENT_STORAGE_URL                    | https://components-file.espressif.com/  | URL of the default file storage server                                                                        |
| IDF_COMPONENT_REGISTRY_PROFILE               | default                                 | Profile in the config file to use for component registry                                                      |
| IDF_COMPONENT_CACHE_PATH                     | \* Depends on OS                        | Cache directory for component manager                                                                         |
| IDF_COMPONENT_VERSION_PROCESS_TIMEOUT        | 300                                     | Timeout in seconds to wait for component processing                                                           |
| IDF_COMPONENT_OVERWRITE_MANAGED_COMPONENTS   | 0                                       | Overwrite files in the managed_component directory, even if they have been modified by the user               |
| IDF_COMPONENT_SUPPRESS_UNKNOWN_FILE_WARNINGS | 0                                       | Ignore unknown files in managed_components directory                                                          |
| IDF_COMPONENT_CHECK_NEW_VERSION              | 1                                       | Check for new versions of components                                                                          |
| IDF_COMPONENT_VERIFY_SSL                     | 1                                       | Verify SSL certificates when making requests to the registry, set it 0 to disable or provide a CA bundle path |

## Contributions Guide

We welcome all contributions to the Component Manager project.

You can contribute by fixing bugs, adding features, adding documentation, or reporting an [issue](https://github.com/espressif/idf-component-manager/issues). We accept contributions via [GitHub Pull Requests](https://docs.github.com/en/github/collaborating-with-pull-requests/proposing-changes-to-your-work-with-pull-requests/about-pull-requests).

Before reporting an issue, make sure you've searched for a similar one that was already created. If you are reporting a new issue, please follow the Issue Template.

## Installing a development version of the component manager

You can install the development version of the component manager from the main branch of this repository:

**On Linux/macOS:**

Go to the directory with your ESP-IDF installation and run:

```bash
# activate ESP-IDF environment
source ./export.sh # or . ./export.fish, if you use fish shell
# remove old version of the component manager
python -m pip uninstall -y idf-component-manager
# install the development version (from the main branch)
python -m pip install git+https://github.com/espressif/idf-component-manager.git@main
```

**On Windows:**

Run `ESP-IDF PowerShell Environment` or `ESP-IDF Command Prompt (cmd.exe)` from the Start menu and run the following command:

```powershell
# remove old version of the component manager
python -m pip uninstall -y idf-component-manager
# install the development version (from the main branch)
python -m pip install git+https://github.com/espressif/idf-component-manager.git@main
```

## Resources

- The Component manager [Documentation](https://docs.espressif.com/projects/idf-component-manager/en/latest/)
- The Python Package Index project page https://pypi.org/project/idf-component-manager/
- The Component Manager section in the [ESP-IDF Programming Guide](https://docs.espressif.com/projects/esp-idf/en/latest/esp32/api-guides/tools/idf-component-manager.html)


# <img src="https://i.imgur.com/nPCcxts.png" height="25" style="height: 25px"> MCStatus

[![discord chat](https://img.shields.io/discord/936788458939224094.svg?logo=Discord)](https://discord.gg/C2wX7zduxC)
![supported python versions](https://img.shields.io/pypi/pyversions/mcstatus.svg)
[![current PyPI version](https://img.shields.io/pypi/v/mcstatus.svg)](https://pypi.org/project/mcstatus/)
[![Docs](https://img.shields.io/readthedocs/mcstatus?label=Docs)](https://mcstatus.readthedocs.io/)
[![Validation](https://github.com/py-mine/mcstatus/actions/workflows/validation.yml/badge.svg)](https://github.com/py-mine/mcstatus/actions/workflows/validation.yml)
[![Tox test](https://github.com/py-mine/mcstatus/actions/workflows/tox-test.yml/badge.svg)](https://github.com/py-mine/mcstatus/actions/workflows/tox-test.yml)

Mcstatus provides an API and command line script to fetch publicly available data from Minecraft servers. Specifically, mcstatus retrieves data by using these protocols: [Server List Ping](https://wiki.vg/Server_List_Ping) and [Query](https://wiki.vg/Query). Because of mcstatus, you do not need to fully understand those protocols and can instead skip straight to retrieving minecraft server data quickly in your own programs.

## Installation

Mcstatus is available on [PyPI](https://pypi.org/project/mcstatus/), and can be installed trivially with:

```bash
python3 -m pip install mcstatus
```

## Usage

### Python API

#### Java Edition

```python
from mcstatus import JavaServer

# You can pass the same address you'd enter into the address field in minecraft into the 'lookup' function
# If you know the host and port, you may skip this and use JavaServer("example.org", 1234)
server = JavaServer.lookup("example.org:1234")

# 'status' is supported by all Minecraft servers that are version 1.7 or higher.
# Don't expect the player list to always be complete, because many servers run
# plugins that hide this information or limit the number of players returned or even
# alter this list to contain fake players for purposes of having a custom message here.
status = server.status()
print(f"The server has {status.players.online} player(s) online and replied in {status.latency} ms")

# 'ping' is supported by all Minecraft servers that are version 1.7 or higher.
# It is included in a 'status' call, but is also exposed separate if you do not require the additional info.
latency = server.ping()
print(f"The server replied in {latency} ms")

# 'query' has to be enabled in a server's server.properties file!
# It may give more information than a ping, such as a full player list or mod information.
query = server.query()
print(f"The server has the following players online: {', '.join(query.players.names)}")
```

#### Bedrock Edition

```python
from mcstatus import BedrockServer

# You can pass the same address you'd enter into the address field in minecraft into the 'lookup' function
# If you know the host and port, you may skip this and use BedrockServer("example.org", 19132)
server = BedrockServer.lookup("example.org:19132")

# 'status' is the only feature that is supported by Bedrock at this time.
# In this case status includes players.online, latency, motd, map, gamemode, and players.max. (ex: status.gamemode)
status = server.status()
print(f"The server has {status.players.online} players online and replied in {status.latency} ms")
```

See the [documentation](https://mcstatus.readthedocs.io) to find what you can do with our library!

### Command Line Interface

This only works with Java servers; Bedrock is not yet supported. Use `mcstatus -h` to see helpful information on how to use this script.

## License

Mcstatus is licensed under the Apache 2.0 license. See LICENSE for full text.


<div align="center">

![biocompute](static/biocompute.png)

# Latch SDK

The Latch SDK is a framework to build and deploy bioinformatics workflows, and
dynamically generate associated interfaces, with a handful of python functions.

It is built directly on [Flyte](https://docs.flyte.org) for all the benefits that the Kubernetes-native
workflow orchestration framework provides - task-level type-safety and
containerization, independent task scheduling, and heterogeneous & highly
scalable computing infrastructure.

[Slack Community](https://join.slack.com/t/latchbiosdk/shared_invite/zt-193ibmedi-WB6mBu2GJ2WejUHhxMOuwg) • [Docs](https://docs.latch.bio) • [Installation](#installation) •
[Quickstart](#configuration) • [Latch](https://latch.bio) • [Latch Verified](https://github.com/latch-verified)

![side-by-side](static/side-by-side.png)

</div>

Workflows developed with the SDK feature:

  + Instant no-code interfaces for accessibility and publication
  + First class static typing
  + Containerization + versioning of every registered change
  + Reliable + scalable managed cloud infrastructure
  + Single line definition of arbitrary resource requirements (eg. CPU, GPU) for serverless execution

The Latch SDK is a framework to build workflows. A collection of existing and
maintained workflows for common biological assays can be found at [Latch
Verified](https://github.com/latch-verified).

### Quickstart

Getting your hands dirty with SDK is the best way to understand how it works.
Run the following three commands in your terminal to register your first
workflow to LatchBio.

**Prerequisite**: ensure that `docker` is present and running on your machine.
(Install docker [here](https://docs.docker.com/get-docker/) if you don't already
have it installed.)

First, install latch through `pip` .

```shell
$ python3 -m pip install latch
```

Then, create some boilerplate code for your new workflow.

```shell
$ latch init testworkflow
```

Finally register the boilerplate code to [LatchBio](latch.bio).

```shell
$ latch register testworkflow
```

This might take 3-10 minutes depending on your network connection. (Subsequent
registers will complete in seconds by reusing the image layers from this initial
register.) The registration process will:

  + Build a docker image containing your workflow code
  + Serialize your code and register it with your LatchBio account
  + Push your docker image to a managed container registry

When registration has completed, you should be able to navigate
[here](https://console.latch.bio/workflows) and see your new workflow in your
account.

If you are having issues with registration or have general questions, please
file an issue on [github](https://github.com/latchbio/latch).

---

### Installation

The SDK is distributed on pip. Install in a fresh virtual environment for best
behavior.

[Virtualenvwrapper](https://virtualenvwrapper.readthedocs.io/en/latest/) is recommended.

```
python3 -m pip install latch
```

_Note that a local installation of docker is required to register workflows_.

---

### Examples

[Latch Verified](https://github.com/latch-verified) features list of well-curated workflows developed by the Latch team.
* [Bulk RNA-seq](https://github.com/latch-verified/bulk-rnaseq)
* [Differential Expression](https://github.com/latch-verified/diff-exp)
* [Pathway Analysis](https://github.com/latch-verified/pathway)

We'll maintain a growing list of well documented examples developed by our community members here. Please open a pull request to feature your own:

**Gene Editing**
  + [Guide Counter](https://github.com/latchbio/wf-guide_counter)
  + [Batch-GE: Gene editing analysis](https://github.com/latchbio/wf-batch_ge)

**Phylogenetics**
  + [Seq-to-tree: Evolutionary history inference](https://github.com/JLSteenwyk/latch_wf_seq_to_tree)
  + [Codon optimization estimation](https://github.com/JLSteenwyk/latch_wf_codon_optimization)
  + [Metamage: Taxonomy classification](https://github.com/jvfe/metamage_latch)

**Single-cell Analysis**
  + [ArchR: Single-cell chromatin accessibility analysis](https://github.com/aa20g217/Archr-Latch-Wf)
  + [emptydropsR: Ambient RNA detection](https://github.com/mrland99/emptydropsR-latch-wf)
  + [scVelo: Single-cell RNA velocity analysis](https://github.com/aa20g217/RNA-velocity-Latch-WF)

**Protein Engineering**
  + [UniRep: mLSTM "babbler" deep representation learner for protein engineering](https://github.com/matteobolner/unirep_latch)
  + [FAMSA: Multiple sequence protein alignment](https://github.com/shivaramakrishna99/famsa-latch)


# LNLDB
[![Deploy](https://www.herokucdn.com/deploy/button.svg)](https://heroku.com/deploy) [![Django CI](https://github.com/WPI-LNL/lnldb/actions/workflows/django-ci.yml/badge.svg)](https://github.com/WPI-LNL/lnldb/actions/workflows/django-ci.yml) [![Coverage Status](https://coveralls.io/repos/WPI-LNL/lnldb/badge.svg?branch=master&service=github)](https://coveralls.io/github/WPI-LNL/lnldb?branch=master) [![Documentation](https://readthedocs.org/projects/lnldb/badge/?version=latest&style=flat)](https://lnldb.readthedocs.io/en/latest/)

## Intro
LNLDB now runs under Python3.x and Django 2.2 or later.

## To Install (Testing)
##### Install required system packages
You are going to need some basic Python/Git tools to run the code. The most important are Python3 and virtualenv (which allows you to install python libs without root). The rest are to compile the python binary libraries, namely MySQL and image manipulation.

###### Linux:
```
sudo apt-get install python3 python3-dev python-pip python-virtualenv git libtiff5-dev libjpeg8-dev zlib1g-dev libfreetype6-dev liblcms2-dev libwebp-dev libmysqlclient-dev
```

###### macOS:
```
curl https://bootstrap.pypa.io/get-pip.py -o get-pip.py
python get-pip.py
pip install virtualenv
```

It is certainly possible to run via Windows practically without changes, but figuring out that setup is left as an
excercise for the user (hint: use pycharm).

##### Get the sources
If you're reading this outside of Github, you can skip this.

```
git clone https://github.com/WPI-LNL/lnldb.git
cd lnldb
```

##### Install Python packages
This uses a lot of library functionalities in the form of plugins. We will make a virtualenv to keep all of this away
from the system Python installation (ie. don't need root), and install our packages directly to the `env` folder.

```
virtualenv env
source env/bin/activate
pip install -r requirements_debug.txt
```

##### Initialize the database
The first line makes/loads the actual database schema, by walking through all of the previous schemas and making necessary
changes to the database one-by-one so that no data is ever lost regardless of versions. On your machine, the database
will by default be an Sqlite file in the runtime folder, on the server, it's WPI's hosted MySQL server.

The second line will populate any data with a useful initial value. Examples include groups, permissions, locations, etc.
If you have updated these or have existing data, don't run this.  You can also use this method to load a backup from the
production database (`dbbak` on server).

The third will create an account to let you log in once you've started the server.
```
python manage.py migrate
python manage.py loaddata fixtures/*.json
python manage.py createsuperuser
```

##### Run the tests

This app includes a number of self-checks to sanity test new code. All new patches are expected to have
tests included, and will be checked automatically when pushed to Github. Also try using the '-n' flag to speed it up.

```
python manage.py test
```

##### Run it!
`python manage.py runserver` or `python manage.py shell_plus`
Note that in future sessions, you must first call `source env/bin/activate` to set up the local path

## Notes

- All server-specific keys, directories, etc. haven't and won't be included. The app checks for a file at
    `lnldb/local_settings.py`, where those and any platform-specific options can be included.

## Installing on heroku

You can now deploy a test server using one button! Well.... almost. You still need to figure out s3, and make a login.

1. Fill in the allowed_hosts field with eg. 'yourappname.herokuapp.com,yourcustomdomain.com'
2. (Not required unless storing uploads) Make an [S3 bucket](https://console.aws.amazon.com/s3/home). Just leave everything disabled. Copy the bucket name and region id (eg. us-east-1) into the form.
3. (Not required unless storing uploads) Make an [IAM identity](https://console.aws.amazon.com/iam/home) to log in under. Copy the id and secret key into the form.
4. (Not required unless storing uploads) Add a IAM Inline Policy to grant permission to read/write on the bucket. A good base is below.
5. Hit deploy and wait.
6. `heroku run python manage.py createsuperuser`

Minimal IAM Policy
```
{
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "s3:GetObject",
                "s3:PutObject",
                "s3:PutObjectAcl",
                "s3:ListBucket"
            ],
            "Resource": [
                "arn:aws:s3:::yourbucket",
                "arn:aws:s3:::yourbucket/*"
            ]
        }
    ]
}
```


# pyiron_atomistics

[![Build Status](https://github.com/pyiron/pyiron_atomistics/workflows/Python%20package/badge.svg)](https://github.com/pyiron//pyiron/actions)
![Anaconda](https://anaconda.org/conda-forge/pyiron_atomistics/badges/downloads.svg)
![Release](https://anaconda.org/conda-forge/pyiron_atomistics/badges/latest_release_date.svg)

pyiron - an integrated development environment (IDE) for computational materials science. It combines several tools in
a common platform:

* Atomic structure objects – compatible to the [Atomic Simulation Environment (ASE)](https://wiki.fysik.dtu.dk/ase/).
* Atomistic simulation codes – like [LAMMPS](http://lammps.sandia.gov) and [VASP](https://www.vasp.at).
* Feedback Loops – to construct dynamic simulation life cycles.
* Hierarchical data management – interfacing with storage resources like SQL and [HDF5](https://support.hdfgroup.org/HDF5/).
* Integrated visualization – based on [NGLview](https://github.com/arose/nglview).
* Interactive simulation protocols - based on [Jupyter notebooks](http://jupyter.org).
* Object-oriented job management – for scaling complex simulation protocols from single jobs to high-throughput simulations.

![Screenshot of pyiron_atomistics running inside jupyterlab.](https://raw.githubusercontent.com/pyiron/pyiron_atomistics/main/docs/images/screenshots.png)

pyiron (called pyron) is developed in the [Computational Materials Design department](https://www.mpie.de/CM) of
[Joerg Neugebauer](https://www.mpie.de/person/43010/2763386) at the [Max Planck Institut für Eisenforschung (Max Planck Institute for iron research)](https://www.mpie.de/2281/en).
While its original focus was to provide a framework to develop and run complex simulation protocols as needed for ab
initio thermodynamics it quickly evolved into a versatile tool to manage a wide variety of simulation tasks. In 2016 the
[Interdisciplinary Centre for Advanced Materials Simulation (ICAMS)](http://www.icams.de) joined the development of the
framework with a specific focus on high throughput applications. In 2018 pyiron was released as open-source project.

**pyiron_atomistics**: This is the documentation page for the basic infrastructure moduls of pyiron.  If you're new to
pyiron and want to get an overview head over to [pyiron](https://pyiron.readthedocs.io/en/latest/).  If you're looking
for the API docs of pyiron_base check [pyiron_base](https://pyiron_base.readthedocs.io/en/latest/).

## Explore pyiron_atomistics
We provide various options to install, explore and run pyiron_atomistics:

* **Workstation Installation (recommeded)**: for Windows, Linux or Mac OS X workstations (interface for local VASP
  executable, support for the latest jupyterlab based GUI)
* **Mybinder.org (beta)**: test pyiron directly in your browser (no VASP license, no visualization, only temporary data
  storage)
* **Docker (for demonstration)**: requires Docker installation (no VASP license, only temporary data storage)

## Join the development
Please contact us if you are interested in using pyiron:

* to interface your simulation code or method
* implementing high-throughput approaches based on atomistic codes
* to learn more about method development and Big Data in material science.

Please also check out the pyiron_atomistics [contributing guidelines](https://github.com/pyiron/pyiron_atomistics/blob/main/CONTRIBUTING.rst).

## Citing
If you use pyiron in your research, please consider citing the following work:

```
@article{pyiron-paper,
    title = {pyiron: An integrated development environment for computational materials science},
    journal = {Computational Materials Science},
    volume = {163},
    pages = {24 - 36},
    year = {2019},
    issn = {0927-0256},
    doi = {https://doi.org/10.1016/j.commatsci.2018.07.043},
    url = {http://www.sciencedirect.com/science/article/pii/S0927025618304786},
    author = {Jan Janssen and Sudarsan Surendralal and Yury Lysogorskiy and Mira Todorova and Tilmann Hickel and Ralf Drautz and Jörg Neugebauer},
    keywords = {Modelling workflow, Integrated development environment, Complex simulation protocols},
}
```


## Injective Python SDK

[![codecov](https://codecov.io/gh/InjectiveLabs/sdk-python/graph/badge.svg?token=RBGK98G6F1)](https://codecov.io/gh/InjectiveLabs/sdk-python)

### Dependencies

**Ubuntu**
```bash
sudo apt install python3.X-dev autoconf automake build-essential libffi-dev libtool pkg-config
```
**Fedora**
```bash
sudo dnf install python3-devel autoconf automake gcc gcc-c++ libffi-devel libtool make pkgconfig
```

**macOS**

```bash
brew install autoconf automake libtool
```

### Quick Start
Installation
```bash
pip install injective-py
```

### Usage
Requires Python 3.9+
Please install `poetry` following the steps described in the [documentation](https://python-poetry.org/docs/#installation)

[Examples](https://github.com/InjectiveLabs/sdk-python/tree/master/examples)
```bash
$ poetry install

# connecting to Injective Exchange API
# and listening for new orders from a specific spot market
$ poetry run python examples/exchange_client/spot_exchange_rpc/8_StreamOrders.py

# sending a msg with bank transfer
# signs and posts a transaction to the Injective Chain
$ poetry run python examples/chain_client/1_MsgSend.py
```
Upgrade `pip` to the latest version, if you see these warnings:
  ```
  WARNING: Value for scheme.platlib does not match. Please report this to <https://github.com/pypa/pip/issues/10151>
  WARNING: Additional context:   user = True   home = None   root = None   prefix = None
  ```

### Development
1. Generate proto binding & build
  ```
  make gen
  poetry build
  ```

2. Install pkg
  ```
  # from local build
  pip uninstall injective-py
  pip install injective-py --no-index --find-links /path/to/injective/sdk-python/dist

  # from pypi
  pip uninstall injective-py
  pip install injective-py
  ```

3. Fetch latest denom config
```
poetry run python pyinjective/fetch_metadata.py
```

Note that the [sync client](https://github.com/InjectiveLabs/sdk-python/blob/master/pyinjective/client.py) has been deprecated as of April 18, 2022. If you are using the sync client please make sure to transition to the [async client](https://github.com/InjectiveLabs/sdk-python/blob/master/pyinjective/async_client.py), for more information read [here](https://github.com/InjectiveLabs/sdk-python/issues/101)

4. Run all unit tests in a development environment
```
poetry run pytest -v
```

## License

Copyright © 2021 - 2022 Injective Labs Inc. (https://injectivelabs.org/)

Originally released by Injective Labs Inc. under:

Apache License

Version 2.0, January 2004

http://www.apache.org/licenses/


[![PyPI version](https://badge.fury.io/py/Truvari.svg)](https://badge.fury.io/py/Truvari)
[![pylint](imgs/pylint.svg)](https://github.com/acenglish/truvari/actions/workflows/pylint.yml)
[![FuncTests](https://github.com/acenglish/truvari/actions/workflows/func_tests.yml/badge.svg?branch=develop&event=push)](https://github.com/acenglish/truvari/actions/workflows/func_tests.yml)
[![coverage](imgs/coverage.svg)](https://github.com/acenglish/truvari/actions/workflows/func_tests.yml)
[![develop](https://img.shields.io/github/commits-since/acenglish/truvari/v4.2.2)](https://github.com/ACEnglish/truvari/compare/v4.2.2...develop)
[![Downloads](https://static.pepy.tech/badge/truvari)](https://pepy.tech/project/truvari)

![Logo](https://raw.githubusercontent.com/ACEnglish/truvari/develop/imgs/BoxScale1_DarkBG.png)
Toolkit for benchmarking, merging, and annotating Structural Variants

📚 [WIKI page](https://github.com/acenglish/truvari/wiki) has detailed documentation.
📈 See [Updates](https://github.com/acenglish/truvari/wiki/Updates) on new versions.
📝 Read our Papers ([#1](https://genomebiology.biomedcentral.com/articles/10.1186/s13059-022-02840-6), [#2](https://rdcu.be/dFQNN)) to learn more.

## 💻 Installation
Truvari uses Python 3.6+ and can be installed with pip:
```
  python3 -m pip install Truvari
```
For details and more installation options, see [Installation](https://github.com/acenglish/truvari/wiki/Installation) on the wiki.

## ⏩ Quick Start

Each sub-command contains help documentation. Start with `truvari -h` to see available commands.

The current most common Truvari use case is for structural variation benchmarking:
```
  truvari bench -b base.vcf.gz -c comp.vcf.gz -o output_dir/
```

Find more matches by harmonizing phased varians using refine:
```
   truvari refine -R -U -r reference.fa --regions output_dir/candidate.refine.bed output_dir/
```

Use Truvari's comparison engine to consolidate redundant variants in a merged multi-sample VCF:
```
    bcftools merge -m none sampleA.vcf.gz sampleB.vcf.gz | bgzip > merge.vcf.gz
    tabix merge.vcf.gz
    truvari collapse -i merge.vcf.gz -o truvari_merge.vcf
```

## 🧬 Truvari Commands

 - [bench](https://github.com/acenglish/truvari/wiki/bench) - Performance metrics from comparison of two VCFs
 - [collapse](https://github.com/acenglish/truvari/wiki/collapse) - Collapse possibly redundant VCF entries
 - [refine](https://github.com/ACEnglish/truvari/wiki/refine) - Automated bench result refinement with phab
 - [anno](https://github.com/acenglish/truvari/wiki/anno) - Add SV annotations to a VCF
 - [phab](https://github.com/ACEnglish/truvari/wiki/phab) - Harmonize variant representations using MSA
 - [consistency](https://github.com/acenglish/truvari/wiki/consistency) - Consistency report between multiple VCFs
 - [vcf2df](https://github.com/acenglish/truvari/wiki/vcf2df) - Turn a VCF into a pandas DataFrame
 - [segment](https://github.com/acenglish/truvari/wiki/segment) - Normalization of SVs into disjointed genomic regions
 - [stratify](https://github.com/acenglish/truvari/wiki/stratify) - Count variants per-region in vcf
 - [divide](https://github.com/ACEnglish/truvari/wiki/divide) - Divide a VCF into independent shards

## 🔎 More Information

All documentation about Truvari is on the [WIKI](https://github.com/acenglish/truvari/wiki). Additional information about using Truvari can be found in [Discussions](https://github.com/ACEnglish/truvari/discussions)