<img src="https://github.com/opensearch-project/opensearch-benchmark/blob/main/opensearch_benchmark.png?raw=true"  height="64px" alt="OpenSearch Benchmark">

OpenSearch Benchmark is the macrobenchmarking framework for OpenSearch.

What is OpenSearch Benchmark?
-----------------------------

If you are looking to performance test OpenSearch, then OpenSearch Benchmark is for you. It can help you with the following tasks:

* Running performance benchmarks and recording results
* Setting up and tearing down OpenSearch clusters for benchmarking
* Managing benchmark data and specifications across OpenSearch versions
* Discovering performance problems by attaching so-called telemetry devices
* Comparing performance results
* Creating customized workloads

We have also put considerable effort into OpenSearch Benchmark to ensure that benchmarking data are reproducible.

Documentation
-------------

Official documentation for OpenSearch Benchmark is [available online](https://opensearch.org/docs/latest/benchmark/index/).

Quick Start
-----------

Want to get started with OpenSearch Benchmark quickly? See [OpenSearch Benchmark's Quick Start guide in the documentation](https://opensearch.org/docs/latest/benchmark/index/).

Getting help
------------

* Quick help: ``opensearch-benchmark --help``
* Want to contribute? Look at [OpenSearch Benchmark's Developer Guide](<https://github.com/opensearch-project/OpenSearch-Benchmark/blob/main/DEVELOPER_GUIDE.md>) for more information
* For any questions or answers, visit our [community forum](<https://discuss.opendistrocommunity.dev/>).
* File improvements or bug reports in our [Github repo](<https://github.com/opensearch-project/OpenSearch-Benchmark/issues>).

How to Contribute
-----------------

See all details in the [contributor guidelines](<https://github.com/opensearch-project/OpenSearch-Benchmark/blob/main/CONTRIBUTING.md>).

License
-------

This software is licensed under the Apache License, version 2 ("ALv2"), quoted below.

Copyright 2015-2022 OpenSearch <https://opensearch.org/>

Licensed under the Apache License, Version 2.0 (the "License"); you may not
use this file except in compliance with the License. You may obtain a copy of
the License at

    http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
License for the specific language governing permissions and limitations under
the License.


<!--- Copyright 2021 eprbell --->

<!--- Licensed under the Apache License, Version 2.0 (the "License"); --->
<!--- you may not use this file except in compliance with the License. --->
<!--- You may obtain a copy of the License at --->

<!---     http://www.apache.org/licenses/LICENSE-2.0 --->

<!--- Unless required by applicable law or agreed to in writing, software --->
<!--- distributed under the License is distributed on an "AS IS" BASIS, --->
<!--- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. --->
<!--- See the License for the specific language governing permissions and --->
<!--- limitations under the License. --->

![RP2 Logo](https://raw.githubusercontent.com/eprbell/rp2/main/docs/images/rp2_header.png)

# RP2 v1.6.0
Privacy-focused, free, powerful crypto tax calculator

[![Static Analysis / Main Branch](https://github.com/eprbell/rp2/actions/workflows/static_analysis.yml/badge.svg)](https://github.com/eprbell/rp2/actions/workflows/static_analysis.yml)
[![Documentation Check / Main Branch](https://github.com/eprbell/rp2/actions/workflows/documentation_check.yml/badge.svg)](https://github.com/eprbell/rp2/actions/workflows/documentation_check.yml)
[![Unix Unit Tests / Main Branch](https://github.com/eprbell/rp2/actions/workflows/unix_unit_tests.yml/badge.svg)](https://github.com/eprbell/rp2/actions/workflows/unix_unit_tests.yml)
[![Windows Unit Tests / Main Branch](https://github.com/eprbell/rp2/actions/workflows/windows_unit_tests.yml/badge.svg)](https://github.com/eprbell/rp2/actions/workflows/windows_unit_tests.yml)
[![CodeQL/Main Branch](https://github.com/eprbell/rp2/actions/workflows/codeql-analysis.yml/badge.svg)](https://github.com/eprbell/rp2/actions/workflows/codeql-analysis.yml)
<!-- markdown-link-check-disable -->
[![Tweet](https://img.shields.io/twitter/url/http/shields.io.svg?style=social)](https://twitter.com/intent/tweet?text=I%20use%20RP2,%20the%20privacy-focused,%20open%20source,%20free,%20non-commercial%20crypto%20tax%20calculator&url=https://github.com/eprbell/rp2/?anything)
<!-- markdown-link-check-enable -->
## Table of Contents
* **[Introduction](https://github.com/eprbell/rp2/blob/main/README.md#introduction)**
  * [How RP2 Operates](https://github.com/eprbell/rp2/blob/main/README.md#how-rp2-operates)
* **[License](https://github.com/eprbell/rp2/blob/main/README.md#license)**
* **[Download](https://github.com/eprbell/rp2/blob/main/README.md#download)**
* **[Installation](https://github.com/eprbell/rp2/blob/main/README.md#installation)**
  * [Ubuntu Linux](https://github.com/eprbell/rp2/blob/main/README.md#installation-on-ubuntu-linux)
  * [macOS](https://github.com/eprbell/rp2/blob/main/README.md#installation-on-macos)
  * [Windows 10](https://github.com/eprbell/rp2/blob/main/README.md#installation-on-windows-10)
  * [Other Unix-like Systems](https://github.com/eprbell/rp2/blob/main/README.md#installation-on-other-unix-like-systems)
* **[Running](https://github.com/eprbell/rp2/blob/main/README.md#running)**
* **[Input and Output Files](https://github.com/eprbell/rp2/blob/main/README.md#input-and-output-files)**
* **[RP2 Ecosystem](https://github.com/eprbell/rp2/blob/main/README.md#rp2-ecosystem)**
  * [List of Ecosystem Projects](https://github.com/eprbell/rp2/blob/main/README.md#list-of-ecosystem-projects)
* **[Reporting Bugs](https://github.com/eprbell/rp2/blob/main/README.md#reporting-bugs)**
* **[Contributing](https://github.com/eprbell/rp2/blob/main/README.md#contributing)**
* **[Developer Documentation](https://github.com/eprbell/rp2/blob/main/README.md#developer-documentation)**
* **[Frequently Asked Questions](https://github.com/eprbell/rp2/blob/main/README.md#frequently-asked-questions)**
* **[Change Log](https://github.com/eprbell/rp2/blob/main/README.md#change-log)**

## Introduction
[RP2](https://github.com/eprbell/rp2) is a privacy-focused, free, non-commercial, open-source, community-driven cryptocurrency tax calculator for multiple countries (currently US, Japan, Spain and Ireland are supported). Preparing crypto taxes can be a daunting and error-prone task, especially if multiple transactions, coins, exchanges and wallets are involved. This task could be delegated to a crypto tax preparation service, but many crypto users value their privacy and prefer not to send their transaction information to third parties unnecessarily. Additionally, many of these services cost money. RP2 solves all of these problems:
* it manages the complexity related to coin flows and tax calculation and it generates [reports that accountants can understand](https://github.com/eprbell/rp2/blob/main/docs/output_files.md#advisor-friendly-report-tax-report-us-output) (in the format of form 8949, for the US case), even if they are not cryptocurrency experts;
* it prioritizes user privacy by storing crypto transactions and tax results on the user's computer and not sending them anywhere else;
* it's 100% free, open-source and non-commercial.

This means that with RP2 there are no transaction limits, no premium versions, no payment requests, no personal data sent to a server (at risk of being hacked), no account creation, no unauditable source code.

Another unique advantage of RP2 is [transparent computation](https://github.com/eprbell/rp2/blob/main/docs/output_files.md#rp2-full-report-transparent-computation): RP2 generates full computation details for every lot fraction, so that it's possible to:
* verify step-by-step how RP2 reaches the final result;
* track down every lot fraction and its accounting details, in case of an audit.

RP2 supports multiple [countries and accounting methods](https://github.com/eprbell/rp2/blob/main/docs/supported_countries.md): supported accounting methods vary country by country.

RP2 reads a configuration file and an input spreadsheet containing crypto transactions. These [input files](https://github.com/eprbell/rp2/blob/main/docs/input_files.md) can be generated either manually or automatically using [DaLI](https://github.com/eprbell/dali-rp2), a RP2 data loader and input generator (which is also privacy-focused, free, non-commercial, open-source and community-driven). After parsing the input, RP2 uses high-precision math to calculate long/short term capital gains, cost bases, balances, average price, in/out lot relationships/fractions, and finally it generates [output files](https://github.com/eprbell/rp2/blob/main/docs/output_files.md).

RP2 has a programmable plugin architecture for [output generators](https://github.com/eprbell/rp2/tree/main/README.dev.md#adding-a-new-report-generator), [accounting methods](https://github.com/eprbell/rp2/tree/main/README.dev.md#adding-a-new-accounting-method) and [countries](https://github.com/eprbell/rp2/tree/main/README.dev.md#adding-support-for-a-new-country). Built-in [output generator plugins](https://github.com/eprbell/rp2/blob/main/docs/output_files.md) vary [country by country](https://github.com/eprbell/rp2/blob/main/docs/supported_countries.md), but RP2's architecture makes it possible to contribute additional generators for different countries or for different country-based cases.

RP2 has extensive [unit test](https://github.com/eprbell/rp2/tree/main/tests/) coverage to reduce the risk of regression.

**IMPORTANT DISCLAIMERS**:
* RP2 offers no guarantee of correctness (read the [license](https://github.com/eprbell/rp2/tree/main/LICENSE)): always verify results with the help of a tax professional.
* The author of RP2 is not a tax professional, but has used RP2 personally for a few years.

### How RP2 Operates
RP2 has been designed to have expressive primitives that can be used as building blocks for most tax scenarios: complex tax events can be described with patterns, built on top of these primitives (see the [FAQ list](https://github.com/eprbell/rp2/blob/main/docs/user_faq.md#tax-scenarios) for examples).

For the US case, RP2 treats virtual currency as property for tax purposes, as per [IRS Virtual Currency Guidance](https://www.irs.gov/newsroom/irs-reminds-taxpayers-to-report-virtual-currency-transactions).

RP2 supports various accounting methods (e.g. FIFO, LIFO, HIFO): however, in and out lots typically don't have matching amounts, so RP2 fractions them, maps in/out lot fractions and computes the resulting cost basis and capital gains for each lot fraction.

RP2 groups lot fractions into the following taxable event categories, each of which has a [specific tax treatment](https://github.com/eprbell/rp2/blob/main/docs/user_faq.md#which-crypto-tax-forms-to-file):
* [AIRDROP](https://github.com/eprbell/rp2/blob/main/docs/user_faq.md#how-to-handle-airdrops): gains from airdrops;
* [DONATE](https://github.com/eprbell/rp2/blob/main/docs/user_faq.md#how-to-handle-donations): donations to charitable organizations;
* [FEE](https://github.com/eprbell/rp2/blob/main/docs/user_faq.md#hhow-to-handle-fee-only-defi-transactions): fee-only transaction, used in some DeFi scenarios (e.g. gas fee for running certain smart contracts);
* [GIFT](https://github.com/eprbell/rp2/blob/main/docs/user_faq.md#how-to-handle-gifts): gifts to parties who are not charitable organizations (not tax-deductible).
* [HARDFORK](https://github.com/eprbell/rp2/blob/main/docs/user_faq.md#how-to-handle-hard-forks): gains from hard forks;
* [INCOME](https://github.com/eprbell/rp2/blob/main/docs/user_faq.md#how-to-handle-miscellaneous-crypto-income): gains from miscellaneous income (e.g. Coinbase Earn);
* [INTEREST](https://github.com/eprbell/rp2/blob/main/docs/user_faq.md#how-to-handle-crypto-interest): gains from interest;
* [MINING](https://github.com/eprbell/rp2/blob/main/docs/user_faq.md#how-to-handle-income-from-mining): gains from mining;
* [MOVE](https://github.com/eprbell/rp2/blob/main/docs/user_faq.md#how-to-handle-a-transfer-of-funds-from-a-wallet-or-exchange-to-another): the fee for moving currency between two accounts controlled by the same owner;
* SELL: specifically, sale and [conversion of a cryptocurrency to another](https://github.com/eprbell/rp2/blob/main/docs/user_faq.md#how-to-handle-conversion-of-a-cryptocurrency-to-another). RP2 splits them in two subcategories:
  * long-term capital gains (if supported by the country plugin: e.g. in the US it's 1 year or more), or
  * short-term capital gains otherwise;
* [STAKING](https://github.com/eprbell/rp2/blob/main/docs/user_faq.md#how-to-handle-income-from-staking): gains from staking;
* [WAGES](https://github.com/eprbell/rp2/blob/main/docs/user_faq.md#how-to-handle-income-from-crypto-wages): income from crypto wages.

For each of these categories RP2 generates an output spreadsheet with transaction details and computed gains/losses (see [Input and Output Files](https://github.com/eprbell/rp2/blob/main/README.md#input-and-output-files) for more details). Users can give this output to their tax preparer with the rest of their tax documentation (see also FAQ on [which tax forms to file](https://github.com/eprbell/rp2/blob/main/docs/user_faq.md#which-crypto-tax-forms-to-file)). Note that buying cryptocurrency using fiat currency is not a taxable event.

**NOTE ON NFTs**: Read the [FAQ on NFTs](https://github.com/eprbell/rp2/blob/main/docs/user_faq.md#how-to-handle-nfts) to learn about how RP2 treats NFTs.

## License
RP2 is released under the terms of Apache License Version 2.0. For more information see [LICENSE](https://github.com/eprbell/rp2/tree/main/LICENSE) or <http://www.apache.org/licenses/LICENSE-2.0>.

## Download
The latest version of RP2 can be downloaded at: <https://pypi.org/project/rp2/>

## Installation
RP2 has been tested on Ubuntu Linux, macOS and Windows 10 but it should work on all systems that have Python version 3.8.0 or greater.

### Installation on Ubuntu Linux
Open a terminal window and enter the following commands:
```
sudo apt-get update
sudo apt-get install python3 python3-pip
```

Then install RP2:
```
pip install rp2
```
### Installation on macOS
First make sure [Homebrew](https://brew.sh) is installed, then open a terminal window and enter the following commands:
```
brew update
brew install python3
```

Then install RP2:
```
pip install rp2
```
### Installation on Windows 10
First make sure [Python](https://python.org) 3.8 or greater is installed (in the Python installer window be sure to click on "Add Python to PATH"), then open a PowerShell window and enter the following:
```
pip install rp2
```

### Installation on Other Unix-like Systems
* install python 3.8 or greater
* install pip3

Then install RP2:
```
pip install rp2
```

## Running
RP2 requires two files as input:
* an ODS-format spreadsheet, containing crypto transactions (ODS-format files can be opened and edited with [LibreOffice](https://www.libreoffice.org/) and many other spreadsheet applications);
* a config file, describing the format of the spreadsheet file: what value each column corresponds to (e.g. timestamp, amount, exchange, fee, etc.) and which cryptocurrencies and exchanges to expect.

The two input files can either:
* be generated automatically using [DaLI](https://github.com/eprbell/dali-rp2), the data loader and input generator for RP2, or
* be prepared manually by the user.

The formats of these files are described in detail in the [Input Files](https://github.com/eprbell/rp2/blob/main/docs/input_files.md) section of the documentation.

Examples of an input spreadsheet and its respective config file:
* [input/crypto_example.ods](https://github.com/eprbell/rp2/tree/main/input/crypto_example.ods)
* [config/crypto_example.ini](https://github.com/eprbell/rp2/tree/main/config/crypto_example.ini).

After reading the input files, RP2 computes taxes and generates output files, which contain information on long/short capital gains, cost bases, balances, average price, in/out lot relationships and fractions, etc. They are described in detail in the [Output Files](https://github.com/eprbell/rp2/blob/main/docs/output_files.md) section of the documentation.

To try RP2 with example files, download [crypto_example.ods](https://github.com/eprbell/rp2/tree/main/input/crypto_example.ods) and [crypto_example.ini](https://github.com/eprbell/rp2/tree/main/config/crypto_example.ini). Let's call `<download_directory>` the location of the downloaded files.

The RP2 executable is country-dependent: `rp2_<country_code>`, where country code is a [ISO 3166-1 alpha-2](https://en.wikipedia.org/wiki/ISO_3166-1_alpha-2), 2-letter identifier (e.g. `rp2_us`, `rp2_jp`, etc).

To generate US tax output for the example files open a terminal window (or PowerShell if on Windows) and enter the following commands:
  ```
  cd <download_directory>
  rp2_us -m fifo -o output -p crypto_example_ crypto_example.ini crypto_example.ods
  ```
Results are generated in the `output` directory and logs are stored in the `log` directory.

The `-m` option is particularly important, because is selects the accounting method: `rp2_us` supports FIFO, LIFO and HIFO (if `-m` is not specified it defaults to FIFO).

To print full command usage information for the `rp2_us` command:
  ```
  rp2_us --help
  ```

## Input and Output Files
Read the [input files](https://github.com/eprbell/rp2/blob/main/docs/input_files.md) and [output files](https://github.com/eprbell/rp2/blob/main/docs/output_files.md) documentation.

## RP2 Ecosystem
This is a call for coders: come and help us expand RP2's functionality!

RP2 is the first component of what could be a powerful, community-driven suite of open-source, crypto tax software. It is intended as the core of a larger project ecosystem, maintained by the community. These projects can extend RP2's capability, usefulness and ease of use in new ways. For example:
* [DaLI data loader plugins](https://github.com/eprbell/dali-rp2/blob/main/README.dev.md#dali-internals): add support for more exchanges and wallets (via REST API and/or CSV files). [Dali](https://github.com/eprbell/dali-rp2), the RP2 data loader, uses them to generate an input ODS file and a config file that can be fed directly to RP2;
* [RP2 plugins](https://github.com/eprbell/rp2/tree/main/README.dev.md#plugin-development): RP2 can be expanded via its programmable plugin architecture, which enables support for new output generators, countries and accounting methods;
* RP2 GUI: make RP2 more user-friendly and accessible to people who are not familiar with the CLI;
* RP2 high-level interface: RP2 captures complex tax events using a few powerful, low-level primitives, aggregated in patterns. A higher level tool, might abstract out these patterns and present them to the user in a friendlier way (for example it may capture a complex concept like DeFi bridging as a primitive, rather than a pattern);
* more...

**Important note**: any RP2 ecosystem project must make user privacy its first priority.

If you'd like to start an ecosystem project, please open an [issue](https://github.com/eprbell/rp2/issues) to let the RP2 community know.

### List of Ecosystem Projects
Here's the current list of projects in the RP2 ecosystem:
* [DaLI](https://github.com/eprbell/dali-rp2): data loader and input generator for RP2 (https://github.com/eprbell/rp2).

## Reporting Bugs
Read the [Contributing](https://github.com/eprbell/rp2/tree/main/CONTRIBUTING.md#reporting-bugs) document.

## Contributing
Read the [Contributing](https://github.com/eprbell/rp2/tree/main/CONTRIBUTING.md) document.

## Developer Documentation
Read the [developer documentation](https://github.com/eprbell/rp2/tree/main/README.dev.md).

## Frequently Asked Questions
Read the [user FAQ list](https://github.com/eprbell/rp2/blob/main/docs/user_faq.md) and the [developer FAQ list](https://github.com/eprbell/rp2/blob/main/docs/developer_faq.md).

## Change Log
Read the [Change Log](https://github.com/eprbell/rp2/tree/main/CHANGELOG.md) document.


# Neo4j Graph Data Science Client

[![Latest version](https://img.shields.io/pypi/v/graphdatascience)](https://pypi.org/project/graphdatascience/)
[![PyPI downloads month](https://img.shields.io/pypi/dm/graphdatascience)](https://pypi.org/project/graphdatascience/)
![Python versions](https://img.shields.io/pypi/pyversions/graphdatascience)
[![Documentation](https://img.shields.io/badge/Documentation-latest-blue)](https://neo4j.com/docs/graph-data-science-client/current/)
[![Discord](https://img.shields.io/discord/787399249741479977?label=Chat&logo=discord)](https://discord.gg/neo4j)
[![Community forum](https://img.shields.io/website?down_color=lightgrey&down_message=offline&label=Forums&logo=discourse&up_color=green&up_message=online&url=https%3A%2F%2Fcommunity.neo4j.com%2F)](https://community.neo4j.com)
[![License](https://img.shields.io/pypi/l/graphdatascience)](https://www.apache.org/licenses/LICENSE-2.0)

`graphdatascience` is a Python client for operating and working with the [Neo4j Graph Data Science (GDS) library](https://github.com/neo4j/graph-data-science).
It enables users to write pure Python code to project graphs, run algorithms, as well as define and use machine learning pipelines in GDS.

The API is designed to mimic the GDS Cypher procedure API in Python code.
It abstracts the necessary operations of the [Neo4j Python driver](https://neo4j.com/docs/python-manual/current/) to offer a simpler surface.
Additionally, the client-specific graph, model, and pipeline objects offer convenient functions that heavily reduce the need to use Cypher to access and operate these GDS resources.

`graphdatascience` is only guaranteed to work with GDS versions 2.0+.

Please leave any feedback as issues on [the source repository](https://github.com/neo4j/graph-data-science-client).
Happy coding!

## Installation

To install the latest deployed version of `graphdatascience`, simply run:

```bash
pip install graphdatascience
```

## Getting started

To use the GDS Python Client, we need to instantiate a GraphDataScience object.
Then, we can project graphs, create pipelines, train models, and run algorithms.

```python
from graphdatascience import GraphDataScience

# Configure the driver with AuraDS-recommended settings
gds = GraphDataScience("neo4j+s://my-aura-ds.databases.neo4j.io:7687", auth=("neo4j", "my-password"), aura_ds=True)

# Import the Cora common dataset to GDS
G = gds.graph.load_cora()
assert G.node_count() == 2708

# Run PageRank in mutate mode on G
pagerank_result = gds.pageRank.mutate(G, tolerance=0.5, mutateProperty="pagerank")
assert pagerank_result["nodePropertiesWritten"] == G.node_count()

# Create a Node Classification pipeline
pipeline = gds.nc_pipe("myPipe")
assert pipeline.type() == "Node classification training pipeline"

# Add a Degree Centrality feature to the pipeline
pipeline.addNodeProperty("degree", mutateProperty="rank")
pipeline.selectFeatures("rank")
features = pipeline.feature_properties()
assert len(features) == 1
assert features[0]["feature"] == "rank"

# Add a training method
pipeline.addLogisticRegression(penalty=(0.1, 2))

# Train a model on G
model, train_result = pipeline.train(G, modelName="myModel", targetProperty="myClass", metrics=["ACCURACY"])
assert model.metrics()["ACCURACY"]["test"] > 0
assert train_result["trainMillis"] >= 0

# Compute predictions in stream mode
predictions = model.predict_stream(G)
assert len(predictions) == G.node_count()
```

The example here assumes using an AuraDS instance.
For additional examples and extensive documentation of all capabilities, please refer to the [GDS Python Client Manual](https://neo4j.com/docs/graph-data-science-client/current/).

Full end-to-end examples in Jupyter ready-to-run notebooks can be found in the [`examples` source directory](https://github.com/neo4j/graph-data-science-client/tree/main/examples):

* [Machine learning pipelines: Node classification](examples/ml-pipelines-node-classification.ipynb)
* [Node Regression with Subgraph and Graph Sample projections](examples/node-regression-with-subgraph-and-graph-sample.ipynb)
* [Product recommendations with kNN based on FastRP embeddings](examples/fastrp-and-knn.ipynb)
* [Sampling, Export and Integration with PyG example](examples/import-sample-export-gnn.ipynb)
* [Load data to a projected graph via graph construction](examples/load-data-via-graph-construction.ipynb)
* [Heterogeneous Node Classification with HashGNN and Autotuning](https://github.com/neo4j/graph-data-science-client/tree/main/examples/heterogeneous-node-classification-with-hashgnn.ipynb)
* [Perform inference using pre-trained KGE models](examples/kge-predict-transe-pyg-train.ipynb)

## Documentation

The primary source for learning everything about the GDS Python Client is the manual, hosted at https://neo4j.com/docs/graph-data-science-client/current/.
The manual is versioned to cover all GDS Python Client versions, so make sure to use the correct version to get the correct information.

## Known limitations

Operations known to not yet work with `graphdatascience`:

* [Numeric utility functions](https://neo4j.com/docs/graph-data-science/current/management-ops/utility-functions/#utility-functions-numeric) (will never be supported)
* [Cypher on GDS](https://neo4j.com/docs/graph-data-science/current/management-ops/create-cypher-db/) (might be supported in the future)
* [Projecting graphs using Cypher Aggregation](https://neo4j.com/docs/graph-data-science/current/management-ops/projections/graph-project-cypher-aggregation/) (might be supported in the future)

## License

`graphdatascience` is licensed under the Apache Software License version 2.0.
All content is copyright © Neo4j Sweden AB.

## Acknowledgements

This work has been inspired by the great work done in the following libraries:

* [pygds](https://github.com/stellasia/pygds) by stellasia
* [gds-python](https://github.com/moxious/gds-python) by moxious


.. _readme:

===============================================================
HydroMT: Automated and reproducible model building and analysis
===============================================================

|pypi| |conda forge| |docs_latest| |docs_stable| |binder| |codecov| |license| |doi| |joss_paper|

What is HydroMT?
----------------
**HydroMT** (Hydro Model Tools) is an open-source Python package that facilitates the process of
building and analyzing spatial geoscientific models with a focus on water system models.
It does so by automating the workflow to go from raw data to a complete model instance which
is ready to run and to analyse model results once the simulation has finished.
HydroMT builds on the latest packages in the scientific and geospatial python eco-system including
xarray_, rasterio_, rioxarray_, geopandas_, scipy_ and pyflwdir_.

Why HydroMT?
------------
Setting up spatial geoscientific models typically requires many (manual) steps
to process input data and might therefore be time consuming and hard to reproduce.
Especially improving models based on global geospatial datasets, which are
rapidly becoming available at increasingly high resolutions, might be challenging.
Furthermore, analyzing model schematization and results from different models,
which often use model-specific peculiar data formats, can be time consuming.
This package aims to make the model building process **fast**, **modular** and **reproducible**
by configuring the model building process from a single *ini* configuration file
and **model- and data-agnostic** through a common model and data interface.

How to use HydroMT?
-------------------
HydroMT can be used as a **command line** application (CLI) which provides commands to *build*,
*update* and *clip* models with a single line, or **from Python** to exploit its rich interface.
You can learn more about how to use HydroMT in its `online documentation. <https://deltares.github.io/hydromt/latest/>`_
For a smooth installing experience, we recommend installing HydroMT and its dependencies
from conda-forge in a clean environment, see `installation guide. <https://deltares.github.io/hydromt/latest/getting_started/installation>`_

HydroMT model plugins
---------------------
HydroMT is commonly used in combination with a **model plugin** which
provides a HydroMT implementation for specific model software. Using the plugins allows to prepare a ready-to-run set of input files from raw geoscientific datasets and analyse model results in a fast and reproducible way.
Known model plugins include:

* hydromt_wflow_: A framework for distributed rainfall-runoff (wflow_sbm) and sediment transport (wflow_sediment) modelling.
* hydromt_delwaq_: A framework for water quality (D-Water Quality) and emissions (D-Emissions) modelling.
* hydromt_sfincs_: A fast 2D hydrodynamic flood model (SFINCS).
* hydromt_fiat_: A flood impact model (FIAT).
* hydromt_delft3dfm_: A flexible mesh hydrodynamic suite for 1D2D and 2D3D modelling (Delft3D FM).

How to cite?
------------
For publications, please cite our JOSS paper |joss_paper|

::
    Eilander et al., (2023). HydroMT: Automated and reproducible model building and analysis. Journal of Open Source Software, 8(83), 4897, https://doi.org/10.21105/joss.04897

To cite a specific software version please use the DOI provided in the Zenodo badge |doi| that points to the latest release.

How to contribute?
-------------------
If you find any issues in the code or documentation feel free to leave an issue on the `github issue tracker. <https://github.com/Deltares/hydromt/issues>`_
You can find information about how to contribute to the HydroMT project at our `contributing page. <https://deltares.github.io/hydromt/latest/dev/contributing>`_

HydroMT seeks active contribution from the (hydro) geoscientific community.
So far, it has been developed and tested with a range of `Deltares <https://www.deltares.nl/en/>`_ models, but
we believe it is applicable to a much wider set of geoscientific models and are
happy to discuss how it can be implemented for your model.

.. _scipy: https://scipy.org/
.. _xarray: https://xarray.pydata.org
.. _geopandas: https://geopandas.org
.. _rioxarray: https://corteva.github.io/rioxarray/stable/
.. _rasterio: https://rasterio.readthedocs.io
.. _pyflwdir: https://deltares.github.io/pyflwdir
.. _hydromt_wflow: https://deltares.github.io/hydromt_wflow
.. _hydromt_sfincs: https://deltares.github.io/hydromt_sfincs
.. _hydromt_delwaq: https://deltares.github.io/hydromt_delwaq
.. _hydromt_fiat: https://deltares.github.io/hydromt_fiat
.. _hydromt_delft3dfm: https://deltares.github.io/hydromt_delft3dfm

.. |pypi| image:: https://img.shields.io/pypi/v/hydromt.svg?style=flat
    :alt: PyPI
    :target: https://pypi.org/project/hydromt/

.. |conda forge| image:: https://anaconda.org/conda-forge/hydromt/badges/version.svg
    :alt: Conda-Forge
    :target: https://anaconda.org/conda-forge/hydromt

.. |codecov| image:: https://codecov.io/gh/Deltares/hydromt/branch/main/graph/badge.svg?token=ss3EgmwHhH
    :alt: Coverage
    :target: https://codecov.io/gh/Deltares/hydromt

.. |docs_latest| image:: https://img.shields.io/badge/docs-latest-brightgreen.svg
    :alt: Latest developers docs
    :target: https://deltares.github.io/hydromt/latest

.. |docs_stable| image:: https://img.shields.io/badge/docs-stable-brightgreen.svg
    :target: https://deltares.github.io/hydromt/stable
    :alt: Stable docs last release

.. |binder| image:: https://mybinder.org/badge_logo.svg
    :alt: Binder
    :target: https://mybinder.org/v2/gh/Deltares/hydromt/main?urlpath=lab/tree/examples

.. |doi| image:: https://zenodo.org/badge/348020332.svg
    :alt: Zenodo
    :target: https://zenodo.org/badge/latestdoi/348020332

.. |license| image:: https://img.shields.io/github/license/Deltares/hydromt?style=flat
    :alt: License
    :target: https://github.com/Deltares/hydromt/blob/main/LICENSE

.. |joss_paper| image:: https://joss.theoj.org/papers/10.21105/joss.04897/status.svg
   :target: https://doi.org/10.21105/joss.04897


Finch
=====

|docs| |pypi| |ci| |license| |fossa| |gitter|

Finch (the bird)
  *Finch is a bird that can adapt to many different climates and ecological niches*

A Web Processing Service for Climate Indicators.

Documentation
-------------

Learn more about Finch in its official documentation at: https://pavics-sdi.readthedocs.io/projects/finch

Submit bug reports, questions and feature requests at: https://github.com/bird-house/finch/issues

Contributing
------------

You can find information about contributing in our `Developer Guide`_.

Please use bump-my-version_ to release a new version.

License
-------

* Free software: Apache Software License 2.0

Credits
-------

This package was created with Cookiecutter_ and the `bird-house/cookiecutter-birdhouse`_ project template.

.. _Cookiecutter: https://github.com/audreyr/cookiecutter
.. _`bird-house/cookiecutter-birdhouse`: https://github.com/bird-house/cookiecutter-birdhouse
.. _`Developer Guide`: https://pavics-sdi.readthedocs.io/projects/finch/en/latest/dev_guide.html
.. _bump-my-version: https://finch.readthedocs.io/en/latest/dev_guide.html#bump-a-new-version

.. |docs| image:: https://readthedocs.org/projects/finch/badge/?version=latest
    :target: https://pavics-sdi.readthedocs.io/projects/finch/en/latest/?badge=latest
    :alt: Documentation Status

.. |pypi| image:: https://img.shields.io/pypi/v/birdhouse-finch.svg
    :target: https://pypi.python.org/pypi/birdhouse-finch

.. |ci| image:: https://github.com/bird-house/finch/actions/workflows/main.yml/badge.svg
    :target: https://github.com/bird-house/finch/actions/workflows/main.yml
    :alt: GitHub CI

.. |fossa| image:: https://app.fossa.com/api/projects/git%2Bgithub.com%2Fbird-house%2Ffinch.svg?type=shield
    :target: https://app.fossa.com/projects/git%2Bgithub.com%2Fbird-house%2Ffinch?ref=badge_shield
    :alt: FOSSA License Scan

.. |license| image:: https://img.shields.io/github/license/bird-house/finch.svg
    :target: https://github.com/bird-house/finch/blob/master/LICENSE.txt
    :alt: GitHub license

.. |gitter| image:: https://badges.gitter.im/bird-house/birdhouse.svg
    :target: https://gitter.im/bird-house/birdhouse?utm_source=badge&utm_medium=badge&utm_campaign=pr-badge&utm_content=badge
    :alt: Join the chat at https://gitter.im/bird-house/birdhouse


<!-- SPDX-License-Identifier: MIT -->
[![PyPi - Version](https://img.shields.io/pypi/v/odxtools)](https://pypi.org/project/odxtools)
[![PyPI - License](https://img.shields.io/pypi/l/odxtools)](LICENSE)
[![CI Status](https://github.com/mercedes-benz/odxtools/actions/workflows/tests.yml/badge.svg?branch=main)](https://github.com/mercedes-benz/odxtools/actions?query=branch%3Amain)

# odxtools

`odxtools` is a set of utilities for working with diagnostic
descriptions of automotive electronic control units using the data
model and the associated technologies of the ODX standard.

[ODX](https://de.wikipedia.org/wiki/ODX) stands for "Open Diagnostic
data eXchange" and is primarily an XML based file format to describe
the diagnostic capabilities of the electronic control units (ECUs) of
complex distributed technical systems (usually cars and trucks). ODX
is an [open standard maintained by ASAM
e.V.](https://www.asam.net/standards/detail/mcd-2-d/) and is also
standardized internationally by
[ISO-22901](https://www.iso.org/standard/41207.html).

Usually, ODX is used to complement the
[UDS](https://en.wikipedia.org/wiki/Unified_Diagnostic_Services)
automotive diagnostics standard -- which itself can be considered to
be an extension of
[OBD-II](https://en.wikipedia.org/wiki/On-board_diagnostics#OBD-II) --
to provide a machine-processable description of the vendor-specific
diagnostics functionality of a vehicle's ECUs. That said, the
functionality which is described by ODX files neither needs to be a
super- nor a subset of OBD-II/UDS, e.g., ODX can be used to describe
diagnostic functionality that uses fundamentally different wire
formats and conventions than the ones mandated by OBD-II/UDS. (In
practice, the ODX-described functionality usually adheres to these
standards, though.)

The functionality provided by `odxtools` encompasses parsing and
internalizing ODX diagnostic database files as well as de- and
encoding the data of diagnostic requests and their responses
send to/received from ECUs in an pythonic manner.

## Table of Contents

- [Use Cases](#use-cases)
- [Installation](#installation)
- [Usage Examples](#usage-examples)
  - [Python snippets](#python-snippets)
- [Using the non-strict mode](#using-the-non-strict-mode)
- [Interactive Usage](#interactive-usage)
  - [Python REPL](#python-repl)
- [Command line usage](#command-line-usage)
  - [Generic parameters](#generic-parameters)
  - [The `list` subcommand](#the-list-subcommand)
  - [The `browse` subcommand](#the-browse-subcommand)
  - [The `snoop` subcommand](#the-snoop-subcommand)
  - [The `find` subcommand](#the-find-subcommand)
  - [The `decode` subcommand](#the-decode-subcommand)
  - [The `compare` subcommand](#the-compare-subcommand)
- [Testing](#testing)
- [Contributing](#contributing)
- [Code of Conduct](#code-of-conduct)
- [Provider Information](#provider-information)
- [Acknowledgements](#acknowledgements)
- [License](#license)

## Use Cases

Here are some of the intended use cases of `odxtools`:

- Prototype development: Interacting with the diagnostic services of
  electronic control units directly from python (requires taping into
  the car's relevant CAN or ethernet bus)
- End-of-production calibration/quality control: Initial set up and
  running a self diagnosis of newly produced cars to ensure that
  everything works as specified
- After-sales: Implementing servicing functionality for workshops, i.e.,
  defining test schedules based on the legally mandated functionality of
  ISO 15031-6 (OBD II) as well as manufacturer-specific routines
- Prototype development (II): Analyzing and debugging diagnostic sessions
  done using third-party software
- Prototype development (III): Implementing bridges to higher-level protocols
  such as HTTP
- Development for mass production: Accelerating the implementation of
  diagnostic servicesfor low-cost ECUs by using `odxtools`-based code
  generators for the diagnostic glue code on system-level languages like
  C++ or rust

Please be aware that some of the use cases listed above are currently
rather aspirational.

## Installation

The easiest way of installing `odxtools` on your system is via `pip`:

```bash
python3 -m pip install odxtools
```

If you want to develop `odxtools` itself, you need to install it from
source using `git`. The first step is to clone the repository:

```bash
cd $BASE_DIR
git clone https://github.com/mercedes-benz/odxtools
```

After this, make sure that all python dependencies are installed:

```bash
cd $BASE_DIR/odxtools
python3 -m pip install -e .
```

Next, you can optionally build a package and install it on the system:

```bash
cd $BASE_DIR/odxtools
python3 -m pip install --upgrade build
python3 -m build
sudo python3 -m pip install dist/odxtools-*.whl
```

Finally, update the `PYTHONPATH` environment variable and the newly
cloned module is ready to be used:

```bash
export PYTHONPATH="$BASE_DIR/odxtools:$PYTHONPATH"
```

Now, you can check whether the installation worked:

```bash
python3 -m odxtools list -a "$YOUR_PDX_FILE"
```

## Usage Examples

### Python snippets

- Load an ODX database from file `somersault.pdx`:

  ```python
  import odxtools

  db = odxtools.load_pdx_file("somersault.pdx")
  ```

- List the names of all available services of the `somersault_lazy` ECU:

  ```python
  # [...]

  ecu = db.ecus.somersault_lazy
  print(f"Available services for {ecu.short_name}: {ecu.services}")
  ```

- Determine the CAN IDs which the `somersault_lazy` ECU uses to send
  and receive diagnostic messages:

  ```python
  # [...]

  print(f"ECU {ecu.short_name} listens for requests on CAN ID 0x{ecu.get_can_receive_id():x}")
  print(f"ECU {ecu.short_name} transmits responses on CAN ID 0x{ecu.get_can_send_id():x}")
  ```

- Encode a `session_start` request to the `somersault_lazy` ECU:

  ```python
  # [...]

  raw_request_data = ecu.services.session_start()

  print(f"Message for session start request of ECU {ecu.short_name}: {raw_request_data}")
  # -> bytearray(b'\x10\x00')
  ```

- Print all mutable parameters of the `session_start` service's first
  positive response:

  ```python
  # [...]

  ecu.services.session_start.positive_responses[0].print_free_parameters_info()
  ```

- Encode the positive response to the `start_session` request:

  ```python
  # [...]

  raw_request_data = ecu.services.session_start()
  raw_response_data = ecu.services.session_start.positive_responses[0].encode(can_do_backward_flips="true", coded_request=raw_request_data)

  print(f"Positive response to session_start() of ECU {ecu.short_name}: {raw_response_data.hex(' ')}")
  # -> Positive response to session_start() of ECU somersault_lazy: 50 01
  ```

- Decode a request:

  ```python
  # [...]

  raw_data = b"\x10\x00"
  decoded_message = ecu.decode(raw_data)
  for x in decoded_message:
    print(f"decoded as '{x.coding_object.short_name}': {x.param_dict}")
  # -> decoded as 'start_session': {'sid': 16, 'id': 0}
  ```

- Decode a response to a request:

  ```python
  # [...]

  raw_request_data = bytes.fromhex("1000")
  raw_response_data = bytes.fromhex("5001")
  decoded_response = ecu.decode_response(raw_response_data, raw_request_data)
  for x in decoded_response:
    print(f"decoded as '{x.coding_object.short_name}': {x.param_dict}")
  # -> decoded as 'session': {'sid': 80, 'can_do_backward_flips': 'true'}
  ```

## Using the non-strict mode

By default, odxtools raises exceptions if it suspects that it cannot
fulfill a requested operation correctly. For example, if the dataset
it is instructed to load is detected to be not conformant with the ODX
specification, or if completing the operation requires missing
features of odxtools. To be able to deal with such cases, odxtools
provides a "non-strict" mode where such issues are ignored, but where
the results are undefined. The following snippet shows how to instruct
odxtools to load a non-conforming file in non-strict mode, and after
this is done, enables the safety checks again:

  ```python
  import odxtools

  [...]

  odxtools.exceptions.strict_mode = False
  botched_db = odxtools.load_file("my_non-conforming_database.pdx")
  odxtools.exceptions.strict_mode = True

  [...]
  ```

## Interactive Usage

### Python REPL

python's interactive read-reval-print-loop (REPL) supports
tab-completion on most plattforms, i.e., in this case, all data can be
conveniently interactivly discovered and this makes `odxtools` a very
convenient tool to explore the capabilities of a given ECU.

A notable exception is the Microsoft Windows platform: Most python
distribtions for Windows do not enable tab-completion by default in
their REPL.  For more convenience in such a scenario, we recommend
using
[ptpython](https://github.com/prompt-toolkit/ptpython/). `ptpython`
can be installed like any other python package, i.e., via `python3 -m
pip install ptpython`. Then, the REPL ought to be started using

```cmd
c:\odxtest>python3 "C:\Python39\Lib\site-packages\ptpython\entry_points\run_ptpython.py"
```

Alternatively, `pyreadline` can be used after installing it via
`python3 -m pip install pyreadline`.  With this, *basic*
tab-completion for python under Windows in [Interactive
Mode](https://docs.python.org/3/tutorial/interpreter.html#interactive-mode)
should work.

## Command line usage

Based the python module, `odxtools` also provides a set of command
line utilities for quick interactive explorations. Amongst others,
these utilities allow the inspection ODX/PDX files, snooping on
diagnostic sessions, etc. If `odxtools` is installed on a system-wide
basis, these commands can be invoked using `odxtools SUBCOMMAND
[PARAMS]`, if the repository has been manually cloned via `git` and
`odxtools` has not been installed on a system-wide basis, the way to
invoke these utilities is via `python3 -m odxtools SUBCOMMAND
[PARAMS]`.

### Generic parameters

Available generic parameters and a list of subcommands can be obtained
using `odxtools --help`:

```bash
$ odxtools --help
usage: odxtools [-h] [--version] {list,browse,snoop,find,decode,compare} ...

Utilities to interact with automotive diagnostic descriptions based on the ODX standard.

Examples:
  For printing all services use:
   odxtools list ./path/to/database.pdx --services
  For browsing the data base and encoding messages use:
   odxtools browse ./path/to/database.pdx

positional arguments:
  {list,browse,snoop,find,decode,compare}
                        Select a sub command
    list                Print a summary of automotive diagnostic files.
    browse              Interactively browse the content of automotive diagnostic files.
    snoop               Live decoding of a diagnostic session.
    find                Find & display services by their name
    decode              Find & print service by hex-data. Can also decode the hex-data to its named parameters.
    compare             Compares two ecu versions or databases with each other. Checks whether diagnostic services and its parameters have changed.

optional arguments:
  -h, --help            show this help message and exit
  --version             Print the odxtools version
```

All subcommands accept the `--help` parameter:

```bash
$ odxtools list --help
usage: odxtools list [-h] [-v VARIANT [VARIANT ...]] [-s [SERVICE [SERVICE ...]]] [-p] [-d] [-a] PDX_FILE
[...]
```

It follows is an inexhaustive list of the subcommands that are
currently available:

### The `list` subcommand

The `list` subcommand is used to parse a `.pdx` database file and
print the relevant parts of its content to the terminal.

```bash
$ odxtools list -h
usage: odxtools list [-h] [-v VARIANT [VARIANT ...]] [-g] [-s [SERVICE [SERVICE ...]]] [-p] [-d] [-a] [-po] PDX_FILE

List the content of automotive diagnostic files (*.pdx)

Examples:
  For displaying only the names of the diagnostic layers use:
    odxtools list ./path/to/database.pdx
  For displaying all content use:
    odxtools list ./path/to/database.pdx --all
  For more information use:
    odxtools list -h

positional arguments:
  PDX_FILE              path to the .pdx file

optional arguments:
  -h, --help            show this help message and exit
  -v VARIANT [VARIANT ...], --variants VARIANT [VARIANT ...]
                        Specifies which variants should be included.
  -g, --global-negative-responses
                        Print a list of the global negative responses for the selected ECUs.
  -s [SERVICE [SERVICE ...]], --services [SERVICE [SERVICE ...]]
                        Print a list of diagnostic services specified in the pdx.
                        If no service names are specified, all services are printed.
  -p, --params          Print a list of all parameters relevant for the selected items.
  -d, --dops            Print a list of all data object properties relevant for the selected items
  -a, --all             Print a list of all diagnostic services and DOPs specified in the pdx
  -po, --plumbing-output
                        Print full objects instead of selected and formatted attributes
```

The options `--variants` and `--services` can be used to specify which services should be printed.
If the `--params` option is specified, the message layout and information about the service parameters (reuest as well as responses) are printed for all specified variants/services.
If the `--global-negative-responses` option is specified, all global negative responses are printed for all specified variants.
If the `--dops` option is specified, a list of all data object properties (their names) is printed for all specified variants/services.
With the parameter `--all` all data of the file that is recognized by `odxtools` is printed.
The default output does not display all information of the specified objects but a selection. To see all object information choose the parameter `--plumbing-output`.

Example:

```bash
$ odxtools list $BASE_DIR/odxtools/examples/somersault.pdx --variants somersault_lazy --services do_forward_flips --params
Overview of diagnostic layers:
    | Name            | Variant Type | Num. of Services | Num. of DOPs | Num. of comparams
----+-----------------+--------------+------------------+--------------+-------------------
  0 | somersault_lazy | ECU-VARIANT  |                5 |           10 |                10

Diagnostic layer: 'somersault_lazy'
 Variant Type: ECU-VARIANT
 Description: Sloppy variant of the somersault ECU (lazy < assiduous)

The services of 'somersault_lazy' are:
 do_forward_flips <ID: OdxLinkId('somersault.service.do_forward_flips')>
  Service description: Do a forward flip.

  Request Properties:
   Request Name: do_forward_flips
   Byte-Array: ---      Hex-String: 0x---
   Service Parameters: [sid, forward_soberness_check, num_flips]

    | Name                    | Byte Pos. | Bit Length | Param. Type | Data Type | Value | Value Desc. | Linked DOP
----+-------------------------+-----------+------------+-------------+-----------+-------+-------------+-----------------
  0 | sid                     |         0 |          8 | CODED-CONST | A_UINT32  | 0xBA  | coded value |
  1 | forward_soberness_check |         1 |          8 | VALUE       | A_UINT32  |       |             | soberness_check
  2 | num_flips               |         2 |          8 | VALUE       | A_UINT32  |       |             | num_flips

  Message format of a request:
           7     6     5     4     3     2     1     0
        +-----+-----+-----+-----+-----+-----+-----+-----+
      0 | sid(8 bits)                                   |
        +-----+-----+-----+-----+-----+-----+-----+-----+
      1 | forward_soberness_check(8 bits)               |
        +-----+-----+-----+-----+-----+-----+-----+-----+
      2 | num_flips(8 bits)                             |
        +-----+-----+-----+-----+-----+-----+-----+-----+

  Positive Response Properties:
   Number of Positive Responses: 1
   Positive Responses: [grudging_forward]
   Service Parameters: [sid, num_flips_done]

    | Name           | Byte Pos. | Bit Length | Parameter Type         | Data Type | Value | Value Desc. | Linked DOP
----+----------------+-----------+------------+------------------------+-----------+-------+-------------+-------------
  0 | sid            |         0 |          8 | CODED-CONST            | A_UINT32  | 0xFA  | coded value |
  1 | num_flips_done |         1 |          8 | MATCHING-REQUEST-PARAM |           |       |             |

  Message format of a positive response:
           7     6     5     4     3     2     1     0
        +-----+-----+-----+-----+-----+-----+-----+-----+
      0 | sid(8 bits)                                   |
        +-----+-----+-----+-----+-----+-----+-----+-----+
      1 | num_flips_done(8 bits)                        |
        +-----+-----+-----+-----+-----+-----+-----+-----+

  Negative Response Properties:
   Number of Negative Responses: 1
   Negative Responses: [flips_not_done]
   Service Parameters: [sid, rq_sid, reason, flips_successfully_done]

    | Name                    |Byte Pos. | Bit Length | Parameter Type         | Data Type | Value     | Value Desc. | Linked DOP
----+-------------------------+----------+------------+------------------------+-----------+-----------+-------------+-------------
  0 | sid                     |        0 |          8 | CODED-CONST            | A_UINT32  | 0x7F      | coded value |
  1 | rq_sid                  |        1 |          8 | MATCHING-REQUEST-PARAM |           |           |             |
  2 | reason                  |        2 |          8 | NRC-CONST              | A_UINT32  | [0, 1, 2] | coded value |
  3 | flips_successfully_done |        3 |          8 | VALUE                  | A_UINT32  |           |             | num_flips

  Number of negative responses: 1
  Message format of a negative response:
           7     6     5     4     3     2     1     0
        +-----+-----+-----+-----+-----+-----+-----+-----+
      0 | sid(8 bits)                                   |
        +-----+-----+-----+-----+-----+-----+-----+-----+
      1 | rq_sid(8 bits)                                |
        +-----+-----+-----+-----+-----+-----+-----+-----+
      2 | reason(8 bits)                                |
        +-----+-----+-----+-----+-----+-----+-----+-----+
      3 | flips_successfully_done(8 bits)               |
        +-----+-----+-----+-----+-----+-----+-----+-----+

```

### The `browse` subcommand

The `browse` subcommand uses
[InquirerPy](https://github.com/kazhala/InquirerPy) to interactively
navigate through the database of a `.pdx` file. For example, using the
`browse` subcommand you can select the ECU and service without
spamming the terminal:

```bash
$ odxtools browse $BASE_DIR/odxtools/examples/somersault.pdx
? Select a Variant.  somersault_lazy
ECU-VARIANT 'somersault_lazy' (Receive ID: 0x7b, Send ID: 0x1c8)
? The variant somersault_lazy offers the following services. Select one!  do_forward_flips
? This service offers the following messages.  Request: do_forward_flips
             7     6     5     4     3     2     1     0
          +-----+-----+-----+-----+-----+-----+-----+-----+
        0 | sid(8 bits)                                   |
          +-----+-----+-----+-----+-----+-----+-----+-----+
        1 | forward_soberness_check(8 bits)               |
          +-----+-----+-----+-----+-----+-----+-----+-----+
        2 | num_flips(8 bits)                             |
          +-----+-----+-----+-----+-----+-----+-----+-----+
     Parameter(short_name='sid', type='CODED-CONST', semantic=None, byte_position=0, bit_length=8, coded_value='0xba')
     Parameter(short_name='forward_soberness_check', type='VALUE', semantic=None, byte_position=1, bit_length=8, dop_ref='somersault.DOP.soberness_check')
      DataObjectProperty('soberness_check', category='LINEAR', internal_type='A_UINT32', physical_type='A_UINT32')
     Parameter(short_name='num_flips', type='VALUE', semantic=None, byte_position=2, bit_length=8, dop_ref='somersault.DOP.num_flips')
      DataObjectProperty('num_flips', category='LINEAR', internal_type='A_UINT32', physical_type='A_UINT32')
[...]
```

### The `snoop` subcommand

The `snoop` subcommand can be used to decode a trace of a or a
currently running diagnostic session.

```bash
$ odxtools snoop -h
usage: odxtools snoop [-h] [--active] [--channel CHANNEL] [--rx RX] [--tx TX] [--variant VARIANT]
                      [--protocol PROTOCOL]
                      PDX_FILE

Live decoding of a diagnostic session.

positional arguments:
  PDX_FILE              path to the .pdx file

options:
  -h, --help            show this help message and exit
  --active, -a          Active mode, sends flow control messages to receive ISO-TP telegrams successfully
  --channel CHANNEL, -c CHANNEL
                        CAN interface name to be used (required in active mode)
  --rx RX, -r RX        CAN ID in which the ECU listens for diagnostic messages
  --tx TX, -t TX        CAN ID in which the ECU sends replys to diagnostic messages  (required in active mode)
  --variant VARIANT, -v VARIANT
                        Name of the ECU variant which the decode process ought to be based on
  --protocol PROTOCOL, -p PROTOCOL
                        Name of the protocol used for decoding
```
Example:
```bash
# create a socketcan `vcan0` interface
sudo ip link add dev vcan0 type vcan
sudo ip link set vcan0 up

# start the snooping on vcan0
odxtools snoop -c vcan0 --variant "somersault_lazy" $BASE_DIR/odxtools/examples/somersault.pdx

# on a different terminal, run the diagnostic session
$BASE_DIR/odxtools/examples/somersaultlazy.py -c vcan0
```

The snoop command will then output the following:

```bash
$ odxtools snoop -c vcan0 --variant "somersault_lazy" $BASE_DIR/odxtools/examples/somersault.pdx
Decoding messages on channel vcan0
Tester: do_forward_flips(forward_soberness_check=18, num_flips=1)
 -> 7fba7f (bytearray(b'\x7f\xba\x7f'), 3 bytes)
Tester: start_session()
 -> session()
Tester: do_forward_flips(forward_soberness_check=18, num_flips=1)
 -> grudging_forward(num_flips_done=bytearray(b'\x01'))
Tester: do_forward_flips(forward_soberness_check=35, num_flips=1)
 -> flips_not_done(rq_sid=bytearray(b'\xba'), reason=0, flips_successfully_done=0)
Tester: do_forward_flips(forward_soberness_check=18, num_flips=3)
 -> grudging_forward(num_flips_done=bytearray(b'\x03'))
Tester: do_forward_flips(forward_soberness_check=18, num_flips=50)
 -> flips_not_done(rq_sid=bytearray(b'\xba'), reason=1, flips_successfully_done=6)
```

### The `find` subcommand

The `find` subcommand can be used to find a service and its associated information by a partial name via cli.

```bash
$ odxtools find -h
usage: odxtools find [-h] [-v VARIANT] -s [SERVICES ...] [-nd] [-ro] [-po] PDX_FILE

Find & print services by name

Examples:
  For displaying the services associated with the partial name 'Reset' without details:
    odxtools find ./path/to/database.pdx -s "Reset" --no-details
  For more information use:
    odxtools find -h

positional arguments:
  PDX_FILE              Location of the .pdx file

options:
  -h, --help            show this help message and exit
  -v VARIANT, --variants VARIANT
                        Specifies which ecu variants should be included.
  -s [SERVICES ...], --service-names [SERVICES ...]
                        Print a list of diagnostic services partially matching given service names
  -nd, --no-details     Don't show all service details
  -ro, --relaxed-output
                        Relax output formatting rules (allow unknown bitlengths for ascii representation)
  -po, --plumbing-output
                        Print full objects instead of selected and formatted attributes
```

Example: Find diagnostic services with the name `session_start`

```bash
$ odxtools find $BASE_DIR/odxtools/examples/somersault.pdx -s session_start

=====================================
somersault_lazy, somersault_assiduous
=====================================

 session_start <ID: OdxLinkId('somersault.service.session_start')>

  Request Properties:
   Request Name: start_session
   Byte-Array: bytearray(b'\x10\x00')   Hex-String: 0x1000
   Service Parameters: [sid, id]

    | Name | Byte Pos. | Bit Length | Param. Type | Data Type | Value | Value Desc. | Linked DOP
----+------+-----------+------------+-------------+-----------+-------+-------------+------------
  0 | sid  |         0 |          8 | CODED-CONST | A_UINT32  | 0x10  | coded value |
  1 | id   |         1 |          8 | CODED-CONST | A_UINT32  | 0x00  | coded value |

  Message format of a request:
           7     6     5     4     3     2     1     0
        +-----+-----+-----+-----+-----+-----+-----+-----+
      0 | sid (8 bits)                                  |
        +-----+-----+-----+-----+-----+-----+-----+-----+
      1 | id (8 bits)                                   |
        +-----+-----+-----+-----+-----+-----+-----+-----+

  Positive Response Properties:
   Number of Positive Responses: 1
   Positive Responses: [session]
   Service Parameters: [sid, can_do_backward_flips]

    | Name                  | Byte Pos. | Bit Length | Param. Type | Data Type        | Value | Value Desc. | Linked DOP
----+-----------------------+-----------+------------+-------------+------------------+-------+-------------+------------
  0 | sid                   |         0 |          8 | CODED-CONST | A_UINT32         | 0x50  | coded value |
  1 | can_do_backward_flips |         1 |          8 | VALUE       | A_UNICODE2STRING |       |             | boolean

  Message format of a positive response:
           7     6     5     4     3     2     1     0
        +-----+-----+-----+-----+-----+-----+-----+-----+
      0 | sid (8 bits)                                  |
        +-----+-----+-----+-----+-----+-----+-----+-----+
      1 | can_do_backward_flips (8 bits)                |
        +-----+-----+-----+-----+-----+-----+-----+-----+

  Negative Response Properties:
   Number of Negative Responses: 1
   Negative Responses: [general_negative_response]
   Service Parameters: [sid, rq_sid, response_code]

    | Name          | Byte Pos.| Bit Length | Parameter Type         | Data Type | Value | Value Desc. | Linked DOP
----+---------------+----------+------------+------------------------+-----------+-------+-------------+------------
  0 | sid           |        0 |          8 | CODED-CONST            | A_UINT32  | 0x7F  | coded value |
  1 | rq_sid        |        1 |          8 | MATCHING-REQUEST-PARAM |           |       |             |
  2 | response_code |        2 |          8 | VALUE                  | A_UINT32  |       |             | error_code

  Message format of a negative response:
           7     6     5     4     3     2     1     0
        +-----+-----+-----+-----+-----+-----+-----+-----+
      0 | sid (8 bits)                                  |
        +-----+-----+-----+-----+-----+-----+-----+-----+
      1 | rq_sid (8 bits)                               |
        +-----+-----+-----+-----+-----+-----+-----+-----+
      2 | response_code (8 bits)                        |
        +-----+-----+-----+-----+-----+-----+-----+-----+

```

### The `decode` subcommand

The `decode` subcommand can be used to decode hex-data to a service, and its associated
parameters.

```bash
$ odxtools decode -h
usage: odxtools decode [-h] [-v VARIANT] -d DATA [-D] PDX_FILE

Decode request by hex-data

Examples:
  For displaying the service associated with the request 10 01 & decoding it:
    odxtools decode ./path/to/database.pdx -D -d '10 01'
  For displaying the service associated with the request 10 01, without decoding it:
    odxtools decode ./path/to/database.pdx -d '10 01'
  For more information use:
    odxtools decode -h

positional arguments:
  PDX_FILE              Location of the .pdx file

options:
  -h, --help            show this help message and exit
  -v VARIANT, --variants VARIANT
                        Specifies which ecu variants should be included.
  -d DATA, --data DATA  Specify data of hex request
  -D, --decode          Decode the given hex data
```

Example: Decode diagnostic services with the request `10 00`

```bash
$ odxtools decode $BASE_DIR/odxtools/examples/somersault.pdx -d '10 00'
Binary data: 10 00
Decoded by service 'session_start' (decoding ECUs: somersault_lazy, somersault_assiduous)
```

Example: Decode diagnostic services with the request `10 00`, and parameters

```bash
$ odxtools decode $BASE_DIR/odxtools/examples/somersault.pdx -d '10 00' -D
Binary data: 10 00
Decoded by service 'session_start' (decoding ECUs: somersault_lazy, somersault_assiduous)
Decoded data:
  sid=16 (0x10)
  id=0 (0x0)
```

### The `compare` subcommand

The `compare` subcommand can be used to compare databases (pdx-files) and diagnostic layers with each other. All diagnostic services as well as its parameters of specified databases and variants are compared with each other and changes are displayed.

#### database comparison:
- new diagnostic layers
- deleted diagnostic layers
- diagnostic layer comparison

#### diagnostic layer comparison:
- new services
- deleted services
- renamed services
- service parameter comparison

#### service parameter comparison:
find changes in following properties:
- Name
- Byte Position
- Bit Length
- Semantic
- Parameter Type
- Coded Value
- Data Type
- Data Object Property (Name, Data Type, Bit Length, Default Value)

```bash
$ odxtools compare -h
usage: odxtools compare [-h] [-v VARIANT [VARIANT ...]] [-db DATABASE [DATABASE ...]] [-nd] [-od] PDX_FILE

Compares two ecu versions or databases with each other. Checks whether diagnostic services and its parameters have changed.

Examples:
  Comparison of two ecu versions:
    odxtools compare ./path/to/database.pdx -v variant1 variant2
  Comparison of two database versions:
    odxtools compare ./path/to/database.pdx -db ./path/to/old-database.pdx
  For more information use:
    odxtools compare -h

positional arguments:
  PDX_FILE              Location of the .pdx file

options:
  -h, --help            show this help message and exit
  -v VARIANT [VARIANT ...], --variants VARIANT [VARIANT ...]
                        Compare specified ecu variants to each other.
  -db DATABASE [DATABASE ...], --database DATABASE [DATABASE ...]
                        Compare specified database file(s) to database file of first input argument.
  -nd, --no-details     Don't show all service parameter details
  -od, --object-details
                        Print all object details instead of just the name
```

Example: Compare the ecu variants `somersault_lazy` and `somersault_assiduous`

```bash
$ odxtools compare $BASE_DIR/odxtools/examples/somersault.pdx -v somersault_lazy somersault_assiduous

Overview of diagnostic layers:
    | Name                 | Variant Type | Num. of Services | Num. of DOPs | Num. of comparams
----+----------------------+--------------+------------------+--------------+-------------------
  0 | somersault_lazy      | ECU-VARIANT  |                5 |           10 |                10
  1 | somersault_assiduous | ECU-VARIANT  |                8 |           10 |                10

Changes in ecu variant somersault_lazy
 (compared to somersault_assiduous)

 Changed diagnostic services for ecu variant: somersault_lazy

 Deleted services
    | Name                 | Semantic   | Hex-Request
----+----------------------+------------+---------------
  0 | set_operation_params | FUNCTION   |
  1 | do_backward_flips    | FUNCTION   |
  2 | headstand            | UNKNOWN    |
```

Example: Compare two databases

```bash
$ odxtools compare $BASE_DIR/odxtools/examples/somersault_modified.pdx -db $BASE_DIR/odxtools/examples/somersault.pdx -nd

Changes in file somersault_modified.pdx
 (compared to somersault.pdx)

Overview of diagnostic layers (for somersault_modified.pdx)
    | Name                 | Variant Type | Num. of Services | Num. of DOPs | Num. of comparams
----+----------------------+--------------+------------------+--------------+-------------------
  0 | somersault           | BASE-VARIANT |                6 |           10 |                10
  1 | somersault_lazy      | ECU-VARIANT  |                5 |           10 |                10
  2 | somersault_assiduous | ECU-VARIANT  |                8 |           10 |                10
  3 | somersault_young     | ECU-VARIANT  |                6 |           10 |                10

Overview of diagnostic layers (for somersault.pdx)
    | Name                 | Variant Type | Num. of Services | Num. of DOPs | Num. of comparams
----+----------------------+--------------+------------------+--------------+-------------------
  0 | somersault           | BASE-VARIANT |                7 |           10 |                10
  1 | somersault_lazy      | ECU-VARIANT  |                5 |           10 |                10
  2 | somersault_assiduous | ECU-VARIANT  |                8 |           10 |                10

 Changed ecu variants:
         New ecu variants:
                         somersault_young
         Deleted ecu variants:

 Changed diagnostic services for ecu variant: somersault_lazy

 Renamed services
    | Name          | Semantic   | Hex-Request   | Old service name
----+---------------+------------+---------------+--------------------
  0 | session_start | SESSION    | 0x1000        | start_session

 Services with parameter changes
    | Name           | Semantic      | Hex-Request | Changed parameters
----+----------------+---------------+-------------+-----------------------------------------------------------------------------------------------
  0 | session_start  | SESSION       | 0x1000      | positive response parameter 'can_do_backward_flips',
  1 | session_stop   | SESSION       | 0x1001      | request parameter 'id', positive response parameter 'can_do_backward_flips',
  2 | tester_present | TESTERPRESENT | 0x3E00      | request parameter 'id', positive response parameter 'status',
  3 | report_status  | CURRENTDATA   | 0x2200      | positive response parameter 'dizzyness_level', positive response parameter 'happiness_level',

 Detailed changes of diagnostic service session_start

Properties of 2. positive response parameter can_do_backward_flips have changed
    | Property      | Old Value   | New Value
----+---------------+-------------+------------------
  0 | Linked DOP    | uint8       | boolean
  1 | DOP data type | A_UINT32    | A_UNICODE2STRING

 Detailed changes of diagnostic service session_stop

Properties of 2. request parameter id have changed
    | Property   |   Old Value |   New Value
----+------------+-------------+-------------
  0 | Bit Length |          16 |           8

Properties of 2. positive response parameter can_do_backward_flips have changed
    | Property      | Old Value   | New Value
----+---------------+-------------+------------------
  0 | Linked DOP    | uint8       | boolean
  1 | DOP data type | A_UINT32    | A_UNICODE2STRING

 Detailed changes of diagnostic service tester_present

Properties of 2. request parameter id have changed
    | Property   | Old Value   | New Value
----+------------+-------------+-------------
  0 | Value      | 0x01        | 0x00

Properties of 2. positive response parameter status have changed
    | Property   | Old Value   | New Value
----+------------+-------------+-------------
  0 | Bit Length | 16          | 8
  1 | Value      | 0x0043      | 0x00

 Detailed changes of diagnostic service report_status

Properties of 2. positive response parameter dizzyness_level have changed
    | Property       | Old Value       | New Value
----+----------------+-----------------+-----------------
  0 | Parameter name | happiness_level | dizzyness_level
  1 | Linked DOP     | happiness_level | dizzyness_level

Properties of 3. positive response parameter happiness_level have changed
    | Property       | Old Value       | New Value
----+----------------+-----------------+-----------------
  0 | Parameter name | dizzyness_level | happiness_level
  1 | Linked DOP     | dizzyness_level | happiness_level

 Changed diagnostic services for ecu variant: somersault_assiduous

 New services
    | Name     | Semantic   | Hex-Request
----+----------+------------+---------------
  0 | flicflac | FUNCTION   |

 Deleted services
    | Name                 | Semantic   | Hex-Request
----+----------------------+------------+---------------
  0 | set_operation_params | FUNCTION   |

 Renamed services
    | Name          | Semantic   | Hex-Request   | Old service name
----+---------------+------------+---------------+--------------------
  0 | session_start | SESSION    | 0x1000        | start_session

 Services with parameter changes
    | Name              | Semantic      | Hex-Request | Changed parameters
----+-------------------+---------------+-------------+---------------------------------------------------------------------------------------------------------------------------------------------
  0 | session_start     | SESSION       | 0x1000      | positive response parameter 'can_do_backward_flips',
  1 | session_stop      | SESSION       | 0x1001      | request parameter 'id', positive response parameter 'can_do_backward_flips',
  2 | tester_present    | TESTERPRESENT | 0x3E00      | request parameter 'id', positive response parameter 'status',
  3 | do_backward_flips | FUNCTION      |             | request parameter 'backward_soberness_check', positive response parameter 'num_flips_done', positive response parameter 'grumpiness_level',
  4 | report_status     | CURRENTDATA   | 0x2200      | positive response parameter 'dizzyness_level', positive response parameter 'happiness_level',
  5 | headstand         | UNKNOWN       |             | request parameter list,

 Detailed changes of diagnostic service session_start

Properties of 2. positive response parameter can_do_backward_flips have changed
    | Property      | Old Value   | New Value
----+---------------+-------------+------------------
  0 | Linked DOP    | uint8       | boolean
  1 | DOP data type | A_UINT32    | A_UNICODE2STRING

 Detailed changes of diagnostic service session_stop

Properties of 2. request parameter id have changed
    | Property   |   Old Value |   New Value
----+------------+-------------+-------------
  0 | Bit Length |          16 |           8

Properties of 2. positive response parameter can_do_backward_flips have changed
    | Property      | Old Value   | New Value
----+---------------+-------------+------------------
  0 | Linked DOP    | uint8       | boolean
  1 | DOP data type | A_UINT32    | A_UNICODE2STRING

 Detailed changes of diagnostic service tester_present

Properties of 2. request parameter id have changed
    | Property   | Old Value   | New Value
----+------------+-------------+-------------
  0 | Value      | 0x01        | 0x00

Properties of 2. positive response parameter status have changed
    | Property   | Old Value   | New Value
----+------------+-------------+-------------
  0 | Bit Length | 16          | 8
  1 | Value      | 0x0043      | 0x00

 Detailed changes of diagnostic service do_backward_flips

Properties of 2. request parameter backward_soberness_check have changed
    | Property       | Old Value      | New Value
----+----------------+----------------+--------------------------
  0 | Parameter name | backward_check | backward_soberness_check

Properties of 2. positive response parameter num_flips_done have changed
    | Property      |   Old Value | New Value
----+---------------+-------------+-------------
  0 | Byte position |           1 |

Properties of 3. positive response parameter grumpiness_level have changed
    | Property       | Old Value   | New Value
----+----------------+-------------+-------------
  0 | Byte position  | 2           |
  1 | Parameter type | CODED-CONST | VALUE

 Detailed changes of diagnostic service report_status

Properties of 2. positive response parameter dizzyness_level have changed
    | Property       | Old Value       | New Value
----+----------------+-----------------+-----------------
  0 | Parameter name | happiness_level | dizzyness_level
  1 | Linked DOP     | happiness_level | dizzyness_level

Properties of 3. positive response parameter happiness_level have changed
    | Property       | Old Value       | New Value
----+----------------+-----------------+-----------------
  0 | Parameter name | dizzyness_level | happiness_level
  1 | Linked DOP     | dizzyness_level | happiness_level

 Detailed changes of diagnostic service headstand

List of request parameters for service headstand is not identical.
    | List     | Values
----+----------+---------------------
  0 | Old list | [sid, id, duration]
  1 | New list | [sid, duration]
```

## Testing

The included unit tests can be run via

```bash
python -m unittest tests/test_*.py
```

The static type checker can be run via
```bash
python3 -m mypy --ignore-missing-imports odxtools
```

## Contributing

We welcome any contributions.  If you want to contribute to this
project, please read the [contributing guide](https://github.com/mercedes-benz/odxtools/blob/main/CONTRIBUTING.md).

## Code of Conduct

Please read our [Code of Conduct](https://github.com/mercedes-benz/foss/blob/master/CODE_OF_CONDUCT.md)
as it is our base for interaction.

## Provider Information

Please visit <https://mbition.io/en/home/index.html> for information on the provider.

Notice: Before you use the program in productive use, please take all necessary precautions,
e.g. testing and verifying the program with regard to your specific use.
The program was tested solely for our own use cases, which might differ from yours.

## Acknowledgements

This work includes research of the project
[SofDCar](https://sofdcar.de/) (19S21002), which is funded by the
[German Federal Ministry for Economic Affairs and
Climate Action](https://www.bmwk.de/).

## License

This project is licensed under the [MIT LICENSE](LICENSE).


# DANDI Client

[![Tests](https://github.com/dandi/dandi-cli/workflows/Tests/badge.svg)](https://github.com/dandi/dandi-cli/actions?query=workflow%3ATests)
[![codecov.io](https://codecov.io/github/dandi/dandi-cli/coverage.svg?branch=master)](https://codecov.io/github/dandi/dandi-cli?branch=master)
[![Conda](https://anaconda.org/conda-forge/dandi/badges/version.svg)](https://anaconda.org/conda-forge/dandi)
[![Gentoo (::science)](https://repology.org/badge/version-for-repo/gentoo_ovl_science/dandi-cli.svg?header=Gentoo%20%28%3A%3Ascience%29)](https://repology.org/project/dandi-cli/versions)
[![GitHub release](https://img.shields.io/github/release/dandi/dandi-cli.svg)](https://GitHub.com/dandi/dandi-cli/releases/)
[![PyPI version fury.io](https://badge.fury.io/py/dandi.svg)](https://pypi.python.org/pypi/dandi/)
[![Documentation Status](https://readthedocs.org/projects/dandi/badge/?version=latest)](https://dandi.readthedocs.io/en/latest/?badge=latest)

The [DANDI Python client](https://pypi.org/project/dandi/) allows you to:

* Download `Dandisets` and individual subject folders or files
* Validate data to locally conform to standards
* Organize your data locally before upload
* Upload `Dandisets`
* Interact with the DANDI archive's web API from Python
* Delete data in the DANDI archive
* Perform other auxiliary operations with data or the DANDI archive

**Note**: This project is under heavy development. See [the issues log](https://github.com/dandi/dandi-cli/issues) or
[Work-in-Progress (WiP)](https://github.com/dandi/dandi-cli/pulls).

## Installation

DANDI Client releases are [available from PyPI](https://pypi.org/project/dandi)
and [conda-forge](https://anaconda.org/conda-forge/dandi).  Install them in your Python (native, virtualenv, or
conda) environment via

    pip install dandi

or

    conda install -c conda-forge dandi

## CLI Tool

This package provides a command line utility with a basic interface
to help you prepare and upload your data to, or obtain data from, the [DANDI archive](http://dandiarchive.org).

```bash
$> dandi
Usage: dandi [OPTIONS] COMMAND [ARGS]...

  A client to support interactions with DANDI archive
  (http://dandiarchive.org).

  To see help for a specific command, run

      dandi COMMAND --help

  e.g. dandi upload --help

Options:
  --version
  -l, --log-level [DEBUG|INFO|WARNING|ERROR|CRITICAL]
                                  Log level (case insensitive).  May be
                                  specified as an integer.  [default: INFO]
  --pdb                           Fall into pdb if errors out
  --help                          Show this message and exit.

Commands:
  delete            Delete dandisets and assets from the server.
  digest            Calculate file digests
  download          Download a file or entire folder from DANDI.
  instances         List known Dandi Archive instances that the CLI can...
  ls                List .nwb files and dandisets metadata.
  move              Move or rename assets in a local Dandiset and/or on...
  organize          (Re)organize NWB files according to their metadata.
  shell-completion  Emit shell script for enabling command completion.
  upload            Upload Dandiset files to DANDI Archive.
  validate          Validate files for data standards compliance.
```
Run `dandi --help` or `dandi <subcommand> --help` (e.g. `dandi upload --help`) to see manual pages.

## Resources

* To learn how to interact with the DANDI archive and for examples on how to use the DANDI Client in various use cases,
see [the handbook](https://www.dandiarchive.org/handbook/)
  (specifically the sections on using the CLI to
[download](https://www.dandiarchive.org/handbook/12_download/) and
[upload](https://www.dandiarchive.org/handbook/13_upload/) `Dandisets`).

* To get help:
  - ask a question: https://github.com/dandi/helpdesk/discussions
  - file a feature request or bug report: https://github.com/dandi/helpdesk/issues/new/choose
  - contact the DANDI team: help@dandiarchive.org

* To understand how to contribute to the dandi-cli repository, see the [DEVELOPMENT.md](./DEVELOPMENT.md) file.


# ⚠ PostHog no longer supports Kubernetes deployments. ⚠

**As of May 2023, [PostHog no longer support Kubernetes deployments](https://posthog.com/blog/sunsetting-helm-support-posthog).** This decision doesn't impact open source ("Hobby") users on [Docker Compose deployments](https://posthog.com/docs/self-host).

## What's next?

To continue using PostHog, you have two options:

### Using PostHog Cloud (Recommended)
We strongly encourage users to move to PostHog Cloud wherever possible so that they always have the latest features and the full benefit of official support. It usually works out much cheaper, too.

PostHog Cloud is SOC 2 compliant and available with either EU or US hosting options. If you already have a self-hosted instance, you can [migrate to PostHog Cloud](https://posthog.com/docs/migrate/migrate-between-posthog-instances). Alternatively, you can choose to [migrate to a open source deployment](https://posthog.com/docs/self-host/open-source/deployment) instead.

### Self-hosting PostHog
If you want to continue using a self-hosted PostHog deployment then you do so without support.

We **strongly recommend** following the [official instructions](https://posthog.com/docs/self-host) to deploy PostHog if you must self-host. Most people who modify or use a non-standard way of running this chart run into issues, which we are unable to help with.

Security updates will continue until May 2024.

# PostHog Helm Chart

[![Latest release of the Helm chart](https://img.shields.io/badge/dynamic/yaml.svg?label=Helm%20chart%20version&url=https://posthog.github.io/charts-clickhouse/index.yaml&query=$.entries.posthog[:1].version&logo=helm)](https://github.com/PostHog/charts-clickhouse)
[![Latest release app version](https://img.shields.io/badge/dynamic/yaml.svg?label=App%20version&url=https://posthog.github.io/charts-clickhouse/index.yaml&query=$.entries.posthog[:1].appVersion)](https://github.com/PostHog/posthog)
[![MIT License](https://img.shields.io/badge/License-MIT-red.svg?style=flat-square)](https://opensource.org/licenses/MIT)
[![Community](https://img.shields.io/badge/PostHog_chat-slack-blue?logo=slack)](https://posthog.com/questions)

-----

🦔 [PostHog](https://posthog.com/) is a developer-friendly, open-source product analytics suite.

This Helm chart bootstraps a [PostHog](https://posthog.com/) installation on a [Kubernetes](http://kubernetes.io) cluster using the [Helm](https://helm.sh) package manager.

## Prerequisites
- Kubernetes >=1.24 <= 1.26
- Helm >= 3.7.0

## Installation
Deployment instructions for the major cloud service providers and on-premise deploys are available [here](https://posthog.com/docs/self-host).

## Changelog
We documented detailed changes for each major release in our [upgrade notes](https://posthog.com/docs/self-host/deploy/upgrade-notes).

## Development
We welcome all contributions to the community, but no longer offer any support.

### Testing
This repo uses several types of test suite targeting different goals:

- **lint tests**: to verify if the Helm templates can be rendered without errors
- **unit tests**: to verify if the rendered Helm templates are as we expect
- **integration tests**: to verify if applying the rendered Helm templates against a Kubernetes target cluster gives us the stack and PostHog installation we expect

#### Lint tests
We use `helm lint` that can be invoked via: `helm lint --strict --set "cloud=local" charts/posthog`

#### Unit tests
In order to run the test suite, you need to install the `helm-unittest` plugin. You can do that by running: `helm plugin install https://github.com/quintush/helm-unittest --version 0.2.8`

For more information about how it works and how to write test cases, please look at the upstream [documentation](https://github.com/quintush/helm-unittest/blob/master/README.md) or to the [tests already available in this repo](https://github.com/PostHog/charts-clickhouse/tree/main/charts/posthog/tests).

To run the test suite you can execute: `helm unittest --helm3 --strict --file 'tests/*.yaml' --file 'tests/clickhouse-operator/*.yaml' charts/posthog`

If you need to update the snapshots, execute:

```
helm unittest --helm3 --strict --file 'tests/*.yaml' --file 'tests/**/*.yaml' charts/posthog -u
```

#### Integration tests
- [kubetest](https://github.com/PostHog/charts-clickhouse/tree/main/ci/kubetest): to verify if applying the rendered Helm templates against a Kubernetes target cluster gives us the stack we expect (example: are the disks encrypted? Can this pod communicate with this service?)
- [k6](https://github.com/PostHog/charts-clickhouse/tree/main/ci/k6): HTTP test used to verify the reliability, performance and compliance of the PostHog installation (example: is the PostHog ingestion working correctly?)
- [e2e - k3s](https://github.com/PostHog/charts-clickhouse/tree/main/.github/workflows/test-helm-chart.yaml): to verify Helm install/upgrade commands on a local k3s cluster
- [e2e - Amazon Web Services](https://github.com/PostHog/charts-clickhouse/tree/main/.github/workflows/test-amazon-web-services-install.yaml), [e2e - DigitalOcean](https://github.com/PostHog/charts-clickhouse/tree/main/.github/workflows/test-digitalocean-install.yaml), [e2e - Google Cloud Platform](https://github.com/PostHog/charts-clickhouse/tree/main/.github/workflows/test-google-cloud-platform-install.yaml): to verify Helm install command on the officially supported cloud platforms

### Running k3s for tests locally

k6 test is using [k3s](https://k3s.io/) for running things on localhost, which might be tricky to get running locally. Here's one method:

```bash
# Install k3s without system daemon
curl -sfL https://get.k3s.io | INSTALL_K3S_SKIP_ENABLE=true sh
# Once done run k3s manually, disabling conflicting services
sudo k3s server --write-kubeconfig-mode 644 --disable traefik --docker --disable-network-policy

# Deploy the chart
helm upgrade --install -f ci/values/k3s.yaml --timeout 20m --create-namespace --namespace posthog posthog ./charts/posthog --wait --wait-for-jobs --debug

# Once done, prepare the data/environment
./ci/setup_ingestion_test.sh

export KUBECONFIG=/etc/rancher/k3s/k3s.yaml
export POSTHOG_API_ADDRESS=$(kubectl get svc -n posthog posthog-web -o jsonpath="{.spec.clusterIP}")
export POSTHOG_EVENTS_ADDRESS=$(kubectl get svc -n posthog posthog-events -o jsonpath="{.spec.clusterIP}")
export "POSTHOG_API_ENDPOINT=http://${POSTHOG_API_ADDRESS}:8000"
export "POSTHOG_EVENT_ENDPOINT=http://${POSTHOG_EVENTS_ADDRESS}:8000"
export "SKIP_SOURCE_IP_ADDRESS_CHECK=true"

# Run test
k6 run ci/k6/ingestion-test.js
```

### Release

To release a new chart, bump the `version` in `charts/posthog/Chart.yaml`. We use [Semantic Versioning](https://semver.org/):

    MAJOR version when you make incompatible API changes
    MINOR version when you add functionality in a backwards compatible manner
    PATCH version when you make backwards compatible bug fixes

Read API here as the chart values interface. When increasing the MAJOR version, ensure to add
appropriate documentation to the [Upgrade notes](https://posthog.com/docs/runbook/upgrade-notes).

Charts are [published on push](https://github.com/PostHog/charts-clickhouse/blob/main/.github/workflows/release-chart.yml)
to the `main` branch.

Note that development charts are also released on PRs such that changes can be tested as required
before merge, e.g. changing staging/dev to use the chart for more end to end validation.