[![CI](https://github.com/exo-lang/exo/actions/workflows/main.yml/badge.svg)](https://github.com/exo-lang/exo/actions/workflows/main.yml)
![GitHub last commit](https://img.shields.io/github/last-commit/exo-lang/exo)
[![codecov](https://codecov.io/gh/exo-lang/exo/branch/master/graph/badge.svg?token=BFIZ0WKP4I)](https://codecov.io/gh/exo-lang/exo)

# Install Exo

## Using Exo

If you're just using Exo, install it using `pip`:
```sh
$ pip install exo-lang
```

## Developing Exo

If you plan to work on the compiler directly, clone this repository and run the following commands:

```sh
$ git submodule update --init --recursive
$ python3.9 -m venv ~/.venv/exo
$ source ~/.venv/exo/bin/activate
```

This will checkout all the required submodules and enable the Exo virtual environment. Next, install the compiler:
```sh
$ python -m pip install -U pip setuptools wheel
$ python -m pip install -r requirements.txt
$ pre-commit install
```

This will make sure you have the submodules checked out and that the pre-commit
scripts (that run an autoformatter, maybe other tools in the future) run.

If you're feeling ambitious, you can also [install Exo from source](#build-exo-from-source).

# Examples

Take a look at `exo/examples` for scheduling examples.

# Exo's scheduling API

## Top-level Python function decorator

1. `@proc` - decorates a Python function which is parsed and compiled as Exo. Replaces
   the function with a `Procedure` object.
2. `@instr` - same as `@proc`, but accepts a hardware instruction as a format string.
3. `@config` - decorates a Python class which is parsed and compiled as an Exo
   configuration object

## Procedure object methods

**Introspection operations**

- `.name()` returns the procedure name.
- `.check_effects()` forces Exo to run effect checking on the procedure.
- `.show_effects()` prints the effects of the procedure.
- `.show_effect(stmt)` prints the effect of the `stmt` in the procedure.
- `.is_instr()` returns `true` if the procedure has a hardware instruction string.
- `.get_instr()` returns the hardware instruction string.
- `.get_ast()` returns a `QAST`, which is an AST representation suitable for
  introspection.

**Execution / interpretation operations**

- `.compile_c(directory, filename)` compiles the procedure into C and stores
  in `filename` in the `directory`.
- `.interpret(**args)` runs Exo interpreter on the procedure.

## Scheduling operations on Procedure objects

**Buffer related operations**

| Operation                                                   | Description                                                                                |
|-------------------------------------------------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| `.reuse_buffer(buf1, buf2)`                                   | Reuses a buffer `buf1` in the use site of `buf2` and removes the allocation of `buf2`.                                                                                |
| `.inline_window(win_stmt)`                                  | Removes the window statement `win_stmt`, which is an alias to the window, and inlines the windowing in its use site.                                                                                |
| `.expand_dim(stmt, alloc_dim, indexing)`                    | Expands the dimension of the allocation statement `stmt` with dimension `alloc_dim` of indexing `indexing`.                                                                                |
| `.bind_expr(new_name, expr)`                                | Binds the right hand side expression `expr` to a newly allocated buffer named `new_name`                                                                                |
| `.stage_mem(win_expr, new_name, stmt_start, stmt_end=None)` | Stages the buffer `win_expr` to the new window expression `new_name` in statement block (`stmt_start` to `stmt_end`), and adds an initialization loop and a write-back loop.                                                                                |
| `.rearrange_dim(alloc, dimensions)`                         | Takes an allocation statement and a list of integers to map the dimension. It rearranges the dimensions of `alloc` in `dimension` order. E.g., if `alloc` were `foo[N,M,K]` and the `dimension` were `[2,0,1]`, it would become `foo[K,N,M]` after this operation.            |
| `.lift_alloc(alloc, n_lifts=1, keep_dims=False)`            | Lifts the allocation statement `alloc` out of `n_lifts` number of scopes. If and For statements are the only statements in Exo which introduce a scope. When lifting the allocation out of a for loop, it will expand its dimension to the loop bound if `keep_dims` is True. |

**Loop related operations**

| Operation                                                           | Description                                                                                |
|---------------------------------------------------------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| `.split(loop, split_const, iter_vars, tail='guard', perfect=False)` | Splits `loop` into an outer and an inner loop. The inner loop bound is `split_const` and the outer and inner loop names are specified by a list of strings `iter_vars`. If `perfect` is True, it will not introduce a tail case. `tail` specifies the tail strategies, where the options are `guard`, `cut`, and `cut_and_guard`. |
| `.fuse_loop(loop1, loop2)`                                          | Fuses two adjacent loops with a common iteration variable.                                                                                |
| `.partition_loop(loop, num)`                                        | Partitions `loop` into two loops, the first running between `0` and `num` and the second between `num+1` and `loop`'s original bound.                                                                                |
| `.reorder(loop1, loop2)`                                            | Reorders two nested loops. `loop2` should be nested directly inside `loop1`. `loop1` will be nested inside `loop2` after this operation.                                                                                |
| `.unroll(loop)`                                                     | Unrolls the loop. The loop needs to have a constant bound.                                                                                |
| `.fission_after(stmt, n_lifts=1)`                                   | Fissions the `n_lifts` number of loops around the `stmt`. The fissioned loops around the `stmt` need to be directly nested with each other and the statements before and after the `stmt` should not have any allocation dependencies.                                                                                |
| `.remove_loop(loop)`                                                | Replaces the loop with its body if the body is idempotent. The system must be able to prove that the loop runs at least once.                                                                                |

**Config related operations**

| Operation                                       | Description                                                                                |
|-------------------------------------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| `.bind_config(expr, config, field)`             | Binds the right hand side `expr` to `config.field`. It will replace the use site of `expr` with `config.field` and introduces a config statement of `config.field = expr`. |
| `.configwrite_root(config, field, expr)`        | Inserts the config statement `config.field = expr` in the beginning of the procedure.                                                                                |
| `.configwrite_after(stmt, config, field, expr)` | Inserts the config statement `config.field = expr` after `stmt`.                                                                                |
| `.delete_config(stmt)`                          | Deletes the configuration statement.                                                                                |

**Other scheduling operations**

| Operation                        | Description                                                                                |
|----------------------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| `.add_assertion(assertion)`      | Asserts the truth of the expression `assertion` at the beginning of the procedure.                                                                                |
| `.lift_if(if, n_lifts=1)`        | Lifts the if statement `if` out of `n_lifts` number of scopes. This is similar to `reorder()`, but for if statements.                                                                        |
| `.eliminate_dead_code(stmt)`     | Eliminates `if` statement if condition is always True or False. Eliminates `for` statement if condition is always False.                                                                     |
| `.delete_pass()`                 | Deletes a `Pass` statement in the procedure.                                                                                |
| `.reorder_stmts(stmt1, stmt2)`   | Reorder two adjacent statements `stmt1` and `stmt2`. After this operation, the order will be `stmt2` `stmt1`.                                                                                |
| `.reorder_before(stmt)`          | Move the statement `stmt` before the previous statement. This is a shorthand for `reorder_stmts()`.                                                                                |
| `.replace(subproc, stmt)`        | Replace the statement with a call to `subproc`. This operation is one of our contributions and is explained in detail in the paper.                                                          |
| `.replace_all(subproc)`          | Eagerly replace every matching statement with a call to `subproc`.                                                                                |
| `.inline(call_site)`             | Inline the function call.                                                                                |
| `.is_eq(another_proc)`           | Returns True if `another_proc` is equivalent to the procedure.                                                                                |
| `.call_eqv(eqv_proc, call_site)` | Replace the function call statement of `call_site` with a call to an equivalent procedure `eqv_proc`.                                                                                |
| `.repeat(directive, *args)`      | Continue to run the directive until it fails. The directive and its arguments are given separately, e.g. `proc.repeat(Procedure.inline, "proc_to_inline(_)")`                                |
| `.simplify()`                    | Simplify the code in the procedure body. Tries to reduce expressions to constants and eliminate dead branches and loops. Uses branch conditions to simplify expressions inside the branches. |
| `.rename(new_name)`              | Rename this procedure to `new_name`.                                                                                |
| `.make_instr(instr_string)`      | Converts this procedure to an instruction procedure with instruction `instr_string`.                                                                                |
| `.partial_eval(*args, **kwargs)` | Specializes this procedure to the given argument values.                                                                                |
| `.set_precision(name, type)`     | Sets the precision type of `name` to `type`.                                                                                |
| `.set_window(name, is_window)`   | If `is_window` is True, it sets the buffer `name` to window type, instead of a tensor type.                                                                                |
| `.set_memory(name, mem_type)`    | Sets a buffer `name`'s memory type to `mem_type`.                                                                                |

# Exo's repository structure

In this repository, folders are structured as follows:

1. `src/exo` is where the core Exo implementation resides.
    - `API.py` defines the stable API. Documentation for this API can be found in the
      section below.
    - `libs/` contains some common memory definitions (`memories.py`) and custom malloc
      implementations. These could be user-defined, but we provide them for convenience.
    - `platforms/` contains instruction definitions that are part of the release. These
      could be user-defined, but we provide them for convenience.
    - Other files are implementation details of Exo (e.g., `typecheck.py` implements
      typecheck), but we will not dwell on these as they are not exposed to users.
2. `apps/` contains some sample applications written in Exo.
3. `dependencies/` contains submodules that Exo's apps and testing depends on.
4. `examples/` contains a Python notebook that we used for live demos. This should be
   ignored.
5. `tests/` contains the Exo test suite.

# Build Exo from source

We make active use of newer Python 3.x features, so please use the same version
of Python as our CI if you're getting errors about unsupported features.

Setting up Exo for development is like any other Python project. We
_strongly_ recommend you use a virtual environment.

```
$ git clone git@github.com:exo-lang/exo.git
$ cd exo/
$ git submodule update --init --recursive
$ python -m venv ~/.venv/exo
$ source ~/.venv/exo/bin/activate
(exo) $ python -m pip install -U pip setuptools wheel
(exo) $ python -m pip install -r requirements.txt
(exo) $ python -m build
(exo) $ pip install dist/*.whl
```

## PySMT

Depending on your setup, getting PySMT to work correctly may be difficult. You
need to independently install a solver such as Z3 or CVC4, and even then getting
the PySMT library to correctly locate that solver may be difficult. We have
included the `z3-solver` package as a requirement, which will hopefully avoid
this issue, but you can also install z3 (or your choice of solver)
independently.

# Notes for Testing

## Dependencies

### Build system (required)

The Exo test harness generates C code and as such needs to compile and link
using an unknown (i.e. system) compiler. To do this, it generates CMake build
files and invokes CMake behind the scenes.

Therefore, you must have CMake **3.21** or newer installed.

By default, CMake will use [Ninja](https://ninja-build.org) as its backend, but
this may be overridden by setting the environment variable `CMAKE_GENERATOR`
to `Unix Makefiles`, in case you do not wish to install Ninja.

### SDE (optional)

For testing x86 features on processors which don't support them (e.g., AVX-512
or AMX), we rely on
the [Intel Software Development Emulator](https://www.intel.com/content/www/us/en/developer/articles/tool/software-development-emulator.html)
as an optional dependency. Tests which rely on this (namely for AMX) look
for `sde64` either in the path defined by the `SDE_PATH` environment variable or
in the system `PATH`, and are skipped if it is not available.

## Running tests

To run the tests, simply type

```
pytest
```

in the root of the project.

## Running Coverage Testing

To run pytest with coverage tests, execute

```
pytest --cov=./ --cov-report=html
```

Then, if you want to see annotated source files, open `./htmlcov/index.html`.

# Publication

The first paper on Exo was published at PLDI '22. You can download the
paper [from ACM Digital Library](https://dl.acm.org/doi/abs/10.1145/3519939.3523446).
If you use Exo, please cite both the compiler and the paper!

```
@inproceedings{pldi22:exo,
  title        = {Exocompilation for Productive Programming of Hardware Accelerators},
  author       = {
    Ikarashi, Yuka and Bernstein, Gilbert Louis and Reinking, Alex and Genc,
    Hasan and Ragan-Kelley, Jonathan
  },
  year         = 2022,
  booktitle    = {
    Proceedings of the 43rd ACM SIGPLAN International Conference on Programming
    Language Design and Implementation
  },
  location     = {San Diego, CA, USA},
  publisher    = {Association for Computing Machinery},
  address      = {New York, NY, USA},
  series       = {PLDI 2022},
  pages        = {703–718},
  doi          = {10.1145/3519939.3523446},
  isbn         = 9781450392655,
  url          = {https://doi.org/10.1145/3519939.3523446},
  abstract     = {
    High-performance kernel libraries are critical to exploiting accelerators
    and specialized instructions in many applications. Because compilers are
    difficult to extend to support diverse and rapidly-evolving hardware
    targets, and automatic optimization is often insufficient to guarantee
    state-of-the-art performance, these libraries are commonly still coded and
    optimized by hand, at great expense, in low-level C and assembly. To better
    support development of high-performance libraries for specialized hardware,
    we propose a new programming language, Exo, based on the principle of
    exocompilation: externalizing target-specific code generation support and
    optimization policies to user-level code. Exo allows custom hardware
    instructions, specialized memories, and accelerator configuration state to
    be defined in user libraries. It builds on the idea of user scheduling to
    externalize hardware mapping and optimization decisions. Schedules are
    defined as composable rewrites within the language, and we develop a set of
    effect analyses which guarantee program equivalence and memory safety
    through these transformations. We show that Exo enables rapid development
    of state-of-the-art matrix-matrix multiply and convolutional neural network
    kernels, for both an embedded neural accelerator and x86 with AVX-512
    extensions, in a few dozen lines of code each.
  },
  numpages     = 16,
  keywords     = {
    program optimization, user-schedulable languages, user-extensible backend
    & scheduling, instruction abstraction, scheduling, hardware
    accelerators
  }
}
```


PyAutoGalaxy: Open-Source Multi Wavelength Galaxy Structure & Morphology
========================================================================

.. image:: https://mybinder.org/badge_logo.svg
   :target: https://mybinder.org/v2/gh/Jammy2211/autogalaxy_workspace/HEAD

.. image:: https://readthedocs.org/projects/pyautogalaxy/badge/?version=latest
   :target: https://pyautogalaxy.readthedocs.io/en/latest/?badge=latest
   :alt: Documentation Status

.. image:: https://github.com/Jammy2211/PyAutoGalaxy/actions/workflows/main.yml/badge.svg
   :target: https://github.com/Jammy2211/PyAutoGalaxy/actions

.. image:: https://github.com/Jammy2211/PyAutoBuild/actions/workflows/release.yml/badge.svg
   :target: https://github.com/Jammy2211/PyAutoBuild/actions

.. image:: https://img.shields.io/badge/code%20style-black-000000.svg
   :target: https://github.com/psf/black

.. image:: https://joss.theoj.org/papers/10.21105/joss.04475/status.svg
   :target: https://doi.org/10.21105/joss.04475

`Installation Guide <https://pyautogalaxy.readthedocs.io/en/latest/installation/overview.html>`_ |
`readthedocs <https://pyautogalaxy.readthedocs.io/en/latest/index.html>`_ |
`Introduction on Binder <https://mybinder.org/v2/gh/Jammy2211/autogalaxy_workspace/release?filepath=introduction.ipynb>`_ |
`HowToGalaxy <https://pyautogalaxy.readthedocs.io/en/latest/howtogalaxy/howtogalaxy.html>`_

The study of a galaxy's structure and morphology is at the heart of modern day Astrophysical research.

**PyAutoGalaxy** makes it simple to model galaxies, for example this Hubble Space Telescope imaging of a spiral
galaxy:

.. image:: https://github.com/Jammy2211/PyAutoGalaxy/blob/main/paper/hstcombined.png?raw=true
        :target: https://github.com/Jammy2211/PyAutoGalaxy/blob/main/paper/hstcombined.png

**PyAutoGalaxy** also fits interferometer data from observatories such as ALMA:

.. image:: https://github.com/Jammy2211/PyAutoGalaxy/blob/main/paper/almacombined.png?raw=true
        :target: https://github.com/Jammy2211/PyAutoGalaxy/blob/main/paper/almacombined.png

Getting Started
---------------

The following links are useful for new starters:

- `The PyAutoGalaxy readthedocs <https://pyautogalaxy.readthedocs.io/en/latest>`_, which includes `an installation guide <https://pyautogalaxy.readthedocs.io/en/latest/installation/overview.html>`_ and `an overview of PyAutoGalaxy's core features <https://pyautogalaxy.readthedocs.io/en/latest/overview/overview_1_galaxies.html>`_.

- `The introduction Jupyter Notebook on Binder <https://mybinder.org/v2/gh/Jammy2211/autogalaxy_workspace/release?filepath=introduction.ipynb>`_, where you can try **PyAutoGalaxy** in a web browser (without installation).

- `The autogalaxy_workspace GitHub repository <https://github.com/Jammy2211/autogalaxy_workspace>`_, which includes example scripts and the `HowToGalaxy Jupyter notebook lectures <https://github.com/Jammy2211/autogalaxy_workspace/tree/master/notebooks/howtogalaxy>`_ which give new users a step-by-step introduction to **PyAutoGalaxy**.

API Overview
------------

Galaxy morphology calculations are performed in **PyAutoGalaaxy** by building a ``Plane`` object from ``LightProfile``
and ``Galaxy`` objects. Below, we create a simple galaxy system where a redshift 0.5
``Galaxy`` with an ``Sersic`` ``LightProfile`` representing a bulge and an ``Exponential`` ``LightProfile``
representing a disk.

.. code-block:: python

    import autogalaxy as ag
    import autogalaxy.plot as aplt

    """
    To describe the galaxy emission two-dimensional grids of (y,x) Cartesian
    coordinates are used.
    """
    grid = ag.Grid2D.uniform(
        shape_native=(50, 50),
        pixel_scales=0.05,  # <- Conversion from pixel units to arc-seconds.
    )

    """
    The galaxy has an elliptical sersic light profile representing its bulge.
    """
    bulge=ag.lp.Sersic(
        centre=(0.0, 0.0),
        ell_comps=ag.convert.ell_comps_from(axis_ratio=0.9, angle=45.0),
        intensity=1.0,
        effective_radius=0.6,
        sersic_index=3.0,
    )

    """
    The galaxy also has an elliptical exponential disk
    """
    disk = ag.lp.Exponential(
        centre=(0.0, 0.0),
        ell_comps=ag.convert.ell_comps_from(axis_ratio=0.7, angle=30.0),
        intensity=0.5,
        effective_radius=1.6,
    )

    """
    We combine the above light profiles to compose a galaxy at redshift 1.0.
    """
    galaxy = ag.Galaxy(redshift=1.0, bulge=bulge, disk=disk)

    """
    We create a Plane, which in this example has just one galaxy but can
    be extended for datasets with many galaxies.
    """
    plane = ag.Plane(
        galaxies=[galaxy],
    )

    """
    We can use the Grid2D and Plane to perform many calculations, for example
    plotting the image of the galaxyed source.
    """
    plane_plotter = aplt.GalaxiesPlotter(plane=plane, grid=grid)
    plane_plotter.figures_2d(image=True)

With **PyAutoGalaxy**, you can begin modeling a galaxy in just a couple of minutes. The example below demonstrates a
simple analysis which fits a galaxy's light.

.. code-block:: python

    import autofit as af
    import autogalaxy as ag

    import os

    """
    Load Imaging data of the strong galaxy from the dataset folder of the workspace.
    """
    dataset = ag.Imaging.from_fits(
        data_path="/path/to/dataset/image.fits",
        noise_map_path="/path/to/dataset/noise_map.fits",
        psf_path="/path/to/dataset/psf.fits",
        pixel_scales=0.1,
    )

    """
    Create a mask for the data, which we setup as a 3.0" circle.
    """
    mask = ag.Mask2D.circular(
        shape_native=dataset.shape_native,
        pixel_scales=dataset.pixel_scales,
        radius=3.0
    )

    """
    We model the galaxy using an Sersic LightProfile.
    """
    light_profile = ag.lp.Sersic

    """
    We next setup this profile as model components whose parameters are free & fitted for
    by setting up a Galaxy as a Model.
    """
    galaxy_model = af.Model(ag.Galaxy, redshift=1.0, light=light_profile)
    model = af.Collection(galaxy=galaxy_model)

    """
    We define the non-linear search used to fit the model to the data (in this case, Dynesty).
    """
    search = af.Nautilus(name="search[example]", n_live=50)

    """
    We next set up the `Analysis`, which contains the `log likelihood function` that the
    non-linear search calls to fit the galaxy model to the data.
    """
    analysis = ag.AnalysisImaging(dataset=masked_dataset)

    """
    To perform the model-fit we pass the model and analysis to the search's fit method. This will
    output results (e.g., dynesty samples, model parameters, visualization) to hard-disk.
    """
    result = search.fit(model=model, analysis=analysis)

    """
    The results contain information on the fit, for example the maximum likelihood
    model from the Dynesty parameter space search.
    """
    print(result.samples.max_log_likelihood())

Support
-------

Support for installation issues, help with galaxy modeling and using **PyAutoGalaxy** is available by
`raising an issue on the GitHub issues page <https://github.com/Jammy2211/PyAutoGalaxy/issues>`_.

We also offer support on the **PyAutoGalaxy** `Slack channel <https://pyautogalaxy.slack.com/>`_, where we also provide the
latest updates on **PyAutoGalaxy**. Slack is invitation-only, so if you'd like to join send
an `email <https://github.com/Jammy2211>`_ requesting an invite.


[![Black](https://img.shields.io/badge/code%20style-black-000000.svg)](https://github.com/psf/black)
[![Build](https://github.com/theislab/ehrapy/actions/workflows/build.yml/badge.svg)](https://github.com/theislab/ehrapy/actions/workflows/build.yml)
[![Codecov](https://codecov.io/gh/theislab/ehrapy/branch/master/graph/badge.svg)](https://codecov.io/gh/theislab/ehrapy)
[![License](https://img.shields.io/github/license/theislab/ehrapy)](https://opensource.org/licenses/Apache2.0)
[![PyPI](https://img.shields.io/pypi/v/ehrapy.svg)](https://pypi.org/project/ehrapy/)
[![Python Version](https://img.shields.io/pypi/pyversions/ehrapy)](https://pypi.org/project/ehrapy)
[![Read the Docs](https://img.shields.io/readthedocs/ehrapy/latest.svg?label=Read%20the%20Docs)](https://ehrapy.readthedocs.io/)
[![Test](https://github.com/theislab/ehrapy/actions/workflows/test.yml/badge.svg)](https://github.com/theislab/ehrapy/actions/workflows/test.yml)
[![pre-commit](https://img.shields.io/badge/pre--commit-enabled-brightgreen?logo=pre-commit&logoColor=white)](https://github.com/pre-commit/pre-commit)

<img src="https://user-images.githubusercontent.com/21954664/156930990-0d668468-0cd9-496e-995a-96d2c2407cf5.png" alt="ehrapy logo">

# ehrapy overview

![fig1](https://github.com/theislab/ehrapy/assets/99650244/aad523a3-b4f9-4a7a-bb61-612af9a6874c)

## Features

-   Exploratory and targeted analysis of Electronic Health Records
-   Quality control & preprocessing
-   Visualization & Exploration
-   Clustering & trajectory inference

## Installation

You can install _ehrapy_ via [pip] from [PyPI]:

```console
$ pip install ehrapy
```

## Usage

Please have a look at the [Usage documentation](https://ehrapy.readthedocs.io/en/latest/usage/usage.html) and the [tutorials](https://ehrapy.readthedocs.io/en/latest/tutorials/index.html).

```python
import ehrapy as ep
```

## Citation

[ Exploratory electronic health record analysis with ehrapy
Lukas Heumos, Philipp Ehmele, Tim Treis, Julius Upmeier zu Belzen, Altana Namsaraeva, Nastassya Horlava, Vladimir A. Shitov, Xinyue Zhang, Luke Zappia, Rainer Knoll, Niklas J. Lang, Leon Hetzel, Isaac Virshup, Lisa Sikkema, Eljas Roellin, Fabiola Curion, Roland Eils, Herbert B. Schiller, Anne Hilgendorff, Fabian J. Theis
medRxiv 2023.12.11.23299816; doi: https://doi.org/10.1101/2023.12.11.23299816 ](https://www.medrxiv.org/content/10.1101/2023.12.11.23299816v1).


# PyScript

## What is PyScript

### Summary

PyScript is a framework that allows users to create rich Python applications in the browser using HTML's interface and the power of [Pyodide](https://pyodide.org/en/stable/), [MicroPython](https://micropython.org/) and [WASM](https://webassembly.org/), and modern web technologies.

To get started see the [Beginning PyScript tutorial](https://docs.pyscript.net/latest/beginning-pyscript/).

For examples see [here](https://pyscript.com/@examples).

Other useful resources:

-   The [official technical docs](https://docs.pyscript.net/).
-   Our current [Home Page](https://pyscript.net/) on the web.
-   A free-to-use [online editor](https://pyscript.com/) for trying PyScript.
-   Our community [Discord Channel](https://discord.gg/BYB2kvyFwm), to keep in touch .

Every Tuesday at 15:30 UTC there is the _PyScript Community Call_ on zoom, where we can talk about PyScript development in the open. Most of the maintainers regularly participate in the call, and everybody is welcome to join.

Every other Thursday at 16:00 UTC there is the _PyScript FUN_ call: this is a call in which everybody is encouraged to show what they did with PyScript.

For more details on how to join the calls and up to date schedule, consult the official calendar:

-   [Google calendar](https://calendar.google.com/calendar/u/0/embed?src=d3afdd81f9c132a8c8f3290f5cc5966adebdf61017fca784eef0f6be9fd519e0@group.calendar.google.com&ctz=UTC) in UTC time;
-   [iCal format](https://calendar.google.com/calendar/ical/d3afdd81f9c132a8c8f3290f5cc5966adebdf61017fca784eef0f6be9fd519e0%40group.calendar.google.com/public/basic.ics).

### Longer Version

PyScript is a meta project that aims to combine multiple open technologies into a framework that allows users to create sophisticated browser applications with Python. It integrates seamlessly with the way the DOM works in the browser and allows users to add Python logic in a way that feels natural both to web and Python developers.

## Try PyScript

To try PyScript, import the appropriate pyscript files into the `<head>` tag of your html page:

```html
<head>
    <link
        rel="stylesheet"
        href="https://pyscript.net/releases/2024.6.2/core.css"
    />
    <script
        type="module"
        src="https://pyscript.net/releases/2024.6.2/core.js"
    ></script>
</head>
<body>
    <script type="py" terminal>
        from pyscript import display
        display("Hello World!") # this goes to the DOM
        print("Hello terminal") # this goes to the terminal
    </script>
</body>
```

You can then use PyScript components in your html page. PyScript currently offers various ways of running Python code:

-   `<script type="py">`: can be used to define python code that is executable within the web page.
-   `<script type="py" src="hello.py">`: same as above, but the python source is fetched from the given URL.
-   `<script type="py" terminal>`: same as above, but also creates a terminal where to display stdout and stderr (e.g., the output of `print()`); `input()` does not work.
-   `<script type="py" terminal worker>`: run Python inside a web worker: the terminal is fully functional and `input()` works.
-   `<py-script>`: same as `<script type="py">`, but it is not recommended because if the code contains HTML tags, they could be parsed wrongly.
-   `<script type="mpy">`: same as above but use MicroPython instead of Python.

Check out the [official docs](https://docs.pyscript.net/) for more detailed documentation.

## How to Contribute

Read the [contributing guide](https://docs.pyscript.net/latest/contributing/) to learn about our development process, reporting bugs and improvements, creating issues and asking questions.

Check out the [developing process](https://docs.pyscript.net/latest/developers/) documentation for more information on how to setup your development environment.

## Governance

The [PyScript organization governance](https://github.com/pyscript/governance) is documented in a separate repository.

## Release

To cut a new release of PyScript simply
[add a new release](https://github.com/pyscript/pyscript/releases) while
remembering to write a comprehensive changelog. A [GitHub action](https://github.com/pyscript/pyscript/blob/main/.github/workflows/publish-release.yml)
will kick in and ensure the release is described and deployed to a URL with the
pattern: https://pyscript.net/releases/YYYY.M.v/ (year/month/version - as per
our [CalVer](https://calver.org/) versioning scheme).

Then, the following three separate repositories need updating:

-   [Documentation](https://github.com/pyscript/docs) - Change the `version.json`
    file in the root of the directory and then `node version-update.js`.
-   [Homepage](https://github.com/pyscript/pyscript.net) - Ensure the version
    referenced in `index.html` is the latest version.
-   [PSDC](https://pyscript.com) - Use discord or Anaconda Slack (if you work at
    Anaconda) to let the PSDC team know there's a new version, so they can update
    their project templates.


#########################################################
**XCP-D** : A Robust Postprocessing Pipeline of fMRI data
#########################################################

.. image:: https://img.shields.io/badge/Source%20Code-pennlinc%2Fxcp__d-purple
   :target: https://github.com/PennLINC/xcp_d
   :alt: GitHub Repository

.. image:: https://readthedocs.org/projects/xcp-d/badge/?version=latest
   :target: http://xcp-d.readthedocs.io/en/latest/?badge=latest
   :alt: Documentation

.. image:: https://img.shields.io/badge/docker-pennlinc/xcp_d-brightgreen.svg?logo=docker&style=flat
   :target: https://hub.docker.com/r/pennlinc/xcp_d/tags/
   :alt: Docker

.. image:: https://circleci.com/gh/PennLINC/xcp_d.svg?style=svg
   :target: https://circleci.com/gh/PennLINC/xcp_d
   :alt: Test Status

.. image:: https://codecov.io/gh/PennLINC/xcp_d/branch/main/graph/badge.svg
   :target: https://codecov.io/gh/PennLINC/xcp_d
   :alt: Codecov

.. image:: https://zenodo.org/badge/309485627.svg
   :target: https://zenodo.org/badge/latestdoi/309485627
   :alt: DOI

.. image:: https://img.shields.io/github/license/pennlinc/xcp_d
   :target: https://opensource.org/licenses/BSD-3-Clause
   :alt: License

This fMRI post-processing and noise regression pipeline is developed by the
`Satterthwaite lab at the University of Pennslyvania <https://www.satterthwaitelab.com/>`_
(**XCP**\; eXtensible Connectivity Pipeline)  and
`Developmental Cognition and Neuroimaging lab at the University of Minnesota
<https://innovation.umn.edu/developmental-cognition-and-neuroimaging-lab/>`_ (**-D**\CAN)
for open-source software distribution.

*****
About
*****

XCP-D paves the final section of the reproducible and scalable route from the MRI scanner to
functional connectivity data in the hands of neuroscientists.
We developed XCP-D to extend the BIDS and NiPrep apparatus to the point where data is most
commonly consumed and analyzed by neuroscientists studying functional connectivity.
Thus, with the development of XCP-D, data can be automatically preprocessed and analyzed in BIDS
format, using NiPrep-style containerized code, all the way from the scanner to functional
connectivity matrices.

XCP-D picks up right where `fMRIprep <https://fmriprep.org>`_ ends, directly consuming the outputs
of fMRIPrep.
XCP-D leverages the BIDS and NiPreps frameworks to automatically generate denoised BOLD images,
parcellated time series, functional connectivity matrices, and quality assessment reports.
XCP-D can also process outputs from: `NiBabies <https://nibabies.readthedocs.io>`_,
`ABCD-BIDS <https://github.com/DCAN-Labs/abcd-hcp-pipeline>`_,
`Minimally preprocessed HCP <https://www.humanconnectome.org/study/hcp-lifespan-development/\
data-releases>`_,
and `UK Biobank <https://doi.org/10.1016/j.neuroimage.2017.10.034>`_ data.

*Please note that XCP is only compatible with HCP-YA versions downloaded c.a. Feb 2023 at the moment.*

.. image:: https://raw.githubusercontent.com/pennlinc/xcp_d/main/docs/_static/xcp_figure_1.png

See the `documentation <https://xcp-d.readthedocs.io/en/latest/>`_ for more details.

Why you should use XCP-D
````````````````````````

XCP-D produces the following commonly-used outputs: matrices, parcellated time series,
dense time series, and additional QC measures.

XCP-D is designed for resting-state or pseudo-resting-state functional connectivity analyses.
XCP-D derivatives may be useful for seed-to-voxel and ROI-to-ROI functional connectivity analyses,
as well as decomposition-based methods, such as ICA or NMF.

When you should not use XCP-D
`````````````````````````````

XCP-D is not designed as a general-purpose postprocessing pipeline.
It is really only appropriate for certain analyses,
and other postprocessing/analysis tools are better suited for many types of data/analysis.

XCP-D derivatives are not particularly useful for task-dependent functional connectivity analyses,
such as psychophysiological interactions (PPIs) or beta series analyses.
It is also not suitable for general task-based analyses, such as standard task GLMs,
as we recommend included nuisance regressors in the GLM step,
rather than denoising data prior to the GLM.

************
Citing XCP-D
************

If you use XCP-D in your research, please use the boilerplate generated by the workflow.
If you need an immediate citations, please cite the following preprint:

   Mehta, K., Salo, T., Madison, T., Adebimpe, A., Bassett, D. S., Bertolero, M., ... & Satterthwaite, T. D.
   (2023).
   XCP-D: A Robust Pipeline for the post-processing of fMRI data.
   *bioRxiv*.
   doi:10.1101/2023.11.20.567926.

Please also cite the Zenodo DOI for the version you're referencing.


  官方网站 | 快速开始 | 更新日志 | 更新推送 | TG 群组

# 项目说明

本项目是基于 RSS 的全自动追番整理下载工具。只需要在 [Mikan Project][mikan] 等网站上订阅番剧，就可以全自动追番。
并且整理完成的名称和目录可以直接被 [Plex][plex]、[Jellyfin][plex] 等媒体库软件识别，无需二次刮削。

## AutoBangumi 功能说明

- 简易单次配置就能持续使用
- 无需介入的 `RSS` 解析器，解析番组信息并且自动生成下载规则。
- 番剧文件整理:

    ```
    Bangumi
    ├── bangumi_A_title
    │   ├── Season 1
    │   │   ├── A S01E01.mp4
    │   │   ├── A S01E02.mp4
    │   │   ├── A S01E03.mp4
    │   │   └── A S01E04.mp4
    │   └── Season 2
    │       ├── A S02E01.mp4
    │       ├── A S02E02.mp4
    │       ├── A S02E03.mp4
    │       └── A S02E04.mp4
    ├── bangumi_B_title
    │   └─── Season 1
    ```

- 全自动重命名，重命名后 99% 以上的番剧可以直接被媒体库软件直接刮削

    ```
  [Lilith-Raws] Kakkou no Iinazuke - 07 [Baha][WEB-DL][1080p][AVC AAC][CHT][MP4].mp4
  >>
   Kakkou no Iinazuke S01E07.mp4
  ```

- 自定义重命名，可以根据上级文件夹对所有子文件重命名。
- 季中追番可以补全当季遗漏的所有剧集
- 高度可自定义的功能选项，可以针对不同媒体库软件微调
- 支持多种 RSS 站点，支持聚合 RSS 的解析。
- 无需维护完全无感使用
- 内置 TDMB 解析器，可以直接生成完整的 TMDB 格式的文件以及番剧信息。

## [Roadmap](https://github.com/users/EstrellaXD/projects/2)

***计划开发的功能：***

- Transmission 的支持。

## Star History

[![Star History Chart](https://api.star-history.com/svg?repos=EstrellaXD/Auto_Bangumi&type=Date)](https://star-history.com/#EstrellaXD/Auto_Bangumi)

## 贡献

欢迎提供 ISSUE 或者 PR, 贡献代码前建议阅读 [CONTRIBUTING.md](CONTRIBUTING.md)。

贡献者名单请见：

## Licence

[MIT licence](https://github.com/EstrellaXD/Auto_Bangumi/blob/main/LICENSE)

[mikan]: https://mikanani.me
[plex]: https://plex.tv
[jellyfin]: https://jellyfin.org


GreaterWMS

Open Source Inventory Management System

![License: APLv2](https://img.shields.io/github/license/GreaterWMS/GreaterWMS)
![Release Version (latest Version)](https://img.shields.io/github/v/release/GreaterWMS/GreaterWMS?color=orange&include_prereleases)
![QR Code Support](https://img.shields.io/badge/QR--Code-Support-orange.svg)
![Docker Support](https://img.shields.io/badge/Docker-Support-orange.svg)
![i18n Support](https://img.shields.io/badge/i18n-Support-orange.svg)

![repo size](https://img.shields.io/github/repo-size/GreaterWMS/GreaterWMS)
![GitHub commit activity](https://img.shields.io/github/commit-activity/m/GreaterWMS/GreaterWMS)
![Contributors](https://img.shields.io/github/contributors/GreaterWMS/GreaterWMS?color=blue)

![GitHub Org's stars](https://img.shields.io/github/stars/GreaterWMS?style=social)
![GitHub Follows](https://img.shields.io/github/followers/Singosgu?style=social)
![GitHub Forks](https://img.shields.io/github/forks/GreaterWMS/GreaterWMS?style=social)
![GitHub Watch](https://img.shields.io/github/watchers/GreaterWMS/GreaterWMS?style=social)

![Python](https://img.shields.io/badge/Python-3.8.10-yellowgreen)
![Django](https://img.shields.io/badge/Django-4.1.2-yellowgreen)
![Quasar Cli](https://img.shields.io/badge/Quasar/cli-1.2.1-yellowgreen)
![Vue](https://img.shields.io/badge/Vue-2.6.0-yellowgreen)
![NodeJS](https://img.shields.io/badge/NodeJS-14.19.3-yellowgreen)

[![YouTube](https://img.shields.io/youtube/channel/subscribers/UCPW1wciGMIEh7CYOdLnsloA?color=red&label=YouTube&logo=youtube&style=for-the-badge)](https://www.youtube.com/channel/UCPW1wciGMIEh7CYOdLnsloA)

#### You can share your vacant warehouse space, use it for those in need, and generate income

## :rocket: Link US

    Home Page

    Demo

  Video Tutorials

  Report Bug

  Request Feature

  中文文档

[//]: # (About the Project)
## :star2: About the Project

This Inventory management system is the currently Ford Asia Pacific after-sales logistics warehousing supply chain process.
After I leave Ford, I start this project in order to help some who need it.
OneAPP Type. Support scanner PDA, mobile APP, desktop exe, website as well.

[//]: # (Function)
## :dart: Function

* [x] Multiple Warehouses
* [x] Supplier Management
* [x] Customer Management
* [x] Scanner PDA
* [x] Cycle Count
* [x] Order Management
* [x] Stock Control
* [x] Safety Stock Show
* [x] API Documents
* [x] IOS APP Support
* [x] Android APP Support
* [x] Electron APP Support
* [x] Auto Update
* [x] i18n Support
* [x] API Documents

[//]: # (Install)
## :compass: Install
Python install
- [python 3.8.10](https://www.python.org/downloads/release/python-3810/)

Nodejs install
- [nodejs 14.19.3](https://nodejs.org/download/release/v14.19.3/)

Twisted install
- Please google how to install Twisted , if you have some problem on install it .

Clone Project
~~~shell
git clone https://github.com/GreaterWMS/GreaterWMS.git
~~~

### docker(Optional)
~~~shell
cd GreaterWMS/
docker-compose up -d
# Change Baseurl
# baseurl GreaterWMS/templates/public/statics/baseurl.txt
docker-compose restart
~~~

  Windows X64

  Centos 7

  Ubuntu 20

[//]: # (development)
## :hammer_and_wrench: How To Run Development Server:

- Run Backend:
~~~shell
cd GreaterWMS
daphne -p 8008 greaterwms.asgi:application
or
daphne -b 0.0.0.0 -p 8008 greaterwms.asgi:application # lan
~~~

- Run Frontend:
~~~shell
cd templates
quasar d
~~~

- Change Request Baseurl
~~~shell
templates/public/statics/baseurl.txt
~~~

- API Documents

~~~shell
baseurl + '/docs/'
~~~

### Companion Mobile APP

- App Source Code

~~~shell
npm install cordova -g

cd app
yarn install
## Development
quasar d -m cordova -T [android, ios]
## Deploy
quasar build -m [android, ios]
~~~

- You can directly use app if you don't want to build it

GreaterWMS is supported by a companion mobile app which allows users access to run the business well.
It can scan the goods by your camera or your PDA scanner.

[IOS](https://apps.apple.com/gb/app/intelligent-warehousing-gwms/id6444078526)

[Android](https://production.56yhz.com/media/GWMS.apks)

## Download Android installer tools

!!! info "Android"

    App store search

    Split APKs Installer

## Directly download installer tools

[Sai](https://po.56yhz.com/media/sai.apk)

- Open Sai APP, choose GWMS.apks then install

[//]: # (publish)
## :trumpet: How To Publish Your APP:

- Web Build:

~~~shell
cd templates
quasar build
~~~

[//]: # (deploy)
## :computer: How To Deploy Server:

  Supervisor Process Guarded

  Nginx Config

If the server has SSL enabled, please use HTTPS and WSS, if SSL is not enabled, use HTTP and WS

The front-end code needs to be rebuilt after modification.

## Show


# Working with translations

You can contribute translations to the Free app [Learn GDScript From Zero](https://github.com/GDQuest/learn-gdscript/) in this repository.

![](images/app-practice-screen.png)

We keep translations separate from the app's source code to simplify the contribution process.

## How to contribute translations

The app offers internationalization support using PO (gettext) files. It's a popular format among translation professionals, and it has native support in Godot.

### Contributing using Weblate

You can contribute translations using the online program Weblate. It's an excellent open-source online app designed for community-driven translations.

![](images/weblate.png)

If you decide to use the platform, please check the open pull requests (translation contributions on this translation repository) first: https://github.com/GDQuest/learn-gdscript-translations/pulls

Weblate will not detect and synchronize with those pull requests until we integrate them into the repository. If there is an open pull request for your target language, please coordinate with contributors and us to avoid losing work.

If there is no open pull request for your desired language, you can get started on Weblate right away.

Translate the app using Weblate today: https://hosted.weblate.org/projects/learn-gdscript-from-zero/

We warmly thank Weblate for giving us access to their free hosted plan for open source projects.

### Adding translations to PO files

To contribute translations, open the subdirectory corresponding to your target language. For example, `es/` for Spanish.

_Note: we hired a professional translator to translate the app to Spanish fully, so we currently don't need contributions in Spanish._

There, you'll find PO translation files. To edit those, you'll need a dedicated program.

There are several free and open-source translation programs you can use to translate PO files. We recommend [Poedit](https://poedit.net/) as it's available on all three major desktop platforms and it offers an accessible interface.

![](images/poedit.png)

Among other features, it allows you to mark translations you're unsure of as "needing work" so another translator can check it before publishing it.

If the directory for your language doesn't exist, you'll need to create it. You should create a directory with the two-letter code corresponding to your language. For example, `fr/` for French and `zh/` for Chinese.

There, you'll need to create PO translation files for your target language from the POT template files. If you need help with that, please ask. We'll be happy to set them up for you.

### Testing your translations in Godot

You'll want to check your translations in Godot when translating the app. To do so, you need to copy your PO files over to the app's repository: [learn-gdscript](https://github.com/GDQuest/learn-gdscript/).

First, you'll want to import the project in Godot.

Then, run the app by pressing F5, open the settings menu, and select the target language. The app will remember your choice when you reopen it.

Once you add many new translations, you can rerun the app to check their length doesn't cause overflowing or other UI issues. As words become longer or shorter in different languages, it's crucial to ensure the interface takes that into account.

If any bug appears in the interface with new translations, please [open a new issue](https://github.com/GDQuest/learn-gdscript/issues) and include a screenshot so we can fix it.

Finally, please note that if the app is running when you save translations, they may not reload in real-time. In that case, you'll need to close the app by pressing F8 in Godot and rerun it by pressing F5.

## Translation files

We split translations for each language into many files. It helps translators to focus on a single lesson or aspect of the app at a time:

* `application.*` contains all translatable strings from the UI of the app. Note that this also includes placeholder text which we sometimes add to our reusable components.
* `classref_database.*` contains entries for the documentation section of the practice side panel, a class reference of sorts that explains some methods and properties to the student. It is based on `course/documentation.csv`.
* `error_database.*` contains explanations and suggestions for GDScript and internal errors. It is based on `lsp/error_database.csv`.
* `lesson-*.*` files contain all translatable strings from the course content. Currently, only content blocks, quizzes, and practices are translated. We currently don't support translatable content in scripts and scenes used by lessons.

## How translations work in Godot and the app

The engine automatically translates UI nodes, while we need to wrap string literals in calls to the `tr()` function.

In your typical Godot project, you would preload translation resources through the project settings.

Instead, this app loads translations dynamically because we separate localization resources into several files per supported language. This applies to both POT files (translation templates) and translated PO files.

### Updating translation templates

The gettext translation format comes with two file extensions: POT for translation templates and PO for translations to a specific language.

PO files derive from POT files, which allow us to track strings that changed between releases finely.

To update POT files, you need to have `babel` and `babel-godot` installed for Python, following [recommendations](https://docs.godotengine.org/en/stable/tutorials/i18n/localization_using_gettext.html#creating-the-po-template-pot-using-pybabel) from the official Godot documentation.

We use a python script to extract docstrings. You can call it from the root of the project like so:

```
python ./i18n/extract.py
```


[![ci-tests](https://github.com/infuseai/piperider-cli/actions/workflows/tests.yaml/badge.svg)](https://github.com/infuseai/piperider-cli/actions/workflows/tests.yaml/badge.svg)
[![codecov](https://codecov.io/gh/InfuseAI/piperider/branch/main/graph/badge.svg?token=iVbQKGM1JA)](https://codecov.io/gh/InfuseAI/piperider)
[![release](https://img.shields.io/github/release/infuseAI/piperider-cli/all.svg?style=flat-square)](https://github.com/infuseAI/piperider-cli/releases)
[![pipy](https://img.shields.io/pypi/v/piperider?style=flat-square)](https://pypi.org/project/piperider/)
[![python](https://img.shields.io/pypi/pyversions/piperider?style=flat-square)](https://pypi.org/project/piperider/)
[![downloads](https://img.shields.io/pypi/dw/piperider?style=flat-square)](https://pypi.org/project/piperider/#files)
[![license](https://img.shields.io/github/license/infuseai/piperider?style=flat-square)](https://github.com/InfuseAI/piperider/blob/main/LICENSE)
[![InfuseAI Discord Invite](https://img.shields.io/discord/664381609771925514?color=%237289DA&label=chat&logo=discord&logoColor=white&style=flat-square)](https://discord.com/invite/5zb2aK9KBV)

   Docs  |
   Discord  |
   Blog

> \[!IMPORTANT\]
> PipeRider has been superseded by [Recce](https://datarecce.io). We recommend that users requiring pre-merge data validation checks migrate to Recce. PipeRider will not longer be updated on a regular basis. You are still welcome to open a PR with bug fixes or feature requests. For questions and help regarding this update, please contact [product@piperider.io](mailto:product@infuseai.io) or leave a message in the [Recce Discord](https://discord.gg/VpwXRC34jz).

# Code review for data in dbt

PipeRider automatically compares your data to highlight the difference in impacted downstream dbt models so you can
merge your Pull Requests with confidence.

### How it works:

- Easy to connect your datasource -> PipeRider leverages
  the [connection profiles in your dbt project](https://docs.getdbt.com/docs/get-started/connection-profiles) to connect
  to the data warehouse
- Generate profiling statistics of your models to get a high-level overview of your data
- Compare target branch changes with the main branch in a HTML report
- Post a quick summary of the data changes to your PR, so others can be confident too

### Core concepts

- **Easy to install**: Leveraging dbt's configuration settings, PipeRider can be installed within 2 minutes
- **Fast comparison**: by collecting profiling statistics (e.g. uniqueness, averages, quantiles, histogram) and metric
  queries, comparing downstream data impact takes little time, speeding up your team's review time
- **Valuable insights**: various profiling statistics displayed in the HTML report give fast insights into your data

# Quickstart

1. **Install PipeRider**

   ```bash
   pip install piperider[<connector>]
   ```

   You can find all supported data source connectors [here](https://docs.piperider.io/reference/supported-data-sources).

1. **Add PipeRider tag on your model**: Go to your dbt project, and add the PipeRider tag on the model you want to
   profile.

   ```sql
   --models/staging/stg_customers.sql
   {{ config(
      tags=["piperider"]
   ) }}

   select ...
   ```

   and show the models would be run by piperider

   ```
    dbt list -s tag:piperider --resource-type model
   ```

1. **Run PipeRider**

   ```bash
   piperider run
   ```

To see the full quick start guide, please refer
to [PipeRider documentation](https://docs.piperider.io/get-started/quick-start)

# Features

- **Model profiling**: PipeRider can profile your [dbt models](https://docs.getdbt.com/docs/build/models) and obtain
  information such as basic data composition, quantiles, histograms, text length, top categories, and more.
- **Metric queries**: PipeRider can integrate with [dbt metrics](https://docs.getdbt.com/docs/build/metrics) and present
  the time-series data of metrics in the report.
- **HTML report**: PipeRider generates a static HTML report each time it runs, which can be viewed locally or shared.
- **Report comparison**: You can compare two previously generated reports or use a single command to compare the
  differences between the current branch and the main branch. The latter is designed specifically for code review
  scenarios. In our pull requests on GitHub, we not only want to know which files have been changed, but also the impact
  of these changes on the data. PipeRider can easily generate comparison reports with a single command to provide this
  information.
- **CI integration**: The key to CI is automation, and in the code review process, automating this workflow is even more
  meaningful. PipeRider can easily integrate into your CI process. When new commits are pushed to your PR branch,
  reports can be automatically generated to provide reviewers with more confidence in the changes made when reviewing.

# Example Report Demo

We use the example project [git-repo-analytics](https://github.com/InfuseAI/git-repo-analytics) to demonstrate how to
use piperider+dbt+duckdb to analyze [dbt-core](https://github.com/dbt-labs/dbt-core) repository. Here is the generated
result (daily update)

[Run Report](https://piperider-github-readme.s3.ap-northeast-1.amazonaws.com/single-run-report/index.html)

[Comparison Report](https://piperider-github-readme.s3.ap-northeast-1.amazonaws.com/comparison-report/index.html)

[Comparison Summary in a PR](https://github.com/InfuseAI/git-repo-analytics/pull/19)

# PipeRider Cloud (beta)

[PipeRider Cloud](http://cloud.piperider.io/) allows you to upload reports and share them with your team members. For
information on pricing plans, please refer to the [pricing page](https://www.piperider.io/#pricing).

# PipeRider Compare Action

PipeRider provides the [PipeRider Compare Action](https://github.com/marketplace/actions/piperider-compare-action) to
quickly integrate into your Github Actions workflow. It has the following features:

- Automatically generates a report comparing the PR branch to the main branch
- Uploads the report to GitHub artifacts or PipeRider cloud
- Adds a comment to the pull request with a comparison summary and a link to the report.

You can refer to
example [workflow yaml](https://github.com/InfuseAI/jaffle_shop/blob/main/.github/workflows/pr-compare.yml) and
the [example pull request](https://github.com/InfuseAI/jaffle_shop/pull/19).

# Development

See [setup dev environment](DEVELOP.md) and the [contributing guildlines](CONTRIBUTING.md) to get started.

**We love chatting with our users! [Let us know](mailto:product@infuseai.io) if you have any questions, feedback, or
need help trying out PipeRider! :heart:**


# Welcome to py4dgeo

[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)
[![GitHub Workflow Status](https://img.shields.io/github/actions/workflow/status/3dgeo-heidelberg/py4dgeo/ci.yml?branch=main)](https://github.com/3dgeo-heidelberg/py4dgeo/actions/workflows/ci.yml)
[![PyPI Release](https://img.shields.io/pypi/v/py4dgeo.svg)](https://pypi.org/project/py4dgeo)
[![Documentation Status](https://readthedocs.org/projects/py4dgeo/badge/)](https://py4dgeo.readthedocs.io/)
[![codecov](https://codecov.io/gh/3dgeo-heidelberg/py4dgeo/branch/main/graph/badge.svg)](https://codecov.io/gh/3dgeo-heidelberg/py4dgeo)

`py4dgeo` is a `C++` library with `Python` bindings for change analysis in multitemporal and 4D point clouds.

Topographic 3D/4D point clouds are omnipresent in geosciences, environmental, ecological and archaeological sciences, robotics, and many more fields and applications. Technology to capture such data using laser scanning and photogrammetric techniques have evolved into standard tools. Dense time series of topographic point clouds are becoming increasing available and require tools for automatic analysis. Moreover, methods considering the full 4D (3D space + time) data are being developed in research and need to be made available in an accessible way with flexible integration into existent workflows.

The **main objective** of `py4dgeo` is to bundle and provide different methods of 3D/4D change analysis in a dedicated, comprehensive Python library.
`py4dgeo` is designed as an international open source project that can be integrated into almost any 3D and GIS software in the geodata domain supporting Python, e.g. as plugins.

`py4dgeo` is under *ongoing active development*.
Below, you find a list of [provided methods](#methods-provided-by-py4dgeo).

## 🔨 Methods provided by py4dgeo

* **M3C2 algorithm** ([Lague et al., 2013](#-literature)) for bitemporal point cloud distance computation. The concept and algorithm is explained [in this tutorial by James Dietrich](https://youtu.be/xJql7h8M2_o), including usage in the graphical software [CloudCompare](www.cloudcompare.org).

* **M3C2-EP** (M3C2-EP; [Winiwarter et al., 2021](#-literature)) for statistical signal-noise separation in change analysis through error propagation. The concept and method are explained in full detail in the related paper.

* **4D objects-by-change** (4D-OBC; [Anders et al., 2021](#-literature)) for time series-based extraction of surface activities *[under active development]*. The concept and method are explained in this scientific talk:

* **Correspondence-driven plane-based M3C2** ([Zahs et al., 2022](#-literature)) for lower uncertainty in 3D topographic change quantification. The concept and method are explained in this scientific talk:

* **Point cloud registration**: Py4dgeo supports to calculate and apply affine transformations to point clouds using a standard ICP implementations. More ICP methods are currently being implemented - stay tuned!

## 🎮 Examples

### Demo notebooks using methods provided by py4dgeo
|                                                                                |   |
|------------------------------------------------------------------------------------------------------------------------------------------------------|---|
| [![Example 1](img/m3c2_change_analysis_thumbnail.png)](https://nbviewer.org/github/3dgeo-heidelberg/py4dgeo/blob/main/demo/m3c2-change_analysis.ipynb)  | [![Example 2](img/m3c2ep_change_analysis_thumbnail.png)](https://nbviewer.org/github/3dgeo-heidelberg/py4dgeo/blob/main/demo/m3c2ep_change_analysis.ipynb)  |
| [![Example 3](img/4dobc_extraction_thumbnail.png)](https://nbviewer.org/github/3dgeo-heidelberg/py4dgeo/blob/main/demo/4dobc-change_analysis.ipynb)  | [![Example 4](img/kmeans_clustering_thumbnail.png)](https://nbviewer.org/github/3dgeo-heidelberg/py4dgeo/blob/main/demo/kmeans_clustering_of_time_series.ipynb)  |
| [![Example 5](img/pca_thumbnail.png)](https://nbviewer.org/github/3dgeo-heidelberg/py4dgeo/blob/main/demo/principal_component_analysis_of_time_series.ipynb)  |  [![Example 6](img/cd-pb-m3c2_change_analysis_thumbnail.png)](https://nbviewer.org/github/3dgeo-heidelberg/py4dgeo/blob/main/jupyter/pbm3c2.ipynb) |
[![Example 7](img/standard_icp_thumbnail.png)](https://nbviewer.org/github/3dgeo-heidelberg/py4dgeo/blob/main/demo/registration_standard_ICP.ipynb) |

## 💻 Installation

### Prerequisites

Using py4dgeo requires the following software installed:

* 64-bit Python `>= 3.8` (32-bit installations might cause trouble during installation of dependencies)

In order to build the package from source, the following tools are also needed.

* A C++17-compliant compiler
* CMake `>= 3.9`
* Doxygen (optional, documentation building is skipped if missing)

### Installing py4dgeo

The preferred way of installing `py4dgeo` is using `pip`.

### Installing the release version using pip

`py4dgeo` can be installed using `pip` to obtain the [current release](https://pypi.org/project/py4dgeo/):

```
python -m pip install py4dgeo
```

### Building from source using pip

The following sequence of commands is used to build `py4dgeo` from source:

```
git clone --recursive https://github.com/3dgeo-heidelberg/py4dgeo.git
cd py4dgeo
python -m pip install -v --editable .
```

The `--editable` flag allows you to change the Python sources of `py4dgeo` without
reinstalling the package. The `-v` flag enables verbose output which gives you
detailed information about the compilation process that you should include into
potential bug reports. To recompile the C++ source, please run `pip install` again.
In order to enable multi-threading on builds from source, your compiler toolchain
needs to support `OpenMP`.

If you want to contribute to the library's development you should also install
its additional Python dependencies for testing and documentation building:

```
python -m pip install -r requirements-dev.txt
```

### Setting up py4dgeo using Docker

Additionally, `py4dgeo` provides a Docker image that allows to explore
the library using JupyterLab. The image can be locally built and run with
the following commands:

```
docker build -t py4dgeo:latest .
docker run -t -p 8888:8888 py4dgeo:latest
```

## 🐍 Documentation of software usage

As a starting point, please have a look to the [Jupyter Notebooks](jupyter) available in the repository and find the `py4dgeo` documentation [on readthedocs](https://py4dgeo.readthedocs.io/en/latest/intro.html).

## 🌐 Published test data

If you are looking for data to test different methods, consider the following open data publications:

### Hourly TLS point clouds of a sandy beach

Vos et al. (2022): [https://doi.org/10.1038/s41597-022-01291-9](https://doi.org/10.1038/s41597-022-01291-9).

### By-weekly TLS point clouds of an Alpine rock glacier

Zahs et al. (2022): [https://doi.org/10.11588/data/TGSVUI](https://doi.org/10.11588/data/TGSVUI).

## 📑 Citation
Please cite py4dgeo when using it in your research and reference the appropriate release version.

```
article{py4dgeo,
author = {py4dgeo Development Core Team}
title = {py4dgeo: library for change analysis in 4D point clouds},
journal = {},
year = {2022},
number = {},
volume = {},
doi = {},
url = {https://github.com/3dgeo-heidelberg/py4dgeo},
}
 ```

## 💟 Funding / Acknowledgements
The initial software development was supported by the [**Scientific Software Center (SSC)**](https://ssc.iwr.uni-heidelberg.de/) in the Open Call 2021. The scientific software project is further supported by the research projects [**CharAct4D**](https://www.uni-heidelberg.de/charact4d) and [**AImon5.0**](https://www.uni-heidelberg.de/aimon).

## 🔔 Contact / Bugs / Feature Requests

You think you have found a bug or have specific request for a new feature? Please open a new issue in the online code repository on Github. Also for general questions please use the issue system.

Scientific requests can be directed to the [3DGeo Research Group Heidelberg](https://uni-heidelberg.de/3dgeo) and its respective members.

## 📜 License

See [LICENSE.md](LICENSE.md).

## 📚 Literature

* Anders, K., Winiwarter, L., Mara, H., Lindenbergh, R., Vos, S.E. & Höfle, B. (2021): Fully automatic spatiotemporal segmentation of 3D LiDAR time series for the extraction of natural surface changes. ISPRS Journal of Photogrammetry and Remote Sensing, 173, pp. 297-308. DOI: [10.1016/j.isprsjprs.2021.01.015](https://doi.org/10.1016/j.isprsjprs.2021.01.015).
* Lague, D., Brodu, N., & Leroux, J. (2013). Accurate 3D comparison of complex topography with terrestrial laser scanner: Application to the Rangitikei canyon (N-Z). ISPRS Journal of Photogrammetry and Remote Sensing, 82, pp. 10-26. DOI: [10.1016/j.isprsjprs.2013.04.009](https://doi.org/10.1016/j.isprsjprs.2013.04.009).
* Winiwarter, L., Anders, K., Höfle, B. (2021): M3C2-EP: Pushing the limits of 3D topographic point cloud change detection by error propagation. ISPRS Journal of Photogrammetry and Remote Sensing, 178, pp. 240–258. DOI: [10.1016/j.isprsjprs.2021.06.011](https://doi.org/10.1016/j.isprsjprs.2021.06.011).
* Zahs, V., Winiwarter, L., Anders, K., Williams, J.G., Rutzinger, M. & Höfle, B. (2022): Correspondence-driven plane-based M3C2 for lower uncertainty in 3D topographic change quantification. ISPRS Journal of Photogrammetry and Remote Sensing, 183, pp. 541-559. DOI: [10.1016/j.isprsjprs.2021.11.018](https://doi.org/10.1016/j.isprsjprs.2021.11.018).


# Paxml (aka Pax)

Pax is a framework to configure and run machine learning experiments on top of Jax.
## Quickstart
### Setting up a Cloud TPU VM

We refer to
[this page](https://cloud.google.com/tpu/docs/users-guide-tpu-vm#managing_tpus)
for more exhaustive documentation about starting a Cloud TPU project. The
following command is sufficient to create a Cloud TPU VM with 8 cores from a
corp machine.

```bash
export ZONE=us-central2-b
export VERSION=tpu-vm-v4-base
export PROJECT=<your-project>
export ACCELERATOR=v4-8
export TPU_NAME=paxml

#create a TPU VM
gcloud compute tpus tpu-vm create $TPU_NAME \
--zone=$ZONE --version=$VERSION \
--project=$PROJECT \
--accelerator-type=$ACCELERATOR
```

If you are using TPU Pod slices, please refer to [this guide](https://cloud.google.com/tpu/docs/jax-pods). Run all the commands from a local machine using [gcloud](https://cloud.google.com/sdk/docs/install) with the `--worker=all` option:

```bash
gcloud compute tpus tpu-vm ssh $TPU_NAME --zone=$ZONE \
--worker=all --command="<commmands>"
```

The following quickstart sections assume you run on a single-host TPU, so you can ssh to the VM and run the commands there.

```bash
gcloud compute tpus tpu-vm ssh $TPU_NAME --zone=$ZONE
```

### Installing Pax

After ssh-ing the VM, you can install the paxml stable release from PyPI, or the dev version from github.

For installing the stable release from PyPI (https://pypi.org/project/paxml/):

```bash
python3 -m pip install -U pip
python3 -m pip install paxml jax[tpu] \
-f https://storage.googleapis.com/jax-releases/libtpu_releases.html
```

If you encounter issues with transitive dependencies and you are using the native Cloud TPU VM environment, please navigate to the corresponding release branch rX.Y.Z and download `paxml/pip_package/requirements.txt`. This file includes the exact versions of all transitive dependencies needed in the native Cloud TPU VM environment, in which we build/test the corresponding release.

```bash
git clone -b rX.Y.Z https://github.com/google/paxml
pip install --no-deps -r paxml/paxml/pip_package/requirements.txt
```

For installing the dev version from github, and for the ease of editing code:

```bash
# install the dev version of praxis first
git clone https://github.com/google/praxis
pip install -e praxis
git clone https://github.com/google/paxml
pip install -e paxml
pip install "jax[tpu]" -f https://storage.googleapis.com/jax-releases/libtpu_releases.html
```

### Run a test model
```bash
# example model using pjit (SPMD)
python3 .local/lib/python3.8/site-packages/paxml/main.py \
--exp=tasks.lm.params.lm_cloud.LmCloudSpmd2BLimitSteps \
--job_log_dir=gs://<your-bucket>

# example model using pmap
python3 .local/lib/python3.8/site-packages/paxml/main.py \
--exp=tasks.lm.params.lm_cloud.LmCloudTransformerAdamLimitSteps \
--job_log_dir=gs://<your-bucket> \
--pmap_use_tensorstore=True
```
### Documentations
Please visit our [docs folder](https://github.com/google/paxml/tree/main/paxml/docs) for documentations and Jupyter Notebook tutorials. Please see the following section for instructions of running Jupyter Notebooks on a Cloud TPU VM.

### Run a notebook
You can run the [example notebooks](paxml/docs/tutorials) in the TPU VM in which you just installed paxml.
####Steps to enable a notebook in a `v4-8`

1. ssh in TPU VM with port forwarding
`gcloud compute tpus tpu-vm ssh $TPU_NAME --project=$PROJECT_NAME    --zone=$ZONE    --ssh-flag="-4 -L 8080:localhost:8080"`

2. install jupyter notebook on the TPU vm and downgrade markupsafe
 ```
 pip install notebook
 pip install markupsafe==2.0.1
 ```
3. export `jupyter` path
`export PATH=/home/$USER/.local/bin:$PATH`

4. scp the [example notebooks](paxml/docs/tutorials) to your TPU VM
`gcloud compute tpus tpu-vm scp  $TPU_NAME:<path inside TPU> <local path of the notebooks>   --zone=$ZONE --project=$PROJECT`

5. start jupyter notebook from the TPU VM and note the token generated by jupyter notebook
 `jupyter notebook --no-browser --port=8080`

6. then in your local browser go to: http://localhost:8080/ and enter the token provided

Note: In case you need to start using a second notebook while the first notebook is still occupying the TPUs, you can run
`pkill -9 python3`
to free up the TPUs.

### Run on GPU

Note: NVIDIA has released an updated version of Pax with H100 FP8 support and broad GPU performance improvements. Please visit the [NVIDIA Rosetta](https://github.com/NVIDIA/JAX-Toolbox/tree/main/rosetta/rosetta/projects/pax) repository for more details and usage instructions.

#### Run PGLE workflow on GPU

The Profile Guided Latency Estimator (PGLE) workflow measures the actual running time
of compute and collectives, the the profile information is fed back into XLA compiler
for a better scheduling decision.

The workflow to use the Profile Guided Latency Estimator workflow in XLA/GPU is:

- 1. Run your workload once, with async collectives and latency hiding scheduler enabled.

You could do so by setting:

```bash
export XLA_FLAGS="--xla_gpu_enable_latency_hiding_scheduler=true"
```

- 2. Collect and post process a profile by using JAX profiler, saving the extracted instruction latencies into a binary protobuf file.

```python
import os
from etils import epath
import jax
from jax.experimental import profiler as exp_profiler

# Define your profile directory
profile_dir = 'gs://my_bucket/profile'
jax.profiler.start_trace(profile_dir)

# run your workflow
# for i in range(10):
#   train_step()

# Stop trace
jax.profiler.stop_trace()
profile_dir = epath.Path(profile_dir)
directories = profile_dir.glob('plugins/profile/*/')
directories = [d for d in directories if d.is_dir()]
rundir = directories[-1]
logging.info('rundir: %s', rundir)

# Post process the profile
fdo_profile = exp_profiler.get_profiled_instructions_proto(os.fspath(rundir))

# Save the profile proto to a file.
dump_dir = rundir / 'profile.pb'
dump_dir.parent.mkdir(parents=True, exist_ok=True)
dump_dir.write_bytes(fdo_profile)

```

After this step, you will get a `profile.pb` file under the `rundir` printed in the code.

- 3. Run the workload again feeding that file into the compilation.

You need to pass the `profile.pb` file to the `--xla_gpu_pgle_profile_file_or_directory_path` flag.

```bash
 export XLA_FLAGS="--xla_gpu_enable_latency_hiding_scheduler=true --xla_gpu_pgle_profile_file_or_directory_path=/path/to/profile/profile.pb"
```

To enable logging in the XLA and check if the profile is good, set the logging level to include `INFO`:

```bash
export TF_CPP_MIN_LOG_LEVEL=0
```

Run the real workflow, if you found these loggings in the running log, it means the profiler is used in the latency hiding scheduler:

```
2023-07-21 16:09:43.551600: I external/xla/xla/service/gpu/gpu_hlo_schedule.cc:478] Using PGLE profile from /tmp/profile/plugins/profile/2023_07_20_18_29_30/profile.pb
2023-07-21 16:09:43.551741: I external/xla/xla/service/gpu/gpu_hlo_schedule.cc:573] Found profile, using profile guided latency estimator
```

### FAQs

1. Pax runs on Jax, you can find details on running Jax jobs on Cloud TPU [here](https://cloud.google.com/tpu/docs/run-calculation-jax), also you can find details on running Jax jobs on a Cloud TPU pod [here](https://cloud.google.com/tpu/docs/jax-pods)

2. If you run into dependency errors, please refer to the `requirements.txt` file in the branch corresponding to the stable release you are installing.
For e.g., for the [stable release 0.4.0](https://pypi.org/project/paxml/0.4.0/) use branch `r0.4.0` and refer to the [requirements.txt](https://github.com/google/paxml/blob/r0.4.0/paxml/pip_package/requirements.txt) for the exact versions of the dependencies used for the stable release.

## Example Convergence Runs
Here are some sample convergence runs on [c4 dataset](https://www.tensorflow.org/datasets/catalog/c4).

### 1B model on c4 dataset

You can run a `1B` params model on c4 dataset on TPU `v4-8`using the config `C4Spmd1BAdam4Replicas`from [c4.py](paxml/tasks/lm/params/c4.py) as follows:

```bash
python3 .local/lib/python3.8/site-packages/paxml/main.py \
--exp=tasks.lm.params.c4.C4Spmd1BAdam4Replicas \
--job_log_dir=gs://<your-bucket>
```
You can observe loss curve and `log perplexity` graph as follows:

### 16B model on c4 dataset

You can run a `16B` params model on c4 dataset on TPU `v4-64`using the config `C4Spmd16BAdam32Replicas`from [c4.py](paxml/tasks/lm/params/c4.py) as follows:

```bash
python3 .local/lib/python3.8/site-packages/paxml/main.py \
--exp=tasks.lm.params.c4.C4Spmd16BAdam32Replicas \
--job_log_dir=gs://<your-bucket>
```
You can observe loss curve and `log perplexity` graph as follows:

### GPT3-XL model on c4 dataset

You can run the GPT3-XL model on c4 dataset on TPU `v4-128`using the config `C4SpmdPipelineGpt3SmallAdam64Replicas`from [c4.py](paxml/tasks/lm/params/c4.py) as follows:

```bash
python3 .local/lib/python3.8/site-packages/paxml/main.py \
--exp=tasks.lm.params.c4.C4SpmdPipelineGpt3SmallAdam64Replicas \
--job_log_dir=gs://<your-bucket>
```
You can observe loss curve and `log perplexity` graph as follows:

## Benchmark on Cloud TPU v4
The [PaLM](https://arxiv.org/abs/2204.02311) paper introduced an efficiency metric called Model FLOPs Utilization (MFU). This is measured as the ratio of the observed throughput (in, for example, tokens per second for a language model) to the theoretical maximum throughput of a system harnessing 100% of peak FLOPs. It differs from other ways of measuring compute utilization because it doesn’t include FLOPs spent on activation rematerialization during the backward pass, meaning that efficiency as measured by MFU translates directly into end-to-end training speed.

To evaluate the MFU of a key class of workloads on TPU v4 Pods with Pax, we carried out an in-depth benchmark campaign on a series of decoder-only Transformer language model (GPT) configurations that range in size from billions to trillions of parameters on the [c4 dataset](https://www.tensorflow.org/datasets/catalog/c4). The following graph shows the training efficiency using the "weak scaling" pattern where we grew the model size in
proportion to the number of chips used.

## Pax on Multislice
The multislice configs in this repo refer to
[1. Singlie slice configs](https://github.com/google/paxml/blob/main/paxml/tasks/lm/params/c4.py)
for syntax / model architecture
and
[2. MaxText repo](https://github.com/google/maxtext)
for config values.

We provide example runs under c4_multislice.py` as a starting point for Pax on multislice.

### Setting up Cloud TPU VMs using Queued Resources

We refer to
[this page](https://cloud.google.com/tpu/docs/queued-resources)
for more exhaustive documentation about using Queued Resources for a multi-slice Cloud TPU project. The
following shows the steps needed to set up TPUs for running example configs in this repo.

```bash
export ZONE=us-central2-b
export VERSION=tpu-vm-v4-base
export PROJECT=<your-project>
export ACCELERATOR=v4-128 # or v4-384 depending on which config you run
```

Say, for running `C4Spmd22BAdam2xv4_128` on 2 slices of v4-128, you'd need to set up TPUs the following way:

```bash
export TPU_PREFIX=<your-prefix> # New TPUs will be created based off this prefix
export QR_ID=$TPU_PREFIX
export NODE_COUNT=<number-of-slices> # 1, 2, or 4 depending on which config you run

#create a TPU VM
gcloud alpha compute tpus queued-resources create $QR_ID --accelerator-type=$ACCELERATOR --runtime-version=tpu-vm-v4-base --node-count=$NODE_COUNT --node-prefix=$TPU_PREFIX
```

### Installing Pax

The setup commands described earlier need to be run on ALL workers in ALL slices. You can 1) ssh into each worker and each slice individually; or 2) use for loop with `--worker=all` flag as the following command.

```bash
for ((i=0; i<$NODE_COUNT; i++))
do
gcloud compute tpus tpu-vm ssh $TPU_PREFIX-$i --zone=us-central2-b --worker=all --command="pip install paxml && pip install orbax==0.1.1 && pip install \"jax[tpu]\" -f https://storage.googleapis.com/jax-releases/libtpu_releases.html"
done
```

### Run a test multislice model
In order to run the multislice configs, open the same number of terminals as your $NODE_COUNT. For our experiments on 2 slices(`C4Spmd22BAdam2xv4_128`), open two terminals. Then, run each of these commands individually from each terminal.

From Terminal 0, run training command for slice 0 as follows:

```bash
export TPU_PREFIX=<your-prefix>
export EXP_NAME=C4Spmd22BAdam2xv4_128
export LIBTPU_INIT_ARGS=\"--xla_jf_spmd_threshold_for_windowed_einsum_mib=0 --xla_tpu_spmd_threshold_for_allgather_cse=10000 --xla_enable_async_all_gather=true --jax_enable_async_collective_offload=true --xla_tpu_enable_latency_hiding_scheduler=true TPU_MEGACORE=MEGACORE_DENSE\"
gcloud compute tpus tpu-vm ssh $TPU_PREFIX-0 --zone=us-central2-b --worker=all \
--command="LIBTPU_INIT_ARGS=$LIBTPU_INIT_ARGS JAX_USE_PJRT_C_API_ON_TPU=1 \
python3 /home/yooh/.local/lib/python3.8/site-packages/paxml/main.py \
--exp=tasks.lm.params.c4_multislice.${EXP_NAME} --job_log_dir=gs://<your-bucket>"
```

From Terminal 1, concurrently run training command for slice 1 as follows:

```bash
export TPU_PREFIX=<your-prefix>
export EXP_NAME=C4Spmd22BAdam2xv4_128
export LIBTPU_INIT_ARGS=\"--xla_jf_spmd_threshold_for_windowed_einsum_mib=0 --xla_tpu_spmd_threshold_for_allgather_cse=10000 --xla_enable_async_all_gather=true --jax_enable_async_collective_offload=true --xla_tpu_enable_latency_hiding_scheduler=true TPU_MEGACORE=MEGACORE_DENSE\"
gcloud compute tpus tpu-vm ssh $TPU_PREFIX-1 --zone=us-central2-b --worker=all \
--command="LIBTPU_INIT_ARGS=$LIBTPU_INIT_ARGS JAX_USE_PJRT_C_API_ON_TPU=1 \
python3 /home/yooh/.local/lib/python3.8/site-packages/paxml/main.py \
--exp=tasks.lm.params.c4_multislice.${EXP_NAME} --job_log_dir=gs://<your-bucket>"
```

### MaxText to Pax
This table covers details on how the MaxText variable names have been translated to Pax.

Note that MaxText has a "scale" which is multiplied to several parameters (base_num_decoder_layers, base_emb_dim, base_mlp_dim, base_num_heads) for final values.

Another thing to mention is while Pax covers DCN and ICN MESH_SHAPE as an array, in MaxText there are separate variables of data_parallelism, fsdp_parallelism and tensor_parallelism for DCN and ICI. Since these values are set as 1 by default, only the variables with value greater than 1 are recorded in this translation table.

That is, `ICI_MESH_SHAPE = [ici_data_parallelism, ici_fsdp_parallelism, ici_tensor_parallelism]` and `DCN_MESH_SHAPE = [dcn_data_parallelism, dcn_fsdp_parallelism, dcn_tensor_parallelism]`

| Pax C4Spmd22BAdam2xv4_128 | | MaxText 2xv4-128.sh | | (after scale is applied) |
|---------------------------|--------------|---------------------------------|----------|-----------------------:|
| | | scale (applied to next 4 variables) | 3 | |
| NUM_LAYERS | 48 | base_num_decoder_layers | 16 | 48 |
| MODEL_DIMS | 6144 | base_emb_dim | 2048 | 6144 |
| HIDDEN_DIMS | 24576 | MODEL_DIMS * 4 (= base_mlp_dim) | 8192 | 24576 |
| NUM_HEADS | 24 | base_num_heads | 8 | 24 |
| DIMS_PER_HEAD | 256 | head_dim | 256 | |
| PERCORE_BATCH_SIZE | 16 | per_device_batch_size | 16 | |
| MAX_SEQ_LEN | 1024 | max_target_length | 1024 | |
| VOCAB_SIZE | 32768 | vocab_size | 32768 | |
| FPROP_DTYPE | jnp.bfloat16 | dtype | bfloat16 | |
| USE_REPEATED_LAYER | TRUE | | | |
| SUMMARY_INTERVAL_STEPS | 10 | | | |
| ICI_MESH_SHAPE | [1, 64, 1] | ici_fsdp_parallelism | 64 | |
| DCN_MESH_SHAPE | [2, 1, 1] | dcn_data_parallelism | 2 | |

# Data inputs

## Intro

Input is an instance of the `BaseInput`
class for getting data into model for train/eval/decode.

```python
class BaseInput:

  def get_next(self):
    pass

  def reset(self):
    pass
```

It acts like an iterator: `get_next()` returns a `NestedMap`, where each field
is a numerical array with batch size as its leading dimension.

Each input is configured by a subclass of `BaseInput.HParams`.
In this page, we use `p` to denote an instance of a `BaseInput.Params`, and it
instantiates to `input`.

## Multihost infeed

In Pax, data is always multihost: Each Jax process will have a separate,
independent `input` instantiated. Their params will have different
`p.infeed_host_index`, set automatically by Pax.

Hence, the local batch size seen on each host is `p.batch_size`, and the global
batch size is `(p.batch_size * p.num_infeed_hosts)`. One will often see
`p.batch_size` set to `jax.local_device_count() * PERCORE_BATCH_SIZE`.

Due to this multihost nature, `input` must be sharded properly.

For training, each `input` must never emit identical batches, and for eval on a
finite dataset, each `input` must terminate after the same number of batches.
The best solution is to have the input implementation properly shard the data,
such that each `input` on different hosts do not overlap. Failing that, one can
also use different random seed to avoid duplicate batches during training.

## Input for eval data

`input.reset()` is never called on training data, but it can for eval (or
decode) data.

For each eval (or decode) run, Pax fetches `N` batches from `input` by calling
`input.get_next()` `N` times. The number of batches used, `N`, can be a fixed
number specified by user, via `p.eval_loop_num_batches`; or `N` can be dynamic
(`p.eval_loop_num_batches=None`) i.e. we call `input.get_next()` until we
exhaust all of its data (by raising `StopIteration` or `tf.errors.OutOfRange`).

If `p.reset_for_eval=True`, `p.eval_loop_num_batches` is ignored and `N` is
determined dynamically as the number of batches to exhaust the data. In this
case, `p.repeat` should be set to False, as doing otherwise would lead to
infinite decode/eval.

If `p.reset_for_eval=False`, Pax will fetch `p.eval_loop_num_batches` batches.
This should be set with `p.repeat=True` so that data are not prematurely
exhausted.

Note that LingvoEvalAdaptor inputs require `p.reset_for_eval=True`.

|                          | `N`: static             | `N`: dynamic            |
| ------------------------ | ----------------------- | ----------------------- |
| `p.reset_for_eval=True`  | Each eval run uses the  | One epoch per eval run. |
:                          : first `N` batches. Not  : `eval_loop_num_batches` :
:                          : supported yet.          : is ignored. Input must  :
:                          :                         : be finite               :
:                          :                         : (`p.repeat=False`)      :
| `p.reset_for_eval=False` | Each eval run uses      | Not supported.          |
:                          : non-overlapping `N`     :                         :
:                          : batches on a rolling    :                         :
:                          : basis, according to     :                         :
:                          : `eval_loop_num_batches` :                         :
:                          : . Input must repeat     :                         :
:                          : indefinitely            :                         :
:                          : (`p.repeat=True`) or    :                         :
:                          : otherwise may raise     :                         :
:                          : exception               :                         :

If running decode/eval on exactly one epoch (i.e. when `p.reset_for_eval=True`),
the input must handle sharding correctly such that each shard raises at the same
step after exactly the same number of batches are produced. This usually means
that the input must pad the eval data. This is done automatically
by`SeqIOInput` and `LingvoEvalAdaptor` (see more below).

### Eval metrics

For the majority of inputs, we only ever call `get_next()` on them to get
batches of data. One type of eval data is an exception to this, where "how to
compute metrics" is also defined on the input object as well.

This is only supported with `SeqIOInput` that defines some canonical eval
benchmark. Specifically, Pax uses `predict_metric_fns` and `score_metric_fns()` defined on the SeqIO task to compute
eval metrics (although Pax does not depend on SeqIO evaluator directly).

## Best practices

When a model uses multiple inputs, either between train/eval or different
training data between pretraining/finetuning, users must ensure that the
tokenizers used by the inputs are identical, especially when importing different
inputs implemented by others.

Users can sanity check the tokenizers by decoding some ids with
`input.ids_to_strings()`.

It's always a good idea to sanity check the data by looking at a few batches.
Users can easily reproduce the param in a colab and inspect the data:

```python
p = ... # specify the intended input param
inp = p.Instantiate()
b = inp.get_next()
print(b)
```

Training data typically should not use a fixed random seed. This is because if
the training job is preempted, training data will start to repeat itself. In
particular, for Lingvo inputs, we recommend setting `p.input.file_random_seed =
0` for training data.

To test for whether sharding is handled correctly, users can manually set
different values for `p.num_infeed_hosts, p.infeed_host_index` and see whether
the instantiated inputs emit different batches.

## Input types

Pax supports 3 types of inputs: SeqIO, Lingvo, and custom.

### SeqIO

`SeqIOInput` can be used to import datasets.

SeqIO inputs handle correct sharding and padding of eval data automatically.

### Lingvo

`LingvoInputAdaptor` can be used to import datasets.

The input is fully delegated to the Lingvo implementation, which may or may not
handle sharding automatically.

For GenericInput based Lingvo input implementation using a fixed
`packing_factor`, we recommend to use
`LingvoInputAdaptorNewBatchSize` to specify a bigger batch size for the inner Lingvo input and put the desired
(usually much smaller) batch size on `p.batch_size`.

For eval data, we recommend using
`LingvoEvalAdaptor` to handle sharding and padding for running eval over one epoch.

### Custom

Custom subclass of `BaseInput`. Users implement their own subclass, typically
with `tf.data` or SeqIO.

Users can also inherit an existing input class to only customize post processing
of batches. For example:

```python
class MyInput(base_input.LingvoInputAdaptor):

  def get_next(self):
    batch = super().get_next()
    # modify batch: batch.new_field = ...
    return batch
```

# Key Pax components

## Hyperparameters

Hyperparameters are an important part of defining models and configuring
experiments.

To integrate better with Python tooling, Pax/Praxis uses a pythonic
dataclass based configuration style for hyperparameters.

```python
class Linear(base_layer.BaseLayer):
  """Linear layer without bias."""

  class HParams(BaseHParams):
    """Associated hyperparams for this layer class.

    Attributes:
      input_dims: Depth of the input.
      output_dims: Depth of the output.
    """
    input_dims: int = 0
    output_dims: int = 0
```

### Nesting

It's also possible to nest HParams dataclasses, in the example below, the
linear_tpl attribute is a nested Linear.HParams.

```python
class FeedForward(base_layer.BaseLayer):
  """Feedforward layer with activation."""

  class HParams(BaseHParams):
    """Associated hyperparams for this layer class.

    Attributes:
      input_dims: Depth of the input.
      output_dims: Depth of the output.
      has_bias: Adds bias weights or not.
      linear_tpl: Linear layer params.
      activation_tpl: Activation layer params.
    """
    input_dims: int = 0
    output_dims: int = 0
    has_bias: bool = True
    linear_tpl: BaseHParams = sub_config_field(Linear.HParams)
    activation_tpl: activations.BaseActivation.HParams = sub_config_field(
        ReLU.HParams)
```

## Layers

A Layer represents an arbitrary function possibly with trainable parameters. A
Layer can contain other Layers as children. Layers are the essential building
blocks of models. Layers inherit from the Flax nn.Module.

Typically layers define two methods:

### setup

This method creates trainable weights and child layers.

### fprop

This method defines the forward propagation function, computing some output
based on the inputs. Additionally, fprop might add summaries or track auxiliary
losses.

### Fiddle and Shared layers
Fiddle is an open-sourced Python-first configuration library
designed for ML applications. Pax/Praxis supports interoperability with Fiddle
Config/Partial(s) and some advanced features like eager error checking and
shared parameters.

```python
fdl_config = Linear.HParams.config(input_dims=1, output_dims=1)

# A typo.
fdl_config.input_dimz = 31337  # Raises an exception immediately to catch typos fast!

fdl_partial = Linear.HParams.partial(input_dims=1)
```

Using Fiddle, layers can be configured to be shared (eg: instantiated only once
with shared trainable weights).

## Model

A model defines solely the network, typically a collection of Layers and defines
interfaces for interacting with the model such as decoding, etc.

Some example base models
include:

*   LanguageModel
*   SequenceModel
*   ClassificationModel

## Task

A Task contains one more more Models and Learner/Optimizers. The simplest Task
subclass is a `SingleTask` which requires the following Hparams:

```python
  class HParams(base_task.BaseTask.HParams):
    """Task parameters.

    Attributes:
      name: Name of this task object, must be a valid identifier.
      model: The underlying JAX model encapsulating all the layers.
      train: HParams to control how this task should be trained.
      metrics: A BaseMetrics aggregator class to determine how metrics are
         computed.
      loss_aggregator: A LossAggregator aggregator class to derermine how the
        losses are aggregated (e.g single or MultiLoss)
      vn: HParams to control variational noise.
```

## Releases
PyPI Version | Commit
------------ | ----------------------------------------
0.1.0        | 546370f5323ef8b27d38ddc32445d7d3d1e4da9a

    Copyright 2022 Google LLC

    Licensed under the Apache License, Version 2.0 (the "License");
    you may not use this file except in compliance with the License.
    You may obtain a copy of the License at

        https://www.apache.org/licenses/LICENSE-2.0

    Unless required by applicable law or agreed to in writing, software
    distributed under the License is distributed on an "AS IS" BASIS,
    WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    See the License for the specific language governing permissions and
    limitations under the License.


# IREE LLVM Sandbox

DISCLAIMER: This is not an officially-supported Google project. It is a sandbox
for quick iteration and experimentation on projects related to the IREE project,
MLIR, and LLVM.

This repository contains experimental work by the IREE team closely related to
LLVM and MLIR, usually with the aim of upstreaming in some form. The main
project is at <https://github.com/google/iree>.

As an experimental project, build greenness, documentation, and polish are
likely to be minimal, as it instead prioritizes easy experimentation.

## License

Licensed under the Apache license with LLVM Exceptions. See [LICENSE](LICENSE)
for more information.

## Subprojects

The repository currently houses the following projects:

* The [Iterators](README-Iterators.md) dialect: database-style iterators for
  expressing computations on streams of data.
* The [Tuple](include/structured/Dialect/Tuple/): ops for manipulation of built-in tuples (used by the Iterators dialect).

## Build Instructions

This project builds as part of the LLVM External Projects facility (see
documentation for the `LLVM_EXTERNAL_PROJECTS` config setting).

It is left to the reader to adapt paths if deviating. We assume below that
projects are checked out to `$HOME/src`.

### Check out Project

In your `$HOME/src` directory, clone this project recursively:

```bash
git clone --recursive https://github.com/google/iree-llvm-sandbox
```

If you have cloned non-recursively already and every time a submodule is
updated, run the following command inside the cloned repository instead:

```bash
git submodule update --recursive --init
```

Define the following environment variables (adapted to your situation), ideally
making them permanent in your `$HOME/.bashrc` or in the `activate` script of
your Python virtual environment (see below):

```bash
export IREE_LLVM_SANDBOX_SOURCE_DIR=$HOME/src/iree-llvm-sandbox
export IREE_LLVM_SANDBOX_BUILD_DIR=${IREE_LLVM_SANDBOX_SOURCE_DIR}/build
```

### Python prerequisites

Create a virtual environment, activate it, and install the dependencies from
[`requirements.txt`](requirements.txt):

```bash
python -m venv ~/.venv/mlirdev
source ~/.venv/mlirdev/bin/activate
python -m pip install --upgrade pip
python -m pip install -r ${IREE_LLVM_SANDBOX_SOURCE_DIR}/requirements.txt
```

For details, see the documentation of the
[MLIR Python Bindings](https://mlir.llvm.org/docs/Bindings/Python/).

### Configure and build main project

Run the command below to set up the build system, possibly adapting it to your
needs. For example, you may choose not to compile `clang`, `clang-tools-extra`,
`lld`, and/or the examples to save compilation time, or use a different variant
than `Debug`.

```bash
cmake \
  -DPython3_EXECUTABLE=$(which python) \
  -DBUILD_SHARED_LIBS=ON \
  -DCMAKE_EXPORT_COMPILE_COMMANDS=TRUE \
  -DCMAKE_BUILD_TYPE=Debug \
  -DLLVM_ENABLE_PROJECTS="mlir;clang;clang-tools-extra" \
  -DLLVM_TARGETS_TO_BUILD="X86" \
  -DLLVM_ENABLE_ASSERTIONS=ON \
  -DLLVM_INCLUDE_TESTS=OFF \
  -DLLVM_INCLUDE_UTILS=ON \
  -DLLVM_INSTALL_UTILS=ON \
  -DLLVM_BUILD_EXAMPLES=ON \
  -DLLVM_EXTERNAL_PROJECTS=structured \
  -DLLVM_EXTERNAL_STRUCTURED_SOURCE_DIR=${IREE_LLVM_SANDBOX_SOURCE_DIR} \
  -DLLVM_ENABLE_LLD=ON \
  -DLLVM_CCACHE_BUILD=ON \
  -DMLIR_INCLUDE_INTEGRATION_TESTS=ON \
  -DMLIR_ENABLE_BINDINGS_PYTHON=ON \
  -DMLIR_ENABLE_PYTHON_BENCHMARKS=ON \
  -S${IREE_LLVM_SANDBOX_SOURCE_DIR}/third_party/llvm-project/llvm \
  -B${IREE_LLVM_SANDBOX_BUILD_DIR} \
  -G Ninja
```

To build, run:

```bash
cd ${IREE_LLVM_SANDBOX_BUILD_DIR} && ninja
```

## Using structured-opt

```bash
"${IREE_LLVM_SANDBOX_BUILD_DIR}"/bin/structured-opt --help
```

## Running tests

You can run all tests with the following command:

```bash
cd ${IREE_LLVM_SANDBOX_BUILD_DIR} && ninja
```

You may also use `lit` to run a subset of the tests. You may

```bash
lit -v ${IREE_LLVM_SANDBOX_BUILD_DIR}/test
lit -v ${IREE_LLVM_SANDBOX_BUILD_DIR}/test/Integration
lit -v ${IREE_LLVM_SANDBOX_BUILD_DIR}/test/Dialect/Iterators/map.mlir
```

## Diagnostics via LSP servers

The [MLIR LSP Servers](https://mlir.llvm.org/docs/Tools/MLIRLSP/) allows editors
to display as-you-type diagnostics, code navigation, and similar features. In
order to extend this functionality to the dialects from this repository, use
the following LSP server binaries:

```bash
${IREE_LLVM_SANDBOX_BUILD_DIR}/bin/mlir-proto-lsp-server
${IREE_LLVM_SANDBOX_BUILD_DIR}/bin/tblgen-lsp-server",
${IREE_LLVM_SANDBOX_BUILD_DIR}/bin/mlir-pdll-lsp-server
```

In VS Code, this is done via the `mlir.server_path`, `mlir.pdll_server_path`,
and `mlir.tablegen_server_path` properties in `settings.json`.

## Hashes of Interest

Repro for experimental results described in the
[arxiv paper](https://arxiv.org/abs/2202.03293):

```bash
git checkout 680c8160edb7aa13b621b28c221288624ebc37e4
echo Please update LLVM to $(cat pinned-llvm-version)
```

Hash before transitioning to schedule dialect only:

```bash
git checkout ea0e5ec37a4d73808e16926c0335cc21fde0286c
echo Please update LLVM to $(cat pinned-llvm-version)
```