  <picture>
    <source media="(prefers-color-scheme: dark)" srcset="https://github.com/flairNLP/fundus/blob/master/resources/logo/svg/logo_darkmode_with_font_and_clear_space.svg">
    <source media="(prefers-color-scheme: light)" srcset="https://github.com/flairNLP/fundus/blob/master/resources/logo/svg/logo_lightmode_with_font_and_clear_space.svg">

  </picture>

A very simple news crawler in Python.
Developed at Humboldt University of Berlin.

[Quick Start](#quick-start) | [Tutorials](#tutorials) | [News Sources](/docs/supported_publishers.md) | [Paper](https://arxiv.org/abs/2403.15279)

---

Fundus is:

* **A static news crawler.**
  Fundus lets you crawl online news articles with only a few lines of Python code!
  Be it from live websites or the CC-NEWS dataset.

* **An open-source Python package.**
  Fundus is built on the idea of building something together.
  We welcome your contribution to  help Fundus [grow](docs/how_to_contribute.md)!

## Quick Start

To install from pip, simply do:

```
pip install fundus
```

Fundus requires Python 3.8+.

## Example 1: Crawl a bunch of English-language news articles

Let's use Fundus to crawl 2 articles from publishers based in the US.

```python
from fundus import PublisherCollection, Crawler

# initialize the crawler for news publishers based in the US
crawler = Crawler(PublisherCollection.us)

# crawl 2 articles and print
for article in crawler.crawl(max_articles=2):
    print(article)
```

That's already it!

If you run this code, it should print out something like this:

```console
Fundus-Article:
- Title: "Feinstein's Return Not Enough for Confirmation of Controversial New [...]"
- Text:  "Democrats jammed three of President Joe Biden's controversial court nominees
          through committee votes on Thursday thanks to a last-minute [...]"
- URL:    https://freebeacon.com/politics/feinsteins-return-not-enough-for-confirmation-of-controversial-new-hampshire-judicial-nominee/
- From:   FreeBeacon (2023-05-11 18:41)

Fundus-Article:
- Title: "Northwestern student government freezes College Republicans funding over [...]"
- Text:  "Student government at Northwestern University in Illinois "indefinitely" froze
          the funds of the university's chapter of College Republicans [...]"
- URL:    https://www.foxnews.com/us/northwestern-student-government-freezes-college-republicans-funding-poster-critical-lgbtq-community
- From:   FoxNews (2023-05-09 14:37)
```

This printout tells you that you successfully crawled two articles!

For each article, the printout details:
- the "Title" of the article, i.e. its headline
- the "Text", i.e. the main article body text
- the "URL" from which it was crawled
- the news source it is "From"

## Example 2: Crawl a specific news source

Maybe you want to crawl a specific news source instead. Let's crawl news articles from Washington Times only:

```python
from fundus import PublisherCollection, Crawler

# initialize the crawler for The New Yorker
crawler = Crawler(PublisherCollection.us.TheNewYorker)

# crawl 2 articles and print
for article in crawler.crawl(max_articles=2):
    print(article)
```

## Example 3: Crawl articles from CC-NEWS

If you're not familiar with CC-NEWS, check out their [paper](https://paperswithcode.com/dataset/cc-news).

````python
from fundus import PublisherCollection, CCNewsCrawler

# initialize the crawler for news publishers based in the US
crawler = CCNewsCrawler(*PublisherCollection.us)

# crawl 2 articles and print
for article in crawler.crawl(max_articles=2):
  print(article)
````

## Tutorials

We provide **quick tutorials** to get you started with the library:

1. [**Tutorial 1: How to crawl news with Fundus**](docs/1_getting_started.md)
2. [**Tutorial 2: How to crawl articles from CC-NEWS**](docs/2_crawl_from_cc_news.md)
3. [**Tutorial 3: The Article Class**](docs/3_the_article_class.md)
4. [**Tutorial 4: How to filter articles**](docs/4_how_to_filter_articles.md)
5. [**Tutorial 5: Advanced topics**](docs/5_advanced_topics)
6. [**Tutorial 6: Logging**](docs/6_logging.md)

If you wish to contribute check out these tutorials:
1. [**How to contribute**](docs/how_to_contribute.md)
2. [**How to add a publisher**](docs/how_to_add_a_publisher.md)

## Currently Supported News Sources

You can find the publishers currently supported [**here**](/docs/supported_publishers.md).

Also: **Adding a new publisher is easy - consider contributing to the project!**

## Evaluation benchmark

Check out our evaluation [benchmark](https://github.com/dobbersc/fundus-evaluation).

| **Scraper** | **Precision**             | **Recall**                | **F1-Score**              |
|-------------|---------------------------|---------------------------|---------------------------|
| [Fundus](https://github.com/flairNLP/fundus)      | **99.89**±0.57 | 96.75±12.75    | **97.69**±9.75 |
| [Trafilatura](https://github.com/adbar/trafilatura) | 90.54±18.86    | 93.23±23.81    | 89.81±23.69    |
| [BTE](https://github.com/dobbersc/fundus-evaluation/blob/master/src/fundus_evaluation/scrapers/bte.py)         | 81.09±19.41    | **98.23**±8.61 | 87.14±15.48    |
| [jusText](https://github.com/miso-belica/jusText)     | 86.51±18.92    | 90.23±20.61    | 86.96±19.76    |
| [news-please](https://github.com/fhamborg/news-please) | 92.26±12.40    | 86.38±27.59    | 85.81±23.29    |
| [BoilerNet](https://github.com/dobbersc/fundus-evaluation/tree/master/src/fundus_evaluation/scrapers/boilernet)   | 84.73±20.82    | 90.66±21.05    | 85.77±20.28    |
| [Boilerpipe](https://github.com/kohlschutter/boilerpipe)  | 82.89±20.65    | 82.11±29.99    | 79.90±25.86    |

## Cite

Please cite the following [paper](https://arxiv.org/abs/2403.15279) when using Fundus or building upon our work:

```bibtex
@misc{dallabetta2024fundus,
      title={Fundus: A Simple-to-Use News Scraper Optimized for High Quality Extractions},
      author={Max Dallabetta and Conrad Dobberstein and Adrian Breiding and Alan Akbik},
      year={2024},
      eprint={2403.15279},
      archivePrefix={arXiv},
      primaryClass={cs.CL}
}
```

## Contact

Please email your questions or comments to [**Max Dallabetta**](mailto:max.dallabetta@googlemail.com?subject=[GitHub]%20Fundus)

## Contributing

Thanks for your interest in contributing! There are many ways to get involved;
start with our [contributor guidelines](docs/how_to_contribute.md) and then
check these [open issues](https://github.com/flairNLP/fundus/issues) for specific tasks.

## License

[MIT](LICENSE)


[![Python package](https://github.com/inductiva/inductiva/actions/workflows/python-package.yml/badge.svg)](https://github.com/inductiva/inductiva/actions/workflows/python-package.yml) [![Python version](https://img.shields.io/pypi/pyversions/inductiva.svg)](https://www.python.org/downloads/) [![Documentation](https://img.shields.io/badge/Documentation-blue.svg)](https://docs.inductiva.ai/en/latest/) [![Benchmarks](https://img.shields.io/badge/Benchmarks-blue.svg)](https://benchmarks.inductiva.ai)  ![coverage badge](https://raw.githubusercontent.com/inductiva/inductiva/Inductiva-badges/badges/cov.svg)

![inductiva_banner](https://github.com/inductiva/inductiva/assets/7538022/85569dd8-3beb-4ac3-8cbe-78e300a28fde)

# Inductiva: a Python package for scaling simulations on the Cloud

Welcome to the official Python library for the Inductiva API version **0.7**.
The Inductiva API allows running a set of open-source physical
simulators on the cloud, easily parallelizing simulations, each running
on hundreds of CPU cores.

Inductiva simplifies the complexities of cloud resource management, and software
configuration, offering a straightforward Python interface for running simulations
on state-of-the-art hardware. This allows scientists and engineers to focus their
time and energy on what matters: running simulations that solve real problems.

To find out more, check our [documentation](https://docs.inductiva.ai/).

If you have any questions or suggestions about the API please
[open an issue on the inductiva’s API Client GitHub repo](https://github.com/inductiva/inductiva/issues),
or contact us via [support@inductiva.ai](mailto:support@inductiva.ai).


# Shiny for Python

[![PyPI Latest Release](https://img.shields.io/pypi/v/shiny.svg)](https://pypi.org/project/shiny/)
[![Build status](https://img.shields.io/github/actions/workflow/status/posit-dev/py-shiny/pytest.yaml?branch=main)](https://img.shields.io/github/actions/workflow/status/posit-dev/py-shiny/pytest.yaml?branch=main)
[![Conda Latest Release](https://anaconda.org/conda-forge/shiny/badges/version.svg)](https://anaconda.org/conda-forge/shiny)
[![Supported Python versions](https://img.shields.io/pypi/pyversions/shiny)](https://pypi.org/project/shiny/)
[![License](https://img.shields.io/github/license/posit-dev/py-shiny)](https://github.com/posit-dev/py-shiny/blob/main/LICENSE)

Shiny for Python is the best way to build fast, beautiful web applications in Python. You can build quickly with Shiny and create simple interactive visualizations and prototype applications in an afternoon. But unlike other frameworks targeted at data scientists, Shiny does not limit your app's growth. Shiny remains extensible enough to power large, mission-critical applications.

To learn more about Shiny see the [Shiny for Python website](https://shiny.posit.co/py/). If you're new to the framework we recommend these resources:

- How [Shiny is different](https://posit.co/blog/why-shiny-for-python/) from Dash and Streamlit.

- How [reactive programming](https://shiny.posit.co/py/docs/reactive-programming.html) can help you build better applications.

- How to [use modules](https://shiny.posit.co/py/docs/workflow-modules.html) to efficiently develop large applications.

- Hosting applications for free on [shinyapps.io](https://shiny.posit.co/py/docs/deploy.html#deploy-to-shinyapps.io-cloud-hosting), [Hugging Face](https://shiny.posit.co/blog/posts/shiny-on-hugging-face/), or [Shinylive](https://shiny.posit.co/py/docs/shinylive.html).

## Join the conversation

If you have questions about Shiny for Python, or want to help us decide what to work on next, [join us on Discord](https://discord.gg/yMGCamUMnS).

## Getting started

To get started with shiny follow the [installation instructions](https://shiny.posit.co/py/docs/install.html) or just install it from pip.

```sh
pip install shiny
```

To install the latest development version:

```sh
# First install htmltools, then shiny
pip install https://github.com/posit-dev/py-htmltools/tarball/main
pip install https://github.com/posit-dev/py-shiny/tarball/main
```

You can create and run your first application with `shiny create`, the CLI will ask you which template you would like to use. You can either run the app with the Shiny extension, or call `shiny run app.py --reload --launch-browser`.

## Development

* Shinylive built from the `main` branch: https://posit-dev.github.io/py-shiny/shinylive/py/examples/
* API documentation for the `main` branch:
    * https://posit-dev.github.io/py-shiny/docs/api/express/
    * https://posit-dev.github.io/py-shiny/docs/api/core/

If you want to do development on Shiny for Python:

```sh
pip install -e ".[dev,test]"
```

Additionally, you can install pre-commit hooks which will automatically reformat and lint the code when you make a commit:

```sh
pre-commit install

# To disable:
# pre-commit uninstall
```


# TF-Keras: the pure-TensorFlow implementation of Keras

This repository hosts the development of the TF-Keras library. It is a pure
TensorFlow implementation of Keras, based on the legacy `tf.keras` codebase.

Note that the "main" version of Keras is now Keras 3 (formerly Keras Core),
which is a multi-backend implementation of Keras, supporting JAX,
PyTorch, and TensorFlow. Keras 3 is being developed at
[keras-team/keras](https://github.com/keras-team/keras).

---
## Support

You can ask questions and join the development discussion:

- In the [TensorFlow forum](https://discuss.tensorflow.org/).
- On the [Keras mailing list](https://groups.google.com/forum/#!forum/keras-users).

---

## Opening an issue

You can also post **bug reports and feature requests** (only)
in [GitHub issues](https://github.com/keras-team/tf-keras/issues).

---

## Opening a PR

We welcome contributions! Before opening a PR, please read
[our contributor guide](https://github.com/keras-team/tf-keras/blob/master/CONTRIBUTING.md),
and the [API design guideline](https://github.com/keras-team/governance/blob/master/keras_api_design_guidelines.md).


  # Errands

  Manage your tasks

  ![CI](https://github.com/mrvladus/Errands/actions/workflows/CI.yml/badge.svg)

## Features
- Add, remove, edit, complete tasks and sub-tasks
- Sync tasks with Nextcloud or other CalDAV providers
- Multiple task lists support
- Add accent color for each task
- Drag and Drop support
- Customizable UI

## Sync
Errands is using [python-caldav](https://github.com/python-caldav/caldav) library for syncing with Nextcloud Tasks and other CalDAV providers.

For now, only **Nextcloud** is supported and known to work well.
Other providers **may not work**. See [python-caldav  documentation](https://caldav.readthedocs.io/en/latest/#compatibility) for more info on compatibility.

## Install
### Flatpak
Errands is available as a Flatpak on Flathub:

It's the **only** supported verion.

### Build flatpak using GNOME Builder
1. Install [GNOME Builder](https://flathub.org/apps/org.gnome.Builder).
2. Click "Clone Repository" with `https://github.com/mrvladus/Errands.git` as the URL.
3. Click on the build button at the top.

## Contribute

### Report a bug
- Make sure you are using latest version from [flathub](https://flathub.org/apps/details/io.github.mrvladus.List).
- See log file at `$HOME/.var/app/io.github.mrvladus.List/data/errands/log.txt`
- Create new issue.
- Copy log if needed.
- Add steps to reproduce the bug if needed.

### Translate
To translate **Errands** to your language see **[TRANSLATIONS.md](TRANSLATIONS.md)**

### Package
The only officially supported format is **flatpak** on [Flathub](https://flathub.org/apps/details/io.github.mrvladus.List)!
Please, **do not package** it in other formats, but if you do, these are dependencies:
- `libadwaita >= 1.5`
- `libsecret`
- `libportal`
- `gtksourceview5`
- `PyGObject`
- `python3-caldav`
- `gnome-online-accounts (optional)`

### Donate
If you like this app, you can support its development. See **[DONATIONS.md](DONATIONS.md)**

## Code of conduct

Errands follows the GNOME project [Code of Conduct](https://wiki.gnome.org/Foundation/CodeOfConduct).


# Prompt flow

[![Python package](https://img.shields.io/pypi/v/promptflow)](https://pypi.org/project/promptflow/)
[![Python](https://img.shields.io/pypi/pyversions/promptflow.svg?maxAge=2592000)](https://pypi.python.org/pypi/promptflow/)
[![PyPI - Downloads](https://img.shields.io/pypi/dm/promptflow)](https://pypi.org/project/promptflow/)
[![CLI](https://img.shields.io/badge/CLI-reference-blue)](https://microsoft.github.io/promptflow/reference/pf-command-reference.html)
[![vsc extension](https://img.shields.io/visual-studio-marketplace/i/prompt-flow.prompt-flow?logo=Visual%20Studio&label=Extension%20)](https://marketplace.visualstudio.com/items?itemName=prompt-flow.prompt-flow)

[![Doc](https://img.shields.io/badge/Doc-online-green)](https://microsoft.github.io/promptflow/index.html)
[![Issue](https://img.shields.io/github/issues/microsoft/promptflow)](https://github.com/microsoft/promptflow/issues/new/choose)
[![Discussions](https://img.shields.io/github/discussions/microsoft/promptflow)](https://github.com/microsoft/promptflow/issues/new/choose)
[![CONTRIBUTING](https://img.shields.io/badge/Contributing-8A2BE2)](https://github.com/microsoft/promptflow/blob/main/CONTRIBUTING.md)
[![License: MIT](https://img.shields.io/github/license/microsoft/promptflow)](https://github.com/microsoft/promptflow/blob/main/LICENSE)

> Welcome to join us to make prompt flow better by
> participating [discussions](https://github.com/microsoft/promptflow/discussions),
> opening [issues](https://github.com/microsoft/promptflow/issues/new/choose),
> submitting [PRs](https://github.com/microsoft/promptflow/pulls).

**Prompt flow** is a suite of development tools designed to streamline the end-to-end development cycle of LLM-based AI applications, from ideation, prototyping, testing, evaluation to production deployment and monitoring. It makes prompt engineering much easier and enables you to build LLM apps with production quality.

With prompt flow, you will be able to:

- **Create and iteratively develop flow**
    - Create executable [flows](https://microsoft.github.io/promptflow/concepts/concept-flows.html) that link LLMs, prompts, Python code and other [tools](https://microsoft.github.io/promptflow/concepts/concept-tools.html) together.
    - Debug and iterate your flows, especially the [interaction with LLMs](https://microsoft.github.io/promptflow/concepts/concept-connections.html) with ease.
- **Evaluate flow quality and performance**
    - Evaluate your flow's quality and performance with larger datasets.
    - Integrate the testing and evaluation into your CI/CD system to ensure quality of your flow.
- **Streamlined development cycle for production**
    - Deploy your flow to the serving platform you choose or integrate into your app's code base easily.
    - (Optional but highly recommended) Collaborate with your team by leveraging the cloud version of [Prompt flow in Azure AI](https://learn.microsoft.com/en-us/azure/machine-learning/prompt-flow/overview-what-is-prompt-flow?view=azureml-api-2).

------

## Installation

To get started quickly, you can use a pre-built development environment. **Click the button below** to open the repo in GitHub Codespaces, and then continue the readme!

[![Open in GitHub Codespaces](https://github.com/codespaces/badge.svg)](https://codespaces.new/microsoft/promptflow?quickstart=1)

If you want to get started in your local environment, first install the packages:

Ensure you have a python environment, `python>=3.9, <=3.11` is recommended.

```sh
pip install promptflow promptflow-tools
```

## Quick Start ⚡

**Create a chatbot with prompt flow**

Run the command to initiate a prompt flow from a chat template, it creates folder named `my_chatbot` and generates required files within it:

```sh
pf flow init --flow ./my_chatbot --type chat
```

**Setup a connection for your API key**

For OpenAI key, establish a connection by running the command, using the `openai.yaml` file in the `my_chatbot` folder, which stores your OpenAI key (override keys and name with --set to avoid yaml file changes):

```sh
pf connection create --file ./my_chatbot/openai.yaml --set api_key=<your_api_key> --name open_ai_connection
```

For Azure OpenAI key, establish the connection by running the command, using the `azure_openai.yaml` file:

```sh
pf connection create --file ./my_chatbot/azure_openai.yaml --set api_key=<your_api_key> api_base=<your_api_base> --name open_ai_connection
```

**Chat with your flow**

In the `my_chatbot` folder, there's a `flow.dag.yaml` file that outlines the flow, including inputs/outputs, nodes,  connection, and the LLM model, etc

> Note that in the `chat` node, we're using a connection named `open_ai_connection` (specified in `connection` field) and the `gpt-35-turbo` model (specified in `deployment_name` field). The deployment_name filed is to specify the OpenAI model, or the Azure OpenAI deployment resource.

Interact with your chatbot by running: (press `Ctrl + C` to end the session)

```sh
pf flow test --flow ./my_chatbot --interactive
```

**Core value: ensuring "High Quality” from prototype to production**

Explore our [**15-minute tutorial**](examples/tutorials/flow-fine-tuning-evaluation/promptflow-quality-improvement.md) that guides you through prompt tuning ➡ batch testing ➡ evaluation, all designed to ensure high quality ready for production.

Next Step! Continue with the **Tutorial**  👇 section to delve deeper into prompt flow.

## Tutorial 🏃♂

Prompt flow is a tool designed to **build high quality LLM apps**, the development process in prompt flow follows these steps: develop a flow, improve the flow quality, deploy the flow to production.

### Develop your own LLM apps

#### VS Code Extension

We also offer a VS Code extension (a flow designer) for an interactive flow development experience with UI.

You can install it from the visualstudio marketplace.

#### Deep delve into flow development

[Getting started with prompt flow](./docs/how-to-guides/quick-start.md): A step by step guidance to invoke your first flow run.

### Learn from use cases

[Tutorial: Chat with PDF](https://github.com/microsoft/promptflow/blob/main/examples/tutorials/e2e-development/chat-with-pdf.md): An end-to-end tutorial on how to build a high quality chat application with prompt flow, including flow development and evaluation with metrics.
> More examples can be found [here](https://microsoft.github.io/promptflow/tutorials/index.html#samples). We welcome contributions of new use cases!

### Setup for contributors

If you're interested in contributing, please start with our dev setup guide: [dev_setup.md](./docs/dev/dev_setup.md).

Next Step! Continue with the **Contributing**  👇 section to contribute to prompt flow.

## Contributing

This project welcomes contributions and suggestions.  Most contributions require you to agree to a
Contributor License Agreement (CLA) declaring that you have the right to, and actually do, grant us
the rights to use your contribution. For details, visit https://cla.opensource.microsoft.com.

When you submit a pull request, a CLA bot will automatically determine whether you need to provide
a CLA and decorate the PR appropriately (e.g., status check, comment). Simply follow the instructions
provided by the bot. You will only need to do this once across all repos using our CLA.

This project has adopted the [Microsoft Open Source Code of Conduct](https://opensource.microsoft.com/codeofconduct/).
For more information see the [Code of Conduct FAQ](https://opensource.microsoft.com/codeofconduct/faq/) or
contact [opencode@microsoft.com](mailto:opencode@microsoft.com) with any additional questions or comments.

## Trademarks

This project may contain trademarks or logos for projects, products, or services. Authorized use of Microsoft
trademarks or logos is subject to and must follow
[Microsoft's Trademark & Brand Guidelines](https://www.microsoft.com/en-us/legal/intellectualproperty/trademarks/usage/general).
Use of Microsoft trademarks or logos in modified versions of this project must not cause confusion or imply Microsoft sponsorship.
Any use of third-party trademarks or logos are subject to those third-party's policies.

## Code of Conduct

This project has adopted the
[Microsoft Open Source Code of Conduct](https://opensource.microsoft.com/codeofconduct/).
For more information see the
[Code of Conduct FAQ](https://opensource.microsoft.com/codeofconduct/faq/)
or contact [opencode@microsoft.com](mailto:opencode@microsoft.com)
with any additional questions or comments.

## Data Collection

The software may collect information about you and your use of the software and
send it to Microsoft if configured to enable telemetry.
Microsoft may use this information to provide services and improve our products and services.
You may turn on the telemetry as described in the repository.
There are also some features in the software that may enable you and Microsoft
to collect data from users of your applications. If you use these features, you
must comply with applicable law, including providing appropriate notices to
users of your applications together with a copy of Microsoft's privacy
statement. Our privacy statement is located at
https://go.microsoft.com/fwlink/?LinkID=824704. You can learn more about data
collection and use in the help documentation and our privacy statement. Your
use of the software operates as your consent to these practices.

### Telemetry Configuration

Telemetry collection is on by default.

To opt out, please run `pf config set telemetry.enabled=false` to turn it off.

## License

Copyright (c) Microsoft Corporation. All rights reserved.

Licensed under the [MIT](LICENSE) license.


# Amazon Braket Python SDK

[![Latest Version](https://img.shields.io/pypi/v/amazon-braket-sdk.svg)](https://pypi.python.org/pypi/amazon-braket-sdk)
[![Supported Python Versions](https://img.shields.io/pypi/pyversions/amazon-braket-sdk.svg)](https://pypi.python.org/pypi/amazon-braket-sdk)
[![Build status](https://github.com/amazon-braket/amazon-braket-sdk-python/actions/workflows/python-package.yml/badge.svg?branch=main)](https://github.com/amazon-braket/amazon-braket-sdk-python/actions/workflows/python-package.yml)
[![codecov](https://codecov.io/gh/amazon-braket/amazon-braket-sdk-python/branch/main/graph/badge.svg?token=1lsqkZL3Ll)](https://codecov.io/gh/amazon-braket/amazon-braket-sdk-python)
[![Documentation Status](https://img.shields.io/readthedocs/amazon-braket-sdk-python.svg?logo=read-the-docs)](https://amazon-braket-sdk-python.readthedocs.io/en/latest/?badge=latest)
[![Stack Overflow](https://img.shields.io/badge/StackExchange-Ask%20questions-blue?logo=stackexchange)](https://quantumcomputing.stackexchange.com/questions/tagged/amazon-braket)

The Amazon Braket Python SDK is an open source library that provides a framework that you can use to interact with quantum computing hardware devices through Amazon Braket.

## Prerequisites
Before you begin working with the Amazon Braket SDK, make sure that you've installed or configured the following prerequisites.

### Python 3.9 or greater
Download and install Python 3.9 or greater from [Python.org](https://www.python.org/downloads/).

### Git
Install Git from https://git-scm.com/downloads. Installation instructions are provided on the download page.

### IAM user or role with required permissions
As a managed service, Amazon Braket performs operations on your behalf on the AWS hardware that is managed by Amazon Braket. Amazon Braket can perform only operations that the user permits. You can read more about which permissions are necessary in the AWS Documentation.

The Braket Python SDK should not require any additional permissions aside from what is required for using Braket. However, if you are using an IAM role with a path in it, you should grant permission for iam:GetRole.

To learn more about IAM user, roles, and policies, see [Adding and Removing IAM Identity Permissions](https://docs.aws.amazon.com/IAM/latest/UserGuide/access_policies_manage-attach-detach.html).

### Boto3 and setting up AWS credentials

Follow the installation [instructions](https://boto3.amazonaws.com/v1/documentation/api/latest/guide/quickstart.html) for Boto3 and setting up AWS credentials.

**Note:** Make sure that your AWS region is set to one supported by Amazon Braket. You can check this in your AWS configuration file, which is located by default at `~/.aws/config`.

### Configure your AWS account with the resources necessary for Amazon Braket
If you are new to Amazon Braket, onboard to the service and create the resources necessary to use Amazon Braket using the [AWS console](https://console.aws.amazon.com/braket/home ).

## Installing the Amazon Braket Python SDK

The Amazon Braket Python SDK can be installed with pip as follows:

```bash
pip install amazon-braket-sdk
```

You can also install from source by cloning this repository and running a pip install command in the root directory of the repository:

```bash
git clone https://github.com/amazon-braket/amazon-braket-sdk-python.git
cd amazon-braket-sdk-python
pip install .
```

### Check the version you have installed
You can view the version of the amazon-braket-sdk you have installed by using the following command:
```bash
pip show amazon-braket-sdk
```

You can also check your version of `amazon-braket-sdk` from within Python:

```
>>> import braket._sdk as braket_sdk
>>> braket_sdk.__version__
```

### Updating the Amazon Braket Python SDK
You can update the version of the amazon-braket-sdk you have installed by using the following command:
```bash
pip install amazon-braket-sdk --upgrade --upgrade-strategy eager
```

## Usage

### Running a circuit on an AWS simulator

```python
import boto3
from braket.aws import AwsDevice
from braket.circuits import Circuit

device = AwsDevice("arn:aws:braket:::device/quantum-simulator/amazon/sv1")

bell = Circuit().h(0).cnot(0, 1)
task = device.run(bell, shots=100)
print(task.result().measurement_counts)
```

The code sample imports the Amazon Braket framework, then defines the device to use (the SV1 AWS simulator). It then creates a Bell Pair circuit, executes the circuit on the simulator and prints the results of the hybrid job. This example can be found in `../examples/bell.py`.

### Running multiple quantum tasks at once

Many quantum algorithms need to run multiple independent circuits, and submitting the circuits in parallel can be faster than submitting them one at a time. In particular, parallel quantum task processing provides a significant speed up when using simulator devices. The following example shows how to run a batch of quantum tasks on SV1:

```python
circuits = [bell for _ in range(5)]
batch = device.run_batch(circuits, shots=100)
# The result of the first quantum task in the batch
print(batch.results()[0].measurement_counts)
```

### Running a hybrid job

```python
from braket.aws import AwsQuantumJob

job = AwsQuantumJob.create(
    device="arn:aws:braket:::device/quantum-simulator/amazon/sv1",
    source_module="job.py",
    entry_point="job:run_job",
    wait_until_complete=True,
)
print(job.result())
```
where `run_job` is a function in the file `job.py`.

The code sample imports the Amazon Braket framework, then creates a hybrid job with the entry point being the `run_job` function. The hybrid job creates quantum tasks against the SV1 AWS Simulator. The hybrid job runs synchronously, and prints logs until it completes. The complete example can be found in `../examples/job.py`.

### Available Simulators
Amazon Braket provides access to two types of simulators: fully managed simulators, available through the Amazon Braket service, and the local simulators that are part of the Amazon Braket SDK.

- Fully managed simulators offer high-performance circuit simulations. These simulators can handle circuits larger than circuits that run on quantum hardware. For example, the SV1 state vector simulator shown in the previous examples requires approximately 1 or 2 hours to complete a 34-qubit, dense, and square circuit (circuit depth = 34), depending on the type of gates used and other factors.
- The Amazon Braket Python SDK includes an implementation of quantum simulators that can run circuits on your local, classic hardware. For example the braket_sv local simulator is well suited for rapid prototyping on small circuits up to 25 qubits, depending on the hardware specifications of your Braket notebook instance or your local environment. An example of how to execute the quantum task locally is included in the repository  `../examples/local_bell.py`.

For a list of available simulators and their features, consult the [Amazon Braket Developer Guide](https://docs.aws.amazon.com/braket/latest/developerguide/braket-devices.html).

### Debugging logs

Quantum tasks sent to QPUs don't always run right away. To view quantum task status, you can enable debugging logs. An example of how to enable these logs is included in repo: `../examples/debug_bell.py`. This example enables quantum task logging so that status updates are continuously printed to the terminal after a quantum task is executed. The logs can also be configured to save to a file or output to another stream. You can use the debugging example to get information on the quantum tasks you submit, such as the current status, so that you know when your quantum task completes.

### Running a Quantum Algorithm on a Quantum Computer
With Amazon Braket, you can run your quantum circuit on a physical quantum computer.

The following example executes the same Bell Pair example described to validate your configuration on a Rigetti quantum computer.

```python
import boto3
from braket.circuits import Circuit
from braket.aws import AwsDevice

device = AwsDevice("arn:aws:braket:::device/qpu/rigetti/Aspen-8")

bell = Circuit().h(0).cnot(0, 1)
task = device.run(bell)
print(task.result().measurement_counts)
```

When you execute your task, Amazon Braket polls for a result. By default, Braket polls for 5 days; however, it is possible to change this by modifying the `poll_timeout_seconds` parameter in `AwsDevice.run`, as in the example below. Keep in mind that if your polling timeout is too short, results may not be returned within the polling time, such as when a QPU is unavailable, and a local timeout error is returned. You can always restart the polling by using `task.result()`.

```python
task = device.run(bell, poll_timeout_seconds=86400)  # 1 day
print(task.result().measurement_counts)
```

To select a quantum hardware device, specify its ARN as the value of the `device_arn` argument. A list of available quantum devices and their features can be found in the [Amazon Braket Developer Guide](https://docs.aws.amazon.com/braket/latest/developerguide/braket-devices.html).

**Important** Quantum tasks may not run immediately on the QPU. The QPUs only execute quantum tasks during execution windows. To find their execution windows, please refer to the [AWS console](https://console.aws.amazon.com/braket/home) in the "Devices" tab.

## Sample Notebooks
Sample Jupyter notebooks can be found in the [amazon-braket-examples](https://github.com/amazon-braket/amazon-braket-examples/) repo.

## Braket Python SDK API Reference Documentation

The API reference, can be found on [Read the Docs](https://amazon-braket-sdk-python.readthedocs.io/en/latest/).

**To generate the API Reference HTML in your local environment**

To generate the HTML, first change directories (`cd`) to position the cursor in the `amazon-braket-sdk-python` directory. Then, run the following command to generate the HTML documentation files:

```bash
pip install tox
tox -e docs
```

To view the generated documentation, open the following file in a browser:
`../amazon-braket-sdk-python/build/documentation/html/index.html`

## Testing

This repository has both unit and integration tests.

To run the tests, make sure to install test dependencies first:

```bash
pip install -e "amazon-braket-sdk-python[test]"
```

### Unit Tests

To run the unit tests:

```bash
tox -e unit-tests
```

You can also pass in various pytest arguments to run selected tests:

```bash
tox -e unit-tests -- your-arguments
```

For more information, please see [pytest usage](https://docs.pytest.org/en/stable/usage.html).

To run linters and doc generators and unit tests:

```bash
tox
```

or if your machine can handle multithreaded workloads, run them in parallel with:

```bash
tox -p auto
```

### Integration Tests

First, configure a profile to use your account to interact with AWS. To learn more, see [Configure AWS CLI](https://docs.aws.amazon.com/cli/latest/userguide/cli-chap-configure.html).

After you create a profile, use the following command to set the `AWS_PROFILE` so that all future commands can access your AWS account and resources.

```bash
export AWS_PROFILE=YOUR_PROFILE_NAME
```
To run the integration tests for local hybrid jobs, you need to have Docker installed and running. To install Docker follow these instructions: [Install Docker](https://docs.docker.com/get-docker/)

Run the tests:

```bash
tox -e integ-tests
```

As with unit tests, you can also pass in various pytest arguments:

```bash
tox -e integ-tests -- your-arguments
```

## Support

### Issues and Bug Reports

If you encounter bugs or face issues while using the SDK, please let us know by posting
the issue on our [Github issue tracker](https://github.com/amazon-braket/amazon-braket-sdk-python/issues/).
For other issues or general questions, please ask on the [Quantum Computing Stack Exchange](https://quantumcomputing.stackexchange.com/questions/ask?Tags=amazon-braket).

### Feedback and Feature Requests

If you have feedback or features that you would like to see on Amazon Braket, we would love to hear from you!
[Github issues](https://github.com/amazon-braket/amazon-braket-sdk-python/issues/) is our preferred mechanism for collecting feedback and feature requests, allowing other users
to engage in the conversation, and +1 issues to help drive priority.

### Code contributors

[![Contributors](https://contrib.rocks/image?repo=amazon-braket/amazon-braket-sdk-python)](https://github.com/amazon-braket/amazon-braket-sdk-python/graphs/contributors)

## License
This project is licensed under the Apache-2.0 License.


![Supported Python versions](https://img.shields.io/badge/python-3.8+-blue.svg)
[![Twitter](https://img.shields.io/twitter/follow/al3xn3ff?label=al3x_n3ff&style=social)](https://twitter.com/intent/follow?screen_name=al3x_n3ff)
[![Twitter](https://img.shields.io/twitter/follow/_zblurx?label=_zblurx&style=social)](https://twitter.com/intent/follow?screen_name=_zblurx)
[![Twitter](https://img.shields.io/twitter/follow/MJHallenbeck?label=MJHallenbeck&style=social)](https://twitter.com/intent/follow?screen_name=MJHallenbeck)
[![Twitter](https://img.shields.io/twitter/follow/mpgn_x64?label=mpgn_x64&style=social)](https://twitter.com/intent/follow?screen_name=mpgn_x64)

🚩 This is the open source repository of NetExec maintained by a community of passionate people
# NetExec - The Network Execution Tool

This project was initially created in 2015 by @byt3bl33d3r, known as CrackMapExec. In 2019 @mpgn_x64 started maintaining the project for the next 4 years, adding a lot of great tools and features. In September 2023 he retired from maintaining the project.

Along with many other contributors, we (NeffIsBack, Marshall-Hallenbeck, and zblurx) developed new features, bug fixes, and helped maintain the original project CrackMapExec.
During this time, with both a private and public repository, community contributions were not easily merged into the project. The 6-8 month discrepancy between the code bases caused many development issues and heavily reduced community-driven development.
With the end of mpgn's maintainer role, we (the remaining most active contributors) decided to maintain the project together as a fully free and open source project under the new name **NetExec** 🚀
Going forward, our intent is to maintain a community-driven and maintained project with regular updates for everyone to use.

<p align="center">
  <!-- placeholder for nxc logo-->
</p>

You are on the **latest up-to-date** repository of the project NetExec (nxc) ! 🎉

- 🚧 If you want to report a problem, open an [Issue](https://github.com/Pennyw0rth/NetExec/issues)
- 🔀 If you want to contribute, open a [Pull Request](https://github.com/Pennyw0rth/NetExec/pulls)
- 💬 If you want to discuss, open a [Discussion](https://github.com/Pennyw0rth/NetExec/discussions)

## Official Discord Channel

If you don't have a Github account, you can ask your questions on Discord!

[![NetExec](https://discordapp.com/api/guilds/1148685154601160794/widget.png?style=banner3)](https://discord.gg/pjwUTQzg8R)

# Documentation, Tutorials, Examples
See the project's [wiki](https://netexec.wiki/) (in development) for documentation and usage examples

# Installation
Please see the installation instructions on the [wiki](https://netexec.wiki/getting-started/installation) (in development)

## Linux
```
sudo apt install pipx git
pipx ensurepath
pipx install git+https://github.com/Pennyw0rth/NetExec
```
# Development
Development guidelines and recommendations in development

# Acknowledgments
All the hard work and development over the years from everyone in the CrackMapExec project

# Code Contributors
Awesome code contributors of NetExec:

[![](https://github.com/mpgn.png?size=50)](https://github.com/mpgn)
[![](https://github.com/Marshall-Hallenbeck.png?size=50)](https://github.com/Marshall-Hallenbeck)
[![](https://github.com/zblurx.png?size=50)](https://github.com/zblurx)
[![](https://github.com/NeffIsBack.png?size=50)](https://github.com/NeffIsBack)
[![](https://github.com/Hackndo.png?size=50)](https://github.com/Hackndo)
[![](https://github.com/XiaoliChan.png?size=50)](https://github.com/XiaoliChan)


# ExLlamaV2

ExLlamaV2 is an inference library for running local LLMs on modern consumer GPUs.

## New in v0.1.0+:

- ExLlamaV2 now supports paged attention via [Flash Attention](https://github.com/Dao-AILab/flash-attention) 2.5.7+
- New generator with dynamic batching, smart prompt caching, K/V cache deduplication and simplified API

![alt_text](doc/dynamic_gen.gif)

## Dynamic generator

The dynamic generator supports all inference, sampling and speculative decoding features of the previous two
generators, consolidated into one API (with the exception of FP8 cache, though the Q4 cache mode is supported and
performs better anyway, see [here](doc/qcache_eval.md).)

The generator is explained in detail [here](doc/dynamic.md).

- Single generation:
  ```python
  output = generator.generate(prompt = "Hello, my name is", max_new_tokens = 200)
  ```
- Batched generation:
    ```python
    outputs = generator.generate(
        prompt = [
            "Hello, my name is",
            "Once upon a time,",
            "Large language models are",
        ],
        max_new_tokens = 200
    )
    ```
- Streamed generation with `asyncio`:
    ```python
    job = ExLlamaV2DynamicJobAsync(
        generator,
        input_ids = tokenizer.encode("You can lead a horse to water"),
        banned_strings = ["make it drink"],
        gen_settings = ExLlamaV2Sampler.Settings.greedy(),
        max_new_tokens = 200
    )
    async for result in job:
        text = result.get("text", "")
        print(text, end = "")
    ```
See the full, updated examples [here](https://github.com/turboderp/exllamav2/tree/master/examples).

## Performance

Some quick tests to compare performance with ExLlama V1. There may be more performance optimizations in the future,
and speeds will vary across GPUs, with slow CPUs still being a potential bottleneck:

| Model      | Mode         | Size  | grpsz | act | 3090Ti  | 4090        |
|------------|--------------|-------|-------|-----|---------|-------------|
| Llama      | GPTQ         | 7B    | 128   | no  | 177 t/s | **198** t/s |
| Llama      | GPTQ         | 13B   | 128   | no  | 109 t/s | **111** t/s |
| Llama      | GPTQ         | 33B   | 128   | yes | 44 t/s  | **48** t/s  |
| OpenLlama  | GPTQ         | 3B    | 128   | yes | 252 t/s | **283** t/s |
| CodeLlama  | EXL2 4.0 bpw | 34B   | -     | -   | 44 t/s  | **50** t/s  |
| Llama2     | EXL2 3.0 bpw | 7B    | -     | -   | 211 t/s | **245** t/s |
| Llama2     | EXL2 4.0 bpw | 7B    | -     | -   | 179 t/s | **207** t/s |
| Llama2     | EXL2 5.0 bpw | 7B    | -     | -   | 159 t/s | **170** t/s |
| Llama2     | EXL2 2.5 bpw | 70B   | -     | -   | 33 t/s  | **37** t/s  |
| TinyLlama  | EXL2 3.0 bpw | 1.1B  | -     | -   | 623 t/s | **730** t/s |
| TinyLlama  | EXL2 4.0 bpw | 1.1B  | -     | -   | 560 t/s | **643** t/s |

## How to

To install from the repo you'll need the CUDA Toolkit and either gcc on Linux or (Build Tools for) Visual Studio
on Windows). Also make sure you have an appropriate version of [PyTorch](https://pytorch.org/get-started/locally/), then run:

```sh
git clone https://github.com/turboderp/exllamav2
cd exllamav2
pip install -r requirements.txt
pip install .

python test_inference.py -m <path_to_model> -p "Once upon a time,"
# Append the '--gpu_split auto' flag for multi-GPU inference
```

A simple console chatbot is included. Run it with:

```sh
python examples/chat.py -m <path_to_model> -mode llama -gs auto
```

The `-mode` argument chooses the prompt format to use. `raw` will produce a simple chatlog-style chat that works with base
models and various other finetunes. Run with `-modes` for a list of all available prompt formats. You can also provide
a custom system prompt with `-sp`.

## Integration and APIs

- [TabbyAPI](https://github.com/theroyallab/tabbyAPI/) is a FastAPI-based server that provides an OpenAI-style web API
compatible with [SillyTavern](https://sillytavernai.com/) and other frontends.

- [ExUI](https://github.com/turboderp/exui) is a simple, standalone single-user web UI that serves an ExLlamaV2 instance
directly with chat and notebook modes.

- [text-generation-webui](https://github.com/oobabooga/text-generation-webui) supports ExLlamaV2 through the **exllamav2**
and **exllamav2_HF** loaders.

- [lollms-webui](https://github.com/ParisNeo/lollms-webui) supports ExLlamaV2 through the exllamav2 binding.

## Installation

### Method 1: Install from source

To install the current dev version, clone the repo and run the setup script:

```sh
git clone https://github.com/turboderp/exllamav2
cd exllamav2
pip install -r requirements.txt
pip install .
```

By default this will also compile and install the Torch C++ extension (`exllamav2_ext`) that the library relies on.
You can skip this step by setting the `EXLLAMA_NOCOMPILE` environment variable:

```sh
EXLLAMA_NOCOMPILE= pip install .
```

This will install the "JIT version" of the package, i.e. it will install the Python components without building the
C++ extension in the process. Instead, the extension will be built the first time the library is used, then cached in
`~/.cache/torch_extensions` for subsequent use.

### Method 2: Install from release (with prebuilt extension)

Releases are available [here](https://github.com/turboderp/exllamav2/releases), with prebuilt wheels that contain the extension binaries. Make sure to grab
the right version, matching your platform, Python version (`cp`) and CUDA version. Crucially, you must also match
the prebuilt wheel with your PyTorch version, since the Torch C++ extension ABI breaks with every new version of
PyTorch.

Either download an appropriate wheel or install directly from the appropriate URL:

```sh
pip install https://github.com/turboderp/exllamav2/releases/download/v0.0.12/exllamav2-0.0.12+cu121-cp311-cp311-linux_x86_64.whl
```

The `py3-none-any.whl` version is the JIT version which will build the extension on first launch. The `.tar.gz` file
can also be installed this way, and it will build the extension while installing.

### Method 3: Install from PyPI

A PyPI package is available as well. This is the same as the JIT version (see above). It can be installed with:

```sh
pip install exllamav2
```

## EXL2 quantization

ExLlamaV2 supports the same 4-bit GPTQ models as V1, but also a new "EXL2" format. EXL2 is based on the same
optimization method as GPTQ and supports 2, 3, 4, 5, 6 and 8-bit quantization. The format allows for mixing quantization
levels within a model to achieve any average bitrate between 2 and 8 bits per weight.

Moreover, it's possible to apply multiple quantization levels to each linear layer, producing something akin to sparse
quantization wherein more important weights (columns) are quantized with more bits. The same remapping trick that lets
ExLlama work efficiently with act-order models allows this mixing of formats to happen with little to no impact on
performance.

Parameter selection is done automatically by quantizing each matrix multiple times, measuring the quantization
error (with respect to the chosen calibration data) for each of a number of possible settings, per layer. Finally, a
combination is chosen that minimizes the maximum quantization error over the entire model while meeting a target
average bitrate.

In my tests, this scheme allows Llama2 70B to run on a single 24 GB GPU with a 2048-token context, producing coherent
and mostly stable output with 2.55 bits per weight. 13B models run at 2.65 bits within 8 GB of VRAM, although currently
none of them uses GQA which effectively limits the context size to 2048. In either case it's unlikely that the model
will fit alongside a desktop environment. For now.

[![chat_screenshot](doc/llama2_70b_chat_thumb.png)](doc/llama2_70b_chat.png)
[![chat_screenshot](doc/codellama_13b_instruct_thumb.png)](doc/codellama_13b_instruct.png)

### Conversion

A script is provided to quantize models. Converting large models can be somewhat slow, so be warned. The conversion
script and its options are explained in [detail here](doc/convert.md)

### Evaluation

A number of evaluaion scripts are provided. See [here](doc/eval.md) for details.

### Community

A test community is provided at https://discord.gg/NSFwVuCjRq
Quanting service free of charge is provided at #bot test. The computation is generiously provided by the Bloke powered by Lambda labs.

### HuggingFace repos

- I've uploaded a few EXL2-quantized models to Hugging Face to play around with, [here](https://huggingface.co/turboderp).

- [LoneStriker](https://huggingface.co/LoneStriker) provides a large number of EXL2 models on Hugging Face.

- [bartowski](https://huggingface.co/bartowski) has some more EXL2 models on HF.


===================================================
django-content-editor -- Editing structured content
===================================================

`Documentation on Read the Docs <http://django-content-editor.readthedocs.org/en/latest/>`_

.. image:: https://github.com/matthiask/django-content-editor/actions/workflows/tests.yml/badge.svg
    :target: https://github.com/matthiask/django-content-editor/
    :alt: CI Status

.. image:: https://readthedocs.org/projects/django-content-editor/badge/?version=latest
    :target: https://django-content-editor.readthedocs.io/en/latest/?badge=latest
    :alt: Documentation Status


[![Build Status](https://img.shields.io/github/actions/workflow/status/fitbenchmarking/fitbenchmarking/release.yml?style=flat-square)](https://github.com/fitbenchmarking/fitbenchmarking/actions/workflows/release.yml)
[![Tests Status](https://img.shields.io/github/actions/workflow/status/fitbenchmarking/fitbenchmarking/main.yml?label=tests&style=flat-square)](https://github.com/fitbenchmarking/fitbenchmarking/actions/workflows/main.yml)
[![Documentation Status](https://img.shields.io/readthedocs/fitbenchmarking?style=flat-square)](https://fitbenchmarking.readthedocs.io/en/latest)
[![Coverage Status](https://img.shields.io/coveralls/github/fitbenchmarking/fitbenchmarking.svg?style=flat-square)](https://coveralls.io/github/fitbenchmarking/fitbenchmarking)
[![Chat](https://img.shields.io/badge/chat-CompareFitMinimizers-lightgrey.svg?style=flat-square&logo=slack)](https://slack.com/)
[![Zenodo](https://img.shields.io/badge/Zenodo-10.5281/zenodo.6597790-blue.svg?style=flat-square)](https://doi.org/10.5281/zenodo.6597790)
[![JOSS](https://img.shields.io/badge/JOSS-10.21105/joss.03127-brightgreen.svg?style=flat-square)](https://doi.org/10.21105/joss.03127)
# FitBenchmarking

FitBenchmarking is an open source tool for comparing different minimizers/fitting frameworks. FitBenchmarking is cross platform and we support Windows, Linux and Mac OS. For questions, feature requests or any other inquiries, please open an issue on GitHub, or send us an e-mail at support@fitbenchmarking.com.

- **Installation Instructions:** https://fitbenchmarking.readthedocs.io/en/latest/users/install_instructions/index.html
- **User Documentation & Example Usage:** https://fitbenchmarking.readthedocs.io/en/latest/users/index.html
- **Community Guidelines:** https://fitbenchmarking.readthedocs.io/en/latest/contributors/guidelines.html
- **Automated Tests:** Run via GitHub Actions, https://github.com/fitbenchmarking/fitbenchmarking/actions, and tests are documented at https://fitbenchmarking.readthedocs.io/en/latest/users/tests.html

The package is the result of a collaboration between STFC’s Scientific Computing Department and ISIS Neutron and Muon Facility and the Diamond Light Source. We also would like to acknowledge support from:

* EU SINE2020 WP-10, which received funding from the European Union’s Horizon2020 research and innovation programme under grant agreement No 654000.
* EPSRC Grant EP/M025179/1  Least Squares: Fit for the Future.
* The Ada Lovelace Centre (ALC). ALC is an integrated, cross-disciplinary data intensive science centre, for better exploitation of research carried out at our large scale National Facilities including the Diamond Light Source (DLS), the ISIS Neutron and Muon Facility, the Central Laser Facility (CLF) and the Culham Centre for Fusion Energy (CCFE).


A comprehensive deep learning framework for digital microscopy.

  Installation •
  Examples •
  Basics •
  Cite us •
  License

We provide tools to create physical simulations of optical systems, to generate and train neural network models, and to analyze experimental data.

# Installation

DeepTrack 2.1 requires at least python 3.6.

To install DeepTrack 2.1, open a terminal or command prompt and run:

    pip install deeptrack

If you have a very recent version of python, you may need to install numpy _before_ DeepTrack. This is a known issue with scikit-image.

### Updating to 2.1 from 2.0

If you are already using DeepTrack 2.0 (pypi version 0.x.x), updating to DeepTrack 2.1 (pypi version 1.x.x) is painless. If you have followed deprecation warnings, no change to your code is needed. There are two breaking changes:

- The deprecated operator `+` to chain features has been removed. It is now only possible using the `>>` operator.
- The deprecated operator `**` to duplicate a feature has been removed. It is now only possible using the `^` operator.

If you notice any other changes in behavior, please report it to us in the issues tab.

# Examples of applications using DeepTrack

DeepTrack is a general purpose deep learning framework for microscopy, meaning you can use it for any task you like. Here, we show some common applications!

 Single particle tracking

    Training a CNN-based single particle tracker using simulated data

    Unsupervised training of a single particle tracker using LodeSTAR

 Multi-particle tracking

     Training LodeSTAR to detect multiple cells from a single image

    Training a UNet-based multi-particle tracker using simulated data

 Particle tracing

      Training MAGIK to trace migrating cells

# Basics to learn DeepTrack 2.1

Everybody learns in different ways! Depending on your preferences, and what you want to do with DeepTrack, you may want to check out one or more of these resources.

## Getting-started guides

We have two separate series of notebooks which aims to teach you all you need to know to use DeepTrack to its fullest. The first is a set of six notebooks with a focus on the application.

1.   deeptrack_introduction_tutorial   gives an overview of how to use DeepTrack 2.1.
2.   tracking_particle_cnn_tutorial  demonstrates how to track a point particle with a convolutional neural network (CNN).
3.   tracking_particle_cnn_tutorial  demonstrates how to track multiple particles using a U-net.
4.   characterizing_aberrations_tutorial  demonstrates how to add and characterize aberrations of an optical device.
5.   distinguishing_particles_in_brightfield_tutorial  demonstrates how to use a U-net to track and distinguish particles of different sizes in brightfield microscopy.
6.   analyzing_video_tutorial  demonstrates how to create videos and how to train a neural network to analyze them.

The second series focuses on individual topics, introducing them in a natural order.

1.   Introducing how to create simulation pipelines and train models.
2.   Demonstrating data generators.
3.   Demonstrating how to customize models using layer-blocks.

## DeepTrack 2.1 in action

Additionally, we have seven more case studies which are less documented, but gives additional insight in how to use DeepTrack with real datasets

1. [MNIST](examples/paper-examples/1-MNIST.ipynb) classifies handwritted digits.
2. [single particle tracking](examples/paper-examples/2-single_particle_tracking.ipynb) tracks experimentally captured videos of a single particle. (Requires opencv-python compiled with ffmpeg to open and read a video.)
3. [single particle sizing](examples/paper-examples/3-particle_sizing.ipynb) extracts the radius and refractive index of particles.
4. [multi-particle tracking](examples/paper-examples/4-multi-molecule-tracking.ipynb) detects quantum dots in a low SNR image.
5. [3-dimensional tracking](examples/paper-examples/5-inline_holography_3d_tracking.ipynb) tracks particles in three dimensions.
6. [cell counting](examples/paper-examples/6-cell_counting.ipynb) counts the number of cells in fluorescence images.
7. [GAN image generation](examples/paper-examples/7-GAN_image_generation.ipynb) uses a GAN to create cell image from masks.

## Model-specific examples

We also have examples that are specific for certain models. This includes
- [*LodeSTAR*](examples/LodeSTAR) for label-free particle tracking.
- [*MAGIK*](deeptrack/models/gnns/) for graph-based particle linking and trace characterization.

## Documentation
The detailed documentation of DeepTrack 2.1 is available at the following link: https://softmatterlab.github.io/DeepTrack2/deeptrack.html

## Video Tutorials

Videos are currently being updated to match with the current version of DeepTrack.

## Cite us!
If you use DeepTrack 2.1 in your project, please cite us here:

```
Benjamin Midtvedt, Saga Helgadottir, Aykut Argun, Jesús Pineda, Daniel Midtvedt, Giovanni Volpe.
"Quantitative Digital Microscopy with Deep Learning."
Applied Physics Reviews 8 (2021), 011310.
https://doi.org/10.1063/5.0034891
```

See also:

<https://www.nature.com/articles/s41467-022-35004-y>:
```
Midtvedt, B., Pineda, J., Skärberg, F. et al.
"Single-shot self-supervised object detection in microscopy."
Nat Commun 13, 7492 (2022).
```

<https://arxiv.org/abs/2202.06355>:
```
Jesús Pineda, Benjamin Midtvedt, Harshith Bachimanchi, Sergio Noé, Daniel  Midtvedt, Giovanni Volpe,1 and  Carlo  Manzo
"Geometric deep learning reveals the spatiotemporal fingerprint ofmicroscopic motion."
arXiv 2202.06355 (2022).
```

<https://doi.org/10.1364/OPTICA.6.000506>:
```
Saga Helgadottir, Aykut Argun, and Giovanni Volpe.
"Digital video microscopy enhanced by deep learning."
Optica 6.4 (2019): 506-513.
```

<https://github.com/softmatterlab/DeepTrack.git>:
```
Saga Helgadottir, Aykut Argun, and Giovanni Volpe.
"DeepTrack." (2019)
```

## Funding

This work was supported by the ERC Starting Grant ComplexSwimmers (Grant No. 677511), the ERC Starting Grant MAPEI (101001267), and the Knut and Alice Wallenberg Foundation.


# Shoonya Backend

Repository for Shoonya's backend.

## Pre-requisites

The project was created using [Python 3.7](https://www.python.org/downloads/). All major dependencies along with the versions are listed in the `backend/deploy/requirements.txt` file.

## Installation

The installation and setup instructions have been tested on the following platforms:

It can be done in 2 ways:-
1) using docker by running all services in containers or
2) directly running the services on local one by one (like, celery and redis).

If your preference is 1 over 2 please be fixed to this environment and follow the steps below it:

- Docker
- Ubuntu 20.04 OR macOs

If you are using a different operating system, you will have to look at external resources (eg. StackOverflow) to correct any errors.

### Create a Virtual Environment

We recommend you to create a virtual environment to install all the dependencies required for the project.

```bash
python3 -m venv <YOUR-ENVIRONMENT-NAME>
source <YOUR-ENVIRONMENT-NAME>/bin/activate # this command may be different based on your OS

# Install dependencies
pip install -r deploy/requirements-dev.txt
```

### Environment file

To set up the environment variables needed for the project, run the following lines:
```bash
cp .env.example ./backend/.env
```

This creates an `.env` file at the root of the project. It is needed to make sure that the project runs correctly. Please go through the file and set the parameters according to your installation.

To create a new secret key, run the following commands (within the virtual environment):
```bash
# Open a Python shell
python backend/manage.py shell

>> from django.core.management.utils import get_random_secret_key
>> get_random_secret_key()
```

Paste the value you get there into the `.env` file.

#### Google Cloud Logging (Optional)

If Google Cloud Logging is being used, please follow these additional steps:

1. Install the `google-cloud-logging` library using the following command:
```bash
pip install google-cloud-logging
```
2. Follow the steps to create a Service Account from the following [Google Cloud Documentation Page](https://cloud.google.com/docs/authentication/production#create_service_account).   This will create a Service Account and generate a JSON Key for the Service Account.
3. Ensure that atleast the Project Logs Writer role (`roles/logging.logWriter`) is assigned to the created Service Account.
4. Add the `GOOGLE_APPLICATION_CREDENTIALS` variable to the `.env` file. This value of this variable should be the path to the JSON Key generated in Step 2. For example,

```bash
GOOGLE_APPLICATION_CREDENTIALS="/path/to/gcloud-key.json"
```

### Docker Installation

`cd` back to the root folder .Once inside, build the docker containers:

```bash
docker-compose -f docker-compose-local.yml build
```

To run the containers:

```bash
docker-compose -f docker-compose-local.yml up -d
```

To share the database with others, just share the postgres_data and the media folder with others.

### Run Migrations (required only for the first time running the project or if you make any changes in the models)
Run the following commands:
```bash
# Check if there are any pending migrations
docker-compose exec web python manage.py makemigrations

# Run all pending migrations
docker-compose exec web python manage.py migrate

# Create a superuser
docker-compose exec web python manage.py createsuperuser

```

If there were no errors, congratulations! The project is up and running.

If your preference is 2 over 1 please be fixed to this environment and follow the steps below it:

- Ubuntu 20.04 OR macOs

You can run the following script and each and every step for setting the codebase will be done directly. Please move to a folder in your local where you would like to store the code and run the script given below there:
```bash
os=$(uname)

if [ "$os" = "Linux" ] || [ "$os" = "Darwin" ]; then

git clone https://github.com/AI4Bharat/Shoonya-Backend.git
cd Shoonya-Backend
git checkout dev
git pull origin dev
cp .env.example ./backend/.env
cd backend
python3 -m venv  venv
source venv/bin/activate

pip install -r ./deploy/requirements.txt

new_secret_key=$(python3 -c "from django.core.management.utils import get_random_secret_key; print(get_random_secret_key())")

env_file=".env"
if sed --version 2>&1 | grep -q 'GNU sed'; then
  sed -i "/^SECRET_KEY=/d" "$env_file"
else
  sed -i.bak "/^SECRET_KEY=/d" "$env_file"
  rm -f "$env_file.bak"
fi

echo "SECRET_KEY='$new_secret_key'" >> "$env_file"

echo "New secret key has been generated and updated in $env_file"

else
  echo "Cannot run this script on: $os"
fi
  ```

### Running background tasks
Please install and run redis from https://redis.io/download/ on port 6379 before starting celery.

To run background tasks for project creation, we need to run the following command in the terminal. This has also been added into the `docker-compose.yml` file.
```bash
celery command - celery -A shoonya_backend.celery worker -l info
celery command - celery -A shoonya_backend.celery beat --loglevel=info
```

You can set use the celery to local by modifying CELERY_BROKER_URL = "redis://localhost:6379/0" in ./backend/shoonya_backend/settings.py.

We can set the concurrency and autoscale in the process as well to manage the number of worker processes in the background. Read more [here](https://stackoverflow.com/a/72366865/9757174).

The commands will be as follows
```bash
celery -A shoonya_backend.celery worker --concurrency=2 --loglevel=info
celery -A shoonya_backend.celery worker --autoscale=10,3 --loglevel=info
```

### Running Linters

In case you want to raise a PR, kindly run linters as specified below. You can install black by running pip install black and use `black`
To run `black` do:

```bash
black ./backend/
```

Happy Coding!!


# Apollo [![Build status](https://github.com/UWCS/apollo/actions/workflows/ci.yaml/badge.svg?branch=master)](https://github.com/UWCS/apollo/actions/workflows/ci.yaml)

Apollo is a [Discord](https://discordapp.com/) bot for the [University of Warwick Computing Society](https://uwcs.co.uk).
It is designed to augment our Discord server with a few of the user services available on our website.

### Installation

#### Dependencies

Apollo uses `pipenv` for dependency and venv management. Install it with `pip install pipenv`, and then run `pipenv install` to install all the required dependencies into a new virtual environment.

#### Development Setup

1. `pipenv install`, as above
2. Copy `config.example.yaml` to `config.yaml` and configure the fields.
3. Copy `alembic.example.ini` to `alembic.ini` and configure any fields you wish to change.
4. Set up the database by running migrations with `alembic upgrade head`.
   - The default database location is `postgresql+psycopg://apollo:apollo@localhost/apollo` (in `config.example.yaml`)
     This requires PostgreSQL to be installed, with a database called `apollo` and a user with name and password `apollo` with access to it.
   - For testing purposes, you may wish to change it to a locally stored file such as `sqlite:///apollo.sqlite3`.
   - Alternatively, see the instructions below for using Docker.
5. On the [Discord Developer Portal](https://discord.com/developers/), make sure that your bot has the required intents.
   - Currently, the Messages and Members intents are necessary.

Run Apollo using `pipenv run python apollo.py`

#### Using Docker

A Dockerfile and docker-compose are provided for easily running Apollo. Assuming you already have docker installed, run `docker compose up` to start both Apollo and a postgres database.

The compose file uses a read-only bind mount to mount `config.yaml` into the container at runtime, not at build time. Copy `config.example.yaml` to `config.yaml` and configure the fields so that compose can do this. You will need to change the database url to `postgresql+psycopg://apollo:apollo@db/apollo` if you wish to connect to the containerised database. Be sure to configure the rest of the fields too: you need a discord bot token.

The docker image builds `alembic.ini` into it by copying the example, as it is rare any values in this wish to be changed on a per-deployment basis.

When you first create the database container, you'll need to apply migrations:

1. Run `docker compose up -d` to start both services in the background. The python one won't work, but leave it running.
   - When changing any source files, the container will have to be rebuilt: `docker compose up --build`
   - For containers running in the background, you can get the outputs using `docker compose logs`
2. Run `docker compose exec apollo alembic upgrade head` to apply the database migrations.
3. Run `docker compose down && docker compose up` to restart both services with the migrations applied.
   - Apollo will now be running in the foreground

Migrations will need to be re-applied every time the database schema changes.

### Contributing
Check [our issues](https://github.com/UWCS/apollo/contribute) for ideas of what to [contribute](https://github.com/UWCS/apollo/contribute). If you want to see a particular feature that isn't mentioned, feel free to work on it (but please do make an issue beforehand, so we can discuss it)

See our wiki for a general overview of the project and and [Contributor Notes](https://techteam.uwcs.co.uk/en/apps/apollo#contributing) for project specific advice and utility.


# ORION
### Operational Routine for the Ingest and Output of Networks

This package takes data sets from various sources and converts them into Knowledge Graphs.

Each data source will go through the following pipeline before it can be included in a graph:

1. Fetch (retrieve an original data source)
2. Parse (convert the data source into KGX files)
3. Normalize (use normalization services to convert identifiers and ontology terms to preferred synonyms)
4. Supplement (add supplementary knowledge specific to that source)

To build a graph use a Graph Spec yaml file to specify the sources you want.

ORION will automatically run each data source specified through the necessary pipeline. Then it will merge the specified sources into a Knowledge Graph.

### Using ORION

Create a parent directory:
```
mkdir ~/ORION_root
```

Clone the code repository:
```
cd ~/ORION_root
git clone https://github.com/RobokopU24/ORION.git
```

Next create directories where data sources, graphs, and logs will be stored.

ORION_STORAGE - for storing data sources
ORION_GRAPHS - for storing knowledge graphs
ORION_LOGS - for storing logs

You can do this manually, or use the script indicated below to set up a standard configuration (Option 1 or 2).

Option 1: Create three directories and set environment variables specifying paths to the locations of those directories.
```
mkdir ~/ORION_root/storage/
export ORION_STORAGE=~/ORION_root/storage/

mkdir ~/ORION_root/graphs/
export ORION_GRAPHS=~/ORION_root/graphs/

mkdir ~/ORION_root/logs/
export ORION_LOGS=~/ORION_root/logs/
```

Option 2: Use this script to create the directories and set the environment variables:
```
cd ~/ORION_root/ORION/
source ./set_up_test_env.sh
```

Next create or select a Graph Spec yaml file where the content of knowledge graphs to be built will be specified.

Use either of the following options, but not both:

Note that running the setup script set_up_test_env.sh will perform Option 1 for you.

Option 1: ORION_GRAPH_SPEC - the name of a Graph Spec file located in the graph_specs directory of ORION
```
export ORION_GRAPH_SPEC=testing-graph-spec.yml
```
Option 2: ORION_GRAPH_SPEC_URL - a URL pointing to a Graph Spec file
```
export ORION_GRAPH_SPEC_URL=https://example.com/example-graph-spec.yml
```

To build a custom graph, alter the Graph Spec file. See the graph_specs directory for examples.

TODO: explain options available in the graph spec (normalization version, source data version can be specified)
```
graphs:
  - graph_id: Example_Graph_ID
    graph_name: Example Graph
    graph_description: This is a description of what is in the graph.
    output_format: neo4j
    sources:
      - source_id: Biolink
      - source_id: HGNC
```

Install Docker to create and run the necessary containers.

By default using docker-compose up will build every graph in your Graph Spec. It runs the command: python /ORION/Common/build_manager.py all.
```
docker-compose up
```
If you want to specify an individual graph you can override the default command with a graph id from your Spec.
```
docker-compose run --rm data_services python /ORION/Common/build_manager.py Example_Graph_ID
```
To run the ORION pipeline for a single data source, you can use:
```
docker-compose run --rm data_services python /ORION/Common/load_manager.py Example_Source
```
To see available arguments and a list of supported data sources:
```
python /ORION/Common/load_manager.py -h
```

### For Developers

To add a new data source to ORION, create a new parser. Each parser extends the SourceDataLoader interface in Common/loader_interface.py.

To implement the interface you will need to write a class that fulfills the following.

Set the class level variables for the source ID and provenance:
```
source_id: str = 'ExampleSourceID'
provenance_id: str = 'infores:example_source'
```

In initialization, call the parent init function first and pass the initialization arguments.
Then set the file names for the data file or files.
```
super().__init__(test_mode=test_mode, source_data_dir=source_data_dir)

self.data_file = 'example_file.gz'
OR
self.example_file_1 = 'example_file_1.csv'
self.example_file_2 = 'example_file_2.csv'
self.data_files = [self.example_file_1, self.example_file_2]
```

Note that self.data_path is set by the parent class and by default refers to a specific directory for the current version of that source in the storage directory.

Implement get_latest_source_version(). This function should return a string representing the latest available version of the source data.

Implement get_data(). This function should retrieve any source data files. The files should be stored with the file names specified by self.data_file or self.data_files. They should be saved in the directory specified by self.data_path.

Implement parse_data(). This function should parse the data files and populate lists of node and edge objects: self.final_node_list (kgxnode), self.final_edge_list (kgxedge).

Finally, add your source to the list of sources in Common/data_sources.py. The source ID string here should match the one specified in the new parser. Also add your source to the SOURCE_DATA_LOADER_CLASS_IMPORTS dictionary, mapping it to the new parser class.

Now you can use that source ID in a graph spec to include your new source in a graph, or as the source id using load_manager.py.

#### Testing and Troubleshooting

After you alter the codebase, or if you are experiencing issues or errors you may want to run tests:
```
docker-compose run --rm data_services pytest /ORION
```


  <picture>
    <source media="(prefers-color-scheme: dark)" srcset="/img/openMINDS_SANDS_logo_dark.png">
    <source media="(prefers-color-scheme: light)" srcset="/img/openMINDS_SANDS_logo_light.png">

  </picture>

# openMINDS_SANDS

[![GitHub][license-shield]][license-url]
[![GitHub contributors][contributors-shield]][contributors-url]

The **openMINDS_SANDS** repository hosts one of the metadata models of the **openMINDS metadata framework**. It defines modular metadata schemas for **s**patial **a**nchoring of **n**euroscience **d**ata **s**tructures.

To generally learn more about the openMINDS metadata framework please go to :arrow_right: [**ReadTheDocs**][docu-url].
Or explore directly the metadata schemas of the openMINDS_SANDS metadata model under :arrow_right: [**schema specifications**][docu-sands-url].

## How to contribute

The openMINDS metadata framework is an open-source project and community contributions are highly appreciated.
If you want to contribute please follow our :arrow_right: [**contribution guidelines**][contribution-url].

[contribution-url]: https://openminds-documentation.readthedocs.io/en/latest/shared/contribution_guidelines.html
[contributors-url]: https://github.com/openMetadataInitiative/openMINDS_SANDS/graphs/contributors
[contributors-shield]: https://img.shields.io/github/contributors/openMetadataInitiative/openMINDS_SANDS
[docu-url]: https://openminds-documentation.readthedocs.io
[docu-sands-url]: https://openminds-documentation.readthedocs.io/en/v3.0/schema_specifications/SANDS.html
[license-url]: https://raw.githubusercontent.com/openMetadataInitiative/openMINDS_SANDS/v3/LICENSE.txt
[license-shield]: https://img.shields.io/github/license/openMetadataInitiative/openMINDS_SANDS


<img src="docs/imgs/acme.png" width="50%">

# Acme: a research framework for reinforcement learning

[![PyPI Python Version][pypi-versions-badge]][pypi]
[![PyPI version][pypi-badge]][pypi]
[![acme-tests][tests-badge]][tests]
[![Documentation Status][rtd-badge]][documentation]

[pypi-versions-badge]: https://img.shields.io/pypi/pyversions/dm-acme
[pypi-badge]: https://badge.fury.io/py/dm-acme.svg
[pypi]: https://pypi.org/project/dm-acme/
[tests-badge]: https://github.com/deepmind/acme/workflows/acme-tests/badge.svg
[tests]: https://github.com/deepmind/acme/actions/workflows/ci.yml
[rtd-badge]: https://readthedocs.org/projects/dm-acme/badge/?version=latest

Acme is a library of reinforcement learning (RL) building blocks that strives to
expose simple, efficient, and readable agents. These agents first and foremost
serve both as reference implementations as well as providing strong baselines
for algorithm performance. However, the baseline agents exposed by Acme should
also provide enough flexibility and simplicity that they can be used as a
starting block for novel research. Finally, the building blocks of Acme are
designed in such a way that the agents can be run at multiple scales (e.g.
single-stream vs. distributed agents).

## Getting started

The quickest way to get started is to take a look at the detailed working code
examples found in the [examples] subdirectory. These show how to instantiate a
number of different agents and run them within a variety of environments. See
the [quickstart notebook][Quickstart] for an even quicker dive into using a
single agent. Even more detail on the internal construction of an agent can be
found inside our [tutorial notebook][Tutorial]. Finally, a full description Acme
and its underlying components can be found by referring to the [documentation].
More background information and details behind the design decisions can be found
in our [technical report][Paper].

> NOTE: Acme is first and foremost a framework for RL research written by
> researchers, for researchers. We use it for our own work on a daily basis. So
> with that in mind, while we will make every attempt to keep everything in good
> working order, things may break occasionally. But if so we will make our best
> effort to fix them as quickly as possible!

[examples]: examples/
[tutorial]: https://colab.research.google.com/github/deepmind/acme/blob/master/examples/tutorial.ipynb
[quickstart]: https://colab.research.google.com/github/deepmind/acme/blob/master/examples/quickstart.ipynb
[documentation]: https://dm-acme.readthedocs.io/
[paper]: https://arxiv.org/abs/2006.00979

## Installation

To get up and running quickly just follow the steps below:

1.  While you can install Acme in your standard python environment, we
    *strongly* recommend using a
    [Python virtual environment](https://docs.python.org/3/tutorial/venv.html)
    to manage your dependencies. This should help to avoid version conflicts and
    just generally make the installation process easier.

    ```bash
    python3 -m venv acme
    source acme/bin/activate
    pip install --upgrade pip setuptools wheel
    ```

1.  While the core `dm-acme` library can be pip installed directly, the set of
    dependencies included for installation is minimal. In particular, to run any
    of the included agents you will also need either [JAX] or [TensorFlow]
    depending on the agent. As a result we recommend installing these components
    as well, i.e.

    ```bash
    pip install dm-acme[jax,tf]
    ```

1.  Finally, to install a few example environments (including [gym],
    [dm_control], and [bsuite]):

    ```bash
    pip install dm-acme[envs]
    ```

1.  **Installing from github**: if you're interested in running the
    bleeding-edge version of Acme, you can do so by cloning the Acme GitHub
    repository and then executing following command from the main directory
    (where `setup.py` is located):

    ```bash
    pip install .[jax,tf,testing,envs]
    ```

## Citing Acme

If you use Acme in your work, please cite the updated accompanying
[technical report][paper]:

```bibtex
@article{hoffman2020acme,
    title={Acme: A Research Framework for Distributed Reinforcement Learning},
    author={
        Matthew W. Hoffman and Bobak Shahriari and John Aslanides and
        Gabriel Barth-Maron and Nikola Momchev and Danila Sinopalnikov and
        Piotr Sta\'nczyk and Sabela Ramos and Anton Raichuk and
        Damien Vincent and L\'eonard Hussenot and Robert Dadashi and
        Gabriel Dulac-Arnold and Manu Orsini and Alexis Jacq and
        Johan Ferret and Nino Vieillard and Seyed Kamyar Seyed Ghasemipour and
        Sertan Girgin and Olivier Pietquin and Feryal Behbahani and
        Tamara Norman and Abbas Abdolmaleki and Albin Cassirer and
        Fan Yang and Kate Baumli and Sarah Henderson and Abe Friesen and
        Ruba Haroun and Alex Novikov and Sergio G\'omez Colmenarejo and
        Serkan Cabi and Caglar Gulcehre and Tom Le Paine and
        Srivatsan Srinivasan and Andrew Cowie and Ziyu Wang and Bilal Piot and
        Nando de Freitas
    },
    year={2020},
    journal={arXiv preprint arXiv:2006.00979},
    url={https://arxiv.org/abs/2006.00979},
}
```

[JAX]: https://github.com/google/jax
[TensorFlow]: https://tensorflow.org
[gym]: https://github.com/openai/gym
[dm_control]: https://github.com/deepmind/dm_env
[dm_env]: https://github.com/deepmind/dm_env
[bsuite]: https://github.com/deepmind/bsuite


katdal
======

This package serves as a data access library to interact with the chunk stores
and HDF5 files produced by the MeerKAT radio telescope and its predecessors
(KAT-7 and Fringe Finder), which are collectively known as *MeerKAT Visibility
Format (MVF)* data sets. It uses memory carefully, allowing data sets to be
inspected and partially loaded into memory. Data sets may be concatenated and
split via a flexible selection mechanism. In addition, it provides a script to
convert these data sets to CASA MeasurementSets.

Quick Tutorial
--------------

Open any data set through a single function to obtain a data set object:

.. code:: python

  import katdal
  d = katdal.open('1234567890.h5')

The ``open`` function automatically determines the version and storage location
of the data set. The versions roughly map to the various instruments::

  - v1 : Fringe Finder (HDF5 file)
  - v2 : KAT-7 (HDF5 file)
  - v3 : MeerKAT (HDF5 file)
  - v4 : MeerKAT (RDB file + chunk store based on objects in Ceph)

Each MVFv4 data set is split into a Redis dump (aka *RDB*) file containing the
metadata in the form of a *telescope state* database, and a *chunk store*
containing the visibility data split into many small blocks or chunks (typically
served by a Ceph object store over the network). The RDB file is the main entry
point to the data set and it can be accessed directly from the MeerKAT SDP
archive if you have the appropriate permissions:

.. code:: python

  # This is just for illustration - the real URL looks a bit different
  d = katdal.open('https://archive/1234567890/1234567890_sdp_l0.rdb?token=AsD3')

Multiple data sets (even of different versions) may also be concatenated
together (as long as they have the same dump rate):

.. code:: python

  d = katdal.open(['1234567890.h5', '1234567891.h5'])

Inspect the contents of the data set by printing the object:

.. code:: python

  print(d)

Here is a typical output::

  ===============================================================================
  Name: 1313067732.h5 (version 2.0)
  ===============================================================================
  Observer: someone  Experiment ID: 2118d346-c41a-11e0-b2df-a4badb44fe9f
  Description: 'Track on Hyd A,Vir A, 3C 286 and 3C 273'
  Observed from 2011-08-11 15:02:14.072 SAST to 2011-08-11 15:19:47.810 SAST
  Dump rate: 1.00025 Hz
  Subarrays: 1
  ID  Antennas                            Inputs  Corrprods
   0  ant1,ant2,ant3,ant4,ant5,ant6,ant7  14      112
  Spectral Windows: 1
  ID  CentreFreq(MHz)  Bandwidth(MHz)  Channels  ChannelWidth(kHz)
   0  1822.000         400.000          1024      390.625
  -------------------------------------------------------------------------------
  Data selected according to the following criteria:
  subarray=0
  ants=['ant1', 'ant2', 'ant3', 'ant4', 'ant5', 'ant6', 'ant7']
  spw=0
  -------------------------------------------------------------------------------
  Shape: (1054 dumps, 1024 channels, 112 correlation products) => Size: 967.049 MB
  Antennas: *ant1,ant2,ant3,ant4,ant5,ant6,ant7  Inputs: 14  Autocorr: yes  Crosscorr: yes
  Channels: 1024 (index 0 - 1023, 2021.805 MHz - 1622.195 MHz), each 390.625 kHz wide
  Targets: 4 selected out of 4 in catalogue
  ID  Name    Type      RA(J2000)     DEC(J2000)  Tags  Dumps  ModelFlux(Jy)
   0  Hyd A   radec      9:18:05.28  -12:05:48.9          333      33.63
   1  Vir A   radec     12:30:49.42   12:23:28.0          251     166.50
   2  3C 286  radec     13:31:08.29   30:30:33.0          230      12.97
   3  3C 273  radec     12:29:06.70    2:03:08.6          240      39.96
  Scans: 8 selected out of 8 total       Compscans: 1 selected out of 1 total
  Date        Timerange(UTC)       ScanState  CompScanLabel  Dumps  Target
  11-Aug-2011/13:02:14 - 13:04:26    0:slew     0:             133    0:Hyd A
              13:04:27 - 13:07:46    1:track    0:             200    0:Hyd A
              13:07:47 - 13:08:37    2:slew     0:              51    1:Vir A
              13:08:38 - 13:11:57    3:track    0:             200    1:Vir A
              13:11:58 - 13:12:27    4:slew     0:              30    2:3C 286
              13:12:28 - 13:15:47    5:track    0:             200    2:3C 286
              13:15:48 - 13:16:27    6:slew     0:              40    3:3C 273
              13:16:28 - 13:19:47    7:track    0:             200    3:3C 273

The first segment of the printout displays the static information of the data
set, including observer, dump rate and all the available subarrays and spectral
windows in the data set. The second segment (between the dashed lines) highlights
the active selection criteria. The last segment displays dynamic information
that is influenced by the selection, including the overall visibility array
shape, antennas, channel frequencies, targets and scan info.

The data set is built around the concept of a three-dimensional visibility array
with dimensions of time, frequency and correlation product. This is reflected in
the *shape* of the dataset:

.. code:: python

  d.shape

which returns ``(1054, 1024, 112)``, meaning 1054 dumps by 1024 channels by 112
correlation products.

Let's select a subset of the data set:

.. code:: python

  d.select(scans='track', channels=slice(200, 300), ants='ant4')
  print(d)

This results in the following printout::

  ===============================================================================
  Name: /Users/schwardt/Downloads/1313067732.h5 (version 2.0)
  ===============================================================================
  Observer: siphelele  Experiment ID: 2118d346-c41a-11e0-b2df-a4badb44fe9f
  Description: 'track on Hyd A,Vir A, 3C 286 and 3C 273 for Lud'
  Observed from 2011-08-11 15:02:14.072 SAST to 2011-08-11 15:19:47.810 SAST
  Dump rate: 1.00025 Hz
  Subarrays: 1
  ID  Antennas                            Inputs  Corrprods
   0  ant1,ant2,ant3,ant4,ant5,ant6,ant7  14      112
  Spectral Windows: 1
  ID  CentreFreq(MHz)  Bandwidth(MHz)  Channels  ChannelWidth(kHz)
   0  1822.000         400.000          1024      390.625
  -------------------------------------------------------------------------------
  Data selected according to the following criteria:
  channels=slice(200, 300, None)
  subarray=0
  scans='track'
  ants='ant4'
  spw=0
  -------------------------------------------------------------------------------
  Shape: (800 dumps, 100 channels, 4 correlation products) => Size: 2.560 MB
  Antennas: ant4  Inputs: 2  Autocorr: yes  Crosscorr: no
  Channels: 100 (index 200 - 299, 1943.680 MHz - 1905.008 MHz), each 390.625 kHz wide
  Targets: 4 selected out of 4 in catalogue
  ID  Name    Type      RA(J2000)     DEC(J2000)  Tags  Dumps  ModelFlux(Jy)
   0  Hyd A   radec      9:18:05.28  -12:05:48.9          200      31.83
   1  Vir A   radec     12:30:49.42   12:23:28.0          200     159.06
   2  3C 286  radec     13:31:08.29   30:30:33.0          200      12.61
   3  3C 273  radec     12:29:06.70    2:03:08.6          200      39.32
  Scans: 4 selected out of 8 total       Compscans: 1 selected out of 1 total
  Date        Timerange(UTC)       ScanState  CompScanLabel  Dumps  Target
  11-Aug-2011/13:04:27 - 13:07:46    1:track    0:             200    0:Hyd A
              13:08:38 - 13:11:57    3:track    0:             200    1:Vir A
              13:12:28 - 13:15:47    5:track    0:             200    2:3C 286
              13:16:28 - 13:19:47    7:track    0:             200    3:3C 273

Compared to the first printout, the static information has remained the same
while the dynamic information now reflects the selected subset. There are many
possible selection criteria, as illustrated below:

.. code:: python

  d.select(timerange=('2011-08-11 13:10:00', '2011-08-11 13:15:00'), targets=[1, 2])
  d.select(spw=0, subarray=0)
  d.select(ants='ant1,ant2', pol='H', scans=(0,1,2), freqrange=(1700e6, 1800e6))

See the docstring of ``DataSet.select`` for more detailed information (i.e.
do ``d.select?`` in IPython). Take note that only one subarray and one spectral
window must be selected.

Once a subset of the data has been selected, you can access the data and
timestamps on the data set object:

.. code:: python

  vis = d.vis[:]
  timestamps = d.timestamps[:]

Note the ``[:]`` indexing, as the ``vis`` and ``timestamps`` properties are
special ``LazyIndexer`` objects that only give you the actual data when
you use indexing, in order not to inadvertently load the entire array into memory.

For the example dataset and no selection the ``vis`` array will have a shape of
``(1054, 1024, 112)``. The time dimension is labelled by ``d.timestamps``, the
frequency dimension by ``d.channel_freqs`` and the correlation product dimension
by ``d.corr_products``.

Another key concept in the data set object is that of *sensors*. These are named
time series of arbitrary data that are either loaded from the data set
(*actual* sensors) or calculated on the fly (*virtual* sensors). Both variants
are accessed through the *sensor cache* (available as ``d.sensor``) and cached
there after the first access. The data set object also provides convenient
properties to expose commonly-used sensors, as shown in the plot example below:

.. code:: python

  import matplotlib.pyplot as plt
  plt.plot(d.az, d.el, 'o')
  plt.xlabel('Azimuth (degrees)')
  plt.ylabel('Elevation (degrees)')

Other useful attributes include ``ra``, ``dec``, ``lst``, ``mjd``, ``u``,
``v``, ``w``, ``target_x`` and ``target_y``. These are all one-dimensional
NumPy arrays that dynamically change length depending on the active selection.

As in katdal's predecessor (scape) there is a ``DataSet.scans`` generator
that allows you to step through the scans in the data set. It returns the
scan index, scan state and target object on each iteration, and updates
the active selection on the data set to include only the current scan.
It is also possible to iterate through the compound scans with the
``DataSet.compscans`` generator, which yields the compound scan index, label
and first target on each iteration for convenience. These two iterators may also
be used together to traverse the data set structure:

.. code:: python

  for compscan, label, target in d.compscans():
      plt.figure()
      for scan, state, target in d.scans():
          if state in ('scan', 'track'):
              plt.plot(d.ra, d.dec, 'o')
      plt.xlabel('Right ascension (J2000 degrees)')
      plt.ylabel('Declination (J2000 degrees)')
      plt.title(target.name)

Finally, all the targets (or fields) in the data set are stored in a catalogue
available at ``d.catalogue``, and the original HDF5 file is still accessible via
a back door installed at ``d.file`` in the case of a single-file data set (v3
or older). On a v4 data set, ``d.source`` provides access to the underlying
telstate for metadata and the chunk store for data.


# Core Solution Services

Core Solution Services (CSS) is a group of foundational microservices, data models and other features, implemented in Python and running on Kubernetes. It supports common functionality needed by solutions built for different industries, like authentication, user management, data storage, and job management, and includes CI/CD using Github Actions and e2e testing.  CSS may be used as a stand-alone platform, or as a foundational layer to accelerate the building of specific applications.  It also includes a generic capability for GenAI use cases.  To date the microservices in CSS have been deployed in production for a variety of customers in the Public Sector division of Google Cloud.

> This codebase uses https://github.com/GoogleCloudPlatform/solutions-builder as a deployment tool and a template for new services.

## Use Cases

We have built an initial use case to support GenAI for Enterprise (GENIE).  GENIE is used as the foundation platform for other GenAI solutions in Google Cloud Public Sector.

### GenAI for Enterprise (GENIE)

We are building core services used for GenAI and Enterprise Search use cases, with built-in flexibility to leverage a number of LLMs like VertexAI (Gemini, Palm2), ChatGPT 3.5/4, Cohere, and more to come.

Highlighted features:

- Enable any enterprise to leverage a chatbot to answer questions using Retrieval Augmented Generation (RAG), based on configured content such as a collection of documents, websites, or databases.
- Support for Vertex model APIs and Vertex Model Garden.
- Easy plug in and use of LLM models available in the market, leveraging [Langchain](https://www.langchain.com/) as a model interface layer.
- Configurable agents based on Langchain.
- A built-in frontend app (using [Streamlit](https://streamlit.io/)) to showcase end-to-end user journeys.
- Cross-platform frontend application powered by [FlutterFlow](https://flutterflow.io/) that delivers end-to-end user flows and seamless digital experience on Android, iOS, web, and desktop platforms.

## Releases

Consult the [CHANGELOG](./CHANGELOG.md) for release information.  The current stable release of Core Solution Services is `v0.2.0`

## Quickstart (for GENIE)

To test the GenAI capabilities in Core Solution Services, we suggest you use Jupyter notebooks. There are several notebooks included in the platform (though be aware these are currently development aids and not polished).

1. Clone the repo locally

2. Create and activate a python 3.9 virtual environment in the LLM Service component.

In the top level directory of the repo:

```
cd components/llm_service
python3 -m venv .venv
source .venv/bin/activate
```

3. Install python requirements

```
pip install -r ../common/src/requirements.txt
pip install -r requirements.txt
pip install jupyter
```

4. Start a notebook server in the notebooks folder

```
cd notebooks
jupyter notebook
```

5. Open one of the notebooks and start experimenting with the libraries and APIs.

## Installing

Please follow the [INSTALL](./INSTALL.md) guide to install the platform. Note we strongly suggest you use a new GCP project to do this.  We also recommend you have the ability to update certain policies for your GCP Organization.

## Development

Please refer to the [DEVELOPMENT](./docs/DEVELOPMENT.md) guide to develop on the platform.

## Deployment with CI/CD

Please refer to the [CICD](./docs/CICD.md) guide to setup automated deployment with Github actions for the platform.