# ⛵ Cluster Template

Welcome to my opinionated and extensible template for deploying a single Kubernetes cluster. The goal of this project is to make it easier for people interested in using Kubernetes to deploy a cluster at home on bare-metal or VMs.

At a high level this project makes use of [makejinja](https://github.com/mirkolenz/makejinja) to read in a [configuration file](./config.sample.yaml) which renders out templates that will allow you to install and manage your Kubernetes cluster with.

## ✨ Features

The features included will depend on the type of configuration you want to use. There are currently **2 different types** of **configurations** available with this template.

1. **"Flux cluster"** - a Kubernetes cluster deployed on-top of [Talos Linux](https://github.com/siderolabs/talos) with an opinionated implementation of [Flux](https://github.com/fluxcd/flux2) using [GitHub](https://github.com/) as the Git provider and [sops](https://github.com/getsops/sops) to manage secrets.

    - **Required:** Some knowledge of [Containers](https://opencontainers.org/), [YAML](https://yaml.org/), and [Git](https://git-scm.com/).
    - **Components:** [flux](https://github.com/fluxcd/flux2), [Cilium](https://github.com/cilium/cilium),[cert-manager](https://github.com/cert-manager/cert-manager), [spegel](https://github.com/spegel-org/spegel), [reloader](https://github.com/stakater/Reloader), and [openebs](https://github.com/openebs/openebs).

2. **"Flux cluster with Cloudflare"** - An addition to "**Flux cluster**" that provides DNS and SSL with [Cloudflare](https://www.cloudflare.com/). [Cloudflare Tunnel](https://www.cloudflare.com/products/tunnel/) is also included to provide external access to certain applications deployed in your cluster.

    - **Required:** A Cloudflare account with a domain managed in your Cloudflare account.
    - **Components:** [ingress-nginx](https://github.com/kubernetes/ingress-nginx/), [external-dns](https://github.com/kubernetes-sigs/external-dns) and [cloudflared](https://github.com/cloudflare/cloudflared).

**Other features include:**

- A [Renovate](https://www.mend.io/renovate)-ready repository with pull request diffs provided by [flux-local](https://github.com/allenporter/flux-local)
- Integrated [GitHub Actions](https://github.com/features/actions) with helpful workflows.

## 💻 Machine Preparation

### System requirements

> [!NOTE]
> 1. The included behaviour of Talos is that all nodes are able to run workloads, **including** the controller nodes. **Worker nodes** are therefore **optional**.
> 2. Do you have 3 or more nodes? It is highly recommended to make 3 of them controller nodes for a highly available control plane.
> 3. Running the cluster on Proxmox VE? My thoughts and recommendations about that are documented [here](https://onedr0p.github.io/home-ops/notes/proxmox-considerations.html).

| Role    | Cores    | Memory        | System Disk               |
|---------|----------|---------------|---------------------------|
| Control | 4 _(6*)_ | 8GB _(24GB*)_ | 120GB _(500GB*)_ SSD/NVMe |
| Worker  | 4 _(6*)_ | 8GB _(24GB*)_ | 120GB _(500GB*)_ SSD/NVMe |
| _\* recommended_ |

1. Head over to <https://factory.talos.dev> and follow the instructions which will eventually lead you to download a Talos Linux iso file (or for SBCs the `.raw.xz`). Make sure to note the schematic ID you will need this later on.

2. Flash the iso or raw file to a USB drive and boot to Talos on your nodes with it.

3. Continue on to 🚀 [**Getting Started**](#-getting-started)

## 🚀 Getting Started

Once you have installed Talos on your nodes, there are six stages to getting a Flux-managed cluster up and runnning.

> [!NOTE]
> For all stages below the commands **MUST** be ran on your personal workstation within your repository directory

### 🎉 Stage 1: Create a Git repository

1. Create a new **public** repository by clicking the big green "Use this template" button at the top of this page.

2. Clone **your new repo** to you local workstation and `cd` into it.

3. Continue on to 🌱 [**Stage 2**](#-stage-2-setup-your-local-workstation-environment)

### 🌱 Stage 2: Setup your local workstation

You have two different options for setting up your local workstation.

- First option is using a `devcontainer` which requires you to have Docker and VSCode installed. This method is the fastest to get going because all the required CLI tools are provided for you in my [devcontainer](https://github.com/onedr0p/cluster-template/pkgs/container/cluster-template%2Fdevcontainer) image.
- The second option is setting up the CLI tools directly on your workstation.

#### Devcontainer method

1. Start Docker and open your repository in VSCode. There will be a pop-up asking you to use the `devcontainer`, click the button to start using it.

2. Continue on to 🔧 [**Stage 3**](#-stage-3-bootstrap-configuration)

#### Non-devcontainer method

1. Install the most recent version of [task](https://taskfile.dev/), see the [installation docs](https://taskfile.dev/installation/) for other supported platforms.

    ```sh
    # Homebrew
    brew install go-task
    # or, Arch
    pacman -S --noconfirm go-task && ln -sf /usr/bin/go-task /usr/local/bin/task
    ```

2. Install the most recent version of [direnv](https://direnv.net/), see the [installation docs](https://direnv.net/docs/installation.html) for other supported platforms.

    ```sh
    # Homebrew
    brew install direnv
    # or, Arch
    pacman -S --noconfirm direnv
    ```

3. [Hook `direnv` into your preferred shell](https://direnv.net/docs/hook.html), then run:

    ```sh
    task workstation:direnv
    ```

    📍 _**Verify** that `direnv` is setup properly by opening a new terminal and `cd`ing into your repository. You should see something like:_

    ```sh
    cd /path/to/repo
    direnv: loading /path/to/repo/.envrc
    direnv: export +ANSIBLE_COLLECTIONS_PATH ...  +VIRTUAL_ENV ~PATH
    ```

4. Install the additional **required** CLI tools

   📍 _**Not using Homebrew or ArchLinux?** Try using the generic Linux task below, if that fails check out the [Brewfile](.taskfiles/Workstation/Brewfile)/[Archfile](.taskfiles/Workstation/Archfile) for what CLI tools needed and install them._

    ```sh
    # Homebrew
    task workstation:brew
    # or, Arch with yay/paru
    task workstation:arch
    # or, Generic Linux (YMMV, this pulls binaires in to ./bin)
    task workstation:generic-linux
    ```

5. Setup a Python virual environment by running the following task command.

    📍 _This commands requires Python 3.11+ to be installed._

    ```sh
    task workstation:venv
    ```

6. Continue on to 🔧 [**Stage 3**](#-stage-3-bootstrap-configuration)

### 🔧 Stage 3: Bootstrap configuration

> [!NOTE]
> The [config.sample.yaml](./config.sample.yaml) file contains config that is **vital** to the bootstrap process.

1. Generate the `config.yaml` from the [config.sample.yaml](./config.sample.yaml) configuration file.

    ```sh
    task init
    ```

2. Fill out the `config.yaml` configuration file using the comments in that file as a guide.

3. Run the following command which will generate all the files needed to continue.

    ```sh
    task configure
    ```

4. Push you changes to git

   📍 _**Verify** all the `./kubernetes/**/*.sops.*` files are **encrypted** with SOPS_

    ```sh
    git add -A
    git commit -m "Initial commit :rocket:"
    git push
    ```

### ⛵ Stage 4: Install Kubernetes

1. Deploy your cluster and bootstrap it. This generates secrets, generates the config files for your nodes and applies them. It bootstraps the cluster afterwards, fetches the kubeconfig file and installs Cilium and kubelet-csr-approver. It finishes with some health checks.

    ```sh
    task talos:bootstrap
    ```

2. ⚠ It might take a while for the cluster to be setup (10+ minutes is normal), during which time you will see a variety of error messages like: "couldn't get current server API group list," "error: no matching resources found", etc. This is a normal. If this step gets interrupted, e.g. by pressing Ctrl + C, you likely will need to [nuke the cluster](#-Nuke) before trying again.

#### Cluster validation

1. The `kubeconfig` for interacting with your cluster should have been created in the root of your repository.

2. Verify the nodes are online

    📍 _If this command **fails** you likely haven't configured `direnv` as [mentioned previously](#non-devcontainer-method) in the guide._

    ```sh
    kubectl get nodes -o wide
    # NAME           STATUS   ROLES                       AGE     VERSION
    # k8s-0          Ready    control-plane,etcd,master   1h      v1.30.1
    # k8s-1          Ready    worker                      1h      v1.30.1
    ```

3. Continue on to 🔹 [**Stage 6**](#-stage-6-install-flux-in-your-cluster)

### 🔹 Stage 6: Install Flux in your cluster

1. Verify Flux can be installed

    ```sh
    flux check --pre
    # ► checking prerequisites
    # ✔ kubectl 1.30.1 >=1.18.0-0
    # ✔ Kubernetes 1.30.1 >=1.16.0-0
    # ✔ prerequisites checks passed
    ```

2. Install Flux and sync the cluster to the Git repository

    📍 _Run `task flux:github-deploy-key` first if using a private repository._

    ```sh
    task flux:bootstrap
    # namespace/flux-system configured
    # customresourcedefinition.apiextensions.k8s.io/alerts.notification.toolkit.fluxcd.io created
    # ...
    ```

1. Verify Flux components are running in the cluster

    ```sh
    kubectl -n flux-system get pods -o wide
    # NAME                                       READY   STATUS    RESTARTS   AGE
    # helm-controller-5bbd94c75-89sb4            1/1     Running   0          1h
    # kustomize-controller-7b67b6b77d-nqc67      1/1     Running   0          1h
    # notification-controller-7c46575844-k4bvr   1/1     Running   0          1h
    # source-controller-7d6875bcb4-zqw9f         1/1     Running   0          1h
    ```

### 🎤 Verification Steps

_Mic check, 1, 2_ - In a few moments applications should be lighting up like Christmas in July 🎄

1. Output all the common resources in your cluster.

    📍 _Feel free to use the provided [kubernetes tasks](.taskfiles/Kubernetes/Taskfile.yaml) for validation of cluster resources or continue to get familiar with the `kubectl` and `flux` CLI tools._

    ```sh
    task kubernetes:resources
    ```

2. ⚠ It might take `cert-manager` awhile to generate certificates, this is normal so be patient.

3. 🏆 **Congratulations** if all goes smooth you will have a Kubernetes cluster managed by Flux and your Git repository is driving the state of your cluster.

4. 🧠 Now it's time to pause and go get some motel motor oil ☕ and admire you made it this far!

## 📣 Flux w/ Cloudflare post installation

#### 🌐 Public DNS

The `external-dns` application created in the `networking` namespace will handle creating public DNS records. By default, `echo-server` and the `flux-webhook` are the only subdomains reachable from the public internet. In order to make additional applications public you must set set the correct ingress class name and ingress annotations like in the HelmRelease for `echo-server`.

#### 🏠 Home DNS

`k8s_gateway` will provide DNS resolution to external Kubernetes resources (i.e. points of entry to the cluster) from any device that uses your home DNS server. For this to work, your home DNS server must be configured to forward DNS queries for `${bootstrap_cloudflare.domain}` to `${bootstrap_cloudflare.gateway_vip}` instead of the upstream DNS server(s) it normally uses. This is a form of **split DNS** (aka split-horizon DNS / conditional forwarding).

> [!TIP]
> Below is how to configure a Pi-hole for split DNS. Other platforms should be similar.
> 1. Apply this file on the Pihole server while substituting the variables
> ```sh
> # /etc/dnsmasq.d/99-k8s-gateway-forward.conf
> server=/${bootstrap_cloudflare.domain}/${bootstrap_cloudflare.gateway_vip}
> ```
> 2. Restart dnsmasq on the server.
> 3. Query an internal-only subdomain from your workstation (any `internal` class ingresses): `dig @${home-dns-server-ip} echo-server-internal.${bootstrap_cloudflare.domain}`. It should resolve to `${bootstrap_cloudflare.ingress_vip}`.

If you're having trouble with DNS be sure to check out these two GitHub discussions: [Internal DNS](https://github.com/onedr0p/cluster-template/discussions/719) and [Pod DNS resolution broken](https://github.com/onedr0p/cluster-template/discussions/635).

... Nothing working? That is expected, this is DNS after all!

#### 📜 Certificates

By default this template will deploy a wildcard certificate using the Let's Encrypt **staging environment**, which prevents you from getting rate-limited by the Let's Encrypt production servers if your cluster doesn't deploy properly (for example due to a misconfiguration). Once you are sure you will keep the cluster up for more than a few hours be sure to switch to the production servers as outlined in `config.yaml`.

📍 _You will need a production certificate to reach internet-exposed applications through `cloudflared`._

#### 🪝 Github Webhook

By default Flux will periodically check your git repository for changes. In order to have Flux reconcile on `git push` you must configure Github to send `push` events to Flux.

> [!NOTE]
> This will only work after you have switched over certificates to the Let's Encrypt Production servers.

1. Obtain the webhook path

    📍 _Hook id and path should look like `/hook/12ebd1e363c641dc3c2e430ecf3cee2b3c7a5ac9e1234506f6f5f3ce1230e123`_

    ```sh
    kubectl -n flux-system get receiver github-receiver -o jsonpath='{.status.webhookPath}'
    ```

2. Piece together the full URL with the webhook path appended

    ```text
    https://flux-webhook.${bootstrap_cloudflare.domain}/hook/12ebd1e363c641dc3c2e430ecf3cee2b3c7a5ac9e1234506f6f5f3ce1230e123
    ```

3. Navigate to the settings of your repository on Github, under "Settings/Webhooks" press the "Add webhook" button. Fill in the webhook url and your `bootstrap_github_webhook_token` secret and save.

## 💥 Nuke

There might be a situation where you want to destroy your Kubernetes cluster. The following command will reset your nodes back to maintenance mode, append `--force` to completely format your the Talos installation. Either way the nodes should reboot after the command has run.

```sh
task talos:nuke
```

## 🤖 Renovate

[Renovate](https://www.mend.io/renovate) is a tool that automates dependency management. It is designed to scan your repository around the clock and open PRs for out-of-date dependencies it finds. Common dependencies it can discover are Helm charts, container images, GitHub Actions, Ansible roles... even Flux itself! Merging a PR will cause Flux to apply the update to your cluster.

To enable Renovate, click the 'Configure' button over at their [Github app page](https://github.com/apps/renovate) and select your repository. Renovate creates a "Dependency Dashboard" as an issue in your repository, giving an overview of the status of all updates. The dashboard has interactive checkboxes that let you do things like advance scheduling or reattempt update PRs you closed without merging.

The base Renovate configuration in your repository can be viewed at [.github/renovate.json5](./.github/renovate.json5). By default it is scheduled to be active with PRs every weekend, but you can [change the schedule to anything you want](https://docs.renovatebot.com/presets-schedule), or remove it if you want Renovate to open PRs right away.

## 🐛 Debugging

Below is a general guide on trying to debug an issue with an resource or application. For example, if a workload/resource is not showing up or a pod has started but in a `CrashLoopBackOff` or `Pending` state.

1. Start by checking all Flux Kustomizations & Git Repository & OCI Repository and verify they are healthy.

    ```sh
    flux get sources oci -A
    flux get sources git -A
    flux get ks -A
    ```

2. Then check all the Flux Helm Releases and verify they are healthy.

    ```sh
    flux get hr -A
    ```

3. Then check the if the pod is present.

    ```sh
    kubectl -n <namespace> get pods -o wide
    ```

4. Then check the logs of the pod if its there.

    ```sh
    kubectl -n <namespace> logs <pod-name> -f
    # or
    stern -n <namespace> <fuzzy-name>
    ```

5. If a resource exists try to describe it to see what problems it might have.

    ```sh
    kubectl -n <namespace> describe <resource> <name>
    ```

6. Check the namespace events

    ```sh
    kubectl -n <namespace> get events --sort-by='.metadata.creationTimestamp'
    ```

Resolving problems that you have could take some tweaking of your YAML manifests in order to get things working, other times it could be a external factor like permissions on NFS. If you are unable to figure out your problem see the help section below.

## ⬆ Upgrading Talos and Kubernetes

### Manual

```sh
# Upgrade Talos to a newer version
# NOTE: This needs to be run once on every node
task talos:upgrade node=? image=?
# e.g.
# task talos:upgrade node=192.168.42.10 image=factory.talos.dev/installer/${schematic_id}:v1.7.4
```

```sh
# Upgrade Kubernetes to a newer version
# NOTE: This only needs to be run once against a controller node
task talos:upgrade-k8s controller=? to=?
# e.g.
# task talos:upgrade-k8s controller=192.168.42.10 to=1.30.1
```

## 👉 Help

- Make a post in this repository's Github [Discussions](https://github.com/onedr0p/cluster-template/discussions).
- Start a thread in the `#support` or `#cluster-template` channels in the [Home Operations](https://discord.gg/home-operations) Discord server.

## ❔ What's next

The cluster is your oyster (or something like that). Below are some optional considerations you might want to review.

### Ship it

To browse or get ideas on applications people are running, community member [@whazor](https://github.com/whazor) created [Kubesearch](https://kubesearch.dev) as a creative way to search Flux HelmReleases across Github and Gitlab.

### Storage

The included CSI (openebs in local-hostpath mode) is a great start for storage but soon you might find you need more features like replicated block storage, or to connect to a NFS/SMB/iSCSI server. If you need any of those features be sure to check out the projects like [rook-ceph](https://github.com/rook/rook), [longhorn](https://github.com/longhorn/longhorn), [openebs](https://github.com/openebs/openebs), [democratic-csi](https://github.com/democratic-csi/democratic-csi), [csi-driver-nfs](https://github.com/kubernetes-csi/csi-driver-nfs),
and [synology-csi](https://github.com/SynologyOpenSource/synology-csi).

## 🙌 Related Projects

If this repo is too hot to handle or too cold to hold check out these following projects.

- [khuedoan/homelab](https://github.com/khuedoan/homelab) - _Modern self-hosting framework, fully automated from empty disk to operating services with a single command._
- [danmanners/aws-argo-cluster-template](https://github.com/danmanners/aws-argo-cluster-template) - _A community opinionated template for deploying Kubernetes clusters on-prem and in AWS using Pulumi, SOPS, Sealed Secrets, GitHub Actions, Renovate, Cilium and more!_
- [ricsanfre/pi-cluster](https://github.com/ricsanfre/pi-cluster) - _Pi Kubernetes Cluster. Homelab kubernetes cluster automated with Ansible and ArgoCD_
- [techno-tim/k3s-ansible](https://github.com/techno-tim/k3s-ansible) - _The easiest way to bootstrap a self-hosted High Availability Kubernetes cluster. A fully automated HA k3s etcd install with kube-vip, MetalLB, and more_

## ⭐ Stargazers

[![Star History Chart](https://api.star-history.com/svg?repos=onedr0p/cluster-template&type=Date)](https://star-history.com/#onedr0p/cluster-template&Date)

## 🤝 Thanks

Big shout out to all the contributors, sponsors and everyone else who has helped on this project.


---

**Mirascope** is an intuitive approach to building with LLMs. Building with Mirascope feels like writing the Python code you’re already used to writing.

## Installation

```bash
pip install mirascope
```

You can also install additional optional dependencies if you’re using those features:

```bash
pip install mirascope[anthropic]  # AnthropicCall, ...
pip install mirascope[groq]       # GroqCall, ...
pip install mirascope[logfire]    # with_logfire decorator, ...
pip install mirascope[all]        # all optional dependencies
```

> Note: escape brackets (e.g. `pip install mirascope\[anthropic\]`) if using zsh

## Getting Started

## Examples

### Colocation

[Colocation](./concepts/philosophy.md#4-colocation) is one of the core tenets of our philosophy. Everything that can impact the quality of a call to an LLM — from the prompt to the model to the temperature — must live together so that we can properly version and test the quality of our calls over time. This is useful since we have all of the information including metadata that we could want for analysis, which is particularly important during rapid development.

```python
import os

from mirascope import tags
from mirascope.openai import OpenAICall, OpenAICallParams

os.environ["OPENAI_API_KEY"] = "YOUR_API_KEY"

@tags(["version:0003"])
class Editor(OpenAICall):
    prompt_template = """
    SYSTEM:
    You are a top class manga editor.

    USER:
    I'm working on a new storyline. What do you think?
    {storyline}
    """

    storyline: str

    call_params = OpenAICallParams(model="gpt-4", temperature=0.4)

storyline = "..."
editor = Editor(storyline=storyline)

print(editor.messages()) # white-space is automatically stripped
# > [{'role': 'system', 'content': 'You are a top class manga editor.'}, {'role': 'user', 'content': "I'm working on a new storyline. What do you think?\n..."}]

critique = editor.call()
print(critique.content)
# > I think the beginning starts off great, but...

print(editor.dump() | critique.dump())
# {
#     "tags": ["version:0003"],
#     "template": "SYSTEM:\nYou are a top class manga editor.\n\nUSER:\nI'm working on a new storyline. What do you think?\n{storyline}",
#     "inputs": {"storyline": "..."},
#     "start_time": 1710452778501.079,
#     "end_time": 1710452779736.8418,
#     "output": {
#         "id": "chatcmpl-92nBykcXyTpxwAbTEM5BOKp99fVmv",
#         "choices": [
#             {
#                 "finish_reason": "stop",
#                 "index": 0,
#                 "logprobs": None,
#                 "message": {
#                     "content": "I think the beginning starts off great, but...",
#                     "role": "assistant",
#                     "function_call": None,
#                     "tool_calls": None,
#                 },
#             }
#         ],
#         "created": 1710452778,
#         "model": "gpt-4-0613",
#         "object": "chat.completion",
#         "system_fingerprint": None,
#         "usage": {"completion_tokens": 25, "prompt_tokens": 33, "total_tokens": 58},
#     },
# }
```

### Chat History

Our template parser makes inserting [chat history](./concepts/chat_history.md) beyond easy:

```python
from openai.types.chat import ChatCompletionMessageParam

from mirascope.openai import OpenAICall

class Librarian(OpenAICall):
    prompt_template = """
    SYSTEM: You are the world's greatest librarian.
    MESSAGES: {history}
    USER: {question}
    """

    question: str
    history: list[ChatCompletionMessageParam] = []

librarian = Librarian(question="", history=[])
while True:
    librarian.question = input("(User): ")
    response = librarian.call()
    librarian.history += [
        {"role": "user", "content": librarian.question},
        {"role": "assistant", "content": response.content},
    ]
    print(f"(Assistant): {response.content}")

#> (User): What fantasy book should I read?
#> (Assistant): Have you read the Name of the Wind?
#> (User): I have! What do you like about it?
#> (Assistant): I love the intricate world-building...
```

### Tools (Function Calling)

We’ve made implementing and using [tools (function calling)](./concepts/tools_(function_calling).md) intuitive:

```python
from typing import Literal

from mirascope.openai import OpenAICall, OpenAICallParams

def get_current_weather(
    location: str, unit: Literal["celsius", "fahrenheit"] = "fahrenheit"
):
    """Get the current weather in a given location."""
    if "tokyo" in location.lower():
        print(f"It is 10 degrees {unit} in Tokyo, Japan")
    elif "san francisco" in location.lower():
        print(f"It is 72 degrees {unit} in San Francisco, CA")
    elif "paris" in location.lower():
        print(f"It is 22 degress {unit} in Paris, France")
    else:
        print(f"I'm not sure what the weather is like in {location}")

class Forecast(OpenAICall):
    prompt_template = "What's the weather in Tokyo?"

    call_params = OpenAICallParams(model="gpt-4", tools=[get_current_weather])

tool = Forecast().call().tool
if tool:
    tool.fn(**tool.args)
	  #> It is 10 degrees fahrenheit in Tokyo, Japan
```

### Chaining

[Chaining](./concepts/chaining_calls.md) multiple calls together is as simple as writing a property:

```python
import os
from functools import cached_property

from mirascope.openai import OpenAICall
from pydantic import computed_field

os.environ["OPENAI_API_KEY"] = "YOUR_API_KEY"

class ChefSelector(OpenAICall):
    prompt_template = """
    Name a chef who is really good at cooking {food_type} food.
    Give me just the name.
    """

    food_type: str

class RecipeRecommender(ChefSelector):
    prompt_template = """
    SYSTEM:
    Imagine that you are chef {chef}.
    Your task is to recommend recipes that you, {chef}, would be excited to serve.

    USER:
    Recommend a recipe using {ingredient}.
    """

    ingredient: str

    @computed_field
    @cached_property
    def chef(self) -> str:
        """Uses `ChefSelector` to select the chef based on the food type."""
        return ChefSelector(food_type=self.food_type).call().content

recommender = RecipeRecommender(food_type="japanese", ingredient="apples")
recipe = recommender.call()
print(recipe.content)
# > Certainly! Here's a recipe for a delicious and refreshing Japanese Apple Salad: ...
```

Of course, you can also chain calls together in sequence rather than through properties and wrap the chain in a function for reusability. You can find an example of this [here](./concepts/chaining_calls.md#chaining-directly-function-chaining).

### Extracting Structured Information

Convenience built on top of tools that makes [extracting structured information](./concepts/extracting_structured_information_using_llms.md) reliable:

```python
from typing import Literal, Type

from mirascope.openai import OpenAIExtractor
from pydantic import BaseModel

class TaskDetails(BaseModel):
    description: str
    due_date: str
    priority: Literal["low", "normal", "high"]

class TaskExtractor(OpenAIExtractor[TaskDetails]):
    extract_schema: Type[TaskDetails] = TaskDetails
    prompt_template = """
    Extract the task details from the following task:
    {task}
    """

    task: str

task = "Submit quarterly report by next Friday. Task is high priority."
task_details = TaskExtractor(task=task).extract()
assert isinstance(task_details, TaskDetails)
print(task_details)
#> description='Submit quarterly report' due_date='next Friday' priority='high'
```

### FastAPI Integration

Since we’ve built our `BasePrompt` on top of [Pydantic](https://pydantic.dev/), we [integrate](./integrations/fastapi.md) with tools like [FastAPI](https://fastapi.tiangolo.com/) out-of-the-box:

```python
import os
from typing import Type

from fastapi import FastAPI
from mirascope.openai import OpenAIExtractor
from pydantic import BaseModel

os.environ["OPENAI_API_KEY"] = "YOUR_OPENAI_API_KEY"

app = FastAPI()

class Book(BaseModel):
    title: str
    author: str

class BookRecommender(OpenAIExtractor[Book]):
    extract_schema: Type[Book] = Book
    prompt_template = "Please recommend a {genre} book."

    genre: str

@app.post("/")
def root(book_recommender: BookRecommender) -> Book:
    """Generates a book based on provided `genre`."""
    return book_recommender.extract()
```

## Supported Providers and Integrations

You can find a list of [supported providers](./concepts/supported_llm_providers.md) with examples of how to use them with Mirascope.

We are constantly working to further integrate Mirascope as seamlessly as possible with as many tools as possible. You can find the [integrations that we currently support](./integrations/client_wrappers.md) in our docs. If there are any integrations that you want, let us know!

## Roadmap

- [ ]  Agents
    - [X]  Easy tool calling and execution
    - [ ]  More convenience around TOOL messages
    - [ ]  Base classes for ReAct agents
    - [ ]  Base classes for Query Planning agents
    - [ ]  Tons of examples...
- [ ]  RAG
    - [X]  ChromaDB
    - [X]  Pinecone
    - [X]  OpenAI Embeddings
    - [X]  Cohere Embeddings
    - [ ]  Hugging Face
    - [ ]  Tons of examples...
- [ ] Mirascope CLI
    - [X] Versioning prompts / calls / extractors
    - [ ] RAG CLI (e.g. versioning stores, one-off vector store interactions)
    - [ ] Versioning integrations with LLMOps tools (e.g. Weave, LangSmith, ...)
    - [ ] LLM Provider Auto-conversion
    - [ ] Templates (`mirascope from_template pinecone_rag_openai_call my_call_name`)
- [X]  Extracting structured information using LLMs
- [X]  Streaming extraction for tools (function calling)
- [ ]  Additional template parsing for more complex messages
    - [X]  Chat History
    - [X]  List + List[List] Convenience
    - [ ]  Additional Metadata
    - [ ]  Vision
- [ ]  Support for more LLM providers:
    - [X]  Anthropic
    - [X]  Cohere
    - [X]  Mistral
    - [X]  Groq
    - [X]  Gemini
    - [ ]  HuggingFace
- [ ]  Integrations
    - [X]  Logfire by Pydantic
    - [X]  Langfuse
    - [X]  Weights & Biases Trace
    - [X]  Weave by Weights & Biases
    - [X]  LangChain / LangSmith
    - [ ]  … tell us what you’d like integrated!
- [ ]  Evaluating prompts and their quality by version

## Versioning

Mirascope uses [Semantic Versioning](https://semver.org/).

## License

This project is licensed under the terms of the [MIT License](https://github.com/Mirascope/mirascope/blob/main/LICENSE).


[![CII Best Practices](https://bestpractices.coreinfrastructure.org/projects/5788/badge)](https://bestpractices.coreinfrastructure.org/projects/5788)

# Alchemy Logging (alog)
The `alog` framework provides tunable logging with easy-to-use defaults and power-user capabilities. The mantra of `alog` is **"Log Early And Often"**. To accomplish this goal, `alog` makes it easy to enable verbose logging at develop/debug time and trim the verbosity at production run time, all while avoiding performance degredation by using lazily evaluated log messages which only render if enabled. The `alog` project maintains language-naitve implementations in many popular programming languages with the goal of giving a consistent application logging experience both when writing your code and when administering a cloud service written in multiple programming languages.

## Channels and Levels
The primary components of the framework are **channels** and **levels** which allow for each log statement to be enabled or disabled when appropriate.

1. **Levels**: Each logging statement is made at a specific level. Levels provide sequential granularity, allowing detailed debugging statements to be placed in the code without clogging up the logs at runtime.

    <details>
    <summary>
    The sequence of levels and their general usage is as follows:
    </summary>

    1. `off`: Disable the given channel completely
    1. `fatal`: A fatal error has occurred. Any behavior after this statement should be regarded as undefined.
    1. `error`: An unrecoverable error has occurred. Any behavior after this statement should be regarded as undefined unless the error is explicitly handled.
    1. `warning`: A recoverable error condition has come up that the service maintainer should be aware of.
    1. `info`: High-level information that is valuable at runtime under moderate load.
    1. `trace`: Used to log begin/end of functions for debugging code paths.
    1. `debug`: High-level debugging statements such as function parameters.
    1. `debug1`: High-level debugging statements.
    1. `debug2`: Mid-level debugging statements such as computed values.
    1. `debug3`: Low-level debugging statements such as computed values inside loops.
    1. `debug4`: Ultra-low-level debugging statements such as data dumps and/or statements inside multiple nested loops.

    </details>

1. **Channels**: Each logging statement is made to a specific channel. Channels are independent of one another and allow for logical grouping of log messages by functionality. A channel can be any string. A channel may have a specific **level** assigned to it, or it may use the configured default level if it is not given a specific level filter.

Using this combination of **Channels** and **Levels**, you can fine-tune what log statements are enabled when you run your application under different circumstances.

## Standard Configuration
There are two primary pieces of configuration when setting up the `alog` environment:

1. **default_level**: This is the level that will be enabled for a given channel when a specific level has not been set in the **filters**.

1. **filters**: This is a mapping from channel name to level that allows levels to be set on a per-channel basis.

## Formatting
All `alog` implementations support two styles of formatting:

1. `pretty`: The **pretty** formatter is designed for development-time logging, making it easy to visually scan your log messages and follow the flow of execution

1. `json`: The **json** formatter is designed for production-runtime logging, making it easy to ingest structured representations of your runtime logs into a cloud log monitoring framework such as [Sysdig](https://sysdig.com/)

As a convenience, a converter is also provided from `json` -> `pretty` so that production logs can be read and visually parsed in a similar manner to development-time logs. The `pretty` -> `json` conversion is not provided since the header formatting for `pretty` logs can be lossy when channel names are truncated.

## Implementations

This section provides the high level details on installing and using each available implementation.

### Python

For full details see [src/python](src/python/README.md)

#### Installation

```sh
pip install alchemy-logging
```

#### Usage

```py
import alog
alog.configure(default_level="info", filters="FOO:debug2,BAR:off")
channel = alog.use_channel("FOO")
channel.debug("Hello alog!")
```

### Typescript

For full details see [src/ts](src/ts/README.md)

#### Installation

```sh
npm install alchemy-logging
```

#### Usage

```ts
import alog from 'alchemy-logging';
alog.configure('info', 'FOO:debug2,BAR:off');
alog.debug('FOO', 'Hello alog!');
```

### Go

For full details see [src/go](src/go/README.md)

#### Installation

```sh
go get github.com/IBM/alchemy-logging/src/go/alog
```

#### Usage

```go
package main
import (
	"github.com/IBM/alchemy-logging/src/go/alog"
)
func main() {
	alog.Config(alog.INFO, alog.ChannelMap{
		"FOO": alog.DEBUG2,
		"BAR": alog.OFF,
	})
	alog.Log("FOO", alog.DEBUG, "Hello alog!")
}
```

### C++

For full details see [src/cpp](src/cpp/README.md)

#### Installation

In your `CMakeLists.txt` using [CPM](https://github.com/cpm-cmake/CPM.cmake):

```cmake
include(cmake/CPM.cmake)
set(ALOG_VERSION main CACHE STRING "The version (point in git history) of alog to use")
CPMAddPackage(
  NAME alog
  GITHUB_REPOSITORY IBM/alchemy-logging
  GIT_TAG ${ALOG_VERSION}
  GIT_SHALLOW true
  OPTIONS
    "BUILD_UNIT_TESTS OFF"
)
```

#### Usage

```cpp
#include <alog/logger.hpp>

int main()
{
    ALOG_SETUP("info", "FOO:debug2,BAR:off");
    ALOG(FOO, debug2, "Hello alog!");
}
```


# spetlr

A python ETL libRary (SPETLR) for Databricks powered by Apache SPark.

Visit SPETLR official webpage: [https://spetlr.com/](https://spetlr.com/)

# NEWS

Support for LTS9.1 is ending. [See the issue for discussions.](https://github.com/spetlr-org/spetlr/issues/61)

TransformerNC class will be removed permanently. [Follow the PR.](https://github.com/spetlr-org/spetlr/pull/82)

# Table of Contents

- [Description](#description)
- [Important Notes](#important-notes)
- [Installation](#installation)
- [Development Notes](#development-notes)
- [Testing](#testing)
- [General Project Info](#general-project-info)
- [Contributing](#contributing)
- [Build Status](#build-status)
- [Releases](#releases)
- [Requirements](#requirements-and-dependencies)
- [Contact](#contact)

# Description

SPETLR has a lot of great tools for working with ETL in Databricks. But to make it easy for you to consider why you need
SPETLR here is a list of the core features:

* ETL framework: A common ETL framework that enables reusable transformations in an object-oriented manner. Standardized
  structures facilitate cooperation in large teams.

* Integration testing: A framework for creating test databases and tables before deploying to production in order to
  ensure reliable and stable data platforms. An additional layer of data abstraction allows full integration testing.

* Handlers: Standard connectors with commonly used options reduce boilerplate.

For more information, visit SPETLR official webpage: [https://spetlr.com/](https://spetlr.com/)

# Important Notes

This package can not be run or tested without access to `pyspark`.
However, installing `pyspark` as part of our installer gave issues when
other versions of `pyspark` were needed. Hence we took out the dependency
from our installer.

# Installation

Install SPETLR from PyPI:
[![PyPI version](https://badge.fury.io/py/spetlr.svg)](https://pypi.org/project/spetlr/)
[![PyPI](https://img.shields.io/pypi/dm/spetlr)](https://pypi.org/project/spetlr/)

```
pip install spetlr
```

# Development Notes

To prepare for development, please install these additional requirements:

- Java 8
- `pip install -r test_requirements.txt`

Then install the package locally

    python setup.py develop

## Testing

### Local tests

After installing the dev-requirements, execute tests by running:

    pytest tests

These tests are located in the `./tests/local` folder and only require a Python interpreter. Pull requests will not be
accepted if these tests do not pass. If you add new features, please include corresponding tests.

### Cluster tests

Tests in the `./tests/cluster` folder are designed to run on a Databricks cluster.
The [Pre-integration Test](https://github.com/spetlr-org/spetlr/blob/main/.github/workflows/pre-integration.yml)
utilizes Azure Resource deployment - and can only be run by the spetlr-org admins.

To deploy the necessary Azure resources to your own Azure Tenant, run the following command:

```powershell
.\.github\deploy\deploy.ps1 -uniqueRunId "yourUniqueId"
```
Be aware that the applied name for *uniqueRunId* should only contain lower case and numbers, and its length should not
exceed 12 characters.

Afterward, execute the following commands:

```powershell
.\.github\submit\build.ps1
.\.github\submit\submit_test_job.ps1
```

# General Project Info

[![Github top language](https://img.shields.io/github/languages/top/spetlr-org/spetlr)](https://github.com/spetlr-org/spetlr)
[![Github stars](https://img.shields.io/github/stars/spetlr-org/spetlr)](https://github.com/spetlr-org/spetlr)
[![Github forks](https://img.shields.io/github/forks/spetlr-org/spetlr)](https://github.com/spetlr-org/spetlr)
[![Github size](https://img.shields.io/github/repo-size/spetlr-org/spetlr)](https://github.com/spetlr-org/spetlr)
[![Issues Open](https://img.shields.io/github/issues/spetlr-org/spetlr.svg?logo=github)](https://github.com/spetlr-org/spetlr/issues)
[![PyPI spetlr badge](https://img.shields.io/pypi/v/spetlr)](https://pypi.org/project/spetlr/)

# Contributing

Feel free to contribute to SPETLR. Any contributions are appreciated - not only new features, but also if you find a way
to improve SPETLR.

If you have a suggestion that can enhance SPETLR, please fork the repository and create a pull request. Alternatively,
you can open an issue with the "enhancement" tag.

1. Fork the Project
2. Create your Feature Branch (`git checkout -b feature/NewSPETLRFeature`)
3. Commit your Changes (`git commit -m 'Add some SEPTLRFeature'`)
4. Push to the Branch (`git push origin feature/NewSPETLRFeature`)
5. Open a Pull Request

# Build Status

[![Post-Integration](https://github.com/spetlr-org/spetlr/actions/workflows/post-integration.yml/badge.svg)](https://github.com/spetlr-org/spetlr/actions/workflows/post-integration.yml)

# Releases

Releases to PyPI is an Github Action which needs to be manually triggered.

[![Release](https://github.com/spetlr-org/spetlr/actions/workflows/release.yml/badge.svg)](https://github.com/spetlr-org/spetlr/actions/workflows/release.yml)
[![PyPI spetlr badge](https://img.shields.io/pypi/v/spetlr)](https://pypi.org/project/spetlr/)

# Requirements and dependencies
The library has three txt-files at the root of the repo. These files defines three levels of requirements:
- `requirements_install.txt` - this file contains the required libraries to be able to install spetlr.
- `requirements_test.txt` - libraries required to run unit- and integration tests
- `requirements_dev.txt` - libraries required in the development process in order to contribute to the repo

All libraries and their dependencies are added with a fixed version to the configuration file `setup.cfg` using the defined requirements from `requirements_install.txt`.

To __upgrade__ the the dependencies in the `setup.cfg` file do the following:

1. Create a new branch
2. Run `upgrade_requirements.ps1` in your terminal
3. Commit the changes the script has made to the cfg file. If there are no changes, everything is up to date.
4. The PR runs all tests and ensure that the library is compliant with any updates

Note that if it is desired to upgrade a dependency, but not to its newest version, it is possible to set the desired version in the `requirements_install.txt`, then this will be respected by the upgrade script.

# Contact

For any inquiries, please use the [SPETLR Discord Server](https://discord.gg/p9bzqGybVW).


# Hyperledger Sawtooth

This project has moved (see below).

Hyperledger Sawtooth was a project to provide an enterprise solution for
building, deploying, and running distributed ledgers (also called blockchains).

# Project Status

This Hyperledger project, Hyperledger Sawtooth, has been archived and is no
longer active within Hyperledger.

Sawtooth is now maintained by the Splinter community. For more information,
visit: https://github.com/splintercommunity/sawtooth-core/

# License

Hyperledger Sawtooth software is licensed under the [Apache License Version 2.0](LICENSE) software license.


OLMo: Open Language Model

OLMo is a repository for training and using AI2's state-of-the-art open language models.
It is built by scientists, for scientists.

## Installation

First install [PyTorch](https://pytorch.org) according to the instructions specific to your operating system.

To install from source (recommended for training/fine-tuning) run:

```bash
git clone https://github.com/allenai/OLMo.git
cd OLMo
pip install -e .[all]
```

Otherwise you can install the model code by itself directly from PyPI with:

```bash
pip install ai2-olmo
```

## Models

### Overview

The core models in the OLMo family released so far are (all trained on the [Dolma dataset](https://huggingface.co/datasets/allenai/dolma)):
| Model | Training Tokens | Context Length | Training Config | W&B Logs | Data Order File(s) ☨ |
|-------|-----------------|:--------------:|-----------------|----------|--------------------|
| [OLMo 1B](https://huggingface.co/allenai/OLMo-1B) | 3 Trillion | 2048 | [configs/official/OLMo-1B.yaml](https://github.com/allenai/OLMo/blob/main/configs/official/OLMo-1B.yaml) | [wandb.ai/…/OLMo-1B](https://wandb.ai/ai2-llm/OLMo-1B/reports/OLMo-1B--Vmlldzo2NzY1Njk1) | [epoch 1](https://olmo-checkpoints.org/ai2-llm/olmo-small/46zc5fly/train_data/global_indices.npy) |
| [OLMo 7B](https://huggingface.co/allenai/OLMo-7B) | 2.5 Trillion | 2048 | [configs/official/OLMo-7B.yaml](https://github.com/allenai/OLMo/blob/main/configs/official/OLMo-7B.yaml) | [wandb.ai/…/OLMo-7B](https://wandb.ai/ai2-llm/OLMo-7B/reports/OLMo-7B--Vmlldzo2NzQyMzk5) | [epoch 1](https://olmo-checkpoints.org/ai2-llm/olmo-medium/wvc30anm/train_data/global_indices.npy), [epoch 2](https://olmo-checkpoints.org/ai2-llm/olmo-medium/wd2gxrza/train_data/global_indices.npy) |
| [OLMo 7B Twin 2T](https://huggingface.co/allenai/OLMo-7B-Twin-2T) | 2 Trillion  | 2048 | [configs/official/OLMo-7B.yaml](https://github.com/allenai/OLMo/blob/main/configs/official/OLMo-7B.yaml) | [wandb.ai/…/OLMo-7B-Twin-2T](https://wandb.ai/ai2-llm/OLMo-7B/reports/OLMo-7B-Twin-2T--Vmlldzo2NzU0NTIz) | [epoch 1](https://olmo-checkpoints.org/ai2-llm/olmo-medium/wvc30anm/train_data/global_indices.npy) |

> ☨ *See [Inspecting training data](#inspecting-training-data) below for usage.*

### Checkpoints

URLs to checkpoints at intermediate steps of the models' trainings can be found in the csv files under [`checkpoints/official/`](https://github.com/allenai/OLMo/blob/main/checkpoints/official). These 'directory' URLs cannot currently be directly accessed, but files within the directory are publicly accessible. These URLs can also be provided to the training script to resume training from the checkpoint (see [Training](#training)). Each checkpoint directory consists of:

- `config.yaml`: the config at that training step.
- `model.pt`, `optim.pt`, `train.pt`: model, optimizer and training state at that training step.

Details about the other types of OLMo checkpoints (including OLMo HF Transformers checkpoints) can be found in [Checkpoints.md](https://github.com/allenai/OLMo/blob/main/docs/Checkpoints.md).

## Inference

You can utilize our Hugging Face integration to run inference on the OLMo Transformers checkpoints:

```python
from transformers import AutoModelForCausalLM, AutoTokenizer

olmo = AutoModelForCausalLM.from_pretrained("allenai/OLMo-1.7-7B-hf")
tokenizer = AutoTokenizer.from_pretrained("allenai/OLMo-1.7-7B-hf")

message = ["Language modeling is "]
inputs = tokenizer(message, return_tensors='pt', return_token_type_ids=False)
response = olmo.generate(**inputs, max_new_tokens=100, do_sample=True, top_k=50, top_p=0.95)
print(tokenizer.batch_decode(response, skip_special_tokens=True)[0])
```

Alternatively, with the Hugging Face pipeline abstraction:

```python
from transformers import pipeline
olmo_pipe = pipeline("text-generation", model="allenai/OLMo-1.7-7B-hf")
print(olmo_pipe("Language modeling is"))
```

### Inference on finetuned checkpoints

If you finetune the model using the code in [Fine-tuning](#fine-tuning), you can use the conversion script to convert a native OLMo checkpoint to a Hugging Face-compatible checkpoint.

```bash
python scripts/convert_olmo_to_hf_new.py --input_dir /path/to/olmo/checkpoint --output_dir /path/to/hf/checkpoint/ --tokenizer_json_path tokenizers/allenai_gpt-neox-olmo-dolma-v1_5.json
```

### Quantization

```python
olmo = AutoModelForCausalLM.from_pretrained("allenai/OLMo-1.7-7B-hf", torch_dtype=torch.float16, load_in_8bit=True)  # requires bitsandbytes
```

The quantized model is more sensitive to typing / cuda, so it is recommended to pass the inputs as inputs.input_ids.to('cuda') to avoid potential issues.

## Reproducibility

### Training

The configs used to train the official OLMo models are provided in the [`configs/official/`](https://github.com/allenai/OLMo/blob/main/configs/official) directory.

Note that while the training and validation data is public and free to download, the paths to the data within those configs are pointed at a CloudFlare R2 bucket, which requires an API key for programmatic access.
So in order to use any of these configs to reproduce a training run you'll first have to download the corresponding data to a location of your choosing and then update the paths in the config accordingly.

You can derive the public HTTP URL from an R2 URL by replacing `r2://olmo-data` with `https://olmo-data.org`.
For example, if the R2 data URL is:

`r2://olmo-data/preprocessed/olmo-mix/v1_5/gpt-neox-20b-pii-special/part-000-00000.npy`

then the corresponding public URL is:

`https://olmo-data.org/preprocessed/olmo-mix/v1_5/gpt-neox-20b-pii-special/part-000-00000.npy`

Once you've updated the data paths in the config you can launch a training run via `torchrun`. For example, to launch the 1B model training on a single 8x GPU node, you would run:

```bash
torchrun --nproc_per_node=8 scripts/train.py configs/official/OLMo-1B.yaml
```

You can use the same method to launch multi-node jobs as well. See [the documentation](https://pytorch.org/docs/stable/elastic/run.html) for `torchrun` to understand the additional arguments you'll need to configure the rendezvous backend / endpoint.

To resume training from a checkpoint, you can pass its path (local or URL)
to `scripts/train.py` with the `--load_path` arguments. For example, to resume training from step 1000 of the OLMo 1B run:

```bash
torchrun --nproc_per_node=8 scripts/train.py configs/official/OLMo-1B.yaml --load_path https://olmo-checkpoints.org/ai2-llm/olmo-small/w1r5xfzt/step1000-unsharded
```

### Inspecting training data

You may be interested in inspecting the exact tokens that composed a particular batch during the training of one of the OLMo models.
We provide tools to do this, but first you'll need to download the data as above (unless you have an R2 API key) and update the corresponding config accordingly.

Then take note of the URL of the data order file you want, which can be found in the [Models Overview](#models-overview) table. For example, the data order file for the first epoch of the OLMo-7B model is [https://olmo-checkpoints.org/ai2-llm/olmo-medium/wvc30anm/train_data/global_indices.npy](https://olmo-checkpoints.org/ai2-llm/olmo-small/46zc5fly/train_data/global_indices.npy).

Once you have that you can use this snippet to inspect the data within a particular batch:

```python
import numpy as np
from cached_path import cached_path

from olmo.config import TrainConfig
from olmo.data import build_memmap_dataset

# Update these paths to what you want:
data_order_file_path = cached_path("https://olmo-checkpoints.org/ai2-llm/olmo-medium/wvc30anm/train_data/global_indices.npy")
train_config_path = "configs/official/OLMo-7B.yaml"

cfg = TrainConfig.load(train_config_path)
dataset = build_memmap_dataset(cfg, cfg.data)
batch_size = cfg.global_train_batch_size
global_indices = np.memmap(data_order_file_path, mode="r+", dtype=np.uint32)

def get_batch_instances(batch_idx: int) -> list[list[int]]:
    batch_start = batch_idx * batch_size
    batch_end = (batch_idx + 1) * batch_size
    batch_indices = global_indices[batch_start:batch_end]
    batch_instances = []
    for index in batch_indices:
        token_ids = dataset[index]["input_ids"].tolist()
        batch_instances.append(token_ids)
    return batch_instances

# Get all 2048 x 2048 token IDs in the first batch.
get_batch_instances(0)
```

## Fine-tuning

To fine-tune an OLMo model using our trainer you'll first need to prepare your dataset by tokenizing it and saving the tokens IDs to a flat numpy memory-mapped array. See [`scripts/prepare_tulu_data.py`](./scripts/prepare_tulu_data.py) for an example with the Tulu V2 dataset, which can be easily modified for other datasets.

Next, prepare your training config. There are many examples in the [`configs/`](https://github.com/allenai/OLMo/blob/main/configs) directory that you can use as a starting point. The most important thing is to make sure the model parameters (the `model` field in the config) match up with the checkpoint you're starting from. To be safe you can always start from the config that comes with the model checkpoint. At a minimum you'll need to make the following changes to the config or provide the corresponding overrides from the command line:

- Update `load_path` to point to the checkpoint you want to start from.
- Set `reset_trainer_state` to `true`.
- Update `data.paths` to point to the `token_ids.npy` file you generated.
- Optionally update `data.label_mask_paths` to point to the `label_mask.npy` file you generated, unless you don't need special masking for the loss.
- Update `evaluators` to add/remove in-loop evaluations.

Once you're satisfied with your training config, you can launch the training job via `torchrun`. For example:

```
torchrun --nproc_per_node=8 scripts/train.py {path_to_train_config} \
    --data.paths=[{path_to_data}/input_ids.npy] \
    --data.label_mask_paths=[{path_to_data}/label_mask.npy] \
    --load_path={path_to_checkpoint} \
    --reset_trainer_state
```

Note: passing CLI overrides like `--reset_trainer_state` is only necessary if you didn't update those fields in your config.

## Evaluation

Additional tools for evaluating OLMo models are available at the [OLMo Eval](https://github.com/allenai/ai2-olmo-eval) repo.

## Citing

```bibtex
@article{OLMo,
  title={OLMo: Accelerating the Science of Language Models},
  author={Dirk Groeneveld and Iz Beltagy and Pete Walsh and Akshita Bhagia and Rodney Kinney and Oyvind Tafjord and A. Jha and Hamish Ivison and Ian Magnusson and Yizhong Wang and Shane Arora and David Atkinson and Russell Authur and Khyathi Raghavi Chandu and Arman Cohan and Jennifer Dumas and Yanai Elazar and Yuling Gu and Jack Hessel and Tushar Khot and William Merrill and Jacob Daniel Morrison and Niklas Muennighoff and Aakanksha Naik and Crystal Nam and Matthew E. Peters and Valentina Pyatkin and Abhilasha Ravichander and Dustin Schwenk and Saurabh Shah and Will Smith and Emma Strubell and Nishant Subramani and Mitchell Wortsman and Pradeep Dasigi and Nathan Lambert and Kyle Richardson and Luke Zettlemoyer and Jesse Dodge and Kyle Lo and Luca Soldaini and Noah A. Smith and Hanna Hajishirzi},
  year={2024},
  url={https://api.semanticscholar.org/CorpusID:267365485},
  journal={arXiv preprint},
}
```


# Datapipe

[Datapipe](https://datapipe.dev/) is a real-time, incremental ETL library for Python with record-level dependency tracking.

The library is designed for describing data processing pipelines and is capable
of tracking dependencies for each record in the pipeline. This ensures that
tasks within the pipeline receive only the data that has been modified, thereby
improving the overall efficiency of data handling.

https://datapipe.dev/

# Development

At the moment these branches are active:

* `master` - current development state, will be promoted to `0.13.x` series
  release once ready
* `v0.13` - current stable version
* `v0.11` - legacy stable version (`v0.12` was skipped due reasons)

# Version compatibility

At the moment, the datapipe library is under active development. Versions:
`v0.*.*`

It should be expected that each minor version is not backward compatible with
the previous one. That is, `v0.7.0` is not compatible with `v0.6.1`. Dependencies
should be fixed to the exact minor version.

After stabilization and transition to the major version `v1.*.*`, the common
rules will apply: all versions with the same major component are compatible.


***
PEX
***
.. image:: https://github.com/pex-tool/pex/workflows/CI/badge.svg?branch=main
    :target: https://github.com/pex-tool/pex/actions?query=branch%3Amain+workflow%3ACI
.. image:: https://img.shields.io/pypi/l/pex.svg
    :target: https://pypi.org/project/pex/
.. image:: https://img.shields.io/pypi/v/pex.svg
    :target: https://pypi.org/project/pex/
.. image:: https://img.shields.io/pypi/pyversions/pex.svg
    :target: https://pypi.org/project/pex/
.. image:: https://img.shields.io/pypi/wheel/pex.svg
    :target: https://pypi.org/project/pex/#files

.. contents:: **Contents**

Overview
========
pex is a library for generating .pex (Python EXecutable) files which are
executable Python environments in the spirit of `virtualenvs <http://virtualenv.org>`_.
pex is an expansion upon the ideas outlined in
`PEP 441 <https://peps.python.org/pep-0441/>`_
and makes the deployment of Python applications as simple as ``cp``.  pex files may even
include multiple platform-specific Python distributions, meaning that a single pex file
can be portable across Linux and OS X.

pex files can be built using the ``pex`` tool.  Build systems such as `Pants
<http://pantsbuild.org/>`_, `Buck <http://facebook.github.io/buck/>`_, and  `{py}gradle <https://github.com/linkedin/pygradle>`_  also
support building .pex files directly.

Still unsure about what pex does or how it works?  Watch this quick lightning
talk: `WTF is PEX? <http://www.youtube.com/watch?v=NmpnGhRwsu0>`_.

pex is licensed under the Apache2 license.

Installation
============

To install pex, simply

.. code-block:: bash

    $ pip install pex

You can also build pex in a git clone using tox:

.. code-block:: bash

    $ tox -e package
    $ cp dist/pex ~/bin

This builds a pex binary in ``dist/pex`` that can be copied onto your ``$PATH``.
The advantage to this approach is that it keeps your Python environment as empty as
possible and is more in-line with what pex does philosophically.

Simple Examples
===============

Launch an interpreter with ``requests``, ``flask`` and ``psutil`` in the environment:

.. code-block:: bash

    $ pex requests flask 'psutil>2,<3'

Save Dependencies From Pip
~~~~~~~~~~~~~~~~~~~~~~~~~~

Or instead freeze your current virtualenv via requirements.txt and execute it anywhere:

.. code-block:: bash

    $ pex $(pip freeze) -o my_virtualenv.pex
    $ deactivate
    $ ./my_virtualenv.pex

Ephemeral Environments
~~~~~~~~~~~~~~~~~~~~~~

Run webserver.py in an environment containing ``flask`` as a quick way to experiment:

.. code-block:: bash

    $ pex flask -- webserver.py

Launch Sphinx in an ephemeral pex environment using the Sphinx entry point ``sphinx:main``:

.. code-block:: bash

    $ pex sphinx -e sphinx:main -- --help

Using Entry Points
~~~~~~~~~~~~~~~~~~

Projects specifying a ``console_scripts`` entry point in their configuration
can build standalone executables for those entry points.

To build a standalone ``pex-tools-executable.pex`` binary that runs the
``pex-tools`` console script found in all pex version ``2.1.35`` and newer distributions:

.. code-block:: bash

    $ pex "pex>=2.1.35" --console-script pex-tools --output-file pex-tools-executable.pex

Specifying A Specific Interpreter
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

You can also build pex files that use a specific interpreter type:

.. code-block:: bash

    $ pex "pex>=2.1.35" -c pex-tools --python=pypy -o pex-tools-pypy-executable.pex

Most pex options compose well with one another, so the above commands can be
mixed and matched, and equivalent short options are available.

For a full list of options, just type ``pex --help``.

Integrating pex into your workflow
==================================

If you use tox (and you should!), a simple way to integrate pex into your
workflow is to add a packaging test environment to your ``tox.ini``:

.. code-block:: ini

    [testenv:package]
    deps = pex
    commands = pex . -o dist/app.pex

Then ``tox -e package`` will produce a relocatable copy of your application
that you can copy to staging or production environments.

Documentation
=============

More documentation about Pex, building .pex files, and how .pex files work
is available at https://docs.pex-tool.org.

Development
===========

Pex uses `tox <https://tox.wiki/en/latest/>`_ for test and development automation. To run
the test suite, just invoke tox:

.. code-block:: bash

    $ tox

If you don't have tox, you can generate a pex of tox:

.. code-block::

    $ pex tox -c tox -o ~/bin/tox

Tox provides many useful commands and options, explained at https://tox.wiki/en/latest/ .
Below, we provide some of the most commonly used commands used when working on Pex, but the
docs are worth acquainting yourself with to better understand how Tox works and how to do more
advanced commands.

To run a specific environment, identify the name of the environment you'd like to invoke by
running ``tox --listenvs-all``, then invoke like this:

.. code-block::

    $ tox -e fmt

To run MyPy:

.. code-block::

    $ tox -e check

All of our tox test environments allow passthrough arguments, which can be helpful to run
specific tests:

.. code-block::

    $ tox -e py37-integration -- -k test_reproducible_build

To run Pex from source, rather than through what is on your PATH, invoke via Python:

.. code-block::

    $ python -m pex


![Ikabot Banner](assets/banner.png)

[![Downloads](https://static.pepy.tech/badge/ikabot)](https://pepy.tech/project/ikabot) ![GitHub Repo stars](https://img.shields.io/github/stars/Ikabot-Collective/ikabot?style=flat) ![licence](https://img.shields.io/github/license/Ikabot-Collective/ikabot?style=flat)

Ikabot is a cross-platform Python program that enhances your Ikariam game by offering premium features for free, focusing on city management, warfare strategies, and streamlined alerts.

-[Discover the features](https://github.com/Ikabot-Collective/ikabot/wiki/Features)-

## Getting Started
- Checkout the [Getting Started guide](https://github.com/Ikabot-Collective/ikabot/wiki/Getting-Started)

## Contribution Guidelines

- Before contributing code, read the [Contribution Guidelines](.github/CONTRIBUTING.md)
  - GitHub issues are for bugs and feature requests.
  - For questions, help and chat in general, join our Discord Server:

     [![Discord server!](https://invidget.switchblade.xyz/3hyxPRj)](http://discord.gg/3hyxPRj)

- Want to say thanks?
  - Hit the :star: Star :star: button

## Contributors

This project exists thanks to all the people who contribute.

## Screenshots

![Ikabot Screenshot](assets/screenshot-1.jpg)


.. image:: https://img.shields.io/badge/lifecycle-maturing-blue
   :target: https://www.tidyverse.org/lifecycle/
.. image:: https://dpcbuild.deltares.nl/app/rest/builds/buildType:id:iMOD6_IMODPython_Windows_Tests/statusIcon.svg
   :target: https://github.com/Deltares/imod-python/commits/master/
.. image:: https://img.shields.io/pypi/l/imod
   :target: https://choosealicense.com/licenses/mit/
.. image:: https://img.shields.io/conda/vn/conda-forge/imod.svg
   :target: https://github.com/conda-forge/imod-feedstock

The ``imod`` Python package is an open source project to make working with
MODFLOW groundwater models in Python easier. It builds on top of popular
packages such as `xarray`_, `pandas`_, `geopandas`_, `dask`_,  and `rasterio`_
to provide a versatile toolset for working with (large) groundwater (modeling)
data:

* Preparing and modifying data from a variety of GIS, scientific, and MODFLOW
  file formats;
* Writing data to MODFLOW-based models;
* Selecting and evaluating for e.g. time series comparison or water budgets;
* Visualizing cross sections, time series, or 3D animations.

We currently support the following MODFLOW-based models:

* `USGS MODFLOW 6`_, structured (DIS) and discretization by vertices (DISV)
  grids only, and not all advanced stress packages (only LAK and UZF)
* `iMOD-WQ`_, which integrates SEAWAT (density-dependent
  groundwater flow) and MT3DMS (multi-species reactive transport calculations)
* `iMODFLOW`_

Development currently focuses on supporting more Modflow 6 functionalities.

This Python package is developed primarily by the Groundwater Management Group
at `Deltares`_. It is used together with a broader set of open source tools and
standards for reproducible modeling and data analysis:

* `Git`_: version control of (Python) scripts;
* `DVC`_: version control of data, on top of Git;
* `netCDF`_: open standard of a flexible, self describing data format;
* `Snakemake`_: workflow manager to turn a collection of scripts into a
  workflow.

Documentation: https://deltares.github.io/imod-python

Source code: https://github.com/Deltares/imod-python

Issues: https://github.com/Deltares/imod-python/issues

.. _Deltares: https://www.deltares.nl
.. _dask: https://dask.org/
.. _xarray: http://xarray.pydata.org/
.. _pandas: http://pandas.pydata.org/
.. _rasterio: https://rasterio.readthedocs.io/en/latest/
.. _geopandas: http://geopandas.org/
.. _Git: https://git-scm.com/
.. _DVC: https://dvc.org/
.. _netCDF: https://www.unidata.ucar.edu/software/netcdf/
.. _Snakemake: https://snakemake.readthedocs.io/en/stable/
.. _USGS MODFLOW 6: https://www.usgs.gov/software/modflow-6-usgs-modular-hydrologic-model
.. _iMODFLOW: https://oss.deltares.nl/web/imod
.. _iMOD-WQ: https://oss.deltares.nl/web/imod


cob_common
===========

## GitHub Actions - Continuous Integration

CI-Status ```kinetic_dev```: [![GHA CI](https://github.com/4am-robotics/cob_common/actions/workflows/main.yml/badge.svg?branch=kinetic_dev)](https://github.com/4am-robotics/cob_common/actions/workflows/main.yml?query=branch%3Akinetic_dev)


cob_command_tools
===========

## GitHub Actions - Continuous Integration

CI-Status ```noetic-devel```: [![GHA CI](https://github.com/4am-robotics/cob_command_tools/actions/workflows/main.yml/badge.svg?branch=noetic-devel)](https://github.com/4am-robotics/cob_command_tools/actions/workflows/main.yml?query=branch%3Anoetic-devel)


# 🕷 ScrapeGraphAI: You Only Scrape Once
[English](https://github.com/VinciGit00/Scrapegraph-ai/blob/main/README.md) | [中文](https://github.com/VinciGit00/Scrapegraph-ai/blob/main/docs/chinese.md) | [日本語](https://github.com/VinciGit00/Scrapegraph-ai/blob/main/docs/japanese.md)
| [코리아노](https://github.com/VinciGit00/Scrapegraph-ai/blob/main/docs/korean.md)
| [Русский](https://github.com/VinciGit00/Scrapegraph-ai/blob/main/docs/russian.md)

[![Downloads](https://img.shields.io/pepy/dt/scrapegraphai?style=for-the-badge)](https://pepy.tech/project/scrapegraphai)
[![linting: pylint](https://img.shields.io/badge/linting-pylint-yellowgreen?style=for-the-badge)](https://github.com/pylint-dev/pylint)
[![Pylint](https://img.shields.io/github/actions/workflow/status/VinciGit00/Scrapegraph-ai/pylint.yml?label=Pylint&logo=github&style=for-the-badge)](https://github.com/VinciGit00/Scrapegraph-ai/actions/workflows/pylint.yml)
[![CodeQL](https://img.shields.io/github/actions/workflow/status/VinciGit00/Scrapegraph-ai/codeql.yml?label=CodeQL&logo=github&style=for-the-badge)](https://github.com/VinciGit00/Scrapegraph-ai/actions/workflows/codeql.yml)
[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg?style=for-the-badge)](https://opensource.org/licenses/MIT)
[![](https://dcbadge.vercel.app/api/server/gkxQDAjfeX)](https://discord.gg/gkxQDAjfeX)

ScrapeGraphAI is a *web scraping* python library that uses LLM and direct graph logic to create scraping pipelines for websites and local documents (XML, HTML, JSON, etc.).

Just say which information you want to extract and the library will do it for you!

## 🚀 Quick install

The reference page for Scrapegraph-ai is available on the official page of PyPI: [pypi](https://pypi.org/project/scrapegraphai/).

```bash
pip install scrapegraphai
```

**Note**: it is recommended to install the library in a virtual environment to avoid conflicts with other libraries 🐱

## 🔍 Demo
Official streamlit demo:

[![My Skills](https://skillicons.dev/icons?i=react)](https://scrapegraph-ai-web-dashboard.streamlit.app)

Try it directly on the web using Google Colab:

[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/drive/1sEZBonBMGP44CtO6GQTwAlL0BGJXjtfd?usp=sharing)

## 📖 Documentation

The documentation for ScrapeGraphAI can be found [here](https://scrapegraph-ai.readthedocs.io/en/latest/).

Check out also the Docusaurus [here](https://scrapegraph-doc.onrender.com/).

## 💻 Usage
There are multiple standard scraping pipelines that can be used to extract information from a website (or local file):
- `SmartScraperGraph`: single-page scraper that only needs a user prompt and an input source;
- `SearchGraph`: multi-page scraper that extracts information from the top n search results of a search engine;
- `SpeechGraph`: single-page scraper that extracts information from a website and generates an audio file.
- `ScriptCreatorGraph`: single-page scraper that extracts information from a website and generates a Python script.

- `SmartScraperMultiGraph`: multi-page scraper that extracts information from multiple pages given a single prompt and a list of sources;
- `ScriptCreatorMultiGraph`: multi-page scraper that generates a Python script for extracting information from multiple pages given a single prompt and a list of sources.

It is possible to use different LLM through APIs, such as **OpenAI**, **Groq**, **Azure** and **Gemini**, or local models using **Ollama**.

### Case 1: SmartScraper using Local Models

Remember to have [Ollama](https://ollama.com/) installed and download the models using the **ollama pull** command.

```python
from scrapegraphai.graphs import SmartScraperGraph

graph_config = {
    "llm": {
        "model": "ollama/mistral",
        "temperature": 0,
        "format": "json",  # Ollama needs the format to be specified explicitly
        "base_url": "http://localhost:11434",  # set Ollama URL
    },
    "embeddings": {
        "model": "ollama/nomic-embed-text",
        "base_url": "http://localhost:11434",  # set Ollama URL
    },
    "verbose": True,
}

smart_scraper_graph = SmartScraperGraph(
    prompt="List me all the projects with their descriptions",
    # also accepts a string with the already downloaded HTML code
    source="https://perinim.github.io/projects",
    config=graph_config
)

result = smart_scraper_graph.run()
print(result)

```

The output will be a list of projects with their descriptions like the following:

```python
{'projects': [{'title': 'Rotary Pendulum RL', 'description': 'Open Source project aimed at controlling a real life rotary pendulum using RL algorithms'}, {'title': 'DQN Implementation from scratch', 'description': 'Developed a Deep Q-Network algorithm to train a simple and double pendulum'}, ...]}
```

### Case 2: SearchGraph using Mixed Models

We use **Groq** for the LLM and **Ollama** for the embeddings.

```python
from scrapegraphai.graphs import SearchGraph

# Define the configuration for the graph
graph_config = {
    "llm": {
        "model": "groq/gemma-7b-it",
        "api_key": "GROQ_API_KEY",
        "temperature": 0
    },
    "embeddings": {
        "model": "ollama/nomic-embed-text",
        "base_url": "http://localhost:11434",  # set ollama URL arbitrarily
    },
    "max_results": 5,
}

# Create the SearchGraph instance
search_graph = SearchGraph(
    prompt="List me all the traditional recipes from Chioggia",
    config=graph_config
)

# Run the graph
result = search_graph.run()
print(result)
```

The output will be a list of recipes like the following:

```python
{'recipes': [{'name': 'Sarde in Saòre'}, {'name': 'Bigoli in salsa'}, {'name': 'Seppie in umido'}, {'name': 'Moleche frite'}, {'name': 'Risotto alla pescatora'}, {'name': 'Broeto'}, {'name': 'Bibarasse in Cassopipa'}, {'name': 'Risi e bisi'}, {'name': 'Smegiassa Ciosota'}]}
```
### Case 3: SpeechGraph using OpenAI

You just need to pass the OpenAI API key and the model name.

```python
from scrapegraphai.graphs import SpeechGraph

graph_config = {
    "llm": {
        "api_key": "OPENAI_API_KEY",
        "model": "gpt-3.5-turbo",
    },
    "tts_model": {
        "api_key": "OPENAI_API_KEY",
        "model": "tts-1",
        "voice": "alloy"
    },
    "output_path": "audio_summary.mp3",
}

# ************************************************
# Create the SpeechGraph instance and run it
# ************************************************

speech_graph = SpeechGraph(
    prompt="Make a detailed audio summary of the projects.",
    source="https://perinim.github.io/projects/",
    config=graph_config,
)

result = speech_graph.run()
print(result)

```

The output will be an audio file with the summary of the projects on the page.

## Sponsors

## 🤝 Contributing

Feel free to contribute and join our Discord server to discuss with us improvements and give us suggestions!

Please see the [contributing guidelines](https://github.com/VinciGit00/Scrapegraph-ai/blob/main/CONTRIBUTING.md).

[![My Skills](https://skillicons.dev/icons?i=discord)](https://discord.gg/uJN7TYcpNa)
[![My Skills](https://skillicons.dev/icons?i=linkedin)](https://www.linkedin.com/company/scrapegraphai/)
[![My Skills](https://skillicons.dev/icons?i=twitter)](https://twitter.com/scrapegraphai)

## 📈 Roadmap
Check out the project roadmap [here](https://github.com/VinciGit00/Scrapegraph-ai/blob/main/docs/README.md)! 🚀

Wanna visualize the roadmap in a more interactive way? Check out the [markmap](https://markmap.js.org/repl) visualization by copy pasting the markdown content in the editor!

## ❤ Contributors
[![Contributors](https://contrib.rocks/image?repo=VinciGit00/Scrapegraph-ai)](https://github.com/VinciGit00/Scrapegraph-ai/graphs/contributors)

## 🎓 Citations
If you have used our library for research purposes please quote us with the following reference:
```text
  @misc{scrapegraph-ai,
    author = {Marco Perini, Lorenzo Padoan, Marco Vinciguerra},
    title = {Scrapegraph-ai},
    year = {2024},
    url = {https://github.com/VinciGit00/Scrapegraph-ai},
    note = {A Python library for scraping leveraging large language models}
  }
```

## Authors

|                    | Contact Info         |
|--------------------|----------------------|
| Marco Vinciguerra  | [![Linkedin Badge](https://img.shields.io/badge/-Linkedin-blue?style=flat&logo=Linkedin&logoColor=white)](https://www.linkedin.com/in/marco-vinciguerra-7ba365242/)    |
| Marco Perini       | [![Linkedin Badge](https://img.shields.io/badge/-Linkedin-blue?style=flat&logo=Linkedin&logoColor=white)](https://www.linkedin.com/in/perinim/)   |
| Lorenzo Padoan     | [![Linkedin Badge](https://img.shields.io/badge/-Linkedin-blue?style=flat&logo=Linkedin&logoColor=white)](https://www.linkedin.com/in/lorenzo-padoan-4521a2154/)  |

## 📜 License

ScrapeGraphAI is licensed under the MIT License. See the [LICENSE](https://github.com/VinciGit00/Scrapegraph-ai/blob/main/LICENSE) file for more information.

## Acknowledgements

- We would like to thank all the contributors to the project and the open-source community for their support.
- ScrapeGraphAI is meant to be used for data exploration and research purposes only. We are not responsible for any misuse of the library.


ToolDelta - Multi Platform Edition

  ToolDelta 是依赖于 PhoenixBuilder | Neo Omega 的多功能扩展组件，可以加载各种各样有趣的插件。

ToolDelta 可以使得你的 中国版《我的世界》基岩版租赁服 的玩法和功能更加多样化。

维基和用户指南 | 交流群

## 目录
- [目录](#目录)
- [注意事项](#注意事项)
- [友情链接](#友情链接)
- [运行和配置](#运行和配置)

## 注意事项
- 遇到平台不兼容的情况请尽快提交 `Issue`
- 已支持 `Termux`

## 友情链接
- [如何使用](https://tooldelta-wiki.tblstudio.cn/learn-use.html)
- [Docker 镜像](https://hub.docker.com/r/wlingzhenyu/tooldelta)

## 运行和配置
 - 直接下载 ToolDelta 可执行文件并运行
    - 发行版 ToolDelta 下载点这里
 - 在 `pip` 安装 ToolDelta 并运行
    - `pip install tooldelta`
    - `echo "import tooldelta;tooldelta.client_title()"`
 - 在docker运行
    ```bash
    docker run -v .:/app -t -i wlingzhenyu/tooldelta:latest
    ```


# ungoogled-chromium-binaries

Website for ungoogled-chromium contributor-submitted binaries (contributor binaries).

**IMPORTANT: These binaries are provided by anyone who are willing to build and submit them. Because these binaries are not necessarily [reproducible](https://reproducible-builds.org/), authenticity cannot be guaranteed.**

[**Link to the website here**](//ungoogled-software.github.io/ungoogled-chromium-binaries/)

[Atom feed for new binaries here](https://ungoogled-software.github.io/ungoogled-chromium-binaries/feed.xml).

This website uses GitHub Pages on the `master` branch.

For problems, suggestions, and questions, please use the Issue tracker.

## Developer info

When creating new pages or adding new versions, use an existing version as a template to follow. If you have questions or suggestions, please create an Issue.

### Publishing new binaries

**Ensure your binary has source code published to a platform repo**. Otherwise, your PR will not be merged.

Prerequisites:
* Python 3 for all scripts under `utilities/`

#### Steps

Steps to publish a new binary. An example of these steps is in the next section.

1. Ensure your binary has source code published to a platform repo.
2. Fork the main binaries repository ([ungoogled-software/ungoogled-chromium-binaries](//github.com/ungoogled-software/ungoogled-chromium-binaries))
    * If this has been done before, pull in new changes from this one if necessary.
3. Create a new Release (i.e. using GitHub's Release feature) in the fork and upload binaries to it. The tag name used must be unique for each Release; it normally matches the ungoogled-chromium version.
4. Upload binaries to the new Release
5. Use `utilities/submit_github_binary.py` to generate an INI file with the correct URLs to the GitHub Release. It requires paths to a locally stored copy of the binaries for computing hashes. Use the `-h` or `--help` argument for more details.
    * If a directory structure in `config/platforms` doesn't exist for the binary's target platform and version, create the necessary directories with the associated `display_name` files in the same manner as existing platforms.
6. Push your changes in the repository; these changes should only be of files in `config/`. Make a pull request against the main repository.
    * This can be bypassed if one is part of the [Binaries Team](//github.com/orgs/ungoogled-software/teams/binaries-team).

Notes:

* `submit_github_binary.py` is currently restricted to generating INI files with URLs to binaries in GitHub releases. If binaries are uploaded elsewhere, then the INI must be created by other means.
* Additional changes can be made to the website configuration before step 5 as necessary.

#### Example

Example command-line steps (with comments, denoted by a hash `#` symbol). Replace `YOURNAME` in the steps with your GitHub username.

**First-time setup**:

```
# In GitHub, fork ungoogled-software/ungoogled-chromium-binaries to YOURNAME/ungoogled-chromium-binaries
git clone https://github.com/YOURNAME/ungoogled-chromium-binaries.git
cd ungoogled-chromium-binaries
git remote add upstream https://github.com/ungoogled-software/ungoogled-chromium-binaries.git
git config branch.master.remote upstream
git config branch.master.pushRemote origin
```

**Publish binaries**:

The following example demonstrates publishing Debian 9 (buster) amd64 packages located in `/home/user/ungoogled-chromium/build/` (which is assumed to be the shell expanded form of `~/ungoogled-chromium/build/`) for ungoogled-chromium version `99.0.1234.567-1`:

```
# In GitHub, create a new Release on YOURNAME/ungoogled-chromium-binaries with a name "99.0.1234.567-1" (without quotes) and a new tag "99.0.1234.567-1" (without quotes; insert it into the tag field). Upload all necessary files from /home/user/ungoogled-chromium/buildspace/ into the Release.
cd ungoogled-chromium-binaries # The same as the one setup above
git pull
# Create the directories debian/ and debian/buster_amd64 with corresponding display_name files in config/platforms/ ONLY if they do NOT exist.
# NOTE: You can use either --git to fetch the latest tag on the current branch, or specify it manually with --tag
./utilities/submit_github_binary.py --git path/to/ungoogled-chromium-debian --username YOURNAME --output config/platforms/debian/buster_amd64/ ~/ungoogled-chromium/build/ungoogled-chromium*
# A commit will be created automatically. Ensure commits are correct before proceeding
git push
# In GitHub, create a pull request in ungoogled-software/ungoogled-chromium-binaries with the new change in YOURNAME/ungoogled-chromium-binaries
```

**Testing changes**

```
python3 -m pip install -r utilities/requirements.txt
python3 utilities/site_generator.py
./utilities/local_server.py
# Open a webpage to http://localhost:8086
```

## External resources

* [github-markdown-css](//github.com/sindresorhus/github-markdown-css)
* [PyAtom](//github.com/sramana/pyatom)


[![Tests](https://github.com/DHI/terracotta/actions/workflows/test.yml/badge.svg)](https://github.com/DHI/terracotta/actions/workflows/test.yml)
[![Documentation Status](https://readthedocs.org/projects/terracotta-python/badge/?version=latest)](https://terracotta-python.readthedocs.io/en/latest/?badge=latest)
[![codecov](https://codecov.io/gh/DHI-GRAS/terracotta/branch/main/graph/badge.svg?token=u16QBwwvvn)](https://codecov.io/gh/DHI-GRAS/terracotta)
[![GitHub release](https://img.shields.io/github/release-pre/dhi-gras/terracotta.svg)](https://github.com/DHI/terracotta/releases)
[![PyPI release](https://img.shields.io/pypi/v/terracotta.svg)](https://pypi.org/project/terracotta)
[![License](https://img.shields.io/github/license/dhi-gras/terracotta.svg)](https://github.com/DHI/terracotta/blob/main/LICENSE)
[![Python versions](https://img.shields.io/pypi/pyversions/terracotta.svg)](https://pypi.org/project/terracotta)

[![Logo](docs/_figures/logo-banner.svg)](#)

Terracotta is a pure Python tile server that runs as a WSGI app on a
dedicated webserver or as a serverless app on AWS Lambda. It is built on a
modern Python stack, powered by awesome open-source software such as
[Flask](http://flask.pocoo.org), [Zappa](https://github.com/Miserlou/Zappa),
and [Rasterio](https://github.com/mapbox/rasterio).

[Read the docs](https://terracotta-python.readthedocs.io/en/latest) |
[Try the demo](https://terracotta-demo-frontend.orangebeach-11aa4896.westeurope.azurecontainerapps.io/) |
[Explore the API](https://terracotta-python.readthedocs.io/en/latest/apidoc.html) |
[Satlas, powered by Terracotta](http://satlas.dk) |
[Docker Image](https://hub.docker.com/r/dhigras/terracotta/tags)

## Why Terracotta?

- It is trivial to get going. Got a folder full of
  [cloud-optimized GeoTiffs](https://www.cogeo.org/) in different
  projections you want to have a look at in your browser?
  `terracotta serve -r {name}.tif` and
  `terracotta connect localhost:5000` get you there.
- We make minimal assumptions about your data, so *you stay in charge*.
  Keep using the tools you know and love to create and organize your
  data, Terracotta serves it exactly as it is.
- Serverless deployment is a first-priority use case, so you don’t have
  to worry about maintaining or scaling your architecture.
- Terracotta instances are self-documenting. Everything the frontend
  needs to know about your data is accessible from only a handful of
  API endpoints.

## The Terracotta workflow

### 1. Optimize raster files

```bash
$ ls -lh
total 1.4G
-rw-r--r-- 1 dimh 1049089 231M Aug 29 16:45 S2A_20160724_135032_27XVB_B02.tif
-rw-r--r-- 1 dimh 1049089 231M Aug 29 16:45 S2A_20160724_135032_27XVB_B03.tif
-rw-r--r-- 1 dimh 1049089 231M Aug 29 16:46 S2A_20160724_135032_27XVB_B04.tif
-rw-r--r-- 1 dimh 1049089 231M Aug 29 16:56 S2A_20170831_171901_25XEL_B02.tif
-rw-r--r-- 1 dimh 1049089 231M Aug 29 16:57 S2A_20170831_171901_25XEL_B03.tif
-rw-r--r-- 1 dimh 1049089 231M Aug 29 16:57 S2A_20170831_171901_25XEL_B04.tif

$ terracotta optimize-rasters *.tif -o optimized/

Optimizing rasters: 100%|██████████████████████████| [05:16<00:00, file=S2A_20170831_...25XEL_B04.tif]
```

### 2. Create a database from file name pattern

```bash
$ terracotta ingest optimized/S2A_{date}_{}_{tile}_{band}.tif -o greenland.sqlite
Ingesting raster files: 100%|███████████████████████████████████████████| 6/6 [00:49<00:00,  8.54s/it]
```

### 3. Serve it up

```bash
$ terracotta serve -d greenland.sqlite
 * Serving Flask app "terracotta.server" (lazy loading)
 * Environment: production
   WARNING: Do not use the development server in a production environment.
   Use a production WSGI server instead.
 * Debug mode: off
 * Running on http://localhost:5000/ (Press CTRL+C to quit)
```

### 4. Explore the running server

#### Manually

You can use any HTTP-capable client, such as `curl`.
```bash
$ curl localhost:5000/datasets?tile=25XEL
{"page":0,"limit":100,"datasets":[{"date":"20170831","tile":"25XEL","band":"B02"},{"date":"20170831","tile":"25XEL","band":"B03"},{"date":"20170831","tile":"25XEL","band":"B04"}]}
```

Modern browsers (e.g. Chrome or Firefox) will render the JSON as a tree.

#### Interactively

Terracotta also includes a web client. You can start the client (assuming the server is running at http://localhost:5000) using
```bash
$ terracotta connect localhost:5000
 * Serving Flask app "terracotta.client" (lazy loading)
 * Environment: production
   WARNING: Do not use the development server in a production environment.
   Use a production WSGI server instead.
 * Debug mode: off
 * Running on http://127.0.0.1:5100/ (Press CTRL+C to quit)
```

Then open the client page (http://127.0.0.1:5100/ in this case) in your browser.

![preview](docs/_figures/workflow-preview.png)

## Development

We gladly accept [bug reports](https://github.com/DHI/terracotta/issues)
and [pull requests](https://github.com/DHI/terracotta/pulls) via GitHub.
For your code to be useful, make sure that it is covered by tests and that
it satisfies our linting practices (via `mypy` and `flake8`).

To run the tests, just install the necessary dependencies via

```bash
$ pip install -e .[test]
```

Then, you can run

```bash
$ pytest
```

from the root of the repository.


# Qiskit IBM Experiment service

**Qiskit** is an open-source SDK for working with quantum computers at the level of circuits, algorithms, and application modules.

This project contains a service that allows accessing the **[IBM Quantum]**
experiment database.

## Installation
The provider can be installed via pip:

```bash
pip install qiskit-ibm-experiment
```

## Provider Setup

1. Create an IBM Quantum account or log in to your existing account by visiting the [IBM Quantum login page].

2. Ensure you have access to the experiment database.

3. Copy (and/or optionally regenerate) your API token from your
   [IBM Quantum account page].

4. Take your token from step 2, here called `MY_API_TOKEN`, and save it by calling `IBMExperimentService.save_account()`:

   ```python
   from qiskit_ibm_experiment import IBMExperimentService
   IBMExperimentService.save_account(token='MY_API_TOKEN')
   ```

   The command above stores your credentials locally in a configuration file called `qiskit-ibm.json`. By default, this file is located in `$HOME/.qiskit`, where `$HOME` is your home directory.

   Once saved you can then instantiate the experiment service without using the API token:

   ```python
   from qiskit_ibm_experiment import IBMExperimentService
   service = IBMExperimentService()

   # display current supported backends
   print(service.backends())

   # get the latest experiments in the DB
   experiment_list = service.experiments()
   ```

   You can also save specific configuration under a given name:

   ```python
   from qiskit_ibm_experiment import IBMExperimentService
   IBMExperimentService.save_account(name='my_config', token='MY_API_TOKEN')
   ```

   And explicitly load it:
   ```python
   from qiskit_ibm_experiment import IBMExperimentService
   service = IBMExperimentService(name='my_config')

   # display current supported backends
   print(service.backends())

### Load Account from Environment Variables
Alternatively, the IBM Provider can discover credentials from environment variables:
```bash
export QISKIT_IBM_EXPERIMENT_TOKEN='MY_API_TOKEN'
export QISKIT_IBM_EXPERIMENT_URL='https://auth.quantum-computing.ibm.com/api'
```

Then instantiate the provider without any arguments and access the backends:
```python
from qiskit_ibm_experiment import IBMExperimentService
service = IBMExperimentService()
```

Environment variable take precedence over the default account saved to disk via `save_account`,
if one exists; but if the `name` parameter is given, the environment variables are ignored.

### Enable Account for Current Session
As another alternative, you can also enable an account just for the current session by instantiating the service with the token

```python
from qiskit_ibm_experiment import IBMExperimentService
service = IBMExperimentService(token='MY_API_TOKEN')
```

## Contribution Guidelines

If you'd like to contribute to IBM Quantum Experiment Service, please take a look at our
[contribution guidelines]. This project adheres to Qiskit's [code of conduct].
By participating, you are expect to uphold to this code.

We use [GitHub issues] for tracking requests and bugs. Please use our [Slack]
for discussion and simple questions. To join our Slack community, use the
invite link [here](https://docs.quantum.ibm.com/support#qiskit).

## Next Steps

Now you're set up and ready to check out some of the other examples from our
[Qiskit Tutorial] repository.

## Authors and Citation

The Qiskit IBM Quantum Experiment Service is the work of [many people] who contribute to the
project at different levels. If you use Qiskit, please cite as per the included
[BibTeX file].

## License

[Apache License 2.0].

[IBM Quantum]: https://www.ibm.com/quantum-computing/
[IBM Quantum login page]:  https://quantum-computing.ibm.com/login
[IBM Quantum account page]: https://quantum-computing.ibm.com/account
[contribution guidelines]: https://github.com/Qiskit/qiskit-ibm-experiment/blob/main/CONTRIBUTING.md
[code of conduct]: https://github.com/Qiskit/qiskit-ibm-experiment/blob/main/CODE_OF_CONDUCT.md
[GitHub issues]: https://github.com/Qiskit/qiskit-ibm-experiment/issues
[slack]: https://qiskit.slack.com
[Stack Exchange]: https://quantumcomputing.stackexchange.com/questions/tagged/qiskit
[Qiskit Tutorial]: https://github.com/Qiskit/qiskit-tutorial
[many people]: https://github.com/Qiskit/qiskit-ibm-experiment/graphs/contributors
[BibTeX file]: https://github.com/Qiskit/qiskit/blob/master/Qiskit.bib
[Apache License 2.0]: https://github.com/Qiskit/qiskit-ibm-experiment/blob/main/LICENSE.txt


# SNEWPY: Supernova Neutrino Early Warning Models for Python

<img src="https://github.com/SNEWS2/snewpy/blob/v1.4/doc/source/snewpy-logo.png?raw=true" alt="snewpy logo: The word 'snewpy' in a monospace font, with an explosion emoji inside the letter 'p'." style="width: 300px; max-width: 100%;" />

[![DOI](https://zenodo.org/badge/221705586.svg)](https://zenodo.org/badge/latestdoi/221705586)
[![PyPI](https://img.shields.io/pypi/v/snewpy)](https://pypi.org/project/snewpy/)
![tests](https://github.com/SNEWS2/snewpy/actions/workflows/tests.yml/badge.svg)
[![Documentation Status](https://readthedocs.org/projects/snewpy/badge/?version=latest)](https://snewpy.readthedocs.io/en/latest/?badge=latest)

SNEWPY is a Python package for working with supernova neutrinos. It offers …

* … a simple and unified interface to hundreds of supernova simulations.
* … a large library of flavor transformations that relate neutrino fluxes produced in the supernova to those reaching a detector on Earth.
* … and a Python interface to SNOwGLoBES which lets you estimate and plot event rates in many different neutrino detectors.

## Installation

Run `pip install snewpy` to install SNEWPY.

SNEWPY includes a large number of supernova models from different simulation groups. Since these models have a size of several 100 MB, they are not included in the initial install but will be downloaded automatically when needed.
Alternatively, you can run the following command to explicitly download models you want to use to a subdirectory named `SNEWPY-models/<model_name>/` in the current directory:

`python -c 'import snewpy; snewpy.get_models()'`

## Usage and Documentation

SNEWPY gives you easy access to hundreds of included SN simulations …
```Python
import astropy.units as u
from snewpy.models.ccsn import Nakazato_2013, Bollig_2016

# Initialise two SN models. This automatically downloads the required data files if necessary.
nakazato = Nakazato_2013(progenitor_mass=20*u.solMass, revival_time=100*u.ms, metallicity=0.004, eos='shen')
bollig = Bollig_2016(progenitor_mass=27*u.solMass)
```

… and many flavor transformations that neutrinos could experience on the way to Earth …
```Python
from snewpy.flavor_transformations import AdiabaticMSW
from snewpy.neutrino import MassHierarchy

# Adiabatic MSW flavor transformation with normal mass ordering
msw_nmo = AdiabaticMSW(mh=MassHierarchy.NORMAL)
```

… letting you quickly calculate the neutrino flux reaching Earth:
```Python
# Assume a SN at the fiducial distance of 10 kpc and normal mass ordering.
flux = bollig.get_flux(distance=10*u.kpc, transformation=msw_nmo)
```

You can also calculate the observed event rate in all neutrino detectors supported by SNOwGLoBES, use the included SN models and flavor transformations in event generators like sntools, and much more.

Jupyter notebooks showcasing the downloadable supernova models available through SNEWPY and much of its functionality are available in the `doc/nb/` subfolder.
Additional example scripts are in the
`python/snewpy/scripts/` subfolder.

Papers describing SNEWPY and the underlying physics are published in the Astrophysical Journal ([DOI:10.3847/1538-4357/ac350f](https://dx.doi.org/10.3847/1538-4357/ac350f), [arXiv:2109.08188](https://arxiv.org/abs/2109.08188)) and the Journal of Open Source Software ([DOI:10.21105/joss.03772](https://dx.doi.org/10.21105/joss.03772)).

For more, see the [full documentation on Read the Docs](https://snewpy.rtfd.io/).

## Contributing

**Your contributions to SNEWPY are welcome!** For minor changes, simply submit a pull request. If you plan larger changes, it’s probably a good idea to open an issue first to coordinate our work.

We use a [Fork & Pull Request](https://docs.github.com/en/get-started/quickstart/fork-a-repo) workflow, which is common on GitHub.
Please see the [Contributing page](https://snewpy.readthedocs.io/en/stable/contributing.html) in our full documentation for details.