    The official Python client for the Huggingface Hub.

        English |
        Deutsch |
        हिंदी |
        한국어 |
        中文（简体）

---

**Documentation**: https://hf.co/docs/huggingface_hub

**Source Code**: https://github.com/huggingface/huggingface_hub

---

## Welcome to the huggingface_hub library

The `huggingface_hub` library allows you to interact with the [Hugging Face Hub](https://huggingface.co/), a platform democratizing open-source Machine Learning for creators and collaborators. Discover pre-trained models and datasets for your projects or play with the thousands of machine learning apps hosted on the Hub. You can also create and share your own models, datasets and demos with the community. The `huggingface_hub` library provides a simple way to do all these things with Python.

## Key features

- [Download files](https://huggingface.co/docs/huggingface_hub/en/guides/download) from the Hub.
- [Upload files](https://huggingface.co/docs/huggingface_hub/en/guides/upload) to the Hub.
- [Manage your repositories](https://huggingface.co/docs/huggingface_hub/en/guides/repository).
- [Run Inference](https://huggingface.co/docs/huggingface_hub/en/guides/inference) on deployed models.
- [Search](https://huggingface.co/docs/huggingface_hub/en/guides/search) for models, datasets and Spaces.
- [Share Model Cards](https://huggingface.co/docs/huggingface_hub/en/guides/model-cards) to document your models.
- [Engage with the community](https://huggingface.co/docs/huggingface_hub/en/guides/community) through PRs and comments.

## Installation

Install the `huggingface_hub` package with [pip](https://pypi.org/project/huggingface-hub/):

```bash
pip install huggingface_hub
```

If you prefer, you can also install it with [conda](https://huggingface.co/docs/huggingface_hub/en/installation#install-with-conda).

In order to keep the package minimal by default, `huggingface_hub` comes with optional dependencies useful for some use cases. For example, if you want have a complete experience for Inference, run:

```bash
pip install huggingface_hub[inference]
```

To learn more installation and optional dependencies, check out the [installation guide](https://huggingface.co/docs/huggingface_hub/en/installation).

## Quick start

### Download files

Download a single file

```py
from huggingface_hub import hf_hub_download

hf_hub_download(repo_id="tiiuae/falcon-7b-instruct", filename="config.json")
```

Or an entire repository

```py
from huggingface_hub import snapshot_download

snapshot_download("stabilityai/stable-diffusion-2-1")
```

Files will be downloaded in a local cache folder. More details in [this guide](https://huggingface.co/docs/huggingface_hub/en/guides/manage-cache).

### Login

The Hugging Face Hub uses tokens to authenticate applications (see [docs](https://huggingface.co/docs/hub/security-tokens)). To login your machine, run the following CLI:

```bash
huggingface-cli login
# or using an environment variable
huggingface-cli login --token $HUGGINGFACE_TOKEN
```

### Create a repository

```py
from huggingface_hub import create_repo

create_repo(repo_id="super-cool-model")
```

### Upload files

Upload a single file

```py
from huggingface_hub import upload_file

upload_file(
    path_or_fileobj="/home/lysandre/dummy-test/README.md",
    path_in_repo="README.md",
    repo_id="lysandre/test-model",
)
```

Or an entire folder

```py
from huggingface_hub import upload_folder

upload_folder(
    folder_path="/path/to/local/space",
    repo_id="username/my-cool-space",
    repo_type="space",
)
```

For details in the [upload guide](https://huggingface.co/docs/huggingface_hub/en/guides/upload).

## Integrating to the Hub.

We're partnering with cool open source ML libraries to provide free model hosting and versioning. You can find the existing integrations [here](https://huggingface.co/docs/hub/libraries).

The advantages are:

- Free model or dataset hosting for libraries and their users.
- Built-in file versioning, even with very large files, thanks to a git-based approach.
- Serverless inference API for all models publicly available.
- In-browser widgets to play with the uploaded models.
- Anyone can upload a new model for your library, they just need to add the corresponding tag for the model to be discoverable.
- Fast downloads! We use Cloudfront (a CDN) to geo-replicate downloads so they're blazing fast from anywhere on the globe.
- Usage stats and more features to come.

If you would like to integrate your library, feel free to open an issue to begin the discussion. We wrote a [step-by-step guide](https://huggingface.co/docs/hub/adding-a-library) with ❤ showing how to do this integration.

## Contributions (feature requests, bugs, etc.) are super welcome 💙💚💛💜🧡❤

Everyone is welcome to contribute, and we value everybody's contribution. Code is not the only way to help the community.
Answering questions, helping others, reaching out and improving the documentations are immensely valuable to the community.
We wrote a [contribution guide](https://github.com/huggingface/huggingface_hub/blob/main/CONTRIBUTING.md) to summarize
how to get started to contribute to this repository.


# Porcupine

Porcupine is an editor written with the notorious Tkinter library. It supports
most things you would expect from an editor, such as autocompletions and syntax
highlighting.

![Screenshot.](screenshot.png)

Most important features:
- Syntax highlighting (supports many programming languages
  and color themes, extensible)
- Autocompletions when pressing tab
- Jump to definition with Ctrl+click
- [Langserver] support
- [Editorconfig][] support
- Git support
- Compiling files inside the editor window
- Running files in a separate terminal or command prompt window
- Automatic indenting and trailing whitespace stripping when Enter is pressed
- Indent/dedent block with Tab and Shift+Tab
- Commenting/uncommenting multiple lines by selecting them and typing a #
- Highlighting matching parentheses
- Line numbers
- Line length marker
- Find/replace
- Code folding
- Multiple files can be opened at the same time like tabs in a web browser
- The tabs can be dragged out of the window to open a new Porcupine window conveniently

[Pygments]: https://pygments.org/
[Langserver]: https://langserver.org/
[Editorconfig]: https://editorconfig.org/

[Porcupine's design](dev-doc/architecture-and-design.md) makes it very easy to customize.
Almost everything is implemented as a plugin that can be disabled easily,
but if you know how to use Python and tkinter, you can also make your own plugins.
Porcupine plugins are so powerful that if you run Porcupine without any plugins, it looks like this:

![Screenshot of porcupine without plugins 1](dev-doc/images/no-plugins-1.png)

![Screenshot of porcupine without plugins 2](dev-doc/images/no-plugins-2.png)

## Installing Porcupine

Development Install

See [CONTRIBUTING.md](CONTRIBUTING.md) for development instructions.

Debian-based Linux distributions (e.g. Ubuntu, Mint)

Open a terminal and run these commands:

    sudo apt install python3-tk python3-pip python3-venv
    sudo apt install --no-install-recommends tkdnd    # for drop_to_open plugin
    python3 -m venv porcupine-venv
    source porcupine-venv/bin/activate
    pip install wheel
    pip install https://github.com/Akuli/porcupine/archive/v2024.03.31.zip
    porcu

To easily run porcupine again later,
go to *Settings* --> *Porcupine Settings*
and check "Show Porcupine in the desktop menu system".
This makes Porcupine show up in the menu just like any other application.

You can uninstall Porcupine by unchecking "Show Porcupine in the desktop menu system" in the settings
and then deleting `porcupine-venv`.

Other Linux distributions

Install Python 3.9 or newer with pip and tkinter somehow.
If you want drag and drop support, also install tkdnd for the Tcl interpreter that tkinter uses.
Then run these commands:

    python3 -m venv porcupine-venv
    source porcupine-venv/bin/activate
    pip install wheel
    pip install https://github.com/Akuli/porcupine/archive/v2024.03.31.zip
    porcu

To easily run porcupine again later,
go to *Settings* --> *Porcupine Settings*
and check "Show Porcupine in the desktop menu system".
This makes Porcupine show up in the menu just like any other application.

You can uninstall Porcupine by unchecking "Show Porcupine in the desktop menu system" in the settings
and then deleting `porcupine-venv`.

MacOS

I don't have a Mac. If you have a Mac, you can help me a lot by installing
Porcupine and letting me know how well it works.

I think you can download Python with tkinter from
[python.org](https://www.python.org/) and then run the commands for
"other Linux distributions" above.

Windows

Download a Porcupine installer from [the releases page](https://github.com/Akuli/porcupine/releases) and run it.
Because I haven't asked Microsoft to trust Porcupine installers,
you will likely get a warning similar to this one:

![Windows thinks it protected your PC](windows-defender.png)

You should still be able to run the installer by clicking "More info".
When installed, you will find Porcupine from the start menu.

## Documentation

If you have just installed Porcupine, have a look at [user-doc/getting-started.md](user-doc/getting-started.md).
If you want to develop Porcupine, see [CONTRIBUTING.md](CONTRIBUTING.md) or [dev-doc/architecture-and-design.md](dev-doc/architecture-and-design.md).

Most of Porcupine's documentation is markdown files in two folders:
- [The `dev-doc` folder](./dev-doc/) contains the documentation for developing Porcupine.
- [The `user-doc` folder](./user-doc/) contains the documentation for using Porcupine.

## FAQ

### What's new in the latest Porcupine release?

See [CHANGELOG.md](CHANGELOG.md).

### Does Porcupine support programming language X?
You will likely get syntax highlighting without any configuring,
and autocompletions with a few lines of configuration file editing.
See [this documentation](user-doc/new-programming-language.md).

### Help! Porcupine doesn't work.
Please install the latest version.
If it still doesn't work, [let me know by creating an issue on
GitHub](https://github.com/Akuli/porcupine/issues/new).

### Is Porcupine written in Porcupine?

Yes. I wrote the very first version in `nano`, but Porcupine has changed a lot since.

### Why is it named Porcupine?

I think because I didn't find other projects named porcupine, but I don't remember exactly.
Originally, Porcupine was named "Akuli's Editor".

### I want an editor that does X, but X is not in the feature list above. Does Porcupine do X?
You can run Porcupine and find out,
or [create an issue on GitHub](https://github.com/Akuli/porcupine/issues/new) and ask.
If you manage to make me excited about X, I might implement it.

### Why did you create a new editor?
Because I can.

### Why did you create a new editor in tkinter?
Because I can.

### How does feature X work?
See [porcupine/](porcupine/)X.py or [porcupine/plugins/](porcupine/plugins/)X.py.

### Why not use editor X?
Because Porcupine is better.

### Is Porcupine based on IDLE?
Of course not. IDLE is an awful mess that you should stay far away from.

### Is Porcupine a toy project or is it meant to be a serious editor?
Porcupine is meant to be a serious editor, in fact you might regret even touching it.
https://www.youtube.com/watch?v=Y3iUoFkDKjU


# Linked Events
[![codecov](https://codecov.io/gh/City-of-Helsinki/linkedevents/branch/master/graph/badge.svg)](https://codecov.io/gh/City-of-Helsinki/linkedevents)
[![Gitter](https://img.shields.io/gitter/room/City-of-Helsinki/heldev.svg?maxAge=2592000)](https://gitter.im/City-of-Helsinki/heldev)

![High-level diagram of Linked Events](./assets/Linked_Events.jpg?raw=true)

#### TL;DR => Linked Events is a REST API which allows you to set up an _**event**_* publication hub.

\*_**event**_  here means a happening where people get together and do something.

Linked Events is event information:

  * *Aggregator* => using Python importers which have the logic to import events information from different data sources
  * *Creator* => by offering PUT/POST `/event` API endpoint with granular user permissions and a hierarchical organization structure supporting different publishing rights for different organizations
  * *Publisher* => by offering API endpoints from which interested parties can retrieve information about events

Linked Events was originally developed for City of Helsinki organization and
you can see the Linked Events API in action for [Helsinki capital region here](https://api.hel.fi/linkedevents/v1/).
It contains data from all Helsinki City Departments as well as data from Helsinki Marketing and the Helmet metropolitan area public libraries. Viewing the API should give a reasonable view for the kind of information Linked Events is targeted for.

Target audience of this README.md are developers who may or may not have a lot of Python
experience and would like to get things running as quickly as possible.
Therefore, instructions written in this README.md should be written accordingly.

## Contributing

The best way to contribute is to open a new PR for discussion. We strive to be able to support various cities with various use cases, so suggestions and new features (as long as they fit in with existing functionality) are welcome.

## How to setup your local development environment
If all you want is a barebone application to work with for your own city:

* Copy `./docker/django/.env.example` to `./docker/django/.env` and change the variable values to your liking.
* Start django application and database server:
  ```
  docker-compose up
  ```

* Access application on [localhost:8000](http://localhost:8000)

* You are done 🔥

If you wish to use locations, addresses and events data from the Helsinki capital region:

* Read [linked-events-importers.md](./linked-events-importers.md#linked-events-importers-and-commands) and decide the importers or commands you would like to use.

* You can then serve the [UI for Linked Events API](https://github.com/City-of-Helsinki/linkedevents-ui) for example by setting authentication keys to `local_settings.py`

* UI app is specific to Helsinki at the moment and requires general Finnish ontology as well as additional Helsinki specific audiences
and keywords to be present. However, UI code should be easily adaptable to your own city if you have an OAuth2 authentication server present

## Development installation on physical or virtual machine

These instructions assume an $INSTALL_BASE, like so:
```bash
INSTALL_BASE=$HOME/linkedevents
```
If you've already cloned this repository, just move repository root into $INSTALL_BASE/linkedevents. Otherwise just clone the repository, like so:
```bash
git clone https://github.com/City-of-Helsinki/linkedevents.git $INSTALL_BASE/linkedevents
```
Prepare Python 3.x virtualenv using your favorite tools and activate it. Plain virtualenv is like so:
```bash
virtualenv -p python3 $INSTALL_BASE/venv
source $INSTALL_BASE/venv/bin/activate
```
Install required Python packages into the virtualenv
```bash
cd $INSTALL_BASE/linkedevents
pip install -r requirements.txt
```
Create the database, like so: (we have only tested on PostgreSQL)
```bash
cd $INSTALL_BASE/linkedevents
sudo -u postgres createuser -R -S linkedevents
# Following is for US locale, we are not certain whether Linkedevents
# behaves differently depending on DB collation & ctype
#sudo -u postgres createdb -Olinkedevents linkedevents
# This is is for Finnish locale
sudo -u postgres createdb -Olinkedevents -Ttemplate0 -lfi_FI.UTF-8 linkedevents
# Create extensions in the database
sudo -u postgres psql linkedevents -c "CREATE EXTENSION postgis;"
sudo -u postgres psql linkedevents -c "CREATE EXTENSION hstore;"
# This fills the database with a basic skeleton
python manage.py migrate
# This adds language fields based on settings.LANGUAGES (which may be missing in external dependencies)
python manage.py sync_translation_fields
# This creates language objects with correct translations
python manage.py create_languages
```

If you wish to install Linkedevents without any Helsinki specific data (an empty database), and instead customize everything for your own city, you have a working install right now.

The last steps are needed if you wish to use location, address or event data from the Helsinki metropolitan region, or if you wish to run the Helsinki UI (https://linkedevents.hel.fi) from https://github.com/City-of-Helsinki/linkedevents-ui. Currently, the UI is specific to Helsinki and requires the general Finnish ontology as well as additional Helsinki specific audiences and keywords to be present, though its code should be easily adaptable to your own city if you have an OAuth2 authentication server present.

The commands below are documented in more detail in [linked-events-importers.md](./linked-events-importers.md#linked-events-importers-and-commands).

```bash
cd $INSTALL_BASE/linkedevents
# Import general Finnish ontology (used by Helsinki UI and Helsinki events)
python manage.py event_import yso --all
# Add keyword set to display in the UI event audience selection
python manage.py add_helsinki_audience
# Add keyword set to display in the UI main category selection
python manage.py add_helsinki_topics
# Import places from Helsinki metropolitan region service registry (used by events from following sources)
python manage.py event_import tprek --places
# Import places from Helsinki metropolitan region address registry (used as fallback locations)
python manage.py event_import osoite --places
# Import events from Helsinki metropolitan region libraries
python manage.py event_import helmet --events
# Import events from Espoo
python manage.py event_import espoo --events
# Import City of Helsinki hierarchical organization for UI user rights management
python manage.py import_organizations https://dev.hel.fi/paatokset/v1/organization/?limit=1000 -c openahjo -s OpenAhjoAPI:ahjo
# Import municipalities in Finland
python manage.py geo_import finland --municipalities
# Import districts in Helsinki
python manage.py geo_import helsinki --divisions
# install API frontend templates:
python manage.py install_templates helevents
```

The last command installs the `helevents/templates/rest_framework/api.html` template,
which contains Helsinki event data summary and license. You may customize the template
for your favorite city by creating `your_favorite_city/templates/rest_framework/api.html`.
For further erudition, take a look at the DRF documentation on [customizing the browsable API](http://www.django-rest-framework.org/topics/browsable-api/#customizing)

After this, everything but search endpoint (/search) is working. See [search](#search)

## Development in macOS (linux/arm64)
* Note: Processor architecture on Apple-silicon machines is ARM64, not AMD64.
  If you have one of these extra tinkering is needed.
  On Intel-silicon machines, this is not necessary.
* Note 2: As AMD64 is emulated on top of ARM64 Linux, this isn't fast.
1. Pre-build linux/AMD64-versions
   1. PostgreSQL-container:
       ```bash
       podman build --platform=linux/amd64 \
         -f docker/postgres/Dockerfile  \
         .
       ```
   1. Django-container:
       ```bash
       podman build --platform=linux/amd64 \
         -f docker/postgres/Dockerfile  \
         .
       ```
2. Now `podman-compose` will find pre-built alternate architecture container images and will run ok:
    ```bash
    podman-compose up
    ```
3. Done, test @ http://127.0.0.1:8000/

### GIS
* Problem on macOS:
    ```text
    django.contrib.gis.geos.error.GEOSException: Error encountered checking Geometry returned from GEOS C function "GEOSGeom_createCollection_r".
    ```
  * See: https://github.com/django/django/pull/15214
* Fix:
  * Edit `linkedevents/settings.py`, add to end:
    ```text
    # macOS:
    GDAL_LIBRARY_PATH="/opt/homebrew/opt/gdal/lib/libgdal.dylib"
    GEOS_LIBRARY_PATH="/opt/homebrew/opt/geos/lib/libgeos_c.dylib"
    ```

### Troubleshooting
* Problem:
  _Error: no container with name or ID "linkedevents-backend" found: no such container_
  * Solution:
    You missed the part with: Copy `./docker/django/.env.example` to `./docker/django/.env`

* Problem:
  TCP-ports not available in host
  * Solution:
    If port forwarding from VM is flaky, native ports are visible.
    Django should be visible in http://127.0.0.1:8080/

## Production notes

Development installation above will give you quite a serviceable production installation for lightish usage. You can serve out the application using your favorite WSGI-capable application server. The WSGI-entrypoint for Linked Events is ```linkedevents.wsgi``` or in file ```linkedevents/wsgi.py```. Former is used by gunicorn, latter by uwsgi. The callable is ```application```.

You will also need to serve out ```static``` and ```media``` folders at ```/static``` and ```/media``` in your URL space.

## Running tests

Tests must be run using an user who can create (and drop) databases and write the directories
your linkedevents installation resides in. Also the template database must include Postgis and
HSTORE-extensions. If you are developing, you probably want to give those
permissions to the database user configured in your development instance. Like so:

```bash
# Change this if you have different DB user
DATABASE_USER=linkedevents
# Most likely you have a postgres system user that can log into postgres as DB postgres user
sudo -u postgres psql << EOF
ALTER USER "$DATABASE_USER" CREATEDB;
\c template1
CREATE EXTENSION IF NOT EXISTS postgis;
CREATE EXTENSION IF NOT EXISTS hstore;
CREATE EXTENSION IF NOT EXISTS pg_trgm;
EOF

```

Afterwards you can run the tests:
```bash
cd $INSTALL_BASE/linkedevents
py.test events
```

Note that search tests will fail unless you configure [search](#search)

## Sending email
The project uses [django-mailer](https://github.com/pinax/django-mailer) for queuing and sending email. When a normal Django email function like `send_mail` is called, the email will be stored into a queue that exists in the database for later sending.

`django-mailer` comes with a number of management commands for interacting with the email queue. Out of those, the `send_mail` command is for us perhaps the most notable one as it is responsible for the actual sending of the queued messages.

To send messages locally, you can run the `send_mail` command:
`./manage.py send_mail`

For more information about `django-mailer` and the management commands, you can refer to the [usage documentation](https://github.com/pinax/django-mailer/blob/master/docs/usage.rst).

## Requirements

Linked Events uses two files for requirements. The workflow is as follows.

`requirements.txt` is not edited manually, but is generated
with `pip-compile`.

`requirements.txt` always contains fully tested, pinned versions
of the requirements. `requirements.in` contains the primary, unpinned
requirements of the project without their dependencies.

In production, deployments should always use `requirements.txt`
and the versions pinned therein. In development, new virtualenvs
and development environments should also be initialised using
`requirements.txt`. `pip-sync` will synchronize the active
virtualenv to match exactly the packages in `requirements.txt`.

In development and testing, to update to the latest versions
of requirements, use the command `pip-compile`. You can
use [requires.io](https://requires.io) to monitor the
pinned versions for updates.

To remove a dependency, remove it from `requirements.in`,
run `pip-compile` and then `pip-sync`. If everything works
as expected, commit the changes.

## Code format

This project uses
[`black`](https://github.com/ambv/black),
[`flake8`](https://github.com/pycqa/flake8) and
[`isort`](https://github.com/pycqa/isort)
for code formatting and quality checking. Project follows the basic
black config, without any modifications.

Basic `black` commands:

* To let `black` do its magic: `black .`
* To see which files `black` would change: `black --check .`

[`pre-commit`](https://pre-commit.com/) can be used to install and
run all the formatting tools as git hooks automatically before a
commit.

## Git blame ignore refs

Project includes a `.git-blame-ignore-revs` file for ignoring certain commits from `git blame`.
This can be useful for ignoring e.g. formatting commits, so that it is more clear from `git blame`
where the actual code change came from. Configure your git to use it for this project with the
following command:

```shell
git config blame.ignoreRevsFile .git-blame-ignore-revs
```

## Commit message format

New commit messages must adhere to the [Conventional Commits](https://www.conventionalcommits.org/)
specification, and line length is limited to 72 characters.

When [`pre-commit`](https://pre-commit.com/) is in use, [`commitlint`](https://github.com/conventional-changelog/commitlint)
checks new commit messages for the correct format.

## Search

Linkedevents uses Elasticsearch for generating results on the /search-endpoint. If you wish to use that functionality, proceed like so:

1. Install elasticsearch

    We've only tested using the rather ancient 1.7 version. If you are using Ubuntu 16.04, 1.7 will be available in the official repository.
    This limitation was originally due to django-haystack not supporting versions above 1.
    As of writing this the django-haystack version in use does support versions 1, 2, 5 and 7.

2. (For Finnish support) Install elasticsearch-analyzer-voikko, libvoikko and needed dictionaries

    `/usr/share/elasticsearch/bin/plugin -i fi.evident.elasticsearch/elasticsearch-analysis-voikko/0.4.0`
    This specific command is for Debian derivatives. The path to `plugin` command might be different on yours. Note that version 0.4.0 is the one compatible with Elasticsearch 1.7

    Installing libvoikko:
    `apt-get install libvoikko1`

    Installing the dictionaries (v5 dictionaries are needed for libvoikko version included in Ubuntu 16.04):

    ```
    wget -P $INSTALL_BASE http://www.puimula.org/htp/testing/voikko-snapshot-v5/dict-morpho.zip
    unzip $INSTALL_BASE/dict-morpho.zip -d /etc/voikko
    ```

3. Configure the thing

    Set the `ELASTICSEARCH_URL` environment variable (or variable in `config_dev.env`, if you are running in development mode) to your elasticsearch instance. The default value is `http://localhost:9200/`.

    Haystack configuration for all Linkedevents languages happens automatically if `ELASTICSEARCH_URL` is set, but you may customize it manually using `local_settings.py` if you know Haystack and wish to do so.

4. Rebuild the search indexes

   `python manage.py rebuild_index`

   You should now have a working /search endpoint, give or take a few.

## Event extensions

It is possible to extend event data and API without touching `events` application by implementing separate extension applications. These extensions will be wired under field `extension_<extension idenfier>` in event API. If not auto enabled (see 6. below), extensions can be enabled per request using query param `extensions` with comma separated identifiers as values, or `all` for enabling all the extensions.

To implement an extension:

1) Create a new Django application, preferably named `extension_<unique identifier for the extension>`.

2) If you need to add new data for events, implement that using model(s) in the extension application.

3) Inherit `events.extensions.EventExtension` and implement needed attributes and methods. See [extensions.py](events/extensions.py) for details.

4) Add `event_extension: <your EventExtension subclass>` attribute to the extension applications's `AppConfig`.

5) Make the extension available by adding the extension application to `INSTALLED_APPS`.

6) If you want to force the extension to be enabled on every request, add the extension's identifier to `AUTO_ENABLED_EXTENSIONS` in Django settings.

For an example extension implementation, see [course extension](extension_course).


# sesdev - deploy and manage SES/Ceph clusters<br/>

`sesdev` is a CLI tool to deploy Ceph clusters (both the upstream and SUSE
downstream versions).

This tool uses [Vagrant](https://www.vagrantup.com/) behind the scenes to create
the VMs and run the deployment scripts.

## Build Status

## Table of Contents

[//]: # (To generate a new TOC, first install https://github.com/ekalinin/github-markdown-toc)
[//]: # (and then run "gh-md-toc README.md")
[//]: # (the new TOC will appear on stdout: the expectation is that the maintainer will do the rest.)

* [Installation](#installation)
   * [Install sesdev on openSUSE or SUSE Linux Enterprise](#install-sesdev-on-opensuse-or-suse-linux-enterprise)
      * [Install KVM/QEMU and Libvirt](#install-kvmqemu-and-libvirt)
      * [Add user to libvirt group](#add-user-to-libvirt-group)
      * [Install Vagrant](#install-vagrant)
         * [Install Vagrant on openSUSE Leap 15.2, SLE-15-SP2 or Tumbleweed](#install-vagrant-on-opensuse-leap-152-sle-15-sp2-or-tumbleweed)
         * [Install Vagrant on openSUSE Leap 15.1, SLE-15-SP1](#install-vagrant-on-opensuse-leap-151-sle-15-sp1)
         * [Vagrant RPM from Hashicorp website](#vagrant-rpm-from-hashicorp-website)
      * [Install sesdev from source](#install-sesdev-from-source)
   * [Install sesdev on Fedora Linux](#install-sesdev-on-fedora-linux)
      * [Install KVM/QEMU and Libvirt](#install-kvmqemu-and-libvirt-1)
      * [Install sesdev from source](#install-sesdev-from-source-1)
   * [Install sesdev on Debian/Ubuntu](#install-sesdev-on-debianubuntu)
   * [Install sesdev from source](#install-sesdev-from-source-2)
      * [Running the unit tests](#running-the-unit-tests)
      * [Install shell autocompletion](#install-shell-autocompletion)
* [Usage](#usage)
   * [Create/deploy a Ceph cluster](#createdeploy-a-ceph-cluster)
      * [Bare bone cluster](#bare-bone-cluster)
      * [CaaSP (with or without Rook/Ceph/SES)](#caasp-with-or-without-rookcephses)
         * [CaaSP k8s cluster](#caasp-k8s-cluster)
         * [CaaSP with Rook/Ceph/SES](#caasp-with-rookcephses)
         * [CaaSP on just one node](#caasp-on-just-one-node)
      * [k3s (with or without Rook/Ceph/SES)](#k3s-with-or-without-rookcephses)
         * [k3s cluster](#k3s-cluster)
         * [k3s with Rook/Ceph/SES](#k3s-with-rookcephses)
         * [k3s with Longhorn](#k3s-with-longhorn)
      * [On a remote libvirt server via SSH](#on-a-remote-libvirt-server-via-ssh)
      * [Using salt instead of DeepSea/ceph-salt CLI](#using-salt-instead-of-deepseaceph-salt-cli)
      * [With a FQDN environment](#with-a-fqdn-environment)
      * [Without the devel repo](#without-the-devel-repo)
      * [With an additional custom zypper repo](#with-an-additional-custom-zypper-repo)
      * [With a set of custom zypper repos completely replacing the default repos](#with-a-set-of-custom-zypper-repos-completely-replacing-the-default-repos)
      * [With custom image paths](#with-custom-image-paths)
      * [With custom default roles](#with-custom-default-roles)
      * [config.yaml examples](#configyaml-examples)
         * [octopus from filesystems:ceph:octopus](#octopus-from-filesystemscephoctopus)
         * [octopus from filesystems:ceph:octopus:upstream](#octopus-from-filesystemscephoctopusupstream)
         * [ses7 from Devel:Storage:7.0](#ses7-from-develstorage70)
         * [ses7 from Devel:Storage:7.0:Pacific](#ses7p-from-develstorage70pacific)
         * [ses7 from Devel:Storage:7.0:CR](#ses7-from-develstorage70cr)
      * [With wire encryption](#with-wire-encryption)
      * [Deploying non-SUSE environments](#deploying-non-suse-environments)
         * [Ubuntu "Bionic Beaver" 18.04](#ubuntu-bionic-beaver-1804)
         * [Ubuntu "Focal Fossa" 20.04](#ubuntu-focal-fossa-2004)
      * [Introspect existing deployments](#introspect-existing-deployments)
         * [List all existing deployments and their overall status](#list-all-existing-deployments-and-their-overall-status)
         * [Get status of individual nodes in an existing deployment](#get-status-of-individual-nodes-in-an-existing-deployment)
         * [Show details of an existing deployment](#show-details-of-an-existing-deployment)
         * [Show roles of nodes in an existing deployment](#show-roles-of-nodes-in-an-existing-deployment)
   * [List existing deployments](#list-existing-deployments)
   * [SSH access to a cluster](#ssh-access-to-a-cluster)
   * [Copy files into and out of a cluster](#copy-files-into-and-out-of-a-cluster)
   * [Services port-forwarding](#services-port-forwarding)
   * [Replace ceph-salt](#replace-ceph-salt)
   * [Replace MGR modules](#replacing-mgr-modules)
   * [Add a repo to a cluster](#add-a-repo-to-a-cluster)
   * [Link two clusters together](#link-two-clusters-together)
   * [Temporarily stop a cluster](#temporarily-stop-a-cluster)
   * [Destroy a cluster](#destroy-a-cluster)
   * [Run "make check"](#run-make-check)
      * [Run "make check" on Tumbleweed from upstream "master" branch](#run-make-check-on-tumbleweed-from-upstream-master-branch)
      * [Run "make check" on openSUSE Leap 15.2 from upstream "octopus" branch](#run-make-check-on-opensuse-leap-152-from-upstream-octopus-branch)
      * [Run "make check" on SLE-15-SP2 from downstream "ses7" branch](#run-make-check-on-sle-15-sp2-from-downstream-ses7-branch)
      * [Other "make check" scenarios](#other-make-check-scenarios)
   * [Custom provisioning](#custom-provisioning)
* [Common pitfalls](#common-pitfalls)
   * [Domain about to create is already taken](#domain-about-to-create-is-already-taken)
   * [Storage pool not found: no storage pool with matching name 'default'](#storage-pool-not-found-no-storage-pool-with-matching-name-default)
   * [When sesdev deployments get destroyed, virtual networks get left behind](#when-sesdev-deployments-get-destroyed-virtual-networks-get-left-behind)
   * [sesdev destroy reported an error](#sesdev-destroy-reported-an-error)
   * ["Failed to connect socket" error when attempting to use remote libvirt server](#failed-to-connect-socket-error-when-attempting-to-use-remote-libvirt-server)
   * [mount.nfs: Unknown error 521](#mountnfs-unknown-error-521)
   * [Problems accessing dashboard on remote sesdev](#problems-accessing-dashboard-on-remote-sesdev)
   * [Error creating IPv6 cluster](#error-creating-ipv6-cluster)
   * [Failed to initialize libnetcontrol](#failed-to-initialize-libnetcontrol)
* [Contributing](#contributing)

## Installation

First, you should have both [QEMU](https://www.qemu.org/) and
[Libvirt](https://libvirt.org/) installed in some machine to host the VMs
created by sesdev (using Vagrant behind the scenes).

### Install sesdev on openSUSE or SUSE Linux Enterprise

#### Install KVM/QEMU and Libvirt

First, if on SUSE Linux Enterprise, make sure you have the Server Applications
Module available on the system. (Links to internal repos are available
[here](https://confluence.suse.com/display/applications/Vagrant+and+Vagrant+Boxes+for+openSUSE+and+SLE#VagrantandVagrantBoxesforopenSUSEandSLE-openSUSELeap15.2andSLE15SP2).)

Run the following commands as root:

```
# zypper -n install -t pattern kvm_server kvm_tools
# systemctl enable libvirtd
# systemctl restart libvirtd
```

#### Add user to libvirt group

If you are running libvirt on the same machine where you installed sesdev, add
your user to the "libvirt" group to avoid "no polkit agent available" errors
when vagrant attempts to connect to the libvirt daemon:

```
# groupadd libvirt
groupadd: group 'libvirt' already exists
# usermod -a -G libvirt $USER
```

Log out, and then log back in. You should now be a member of the "libvirt"
group.

#### Install Vagrant

sesdev needs Vagrant to work. Vagrant can be installed in a number of ways,
depending on your environment:

##### Install Vagrant on openSUSE Leap 15.2, SLE-15-SP2 or Tumbleweed

On very new OSes like these, Vagrant is included in the operating system's base
repos. Just install the ``vagrant`` and ``vagrant-libvirt`` packages.

For SLE-15-SP2, the packages are available via the
[SUSE Package Hub](https://packagehub.suse.com/). (Links to internal repos are available
[here](https://confluence.suse.com/display/applications/Vagrant+and+Vagrant+Boxes+for+openSUSE+and+SLE#VagrantandVagrantBoxesforopenSUSEandSLE-openSUSELeap15.2andSLE15SP2).)

##### Install Vagrant on openSUSE Leap 15.1, SLE-15-SP1

To install Vagrant on these systems, run the following commands as root:

```
# zypper ar https://download.opensuse.org/repositories/Virtualization:/vagrant/<repo> vagrant_repo
# zypper ref
# zypper -n install vagrant vagrant-libvirt
```

Where `<repo>` can be any of the openSUSE build targets currently enabled for
the [Virtualization:vagrant/vagrant package in the openSUSE Build Service](https://build.opensuse.org/package/show/Virtualization:vagrant/vagrant).

Be aware that ``Virtualization:vagrant`` is a development project where updates
to the latest official openSUSE vagrant packages are prepared. That means the
vagrant packages in this repo will tend to be new and, sometimes, even broken.
In that case, read on to the next section.

##### Vagrant RPM from Hashicorp website

If you find that, for whatever reason, you cannot get a working vagrant package
from OBS, it is possible to install vagrant from the official RPMs published on
[the Hashicorp website](https://www.vagrantup.com/downloads.html).

To install vagrant and its libvirt plugin from Hashicorp, the following
procedure has been known to work with vagrant 2.4.0 (run the commands as root):

1. download vagrant RPM from https://releases.hashicorp.com/vagrant/
2. install gcc, make and libvirt-devel (``zypper install gcc make libvirt-devel``)
3. install vagrant (``rpm -i <the RPM you just downloaded>``)
4. delete file that causes libvirt plugin compilation to fail
   (``rm /opt/vagrant/embedded/lib/libreadline.so.8``)

Finally, run the following command as the user you run sesdev with:

    vagrant plugin install vagrant-libvirt

#### Install sesdev from source

Proceed to [Install sesdev from source](#install-sesdev-from-source-2),
then refer to the [Usage](#usage) chapter, below, for further information.

### Install sesdev on Fedora Linux

#### Install KVM/QEMU and Libvirt

Run the following commands as root:

```
# dnf install qemu-common qemu-kvm libvirt-daemon-kvm \
libvirt-daemon libvirt-daemon-driver-qemu vagrant-libvirt
# systemctl enable libvirtd
# systemctl restart libvirtd
```

#### Install sesdev from source

Proceed to [Install sesdev from source](#install-sesdev-from-source-2),
then refer to the [Usage](#usage) chapter, below, for further information.

### Install sesdev on Debian/Ubuntu

sesdev is known to work on recent Ubuntu versions.  Follow the instructions given in
[Install sesdev from source](#install-sesdev-from-source-2).

### Install sesdev from source

sesdev uses the libvirt API Python bindings, and these cannot be installed via
pip unless the RPM packages "gcc", "python3-devel", and "libvirt-devel" are
installed, first. Also, in order to clone the sesdev git repo, the "git-core"
package is needed. So, before proceeding, make sure that all of these packages
are installed in the system:

#### openSUSE

```
# zypper -n install gcc git-core libvirt-devel python3-devel python3-virtualenv
```

#### Debian / Ubuntu

```
# apt-get install -y git gcc libvirt-dev \
virtualenv python3-dev python3-venv python3-virtualenv
```

#### Fedora

```
# dnf install -y git-core gcc libvirt-devel \
python3-devel python3-virtualenv
```

Now you can proceed to clone the sesdev source code repo and bootstrap it:

```
$ git clone https://github.com/SUSE/sesdev.git
$ cd sesdev
$ ./bootstrap.sh
```

Before you can use `sesdev`, you need to activate the Python virtual environment
created by the `bootstrap.sh` script. The script tells you how to do this, but
we'll give the command here, anyway:

```
source venv/bin/activate
```

At this point, sesdev should be installed and ready to use: refer to the
[Usage](#usage) chapter, below, for further information.

To leave the virtual environment, simply run:

```
deactivate
```

CAVEAT: Remember to re-run `./bootstrap.sh` after each git pull.

#### Running the unit tests

If you are preparing a code change for submission and would like to run the
unit tests on it, make sure you have installed sesdev from source, as described
above, and the virtualenv is active. Then, follow the instructions below.

First, make sure you have installed sesdev from source following the
instructions from [here](#install-sesdev-from-source).

Second, make sure your virtualenv is active (`source venv/bin/activate`).

At this point, run `tox --version` to check if tox is already installed on your
system. If it is not, then run `pip3 install tox` to install it in the Python
virtual environment.

Finally, inspect the list of testing environments in `tox.ini` and choose one or
more that you are interested in. Here is an example, but the actual output might
be different:

```
$ tox --listenvs
py3
lint
```

(This means you have two testing environments to choose from: `py3` and `lint`.)

Finally, run your chosen test environment(s):

```
tox -e py3
tox -e lint
```

If you don't know which testing environment to choose, the command `tox` will
run _all_ the testing environments.

#### Install Shell Autocompletion

First, generate the autocompletion code for the shell of your choice. This
example assumes the bash shell, but zsh and fish are supported too and work
analogous:

```
sesdev shell-completion bash > ~/.sesdev-completion.sh
```

Then source it in your shell's rc file, for bash that is `~/.bashrc`:

```
source ~/.sesdev-completion.sh
```

## Usage

Run `sesdev --help` or `sesdev <command> --help` to get the available
options and description of the commands.

### Create/deploy a Ceph cluster

To create a single node Ceph cluster based on nautilus/leap-15.1 on your local
system, run the following command:

```
$ sesdev create nautilus --single-node mini
```

The `mini` argument is the ID of the deployment. It is optional: if you omit it,
sesdev will assign an ID as it sees fit. You can create many deployments by
giving them different IDs.

To create a multi-node Ceph cluster, you can specify the nodes and their roles
using the `--roles` option.

The roles of each node are grouped in square brackets, separated by commas. The
nodes are separated by commas, too.

The following roles can be assigned:

* `master` - The master node, running management components like the Salt master
* `admin` - signifying that the node should get ceph.conf and keyring [1]
* `bootstrap` - The node where `cephadm bootstrap` will be run
* `client` - Various Ceph client utilities
* `nfs` - NFS (Ganesha) gateway [2] [4]
* `grafana` - Grafana metrics visualization (requires Prometheus) [3]
* `igw` - iSCSI target gateway
* `mds` - CephFS MDS
* `mgr` - Ceph Manager instance
* `mon` - Ceph Monitor instance
* `prometheus` - Prometheus monitoring [3]
* `rgw` - Ceph Object Gateway
* `storage` - OSD storage daemon [3]
* `suma` - SUSE Manager (octopus only)

[1] CAVEAT: sesdev applies the `admin` role to all nodes, regardless of whether
or not the user specified it explicitly on the command line or in `config.yaml`.

[2] The `nfs` role may also be used -- by itself on a dedicated VM -- when
deploying a CaaSP cluster. See [Rook and CaaSP based Ceph
cluster](#rook-and-caasp-based-ceph-cluster) for more information.

[3] Do not use the `storage` role when deploying Rook/Ceph over CaaSP. See
[Rook and CaaSP based Ceph cluster](#rook-and-caasp-based-ceph-cluster) for more
information.

[4] Not currently supported by the `octopus`, `pacific`, or `master` roles.

The following example will generate a cluster with four nodes: the master (Salt
Master) node that is also running a MON daemon; a storage (OSD) node that
will also run a MON, a MGR and an MDS and serve as the bootstrap node; another
storage (OSD) node with MON, MGR, and MDS; and a fourth node that will run an iSCSI
gateway, an NFS (Ganesha) gateway, and an RGW gateway.

```
$ sesdev create nautilus --roles="[master, mon], [bootstrap, storage, mon, mgr, mds], \
  [storage, mon, mgr, mds], [igw, nfs, rgw]"
```

#### Bare bone cluster

An important use case of sesdev is to create "bare bone" clusters: i.e.,
clusters with almost nothing running on them, but ready for manual testing of
deployment procedures, or just playing around.

Some caveats apply:

1. These caveats apply only to core (Ceph) deployment versions. Rook/CaaSP is
   different: see [Rook and CaaSP based Ceph cluster](#rook-and-caasp-based-ceph-cluster)
   for details.
2. For `nautilus`, and `ses6`, the only role required is `master` and
   you can use `--stop-before-deepsea-stage` to control how many DeepSea stages
   are run.
3. For `octopus`, `ses7`, `ses7p`, and `pacific`, the only roles required are
   `master` and `bootstrap`. While it is possible to stop the deployment script
   at various stages (see `sesdev create octopus --help` for details), in
   general sesdev will try to deploy Ceph services/daemons according to the
   roles given by the user.
4. You can specify a node with no roles like so: `[]`
5. Ordinarily, a node gets extra disks ("OSD disks") only when the `storage`
   role is specified. However, to facilitate deployment of "bare bone" clusters,
   sesdev will also create and attach disks if the user explicitly gives the
   `--num-disks` option.
6. Disks will not be created/attached to nodes that have only the `master`
   role and no other roles.

Example:

```
sesdev create octopus --roles="[master],[mon,mgr,bootstrap],[],[]" --num-disks 3
```

This will bootstrap an octopus cluster with:

1. an "admin node" (`[master]`)
2. a bootstrap node (`[mon,mgr,bootstrap]`)
3. two empty nodes (`[]`) ready for "Day 2" operations

#### CaaSP (with or without Rook/Ceph/SES)

##### CaaSP k8s cluster

To create CaaSP k8s cluster that has a `loadbalancer` node, 2 `worker` nodes and
a `master` node:

```
$ sesdev create caasp4
```

By default it just creates and configures a CaaSP cluster, and workers don't
have any disks unless the `--deploy-ses` (see below) or `--num-disks` options
are given.

To create workers with disks and without a `loadbalancer` role:

```
$ sesdev create caasp4 --roles="[master], [worker], [worker]" --disk-size 6 --num-disks 2
```

Note: sesdev does not support sharing of roles on a single `caasp4` node. Each
node must have one and only one role. However, it is still possible to deploy
a single-node cluster (see below). In this case the master node will also
function as a worker node even though the `worker` role is not explicitly given.

For persistent storage, there are two options: either deploy SES with Rook (see
below), or specify an `nfs` role -- always by itself on a dedicated node. In the
latter case, sesdev will create a node acting as an NFS server as well as an NFS
client pod in the CaaSP cluster, providing a persistent store for other
(containerized) applications.

##### CaaSP with Rook/Ceph/SES

To have sesdev deploy Rook on the CaaSP cluster, give the `--deploy-ses` option.
The default disk size is 8G, number of worker nodes 2, number of disks per
worker node 3:

```
$ sesdev create caasp4 --deploy-ses
```

Note: sesdev does not support sharing of roles on a single `caasp4` node. Each
node must have one and only one role. However, it is still possible to deploy
a single-node cluster (see below). In this case the master node will also
function as a worker node even though the `worker` role is not explicitly given.

Note: the `storage` role should never be given in a `caasp4` cluster. By
default, Rook will will look for any spare block devices on worker nodes (i.e.,
all block devices but the first (OS disk) of any given worker) and create OSD
pods for them. Just be aware that sesdev will not create these "spare block
devices" unless you explicitly pass either the `--num-disks` or the
`--deploy-ses` option (or both).

##### CaaSP on just one node

To create a single-node CaaSP cluster, use `--single-node` option. This may be
given in combination with `--deploy-ses`, or by itself. For example, the
following command creates a CaaSP cluster on one node with four disks (8G) and
also deploy SES/Ceph on it, using Rook:

```
$ sesdev create caasp4 --single-node --deploy-ses
```

Note: since passing `--single-node` without an explicit deployment name causes
the name to be set to `DEPLOYMENT_VERSION-mini`, the resulting cluster from the
example above would be called `caasp4-mini`.

#### k3s (with or without Rook/Ceph/SES)

##### k3s cluster

To create a k3s cluster that has 4 `worker` nodes and
a `master` node:

```
$ sesdev create k3s
```

This uses `curl -sfL https://get.k3s.io | -` to install k3s,
and `curl -fsSL https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-3 | bash` to install helm.

By default it just creates and configures a k3s cluster, and workers don't
have any disks unless the `--deploy-ses`, `--deploy-longhorn` (see below)
or `--num-disks` options are given.

##### k3s with Rook/Ceph/SES

To have sesdev deploy Rook on the k3s cluster, give the `--deploy-ses` option.
The default disk size is 8G, number of worker nodes 4, number of disks per
worker node 3:

```
$ sesdev create k3s --deploy-ses
```

##### k3s with Longhorn

To have sesdev deploy [Longhorn](https://longhorn.io) instead of Ceph, give
the `--deploy-longhorn` option.  By default this will deploy 4 worker nodes
each with one additional 8G disk, mounted at /var/lib/longhorn, and will
install the latest stable version of Longhorn:

```
$ sesdev create k3s --deploy-longhorn
```

To deploy a specific version of Longorn, use the `--longhorn-version` option:

```
$ sesdev create k3s --deploy-longhorn --longhorn-version=1.4.1
```

Currently Longhorn deployments will only use _one_ disk.  If more are
specified using the `--num-disks` option, only the first disk will be
mounted for use by Longhorn.  All other additional disks will remain
untouched.

#### On a remote libvirt server via SSH

If you would like to start the cluster VMs on a remote server via libvirt/SSH,
create a configuration file `$HOME/.sesdev/config.yaml` with the following
content:

```
libvirt_use_ssh: true
libvirt_user: <ssh_user>
libvirt_private_key_file: <private_key_file>   # defaults to $HOME/.ssh/id_rsa
libvirt_host: <hostname|ip address>
```

Note that passwordless SSH access to this user@host combination needs to be
configured and enabled.

#### Using salt instead of DeepSea/ceph-salt CLI

By default, sesdev will use the DeepSea CLI to run the DeepSea Stages (nautilus,
ses6) or the "ceph-salt" command to apply the ceph-salt Salt Formula
(ses7, octopus, pacific).

If you would rather use Salt directly, give the `--salt` option on the `sesdev
create` command line.

#### With a FQDN environment

In some cases you might want to deploy a Ceph cluster in an environment where

    hostname

returns an FQDN and

    hostname -s

returns the short hostname (defined as a string containing no `.` characters).
DeepSea and ceph-salt should have no problem with this. You can tell sesdev
to set the hostname to the FQDN by passing the `--fqdn` option to `sesdev
create`.

#### Without the devel repo

The "core" deployment targets (nautilus, ses6, octopus, ses7, ses7p,
pacific) all have a concept of a "devel" repo where updates to the
Ceph/storage-related packages are staged. Since users frequently want to install
the "latest, greatest" packages, the "devel" repo is added to all nodes by
default. However, there are times when this is not desired: when using sesdev to
simulate update/upgrade scenarios, for example.

To deploy a Ceph cluster without the "devel" repo, give the `--product` option
on the `sesdev create` command line.

#### With an additional custom zypper repo

Each deployment version (e.g. "octopus", "nautilus") is associated with
a set of zypper repos which are added on each VM that is created.

There are times when you may need to add additional zypper repo(s)
to all the VMs prior to deployment. In such a case, add one or more `--repo`
options to the command line, e.g.:

```
$ sesdev create nautilus --single-node --repo [URL_OF_REPO]
```

By default, the custom repo(s) will be added with an elevated priority,
to ensure that packages from these repos will be installed even if higher
RPM versions of those packages exist. If this behavior is not desired,
add `--no-repo-priority` to disable it.

#### With a set of custom zypper repos completely replacing the default repos

If the default zypper repos that are added to each VM prior to deployment are
completely wrong for your use case, you can override them via
`~/.sesdev/config.yaml`.

To do this, you have to be familiar with two of sesdev's internal dictionaries:
`OS_REPOS` and `VERSION_DEVEL_REPOS`. The former specifies repos that are
added to all VMs with a given operating system, regardless of the Ceph version
being deployed, and the latter specifies additional repos that are added to VMs
depending on the Ceph version being deployed. Refer to `seslib/__init__.py` for
the current defaults.

To override `OS_REPOS`, add an `os_repos:` stanza to your `~/.sesdev/config.yaml`.

To override `VERSION_DEVEL_REPOS`, add a `version_devel_repos:` stanza to your `~/.sesdev/config.yaml`.

Please note that you need not copy-paste any parts of these internal
dictionaries from the source code into your config. You can selectively override
only those parts that you need. For example, the following config snippet will
override the default additional repos for "octopus" deployments on "leap-15.2",
but it will not change the defaults for any of the other deployment versions:

```
version_devel_repos:
    octopus:
        leap-15.2:
            - 'https://download.opensuse.org/repositories/filesystems:/ceph:/octopus/openSUSE_Leap_15.2'
```

If you need a higher priority on one or more of the repos,
`version_devel_repos` supports a "magic priority prefix" on the repo URL,
like so:

```
version_devel_repos:
    octopus:
        leap-15.2:
            - '96!https://download.opensuse.org/repositories/filesystems:/ceph:/octopus/openSUSE_Leap_15.2'
```

This would cause the zypper repo to be added at priority 96.

#### With custom image paths

In Ceph versions "octopus" and newer, the Ceph daemons run inside containers.
When the cluster is bootstrapped, a container image is downloaded from a remote
registry. The default image paths are set by the internal dictionaries
`IMAGE_PATHS_DEVEL` and `IMAGE_PATHS_PRODUCT`. You can specify a different
image path using the `--image-path` option to e.g., `sesdev create octopus`.

If you would like to permanently specify a different image path for one or more
Ceph versions, you can override the defaults by adding a stanza like the
following to your `~/.sesdev/config.yaml`:

```
image_paths_devel:
    octopus:
        ceph: 'registry.opensuse.org/filesystems/ceph/octopus/images/ceph/ceph'
```

#### With custom registry

In case there is a need to use some container registry mirror, it is possible
to override registry location, and disable ssl if required. For example,
similar record can be add to the `~/.sesdev/config.yaml`.

```
container_registry:
    prefix: 'registry.suse.de'
    location: '1.2.3.4:5000'
    insecure: True
```

#### With custom vagrant boxes

Custom libvirt image vagrant box can be provided using `os_box` record for each `os`:

```
os_box:
    sles-15-sp2: 'http://1.2.3.4/mirror/SLE-15-SP2/images/SLES15-SP2-Vagrant.x86_64-libvirt.box'
```

#### With custom default roles

When the user does not give the `--roles` option on the command line, `sesdev`
will use the default roles for the given deployment version. These defaults can
be changed by adding a `version_default_roles` stanza to your `~/.sesdev/config.yaml`:

```
version_default_roles:
    octopus:
        - [master, mon, mgr, storage]
        - [mon, mgr, storage]
        - [mon, mgr, storage]
```

#### config.yaml examples

##### octopus from filesystems:ceph:octopus

This is the default, so no tweaking of config.yaml is necessary. Just:

```
sesdev create octopus
```

##### octopus from filesystems:ceph:octopus:upstream

Run `sesdev create octopus` with the following options:

```
sesdev create octopus \
    --repo-priority \
    --repo https://download.opensuse.org/repositories/filesystems:/ceph:/octopus:/upstream/openSUSE_Leap_15.2 \
    --image-path registry.opensuse.org/filesystems/ceph/octopus/upstream/images/ceph/ceph
```

Alternatively, add the following to your `config.yaml` to always use these
options when deploying `octopus` clusters:

```
version_devel_repos:
    octopus:
        leap-15.2:
            - 'https://download.opensuse.org/repositories/filesystems:/ceph:/octopus/openSUSE_Leap_15.2'
            - '96!https://download.opensuse.org/repositories/filesystems:/ceph:/octopus:/upstream/openSUSE_Leap_15.2'
image_paths_devel:
    octopus:
        ceph: 'registry.opensuse.org/filesystems/ceph/octopus/upstream/images/ceph/ceph'
```

Note: The elevated priority on the `filesystems:ceph:octopus:upstream`
repo is needed to ensure that the ceph package from that project gets installed
even if RPM evaluates its version number to be lower than that of the ceph
packages in the openSUSE Leap 15.2 base and `filesystems:ceph:octopus` repos.

##### ses7 from Devel:Storage:7.0

This is the default, so no tweaking of config.yaml is necessary. Just:

```
sesdev create ses7
```

Note that this will work even if there is no ceph package visible at
https://build.suse.de/project/show/Devel:Storage:7.0 since it uses the
installation media repo, not the "SLE_15_SP2" repo.

##### ses7p from Devel:Storage:7.0:Pacific

This is the default, so no tweaking of config.yaml is necessary. Just:

```
sesdev create ses7p
```

Note that this will work even if there is no ceph package visible at
https://build.suse.de/project/show/Devel:Storage:7.0:Pacific since it uses the
installation media repo, not the "SLE_15_SP3" repo.

##### ses7 from Devel:Storage:7.0:CR

The ceph package in `Devel:Storage:7.0:CR` has the same version as
the one in `filesystems:ceph:master:upstream`, so the procedure for
using it is similar:

```
sesdev create ses7 \
    --repo-priority \
    --repo http://download.suse.de/ibs/Devel:/Storage:/7.0:/CR/SLE_15_SP2/ \
    --image-path registry.suse.de/devel/storage/7.0/cr/containers/ses/7/ceph/ceph
```

Alternatively, add the following to your `config.yaml` to always use
these options when deploying `ses7` clusters:

```
version_devel_repos:
    ses7:
        sles-15-sp2:
            - 'http://download.suse.de/ibs/SUSE:/SLE-15-SP2:/Update:/Products:/SES7/images/repo/SUSE-Enterprise-Storage-7-POOL-x86_64-Media1/'
            - 'http://download.suse.de/ibs/Devel:/Storage:/7.0/images/repo/SUSE-Enterprise-Storage-7-POOL-x86_64-Media1/'
            - '96!http://download.suse.de/ibs/Devel:/Storage:/7.0:/CR/SLE_15_SP2/'
image_paths_devel:
    ses7:
        ceph: 'registry.suse.de/devel/storage/7.0/cr/containers/ses/7/ceph/ceph'
```

Note: The elevated priority on the `Devel:Storage:7.0:CR` repo is needed to
ensure that the ceph package from that project gets installed even if RPM
evaluates its version number to be lower than that of the ceph packages in the
SES7 Product and `Devel:Storage:7.0` repos.

#### With wire encryption

The "octopus", "ses7", "ses7p", and "pacific" deployment versions can be told to
use wire encryption (a feature of the Ceph Messenger v2), where Ceph encrypts
its own network traffic.

In order to deploy a cluster with Messenger v2 encryption, we need to
either prioritise 'secure' over 'crc' mode, or only provide 'secure' mode.

The specific ceph options used to accomplish this are:

- `ms_cluster_mode`
- `ms_service_mode`
- `ms_client_mode`

By default all of these are set to `crc secure`, which prioritises `crc`
over full encryption (`secure`).

To tell sesdev to deploy a cluster with wire encryption active, provide one of
the following two options:

`--msgr2-secure-mode`: This sets the above 3 options to just 'secure'.

`--msgr2-prefer-secure`: This changes the order to `secure crc` so secure
is prefered over crc.

These only effect msgr2, so anything talking msgr1 (like the RBD and CephFS
kernel clients) will be unencrypted.

#### Deploying non-SUSE environments

sesdev has limited ability to deploy non-SUSE environments. Read on for details.

##### Ubuntu "Bionic Beaver" 18.04

Ubuntu Bionic is supported with the `octopus` deployment version. For example:

```
sesdev create octopus --os ubuntu-bionic
sesdev create octopus --single-node --os ubuntu-bionic
```

This will create Ubuntu 18.04 VMs and bootstrap a Ceph Octopus cluster on them
using `cephadm bootstrap`. To stop the deployment before bootstrap, give the
`--stop-before-cephadm-bootstrap` option.

##### Ubuntu "Focal Fossa" 20.04

Ubuntu Focal is supported with the `octopus`, `pacific`, `quincy`, `reef`
deployment versions, and this os is the default for `quincy` and `reef`.
For example:

```
sesdev create pacific --os ubuntu-focal
sesdev create pacific --single-node --os ubuntu-focal
```

This will create Ubuntu 20.04 VMs and bootstrap a Ceph Pacific cluster on them
using `cephadm bootstrap`. To stop the deployment before bootstrap, give the
`--stop-before-cephadm-bootstrap` option.

When deploying Ceph Quincy there is no need to specify `--os` option:

```
sesdev create quincy
sesdev create quincy --single-node
```

### Introspect existing deployments

Please note that the `sesdev status` and `sesdev show` commands take
a `--format` option, which can be used to make the command produce JSON output
(easily parsable by computer programs) as opposed to the default format
(intended to be read by humans).

#### List all existing deployments and their overall status

```
$ sesdev status
```

#### Get status of individual nodes in an existing deployment

```
$ sesdev status <deployment_id> [NODE]
```

For example, if I want to see the status of all nodes in deployment "foo":

```
$ sesdev status foo
```

If I want to see the status of just one node3 in deployment "foo":

```
$ sesdev status foo node3
```

#### Show details of a single existing deployment

```
$ sesdev show --detail <deployment_id>
```

#### Show roles of nodes in an existing deployment

The following command provides all details of a deployment, including the roles
of all nodes:

```
$ sesdev show --detail <deployment_id>
```

If you need to find which node of a deployment contains role "foo", try this:

```
$ sesdev show --nodes-with-role=<role> <deployment_id>
```

### SSH access to a cluster

```
$ sesdev ssh <deployment_id> [NODE]
```

Spawns an SSH shell to the master node, or to node `NODE` if explicitly
specified. You can check the existing node names with the following command:

```
$ sesdev show <deployment_id>
```

### Copy files into and out of a cluster

`sesdev` provides a subset of `scp` functionality. For details, see:

```
$ sesdev scp --help
```

### Services port-forwarding

It's possible to use an SSH tunnel to enble TCP port-forwarding for a service
running in the cluster. Currently, the following services can be forwarded:

* dashboard - The Ceph Dashboard (nautilus and above)
* grafana - Grafana metrics dashboard
* suma - SUSE Manager (octopus only)

```
$ sesdev tunnel <deployment_id> dashboard
```

The command will output the URL that you can use to access the dashboard.

### Replace ceph-salt

For deployments that used ceph-salt, it's possible to replace the ceph-salt
installed by sesdev with a different one:

```
$ sesdev replace-ceph-salt --local <path> <deployment_id>
```

Assuming `<path>` points to ceph-salt source code, the command will work
regardless of whether ceph-salt was originally installed from source or
from RPM.

### Replace MGR modules

It's possible to replace Ceph MGR modules with a version found in a github PR,
git branch or in a local repository.

This can be helpful to test PRs in a cluster with all services enabled.

```
$ sesdev replace-mgr-modules <deployment_id> <pr>
```

### Add a repo to a cluster

A custom repo can be added to all nodes of a running cluster using the following
command:

```
$ sesdev add-repo <deployment_id> <repo_url>
```

If the repo URL is omitted, the "devel" repo (as defined for the Ceph version
deployed) will be added.

If you want to also update packages on all nodes to the versions in that repo,
give the `--update` option. For example, one can test an update scenario by
deploying a cluster with the `--product` option and then updating the cluster to
the packages staged in the "devel" project:

```
$ sesdev add-repo --update <deployment_id>
```

### Link two clusters together

When sesdev deploys a Ceph cluster, the "public network" of the cluster points
at a virtual network that was created by libvirt together with the cluster VMs.
Although Ceph calls it the "public network", this network is actually *private*
in the sense that, due to iptables rules created by libvirt, packets from this
network cannot reach the "public networks" of other Ceph clusters deployed by
sesdev, even though they are all on the same host (the libvirt host).

Under ordinary circumstances, this is a good thing because it prevents packets
from one sesdev environment from reaching other sesdev environments. But there
are times when one might wish the various libvirt networks were not so isolated
from each other -- such as when trying to set up RGW Multisite, RBD Mirroring,
or CephFS Snapshot Sync between two sesdev clusters.

If you need your clusters to be able to communicate with each other over
the network and you are desperate enough to mess with iptables on the libvirt
host to accomplish it, run the following commands as root on the libvirt
host:

```
# iptables -F LIBVIRT_FWI
# iptables -A LIBVIRT_FWI -j ACCEPT
```

The LIBVIRT_FWI chain (part of the FORWARD table) contains the rules ensuring
that Vagrant environments cannot see or communicate with one another over the
network. The first command flushes the chain (deletes all these rules), and the
second one replaces them all with a single rule which unconditionally accepts
any packets that are processed through this chain. This has the effect of
completely opening up all libvirt VMs to communicate with all other libvirt VMs
on the same host.

It can also be useful to add lines to `/etc/hosts` and
`/root/.ssh/authorized_keys` on the two clusters so nodes on the "other"
cluster can be referred to by their Fully Qualified Domain Names (FQDNs, e.g.
"master.octopus2.test") and to facilitate SSHing between the two clusters. This
can be accomplished very easily by issuing the following command:

```
$ sesdev link <deployment_id_1> <deployment_id_2>
```

where `<deployment_id_1>` and `<deployment_id_2>` are the deployment IDs of two
existing sesdev clusters.

### Temporarily stop a cluster

A running cluster can be stopped by running the following command:

```
$ sesdev stop <deployment_id>
```

### Destroy a cluster

To remove a cluster (both the deployed VMs and the configuration), use the
following command:

```
$ sesdev destroy <deployment_id>
```

It has been reported that vagrant-libvirt sometimes leaves networks behind when
destroying domains (i.e. the VMs associated with a sesdev deployment). If this
bothers you, `sesdev destroy` has a `--destroy-networks` option you can use.

### Run "make check"

If your libvirtd machine has enough memory, you can use sesdev to run "make
check" in various environments. Use

```
$ sesdev create makecheck --help
```

to see the available options.

RAM CAVEAT: the default RAM amount for the makecheck might not be sufficient.
If you have plenty of memory on your libvirtd machine, running with higher
values of `--ram` (the higher, the better) is recommended.

CPUS CAVEAT: using the `--cpus` option, it is also possible increase the number
of (virtual) CPUs available for the build, but values greater than four have not
been well tested.

The `sesdev create makecheck` command will (1) deploy a VM, (2) create an
"ordinary" (non-root) user with passwordless sudo privileges and, as this
user (3) clone the specified Ceph repo and check out the specified branch,
(4) run `install-deps.sh`, and (5) run `run-make-check.sh`.

The following sub-sections provide instructions on how to reproduce some
common "make check" scenarios.

#### Run "make check" on Tumbleweed from upstream "master" branch

This is the default. Just:

```
$ sesdev create makecheck
```

#### Run "make check" on openSUSE Leap 15.2 from upstream "octopus" branch

```
$ sesdev create makecheck --os leap-15.2 --ceph-branch octopus
```

(It is not necessary to give `--ceph-repo https://github.com/ceph/ceph` here,
since that is the default.)

#### Run "make check" on SLE-15-SP2 from downstream "ses7" branch

```
$ sesdev create makecheck --os sles-15-sp2 \
      --ceph-repo https://github.com/SUSE/ceph \
      --ceph-branch ses7
```

#### Other "make check" scenarios

More combinations are supported than are described here. Compiling
the respective `sesdev create makecheck` commands for these environments is left
as an exercise for the reader.

### Custom provisioning

If you like to add configuration files or run arbitrary commands on each VM on
deployment, you can do so by providing these files in the
`~/.sesdev/.user_provision` directory.

Note that all configuration files are copied to all the VMs on deployment, as
well as the `provision.sh` file is executed on all VMs on deployment.

#### Configuration files

To have configuration added automatically to each VM, simply put them into the
`~/.sesdev/.user_provision/config` directory. All files in this directory will be
copied to `/root` on the hosts.

##### Example

Create a file `~/.sesdev/.user_provision/config/.vimrc` and it will be copied to
`/root/.vimrc` on each host on deployment, so you will always have your personal
Vim configuration on all hosts across all deployments.

#### Running arbitrary commands

Running any commands is achieved by creating a
`~/.sesdev/.user_provision/provision.sh` file. The script will be executed after
the deployment of a VM has been successfully completed.

#### Triggering custom provisioning manually

Custom provisioning can be triggered manually by issuing `sesdev user-provision
<deployment-id>` or `sesdev user-provision <deployment-id> <host>` respectively.
The former command applies the custom provisioning to all VMs in the deployment,
whereas the latter variant only to a single VM.

## Common pitfalls

This section describes some common pitfalls and how to resolve them.

### Domain about to create is already taken

#### Symptom

After deleting the `~/.sesdev` directory, `sesdev create` fails because
Vagrant throws an error message containing the words "domain about to create is
already taken".

#### Analysis

As described
[here](https://github.com/vagrant-libvirt/vagrant-libvirt/issues/658#issuecomment-335352340),
this typically occurs when the `~/.sesdev` directory is deleted. The libvirt
environment still has the domains, etc. whose metadata was deleted, and Vagrant
does not recognize the existing VM as one it created, even though the name is
identical.

#### Resolution

As described
[here](https://github.com/vagrant-libvirt/vagrant-libvirt/issues/658#issuecomment-380976825),
this can be resolved by manually deleting all the domains (VMs) and volumes
associated with the old deployment (note: the commands must be run as root):

```
# virsh list --all
# # see the names of the "offending" machines. For each, do:
# virsh destroy <THE_MACHINE>
# virsh undefine <THE_MACHINE>
# virsh vol-list default
# # For each of the volumes associated with one of the deleted machines, do:
# virsh vol-delete --pool default <THE_VOLUME>
```

### Storage pool not found: no storage pool with matching name 'default'

#### Symptom

You run `ses create` but it does nothing and gives you a traceback ending with
an error:

```
libvirt.libvirtError: Storage pool not found: no storage pool with matching name 'default'
```

#### Analysis

For whatever reason, your libvirt deployment does not have a default pool
defined. You can verify this by running the following command as root:

```
# virsh pool-list
```

In a working deployment, it says:

```
 Name      State    Autostart
-------------------------------
 default   active   no
```

but in this case the "default" storage pool is missing. (One user hit this when
deploying sesdev on SLE-15-SP1.)

#### Resolution

The "libvirt-daemon" RPM owns a directory `/var/lib/libvirt/images` which is
intended to be associated with the default storage pool:

```
# rpm -qf /var/lib/libvirt/images
libvirt-daemon-5.1.0-lp151.7.6.1.x86_64
```

Assuming this directory exists and is empty, you can simply create a storage
pool called "default" that points to this directory, and the issue will be
resolved (run the commands as root):

```
# virsh pool-define /dev/stdin <<EOF
<pool type='dir'>
  <name>default</name>
  <target>
    <path>/var/lib/libvirt/images</path>
  </target>
</pool>
EOF
# virsh pool-start default
# virsh pool-autostart default
```

Credits to Federico Simoncelli for the resolution, which I took from
[his post here](https://github.com/simon3z/virt-deploy/issues/8#issuecomment-73111541)

### When sesdev deployments get destroyed, virtual networks get left behind

#### Symptom

You create and destroy a sesdev deployment, perhaps even several
times, and then you notice that virtual networks get left behind. For example,
after several create/destroy cycles on deployment "foo":

```
# virsh net-list
 Name              State    Autostart   Persistent
----------------------------------------------------
 foo0              active   yes         yes
 foo1              active   yes         yes
 foo10             active   yes         yes
 foo2              active   yes         yes
 foo3              active   yes         yes
 foo4              active   yes         yes
 foo5              active   yes         yes
 foo6              active   yes         yes
 foo7              active   yes         yes
 foo8              active   yes         yes
 foo9              active   yes         yes
 vagrant-libvirt   active   no          yes
```

#### Analysis

It has been reported that vagrant-libvirt sometimes leaves networks behind when
it destroys domains (i.e. the VMs associated with a sesdev deployment). We do
not currently know why, or under what conditions, this happens.

#### Resolution

If this behavior bothers you, `sesdev destroy` has a `--destroy-networks` option
you can use. Of course, `sesdev destroy --destroy-networks` only works for the
network(s) associated with the VMs in the deployment being destroyed. To quickly
destroy a bunch of networks, construct a script like this one:

```
#!/bin/bash
read -r -d '' NETZ <<EOF
foo0
foo1
foo2
foo3
foo4
foo5
foo6
foo7
foo8
foo9
foo10
EOF
for net in $NETZ ; do
    virsh net-destroy $net
    virsh net-undefine $net
done
```

The script should be run as root on the libvirt server.

An unsupported, user-contributed version of this script -- `contrib/nukenetz.sh`
-- can be found in the source-code tree.

Also, read the [next section](#sesdev-destroy-reported-an-error) for more
relevant information.

### sesdev destroy reported an error

#### Symptom

You ran `sesdev destroy` but there were errors and you suspect that a deployment
(or deployments) might not have been completely destroyed.

#### Analysis

The command `sesdev destroy` has been known to fail, leaving a deployment "not
completely destroyed".

A sesdev deployment `DEP_ID` consists of several components:

* a subdirectory under `~/.sesdev/DEP_ID`
* some number of libvirt domains
* some number of libvirt storage volumes in the `default` storage pool
* some number of libvirt networks

and the names of all the libvirt domains, volumes, and networks used by domain
`DEP_ID` can be expected to begin with `DEP_ID`. For example, if the DEP_ID
is "octopus", the associated libvirt artifacts will have names starting with
"octopus".

#### Resolution

Use the following commands to check for vestiges of your deployment:

```
sudo virsh list --all | grep '^ DEP_ID'
sudo virsh vol-list default | grep '^ DEP_ID'
sudo virsh net-list | grep '^ DEP_ID'
(cd ~/.sesdev ; ls -d1 */ | grep '^DEP_ID')
```

Then, **assuming you use your libvirt instance is dedicated to `sesdev` and not
used for anything else**, you could use the following commands to delete
everything you found, and that would clean up the partially-destroyed
deployment:

```
sudo virsh destroy LIBVIRT_DOMAIN
sudo virsh undefine LIBVIRT_DOMAIN
sudo virsh vol-delete --pool default LIBVIRT_STORAGE_POOL
sudo virsh net-destroy LIBVIRT_NETWORK
sudo virsh net-undefine LIBVIRT_NETWORK
```

### "Failed to connect socket" error when attempting to use remote libvirt server

#### Symptom

When attempting to create or list deployments on a remote libvirt/SSH server,
sesdev barfs out a Python traceback ending in:

```
libvirt.libvirtError: Failed to connect socket to
'/var/run/libvirt/libvirt-sock': No such file or directory
```

#### Analysis

When told to use remote libvirt/SSH, sesdev expects that there won't be any
libvirtd instance running locally. This Python traceback is displayed when

1. sesdev is configured to use remote libvirt/SSH, **and**
2. libvirtd.service is running locally

#### Resolution

Stop the local libvirtd.service.

### mount.nfs: Unknown error 521

#### Symptom

When the `--synced-folder` option is provided, the deployment fails with
something like:

    mount -o vers=3,udp 192.168.xxx.xxx:/home/$USER/.sesdev/$NAME /$PATH

    Stderr from the command:

    mount.nfs: Unknown error 521

#### Analysis

This indicates that your nfs-server is not working properly or hasn't started yet.

#### Resolution

Please make sure that your nfs-server is up and running without errors:

```
# systemctl status nfs-server
```

If this doesn't report back with `active`, please consider running:

```
# systemctl restart nfs-server
# systemctl enable nfs-server
```

### Problems accessing dashboard on remote sesdev

#### Symptom

I'm running sesdev on a remote machine and I want to access the dashboard of
a cluster deployed by sesdev on that machine. Since the machine is remote, I
can't just fire up a browser on it. I would like to point a browser that I have
running locally (e.g. on a laptop) at the dashboard deployed by sesdev on the
remote machine. I've tried a bunch of stuff, but I just can't seem to make it
work.

#### Analysis

There are two possible pitfalls you could be hitting. First: if you do

```
sesdev tunnel DEP_ID dashboard
```

sesdev will choose an IP address essentially at random. Your remote sesdev
machine very likely has multiple IP addresses and sesdev, in accordance with
Murphy's Law, sesdev is choosing an IP address which is not accessible from
the machine where the browser is running.

However, even when specifying `--local-address CORRECT_IP_ADDRESS`, it still
might not work if there are other dashboard instances (sesdev or bare metal)
running on the remote machine and already listening on the port where the newly
deployed dashboard is listening. In other words, there might be other dashboards
running on the machine that you're not aware of.

Things are further confused by the nomenclature of the `sesdev tunnel` command.
What sesdev refers to as "local address/port" is actually the address/port on
the remote machine (remote to you, but local to sesdev itself). What it refers
to as "remote port" is the port that is being tunneled (the one inside the VM,
on which the dashboard is listening).

#### Resolution

First, you have to be really sure that the "local IP address" you feed into the
`sesdev tunnel` command is (1) a valid IP address of the sesdev machine that (2)
is accessible from the browser running on your local machine.

Once you are sure of the correct IP address, use `sesdev ssh DEP_ID` to enter
the cluster and run

```
ceph mgr services
```

This will tell you the node where the dashboard is running, and the port that
it's listening on, and the protocol to use (http or https). Carefully write down
of all three pieces of information. Now, do:

```
sesdev tunnel DEP_ID \
    --node NODE_WHERE_DASHBOARD_IS_RUNNING \
    --remote-port PORT_WHERE_DASHBOARD_IS_LISTENING \
    --local-address CORRECT_IP_ADDRESS \
    --local-port ANY_ARBITRARY_HIGH_NUMBERED_PORT
```

The output of this command will say

```
You can now access the service in: CORRECT_IP_ADDRESS:ANY_ARBITRARY_HIGH_NUMBERED_PORT
```

Now, you probably can't just paste that URL into your browser, because the
dashboard is likely using SSL (the default). Instead, refer to your notes to
determine the protocol the dashboard is using (probably "https", but might be
"http" if SSL is disabled), and then fashion a fully-qualified URL like so:

```
PROTOCOL://CORRECT_IP_ADDRESS:ANY_ARBITRARY_HIGH_NUMBERED_PORT
```

One final note: it's a good practice to use a different
`ANY_ARBITRARY_HIGH_NUMBERED_PORT` every time you run `sesdev tunnel`. This is
because of ``https://github.com/SUSE/sesdev/issues/276``.

### Error creating IPv6 cluster

#### Symptom

I'm running `sesdev create` with `--ipv6` option, and I'm getting the following error:

```
Error while activating network: Call to virNetworkCreate failed: internal error:
Check the host setup: enabling IPv6 forwarding with RA routes without accept_ra
set to 2 is likely to cause routes loss. Interfaces to look at: enp0s25.
```

#### Resolution

Set "Accept Router Advertisements" to 2 ("Overrule forwarding behaviour"), by running:

```
sysctl -w net.ipv6.conf.<if>.accept_ra=2
```

Where `<if>` is the network interface from the error, or `all` if you want to apply
the config to all network interfaces.

### Failed to initialize libnetcontrol

#### Symptom

After starting `libvirtd.service`, `systemctl status libvirtd.service` says

```
Failed to intialize libnetcontrol.  Management of interface devices is disabled
```

(Yes, it really says "intialize" instead of "initialize".)

#### Analysis

At present, `libvirtd` works together well with the `wicked` network management
system. It does not work so well with `NetworkManager`, so if you see this
message it probably means you are using `NetworkManager`.

These two - `wicked` and `NetworkManager` - are mutually exclusive: you must
have one or the other, and you cannot have both at the same time.

#### Resolution

The resolution is to disable `NetworkManager` by enabling `wicked` and
configuring it properly (i.e. so you don't experience any loss in network
connectivity).

Refer to your operating system's documentation to learn how to configure
networking with `wicked`. For example, for openSUSE Leap 15.2 you can refer
to Reference -> System -> Basic Networking:

https://doc.opensuse.org/documentation/leap/reference/html/book-opensuse-reference/cha-network.html

Once you have `wicked` running without any loss of network connectivity, proceed
to restart libvirtd:

```
# systemctl restart libvirtd.service
```

After this, the `Failed to intialize libnetcontrol` message should no longer
appear in the journal (log) of `libvirtd.service`.

## Contributing

If you would like to submit a patch to sesdev, or otherwise participate in the
sesdev community, please read the files `CONTRIBUTING.rst` and
`CODE_OF_CONDUCT.md` in the top-level directory of the source code
distribution. These files can also be found on-line:

https://github.com/SUSE/sesdev/blob/master/CONTRIBUTING.rst
https://github.com/SUSE/sesdev/blob/master/CODE_OF_CONDUCT.md


Raven : Hydrological modeling and analytics
===========================================

.. image:: https://readthedocs.org/projects/pavics-raven/badge/?version=latest
    :target: https://pavics-raven.readthedocs.io/en/latest/?badge=latest
    :alt: Documentation Status

.. image:: https://github.com/Ouranosinc/raven/actions/workflows/main.yml/badge.svg
    :target: https://github.com/Ouranosinc/raven/actions/workflows/main.yml
    :alt: Build status

.. image:: https://img.shields.io/github/license/Ouranosinc/raven.svg
    :target: https://github.com/Ouranosinc/raven/blob/main/LICENSE.txt
    :alt: GitHub license

.. image:: https://badges.gitter.im/bird-house/birdhouse.svg
    :target: https://gitter.im/bird-house/birdhouse?utm_source=badge&utm_medium=badge&utm_campaign=pr-badge&utm_content=badge
    :alt: Join the chat at https://gitter.im/bird-house/birdhouse

.. image:: https://app.fossa.com/api/projects/git%2Bgithub.com%2FOuranosinc%2Fraven.svg?type=shield
    :target: https://app.fossa.com/projects/git%2Bgithub.com%2FOuranosinc%2Fraven?ref=badge_shield
    :alt: FOSSA

.. image:: https://zenodo.org/badge/135511617.svg
    :target: https://zenodo.org/badge/latestdoi/135511617
    :alt: DOI

Raven (the bird)
  *Raven offers processes related to hydrological modeling, and in particular, the Raven hydrological modelling framework.*

Raven is an open source server project offering data collection and preparation, as well as geoprocessing and catchment delineation through the Web Processing Service (WPS) standard. Raven processes can be embedded in a graphical user interface or accessed directly from a programming environment. From Python, birdy_ WPSClient provides a user-friendly python interface to Raven's WPS processes for geospatial processing.

The properties of custom watersheds can be extracted from a Digital Elevation Model and a land-use database.

Raven can be compiled and installed, or simply deployed using docker. A hosted version is available at  https://pavics.ouranos.ca/twitcher/ows/proxy/raven.

Documentation
-------------

Learn more about Raven in its official documentation at
https://pavics-raven.readthedocs.io.

Submit bug reports, questions and feature requests at
https://github.com/Ouranosinc/raven/issues

Contributing
------------

You can find information about contributing in our `Developer Guide`_.

Please use bumpversion_ to release a new version.

License
-------

Free software: MIT license

Credits
-------

This project was funded by the CANARIE_ research software program.

Hydrological models are based on the `Raven`_ modeling framework.

This package was created with Cookiecutter_ and the `bird-house/cookiecutter-birdhouse`_ project template.

.. _`birdy`: https://birdy.readthedocs.io
.. _`xarray`: http://xarray.pydata.org
.. _`xclim`: https://xclim.readthedocs.io
.. _`Raven`: http://raven.uwaterloo.ca
.. _`CANARIE`: https://www.canarie.ca
.. _Cookiecutter: https://github.com/audreyr/cookiecutter
.. _`bird-house/cookiecutter-birdhouse`: https://github.com/bird-house/cookiecutter-birdhouse
.. _`Developer Guide`: https://pavics-raven.readthedocs.io/en/latest/dev_guide.html
.. _bumpversion: https://pavics-raven.readthedocs.io/en/latest/dev_guide.html#bump-a-new-version