# README
[![FOSSA Status](https://app.fossa.com/api/projects/git%2Bgithub.com%2Fsplunk%2Fsplunk-connect-for-syslog.svg?type=shield)](https://app.fossa.com/projects/git%2Bgithub.com%2Fsplunk%2Fsplunk-connect-for-syslog?ref=badge_shield)

Splunk Connect for Syslog is an open source packaged solution to
get data into Splunk using syslog-ng (OSE) and the Splunk
HTTP event Collector.

## Purpose

Splunk Connect for Syslog (SC4S) is a community project that helps reduce the pain of getting syslog data sources into Splunk. Splunk Connect for Syslog should be used by any Splunk customer needing to onboard data sources via syslog to Splunk. The primary pain points SC4S addresses include the following:

* Lack of deep syslog expertise in the community
* Inconsistency between syslog server deployments, which creates a support challenge
* Data sources tagged with catch-all sourcetype “syslog”, which limits Splunk analytics
* Uneven data distribution between Splunk indexers, which impacts search performance

## Usage

For full usage instructions, please visit the Splunk Connect for Syslog [documentation](https://splunk.github.io/splunk-connect-for-syslog/).

## Getting Support

Thank you for considering SC4S for your Splunk needs.

**Splunk Support**: If you are an existing Splunk customer with access to the Support Portal, create a support ticket for the quickest resolution to any issues you experience. Here are some examples of when it may be appropriate to create a support ticket:
- If you experience an issue with the current version of SC4S, such as a feature gap or a documented feature that is not working as expected.
- If you have difficulty with the configuration of SC4S, either at the back end or with the out-of-box parsers or index configurations.
- If you experience performance issues and need help understanding the bottlenecks.
- If you have any questions or issues with the SC4S documentation.

**GitHub Issues**: For all enhancement requests, please feel free to create GitHub issues. We prioritize and work on issues based on their priority and resource availability. You can help us by tagging the requests with the appropriate labels.

_Splunk Developers are active in the external usergroup on best effort basis, Please use support case/github issues to resolve your issues quickly_

## Contributing

We welcome feedback and contributions from the community! Please see our [contribution guidelines](/docs/CONTRIBUTING.md) for more information on how to get involved. PR contributions require acceptance of both the code of conduct and the contributor license agreement.

This repository uses `pre-commit`. After installing dependencies, please do
```bash
pre-commit install
```

## License

* Configuration and documentation licensed subject to [CC0](LICENSE-CC0)

* Code and scripts licensed subject to [BSD-2-Clause](LICENSE-BSD2)

* Third Party Axoflow image of syslog-ng [License](https://github.com/axoflow/axosyslog-docker/blob/main/LICENSE.)

* Third Party Syslog-NG (OSE) [License](https://github.com/balabit/syslog-ng)

[![FOSSA Status](https://app.fossa.com/api/projects/git%2Bgithub.com%2Fsplunk%2Fsplunk-connect-for-syslog.svg?type=large)](https://app.fossa.com/projects/git%2Bgithub.com%2Fsplunk%2Fsplunk-connect-for-syslog?ref=badge_large)

## References

* Syslog-ng Documentation provided by Axoflow [Docs](https://axoflow.com/docs/axosyslog/)


# Build an offline, airgapped Bitcoin signing device for less than $50!

![Image of SeedSigners in Mini Pill Enclosures](docs/img/Mini_Pill_Main_Photo.jpg)

---------------

* [Project Summary](#project-summary)
* [Shopping List](#shopping-list)
* [Software Installation](#software-installation)
  * [Verifying the Software](#verifying-the-software)
* [Enclosure Designs](#enclosure-designs)
* [SeedQR Printable Templates](#seedqr-printable-templates)
* [Build from Source](#build-from-source)
* [Developer Local Build Instructions](#developer-local-build-instructions)

---------------

# Project Summary

[![CI](https://github.com/SeedSigner/seedsigner/actions/workflows/tests.yml/badge.svg)](https://github.com/SeedSigner/seedsigner/actions/workflows/tests.yml)
[![Build](https://github.com/SeedSigner/seedsigner/actions/workflows/build.yml/badge.svg)](https://github.com/SeedSigner/seedsigner/actions/workflows/build.yml)

The goal of SeedSigner is to lower the cost and complexity of Bitcoin multi-signature wallet use. To accomplish this goal, SeedSigner offers anyone the opportunity to build a verifiably air-gapped, stateless Bitcoin signing device using inexpensive, publicly available hardware components (usually < $50). SeedSigner helps users save with Bitcoin by assisting with trustless private key generation and multisignature (aka "multisig") wallet setup, and helps users transact with Bitcoin via a secure, air-gapped QR-exchange signing model.

Additional information about the project can be found at [SeedSigner.com](https://seedsigner.com).

You can follow [@SeedSigner](https://twitter.com/SeedSigner) on Twitter for the latest project news and developments.

If you have specific questions about the project, our [Telegram Group](https://t.me/joinchat/GHNuc_nhNQjLPWsS) is a great place to ask them.

### Feature Highlights:
* Calculate the final word (aka checksum) of a 12- or 24-word BIP39 seed phrase
* Create a 24-word BIP39 seed phrase with 99 dice rolls or a 12-word with 50 rolls [(Verifying dice seed generation)](docs/dice_verification.md)
* Create a 12- or 24-word BIP39 seed phrase via image entropy from the onboard camera
* Temporarily stores seeds in memory while the device is powered; all memory is wiped when power is removed
* SD card removable after boot to ensure no secret data can be written to it
* Guided interface to manually transcribe a seed to the SeedQR format for instant seed loading [(demo video here)](https://youtu.be/c1-PqTNx1vc)
* BIP39 passphrase (aka "word 25") support
* Native Segwit Multisig XPUB generation
* PSBT-compliant; scan and parse transaction data from animated QR codes
* Sign transactions & transfer XPUB data using animated QR codes [(demo video here)](https://youtu.be/LPqvdQ2gSzs)
* Live preview during image entropy seed generation and QR scanning UX
* Optimized seed word entry interface
* Support for Bitcoin Mainnet & Testnet
* Support for custom user-defined derivation paths
* On-demand receive address verification
* Address Explorer for single sig and multisig wallets
* User-configurable QR code display density
* Responsive, event-driven user interface

### Considerations:
* Built for compatibility with Specter Desktop, Sparrow, and BlueWallet Vaults
* Device takes up to 60 seconds to boot before menu appears (be patient!)
* Always test your setup before transferring larger amounts of bitcoin (try Testnet first!)
* Taproot not quite yet supported
* Slightly rotating the screen clockwise or counter-clockwise should resolve lighting/glare issues
* If you think SeedSigner adds value to the Bitcoin ecosystem, please help us spread the word! (tweets, pics, videos, etc.)

### Planned Upcoming Improvements / Functionality:
* Multi-language support
* Significantly faster boot time
* Reproducible builds
* Port to MicroPython to broaden the range of compatible hardware to include low-cost microcontrollers
* Other optimizations based on user feedback!

---------------

# Shopping List

To build a SeedSigner, you will need:

* Raspberry Pi Zero (preferably version 1.3 with no WiFi/Bluetooth capability, but any Raspberry Pi 2/3/4 or Zero model will work, Raspberry Pi 1 devices will require a hardware modification to the Waveshare LCD Hat, as per the [instructions here](./docs/legacy_hardware.md))
* Waveshare 1.3" 240x240 pxl LCD (correct pixel count is important, more info at https://www.waveshare.com/wiki/1.3inch_LCD_HAT)
* Pi Zero-compatible camera (tested to work with the Aokin / AuviPal 5MP 1080p with OV5647 Sensor)

Notes:
* You will need to solder the 40 GPIO pins (20 pins per row) to the Raspberry Pi Zero board. If you don't want to solder, purchase "GPIO Hammer Headers" for a solderless experience.
* Other cameras with the above sensor module should work, but may not fit in the Orange Pill enclosure
* Choose the Waveshare screen carefully; make sure to purchase the model that has a resolution of 240x240 pixels

---------------

# Software Installation

## A Special Note On Minimizing Trust
As is the nature of pre-packaged software downloads, downloading and using the prepared SeedSigner release images means implicitly placing trust in the people preparing those images; in our project the released images are prepared and signed by the eponymous creator of the project, SeedSigner "the person". That individual is additionally the only person in possession of the PGP keys that are used to sign the release images.

Starting with v0.7.0, the images distributed via GitHub are reproducible. This means you and others can verify the released images are byte-for-byte the same when built from source. You can contribute to this project by building from source and sharing the hash of the final images.

Instructions to build a SeedSigner OS image (using precisely the same process that is used to create the prepared release images) have been made available. We have put a lot of thought and work into making these instructions easy to understand and follow, even for less technical users. These instructions can be found [here](https://github.com/SeedSigner/seedsigner-os/blob/main/docs/building.md).

## Downloading the Software

Download the current Version (0.7.0) software image that is compatible with your  Raspberry Pi Hardware. The Pi Zero 1.3 is the most common and recommended board.
| Board                 | Download Image Link/Name          |
| --------------------- | --------------------------------- |
|**[Raspberry Pi Zero 1.3](https://www.raspberrypi.com/products/raspberry-pi-zero/)**      |[`seedsigner_os.0.7.0.pi0.img`](https://github.com/SeedSigner/seedsigner/releases/download/0.7.0/seedsigner_os.0.7.0.pi0.img)      |
|[Raspberry Pi Zero W](https://www.raspberrypi.com/products/raspberry-pi-zero-w/)    |[`seedsigner_os.0.7.0.pi0.img`](https://github.com/SeedSigner/seedsigner/releases/download/0.7.0/seedsigner_os.0.7.0.pi0.img)      |
|[Raspberry Pi Zero 2 W](https://www.raspberrypi.com/products/raspberry-pi-zero-2-w/)  |[`seedsigner_os.0.7.0.pi02w.img`](https://github.com/SeedSigner/seedsigner/releases/download/0.7.0/seedsigner_os.0.7.0.pi02w.img)    |
|[Raspberry Pi 1 Model B/B+](https://www.raspberrypi.com/products/raspberry-pi-1-model-b-plus/) |[`seedsigner_os.0.7.0.pi0.img`](https://github.com/SeedSigner/seedsigner/releases/download/0.7.0/seedsigner_os.0.7.0.pi0.img)      |
|[Raspberry Pi 2 Model B](https://www.raspberrypi.com/products/raspberry-pi-2-model-b/) |[`seedsigner_os.0.7.0.pi2.img`](https://github.com/SeedSigner/seedsigner/releases/download/0.7.0/seedsigner_os.0.7.0.pi2.img)      |
|[Raspberry Pi 3 Model B](https://www.raspberrypi.com/products/raspberry-pi-3-model-b/) |[`seedsigner_os.0.7.0.pi02w.img`](https://github.com/SeedSigner/seedsigner/releases/download/0.7.0/seedsigner_os.0.7.0.pi02w.img)    |
|[Raspberry Pi 4 Model B](https://www.raspberrypi.com/products/raspberry-pi-4-model-b/) |[`seedsigner_os.0.7.0.pi4.img`](https://github.com/SeedSigner/seedsigner/releases/download/0.7.0/seedsigner_os.0.7.0.pi4.img)      |
|[Raspberry Pi 400](https://www.raspberrypi.com/products/raspberry-pi-400-unit/) |[`seedsigner_os.0.7.0.pi4.img`](https://github.com/SeedSigner/seedsigner/releases/download/0.7.0/seedsigner_os.0.7.0.pi4.img)      |

Note: If you have physically removed the WiFi component from your board, you will still use the image file of the original(un-modified) hardware. (Our files are compiled/based on the *processor* architecture). Although it is better to spend a few minutes upfront to determine which specific Pi hardware/model you have, if you are still unsure which hardware you have, you can try using the pi0.img file. Making an incorrect choice here will not ruin your board, because this is software, not firmware.

**also download** these 2 signature verification files to the same folder
[The Plaintext manifest file](https://github.com/SeedSigner/seedsigner/releases/download/0.7.0/seedsigner.0.7.0.sha256.txt)
[The Signature of the manifest file](https://github.com/SeedSigner/seedsigner/releases/download/0.7.0/seedsigner.0.7.0.sha256.txt.sig)

Users familiar with older versions of the SeedSigner software might be surprised with how fast their software downloads now are, because since version 0.6.0 the software image files are now 100x smaller! Each image file is now under 42 Megabytes so your downloads and verifications will be very quick now (and might even seem *too* quick)!

Once the files have all finished downloading, follow the steps below to verify the download before continuing on to write the software onto a MicroSD card. Next, insert the MicroSD into your assembled hardware and connect the USB power. Allow about 45 seconds for our logo to appear, and then you can begin using your SeedSigner!

[Our previous software versions are available here](https://github.com/SeedSigner/seedsigner/releases). Choose a specific version and then expand the *Assets* sub-heading to display the .img file binary and also the 2 associated signature files. **Note:** The prior version files will have lower numbers than the scripts and examples provided in this document, but the naming format will be the same, so you can edit them as required for signature verification etc.

## Verifying that the downloaded files are authentic (optional but highly recommended!)

You can quickly verify that the software you just downloaded is both authentic and unaltered, by following these instructions.
We assume you are running the commands from a computer where both [GPG](https://gnupg.org/download/index.html) and [shasum](https://command-not-found.com/shasum) are already installed, and that you also know [how to navigate on a terminal](https://terminalcheatsheet.com/guides/navigate-terminal).

> You must run the following verification before opening or mounting the .img file.
> Some operating systems modify the file on mount causing verification to fail.

### Step 1. Verify that the signature (.sig) file is genuine:

Run GPG's *fetch-keys* command to import the SeedSigner projects public key from the popular online keyserver called *Keybase.io*, into your computers *keychain*.

```
gpg --fetch-keys https://keybase.io/seedsigner/pgp_keys.asc
```
The result should confirm that 1 key was *either* imported or updated. *Ignore* any key ID's or email addresses shown.

![SS - Fetchkeys-Keybase PubKey import with Fingerprint shown (New import or update of the key)v3-100pct](https://user-images.githubusercontent.com/91296549/221334414-adc3616c-462e-490e-8492-3dfee367d13a.jpg)

Next, you will run the *verify* command on the signature (.sig) file. (*Verify* must be run from inside the same folder that you downloaded the files into earlier. The `*`'s in this command will auto-fill the version from your current folder, so it should be copied and pasted as-is.)
```
gpg --verify seedsigner.0.7.*.sha256.txt.sig
```

When the verify command completes successfully, it should display output like this:

![SS - Verify Command - GPG on Linux - Masked_v4-100pct](https://user-images.githubusercontent.com/91296549/221334135-8ad1f1af-26d2-429a-91ce-ad41703ed38c.jpg)
The result must display "**Good signature**".  Ignore any email addresses - *only*  matching Key fingerprints count here. Stop immediately if it displays "*Bad signature*"!

On the *last* output line, look at your *rightmost* 16 characters (the 4 blocks of 4).
**Crucially, we must now check WHO that Primary key fingerprint /ID belongs to.** We will start by looking at Keybase.io to see if it is the *SeedSigner project* 's public key or not.

 About the warning message:

  Since you are about to match the outputted fingerprint/ID against the proofs at Keybase.io/SeedSigner, and thereby confirm who the pubkey really belongs to-, you can safely ignore this warning message:

```
> WARNING: This key is not certified with a trusted signature!
> There is no indication that the signature belongs to the owner.
 ```

 More about how the verify command works:

The verify command will attempt to decrypt the signature file (sha256.sig) by trying each public key already imported into your computer. If the public key we just imported (via fetch-keys), manages to: (a) successfully decrypt the .sig file , and (b), that result matches exactly to the clear-text equivalent (.sha256) of the .sig file, then its "a good signature"!

Crucially, we must still manually check who *exactly* owns the Key ID which gave us that "Good signature". Thats what the warning message means- Who does the matching key really belong to? We will start by looking at keybase.io to see if it is "The SeedSigner project"'s public Key or not.
Note that it is the file hashes of .sig and .sha256 that *verify* compares, not their raw contents.

Now to determine ***who*** the Public key ID belongs to: Goto [Keybase.io/SeedSigner](https://keybase.io/seedsigner)

![SS - Keybase Website PubKey visual matching1_Cropped-80pct](https://user-images.githubusercontent.com/91296549/215326193-97c84e35-5570-4e52-bf3f-e86d367c8908.jpg)

**You must now *manually* compare: The 16 character fingerprint ID (as circled in red above) to, those *rightmost* 16 characters from your *verify* command.**

**If they match exactly, then you have successfully confirmed that your .sig file is authentically from the SeedSigner Project!**

Learn more about how keybase.io helps you check that someone (online) is who they say they are:

Keybase.io allows you to independently verify that the public key saved on Keybase.io, is both authentic and that it belongs to the organization it claims to represent.
 Keybase has already checked the three pubkey file locations  cryptographically when they were saved there. You can further verify the key publications if you would like:

 - *via Keybase*: By clicking on any of the three blue badges to see that the "proof" was published at that location. (The blue badge marked as tweet, is in the most human-readable form and it is also a bi-directional link on Twitter)
or,
 - *without keybase (out-of-band)*: By using these 3 links directly: [Twitter](https://twitter.com/SeedSigner/status/1530555252373704707), [Github](https://gist.github.com/SeedSigner/5936fa1219b07e28a3672385b605b5d2) and [SeedSigner.com](https://seedsigner.com/keybase.txt). This method can be used if you would like to make an even deeper, independent inspection without relying on Keybase at all, or if the Keybase.io site is no longer valid or it is removed entirely.

Once you have used one of these methods, you will know if the Public Key stored on Keybase, is genuinely from the SeedSinger Project or not.

If the two ID's do *not* match, then you must stop here immediately. Do not continue. Contact us for assistance in the Telegram group address above.

### Step 2. Verifying that the *software images/binaries* are genuine

Now that you have confirmed that you do have the real SeedSigner Project's Public Key (ie the 16 characters match) - you can return to your terminal window. Running  the *shasum* command, is the final verification step and will confirm (via file hashing) that the software code/image files, were also not altered since publication, or even during your download process.
(Prior to version 0.6.0  , your verify command will check the .zip file which contains the binary files.)

 **On Linux or OSX:** Run this command
```
shasum -a 256 --ignore-missing --check seedsigner.0.7.*.sha256.txt
```

**On Windows (inside Powershell):** Run this command
```
CertUtil -hashfile  seedsigner_os.0.7.0.Insert_Your_Pi_Models_binary_here_For_Example_pi02w.img SHA256
```
On Windows, you must then manually compare the resulting file hash value to the corresponding hash value shown inside the .SHA256 cleartext file.

Wait up to 30 seconds for the command to complete, and it should display:
```
seedsigner_os.0.7.x.[Your_Pi_Model_For_Example:pi02w].img: OK
```
**If you receive the "OK" message** for your **seedsigner_os.0.7.x.[Your_Pi_Model_For_Example:pi02w].img file**, as shown above, then your verification is fully complete!
**All of your downloaded files have now been confirmed as both authentic and unaltered!** You can proceed to create/write your MicroSD card😄😄 !!

If your file result shows "FAILED", then you must stop here immediately. Do not continue. Contact us for assistance at  the Telegram group address above.

Please recognize that this process can only validate the software to the extent that the entity that first published the key is an honest actor, and their private key is not compromised or somehow being used by a malicious actor.

## Writing the software onto your MicroSD card

To write the SeedSigner software onto your MicroSD card, there are a few options available:
| Application              | Description                                                                                | Platform and official Source                                                         |
|--------------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------|--------------------------------------------------------------------------------------|
| Balena Etcher            | The application is called Etcher, and the company that wrote it is called Balena.  Hence *Etcher by Balena* or *Balena Etcher*                                                  | [Available for Windows, Mac and Linux](https://www.balena.io/etcher#download-etcher) |
| Raspberry Pi Imager      | Produced by the Raspberry Pi organization.                                                                                | [Available for Windows, Mac and Linux](https://www.raspberrypi.com/software/)        |
| DD Command Line Utility  | Built-in to Linux and MacOS, the DD (Data Duplicator) is a tool for advanced users.  If not used carefully it can accidentally format the incorrect disk!   | Built-in to Linux and MacOS                                                        |

Be sure to download the software from the genuine publisher.
Either of the Etcher or Pi Imager software is recommended.  Some SeedSigner users have reported a better experience with one or the other. So, if the one application doesn’t work well for your particular machine, then please try the other one.

### **General Considerations:**
The writing and verify steps are very quick from version 0.6.0 upwards, so please pay close attention to your screen.
Make sure to set any write-protection physical slider on the MicroSD Card Adapter to UN-locked.
You also don’t need to pre-format the MicroSD beforehand.  You *dont* need to unzip any .zip file beforehand.
Current Etcher and Pi Imager software will perform a verify action (by default) to make sure the card was written successfully! Watching for that verify step to complete successfully, can save you a lot of headaches if you later need to troubleshoot issues where your SeedSigner device doesn’t boot up at power on.
Writing the MicroSd card is also known as flashing.
It will overwrite everything on the MicroSD card.
If the one application fails for you, then please try again using our other recommended application.
Advanced users may want to try the Linux/MacOS *DD* command instead of using Etcher or Pi Imager, however, a reminder is given that DD can overwrite the wrong disk if you are not careful !
#### **Specific considerations for Windows users:**
Use the Pi imager software as your first choice on Windows. Windows can sometimes flag the writing of a MicroSD as risky behaviour and hence it may prevent this activity. If this happens, your writing/flashing will fail, hang or wont even begin, in which case you should to try to run the Etcher/Pi-Imager app "As administrator", (right-click and choose that option). It can also be blocked by windows security in some cases, so If you have the (non-default) *Controlled Folder Access* option set to active, try turning that *off* temporarily.

---------------

# Enclosure Designs

### Open Pill

The Open Pill enclosure design is all about quick, simple and inexpensive depoloyment of a SeedSigner device. The design does not require any additional hardware and can be printed using a standard FDM 3D printer in about 2 hours, no supports necessary. A video demonstrating the assembly process can be found [here](https://youtu.be/gXPFJygZobEa). To access the design file and printable model, click [here](https://github.com/SeedSigner/seedsigner/tree/main/enclosures/open_pill).

### Orange Pill

The Orange Pill enclosure design offers a more finished look that includes button covers and a joystick topper. You'll also need the following additional hardware to assemble it:

* 4 x F-F M2.5 spacers, 10mm length
* 4 x M2.5 pan head screws, 6mm length
* 4 x M2.5 pan head screws, 12mm length

The upper and lower portions of the enclosure can be printed using a standard FDM 3D printer, no supports necessary. The buttons and joystick nub should ideally be produced with a SLA/resin printer. An overview of the entire assembly process can be found [here](https://youtu.be/aIIc2DiZYcI). To access the design files and printable models, click [here](https://github.com/SeedSigner/seedsigner/tree/main/enclosures/orange_pill).

### Community Designs

* [Lil Pill](https://cults3d.com/en/3d-model/gadget/lil-pill-seedsigner-case) by @_CyberNomad
* [OrangeSurf Case](https://github.com/orangesurf/orangesurf-seedsigner-case) by @OrangeSurfBTC
* [PS4 SeedSigner](https://www.thingiverse.com/thing:5363525) by @Silexperience
* [OpenPill Faceplate](https://www.printables.com/en/model/179924-seedsigner-open-pill-cover-plates-digital-cross-jo) by @Revetuzo
* [Waveshare CoverPlate](https://cults3d.com/en/3d-model/various/seedsigner-coverplate-for-waveshare-1-3-inch-lcd-hat-with-240x240-pixel-display) by @Adathome1

---------------

# SeedQR Printable Templates
You can use SeedSigner to export your seed to a hand-transcribed SeedQR format that enables you to instantly load your seed back into SeedSigner.

[More information about SeedQRs](docs/seed_qr/README.md)

Standard SeedQR templates:
* [12-word SeedQR template dots (25x25)](docs/seed_qr/printable_templates/dots_25x25.pdf)
* [24-word SeedQR template dots (29x29)](docs/seed_qr/printable_templates/dots_29x29.pdf)
* [12-word SeedQR template grid (25x25)](docs/seed_qr/printable_templates/grid_25x25.pdf)
* [24-word SeedQR template grid (29x29)](docs/seed_qr/printable_templates/grid_29x29.pdf)
* [Baseball card template: 24-word SeedQR (29x29)](docs/seed_qr/printable_templates/trading_card_29x29_w24words.pdf)

CompactSeedQR templates:
* [12-word CompactSeedQR template dots (21x21)](docs/seed_qr/printable_templates/dots_21x21.pdf)
* [24-word CompactSeedQR template dots (25x25)](docs/seed_qr/printable_templates/dots_25x25.pdf)
* [12-word CompactSeedQR template grid (21x21)](docs/seed_qr/printable_templates/grid_21x21.pdf)
* [24-word CompactSeedQR template grid (25x25)](docs/seed_qr/printable_templates/grid_25x25.pdf)
* [Baseball card template: 12-word Compact SeedQR (21x21)](docs/seed_qr/printable_templates/trading_card_21x21_w12words.pdf)
* [Baseball card template: 24-word Compact SeedQR (25x25)](docs/seed_qr/printable_templates/trading_card_25x25_w24words.pdf)

2-sided SeedQR templates - 8 per sheet
Printing settings - (2-sided)("flip on long edge")("Actual Size")
If printing on cardstock, adjust your printer settings via its control panel

A4 templates(210mm * 297mm):
* [21x21 - stores 12-word seeds ONLY in CompactSeedQR format ONLY](docs/seed_qr/printable_templates/21x21_A4_trading_card_2sided.pdf)
* [25x25 - stores 12-word or 24 word seeds depending on SeedQR format](docs/seed_qr/printable_templates/25x25_A4_trading_card_2sided.pdf)
* [29x29 - stores 24-word seeds ONLY as plaintext SeedQR format ONLY](docs/seed_qr/printable_templates/29x29_A4_trading_card_2sided.pdf)

Letter templates(8.5in * 11in):
* [21x21 - stores 12-word seeds ONLY in CompactSeedQR format ONLY](docs/seed_qr/printable_templates/21x21_letter_trading_card_2sided.pdf)
* [25x25 - stores 12-word or 24 word seeds depending on format](docs/seed_qr/printable_templates/25x25_letter_trading_card_2sided.pdf)
* [29x29 - stores 24-word seeds ONLY as plaintext SeedQR format ONLY](docs/seed_qr/printable_templates/29x29_letter_trading_card_2sided.pdf)
---------------

# Build from Source
See the [SeedSigner OS repo](https://github.com/SeedSigner/seedsigner-os/) for instructions.

# Developer Local Build Instructions
Raspberry Pi OS is commonly used for development. See the [Raspberry Pi OS Build Instructions](docs/raspberry_pi_os_build_instructions.md)


#
OPTIMADE Python tools

Latest release	Build status	Activity

The aim of [OPTIMADE](https://optimade.org) is to develop a common API, compliant with the [JSON:API 1.0](http://jsonapi.org/format/1.0/) specification.
This is to enable interoperability among databases that serve crystal structures and calculated properties of existing and hypothetical materials.

This repository contains a library of tools for implementing and consuming [OPTIMADE APIs](https://www.optimade.org) using Python:

1. [pydantic](https://github.com/pydantic/pydantic) data models for all [OPTIMADE entry types](https://www.optimade.org/optimade-python-tools/latest/all_models/) and endpoint responses, and a [Lark](https://github.com/lark-parser/lark) [EBNF grammar](https://en.wikipedia.org/wiki/Extended_Backus%E2%80%93Naur_form) implementation for the OPTIMADE filter language.
1. Adapters to map OPTIMADE data to and from many commonly used atomistic Python frameworks (e.g., [pymatgen](https://pymatgen.org/), [ASE](https://wiki.fysik.dtu.dk/ase/)) and crystallographic file types (e.g., [CIF](https://www.iucr.org/resources/cif)), using the `optimade.adapters` module.
1. A configurable reference server implementation that can make use of either MongoDB or Elasticsearch database backends out-of-the-box, and is readily extensible to other backends. Try it out on the [demo site](https://optimade.fly.dev)! The OpenAPI schemas of the server are used to construct the [OPTIMADE schemas](https://schemas.optimade.org/) site.
1. An [OPTIMADE client](https://www.optimade.org/optimade-python-tools/latest/getting_started/client/) (`optimade-get`) that can query multiple [OPTIMADE providers](https://optimade.org/providers-dashboard) concurrently with a given filter, at the command-line or from Python code.
1. A fuzzy API validator tool, which may be called from the shell (`optimade-validator`) or used as a GitHub Action from [optimade-validator-action](https://github.com/Materials-Consortia/optimade-validator-action); this validator is used to construct the [providers dashboard](https://optimade.org/providers-dashboard).

## Documentation

This document, guides, and the full module API documentation can be found online at [https://optimade.org/optimade-python-tools](https://optimade.org/optimade-python-tools).
In particular, documentation of the OPTIMADE API response data models (implemented here with [pydantic](https://github.com/pydantic/pydantic)) can be found online under [OPTIMADE Data Models](https://optimade.org/optimade-python-tools/latest/all_models).

The release history and changelog can be found in [the changelog](CHANGELOG.md).

## Installation

Detailed installation instructions for different use cases (e.g., using the library or running a server) can be found in [the installation documentation](INSTALL.md).

The latest stable version of this package can be obtained from [PyPI](https://pypi.org/project/optimade):

```shell
pip install optimade
```

The latest development version of this package can be obtained from the master branch of this repository:

```shell
git clone https://github.com/Materials-Consortia/optimade-python-tools
```

## Supported OPTIMADE versions

Each release of the `optimade` package from this repository only targets one version of the OPTIMADE specification, summarised in the table below.

        OPTIMADE API version
        optimade requirements

        v1.0.0
        optimade<=0.12.9

        v1.1.0
        optimade~=0.16

## Contributing and Getting Help

All development of this package (bug reports, suggestions, feedback and pull requests) occurs in the [optimade-python-tools GitHub repository](https://github.com/Materials-Consortia/optimade-python-tools).
Contribution guidelines and tips for getting help can be found in the [contributing notes](CONTRIBUTING.md).

## How to cite

If you use this package to access or serve OPTIMADE data, we kindly request that you consider citing the following:

- Andersen *et al.*, OPTIMADE, an API for exchanging materials data, *Sci. Data* **8**, 217 (2021) [10.1038/s41597-021-00974-z](https://doi.org/10.1038/s41597-021-00974-z)
- Evans *et al.*, optimade-python-tools: a Python library for serving and consuming materials data via OPTIMADE APIs. *Journal of Open Source Software*, **6**(65), 3458 (2021) [10.21105/joss.03458](https://doi.org/10.21105/joss.03458)

## Links

- [OPTIMADE Specification](https://github.com/Materials-Consortia/OPTIMADE/blob/develop/optimade.rst), the human-readable specification that this library is based on.
- [optimade-validator-action](https://github.com/Materials-Consortia/optimade-validator-action), a GitHub action that can be used to validate implementations from a URL (using the validator from this repo).
- [OpenAPI](https://github.com/OAI/OpenAPI-Specification), the machine-readable format used to specify the OPTIMADE API in [`openapi.json`](openapi/openapi.json) and [`index_openapi.json`](openapi/index_openapi.json).
- [Interactive documentation](https://petstore.swagger.io/?url=https://raw.githubusercontent.com/Materials-Consortia/optimade-python-tools/master/openapi/openapi.json) generated from [`openapi.json`](openapi/openapi.json) (see also [interactive JSON editor](https://editor.swagger.io/?url=https://raw.githubusercontent.com/Materials-Consortia/optimade-python-tools/master/openapi/openapi.json)).
- [pydantic](https://pydantic-docs.helpmanual.io/), the library used for generating the OpenAPI schema from [Python models](https://www.optimade.org/optimade-python-tools/all_models/).
- [FastAPI](https://fastapi.tiangolo.com/), the framework used for generating the reference implementation expressed by the [`openapi.json`](openapi/openapi.json) specification.
- [Lark](https://github.com/lark-parser/lark), the library used to parse the filter language in OPTIMADE queries.


# Aries Agent Test Harness: Smashing Complexity in Interoperability Testing<!-- omit in toc -->

[![License](https://img.shields.io/badge/License-Apache%202.0-blue.svg)](LICENSE)

The Aries Agent Test Harness (AATH) is a [BDD](https://en.wikipedia.org/wiki/Behavior-driven_development)-based test execution engine and set of tests for evaluating the interoperability of Aries Agents and Agent Frameworks. The tests are agnostic to the components under test but rather are designed based on the [Aries RFCs](https://github.com/hyperledger/aries) and the interaction protocols documented there. The AATH enables the creation of an interop lab much like the [labs](https://www.iol.unh.edu/) used by the telcos when introducing new hardware into the markets—routers, switchers and the like. Aries agent and agent framework builders can easily incorporate these tests into the their CI/CD pipelines to ensure that interoperability is core to the development process.

Want to see the Aries Agent Test Harness in action? Give it a try using a git, docker and bash enabled system. Once you are in a bash shell, run the following commands to execute a set of RFC tests using the [Aries Cloud Agent - Python](https://github.com/hyperledger/aries-cloudagent-python):

```bash
git clone https://github.com/hyperledger/aries-agent-test-harness
cd aries-agent-test-harness
./manage build -a acapy -a javascript
./manage run -d acapy -b javascript -t @AcceptanceTest -t ~@wip

```

The commands take a while to run (you know...building modern apps always means downloading half the internet...), so while you wait, here's what's happening:

- The AATH `./manage build` command builds Test Agent docker images for the Aries Cloud Agent Python (ACA-Py) and Aries Framework JavaScript (AFJ) agent frameworks and the test harness.
- The AATH `./manage run` command executes a set of tests (those tagged "AcceptanceTest" but not tagged "@wip") with the ACA-Py test agent playing most of the roles—Acme, Faber and Mallory, while the AFJ test agent plays the role of Bob.

It's that last part makes the AATH powerful. On every run, different AATH-enabled components can be assigned any role (Acme, Bob, Faber, Mallory). For some initial pain (AATH-enabling a component), interoperability testing becomes routine, and we can make hit our goal: to make interoperability boring.

Interesting to you? Read on for more about the architecture, how to build tests, how to AATH-enable the Aries agents and agent frameworks that you are building and how you can run these tests on a continuous basis. For a brief set of slides covering the process and goals, check [this](https://docs.google.com/presentation/d/1tNttxlHE8iwyOr7LDOZ6VrCwh8AuINfzuI2Gl0WEld0/edit?usp=sharing) out.

We'd love to have help in building out a full Aries interoperability lab.

## Contents<!-- omit in toc -->

- [Architecture](#architecture)
- [Aries Agent Test Harness Terminology](#aries-agent-test-harness-terminology)
- [Test Script Guidelines](#test-script-guidelines)
- [Aries Agent Backchannels](#aries-agent-backchannels)
  - [Implemented Backchannels](#implemented-backchannels)
- [The `manage` bash script](#the-manage-bash-script)
- [Using AATH Agents as Services](#using-aath-agents-as-services)
  - [Use Cases](#use-cases)
    - [Debugging within AATH](#debugging-within-aath)
    - [Aries Mobile Test Harness](#aries-mobile-test-harness)
- [Extra Backchannel-Specific Parameters](#extra-backchannel-specific-parameters)
- [Custom Configurations for Agents](#custom-configurations-for-agents)
  - [Use Cases](#use-cases-1)
    - [Aries Mobile Test Harness](#aries-mobile-test-harness-1)
- [Test Tags](#test-tags)
  - [Running Tagged Tests](#running-tagged-tests)
  - [Test Coverage](#test-coverage)
  - [Test Reporting](#test-reporting)
- [Adding Runsets](#adding-runsets)

## Architecture

The following diagram provides an overview of the architecture of the AATH.

![Aries Agent Test Harness Architecture](docs/assets/aath-arch/aath-arch.png)

- All of the executable elements run in "Test Agent" docker containers.
- The components-under-test (e.g. Acme, Bob and Mallory) are externally identical, enabling the selection of the components to be tested a runtime decision.
  - An identical HTTP interface exists between the test harness and each test docker container.
  - A "backchannel" within each test container handles the commands from the test harness, converting them into actions understood by the agent or agent framework.
    - This is the hard part—creating a backchannel for each agents/agent frameworks
- The general roles of the test participants are:
  - Acme: an enterprise issuer/verifier
  - Bob: a holder/prover
  - Faber: another enterprise issuer/verifier
  - Mallory: a malicious holder/prover
  - Other roles could be added, perhaps Carol (another holder/prover) and Device (an IOT device).
- The test harness is the Python [behave](https://behave.readthedocs.io/en/latest/) engine, with the "features" (as they are called in BDD-speak) written in [Gherkin](https://behave.readthedocs.io/en/latest/gherkin.html#gherkin-feature-testing-language) and the feature steps in Python.
- Dockerfiles are used to package the backchannel and the component under test (an agent or agent framework) into a single container image with a pre-defined set of port mappings.
  - One of the ports is for the test-harness-to-backchannel interface, while the rest can be used as needed by the component under test and it's backchannel.
- At test execution time, command line parameters enable the selection of any of the docker images to be used to play any of the test case roles. Further, a flexible test case tagging mechanism enables command line (or configuration file) selection of the tests cases to be run.
  - Since the components being tested will have different capabilities (e.g., issuer, prover, verifier), they may only be selected to play certain roles. For example, a mobile wallet app might only play the roles of Bob and Mallory, and never be selected to play the role of Acme.
  - Likewise, only some tests will be selected for execution based on the interoperable capabilities of the components being tested, for example, a component might only support AIP 2.0.
- A bash script (`./manage`) processes the command line options and orchestrates the docker image building and test case running.
- The `./manage` script also supports running the services needed by the tests, such as a [von-network](https://github.com/bcgov/von-network) Indy instance, an [Indy tails service](https://github.com/bcgov/indy-tails-server), a universal resolver and a `did:orb` instance.
  - Environment variables can also be used to configure a test run to use public services, such as the BCovrin test Indy instance. You'll find examples of using environment variables to use those services in various documentation files in this repo.
- A special Test Agent called `mobile` can be used in the `Bob` role to test mobile wallet apps on phones. See [this document](./MOBILE_AGENT_TESTING.md) for details.

## Aries Agent Test Harness Terminology

There are a couple of layers of abstraction involved in the test harness architecture, and it's worth formalizing some terminology to make it easier to communicate about what's what when we are are running tests.

- **Test Harness**: An engine that executes the selected tests and collects the results.
- **Backchannel**: A piece of integration software that receives HTTP requests from the Test Harness and interacts with the agent or agent framework under test to execute the requests. A backchannel is needed for each agent or agent framework. The same backchannel will likely work for multiple agent and agent framework versions and/or configurations.
- **Components under Test (CUTs)**: external Aries agents or agent frameworks being tested.
  - The CUT might be an agent framework like aries-framework-dotnet or aries-cloudagent-python. In those cases, the backchannel is a controller that directly operates an instance of the agent framework.
  - The CUT might be a full agent; a combination of an agent framework instance and a controller. In those cases, the backchannel must be able to operate the controller, which in turn controls the agent framework. For example, a mobile agent has a user interface (a controller), perhaps with some automation rules. Its' AATH backchannel will interact with the controller, not the agent framework directly.
  - There may be multiple CUTs that operate with a single backchannel. For example, different configurations of an agent framework or different versions.
- **Test Agent (TA)**: The combined instance of a CUT and a backchannel, instantiated as a single docker image.

## Test Script Guidelines

AATH test scripts are written in the [Gherkin](https://behave.readthedocs.io/en/latest/gherkin.html#gherkin-feature-testing-language) language, using the python [behave](https://behave.readthedocs.io/en/latest/) framework. Guidelines for writing test scripts are located [here](./TEST_DEV_GUIDE.md).

## Aries Agent Backchannels

Backchannels are the challenging part of the AATH. In order to participate in the interoperability testing, each CUT builder must create and maintain a backchannel that converts requests from the test harness into commands for the component under test. In some cases, that's relatively easy, such as with [Aries Cloud Agent - Python](https://github.com/hyperledger/aries-cloudagent-python). An ACA-Py controller uses an HTTP interface to control an ACA-Py instance, so the ACA-Py backchannel is "just another" ACA-Py controller. In other cases, it may be more difficult, calling for the component under test to be embedded into a web service.

We have created a proof-of-concept Test Agent to support manual testing with mobile agents, described [here](./MOBILE_AGENT_TESTING.md).

A further complication is that as tests are added to the test suite, the backchannel interface expands, requiring that backchannel maintainers extend their implementation to be able to run the new tests. Note that the test engine doesn't stop if the backchannel steps are not implemented, however, such tests will be marked as `fail`ing on test runs, usually with an HTTP `404` error.

### Implemented Backchannels

Backchannels can be found in the [`aries-backchannels`](aries-backchannels) folder of this repo. For more information on building a backchannel, see the documentation in the [`aries-backchannels` README](aries-backchannels/README.md), and look at the code of the existing backchannels. To get help in building a backchannel for a component you want tested, please use GitHub issues and/or ask questions on the [Hyperledger Discord](https://discord.gg/hyperledger) `#aries-agent-test-harness` channel.

Three backchannels have been implemented, for the [ACA-PY](https://github.com/hyperledger/aries-cloudagent-python), [AriesVCX](https://github.com/hyperledger/aries-vcx), [.NET](https://github.com/hyperledger/aries-framework-dotnet) and [JavaScript](https://github.com/hyperledger/aries-framework-javascript.git) Aries agent frameworks. The ACA-Py is built on a common Python base (./aries-backchannels/python/aries_backchannel.py) that sets up the backchannel API listener and performs some basic request validation and dispatching. On the other hand AriesVCX is build on their preferred language (Rust). The ACA-PY (./aries-backchannels/acapy/acapy_backchannel.py) and AriesVCX (./aries-backchannels/aries-vcx) implementations are good example to extend the base to add support for their respective agent frameworks.

There is also a backchannel to support (manual) testing with [mobile](./aries-backcgannels/mobile) agents. This backchannel doesn't control the mobile agent directly, rather it will prompt the tester to manually accept connection requests, credential offers etc. Use of the mobile backchannel is described [here](./MOBILE_AGENT_TESTING.md).

## The `manage` bash script

The AATH `./manage` script in the repo root folder is used to manage running builds of TA images and initiate test runs. Running the script with no arguments or just `help` to see the script's usage information. The following summarizes the key concepts.

`./manage` is a bash script, so you must be in a bash compatible shell to run the AATH. You must also have an operational docker installation and git installed. Pretty normal stuff for Aries Agent development. As well, the current AATH requires access to a running Indy network. A locally running instance of [VON-Network](https://github.com/bcgov/von-network) is one option, but you can also pass in environment variables for the LEDGER_URL, GENESIS_URL or GENESIS_FILE to use a remote network. For example `LEDGER_URL_CONFIG=http://test.bcovrin.vonx.io`

Before running tests, you must build the TA and harness docker images. Use `./manage build -a <TA>` to build the docker images for a TA, and the test harness itself. You may specify multiple `-a` parameters to build multiple TAs at the same time. Leaving off the `-a` option builds docker images for all of the TAs found in the repo. It takes a long time to run...

There are two options for testing [ACA-PY](https://github.com/hyperledger/aries-cloudagent-python) - you can build and run `acapy`, which builds the backchannel based on the latest released code, or you can build and run `acapy-main`, which builds the backchannel based on the latest version of the `main` branch. (Note that to build the backchannel based on a different repo/branch, edit [this file](https://github.com/hyperledger/aries-agent-test-harness/blob/main/aries-backchannels/acapy/requirements-main.txt) to specify the repo/branch you want to test, and then build/run `acapy-main`.)

To run the tests, use the `./manage run...` sub-command. the `run` command requires defining what TAs will be used for Acme (`-a <TA>`), Bob (`-b <TA>`) and Mallory (`-m <TA>`). To default all the agents to use a single component, use `-d <TA>`. Parameters are processed in order, so you can use `-d` to default the agents to one, and then use `-b` to use a different TA for Bob.

There are two ways to control the behave test engine's selection of test cases to run. First, you can specify one or more `-t <tag>` options to select the tests associated with specific tags. See the guidance on using tags with behave [here](https://behave.readthedocs.io/en/stable/behave.html#tag-expression). Note that each `-t` option is passed to behave as a `--tags <tag>` parameter, enabling control of the ANDs and ORs handling of tags. Specifically, each separate `-t` option is ANDed with the rest of the `-t` options. To OR tags, use a single `-t` option with commas (`,`) between the tags. For example, specify the options `-t @t1,@t2 -t @f1` means to use "tests tagged with `(t1 or t2) AND f1`." To get a full list of possible tags to use in this run command, use the `./manage tags` command.

> Note that the `<tag>` arguments passed in on the command line **cannot** have a space, even if you double-quote the tag or escape the space. This is because the args are going through multiple layers shells (the script, calling docker, calling a script in the docker instance that in turn calls behave...). In all that argument passing, the wrappers around the args get lost. That should be OK in most cases, but if it is a problem, we have the `-i` option as follows...

To enable full control over behave's behaviour (if you will...), the `-i <ini file>` option can be used to pass a behave "ini" format file into the test harness container. The ini file enables full control over the behave engine, add handles the shortcoming of not being able to pass tags arguments with spaces in them. See the behave configuration file options [here](https://behave.readthedocs.io/en/stable/behave.html#configuration-files). Note that the file name can be whatever you want. When it lands in the test harness container, it will be called `behave.ini`. There is a default ini file located in `aries-agent-test-harness/aries-test-harness/behave.ini`. This ini file is picked up and used by the test harness without the -i option. To run the tests with a custom behave ini file, follow this example,

```bash
./manage run -d acapy -t @AcceptanceTest -t ~@wip -i aries-test-harness/MyNewBehaveConfig.ini
```

For a full inventory of tests available to run, use the `./manage tests`. Note that tests in the list tagged @wip are works in progress and should generally not be run.

## Using AATH Agents as Services
You may have the need to utilize the agents and their controller/backchannels separately from running interop tests with them. This can be for debugging AATH test code, or for something outside of AATH, like Aries Mobile Test Harness (AMTH) tests. To assist in this requirement the manage script can start 1-n agents of any aries framework that exists in AATH. This is done as follows:

```
./manage start -a acapy-main
```
The command above will only start Acme as ACA-py. No other agents (Bob, Faber, etc.) will be started.
```
NGROK_AUTHTOKEN=2ZrwpFakeAuthToken_W4VDBxavAzdB5K3wsDGz LEDGER_URL_CONFIG=http://test.bcovrin.vonx.io TAILS_SERVER_URL_CONFIG=https://tails.vonx.io AGENT_CONFIG_FILE=/aries-backchannels/acapy/auto_issuer_config.yaml ./manage start -a afgo-interop -b acapy-main -n
```
The second command above, will start Acme as AFGO, and Bob as ACA-py, utilizing an external Ledger and Tails Server, with a custom configuration to start ACA-py with. It will also start ngrok which is usually needed for mobile testing in AMTH.

To stop any agents started in this manner just run `./manage stop`.

### Use Cases
#### Debugging within AATH
When running test code in a debugger, you may not always want or need all the agents running when doing your debugging. Your test may only utilize Acme and Bob, and have no need for Faber and Mallory. This feature will allow you to start only the agents needed in your test you are debugging. The following example will run ACA-py as Acme and Bob with no other agents running.
```
./manage start -a acapy-main -b acapy-main
```
#### Aries Mobile Test Harness
Aries Mobile Test Harness (AMTH) is a testing stack used to test mobile Aries wallets. To do this end to end, mobile tests need issuers, verifiers, and maybe mediators. Instead of AMTH managing a set of controllers and agents, AMTH can point to an Issuer or Verifier controller/agent URL. AMTH can take advantage of the work done across aries frameworks and backchannels to assign AATH agents as issuers or verifiers in testing aries wallets. For example, the BC Wallet tests in AMTH are utilizing ACA-py agents in AATH as an issuer and verifier. This is done by executing the following.
From within aries-agent-test-harness
```
./manage start -a acapy-main -b acapy-main
```
From within aries-mobile-test-harness
```
LEDGER_URL_CONFIG=http://test.bcovrin.vonx.io REGION=us-west-1 ./manage run -d SauceLabs -u <device-cloud-username> -k <device-cloud-access-key> -p iOS -a AriesBifold-114.ipa -i http://0.0.0.0:9020 -v http://0.0.0.0:9030 -t @bc_wallet -t @T001-Connect
```
The URLs for issuer and verifier are pointers to the backchannel controllers for Acme and Bob in AATH, so that these test take advantage of the work done there.

## Extra Backchannel-Specific Parameters

You can pass backchannel-specific parameters as follows:

```bash
BACKCHANNEL_EXTRA_acapy_main="{\"wallet-type\":\"indy\"}" ./manage run -d acapy-main -t @AcceptanceTest -t ~@wip
```

The environment variable name is of the format `-<agent_name>`, where `<agent_name>` is the name of the agent (e.g. `acapy-main`) with hyphens replaced with underscores (i.e. `acapy_main`).

The contents of the environment variable are backchannel-specific. For aca-py it is a JSON structure containing parameters to use for agent startup.

The above example runs all the tests using the `indy` wallet type (vs `askar`, which is the default).

## Custom Configurations for Agents
  Alternatively to the Extra Backchannel-Specific Parameters above, you can also pass a configuration file through to your agent when it starts (only works if your agent is started by your backchannel). The AATH tests have a predefined set of options needed for the test flow to function properly so, adding this configuration to AATH test execution may have side effects causing the interop tests to fail. However, this is helpful when using the agents as services outside of AATH tests like with Mobile Wallet tests in Aries Mobile Test Harness, where the agents usually will benefit from having auto options turned on. You can pass through your config file using the environment variable AGENT_CONFIG_FILE as follows:
  ```
  NGROK_AUTHTOKEN=2ZrwpFakeAuthToken_W4VDBxavAzdB5K3wsDGz LEDGER_URL_CONFIG=http://test.bcovrin.vonx.io TAILS_SERVER_URL_CONFIG=https://tails.vonx.io AGENT_CONFIG_FILE=/aries-backchannels/acapy/auto_issuer_config.yaml ./manage start -b acapy-main -n
  ```
The config file should live in the `aries-backchannels/<agent>` folder so it gets copied into the agent container automatically. Currently only the acapy backchannel supports this custom configuration in this manner.

### Use Cases
#### Aries Mobile Test Harness
When using AATH agents as a service for AMTH, these agent services will need to be started with differet or extra parameters on the agents than AATH starts them with by default. Mobile test issuers and verifiers may need the auto parameters turned on, like `--auto-accept-requests`, `--auto-respond-credential-proposal`, etc. The only way to do this when using the AATH agents is through using this configuration file handling. There is an existing file in `aries-backchannels/acapy` called auto_isser_config.yaml that is there to support this requirement for the BC wallet. This works in BC Wallet as follows;
From within aries-agent-test-harness
```
NGROK_AUTHTOKEN=2ZrwpFakeAuthToken_W4VDBxavAzdB5K3wsDGz LEDGER_URL_CONFIG=http://test.bcovrin.vonx.io TAILS_SERVER_URL_CONFIG=https://tails.vonx.io AGENT_CONFIG_FILE=/aries-backchannels/acapy/auto_issuer_config.yaml ./manage start -a acapy-main -b acapy-main -n
```
From within aries-mobile-test-harness
```
LEDGER_URL_CONFIG=http://test.bcovrin.vonx.io REGION=us-west-1 ./manage run -d SauceLabs -u <device-cloud-username> -k <device-cloud-access-key> -p iOS -a AriesBifold-114.ipa -i http://0.0.0.0:9020 -v http://0.0.0.0:9030 -t @bc_wallet -t @T001-Connect
```

## Test Tags

The test harness has utilized tags in the BDD feature files to be able to narrow down a test set to be executed at runtime. The general AATH tags currently utilized are as follows:

- @AcceptanceTest - Tests based on requirements specifically stated in the RFC.
- @DerivedFunctionalTest - Tests derived on requirements but not specifically stated in the RFC.
- @P1, @P2, @P3, @P4 - Test Priority.
- @NegativeTest - Test that attempts to break the software. ie. change workflow order, use invalid data, etc.
- @ExceptionTest - Tests that are based on requirements that suggest exception cases.
- @SmokeTest - Tests that can be used as a builds smoke or sanity tests.
- @NeedsReview - Tests that have not been reviewed or approved.
- @ReviewedApproved - Tests that have been reviewed and approved.
- @wip - Tests that are a work in progress and incomplete.
- @Done - Finished tests that are expected to Pass if executed against a Test Agent.
- @AIP10, @AIP20 - The Aries Interop Profile version for which the tests are written.
- @MobileTest - The test (likely) works when using the "mobile" Test Agent.
- @T01-AIP10-RFC0160 - Unique Test Identifiers.

Proposed Connection Protocol Tags

- @MultiUseInvite - Test utilizes a multi-use invite. Not using this tag and the test expects the invite to be single use.
- @SingleTryOnException
- @RetryableOnException

To get a list of all the tags in the current test suite, run the command: `./manage tags`

To get a list of the tests (scenarios) and the associated tags, run the command: `./manage tests`

### Running Tagged Tests

Using tags, one can just run Acceptance Tests...

```bash
./manage run -d acapy -t @AcceptanceTest
```

or all Priority 1 Acceptance Tests, but not the ones flagged Work In Progress...

```bash
./manage run -d acapy -t @P1 -t @AcceptanceTest -t ~@wip
```

or derived functional tests

```bash
./manage run -d acapy -t @DerivedFunctionalTest
```

or all the ExceptionTests...

```bash
./manage run -t @ExceptionTest
```

Using AND, OR in Test Execution Tags
Stringing tags together in one `-t` with commas as separators is equivalent to an `OR`. The separate `-t` options is equivalent to an `AND`.

```bash
./manage run -d acapy-main -t @RFC0453,@RFC0454 -t ~@wip -t ~@CredFormat_JSON-LD
```

So the command above will run tests from RFC0453 or RFC0454, without the wip tag, and without the CredFormat_JSON-LD tag.

To read more on how one can control the execution of test sets based on tags see the [behave documentation](https://behave.readthedocs.io/en/stable/tutorial.html#controlling-things-with-tags)

The option `-i <inifile>` can be used to pass a file in the `behave.ini` format into behave. With that, any behave configuration settings can be specified to control how behave behaves. See the behave documentation about the `behave.ini` configuration file [here](https://behave.readthedocs.io/en/stable/behave.html#configuration-files).

### Test Coverage

To read about what protocols and features from Aries Interop Profile 1.0, see the [Test Coverage Matrix](./TEST-COVERAGE.md).

### Test Reporting

For information on enhanced test reporting with the Aries Agent Test Harness, see [Advanced Test Reporting](./TEST_REPORTING.md).

## Adding Runsets

Runsets are GHA based workflows that automate the execution of your interop tests and reporting of the results.

These workflows are contained in the .github/workflows folder and must be named `test-harness-<name>.yml`.  Refer to the existing files for examples on how to create one specific to your use case.  In most cases you will be able to copy an existing file and change a few parameters.

Test execution is controlled by the [`test-harness-runner`](./.github/workflows/test-runner.yml).  This workflow will dynamically pick up and run any workflow conforming to the `test-harness-*.yml` naming convention.  Specific test harnesses can be excluded by adding their file name pattern to the `ignore_files_starts_with` list separated by a `,`.  The test harnesses are run by the **Run Test Harness** job which uses a throttled matrix strategy.  The number of concurrent test harness runs can be controlled by setting the `max-parallel` parameter to an appropriate number.


===========
alfasim-sdk
===========

.. image:: https://img.shields.io/pypi/v/alfasim-sdk.svg
        :target: https://pypi.python.org/pypi/alfasim-sdk

.. image:: https://github.com/ESSS/alfasim-sdk/workflows/ALFAsim%20SDK/badge.svg
    :target: https://github.com/ESSS/alfasim-sdk/actions?query=workflow%3A%22ALFAsim+SDK%22

.. image:: https://codecov.io/gh/ESSS/alfasim-sdk/branch/master/graph/badge.svg
    :target: https://codecov.io/gh/ESSS/alfasim-sdk

.. image:: https://readthedocs.org/projects/alfasim-sdk/badge/?version=latest
    :target: https://alfasim-sdk.readthedocs.io/en/latest/?badge=latest
    :alt: Documentation Status

.. image:: https://img.shields.io/badge/code%20style-black-000000.svg
    :target: https://github.com/python/black

.. image:: https://sonarcloud.io/api/project_badges/measure?project=ESSS_alfasim-sdk&metric=alert_status
    :target: https://sonarcloud.io/project/overview?id=ESSS_alfasim-sdk

ALFAsim API/SDK

* Free software: MIT license
* Documentation: https://alfasim-sdk.readthedocs.io.

Features
--------

* TODO

Credits
-------

This package was created with Cookiecutter_ and the `audreyr/cookiecutter-pypackage`_ project template.

.. _Cookiecutter: https://github.com/audreyr/cookiecutter
.. _`audreyr/cookiecutter-pypackage`: https://github.com/audreyr/cookiecutter-pypackage


    ETA: Extensible Toolkit for Analytics

**An open and extensible computer vision, machine learning and video analytics
infrastructure.**

[![PyPI python](https://img.shields.io/pypi/pyversions/voxel51-eta)](https://pypi.org/project/voxel51-eta)
[![PyPI version](https://badge.fury.io/py/voxel51-eta.svg)](https://pypi.org/project/voxel51-eta)
[![pre-commit](https://img.shields.io/badge/pre--commit-enabled-brightgreen?logo=pre-commit&logoColor=white)](https://github.com/pre-commit/pre-commit)
[![License](https://img.shields.io/badge/License-Apache%202.0-blue.svg)](LICENSE)
[![Twitter](https://img.shields.io/twitter/follow/Voxel51?style=social)](https://twitter.com/voxel51)

## Requirements

ETA is very portable:

-   Installable on Mac or Linux
-   Supports Python 3.6 or later
-   Supports TensorFlow 1.X and 2.X
-   Supports OpenCV 2.4+ and OpenCV 3.0+
-   Supports CPU-only and GPU-enabled installations
-   Supports CUDA 8, 9 and 10 for GPU installations

## Installation

You can install the latest release of ETA via `pip`:

```shell
pip install voxel51-eta
```

This will perform a [lite installation of ETA](#lite-installation). If you use
an ETA feature that requires additional dependencies (e.g., `ffmpeg` or
`tensorflow`), you will be prompted to install the relevant packages.

## Docker Installation

If you prefer to operate via Docker, see the
[Docker Build Guide](https://github.com/voxel51/eta/blob/develop/docs/docker_build_guide.md)
for simple instructions for building a Docker image with an ETA environment
installed.

## Installation from source

#### Step 0: Setup your Python environment

It is assumed that you already have
[Python installed](https://www.python.org/downloads) on your machine.

> **IMPORTANT:** ETA assumes that the version of Python that you intend to use
> is accessible via `python` and `pip` on your path. In particular, for Python
> 3 users, this means that you may need to alias `python3` and `pip3` to
> `python` and `pip`, respectively.

We strongly recommend that you install ETA
[in a virtual environment](https://github.com/voxel51/eta/blob/develop/docs/virtualenv_guide.md)
to maintain a clean workspace.

#### Step 1: Clone the repository

```shell
git clone https://github.com/voxel51/eta
cd eta
```

#### Step 2: Run the install script

```shell
bash install.bash
```

Note that the install script supports flags that control things like (on macOS)
whether `port` or `brew` is used to install packages. Run
`bash install.bash -h` for more information.

For Linux installs, the script inspects your system to see if CUDA is installed
via the `lspci` command. If CUDA is available, TensorFlow is installed with GPU
support.

The table below lists the version of TensorFlow that will be installed by the
installer, as recommended by the
[tested build configurations](https://www.tensorflow.org/install/source#tested_build_configurations):

| CUDA Version Found | TensorFlow Version Installed |
| ------------------ | ---------------------------- |
| CUDA 8             | `tensorflow-gpu~=1.4`        |
| CUDA 9             | `tensorflow-gpu~=1.12`       |
| CUDA 10            | `tensorflow-gpu~=1.15`       |
| Other CUDA         | `tensorflow-gpu~=1.15`       |
| No CUDA            | `tensorflow~=1.15`           |

> Note that ETA also supports TensorFlow 2.X. The only problems you may face
> when using ETA with TensorFlow 2 are when trying to run inference with
> [ETA models](https://github.com/voxel51/eta/blob/develop/eta/models/manifest.json)
> that only support TensorFlow 1. A notable case here are TF-slim models. In
> such cases, you should see an informative error message alerting you of the
> requirement mismatch.

### Lite installation

Some ETA users are only interested in using the core ETA library defined in the
`eta.core` package. In such cases, you can perform a lite installation using
the `-l` flag of the install script:

```shell
bash install.bash -l
```

Lite installation omits submodules and other large dependencies that are not
required in order for the core library to function. If you use an ETA feature
that requires additional dependencies (e.g., `ffmpeg` or `tensorflow`), you
will be prompted to install the relevant packages.

### Developer installation

If you are interested in contributing to ETA or generating its documentation
from source, you should perform a developer installation using the `-d` flag of
the install script:

```shell
bash install.bash -d
```

## Setting up your execution environment

When the root `eta` package is imported, it tries to read the `eta/config.json`
file to configure various package-level constants. Many advanced ETA features
such as pipeline building, model management, etc. require a properly configured
environment to function.

To setup your environment, create a copy the example configuration file:

```shell
cp config-example.json eta/config.json
```

If desired, you can edit your config file to customize the various paths,
change default constants, add environment variables, customize your default
`PYTHONPATH`, and so on. You can also add additional paths to the
`module_dirs`, `pipeline_dirs`, and `models_dirs` sections to expose custom
modules, pipelines, and models to your system.

Note that, when the config file is loaded, any `{{eta}}` patterns in directory
paths are replaced with the absolute path to the `eta/` directory on your
machine.

The default config includes the `modules/`, `pipelines/`, and `models/`
directories on your module, pipeline, and models search paths, respectively.
These directories contain the necessary information to run the standard
analytics exposed by the ETA library. In addition, the relative paths
`./modules/`, `./pipelines/`, and `./models/` are added to their respective
paths to support the typical directory structure that we adopt for our custom
projects.

### CLI

Installing ETA automatically installs `eta`, a command-line interface (CLI) for
interacting with the ETA Library. This utility provides access to many useful
features of ETA, including building and running pipelines, downloading models,
and interacting with remote storage.

To explore the CLI, type `eta --help`, and see the
[CLI Guide](https://github.com/voxel51/eta/blob/develop/docs/cli_guide.md) for
complete information.

## Quickstart

Get your feet wet with ETA by running some of examples in the
[examples folder](https://github.com/voxel51/eta/tree/develop/eta/examples).

Also, see the [docs folder](https://github.com/voxel51/eta/tree/develop/docs)
for more documentation about the various components of the ETA library.

## Organization

The ETA package is organized as described below. For more information about the
design and function of the various ETA components, read the documentation in
the [docs folder](https://github.com/voxel51/eta/tree/develop/docs).

| Directory         | Description                                                                                |
| ----------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `eta/classifiers` | wrappers for performing inference with common classifiers                                                                                |
| `eta/core`        | the core ETA library, which includes utilities for working with images, videos, embeddings, and much more                                                                                |
| `eta/detectors`   | wrappers for performing inference with common detectors                                                                                |
| `eta/docs`        | documentation about the ETA library                                                                                |
| `eta/examples`    | examples of using the ETA library                                                                                |
| `eta/models`      | library of ML models. The `manifest.json` file in this folder enumerates the models, which are downloaded to this folder as needed. See the [Models developer's guide](https://github.com/voxel51/eta/blob/develop/docs/models_dev_guide.md) for more information about ETA's model registry |
| `eta/modules`     | library of video processing/analytics modules. See the [Module developer's guide](https://github.com/voxel51/eta/blob/develop/docs/modules_dev_guide.md) for more information about ETA modules                                                                                |
| `eta/pipelines`   | library of video processing/analytics pipelines. See the [Pipeline developer's guide](https://github.com/voxel51/eta/blob/develop/docs/pipelines_dev_guide.md) for more information about ETA pipelines                                                                                |
| `eta/resources`   | resources such as media, templates, etc                                                                                |
| `eta/segmenters`  | wrappers for performing inference with common semantic segmenters                                                                                |
| `eta/tensorflow`  | third-party TensorFlow repositories that ETA builds upon                                                                                |

## Generating Documentation

This project uses
[Sphinx-Napoleon](https://pypi.python.org/pypi/sphinxcontrib-napoleon) to
generate its documentation from source.

To generate the documentation, you must install the developer dependencies by
running the `install.bash` script with the `-d` flag.

Then you can generate the docs by running:

```shell
bash sphinx/generate_docs.bash
```

To view the documentation, open the `sphinx/build/html/index.html` file in your
browser.

## Uninstallation

```shell
pip uninstall voxel51-eta
```

## Acknowledgements

This project was gratefully supported by the
[NIST Public Safety Innovation Accelerator Program](https://www.nist.gov/news-events/news/2017/06/nist-awards-385-million-accelerate-public-safety-communications).

## Citation

If you use ETA in your research, feel free to cite the project (but only if you
love it 😊):

```bibtex
@article{moore2017eta,
  title={ETA: Extensible Toolkit for Analytics},
  author={Moore, B. E. and Corso, J. J.},
  journal={GitHub. Note: https://github.com/voxel51/eta},
  year={2017}
}
```


# GCR Catalogs

[![Conda Version](https://img.shields.io/conda/vn/conda-forge/lsstdesc-gcr-catalogs.svg)](https://anaconda.org/conda-forge/lsstdesc-gcr-catalogs)
[![doi:10.3847/1538-4365/aaa6c3](https://img.shields.io/badge/Paper%20DOI-10.3847%2F1538--4365%2Faaa6c3-brightgreen.svg)](https://doi.org/10.3847/1538-4365/aaa6c3)
[![arXiv:1709.09665](https://img.shields.io/badge/astro--ph.IM-arXiv%3A1709.09665-B31B1B.svg)](https://arxiv.org/abs/1709.09665)

`GCRCatalogs` is a Python package that serves as a repository of various
galaxy catalogs and sky catalogs for the LSST Dark Energy Science Collaboration (DESC).
It provides a unified user interface to access all catalogs by using
the [Generic Catalog Reader (GCR)](https://github.com/yymao/generic-catalog-reader) base class.

This package is also used by the [DESCQA](https://github.com/LSSTDESC/descqa)
validation framework, and the concept and description of this reader interface
can be found in the [DESCQA paper](https://doi.org/10.3847/1538-4365/aaa6c3)
and also the [GCR repo](https://github.com/yymao/generic-catalog-reader).

The instructions below are intended for DESC members.
**If you are using public DESC data sets with `GCRCatalogs`, please follow the instructions on the [DESC Data Portal: data.lsstdesc.org](https://data.lsstdesc.org).**

## Available Catalogs

Below is a list of most-used catalogs. To find a complete, most up-to-date list of all available catalogs,
run the following code:

```python
import GCRCatalogs

# List all catalogs that are recommended for general consumption
GCRCatalogs.get_available_catalog_names()

# List all catalogs, including those may not be intended for general consumption
GCRCatalogs.get_available_catalog_names(include_default_only=False)

# List all catalogs whose names contain the word "object"
GCRCatalogs.get_available_catalog_names(include_default_only=False, name_contains="object")

# List all catalogs whose names start with the word "buzzard"
GCRCatalogs.get_available_catalog_names(include_default_only=False, name_startswith="buzzard")
```

(*Note*: remove `False` in the above line to only see recommended catalogs.)

Each catalog is specified by a YAML config file,
which can be found [here](GCRCatalogs/catalog_configs).

You can also find an overview and more detailed description of all the data
products of DESC Data Challenge 2 at the
["DC2 Data Product Overview"](https://confluence.slac.stanford.edu/x/oJgHDg)
Confluence page (*DESC member only*).

### Extragalactic Catalogs and Add-ons

#### DiffSky Series
*by Andrew Hearin, Eve Kovacs, Patricia Larsen, Esteban Rangel, Katrin Heitmann et al.*

- `roman_rubin_v1.1.3_elais`: This catalog is a variant of roman_rubin_v1.1.2_elais that uses an improved tuning of the SED model parameters. The resulting luminosity distributions are in better agreement with the validation data (COSMOS 2020 and DESCQA tests).

- `roman_rubin_v1.1.2_elais`: This catalog was produced for the joint roman-desc image simulations using differentiable, forward modeling techniques. Predictions for the galaxy SEDs are based on their star-formation hsitories. Sky area covers the ELAIS field and is ~110 sq. deg.

- `roman_rubin_v1.1.1_elais`: DEPRECATED. This catalog was produced for the joint roman-desc image simulations using differentiable, forward modeling techniques. This catalog is deprecated as it contains a serious bug resulting in satellite galaxies that are too bright.

#### SkySim5000

*by Andrew Hearin, Danila Korytov, Eve Kovacs, Esteban Rangel, Patricia Larsen, Joe Hollowed, Andrew Benson, Katrin Heitmann et al.*

- `skysim5000_v1.2`: latest SkySim5000 catalog, with bug fixes for ellipticity and concentration values; tidal torque fields added, full sky area (~5000 sq. deg.)
  - `skysim5000_v1.2_image`: same but only covers the image simulation sky area (~400 sq. deg.)
  - `skysim5000_v1.2_small`: same but only covers a small  sky area (~50 sq. deg.) for testing purposes
  - `skysim5000_v1.1.2`: preliminary version of new SkySim5000 catalog, full sky area (~5000 sq. deg.) (0<z<1)
  - `skysim5000_v1.1.1`: previous SkySim5000 catalog, full sky area (~5000 sq. deg.)
  - `skysim5000_v1.1.1_image`: same but only covers the image simulation sky area (~400 sq. deg.)
  - `skysim5000_v1.1.1_small`: same but only covers a small  sky area (~50 sq. deg.) for testing purposes
  - `skysim5000_v1.1.1_parquet`: same but using parquet files as the underlying format for better performance; no native quantities
  - `skysim5000_v1.1.1_redmapper_v0.8.5`: Redmapper catalog (v0.8.5) for `skysim5000_v1.1.1` (provided by Eli Rykoff).
  - `skysim5000_v1.1.1_redmagic_v0.8.5_highdens`: Redmagic catalog (v0.8.5) for `skysim5000_v1.1.1` for the high density sample (provided by Eli Rykoff).
  - `skysim5000_v1.1.1_redmagic_v0.8.5_highlum`: Redmagic catalog (v0.8.5) for `skysim5000_v1.1.1` for the high luminosity sample (provided by Eli Rykoff).

#### cosmoDC2

*by Andrew Benson, Andrew Hearin, Katrin Heitmann, Danila Korytov, Eve Kovacs, Patricia Larsen et al.*

- `cosmoDC2_v1.1.4_image`: latest cosmoDC2 catalog (used for Run 2.1+)
- `cosmoDC2_v1.1.4_small`: 17 contiguous healpixels of `cosmoDC2_v1.1.4_image` for testing purposes
- `cosmoDC2_v1.1.4_redmapper_v0.8.1`: Redmapper catalog (v0.8.1) for `cosmoDC2_v1.1.4_image` (provided by Eli Rykoff).
- `cosmoDC2_v1.1.4_redmagic_v0.8.1_highdens`: Redmagic catalog (v0.8.1) for `cosmoDC2_v1.1.4_image` for the high density sample (provided by Eli Rykoff).
- `cosmoDC2_v1.1.4_redmagic_v0.8.1_highlum`: Redmagic catalog (v0.8.1) for `cosmoDC2_v1.1.4_image` for the high luminosity sample (provided by Eli Rykoff).
- `cosmoDC2_v1.1.4_wazp_v1.0_truez`: WaZP catalog (v1.0) for `cosmoDC2_v1.1.4` using true redshifts (provided by Michel Aguena).
- `cosmoDC2_v1.1.4_wazp_v1.0_flexzboost_v1`: WaZP catalog (v1.0) for `cosmoDC2_v1.1.4_image_with_photozs_flexzboost_v1` using FlexZBoost redshifts (provided by Michel Aguena).
- `cosmoDC2_v1.1.4_image_with_photozs_v1` and `cosmoDC2_v1.1.4_small_with_photozs_v1`: containing photo-z for cosmoDC2 v1.1.4 (provided by Sam Schmidt)
- `cosmoDC2_v1.1.4_image_with_photoz_calib` and `cosmoDC2_v1.1.4_small_with_photoz_calib`: containing columns that identify DESI-like QSOs, LRGs, ELGs, or a magnitude limited sample in cosmoDC2 v1.1.4 (provided by Chris Morrison)

#### protoDC2

*by Andrew Benson, Andrew Hearin, Katrin Heitmann, Danila Korytov, Eve Kovacs, Patricia Larsen et al.*

- `protoDC2`: full catalog
- `protoDC2_test`: same as `protoDC2` but this one skips time-consuming md5 check
- `proto-dc2_vX.X_test.yaml`: some other versions of the protoDC2 catalog

#### Buzzard

*by Joe DeRose, Risa Wechsler, Eli Rykoff et al.*

- `buzzard`: full catalog, DES Y3 area
- `buzzard_test`: same as `buzzard` but a small subset for testing purpose / faster access
- `buzzard_high-res`: higher resolution, smaller sky area
- `buzzard_v2.0.0_x`: different realizations of the version of the `buzzard` catalog documented in [arXiv:1901.02401](https://arxiv.org/abs/1901.02401).

### DC2 Runs Data Products and Add-ons

*by LSST DESC, compiled by the DC2 Team*

#### Run 3.1 Truth Catalogs

- `dc2_truth_run3.1i_agn_auxiliary_info`: Useful derived information for AGN.
- `dc2_truth_run3.1i_agn_truth_summary`: AGN truth summary table.
- `dc2_truth_run3.1i_agn_variability_truth`: AGN variable truth information
- `dc2_truth_run3.1i_lensed_agn_truth_summary`: Truth summary table for lensed AGN.
- `dc2_truth_run3.1i_lensed_agn_variability_truth`: Variable truth information for lensed AGN.
- `dc2_truth_run3.1i_lensed_host_truth_summary`: Truth summary table for lensed host galaxies.
- `dc2_truth_run3.1i_lensed_sne_truth_summary`: Truth summary table for lensed SNe.
- `dc2_truth_run3.1i_lensed_sn_variability_truth`: Variable truth information for lensed SNe.

#### Run 2.2 Object Catalogs

##### DR6 (up to Year 5)

- `dc2_object_run2.2i_dr6`: static object catalog for Run 2.2i DR6 (WFD and DDF visits; a small sky region in the upper right corner of the footprint was excluded in the current version (v1))
- `dc2_object_run2.2i_dr6_with_addons`: same as `dc2_object_run2.2i_dr6` but with all available add-on catalogs (currently including metacal and truth-match; photo-z add-on not yet available. Use `dc2_object_run2.2i_dr6a_with_photoz` for a preview of photo-z.)
- `dc2_redmapper_run2.2i_dr6_wfd_v0.8.1`: Redmapper catalog (v0.8.1) for `dc2_object_run2.2i_dr6` (provided by Eli Rykoff).
- `dc2_redmagic_run2.2i_dr6_wfd_v0.8.1_highdens`: Redmagic catalog (v0.8.1) for `dc2_object_run2.2i_dr` for the high density sample (provided by Eli Rykoff).
- `dc2_redmagic_run2.2i_dr6_wfd_v0.8.1_highlum`: Redmagic catalog (v0.8.1) for `dc2_object_run2.2i_dr` for the high luminosity sample (provided by Eli Rykoff).

##### DR2 (up to Year 1)

- `dc2_object_run2.2i_dr2_wfd`: static object catalog for Run2.2i DR2 (WFD visits)
- `dc2_object_run2.2i_dr2_wfd_with_addons`: same as `dc2_object_run2.2i_dr2_wfd` but with all available add-on catalogs
  (currently including only truth-match; metacal, photo-z not yet available)

##### DR3 (up to Year 2)

Note: DR3 processing is not fully completed; a few tracts are missing. Here `dr3a` is a preview of DR3.

- `dc2_object_run2.2i_dr3a`: static object catalog for Run 2.2i DR3 (preview)
- `dc2_object_run2.2i_dr3a_with_metacal`: `dc2_object_run2.2i_dr3a` + metacal (preview; missing more tracts)
- `dc2_object_run2.2i_dr3a_with_photoz`: `dc2_object_run2.2i_dr3a` + photo-z (preview)

#### Run 2.2 Truth Catalogs

- `dc2_run2.2i_truth_merged_summary`: combined truth summary table (galaxies, stars, SNe); partitioned by tract.
- `dc2_run2.2i_truth_galaxy_summary`: galaxy truth summary table, partitioned by healpixel, like cosmoDC2.
- `dc2_run2.2i_truth_sn_summary`: SN truth summary table.
- `dc2_run2.2i_truth_sn_variability`: SN variability truth information.
- `dc2_run2.2i_truth_star_summary`: star truth summary table.
- `dc2_run2.2i_truth_star_variability`: star variability truth information.

#### Run 1.2 Object Catalogs

- `dc2_object_run1.2i`: static object catalog for Run 1.2i (with only DPDD columns and native columns needed for the DPDD columns)
- `dc2_object_run1.2i_with_photoz`: same as `dc2_object_run1.2i` but with photo-z's (columns that start with `photoz_`). Photo-z provided by Sam Schmidt.
- `dc2_object_run1.2i_all_columns`: static object catalog for Run 1.2i (with DPDD and all native columns, slower to access)
- `dc2_object_run1.2i_tract4850`, `dc2_object_run1.2i_tract5063`: same as `dc2_object_run1.2i_all_columns` but only has one tract for testing purpose / faster access
- `dc2_object_run1.2p`: static object catalog for Run 1.2p (with only DPDD columns and native columns needed for the DPDD columns)
- `dc2_object_run1.2p_all_columns`: static object catalog for Run 1.2p (with DPDD and all native columns, slower to access)
- `dc2_object_run1.2p_tract4850`: same as `dc2_object_run1.2p_all_columns` but only has one tract (4850)for testing purpose / faster access

#### Run 1.2 Truth Catalogs

- `dc2_truth_run1.2_static`: truth catalog for Run 1.2 (static objects only, corresponds to `proto-dc2_v3.0`)
- `dc2_truth_run1.2_variable_lightcurve`: light curves of variable objects in the truth catalog for Run 1.2
- `dc2_truth_run1.2_variable_summary`: summary table of variable objects in the truth catalog for Run 1.2

#### Run 1.2 DIA Source Catalogs

- `dc2_dia_source_run1.2p_test`: DIASource Table catalog for a test DIA processing of Tract+Paptch 4849+6,6 for Run 1.2p (with only DPDD columns and native columns needed for the DPDD columns).

#### Run 1.2 Forced Source Catalogs

- `dc2_forced_source_run1.2p`: Forced Source Table catalog for Run 1.2p (with only DPDD columns and native columns needed for the DPDD columns).  This is the forced-position photometry based on the positions in the Object Table.

#### Run 1.2 Source Catalogs

- `dc2_source_run1.2i`: Source Table catalog for Run 1.2i (with only DPDD columns and native columns needed for the DPDD columns)

#### Run 1.2 e-images

- `dc2_eimages_run1.2i_visit-181898`: one visit of e-images for Run 1.2i
- `dc2_eimages_run1.2p_visit-181898`: one visit of e-images for Run 1.2p

## Using GCRCatalogs at NERSC

All catalogs available in `GCRCatalogs` are physically located at NERSC
(most are mirrored at IN2P3-CC as well).
At NERSC, you need to be in the `lsst` user group to access them.
You can find instructions about getting a NERSC account and joining `lsst` group
at [this Confluence page](https://confluence.slac.stanford.edu/x/mgRTD)
(*DESC members only*).

### With Jupyter notebooks

It is recommended that you first install DESC-specific kernels for your
NERSC jupyter environment (*you only need to do this once*).
To do so, log in to cori.nersc.gov and run:
```bash
source /global/common/software/lsst/common/miniconda/kernels/setup.sh
```
Detailed instructions can also be found at [this Confluence page](https://confluence.slac.stanford.edu/x/1_ubDQ)
(*DESC members only*).

Then, you can [start a NERSC notebook server](https://jupyter.nersc.gov)
and open a notebook with the `desc-python` or `desc-stack` kernel.
`GCRCatalogs` and necessary dependencies are already installed in these two kernels.
You can check if it works simply by running:

```python
import GCRCatalogs
```

If you don't have these DESC-specific kernels installed, you can modify
`sys.path` at run time (*not recommended*).
At the very first cell of your notebook, run:

```python
import sys
sys.path.insert(0, '/global/common/software/lsst/common/miniconda/current/lib/python3.6/site-packages')
```

### In a terminal or in a Python script

You can activate DESC Python environment by running the following line on NERSC
(needs to be in `bash` or `zsh`):

```bash
source /global/common/software/lsst/common/miniconda/setup_current_python.sh
```

If you want to use `GCRCatalogs` in a Python script, you can either activate DESC
Python environment before you run the script, or edit the hashbang line of the script to be:

```bash
#!/global/common/software/lsst/common/miniconda/current/envs/stack/bin/python
```

### Using the latest version of GCRCatalogs

If you need to use a newer version of `GCRCatalogs` than the one installed in the DESC Python environment,
here's what you need to do:

1. Clone this repo (on a NERSC machine):
   ```bash
   git clone git@github.com:LSSTDESC/gcr-catalogs.git
   ```
   (Note that if you want to use a PR, you need to clone the corresponding branch.)

2. Add the path to `sys.path` in your notebook.
   ```python
   import sys
   sys.path.insert(0, '/path/to/cloned/gcr-catalogs')
   ```
   (Note that if you use `sys.path` for the DESC Python environment, you should add the line above *right after* you insert the DESC Python environment.)

If you are running DESCQA and want to use your cloned `GCRCatalogs`, you can add the path to `-p` option:
```bash
./run_master.sh -t <tests> -c <catalogs> -p /path/to/cloned/gcr-catalogs
```
See [more instrcutions for DESCQA here](https://github.com/LSSTDESC/descqa/blob/master/CONTRIBUTING.md#master-script-options).

`GCRCatalogs` is also pip-installable, in case you need to install,
say the master branch of `GCRCatalogs` in your own Python environment
(*no, in most cases you don't need this*):
```bash
pip install https://github.com/LSSTDESC/gcr-catalogs/archive/master.zip
```

## Usage and Examples

Here's the very basic usage of `GCRCatalogs`.
Scroll down to see the example notebooks and more advanced usages.

```python
import GCRCatalogs

# see all available catalogs
print(GCRCatalogs.get_available_catalogs(names_only=True))

# load a catalog
catalog = GCRCatalogs.load_catalog('protoDC2')

# load a catalog with runtime custom options
# (one needs to check catalog configs to know the keywords)
catalog = GCRCatalogs.load_catalog('cosmoDC2_v1.1.4_image', config_overwrite={'healpix_pixels': [8786, 8787, 8788]})
catalog = GCRCatalogs.load_catalog('dc2_object_run2.2i_dr6', config_overwrite={'tracts': [3638, 3639, 3640])

# see all available quantities
print(sorted(catalog.list_all_quantities()))

# load quantities
data = catalog.get_quantities(['ra', 'dec'])
```

- You can find quantity definitions in [`GCRCatalogs/SCHEMA.md`](https://github.com/LSSTDESC/gcr-catalogs/blob/master/GCRCatalogs/SCHEMA.md).

- See [this notebook](examples/GCRCatalogs%20Demo.ipynb) for a detailed tutorial on how to use `GCRCatalogs`.

- See [this notebook](examples/CompositeReader%20example.ipynb) for a example of using the composite catalog feature.

- See [this notebook](examples/CLF%20Test.ipynb) for an actual application (the Conditional Luminosity Function test) using GCR Catalogs. (Thanks to Joe DeRose for providing the CLF test example!)

- You can find more tutorial notebooks that use `GCRCatalogs` in [`LSSTDESC/DC2-analysis`](https://github.com/LSSTDESC/DC2-analysis/tree/master/tutorials).

- See also the [GCR documentation](https://yymao.github.io/generic-catalog-reader/) for the complete GCR API.


 .. image:: https://coveralls.io/repos/watchdogpolska/feder/badge.svg?branch=master&service=github
    :target: https://coveralls.io/github/watchdogpolska/feder?branch=master

 .. image:: https://github.com/watchdogpolska/feder/workflows/Python%20package/badge.svg
    :target: https://github.com/watchdogpolska/feder/actions?workflow=Python+package

 .. image:: https://img.shields.io/github/license/watchdogpolska/feder.svg

 .. image:: https://david-dm.org/watchdogpolska/feder/dev-status.svg
     :target: https://david-dm.org/watchdogpolska/feder#info=devDependencies
     :alt: devDependency Status

Feder
=====

Mechanizm do automatycznego wysyłania wniosków o informację do dużej liczby podmiotów, automatycznego przyjmowania odpowiedzi, udostępnienia otrzymanych odpowiedzi do analizy pod wybranym kątem przez masowo angażowanych do tego zadania obywateli oraz  upubliczniania zweryfikowanych odpowiedzi i zestawień uzyskanych danych.

Testowanie w lokalnym środowisku
--------------------------------

Zbudowanie usługi:

.. code-block:: bash

    make build  # zbudowanie wszystkich zależności; może zająć trochę czasu przy pierwszym wywołaniu
    make wait_web  # inicjalizacja lokalnych zależności
    docker-compose up  # inicjalizacja samej aplikacji; po wykonaniu, aplikacja powinna być widoczna w localhost:8000

Inicjalizacja bazy danych:

.. code-block:: bash

    make importterc  # wypełnienie bazy jednostek organizacyjnych
    make createsuperuser  # stworzenie konta administratora

Inicjalizacja jest konieczna przed pierwszym użyciem. W trakcie wywoływania komend aplikacja powinna działać w tle, tj. wcześniej
należy wywołać komendę

.. code-block:: bash

    docker-compose up

Statyczna kontrola kodu
-----------------------

Statyczna kontrola kodu w projekcie służy do sprawdzenia, czy kod spełnia określone standardy jakości.

Wszystkie sprawdzenia kodu statycznego można uruchamiać za pomocą `pre-commit run`_. Są one
także weryfikowane w środowisku `GitHub Actions`_, przy każdej
propozycji zmian zawartości repozytorium.

.. _pre-commit run: https://pre-commit.com/
.. _GitHub Actions: https://github.com/watchdogpolska/feder/actions

W celu instalacji narzędzia do statycznej kontroli kodu, należy wykonać:

.. code-block:: bash

    pip install -r requirements/dev.txt

W celu wykonania kontroli statycznej kodu, należy wykonać:

.. code-block:: bash

    git add .
    make lint

Aby włączyć automatyczne sprawdzanie kodu przed stworzeniem zmiany (*commit*), należy wykonać:

.. code-block:: bash

    pre-commit install


# SahasrahBot

This really wasn't meant to be publically viewable, but since people asked for the code here it is.

This is mostly here for transparency purposes, as this bot is responsible for rolling games for racing.  However, I cannot provide any support with setting this bot up locally or getting a local dev environment going.

Someday I'll migrate to a proper ORM and have a proper database migration setup.   Maybe then it'll be easier to then have feature flags so people can run this on their own.

If there's any feature within this bot that you'd like to use for your own project, feel free to use it within the terms of the license agreement attached.  I'd also be happy to make it a separate module, or add it to pyz3r, as well.

Thanks,

Thomas Prescott, aka Synack

## Installing dependencies

1.  pip install --user virtualenv
2.  pip install poetry
3.  virtualenv .venv
4.  poetry install