# Professional Services

Common solutions and tools developed by Google Cloud's Professional Services
team.

## Disclaimer

This repository and its contents are not an officially supported Google product.

## License

All solutions within this repository are provided under the
[Apache 2.0](https://www.apache.org/licenses/LICENSE-2.0) license. Please see
the [LICENSE](/LICENSE) file for more detailed terms and conditions.

[![Open in Cloud Shell](http://gstatic.com/cloudssh/images/open-btn.svg)](https://console.cloud.google.com/cloudshell/editor?cloudshell_git_repo=https%3A%2F%2Fgithub.com%2FGoogleCloudPlatform%2Fprofessional-services.git)

## Examples

The examples folder contains example solutions across a variety of Google Cloud
Platform products. Use these solutions as a reference for your own or extend
them to fit your particular use case.

*   [Anthos Service Mesh Multi-Cluster](examples/anthos-service-mesh-multicluster) -
    Solution to federate two private GKE clusters using Anthos Service Mesh.
*   [Anthos CICD with Gitlab](examples/anthos-cicd-with-gitlab) - A step-by-step
    guide to create an example CI/CD solution using Anthos and Gitlab.
*   [Audio Content Profiling](examples/ml-audio-content-profiling) - A tool that
    builds a pipeline to scale the process of moderating audio files for
    inappropriate content using machine learning APIs.
*   [Bigdata generator](tools/bigdata-generator/) -
    Solution that generates large amounts of data for stress-testing bigdata solutions (e.g BigQuery).
    For each of the fields you want to generate, you can specify rules for generating their values. The generated data can stored in BigQuery or GCS (Avro, CSV).
*   [BigQuery Analyze Realtime Reddit Data](examples/bigquery-analyze-realtime-reddit-data/) -
    Solution to deploy a ([reddit](https://www.reddit.com)) social media data collection
    architecture on Google Cloud Platform.  Analyzes reddit comments in realtime and
    provides free natural-language processing / sentiment.
*   [BigQuery Audit Log Dashboard](examples/bigquery-audit-log) - Solution to
    help audit BigQuery usage using Data Studio for visualization and a sample
    SQL script to query the back-end data source consisting of audit logs.
*   [BigQuery Audit Log Anomaly Detection](examples/bigquery-auditlog-anomaly-detection) -
    Sample of using BigQuery audit logs for automated anomaly detection and
    outlier analysis. Generates user friendly graphs for quick bq environment
    analysis.
*   [BigQuery Automated Email Exports](examples/bq-email-exports) - Serverless
    solution to automate the sending of BigQuery export results via email on a
    scheduled interval. The email will contain a link to a signed or unsigned
    URL, allowing the recipient to view query results as a JSON, CSV, or Avro
    file.
*   [BigQuery Automated Schema Management](tools/bqman) - Command-line utility
    for automated provisioning and management of BigQuery datasets and tables.
*   [BigQuery Billing Dashboard](examples/bigquery-billing-dashboard) - Solution
    to help displaying billing info using Data Studio for visualization and a
    sample SQL script to query the back-end billing export table in BigQuery.
*   [BigQuery Cross Project Slot Monitoring](examples/bigquery-cross-project-slot-monitoring) -
    Solution to help monitoring slot utilization across multiple projects, while
    breaking down allocation per project.
*   [BigQuery Data Consolidator](tools/bigquery-data-consolidator) - Solution to
    consolidate data within an organization from multiple projects into one target
    Dataset/Table where all Source tables are of same schema (like Billing Exports!); specifically
    useful for data consolidation and further reporting in Cloud FinOps engagements.
*   [BigQuery DDL Validator](examples/bigquery-ddl-validator) -
    A utility that will read the Legacy DDL and compare it against the previously extracted DDL and
    produce an output with the name of the objects where the DDL is no longer matching.
*   [BigQuery Group Sync For Row Level Access](examples/bigquery-row-access-groups) -
    Sample code to synchronize group membership from G Suite/Cloud Identity into
    BigQuery and join that with your data to control access at row level.
*   [BigQuery Long Running Optimization Utility](examples/bigquery-long-running-optimizer) -
    A utility that reads the entire SQL and provides a list of suggestions that would help to optimize the query and avoid the long running issues.
*   [BigQuery Oracle DDL Migration Utility](examples/bigquery-oracle-ddl-migration-utility) -
    Oracle DDL Migration Utility to migrate the tables schema (DDL) from Oracle DB to BigQuery.
    The utility leverages BigQuery Translation API and offers additional features
    such as adding partitioning, clustering, metadata columns and prefixes to table names.
*   [BigQuery Pipeline Utility](tools/bqpipeline) - Python utility class for
    defining data pipelines in BigQuery.
*   [BigQuery Remote Function](examples/bq-remote-function) - It allows user to implement custom
    services or libraries in languages other than SQL or Javascript which are not part of UDFs.
    The utility contains sample string format Java code to deploy cloud run gen2 instance and invoke
    the service from BigQuery using remote function.
*   [BigQuery Amazon S3 Migration Tool](tools/bigquery-s3tobq) - Bigquery Migration Tool to transfer data
    from files in Amazon S3 to BigQuery Tables based on configuration provided.
*   [BigQuery Snowflake TabRle Migration Tool](examples/bigquery-snowflake-tables-migration-utility) -
    BigQuery Snowflake Table Migration Tool helps to migrate the table DDL's from Snowflake to BigQuery.
    The utility leverages BigQuery Translation API and offers additional features
    such as adding partitioning, clustering, metadata columns and prefixes to table names.
*   [BigQuery Table Access Pattern Analysis](examples/bigquery-table-access-pattern-analysis) -
    Sample code to analyse data pipeline optimisation points, by pinpointing
    suboptimal pipeline scheduling between tables in a data warehouse ELT job.
*   [BigQuery Tink Toolkit](tools/bigquery-tink-toolkit) - Python utility class
    for working with Tink-based cryptography in on-prem or GCP systems in a way
    that is interoperable with BigQuery's field-level encryption. Includes a
    sample PySpark job and a script for generating and uploading KMS-encrypted
    Tink keysets to BigQuery.
*   [BigQuery to XML Export](tools/bigquery-to-xml) - Python tool that takes a
    BigQuery query and returns the output as an XML string.
*   [BigQuery Translation Validator](examples/bigquery-translation-validator-utility) - A python utility to compare 2 SQL Files and  point basic differences like column names,
    table names, joins, function names, is-Null and query syntax.
*   [BigQuery Generic DDL Migration Utility](examples/bigquery-generic-ddl-migration-utility) -
    Generic DDL Migration Utility to migrate the tables schema (DDL) from Database(Oracle, Snowflake, MSSQL, Vertica, Neteeza) DB to BigQuery.
    The utility leverages BigQuery Translation API and offers additional features
    such as adding partitioning, clustering, metadata columns and prefixes to table names.
*   [Bigtable Dataflow Cryptocurrencies Exchange RealTime Example](examples/cryptorealtime) -
    Apache Beam example that reads from the Crypto Exchanges WebSocket API as
    Google Cloud Dataflow pipeline and saves the feed in Google Cloud Bigtable.
    Real time visualization and query examples from GCP Bigtable running on
    Flask server are included.
*   [Bigtable Dataflow Update Table Key Pipeline](examples/bigtable-change-key) -
    Dataflow pipeline with an example of how to update the key of an existing
    table. It works with any table, regardless the schema. It shows how to
    update your key for a table with existing data, to try out different
    alternatives to improve performance.
*   [Carbon Footprint Reporting](examples/carbon-foortprint-dashboard) - Example of
    using the prebuilt Data studio & Looker template for analysing GCP Carbon Footprint Estimates.
*   [Cloud Audit Log Samples](examples/audit-log-examples/) - A sample
    collection of Audit Logs for Users and Customers to better the structure,
    contents, and values contained in various log events.
*   [Cloud Build Application CICD Examples](examples/cloudbuild-application-cicd) -
    Cloud Build CI/CD Examples for Applications like containerization &
    deployment to Cloud Run.
*   [Cloud Build with Proxy Running in Background](examples/cloudbuild-with-tcp-proxy) -
    Examples of cloudbuild with docker-compose running tcp proxy in the
    background for all build steps.
*   [Cloud Composer CI/CD](examples/cloud-composer-cicd) - Examples of using
    Cloud Build to deploy airflow DAGs to Cloud Composer.
*   [Cloud Composer Deployment in Shared VPC](examples/composer-shared-vpc) -
    Terraform code to deploy cloud composer in shared VPC environment.
*   [Cloud Composer Dependency Management](examples/cloud-composer-dependency-management-example) - Example of
    Cloud Composer Dependency Management designed to orchestrate complex task dependencies within Apache Airflow which addresses the challenge of managing parent-child DAG relationships across varying temporal frequencies (yearly, monthly, weekly etc)
*   [Cloud Composer Examples](examples/cloud-composer-examples) - Examples of
    using Cloud Composer, GCP's managed Apache Airflow service.
*   [Cloud Data Fusion Functions and Plugins](examples/cloud-datafusion-functions-plugins) -
    Examples of Cloud Data Fusion Functions and Plugins.
*   [Cloud DNS load balancing](examples/cloud-dns-load-balancing) - Multi-region HA setup for GCE VMs and Cloud Run based applications utilizing Cloud DNS load balancing and multiple Google Cloud load balancer types.
*   [Cloud DNS public zone monitoring](examples/cloud-dns-public-zone-dashboard) - Visualizing Cloud DNS public zone query data using log-based metrics and Cloud Monitoring.
*   [Cloud Function Act As](examples/cloud-function-act-as) - Example of
    executing a Cloud Function on behalf and with IAM permissions of the GitHub
    Workload Identity caller.
*   [Cloud Function VM Delete Event Handler Example](examples/gcf-pubsub-vm-delete-event-handler) -
    Solution to automatically delete A records in Cloud DNS when a VM is
    deleted. This solution implements a [Google Cloud Function][gcf]
    [Background Function][gcf-bg] triggered on `compute.instances.delete` events
    published through [Stackdriver Logs Export][logs-export].
*   [Certificate Authority Service Hierarchy](examples/certificate-authority-service-hierarchy) - Root and Subordinate Certificate Authority Service CA Pools and CAs with examples for domain ownership validation and sample load test script.
*   [Cloud Run to BQ](examples/cloudrun-to-bq) - Solution to accept events/data
    on HTTP REST Endpoint and insert into BQ.
*   [Cloud SQL Custom Metric](examples/cloud-sql-custom-metric) - An example of
    creating a Stackdriver custom metric monitoring Cloud SQL Private Services
    IP consumption.
*   [Cloud Support API](examples/cloud-support) - Sample code using Cloud
    Support API
*   [CloudML Bank Marketing](examples/cloudml-bank-marketing) - Notebook for
    creating a classification model for marketing using CloudML.
*   [CloudML Bee Health Detection](examples/cloudml-bee-health-detection) -
    Detect if a bee is unhealthy based on an image of it and its subspecies.
*   [CloudML Churn Prediction](examples/cloudml-churn-prediction) - Predict
    users' propensity to churn using Survival Analysis.
*   [CloudML Customer Support and Complaint Handling](examples/cloudml-support-routing) -
    BigQuery + AutoML pipeline classifying customer complaints based on expected
    resolution; adaptable to other support communications use cases.
*   [CloudML Deep Collaborative Filtering](examples/cloudml-collaborative-filtering) -
    Recommend songs given either a user or song.
*   [CloudML Energy Price Forecasting](examples/cloudml-energy-price-forecasting) -
    Predicting the future energy price based on historical price and weather.
*   [CloudML Fraud Detection](examples/cloudml-fraud-detection) - Fraud
    detection model for credit-cards transactions.
*   [CloudML Scikit-learn Pipeline](examples/cloudml-sklearn-pipeline) - This is
    a example for building a scikit-learn-based machine learning pipeline
    trainer that can be run on AI Platform. The pipeline can be trained locally
    or remotely on AI platform. The trained model can be further deployed on AI
    platform to serve online traffic.
*   [CloudML Sentiment Analysis](examples/cloudml-sentiment-analysis) -
    Sentiment analysis for movie reviews using TensorFlow `RNNEstimator`.
*   [CloudML TensorFlow Profiling](examples/tensorflow-profiling-examples) -
    TensorFlow profiling examples for training models with CloudML
*   [Data Generator](examples/dataflow-data-generator) - Generate random data
    with a custom schema at scale for integration tests or demos.
*   [Dataflow BigQuery Transpose Example](examples/dataflow-bigquery-transpose) -
    An example pipeline to transpose/pivot/rotate a BigQuery table.
*   [Dataflow Custom Templates Example](examples/dataflow-custom-templates) - An
    example that demonstrates how to build custom Dataflow templates.
*   [Dataflow Elasticsearch Indexer](examples/dataflow-elasticsearch-indexer) -
    An example pipeline that demonstrates the process of reading JSON documents
    from Cloud Pub/Sub, enhancing the document using metadata stored in Cloud
    Bigtable and indexing those documents into
    [Elasticsearch](https://www.elastic.co/).
*   [Dataflow BigQuery to AlloyDB](examples/dataflow-bigquery-to-alloydb/) -
    Example that shows how to move data from BigQuery to an AlloyDB table using Dataflow.
*   [Dataflow Flex Template in Restricted Networking Env](examples/dataflow-flex-python/) -
    Example implements a python flex template which can be run in an environment
    where workers can not download python packages due to egress traffic restrictions.
*   [Dataflow Python Examples](examples/dataflow-python-examples) - Various ETL
    examples using the Dataflow Python SDK.
*   [Dataflow Scala Example: Kafka2Avro](examples/dataflow-scala-kafka2avro) -
    Example to read objects from Kafka, and persist them encoded in Avro in
    Google Cloud Storage, using Dataflow with SCIO.
*   [Dataflow Streaming Benchmark](examples/dataflow-streaming-benchmark) -
    Utility to publish randomized fake JSON messages to a Cloud Pub/Sub topic at
    a configured QPS.
*   [Dataflow Streaming Schema Changes Handler](examples/dataflow-streaming-schema-handler) -
    Dataflow example to handle schema changes using schema enforcement and DLT
    approach
*   [Dataflow Streaming XML to GCS](examples/dataflow-xml-pubsub-to-gcs) -
    Dataflow example to handle streaming of xml encoded messages and write them to Google Cloud Storage
*   [Dataflow DLP Hashpipeline](examples/dataflow-dlp-hash-pipeline) - Match DLP
    Social Security Number findings against a hashed dictionary in Firestore.
    Use Secret Manager for the hash key.
*   [Dataflow Template Pipelines](https://github.com/GoogleCloudPlatform/DataflowTemplates) -
    Pre-implemented Dataflow template pipelines for solving common data tasks on
    Google Cloud Platform.
*   [Dataflow Production Ready](examples/dataflow-production-ready) - Reference
    implementation for best practices around Beam, pipeline structuring, testing
    and continuous deployment.
*   [Dataflow XML to BigQuery](examples/dataflow-xmlio-to-bq) - Example of
    loading XML data into BigQuery with DataFlow via XMLIO.
*   [Dataproc GCS Connector](examples/dataproc-gcs-connector) - Install and test
    unreleased features on the GCS Connector for Dataproc.
*   [Dataproc Job Optimization Guide](examples/dataproc-job-optimization-guide) - Step-by-step
    guide for optimizing a sample Dataproc Job.
*   [Dataproc Persistent History Server for Ephemeral Clusters](examples/dataproc-persistent-history-server) -
    Example of writing logs from an ephemeral cluster to GCS and using a
    separate single node cluster to look at Spark and YARN History UIs.
*   [Dataproc Lifecycle Management via Composer](examples/dataproc-lifecycle-via-composer) - Ephemeral Dataproc lifecycle management and resources optimization via Composer, Terraform template to deploy Composer and additional reqs, Dynamically generated DAGs from jobs config files.
*   [Dataproc Running Notebooks](examples/dataproc-running-notebooks) - Orchestrating the workflow of running Jupyter Notebooks on a Dataproc cluser via PySpark job
*   [dbt-on-cloud-composer](examples/dbt-on-cloud-composer) - Example of using
    dbt to manage BigQuery data pipelines, utilizing Cloud Composer to run and
    schedule the dbt runs.
*   [Data Format Description Language (DFDL) Processesor with Firestore and
    Pubsub](examples/dfdl-firestore-pubsub-example) - Example to process a
    binary using DFDL definition and Daffodil libraries. The DFDL definition is
    stored in firestore, the request to process is done through a pubsub
    subcription and the output is published is a JSON format in a Pubsub topic.
*   [Data Format Description Language (DFDL) Processesor with Bigtable and
    Pubsub](examples/dfdl-bigtable-pubsub-example) - Example to process a binary
    using DFDL definition and Daffodil libraries. The DFDL definition is stored
    in bigtable, the request to process is done through a pubsub subcription and
    the output is published is a JSON format in a Pubsub topic.
*   [Dialogflow Webhook Example](examples/dialogflow-webhook-example) - Webhook
    example for dialogflow in Python.
*   [Dialogflow CX Private Webhook Example](examples/dialogflowcx-private-webhook-example) -
    Webhook example for Dialogflow CX in Python.
*   [Dialogflow Middleware Example](examples/ccai-dialogflow-middleware) -
    Dialogflow middleware example in Java.
*   [Dialogflow Entities Creation and Update](examples/dialogflow-entities-example) -
    Creation and update of entities for Dialogflow in Python.
*   [DLP API Examples](examples/dlp) - Examples of the DLP API usage.
*   [Ephemeral Projects](examples/ephemeral-projects) - Creating short lived gcp projects for sandbox purposes.
*   [GCE Access to Google AdminSDK](examples/gce-to-adminsdk) - Example to help
    manage access to Google's AdminSDK using GCE's service account identity
*   [GCS Hive External Table File Optimization](examples/gcs-hive-external-table-file-optimization) -
    Example solution to showcase impact of file count, file size, and file
    type on Hive external tables and query speeds.
*   [GCS to BQ using serverless services](examples/gcs-to-bq-serverless-services) -
    Example to ingest GCS to BigQuery using serverless services such as Cloud
    Function, Pub/Sub and Serverless Spark.
*   [GDCE Terraform Example](examples/gdce-terraform-example) - Example for provisioning GDCE
    resources using terraform.
*   [GKE HA setup using spot VMs](examples/gke-ha-setup-using-spot-vms/) -
    Example for running an application with high availability requirements on
    GKE spot nodes using on-demand nodes as fallback
*   [Grpc Server connected to Spanner Database](examples/grpc_spanner_example) -
    Basic example of a Grpc server that is connected to a Spanner database.
*   [Grpc Server connected to Redis](examples/grpc_redis_example) - Basic
    example of a Grpc server that is connected to Redis.
*   [Gitlab KAS agent for GKE](examples/gitlab-kas-gke) - Terraform solution for          deploying a Gitlab KAS agent for synchronizing container deployments from Gitlab repos into a GKE cluster
*   [Home Appliance Status Monitoring from Smart Power Readings](examples/e2e-home-appliance-status-monitoring) -
    An end-to-end demo system featuring a suite of Google Cloud Platform
    products such as IoT Core, ML Engine, BigQuery, etc.
*   [IAP User Profile](examples/iap-user-profile) - An example to retrieve user
    profile from an IAP-enabled GAE application.
*   [IoT Nirvana](examples/iot-nirvana) - An end-to-end Internet of Things
    architecture running on Google Cloud Platform.
*   [Kubeflow Pipelines Sentiment Analysis](examples/kubeflow-pipelines-sentiment-analysis) -
    Create a Kubeflow Pipelines component and pipelines to analyze sentiment for
    New York Times front page headlines using Cloud Dataflow (Apache Beam Java)
    and Cloud Natural Language API.
*   [Kubeflow Fairing Example](examples/kubeflow-fairing-example) - Provided
    three notebooks to demonstrate the usage of Kubeflow Faring to train machine
    learning jobs (Scikit-Learn, XGBoost, Tensorflow) locally or in the Cloud
    (AI platform training or Kubeflow cluster).
*   [Left-Shift Validation Pre-Commit Hook](examples/left-shift-validation-pre-commit-hook/) -
    An example that uses a set of Bash scripts to set up a pre-commit hook that
    validates Kubernetes resources with Gatekeeper constraints and constraint
    templates from your choice of sources.
*   [LookerStudio Cost Optimization Dashboard](examples/cost-optimization-dashboard) -
    SQL scripts to help build Cost Optimization LookerStudio Dashboard.
*   [Personal Workbench Notebooks Deployer](examples/personal-workbench-notebooks-deployer) - Terraform sample modules to provision Dataproc Hub using personal auth clusters, and workbench managed notebooks for individual analytical users.
*   [Project factory with Terragrunt](examples/terragrunt-project-factory-gcp/) -
    This implements a `State-Scalable` project factory pattern for creating Google Cloud Platform projects using Terragrunt and public Terraform modules
*   [Python CI/CD with Cloud Builder and CSR](examples/python-cicd-with-cloudbuilder) -
    Example that uses Cloud Builder and Cloud Source Repositories to automate
    testing and linting.
*   [Pub/Sub Client Batching Example](examples/pubsub-publish-avro-example) -
    Batching in Pub/Sub's Java client API.
*   [QAOA](examples/qaoa) - Examples of parsing a max-SAT problem in a
    proprietary format, for Quantum Approximate Optimization Algorithm (QAOA)
*   [Redis Cluster on GKE Example](examples/redis-cluster-gke) - Deploying Redis
    cluster on GKE.
*   [Spanner Interleave Subquery](examples/spanner-interleave-subquery) -
    Example code to benchmark Cloud Spanner's subqueries for interleaved tables.
*   [Spanner Change Stream to BigQuery using Dataflow](examples/spanner-changestreams-bigquery) -
    Terraform code to deploy Spanner change stream and publish changes to BigQuery using Dataflow Streaming Job.
*   [Spinnaker](examples/spinnaker) - Example pipelines for a Canary /
    Production deployment process.
*   [STS Metrics from STS Notification](examples/sts-metrics) - Example code to
    generate custom metrics from STS notification.
*   [TensorFlow Serving on GKE and Load Testing](examples/tf-load-testing) -
    Examples how to implement Tensorflow model inference on GKE and to perform a
    load testing of such solution.
*   [TensorFlow Unit Testing](examples/tensorflow-unit-testing) - Examples how
    to write unit tests for TensorFlow ML models.
*   [Terraform Internal HTTP Load Balancer](examples/terraform-ilb) - Terraform
    example showing how to deploy an internal HTTP load balancer.
*   [Terraform NetApp CVS](examples/tf-netapp-cvs) - This example shows how to deploy NetApp CVS volumes using
    terraform.
*   [Terraform Resource Change Policy Library](examples/terraform-resource-change-policy-library) -
    Contains a library of policies written in the
    [OPA Constraint Framework](https://github.com/open-policy-agent/frameworks/blob/master/constraint/README.md)
    format to be used by `gcloud beta terraform vet` to validate Terraform resource
    changes in a CI/CD pipeline.
*   [Uploading files directly to Google Cloud Storage by using Signed URL](examples/direct-upload-to-gcs) -
    Example architecture to enable uploading files directly to GCS by using
    [Signed URL](https://cloud.google.com/storage/docs/access-control/signed-urls).
*   [TSOP object transfer Log prosessor](examples/tsop-log-processor/) - This example shows
    how to log object transfer logs by TSOP to Cloud Logging.
*   [GCS CSV files to BigQuery](https://github.com/GoogleCloudPlatform/DataflowTemplates/blob/main/v1/README_GCS_CSV_to_BigQuery.md) - This example shows how to load files in CSV format stored in GCS to load to BigQuery tables. The files can be uncompressed or be compressed in formats such as Bzip2, GZIP and etc. See https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/io/Compression.html for the list of support compression method.

## Tools

The tools folder contains ready-made utilities which can simplify Google Cloud
Platform usage.

*   [Agile Machine Learning API](tools/agile-machine-learning-api) - A web
    application which provides the ability to train and deploy ML models on
    Google Cloud Machine Learning Engine, and visualize the predicted results
    using LIME through simple post request.
*   [Airflow DAG Metadata Generator](tools/airflow-dag-metadata-generator) - Use Google's
    generative models to analyze Airflow DAGs and supplement them with generated `description`,
    `tags`, and `doc_md` values.
*   [Airflow States Collector](tools/airflow-states-collector) -
      A tool that creates and uploads an airflow dag to the dags GCS folder. The dag incrementally collect airflow task states and stores to BQ. It also autogenerates a LookerStudio dashboard querying the BQ view.
*   [Airpiler](tools/airpiler) - A python script to convert Autosys JIL files to
    dag-factory format to be executed in Cloud Composer (managed airflow
    environment).
*   [Ansible Module for Anthos on Bare Metal](tools/anthosbm-ansible-module) -
    Ansible module for installation of Anthos on Bare Metal
*   [Anthos Bare Metal Installer](tools/anthosbm-ansible-module) - An
    [ansible](https://www.ansible.com/resources/get-started) playbook that can
    be used to install
    [Anthos Bare Metal](https://cloud.google.com/anthos/clusters/docs/bare-metal).
*   [Apache Beam Client Throttling](tools/apachebeam-throttling) - A library
    that can be used to limit the number of requests from an Apache Beam
    pipeline to an external service. It buffers requests to not overload the
    external service and activates client-side throttling when the service
    starts rejecting requests due to out of quota errors.
*   [API Key Rotation Checker](tools/api-key-rotation) - A tool that checks your
    GCP organization for API keys and compares them to a customizable rotation
    period. Regularly rotating API keys is a Google and industry standard
    recommended best practice.
*   [AssetInventory](tools/asset-inventory) - Import Cloud Asset Inventory
    resourcs into BigQuery.
*   [BigQuery Discount Per-Project Attribution](tools/kunskap) - A tool that
    automates the generation of a BigQuery table that uses existing exported
    billing data, by attributing both CUD and SUD charges on a per-project
    basis.
*   [BigQuery Policy Tag Utility](tools/bqtag) - Utility class for tagging BQ
    Table Schemas with Data Catalog Taxonomy Policy Tags. Create BQ Authorized
    Views using Policy Tags. Helper utility to provision BigQuery Dataset, Data
    Catalog Taxonomy and Policy Tags.
*   [BigQuery Query Plan Exporter](tools/bigquery-query-plan-exporter) - Command
    line utility for exporting BigQuery query plans in a given date range.
*   [BigQuery Query Plan Visualizer](tools/bq-visualizer) - A web application
    which provides the ability to visualise the execution stages of BigQuery
    query plans to aid in the optimization of queries.
*   [BigQuery z/OS Mainframe Connector](tools/bigquery-zos-mainframe-connector) -
    A utility used to load COBOL MVS data sets into BigQuery and execute query
    and load jobs from the IBM z/OS Mainframe.
*   [Boolean Organization Policy Enforcer](tools/boolean-org-policy-enforcer) -
    A tool to find the projects that do not set a boolean organization policy to
    its expected state, subsequently, set the organization policy to its
    expected set.
*   [Capacity Planner CLI](tools/capacity-planner-cli) - A stand-alone tool to
    extract peak resource usage values and corresponding timestamps for a given
    GCP project, time range and timezone.
*   [Capacity Planner Sheets Extension](tools/capacity-planner-sheets-extension) -
    A Google Sheets extension to extract peak resource usage values and corresponding
    timestamps for a given GCP project, time range and timezone.
*   [CloudConnect](tools/cloudconnect) - A package that automates the setup of
    dual VPN tunnels between AWS and GCP.
*   [Cloudera Parcel GCS Connector](tools/cloudera-parcel-gcsconnector) - This
    script helps you create a Cloudera parcel that includes Google Cloud Storage
    connector. The parcel can be deployed on a Cloudera managed cluster. This
    script helps you create a Cloudera parcel that includes Google Cloud Storage
    connector. The parcel can be deployed on a Cloudera managed cluster.
*   [Cloud AI Vision Utilities](tools/cloud-vision-utils) - This is an
    installable Python package that provides support tools for Cloud AI Vision.
    Currently there are a few scripts for generating an AutoML Vision dataset
    CSV file from either raw images or image annotation files in PASCAL VOC
    format.
*   [Cloud Composer Backup and Recovery](tools/cloud-composer-backup-restore) - A
    command line tool for applying backup and recovery operations on Cloud
    Composer Airflow environments.
*   [Cloud Composer DAG Validation](tools/cloud-composer-dag-validation) - An automated process for running validation and testing against DAGs in Composer.
*   [Cloud Composer Migration Complexity Assessment](tools/cloud-composer-migration-complexity-assessment) - An Airflow DAG that uses a variety
    of tools to analyze a Cloud Composer 1 environment, determine a work estimate, and
    accelerate the conversion of airflow 1 dags to airflow 2 dags.
*   [Cloud Composer Migration Terraform Generator](tools/cloud-composer-migration-terraform-generator) - Analyzes an existing Cloud Composer 1
    / Airflow 1 environment and generates terraform. Configures new Cloud Composer 2
    environment to meet your workload demands.
*   [CUD Prioritized Attribution](tools/cuds-prioritized-attribution) - A tool
    that allows GCP customers who purchased Committed Use Discounts (CUDs) to
    prioritize a specific scope (e.g. project or folder) to attribute CUDs first
    before letting any unconsumed discount float to other parts of an
    organization.
*   [Custom Organization Policy Library](tools/custom-organization-policy-library) - A library
    of custom organization policy constraints and samples. It includes tools to easily generate policies for provisioning across your organization using either Google Cloud (gcloud) or Terraform.
*   [Custom Role Analyzer](tools/custom-roles-analyzer) - This tool will provide
    useful insights with respect to custom roles at organization level as well
    as project level to find predefined roles from which the custom role is
    built.
*   [Custom Role Manager](tools/custom-role-manager) - Manages organization- or
    project-level custom roles by combining predefined roles and including and
    removing permissions with wildcards. Can run as Cloud Function or output
    Terraform resources.
*   [Dataproc Event Driven Spark Recommendations](tools/dataproc-event-driven-spark-recommendations/) -
    Use Google Cloud Functions to analyze Cloud Dataproc clusters and recommend
    best practices for Apache Spark jobs.  Also logs cluster configurations for
    future reference.
*   [Dataproc Scheduled Cluster Sizing](tools/dataproc-scheduled-cluster-sizing/) -
    Use Google Cloud Scheduler an Google Cloud Functions to schedule the resizing
    of a Dataproc cluster.  Changes the primary and secondary worker count.
*   [DataStream Deployment Automation](tools/datastream-deployment-python-automation) -
    Python script to automate the deployment of Google Cloud DataStream. This
    script will create connection profiles, create stream and start stream.
*   [DLP to Data Catalog](tools/dlp-to-data-catalog/) - Inspect your tables using Data Loss  Prevention for PII data and automatically tag it on Data Catalog using Python.
*   [DNS Sync](tools/dns-sync) - Sync a Cloud DNS zone with GCE resources.
    Instances and load balancers are added to the cloud DNS zone as they start
    from compute_engine_activity log events sent from a pub/sub push
    subscription. Can sync multiple projects to a single Cloud DNS zone.
*   [Firewall Enforcer](tools/firewall-enforcer) - Automatically watch & remove
    illegal firewall rules across organization. Firewall rules are monitored by
    a Cloud Asset Inventory Feed, which trigger a Cloud Function that inspects
    the firewall rule and deletes it if it fails a test.
*   [GCE Disk Encryption Converter](tools/gce-google-keys-to-cmek) - A tool that
    converts disks attached to a GCE VM instance from Google-managed keys to a
    customer-managed key stored in Cloud KMS.
*   [GCE switch disk-type](tools/gce-change-disktype) - A tool that changes type
    of disks attached to a GCE instance.
*   [GCE Quota Sync](tools/gce-quota-sync) - A tool that fetches resource quota
    usage from the GCE API and synchronizes it to Stackdriver as a custom
    metric, where it can be used to define automated alerts.
*   [GCE Usage Log](tools/gce-usage-log) - Collect GCE instance events into a
    BigQuery dataset, surfacing your vCPUs, RAM, and Persistent Disk, sliced by
    project, zone, and labels.
*   [GCP Architecture Visualizer](https://github.com/forseti-security/forseti-visualizer) -
    A tool that takes CSV output from a Forseti Inventory scan and draws out a
    dynamic hierarchical tree diagram of org -> folders -> projects ->
    gcp_resources using the D3.js javascript library.
*   [GCP AWS HA VPN Connection terraform ](tools/gcp-aws-ha-vpn) - Terraform
    script to setup HA VPN between GCP and AWS.
*   [GCP Azure HA VPN Connection Terraform](tools/gcp-azure-ha-vpn) - Terraform
    code to setup HA VPN between GCP and Microsoft Azure.
*   [GCP Organization Hierarchy Viewer](tools/gcp-org-hierarchy-viewer) - A CLI
    utility for visualizing your organization hierarchy in the terminal.
*   [GCPViz](tools/gcpviz) - a visualization tool that takes input from
    [Cloud Asset Inventory](https://cloud.google.com/asset-inventory/docs/overview),
    creates relationships between assets and outputs a format compatible with
    [graphviz](http://graphviz.gitlab.io/).
*   [GCS Bucket Mover](tools/gcs-bucket-mover) - A tool to move user's bucket,
    including objects, metadata, and ACL, from one project to another.
*   [GCS to BigQuery](tools/gcs2bq) - A tool fetches object metadata from all
    Google Cloud Storage buckets and exports it in a format that can be imported
    into BigQuery for further analysis.
*   [GCS Usage Recommender](tools/gcs-usage-recommender) - A tool that generates
    bucket-level intelligence and access patterns across all projects for a GCP
    project to generate recommended object lifecycle management.
*   [GCVE2BQ](tools/gcve2bq) - A tool for scheduled exports of VM, datastore and ESXi
    utilization data from vCenter to BigQuery for billing and reporting use cases.
*   [GKE AutoPSC Controller](tools/gke-autopsc-controller) - Google Kubernetes Engine
    controller, to setup PSC ServiceAttachment for Gateway API managed Forwarding Rules.
*   [Global DNS -> Zonal DNS Project Bulk Migration](tools/gdns-zdns-project-bulk-migration) -
    A shell script for gDNS-zDNS project bulk migration.
*   [GKE Billing Export](tools/gke-billing-export) - Google Kubernetes Engine
    fine grained billing export.
*   [gmon](tools/gmon/) - A command-line interface (CLI) for Cloud Monitoring
    written in Python.
*   [Google Cloud Support Slackbot](tools/google-cloud-support-slackbot) - Slack
    application that pulls Google Cloud support case information via the Cloud
    Support API and pushes the information to Slack
*   [GSuite Exporter Cloud Function](tools/gsuite-exporter-cloud-function/) - A
    script that deploys a Cloud Function and Cloud Scheduler job that executes
    the GSuite Exporter tool automatically on a cadence.
*   [GSuite Exporter](tools/gsuite-exporter/) - A Python package that automates
    syncing Admin SDK APIs activity reports to a GCP destination. The module
    takes entries from the chosen Admin SDK API, converts them into the
    appropriate format for the destination, and exports them to a destination
    (e.g: Stackdriver Logging).
*   [Hive to BigQuery](tools/hive-bigquery/) - A Python framework to migrate
    Hive table to BigQuery using Cloud SQL to keep track of the migration
    progress.
*   [IAM Permissions Copier](tools/iam-permissions-copier) - This tool allows
    you to copy supported GCP IAM permissions from unmanaged users to managed
    Cloud Identity users.
*   [IAM Recommender at Scale](tools/iam-recommender-at-scale) - A python
    package that automates applying iam recommendations.
*   [Instance Mapper](tools/instance_mapper) - Maps different IaaS VM instance
    types from EC2 and Azure Compute to Google Cloud Platform instance types
    using a customizable score-based method. Also supports database instances.
*   [IPAM Autopilot](tools/ipam-autopilot) - A simple tool for managing IP
    address ranges for GCP subnets.
*   [K8S-2-GSM](tools/k8s-2-gsm) - A containerized golang app to migrate Kubernetes
    secrets to Google Secrets Manger
    (to leverage [CSI secret driver](https://secrets-store-csi-driver.sigs.k8s.io/)).
    [LabelMaker](tools/labelmaker) - A tool that reads key:value pairs from a
    json file and labels the running instance and all attached drives
    accordingly.
*   [Logbucket Global to Regional](tools/logbucket-global-to-regional) - Utility
    to change _Default sink destination to regional log buckets
*   [Machine Learning Auto Exploratory Data Analysis and Feature Recommendation](tools/ml-auto-eda) -
    A tool to perform comprehensive auto EDA, based on which feature
    recommendations are made, and a summary report will be generated.
*   [Maven Archetype Dataflow](tools/maven-archetype-dataflow) - A maven
    archetype which bootstraps a Dataflow project with common plugins
    pre-configured to help maintain high code quality.
*   [Netblock Monitor](tools/netblock-monitor) - An Apps Script project that
    will automatically provide email notifications when changes are made to
    Google’s IP ranges.
*   [OpenAPI to Cloud Armor converter](tools/openapi-to-cloud-armor) - A simple
    tool to generate Cloud Armor policies from OpenAPI specifications.
*   [Permission Discrepancy Finder](tools/permission-discrepancy-finder) - A
    tool to find the principals with missing permissions on a resource within a
    project, subsequently, grants them the missing permissions.
*   [Pubsub2Inbox](tools/pubsub2inbox) - A generic Cloud Function-based tool
    that takes input from Pub/Sub messages and turns them into email, webhooks
    or GCS objects.
*   [Quota Manager](tools/quota-manager) - A python module to programmatically
    update GCP service quotas such as bigquery.googleapis.com.
*   [Quota Monitoring and Alerting](tools/quota-monitoring-alerting) - An
    easy-to-deploy Data Studio Dashboard with alerting capabilities, showing
    usage and quota limits in an organization or folder.
*   [Ranger Hive Assessment for BigQuery/BigLake IAM migration](tools/ranger-to-bigquery-biglake-assessment) -
    A tool that assesses which Ranger authorization rules can be migrated
    or not to BigQuery/BigLake IAM.
*   [Reddit Comment Streaming](tools/reddit-comment-streaming/) -
    Use PRAW, TextBlob, and Google Python API to collect and analyze
    reddit comments. Pushes comments to a Google Pub/sub Topic.
*   [Secret Manager Helper](tools/secret-manager-helper) - A Java library to
    make it easy to replace placeholder strings with Secret Manager secret
    payloads.
*   [Service Account Provider](tools/service-account-provider) - A tool to
    exchange GitLab CI JWT tokens against GCP IAM access tokens, in order to
    allow GitLab CI jobs to access Google Cloud APIs
*   [Site Verification Group Sync](tools/site-verification-group-sync) - A tool
    to provision "verified owner" permissions (to create GCS buckets with custom
    dns) based on membership of a Google Group.
*   [SLO Generator](tools/slo-generator/) - A Python package that automates
    computation of Service Level Objectives, Error Budgets and Burn Rates on
    GCP, and export the computation results to available exporters (e.g: PubSub,
    BigQuery, Stackdriver Monitoring), using policies written in JSON format.
*   [Snowflake_to_BQ](tools/snowflake2bq/) - A shell script to transfer tables
    (schema & data) from Snowflake to BigQuery.
*   [SPIFFE GCP Proxy](tools/spiffe-gcp-proxy) - A tool to ease the integration
    of [SPIFFE](https://spiffe.io/) supported On-Prem workloads with GCP APIs
    using Workload Identity Federation
*   [STS Job Manager](tools/sts-job-manager/) - A petabyte-scale bucket
    migration tool utilizing
    [Storage Transfer Service](https://cloud.google.com/storage-transfer-service)
*   [VM Migrator](tools/vm-migrator) - This utility automates migrating Virtual
    Machine instances within GCP. You can migrate VM's from one zone to another
    zone/region within the same project or different projects while retaining
    all the original VM properties like disks, network interfaces, ip, metadata,
    network tags and much more.
*   [VPC Flow Logs Analysis](tools/vpc-flowlogs-analysis) - A configurable Log
    sink + BigQuery report that shows traffic attributed to the projects in the
    Shared VPCs.
*   [VPC Flow Logs Enforcer](tools/vpc-flowlogs-enforcer) - A Cloud Function
    that will automatically enable VPC Flow Logs when a subnet is created or
    modified in any project under a particular folder or folders.
*   [VPC Flow Logs Top Talkers](tools/vpc-flowlogs-toptalkers) - A configurable
    Log sink + BigQuery view to generate monthly/daily aggregate traffic reports
    per subnet or host, with the configurable labelling of IP ranges and ports.
*   [Webhook Ingestion Data Pipeline](tools/webhook-ingestion-pipeline) - A
    deployable app to accept and ingest unauthenticated webhook data to
    BigQuery.
*   [XSD to BigQuery Schema Generator](tools/xsd-to-bigquery-schema) - A command
    line tool for converting an XSD schema representing deeply nested and
    repeated XML content into a BigQuery compatible table schema represented in
    JSON.
*   [Numeric Family Recommender - Oracle](tools/numeric-family-recommender-oracle) - The Numeric Family
    Recommender is a database script that recommends the best numeric data type for the NUMBER data type
    when migrating from legacy databases like Oracle to Google Cloud platforms like BigQuery, AlloyDB,
    Cloud SQL for PostgreSQL, and Google Cloud Storage.
*   [Composer DAG Load Generator](tools/cloud-composer-dag-generator) - This is an automatic
    DAG generator tool which can be used to create test workload on a cloud composer environmnet
    and to test differents airflows configurations or to do fine tune using the composer/airflow metrics.
*   [Gradio and Generative AI Example](examples/genai-gradio-example) - The example code allows developers
    to create rapid Generative AI PoC applications with Gradio and Gen AI agents.
*   [Memorystore Cluster Ops Framework](tools/memorystore-cluster-ops-framework) - This is a framework that
    provides the tools to apply cluster level operations that enable capabilities like cluster backups, migration & validation, etc.
    The framework can be extended for other use cases as required.
    The framework uses RIOT to bridge current product gaps with Memorystore Clusters
*   [ML Project Generator](tools/ml-project-generator) - A utility to create a Production grade ML project template with the best productivity tools installed like auto-formatting, license checks, linting, etc.

## Contributing

See the contributing [instructions](/CONTRIBUTING.md) to get started
contributing.

## Contact

Questions, issues, and comments should be directed to
[professional-services-oss@google.com](mailto:professional-services-oss@google.com).

[gcf]: https://cloud.google.com/functions/
[gcf-bg]: https://cloud.google.com/functions/docs/writing/background
[logs-export]: https://cloud.google.com/logging/docs/export/


[![PyPI version](https://badge.fury.io/py/arviz.svg)](https://badge.fury.io/py/arviz)
[![Azure Build Status](https://dev.azure.com/ArviZ/ArviZ/_apis/build/status/arviz-devs.arviz?branchName=main)](https://dev.azure.com/ArviZ/ArviZ/_build/latest?definitionId=1&branchName=main)
[![codecov](https://codecov.io/gh/arviz-devs/arviz/branch/main/graph/badge.svg)](https://codecov.io/gh/arviz-devs/arviz)
[![Code style: black](https://img.shields.io/badge/code%20style-black-000000.svg)](https://github.com/ambv/black)
[![Gitter chat](https://badges.gitter.im/gitterHQ/gitter.png)](https://gitter.im/arviz-devs/community)
[![DOI](http://joss.theoj.org/papers/10.21105/joss.01143/status.svg)](https://doi.org/10.21105/joss.01143) [![DOI](https://zenodo.org/badge/DOI/10.5281/zenodo.2540945.svg)](https://doi.org/10.5281/zenodo.2540945)
[![Powered by NumFOCUS](https://img.shields.io/badge/powered%20by-NumFOCUS-orange.svg?style=flat&colorA=E1523D&colorB=007D8A)](https://numfocus.org)

ArviZ (pronounced "AR-_vees_") is a Python package for exploratory analysis of Bayesian models. It includes functions for posterior analysis, data storage, model checking, comparison and diagnostics.

### ArviZ in other languages
ArviZ also has a Julia wrapper available [ArviZ.jl](https://julia.arviz.org/).

## Documentation

The ArviZ documentation can be found in the [official docs](https://python.arviz.org/en/latest/index.html).
First time users may find the [quickstart](https://python.arviz.org/en/latest/getting_started/Introduction.html)
to be helpful. Additional guidance can be found in the
[user guide](https://python.arviz.org/en/latest/user_guide/index.html).

## Installation

### Stable
ArviZ is available for installation from [PyPI](https://pypi.org/project/arviz/).
The latest stable version can be installed using pip:

```
pip install arviz
```

ArviZ is also available through [conda-forge](https://anaconda.org/conda-forge/arviz).

```
conda install -c conda-forge arviz
```

### Development
The latest development version can be installed from the main branch using pip:

```
pip install git+git://github.com/arviz-devs/arviz.git
```

Another option is to clone the repository and install using git and setuptools:

```
git clone https://github.com/arviz-devs/arviz.git
cd arviz
python setup.py install
```

-------------------------------------------------------------------------------
## [Gallery](https://python.arviz.org/en/latest/examples/index.html)

  And more...

## Dependencies

ArviZ is tested on Python 3.10, 3.11 and 3.12, and depends on NumPy, SciPy, xarray, and Matplotlib.

## Citation

If you use ArviZ and want to cite it please use [![DOI](http://joss.theoj.org/papers/10.21105/joss.01143/status.svg)](https://doi.org/10.21105/joss.01143)

Here is the citation in BibTeX format

```
@article{arviz_2019,
  doi = {10.21105/joss.01143},
  url = {https://doi.org/10.21105/joss.01143},
  year = {2019},
  publisher = {The Open Journal},
  volume = {4},
  number = {33},
  pages = {1143},
  author = {Ravin Kumar and Colin Carroll and Ari Hartikainen and Osvaldo Martin},
  title = {ArviZ a unified library for exploratory analysis of Bayesian models in Python},
  journal = {Journal of Open Source Software}
}
```

## Contributions
ArviZ is a community project and welcomes contributions.
Additional information can be found in the [Contributing Readme](https://github.com/arviz-devs/arviz/blob/main/CONTRIBUTING.md)

## Code of Conduct
ArviZ wishes to maintain a positive community. Additional details
can be found in the [Code of Conduct](https://github.com/arviz-devs/arviz/blob/main/CODE_OF_CONDUCT.md)

## Donations
ArviZ is a non-profit project under NumFOCUS umbrella. If you want to support ArviZ financially, you can donate [here](https://numfocus.org/donate-to-arviz).

## Sponsors
[![NumFOCUS](https://www.numfocus.org/wp-content/uploads/2017/07/NumFocus_LRG.png)](https://numfocus.org)


  <picture>
    <source media="(prefers-color-scheme: dark)" srcset="https://github.com/holoviz/panel/raw/main/doc/_static/logo_horizontal_dark_theme.png">

  </picture>

# Panel: The powerful data exploration & web app framework for Python

Panel is an [open-source](https://github.com/holoviz/panel/blob/main/LICENSE.txt) Python library that lets you **easily build powerful tools, dashboards and complex applications entirely in Python**. It has a batteries-included philosophy, putting the PyData ecosystem, powerful data tables and much more at your fingertips. High-level reactive APIs and lower-level callback based APIs ensure you can quickly build exploratory applications, but you aren't limited if you build complex, multi-page apps with rich interactivity. Panel is a member of the [HoloViz](https://holoviz.org/) ecosystem, your gateway into a connected ecosystem of data exploration tools.

---

Enjoying Panel? Show your support with a [Github star](https://github.com/holoviz/panel) — it’s a simple click that means the world to us and helps others discover it too! ⭐

---

Downloads

Build Status

Coverage

Latest dev release

Latest release

Docs

Notebooks

Support

[Home](https://panel.holoviz.org/) | [Installation instructions](#installation-instructions) | [Getting Started Guide](https://panel.holoviz.org/getting_started/index.html) | [Reference Guides](https://panel.holoviz.org/reference/index.html) | [Examples](#examples) | [License](#license) | [Support](#support--feedback)

## Panel works with the tools you know and love

[Panel](https://panel.holoviz.org/) makes it easy to combine widgets, plots, tables and other viewable Python objects into custom analysis tools, applications, and dashboards.

[![Panel NYC Taxi Linked Brushing](https://assets.holoviz.org/panel/readme/linked_brushing.gif)](https://panel.holoviz.org/reference/templates/FastGridTemplate.html)

Panel works really well with the visualization tools you already know and love like [Altair/ Vega](https://panel.holoviz.org/reference/panes/Vega.html), [Bokeh](https://panel.holoviz.org/reference/panes/Bokeh.html), [Datashader](https://datashader.org/), [Deck.gl/ pydeck](https://panel.holoviz.org/reference/panes/DeckGL.html), [Echarts/ pyecharts](https://panel.holoviz.org/reference/panes/ECharts.html), [Folium](https://panel.holoviz.org/reference/panes/Folium.html), [HoloViews](https://holoviews.org/), [hvPlot](https://hvplot.holoviz.org), [plotnine](https://panel.holoviz.org/reference/panes/Matplotlib.html), [Matplotlib](https://panel.holoviz.org/reference/panes/Matplotlib.html), [Plotly](https://panel.holoviz.org/reference/panes/Plotly.html), [PyVista/ VTK](https://panel.holoviz.org/reference/panes/VTK.html), [Seaborn](https://panel.holoviz.org/gallery/styles/SeabornStyle.html) and more. Panel also works with the [ipywidgets](https://panel.holoviz.org/reference/panes/IPyWidget.html) ecosystem.

[![Pythons DataViz works with Panel](https://assets.holoviz.org/panel/readme/dataviz.gif)](https://panel.holoviz.org/reference/index.html#panes)

Panel provides bi-directional communication making it possible to react to clicks, selections, hover etc. events.

[![Vega Selections](https://assets.holoviz.org/panel/readme/vega_selections.gif)](https://panel.holoviz.org/reference/panes/Vega.html)

You can develop in [Jupyter Notebooks](http://jupyter.org) as well as editors like [VS Code](https://code.visualstudio.com/), [PyCharm](https://www.jetbrains.com/pycharm/) or [Spyder](https://www.spyder-ide.org/).

Panel provides a unique combination of deployment options. You can share your data and models as

- a web application running on the [Tornado](https://www.tornadoweb.org/en/stable/) (default), [Flask](https://flask.palletsprojects.com/), [Django](https://www.djangoproject.com/) or [Fast API](https://fastapi.tiangolo.com/) web server.
- a stand alone client side application powered by [Pyodide](https://pyodide.org/en/stable/) or [PyScript](https://pyscript.net/) via [`panel convert`](https://panel.holoviz.org/how_to/wasm/convert.html).
- an interactive Jupyter notebook component.
- a static `.html` web page, a `.gif` video, a `.png` image and more.

Panel has something to offer for every one *from beginner to data pro*.

## Panel is a member of the HoloViz ecosystem

Panel is a member of the ambitious [HoloViz](https://holoviz.org/) dataviz ecosystem and has first class support for the other members like [hvPlot](https://hvplot.holoviz.org) (simple .hvplot plotting api), [HoloViews](https://holoviews.org/) (powerful plotting api), and [Datashader](https://datashader.org/) (big data viz).

Panel is built on top of [Param](https://param.holoviz.org). Param enables you to annotate your code with parameter ranges, documentation, and dependencies between parameters and code. With this approach,

- you don't ever have to commit to whether your code will be used in a notebook, a data app, in batch processing, or reports.
- you will write less code and be able to develop large, maintainable code bases!

## Mini getting-started

Head over to the [getting started guide](https://panel.holoviz.org/getting_started/index.html) for more!

### Installation Instructions

Panel can be installed on Linux, Windows, or Mac with ``conda``:

```bash
conda install panel
```

or with ``pip``:

```bash
pip install panel
```

See the [Environments](#environments) section below for additional instructions for your environment.

### Interactive data apps

Bring your data or model

```python
def model(n=5):
    return "⭐"*n
```

Bind it to a Panel *widget* and *lay it out*.

```python
import panel as pn

pn.extension()

slider = pn.widgets.IntSlider(value=5, start=1, end=5)

interactive_model = pn.bind(model, n=slider)

layout = pn.Column(slider, interactive_model)
```

![Panel Notebook Example](https://assets.holoviz.org/panel/readme/notebook.gif)

For deployment on a web server wrap it in a nice template.

```python
pn.template.FastListTemplate(
    site="Panel", title="Example", main=[layout],
).servable()
```

Start the server with

```bash
panel serve name_of_script.py --show
```

or

```bash
panel serve name_of_notebook.ipynb --show
```

![Panel Example App](https://assets.holoviz.org/panel/readme/example_app.gif)

## Examples

[![Panel Gallery](https://assets.holoviz.org/panel/readme/gallery.jpg)](https://panel.holoviz.org/gallery/index.html)

[![Panel Chat Examples](https://assets.holoviz.org/panel/readme/panel-chat-examples.jpg)](https://holoviz-topics.github.io/panel-chat-examples/)

[![Awesome Panel Gallery](https://assets.holoviz.org/panel/readme/awesome_panel.jpg)](https://www.awesome-panel.org)

## Get started

Develop applications in your favorite notebook or editor environment, including Jupyter(Lab) notebooks, VSCode, Google Colab and many more, [see our getting started guide](https://panel.holoviz.org/getting_started/installation.html#developing-in-different-editors) for more details.

## Support & Feedback

- Usage questions and showcases -> [HoloViz Community](https://holoviz.org/community.html)
- Bug reports and feature requests -> [Github](https://github.com/holoviz/panel)
- Developer discussions -> [Discord](https://discord.gg/rb6gPXbdAr)

For more detail check out the [HoloViz Community Guide](https://holoviz.org/community.html).

## Contributing ❤

Check out the [Contributing Guide](CONTRIBUTING.MD).

## License

Panel is completely free and open-source. It is licensed under the [BSD 3-Clause License](https://opensource.org/licenses/BSD-3-Clause).

## Sponsors

The Panel project is also very grateful for the sponsorship by the organizations and companies below:


[![Build Status](https://github.com/pyro-ppl/numpyro/workflows/CI/badge.svg)](https://github.com/pyro-ppl/numpyro/actions)
[![Documentation Status](https://readthedocs.org/projects/numpyro/badge/?version=latest)](https://numpyro.readthedocs.io/en/latest/?badge=latest)
[![Latest Version](https://badge.fury.io/py/numpyro.svg)](https://pypi.python.org/pypi/numpyro)
# NumPyro

Probabilistic programming powered by [JAX](https://github.com/google/jax) for autograd and JIT compilation to GPU/TPU/CPU.

[Docs and Examples](https://num.pyro.ai) | [Forum](https://forum.pyro.ai/)

----------------------------------------------------------------------------------------------------

## What is NumPyro?

NumPyro is a lightweight probabilistic programming library that provides a NumPy backend for [Pyro](https://github.com/pyro-ppl/pyro). We rely on [JAX](https://github.com/google/jax) for automatic differentiation and JIT compilation to GPU / CPU. NumPyro is under active development, so beware of brittleness, bugs, and changes to the API as the design evolves.

NumPyro is designed to be *lightweight* and focuses on providing a flexible substrate that users can build on:

 - **Pyro Primitives:** NumPyro programs can contain regular Python and NumPy code, in addition to [Pyro primitives](https://pyro.ai/examples/intro_part_i.html) like `sample` and `param`. The model code should look very similar to Pyro except for some minor differences between PyTorch and Numpy's API. See the [example](https://github.com/pyro-ppl/numpyro#a-simple-example---8-schools) below.
 - **Inference algorithms:** NumPyro supports a number of inference algorithms, with a particular focus on MCMC algorithms like Hamiltonian Monte Carlo, including an implementation of the No U-Turn Sampler. Additional MCMC algorithms include [MixedHMC](https://num.pyro.ai/en/latest/mcmc.html#numpyro.infer.mixed_hmc.MixedHMC) (which can accommodate discrete latent variables) as well as [HMCECS](https://num.pyro.ai/en/latest/mcmc.html#numpyro.infer.hmc_gibbs.HMCECS) (which only computes the likelihood for subsets of the data in each iteration). One of the motivations for NumPyro was to speed up Hamiltonian Monte Carlo by JIT compiling the verlet integrator that includes multiple gradient computations. With JAX, we can compose `jit` and `grad` to compile the entire integration step into an XLA optimized kernel. We also eliminate Python overhead by JIT compiling the entire tree building stage in NUTS (this is possible using [Iterative NUTS](https://github.com/pyro-ppl/numpyro/wiki/Iterative-NUTS)). There is also a basic Variational Inference implementation together with many flexible (auto)guides for Automatic Differentiation Variational Inference (ADVI). The variational inference implementation supports a number of features, including support for models with discrete latent variables (see [TraceGraph_ELBO](https://num.pyro.ai/en/latest/svi.html#numpyro.infer.elbo.TraceGraph_ELBO) and [TraceEnum_ELBO](https://num.pyro.ai/en/latest/svi.html#numpyro.infer.elbo.TraceEnum_ELBO)).
 - **Distributions:** The [numpyro.distributions](https://numpyro.readthedocs.io/en/latest/distributions.html) module provides distribution classes, constraints and bijective transforms. The distribution classes wrap over samplers implemented to work with JAX's [functional pseudo-random number generator](https://github.com/google/jax#random-numbers-are-different). The design of the distributions module largely follows from [PyTorch](https://pytorch.org/docs/stable/distributions.html). A major subset of the API is implemented, and it contains most of the common distributions that exist in PyTorch. As a result, Pyro and PyTorch users can rely on the same API and batching semantics as in `torch.distributions`. In addition to distributions, `constraints` and `transforms` are very useful when operating on distribution classes with bounded support. Finally, distributions from TensorFlow Probability ([TFP](https://num.pyro.ai/en/latest/distributions.html?highlight=tfp#numpyro.contrib.tfp.distributions.TFPDistribution)) can directly be used in NumPyro models.
 - **Effect handlers:** Like Pyro, primitives like `sample` and `param` can be provided nonstandard interpretations using effect-handlers from the [numpyro.handlers](https://numpyro.readthedocs.io/en/latest/handlers.html) module, and these can be easily extended to implement custom inference algorithms and inference utilities.

## A Simple Example - 8 Schools

Let us explore NumPyro using a simple example. We will use the eight schools example from Gelman et al., Bayesian Data Analysis: Sec. 5.5, 2003, which studies the effect of coaching on SAT performance in eight schools.

The data is given by:

```python
>>> import numpy as np

>>> J = 8
>>> y = np.array([28.0, 8.0, -3.0, 7.0, -1.0, 1.0, 18.0, 12.0])
>>> sigma = np.array([15.0, 10.0, 16.0, 11.0, 9.0, 11.0, 10.0, 18.0])

```
, where `y` are the treatment effects and `sigma` the standard error. We build a hierarchical model for the study where we assume that the group-level parameters `theta` for each school are sampled from a Normal distribution with unknown mean `mu` and standard deviation `tau`, while the observed data are in turn generated from a Normal distribution with mean and standard deviation given by `theta` (true effect) and `sigma`, respectively. This allows us to estimate the population-level parameters `mu` and `tau` by pooling from all the observations, while still allowing for individual variation amongst the schools using the group-level `theta` parameters.

```python
>>> import numpyro
>>> import numpyro.distributions as dist

>>> # Eight Schools example
... def eight_schools(J, sigma, y=None):
...     mu = numpyro.sample('mu', dist.Normal(0, 5))
...     tau = numpyro.sample('tau', dist.HalfCauchy(5))
...     with numpyro.plate('J', J):
...         theta = numpyro.sample('theta', dist.Normal(mu, tau))
...         numpyro.sample('obs', dist.Normal(theta, sigma), obs=y)

```

Let us infer the values of the unknown parameters in our model by running MCMC using the No-U-Turn Sampler (NUTS). Note the usage of the `extra_fields` argument in [MCMC.run](https://num.pyro.ai/en/latest/mcmc.html#numpyro.infer.mcmc.MCMC.run). By default, we only collect samples from the target (posterior) distribution when we run inference using `MCMC`. However, collecting additional fields like potential energy or the acceptance probability of a sample can be easily achieved by using the `extra_fields` argument. For a list of possible fields that can be collected, see the [HMCState](https://num.pyro.ai/en/latest/mcmc.html#numpyro.infer.hmc.HMCState) object. In this example, we will additionally collect the `potential_energy` for each sample.

```python
>>> from jax import random
>>> from numpyro.infer import MCMC, NUTS

>>> nuts_kernel = NUTS(eight_schools)
>>> mcmc = MCMC(nuts_kernel, num_warmup=500, num_samples=1000)
>>> rng_key = random.PRNGKey(0)
>>> mcmc.run(rng_key, J, sigma, y=y, extra_fields=('potential_energy',))

```

We can print the summary of the MCMC run, and examine if we observed any divergences during inference. Additionally, since we collected the potential energy for each of the samples, we can easily compute the expected log joint density.

```python
>>> mcmc.print_summary()  # doctest: +SKIP

                mean       std    median      5.0%     95.0%     n_eff     r_hat
        mu      4.14      3.18      3.87     -0.76      9.50    115.42      1.01
       tau      4.12      3.58      3.12      0.51      8.56     90.64      1.02
  theta[0]      6.40      6.22      5.36     -2.54     15.27    176.75      1.00
  theta[1]      4.96      5.04      4.49     -1.98     14.22    217.12      1.00
  theta[2]      3.65      5.41      3.31     -3.47     13.77    247.64      1.00
  theta[3]      4.47      5.29      4.00     -3.22     12.92    213.36      1.01
  theta[4]      3.22      4.61      3.28     -3.72     10.93    242.14      1.01
  theta[5]      3.89      4.99      3.71     -3.39     12.54    206.27      1.00
  theta[6]      6.55      5.72      5.66     -1.43     15.78    124.57      1.00
  theta[7]      4.81      5.95      4.19     -3.90     13.40    299.66      1.00

Number of divergences: 19

>>> pe = mcmc.get_extra_fields()['potential_energy']
>>> print('Expected log joint density: {:.2f}'.format(np.mean(-pe)))  # doctest: +SKIP
Expected log joint density: -54.55

```

The values above 1 for the split Gelman Rubin diagnostic (`r_hat`) indicates that the chain has not fully converged. The low value for the effective sample size (`n_eff`), particularly for `tau`, and the number of divergent transitions looks problematic. Fortunately, this is a common pathology that can be rectified by using a [non-centered paramaterization](https://mc-stan.org/docs/2_18/stan-users-guide/reparameterization-section.html) for `tau` in our model. This is straightforward to do in NumPyro by using a [TransformedDistribution](https://num.pyro.ai/en/latest/distributions.html#transformeddistribution) instance together with a [reparameterization](https://num.pyro.ai/en/latest/handlers.html#reparam) effect handler. Let us rewrite the same model but instead of sampling `theta` from a `Normal(mu, tau)`, we will instead sample it from a base `Normal(0, 1)` distribution that is transformed using an [AffineTransform](https://num.pyro.ai/en/latest/distributions.html#affinetransform). Note that by doing so, NumPyro runs HMC by generating samples `theta_base` for the base `Normal(0, 1)` distribution instead. We see that the resulting chain does not suffer from the same pathology — the Gelman Rubin diagnostic is 1 for all the parameters and the effective sample size looks quite good!

```python
>>> from numpyro.infer.reparam import TransformReparam

>>> # Eight Schools example - Non-centered Reparametrization
... def eight_schools_noncentered(J, sigma, y=None):
...     mu = numpyro.sample('mu', dist.Normal(0, 5))
...     tau = numpyro.sample('tau', dist.HalfCauchy(5))
...     with numpyro.plate('J', J):
...         with numpyro.handlers.reparam(config={'theta': TransformReparam()}):
...             theta = numpyro.sample(
...                 'theta',
...                 dist.TransformedDistribution(dist.Normal(0., 1.),
...                                              dist.transforms.AffineTransform(mu, tau)))
...         numpyro.sample('obs', dist.Normal(theta, sigma), obs=y)

>>> nuts_kernel = NUTS(eight_schools_noncentered)
>>> mcmc = MCMC(nuts_kernel, num_warmup=500, num_samples=1000)
>>> rng_key = random.PRNGKey(0)
>>> mcmc.run(rng_key, J, sigma, y=y, extra_fields=('potential_energy',))
>>> mcmc.print_summary(exclude_deterministic=False)  # doctest: +SKIP

                   mean       std    median      5.0%     95.0%     n_eff     r_hat
           mu      4.08      3.51      4.14     -1.69      9.71    720.43      1.00
          tau      3.96      3.31      3.09      0.01      8.34    488.63      1.00
     theta[0]      6.48      5.72      6.08     -2.53     14.96    801.59      1.00
     theta[1]      4.95      5.10      4.91     -3.70     12.82   1183.06      1.00
     theta[2]      3.65      5.58      3.72     -5.71     12.13    581.31      1.00
     theta[3]      4.56      5.04      4.32     -3.14     12.92   1282.60      1.00
     theta[4]      3.41      4.79      3.47     -4.16     10.79    801.25      1.00
     theta[5]      3.58      4.80      3.78     -3.95     11.55   1101.33      1.00
     theta[6]      6.31      5.17      5.75     -2.93     13.87   1081.11      1.00
     theta[7]      4.81      5.38      4.61     -3.29     14.05    954.14      1.00
theta_base[0]      0.41      0.95      0.40     -1.09      1.95    851.45      1.00
theta_base[1]      0.15      0.95      0.20     -1.42      1.66   1568.11      1.00
theta_base[2]     -0.08      0.98     -0.10     -1.68      1.54   1037.16      1.00
theta_base[3]      0.06      0.89      0.05     -1.42      1.47   1745.02      1.00
theta_base[4]     -0.14      0.94     -0.16     -1.65      1.45    719.85      1.00
theta_base[5]     -0.10      0.96     -0.14     -1.57      1.51   1128.45      1.00
theta_base[6]      0.38      0.95      0.42     -1.32      1.82   1026.50      1.00
theta_base[7]      0.10      0.97      0.10     -1.51      1.65   1190.98      1.00

Number of divergences: 0

>>> pe = mcmc.get_extra_fields()['potential_energy']
>>> # Compare with the earlier value
>>> print('Expected log joint density: {:.2f}'.format(np.mean(-pe)))  # doctest: +SKIP
Expected log joint density: -46.09

```

Note that for the class of distributions with `loc,scale` parameters such as `Normal`, `Cauchy`, `StudentT`, we also provide a [LocScaleReparam](https://num.pyro.ai/en/latest/reparam.html#loc-scale-decentering) reparameterizer to achieve the same purpose. The corresponding code will be

    with numpyro.handlers.reparam(config={'theta': LocScaleReparam(centered=0)}):
        theta = numpyro.sample('theta', dist.Normal(mu, tau))

Now, let us assume that we have a new school for which we have not observed any test scores, but we would like to generate predictions. NumPyro provides a [Predictive](https://num.pyro.ai/en/latest/utilities.html#numpyro.infer.util.Predictive) class for such a purpose. Note that in the absence of any observed data, we simply use the population-level parameters to generate predictions. The `Predictive` utility conditions the unobserved `mu` and `tau` sites to values drawn from the posterior distribution from our last MCMC run, and runs the model forward to generate predictions.

```python
>>> from numpyro.infer import Predictive

>>> # New School
... def new_school():
...     mu = numpyro.sample('mu', dist.Normal(0, 5))
...     tau = numpyro.sample('tau', dist.HalfCauchy(5))
...     return numpyro.sample('obs', dist.Normal(mu, tau))

>>> predictive = Predictive(new_school, mcmc.get_samples())
>>> samples_predictive = predictive(random.PRNGKey(1))
>>> print(np.mean(samples_predictive['obs']))  # doctest: +SKIP
3.9886456

```

## More Examples

For some more examples on specifying models and doing inference in NumPyro:

 - [Bayesian Regression in NumPyro](https://nbviewer.jupyter.org/github/pyro-ppl/numpyro/blob/master/notebooks/source/bayesian_regression.ipynb) - Start here to get acquainted with writing a simple model in NumPyro, MCMC inference API, effect handlers and writing custom inference utilities.
 - [Time Series Forecasting](https://nbviewer.jupyter.org/github/pyro-ppl/numpyro/blob/master/notebooks/source/time_series_forecasting.ipynb) - Illustrates how to convert for loops in the model to JAX's `lax.scan` primitive for fast inference.
 - [Annotation examples](https://num.pyro.ai/en/stable/examples/annotation.html) - Illustrates how to utilize the enumeration mechanism to perform inference for models with discrete latent variables.
 - [Baseball example](https://github.com/pyro-ppl/numpyro/blob/master/examples/baseball.py) - Using NUTS for a simple hierarchical model. Compare this with the baseball example in [Pyro](https://github.com/pyro-ppl/pyro/blob/dev/examples/baseball.py).
 - [Hidden Markov Model](https://github.com/pyro-ppl/numpyro/blob/master/examples/hmm.py) in NumPyro as compared to [Stan](https://mc-stan.org/docs/2_19/stan-users-guide/hmms-section.html).
 - [Variational Autoencoder](https://github.com/pyro-ppl/numpyro/blob/master/examples/vae.py) - As a simple example that uses Variational Inference with neural networks. [Pyro implementation](https://github.com/pyro-ppl/pyro/blob/dev/examples/vae/vae.py) for comparison.
 - [Gaussian Process](https://github.com/pyro-ppl/numpyro/blob/master/examples/gp.py) - Provides a simple example to use NUTS to sample from the posterior over the hyper-parameters of a Gaussian Process.
 - [Horseshoe Regression](https://github.com/pyro-ppl/numpyro/blob/master/examples/horseshoe_regression.py) - Shows how to implemement generalized linear models equipped with a Horseshoe prior for both binary-valued and real-valued outputs.
 - [Statistical Rethinking with NumPyro](https://github.com/fehiepsi/rethinking-numpyro) - [Notebooks](https://nbviewer.jupyter.org/github/fehiepsi/rethinking-numpyro/tree/master/notebooks/) containing translation of the code in Richard McElreath's [Statistical Rethinking](https://xcelab.net/rm/statistical-rethinking/) book second version, to NumPyro.
 - Other model examples can be found in the [examples](https://num.pyro.ai/en/stable/) site.

Pyro users will note that the API for model specification and inference is largely the same as Pyro, including the distributions API, by design. However, there are some important core differences (reflected in the internals) that users should be aware of. e.g. in NumPyro, there is no global parameter store or random state, to make it possible for us to leverage JAX's JIT compilation. Also, users may need to write their models in a more *functional* style that works better with JAX. Refer to [FAQs](#frequently-asked-questions) for a list of differences.

## Overview of inference algorithms

We provide an overview of most of the inference algorithms supported by NumPyro and offer some guidelines about which inference algorithms may be appropriate for different classes of models.

### MCMC

- [NUTS](https://num.pyro.ai/en/latest/mcmc.html#numpyro.infer.hmc.NUTS), which is an adaptive variant of [HMC](https://num.pyro.ai/en/latest/mcmc.html#numpyro.infer.hmc.HMC), is probably the most commonly used inference algorithm in NumPyro. Note that NUTS and HMC are not directly applicable to models with discrete latent variables, but in cases where the discrete variables have finite support and summing them out (i.e. enumeration) is tractable, NumPyro will automatically sum out discrete latent variables and perform NUTS/HMC on the remaining continuous latent variables.
As discussed above, model [reparameterization](https://num.pyro.ai/en/latest/reparam.html#module-numpyro.infer.reparam) may be important in some cases to get good performance. Note that, generally speaking, we expect inference to be harder as the dimension of the latent space increases. See the [bad geometry](https://num.pyro.ai/en/latest/tutorials/bad_posterior_geometry.html) tutorial for additional tips and tricks.
- [MixedHMC](https://num.pyro.ai/en/latest/mcmc.html#numpyro.infer.mixed_hmc.MixedHMC) can be an effective inference strategy for models that contain both continuous and discrete latent variables.
- [HMCECS](https://num.pyro.ai/en/latest/mcmc.html#numpyro.infer.hmc_gibbs.HMCECS) can be an effective inference strategy for models with a large number of data points. It is applicable to models with continuous latent variables. See [here](https://num.pyro.ai/en/latest/examples/covtype.html) for an example.
- [BarkerMH](https://num.pyro.ai/en/latest/mcmc.html#numpyro.infer.barker.BarkerMH) is a gradient-based MCMC method that may be competitive with HMC and NUTS for some models. It is applicable to models with continuous latent variables.
- [HMCGibbs](https://num.pyro.ai/en/latest/mcmc.html#numpyro.infer.hmc_gibbs.HMCGibbs) combines HMC/NUTS steps with custom Gibbs updates. Gibbs updates must be specified by the user.
- [DiscreteHMCGibbs](https://num.pyro.ai/en/latest/mcmc.html#numpyro.infer.hmc_gibbs.DiscreteHMCGibbs) combines HMC/NUTS steps with Gibbs updates for discrete latent variables. The corresponding Gibbs updates are computed automatically.
- [SA](https://num.pyro.ai/en/latest/mcmc.html#numpyro.infer.sa.SA) is the only MCMC method in NumPyro that does not leverage gradients. It is only applicable to models with continuous latent variables. It is expected to perform best for models whose latent dimension is low to moderate. It may be a good choice for models with non-differentiable log densities. Note that SA generally requires a *very* large number of samples, as mixing tends to be slow. On the plus side individual steps can be fast.

Like HMC/NUTS, all remaining MCMC algorithms support enumeration over discrete latent variables if possible (see [restrictions](https://pyro.ai/examples/enumeration.html#Restriction-1:-conditional-independence)). Enumerated sites need to be marked with `infer={'enumerate': 'parallel'}` like in the [annotation example](https://num.pyro.ai/en/stable/examples/annotation.html).

### Nested Sampling
- [NestedSampler](https://num.pyro.ai/en/latest/contrib.html#nested-sampling) offers a wrapper for [jaxns](https://github.com/Joshuaalbert/jaxns). See [JAXNS's readthedocs](https://jaxns.readthedocs.io/en/latest/) for examples and [Nested Sampling for Gaussian Shells](https://num.pyro.ai/en/stable/examples/gaussian_shells.html) example for how to apply the sampler on numpyro models. Can handle arbitrary models, including ones with discrete RVs, and non-invertible transformations.

### Stochastic variational inference
- Variational objectives
    - [Trace_ELBO](https://num.pyro.ai/en/latest/svi.html#numpyro.infer.elbo.Trace_ELBO) is our basic ELBO implementation.
    - [TraceMeanField_ELBO](https://num.pyro.ai/en/latest/svi.html#numpyro.infer.elbo.TraceMeanField_ELBO) is like `Trace_ELBO` but computes part of the ELBO analytically if doing so is possible.
    - [TraceGraph_ELBO](https://num.pyro.ai/en/latest/svi.html#numpyro.infer.elbo.TraceGraph_ELBO) offers variance reduction strategies for models with discrete latent variables. Generally speaking, this ELBO should always be used for models with discrete latent variables.
    - [TraceEnum_ELBO](https://num.pyro.ai/en/latest/svi.html#numpyro.infer.elbo.TraceEnum_ELBO) offers variable enumeration strategies for models with discrete latent variables. Generally speaking, this ELBO should always be used for models with discrete latent variables when enumeration is possible.
- Automatic guides (appropriate for models with continuous latent variables)
    - [AutoNormal](https://num.pyro.ai/en/latest/autoguide.html#numpyro.infer.autoguide.AutoNormal) and [AutoDiagonalNormal](https://num.pyro.ai/en/latest/autoguide.html#numpyro.infer.autoguide.AutoDiagonalNormal) are our basic mean-field guides. If the latent space is non-euclidean (due to e.g. a positivity constraint on one of the sample sites) an appropriate bijective transformation is automatically used under the hood to map between the unconstrained space (where the Normal variational distribution is defined) to the corresponding constrained space (note this is true for all automatic guides). These guides are a great place to start when trying to get variational inference to work on a model you are developing.
    - [AutoMultivariateNormal](https://num.pyro.ai/en/latest/autoguide.html#numpyro.infer.autoguide.AutoMultivariateNormal) and [AutoLowRankMultivariateNormal](https://num.pyro.ai/en/latest/autoguide.html#numpyro.infer.autoguide.AutoLowRankMultivariateNormal) also construct Normal variational distributions but offer more flexibility, as they can capture correlations in the posterior. Note that these guides may be difficult to fit in the high-dimensional setting.
    - [AutoDelta](https://num.pyro.ai/en/latest/autoguide.html#numpyro.infer.autoguide.AutoDelta) is used for computing point estimates via MAP (maximum a posteriori estimation). See [here](https://github.com/pyro-ppl/numpyro/blob/bbe1f879eede79eebfdd16dfc49c77c4d1fc727c/examples/zero_inflated_poisson.py#L101) for example usage.
    - [AutoBNAFNormal](https://num.pyro.ai/en/latest/autoguide.html#numpyro.infer.autoguide.AutoBNAFNormal) and [AutoIAFNormal](https://num.pyro.ai/en/latest/autoguide.html#numpyro.infer.autoguide.AutoIAFNormal) offer flexible variational distributions parameterized by normalizing flows.
    - [AutoDAIS](https://num.pyro.ai/en/latest/autoguide.html#numpyro.infer.autoguide.AutoDAIS) is a powerful variational inference algorithm that leverages HMC. It can be a good choice for dealing with highly correlated posteriors but may be computationally expensive depending on the nature of the model.
    - [AutoSurrogateLikelihoodDAIS](https://num.pyro.ai/en/latest/autoguide.html#numpyro.infer.autoguide.AutoSurrogateLikelihoodDAIS) is a powerful variational inference algorithm that leverages HMC and that supports data subsampling.
    - [AutoSemiDAIS](https://num.pyro.ai/en/latest/autoguide.html#numpyro.infer.autoguide.AutoSemiDAIS) constructs a posterior approximation like [AutoDAIS](https://num.pyro.ai/en/latest/autoguide.html#numpyro.infer.autoguide.AutoDAIS) for local latent variables but provides support for data subsampling during ELBO training by utilizing a parametric guide for global latent variables.
    - [AutoLaplaceApproximation](https://num.pyro.ai/en/latest/autoguide.html#numpyro.infer.autoguide.AutoLaplaceApproximation) can be used to compute a Laplace approximation.

### Stein Variational Inference

See the [docs](https://num.pyro.ai/en/latest/contrib.html#stein-variational-inference) for more details.

## Installation

> **Limited Windows Support:** Note that NumPyro is untested on Windows, and might require building jaxlib from source. See this [JAX issue](https://github.com/google/jax/issues/438) for more details. Alternatively, you can install [Windows Subsystem for Linux](https://docs.microsoft.com/en-us/windows/wsl/) and use NumPyro on it as on a Linux system. See also [CUDA on Windows Subsystem for Linux](https://developer.nvidia.com/cuda/wsl) and [this forum post](https://forum.pyro.ai/t/numpyro-with-gpu-works-on-windows/2690) if you want to use GPUs on Windows.

To install NumPyro with the latest CPU version of JAX, you can use pip:

```
pip install numpyro
```

In case of compatibility issues arise during execution of the above command, you can instead force the installation of a known
compatible CPU version of JAX with

```
pip install numpyro[cpu]
```

To use **NumPyro on the GPU**, you need to install CUDA first and then use the following pip command:
```
pip install numpyro[cuda] -f https://storage.googleapis.com/jax-releases/jax_cuda_releases.html
```
If you need further guidance, please have a look at the [JAX GPU installation instructions](https://github.com/google/jax#pip-installation-gpu-cuda).

To run **NumPyro on Cloud TPUs**, you can look at some [JAX on Cloud TPU examples](https://github.com/google/jax/tree/master/cloud_tpu_colabs).

For Cloud TPU VM, you need to setup the TPU backend as detailed in the [Cloud TPU VM JAX Quickstart Guide](https://cloud.google.com/tpu/docs/jax-quickstart-tpu-vm).
After you have verified that the TPU backend is properly set up,
you can install NumPyro using the `pip install numpyro` command.

> **Default Platform:** JAX will use GPU by default if CUDA-supported `jaxlib` package is installed. You can use [set_platform](https://num.pyro.ai/en/stable/utilities.html#set-platform) utility `numpyro.set_platform("cpu")` to switch to CPU at the beginning of your program.

You can also install NumPyro from source:

```
git clone https://github.com/pyro-ppl/numpyro.git
cd numpyro
# install jax/jaxlib first for CUDA support
pip install -e .[dev]  # contains additional dependencies for NumPyro development
```

You can also install NumPyro with conda:
```
conda install -c conda-forge numpyro
```

## Frequently Asked Questions

1. Unlike in Pyro, `numpyro.sample('x', dist.Normal(0, 1))` does not work. Why?

   You are most likely using a `numpyro.sample` statement outside an inference context. JAX does not have a global random state, and as such, distribution samplers need an explicit random number generator key ([PRNGKey](https://jax.readthedocs.io/en/latest/jax.random.html#jax.random.PRNGKey)) to generate samples from. NumPyro's inference algorithms use the [seed](https://num.pyro.ai/en/latest/handlers.html#seed) handler to thread in a random number generator key, behind the scenes.

   Your options are:

   - Call the distribution directly and provide a `PRNGKey`, e.g. `dist.Normal(0, 1).sample(PRNGKey(0))`
   - Provide the `rng_key` argument to `numpyro.sample`. e.g. `numpyro.sample('x', dist.Normal(0, 1), rng_key=PRNGKey(0))`.
   - Wrap the code in a `seed` handler, used either as a context manager or as a function that wraps over the original callable. e.g.

        ```python
        with handlers.seed(rng_seed=0):  # random.PRNGKey(0) is used
            x = numpyro.sample('x', dist.Beta(1, 1))    # uses a PRNGKey split from random.PRNGKey(0)
            y = numpyro.sample('y', dist.Bernoulli(x))  # uses different PRNGKey split from the last one
        ```

     , or as a higher order function:

        ```python
        def fn():
            x = numpyro.sample('x', dist.Beta(1, 1))
            y = numpyro.sample('y', dist.Bernoulli(x))
            return y

        print(handlers.seed(fn, rng_seed=0)())
        ```

2. Can I use the same Pyro model for doing inference in NumPyro?

   As you may have noticed from the examples, NumPyro supports all Pyro primitives like `sample`, `param`, `plate` and `module`, and effect handlers. Additionally, we have ensured that the [distributions](https://numpyro.readthedocs.io/en/latest/distributions.html) API is based on `torch.distributions`, and the inference classes like `SVI` and `MCMC` have the same interface. This along with the similarity in the API for NumPy and PyTorch operations ensures that models containing Pyro primitive statements can be used with either backend with some minor changes. Example of some differences along with the changes needed, are noted below:

   - Any `torch` operation in your model will need to be written in terms of the corresponding `jax.numpy` operation. Additionally, not all `torch` operations have a `numpy` counterpart (and vice-versa), and sometimes there are minor differences in the API.
   - `pyro.sample` statements outside an inference context will need to be wrapped in a `seed` handler, as mentioned above.
   - There is no global parameter store, and as such using `numpyro.param` outside an inference context will have no effect. To retrieve the optimized parameter values from SVI, use the [SVI.get_params](https://num.pyro.ai/en/latest/svi.html#numpyro.infer.svi.SVI.get_params) method. Note that you can still use `param` statements inside a model and NumPyro will use the [substitute](https://num.pyro.ai/en/latest/handlers.html#substitute) effect handler internally to substitute values from the optimizer when running the model in SVI.
   - PyTorch neural network modules will need to rewritten as [stax](https://github.com/google/jax#neural-net-building-with-stax), [flax](https://flax.readthedocs.io/en/latest/), or [haiku](https://dm-haiku.readthedocs.io/en/latest/) neural networks. See the [VAE](https://num.pyro.ai/en/latest/examples/vae.html) and [ProdLDA](https://num.pyro.ai/en/stable/examples/prodlda.html) examples for differences in syntax between the two backends.
   - JAX works best with functional code, particularly if we would like to leverage JIT compilation, which NumPyro does internally for many inference subroutines. As such, if your model has side-effects that are not visible to the JAX tracer, it may need to rewritten in a more functional style.

   For most small models, changes required to run inference in NumPyro should be minor. Additionally, we are working on [pyro-api](https://github.com/pyro-ppl/pyro-api) which allows you to write the same code and dispatch it to multiple backends, including NumPyro. This will necessarily be more restrictive, but has the advantage of being backend agnostic. See the [documentation](https://pyro-api.readthedocs.io/en/latest/dispatch.html#module-pyroapi.dispatch) for an example, and let us know your feedback.

3. How can I contribute to the project?

   Thanks for your interest in the project! You can take a look at beginner friendly issues that are marked with the [good first issue](https://github.com/pyro-ppl/numpyro/issues?q=is%3Aissue+is%3Aopen+label%3A%22good+first+issue%22) tag on Github. Also, please feel to reach out to us on the [forum](https://forum.pyro.ai/).

## Future / Ongoing Work

In the near term, we plan to work on the following. Please open new issues for feature requests and enhancements:

 - Improving robustness of inference on different models, profiling and performance tuning.
 - Supporting more functionality as part of the [pyro-api](https://github.com/pyro-ppl/pyro-api) generic modeling interface.
 - More inference algorithms, particularly those that require second order derivatives or use HMC.
 - Integration with [Funsor](https://github.com/pyro-ppl/funsor) to support inference algorithms with delayed sampling.
 - Other areas motivated by Pyro's research goals and application focus, and interest from the community.

## Citing NumPyro

The motivating ideas behind NumPyro and a description of Iterative NUTS can be found in this [paper](https://arxiv.org/abs/1912.11554) that appeared in NeurIPS 2019 Program Transformations for Machine Learning Workshop.

If you use NumPyro, please consider citing:

```
@article{phan2019composable,
  title={Composable Effects for Flexible and Accelerated Probabilistic Programming in NumPyro},
  author={Phan, Du and Pradhan, Neeraj and Jankowiak, Martin},
  journal={arXiv preprint arXiv:1912.11554},
  year={2019}
}
```

as well as

```
@article{bingham2019pyro,
  author    = {Eli Bingham and
               Jonathan P. Chen and
               Martin Jankowiak and
               Fritz Obermeyer and
               Neeraj Pradhan and
               Theofanis Karaletsos and
               Rohit Singh and
               Paul A. Szerlip and
               Paul Horsfall and
               Noah D. Goodman},
  title     = {Pyro: Deep Universal Probabilistic Programming},
  journal   = {J. Mach. Learn. Res.},
  volume    = {20},
  pages     = {28:1--28:6},
  year      = {2019},
  url       = {http://jmlr.org/papers/v20/18-403.html}
}
```


# Passive BLE Monitor integration

![BLE_sensors](https://raw.githubusercontent.com/custom-components/ble_monitor/master/pictures/sensors.jpg)

This custom component for [Home Assistant](https://www.home-assistant.io) passively monitors [many different BLE devices](https://custom-components.github.io/ble_monitor/devices) of several different [brands](https://custom-components.github.io/ble_monitor/by_brand). BLE Monitor can also be used as device tracker for BLE devices with a static MAC address or with the UUID.

## More info

- [Documentation](https://custom-components.github.io/ble_monitor/#introduction)
- [Installation instructions](https://custom-components.github.io/ble_monitor/Installation)
- [Configuration](https://custom-components.github.io/ble_monitor/configuration_params)
- [Supported devices](https://custom-components.github.io/ble_monitor/devices)
- [Forward data from ESPhome](https://custom-components.github.io/ble_monitor/parse_data)
- [DIY sensors](https://custom-components.github.io/ble_monitor/bthome)
- [FAQ](https://custom-components.github.io/ble_monitor/faq)
- [New sensor request](https://custom-components.github.io/ble_monitor/sensor_request)
- [Developer documentation](https://custom-components.github.io/ble_monitor/developer_docs)
- [Forum](https://community.home-assistant.io/t/passive-ble-monitor-integration/)
- [Report issues](https://github.com/custom-components/ble_monitor/issues)

## Supported sensor brands

![BLE_sensors](https://raw.githubusercontent.com/custom-components/ble_monitor/master/pictures/sensors_2.png)

- Acconeer
- Air Mentor
- Amazfit
- ATC (custom firmware for Xiaomi/Qingping sensors)
- BlueMaestro
- Blustream (Taylor TaylorSense, D'Addario Humiditrak)
- BTHome
- b-parasite
- Chef iQ
- Ela
- Govee
- Grundfos
- HolyIOT
- Hörmann
- HHCC
- Inkbird
- iNode
- Jaalee
- Jinou
- Kegtron
- KKM
- Mikrotik
- Moat
- Oras
- Oral-B
- Qingping
- Relsib
- Ruuvitag
- Sensirion
- SensorPush
- SmartDry
- Switchbot
- Teltonika
- Thermobeacon (Thermoplus, Brifit, Oria)
- Thermopro
- Tilt
- Xiaogui (Scale)
- Xiaomi (MiBand)
- Xiaomi (MiBeacon sensors)
- Xiaomi (MiScale)

## Why isn't ESPHome Bluetooth Proxy not working with BLE monitor?

Please note that ESPHome Bluetooth Proxies cannot forward data to BLE monitor. Check out [this page](https://custom-components.github.io/ble_monitor/parse_data) for alternative solutions.

## Important announcement about the future of BLE monitor

Home Assistant 2022.8 has (improved) support for passive BLE devices directly in Home Assistant. For each brand, a core BLE integration will be developed, such that maintenance can be divided over more people, using the latest Bluetooth packages (bleak). I'm working together with the Home Assistant devs to move sensors from BLE Monitor to Home Assistant core integrations. During the transition, BLE monitor will still be available, but it is possible that the core HA Bluetooth integrations will not work niceley parallel to BLE monitor. **If it is not working together (common symptom is that core integrations stop updating after a while), try to enable active scan in the BLE Monitor.** My advise, when all your sensors are available in Home Assistant, make the move. The aim is to have all sensors moved into Home Assistant as core integration. After the move, BLE monitor will probably be deprecated. If you want to help moving sensors from BLE monitor, feel free to help. Check out the links below.

**Some interesting links**

- Most pypi packages for the BLE parsing will be developed and collected here: https://github.com/Bluetooth-Devices

**Available as official HA integration**

The following integrations are available as official Home Assistant integration.
- BlueMaestro
- b-parasite (will be using BTHome with new firmware)
- BTHome
- Govee
- HHCC
- iBeacon
- Inkbird
- Kegtron
- Moat
- Oral-B
- Qingping
- RuuviTag
- SensorPush
- Sensirion (MyCO2 gadget)
- Thermobeacon
- Thermopro / Sensorpro
- Thermobeacon (Thermoplus, Brifit, Oria)
- Tilt
- Xiaomi (part 1 and 2)
- Device tracking based on MAC address (Bluetooth LE tracker integration)