# Overview
This is the next gen (v4) of AutoRest Java generator. It's built on AutoRest v3, written in Java, and supports OpenAPI3. It generates clients that work with `com.azure:azure-core`.

# Prerequisites
You need to have the following installed on your machine:

- Node.JS v12+
- Java 11+
- Maven 3

You need to have [autorest](https://www.npmjs.com/package/autorest) installed through NPM:

```bash
npm install -g autorest
```

# Usage
To use the latest released preview(https://github.com/Azure/autorest.java/releases), run
```bash
autorest --java
    --use:@autorest/java@4.x.x
    --input-file:path/to/specs.json
    --output-folder:where/to/generate/java/files
    --namespace:specified.java.package
```
The first time running it will take a little longer to download and install all the components.

To build from source code, clone this repo and checkout to main branch. Make sure all prerequisites are met, and run

```bash
mvn package -P local
```

This will build a file `javagen-jar-with-dependencies.jar` under `javagen` module, a `preprocess-jar-with-dependencies.jar` under `preprocessor` module, a `fluentgen-jar-with-dependencies.jar` under `fluentgen` module, and a `fluentnamer-jar-with-dependencies.jar` under `fluentnamer` module.

And then run AutoRest

```bash
autorest --java
    --use:where/this/repo/is/cloned/autorest.java
    --input-file:path/to/specs.json
    --output-folder:where/to/generate/java/files
    --namespace:specified.java.package
```

Java files will be generated under `where/to/generate/java/files/src/main/java/specified/java/package`.

To debug, add `--java.debugger` to the argument list. The JVM will suspend at the beginning of the execution. Then attach a remote debugger in your IDE to `localhost:5005`. **Make sure you detach the debugger before killing the AutoRest process. Otherwise it will fail to shutdown the JVM and leave it orphaned. (which can be killed in the Task Manager)**

# Recommended Settings

- `use-default-http-status-code-to-exception-type-mapping: true`, use subclass of `HttpResponseException` defined in azure-core, when service returns unexpected status code.
- `generic-response-type: true`, use `ResponseBase<>`, instead a subclass of `ResponseBase<>`, for response with HTTP headers. Performance and reflective access improvement.
- `required-fields-as-ctor-args`, create a constructor taking required properties, for model classes. The correctness of this constructor highly depends on the correctness of Swagger.
- `output-model-immutable: true`, make model classes immutable, if model is used only in response.
- `enable-sync-stack: true`, let sync method invoke sync RestProxy method.
- `enable-page-size: true`, let `maxpagesize` query parameter be supplied via `byPage` API in `PagedFlux` or `PagedIterable`.
- `use-key-credential: true`, use `KeyCredential` in builder for API key.

- `generate-client-as-impl: true`, generate Client in implementation package, for customization.
- `models-subpackage: implementation.models`, generate model classes in implementation package.
- `custom-types-subpackage: models`, generate selected model classes in models package.

# Settings
Settings can be provided on the command line through `--name:value` or in a README file through `name: value`. The list of settings for AutoRest in general can be found at https://github.com/Azure/autorest/blob/main/docs/user/command-line-interface.md. The list of settings for AutoRest.Java specifically are listed below:

|Option                                                                 | Description |
|------------------|-------------|
|`--client-side-validations`|Generate validations for required parameters and required model properties. Default is false.|
|`--generate-client-as-impl`|Append "Impl" to the names of service clients and method groups and place them in the `implementation` sub-package. Default is false.|
|`--generate-client-interfaces`|Implies `--generate-client-as-impl` and generates interfaces for all the "Impl"s. Default is false.|
|`--generate-sync-async-clients`|Implies `--generate-client-as-impl` and generates sync and async convenience layer clients for all the "Impl"s. Default is false.|
|`--generate-builder-per-client`|Requires `--generate-sync-async-clients`, and generates one ClientBuilder for each Client. Default is false.|
|`--implementation-subpackage=STRING`|The sub-package that the Service client and Method Group client implementation classes will be put into. Default is `implementation`.|
|`--models-subpackage=STRING`|The sub-package that Enums, Exceptions, and Model types will be put into. Default is `models`.|
|`--sync-methods=all\|essential\|none`|Specifies mode for generating sync wrappers. Supported value are
  `essential` - generates only one sync returning body or header (default)
  `all` - generates one sync method for each async method
  `none` - does not generate any sync methods|
|`--required-parameter-client-methods`|Indicates whether client method overloads with only required parameters should be generated. Default is false.|
|`--custom-types=COMMA,SEPARATED,STRINGS`|Specifies a list of files to put in the package specified in `--custom-types-subpackage`.|
|`--custom-types-subpackage=STRING`|The sub-package that the custom types should be generated in. The types that custom types reference, or inherit from will also be automatically moved to this sub-package. **Recommended usage**: You can set this value to `models` and set `--models-subpackage=implementation.models`to generate models to `implementation.models` by default and pick specific models to be public through `--custom-types=`.|
|`--client-type-prefix=STRING`|The prefix that will be added to each generated client type.|
|`--service-interface-as-public`|Indicates whether to generate service interfaces as public. This resolves `SecurityManager` issues to prevent reflectively access non-public APIs. Default is true.|
|`--client-flattened-annotation-target=TYPE,FIELD,NONE,DISABLED`|Indicates the target of `@JsonFlatten` annotation for `x-ms-client-flatten`. Default is `TYPE`.|
|`--disable-client-builder`|Indicates whether to disable generating the `ClientBuilder` class. This is for SDK that already contains a hand-written `ClientBuilder` class. Default is false.|
|`--polling`|Configures how to generate long running operations. See [Polling Configuration](#polling-configuration) to see more details on how to use this flag.|
|`--service-name`|Service name used in Client class and other documentations. If not provided, service name is deduced from `title` configure (from swagger or readme).|
|`--partial-update`|Indicates whether to support partial update for `Client`/`AsyncClient` classes and `ClientBuilder` class.|
|`--default-http-exception-type`|The fully-qualified class name that should be used as the default HTTP exception type. The class must extend from `HttpResponseException`.|
|`--use-default-http-status-code-to-exception-type-mapping`|Indicates whether a default HTTP status code to exception mapping should be used if one isn't provided.|
|`--http-status-code-to-exception-type-mapping`|The HTTP status code to exception mapping that should be used. All exception types must be fully-qualified and extend from `HttpResponseException`.|
|`--generic-response-type`|Indicates that generic response types are used instead of named response types that extend the generic type.|
|`--enable-sync-stack`|Indicates that sync method invokes sync RestProxy method. By default, sync method invokes async RestProxy method.|
|`--required-fields-as-ctor-args`|Indicates that class models have constructor taking required properties.|
|`--output-model-immutable`|Indicates that output-only models be generated as immutable, and without public constructor.|
|`--use-input-stream-for-binary`|Indicates that `InputStream` is used for binary response body. By default, `BinaryData` is used.|
|`--enable-page-size`|Indicates that `maxpagesize` query parameter be supplied via `byPage` API in `PagedFlux` or `PagedIterable`, instead of via client method parameter.|
|`--use-key-credential`|Indicates that builder uses `KeyCredential` for API key.|
|`--graal-vm-config`|Generates GraalVM config under `resources/META-INF/native-image`.|

## Settings for minimal data-plane clients

`data-plane` option enables the generator to generate code for minimal data-plane clients.

`data-plane` option will change the default value for some vanilla options.
For example, `generate-client-interfaces`, `generate-client-as-impl`, `generate-sync-async-clients`, `generate-builder-per-client`, `enable-sync-stack`, `required-fields-as-ctor-args` option is by default `true`.
`polling` is by default enabled as default settings globally (`polling={}`).

`sdk-integration` option can be used for integrating to [azure-sdk-for-java](https://github.com/Azure/azure-sdk-for-java/).

`generate-samples` option can be used for generating samples for client.
`generate-tests` option can be used for generating tests for client.

See [documentation](docs/generate/dataplane.md) for frequently used options, and their effect on generated clients.

## Additional settings for Fluent

`fluent` option enables the generator extension for [Azure Management Libraries for Java](https://aka.ms/azsdk/java/mgmt).

Following settings only works when `fluent` option is specified.

| Option      | Description |
| ----------- | ----------- |
| `--fluent` | Enum. `LITE` for Fluent Lite; `PREMIUM` for Fluent Premium. Case insensitive. Default is `PREMIUM` if provided as other values. |
| `--fluent-subpackage` | String. The sub-package that vanilla client and builder will be put into. Default is `fluent`. |
| `--pom-file` | String. Name for Maven POM file. Default is `pom.xml`. |
| `--package-version` | String. Version number for Maven artifact. Default is `1.0.0-beta.1`. |
| `--service-name` | String. Service name used in Manager class and other documentations. If not provided, service name is deduced from `title` configure (from swagger or readme). |
| `--sdk-integration` | Boolean. Integrate to [azure-sdk-for-java](https://github.com/Azure/azure-sdk-for-java/). Default is `false`. Provide `output-folder` as absolute path for best performance. |
| `--generate-samples` | Boolean. Generate samples from `x-ms-examples` in swagger. Default is `false`. |
| `--add-inner` | CSV. Treat as inner class (move to `fluent.models` namespace, append `Inner` to class name). |
| `--remove-inner` | CSV. Exclude from inner classes. |
| `--rename-model` | CSV. Rename classes. Each item is of pattern `from:to`. |
| `--remove-model` | CSV. Remove classes. |
| `--preserve-model` | CSV. Preserve classes from clean-up. |
| `--remove-operation-group` | CSV. Remove operation groups. |
| `--name-for-ungrouped-operations` | String. Name for ungrouped operation group. Default to `ResourceProviders` for Lite. |
| `--resource-collection-associations` | Array. Prioritized association from resource model to resource collection. |

`fluent` option will change the default value for some vanilla options.
For example, `generate-client-interfaces`, `required-parameter-client-methods` option is by default `true`.

The code formatter would require Java 11+ runtime.

## Polling configuration
Polling configurations can be set through `--polling` setting globally or for each operation. The format is a key value map specified below:

```
polling:
  {operationId}:
    strategy: {strategy}
    sync-strategy: {sync-strategy}
    intermediate-type: {intermediate-type}
    final-type: {final-type}
    poll-interval: {poll-interval}
```

With the fields specified below:

|Field|Type|Required|Description|Example|
|-----|----|--------|-----------|-------|
|operationId|String|true|The `operationId` of the operation. For global polling configuration, use `default`. Case insensitive.|`Pets_put`|
|strategy|String|false|The invocation to construct a polling strategy. Use fully qualified class name if outside the implementation subpackage specified in `namespace` & `implementation-subpackage`. Use dynamic literals `{httpPipeline}`, `{context}`, `{serializerAdapter}`, `{endpoint}`, `{serviceVersion}`, `{intermediate-type}`, `{final-type}` if these components are required to construct the polling strategy. Default is `new DefaultPollingStrategy<>(new PollingStrategyOptions({httpPipeline}).setEndpoint({endpoint}).setContext({context}).setServiceVersion({serviceVersion}))`.|`new com.azure.core.util.polling.OperationResourcePollingStrategy<>({httpPipeline}, {context})`|
|sync-stragety|String|false|Similar to `stragety`. It refers to the synchronous operation resource polling strategy.||
|intermediate-type|String|false|The type of the polling intermediate type. Use fully qualified class name if outside the base package specified in `namespace`. Default is the return type specified on the operation in Swagger, or `BinaryData` if the operation returns `void`.|`PollResult`,`com.azure.core.util.BinaryData`|
|final-type|String|false|The type of the final result type. Use fully qualified class name if outside the base package specified in `namespace`. Default is the return type specified on the operation in Swagger, or `BinaryData` if the operation returns `void`.|`Pet`,`com.azure.core.util.BinaryData`|
|poll-interval|integer|false|The default interval in seconds to poll with (can be modified by users in `PollerFlux` and `SyncPoller`. Default is 1.|30|

To use default settings globally, use `--polling={}`.

## HTTP Status Code to Exception Type Handling

By default, Swagger definitions will contain an exception type to be thrown when any error HTTP status code is seen. Swaggers
may also define the error HTTP status code to exception type mapping, but isn't always the case. Or, it may be needed to change
the default exception type to a custom type. In these cases it is possible to configure the error HTTP status code to exception
type handling by using the configurations `--default-http-exception-type`, `--use-default-http-status-code-to-exception-type-mapping`,
and `--http-status-code-to-exception-type-mapping`.

NOTE: Using the configurations on an already shipped library will likely result in runtime breaking changes as most
libraries are already using a sub-class type of `HttpResponseException` and changing sub-types is a breaking change.

### Default Http Exception Type

The default exception type is used for all error HTTP response status codes that do not have an explicit exception type
configured. Generally, there is no error HTTP response status code to exception type mapping, so this will become your
general exception type. The following is an example of configuring `com.azure.core.exception.HttpResponseException` as
the default exception type:

`autorest --default-http-exception-type=com.azure.core.exception.HttpResponseException`

or in the Swagger README configuration:

```
default-http-exception-type: com.azure.core.exception.HttpResponseException
```

If this configuration isn't used the default exception type will be based on the Swagger definition.

### Use Default HTTP Status Code to Exception Type Mapping

The default error HTTP status code to exception type map is the following:

|Status Code|Exception Type|
|-----------|--------------|
|401|com.azure.core.exception.ClientAuthenticationException|
|404|com.azure.core.exception.ResourceNotFoundException|
|409|com.azure.core.exception.ResourceModifiedException|

By default, this mapping isn't used, to use it pass the `--use-default-http-status-code-to-exception-type-mapping` to
the generator or in the Swagger README configuration:

```
use-default-http-status-code-to-exception-type-mapping: true
```

### Custom HTTP Status Code to Exception Type Mapping

If you need to have a custom mapping it is possible to do so by using `--http-status-code-to-exception-type-mapping`.
This requires additional configuration in the Swagger README definition as it is a mapping of HTTP status code to
exception. The following is an example:

```
http-status-code-to-exception-type-mapping:
  404: com.azure.core.exception.ResourceNotFoundException
  412: com.azure.core.exception.ResourceExistsException
```

### Example

The HTTP status code to exception type handling configurations can be used together to offer a nice baseline default
with additional customizations as need. For example, if you wanted to configure the default exception type, use most of
the default error HTTP status code to exception type, and customize one error HTTP status code exception type the following
configurations can be used.

```
default-http-exception-type: com.azure.core.exception.HttpResponseException
use-default-http-status-code-to-exception-type-mapping: true
http-status-code-to-exception-type-mapping:
  404: com.azure.core.exception.ResourceExistsException
```

This results in the following error HTTP response to exception type mapping:

|Status Code|Exception Type|
|-----------|--------------|
|401|com.azure.core.exception.ClientAuthenticationException|
|404|com.azure.core.exception.ResourceExistsException|
|409|com.azure.core.exception.ResourceModifiedException|
|*|com.azure.core.exception.HttpResponseException|

Notices how 404 changes from the default `com.azure.core.exception.ResourceNotFoundException` to
`com.azure.core.exception.ResourceExistsException`.

# Minimal Data-Plane Clients

You can generate the output as minimal data-plane clients with `--data-plane` flag.
The models will not be generated and the methods in the clients will be generated as [protocol methods](https://github.com/Azure/azure-sdk-for-java/wiki/Protocol-Methods).

The generated code has the following structure:

```
- pom.xml
- /src/main/java
  |
  - module-info.java
  - /com/azure/<group>/<service>
    |
    - <Service>Builder.java
    - <Service>Client.java
    - <Service>AsyncClient.java
    - /implementation
      |
      - <Service>ClientImpl.java
```

and requires latest `azure-core` as a dependency.

# Customizations

For guidance on using the post-code generation customization framework see its [documentation](https://github.com/Azure/autorest.java/tree/main/customization-base/README.md).

# Project structure
## extension-base
This contains the base classes and utilities for creating an AutoRest extension in Java. It handles the JSON RPC communications with AutoRest core, provides JSON and YAML parsers, and provides the POJO models for the code model output from [modelerfour](https://github.com/Azure/autorest.modelerfour/).

Extend from `NewPlugin.java` class if you are writing a new extension in Java.

## javagen
This contains the actual generator extension, including mappers that maps a code model to a Java client model, and templates that writes the Java client models into .java files.

## fluentgen
This contains the [generator extension for Azure Management Libraries](#additional-settings-for-fluent).

## tests
This contains the generated classes from the [test swaggers](https://github.com/Azure/autorest.testserver/tree/main/swagger) in `src/main`. The code here should always be kept up-to-date with the output of the generator in `javagen`.

This also contains test code for these generated code under `src/test`. Running the tests will hit the test server running locally (see https://github.com/Azure/autorest.testserver for instructions) and verify the correctness of the generated code.

# Contributing

This project welcomes contributions and suggestions.  Most contributions require you to agree to a
Contributor License Agreement (CLA) declaring that you have the right to, and actually do, grant us
the rights to use your contribution. For details, visit https://cla.microsoft.com.

When you submit a pull request, a CLA-bot will automatically determine whether you need to provide
a CLA and decorate the PR appropriately (e.g., label, comment). Simply follow the instructions
provided by the bot. You will only need to do this once across all repos using our CLA.

This project has adopted the [Microsoft Open Source Code of Conduct](https://opensource.microsoft.com/codeofconduct/).
For more information see the [Code of Conduct FAQ](https://opensource.microsoft.com/codeofconduct/faq/) or
contact [opencode@microsoft.com](mailto:opencode@microsoft.com) with any additional questions or comments.

### Autorest plugin configuration
- Please don't edit this section unless you're re-configuring how the Java extension plugs in to AutoRest
AutoRest needs the below config to pick this up as a plug-in - see https://github.com/Azure/autorest/blob/main/docs/developer/architecture/AutoRest-extension.md

#### Javagen

``` yaml $(java) && !$(fluent) && !$(android)
use: $(this-folder)/javagen
```

``` yaml $(java) && $(fluent) && !$(android)
use: $(this-folder)/fluentgen
```

```  yaml $(java) && $(android)
use: $(this-folder)/androidgen
```

#### Help

```yaml
help-content:
  java:
    activationScope: java
    categoryFriendlyName: Java Generator
    settings:
      - key: client-side-validations
        type: bool
        description: Generate validations for required parameters and required model properties. Default is false.
      - key: generate-client-as-impl
        type: bool
        description: Append "Impl" to the names of service clients and method groups and place them in the `implementation` sub-package. Default is false.
      - key: generate-client-interfaces
        type: bool
        description: Implies `--generate-client-as-impl` and generates interfaces for all the "Impl"s. Default is false.
      - key: generate-sync-async-clients
        type: bool
        description: Implies `--generate-client-as-impl` and generates sync and async convenience layer clients for all the "Impl"s. Default is false.
      - key: generate-builder-per-client
        type: bool
        description: Requires `--generate-sync-async-clients`, and generates one ClientBuilder for each Client. Default is false.
      - key: implementation-subpackage=STRING
        type: string
        description: The sub-package that the Service client and Method Group client implementation classes will be put into. Default is `implementation`.
      - key: models-subpackage=STRING
        type: string
        description: The sub-package that Enums, Exceptions, and Model types will be put into. Default is `models`.
      - key: sync-methods
        type: string
        description: \[all|essential|none] Specifies mode for generating sync wrappers. Supported value are <br>  `essential` - generates only one sync returning body or header (default) <br>  `all` - generates one sync method for each async method<br>  `none` - does not generate any sync methods
      - key: required-parameter-client-methods
        type: bool
        description: Indicates whether client method overloads with only required parameters should be generated. Default is false.
      - key: custom-types
        type: string
        description: \[COMMA,SEPARATED,STRINGS] Specifies a list of files to put in the package specified in `--custom-types-subpackage`.
      - key: custom-types-subpackage
        type: string
        description: The sub-package that the custom types should be generated in. The types that custom types reference, or inherit from will also be automatically moved to this sub-package. **Recommended usage** You can set this value to `models` and set `--models-subpackage=implementation.models`to generate models to `implementation.models` by default and pick specific models to be public through `--custom-types=`.
      - key: client-type-prefix
        type: string
        description: The prefix that will be added to each generated client type.
      - key: client-flattened-annotation-target
        type: string
        description: \[TYPE,FIELD,NONE,DISABLED] Indicates the target of `@JsonFlatten` annotation for `x-ms-client-flatten`. Default is `TYPE`.
      - key: disable-client-builder
        type: bool
        description: Indicates whether to disable generating the `ClientBuilder` class. This is for SDK that already contains a hand-written `ClientBuilder` class. Default is false.
      - key: polling
        type: string
        description: Configures how to generate long running operations. See [Polling Configuration](https://github.com/Azure/autorest.java#polling-configuration) to see more details on how to use this flag.
      - key: service-name
        type: string
        description: Service name used in Client class and other documentations. If not provided, service name is deduced from `title` configure (from swagger or readme).
      - key: partial-update
        type: bool
        description: Indicates whether to support partial update for `Client`/`AsyncClient` classes and `ClientBuilder` class. Default is false.
      - key: enable-sync-stack
        type: bool
        description: Indicates that sync method invokes sync RestProxy method. By default, sync method invokes async RestProxy method.
      - key: required-fields-as-ctor-args
        type: bool
        description: Indicates that class models have constructor taking required properties.
      - key: output-model-immutable
        type: bool
        description: Indicates that output-only models be generated as immutable, and without public constructor. Default is false.
      - key: data-plane
        type: bool
        description:  Indicates whether to generate code for minimal clients. Default is false.

  javafluent:
    activationScope: fluent
    categoryFriendlyName: Java fluent Generator
    settings:
      - key: fluent
        type: string
        description: Enum. `LITE` for Fluent Lite; `PREMIUM` for Fluent Premium. Case insensitive. Default is `PREMIUM` if provided as other values.
      - key: fluent-subpackage
        type: string
        description: String. The sub-package that vanilla client and builder will be put into. Default is `fluent`.
      - key: pom-file
        type: string
        description: String. Name for Maven POM file. Default is `pom.xml`.
      - key: package-version
        type: string
        description: String. Version number for Maven artifact. Default is `1.0.0-beta.1`.
      - key: service-name
        type: string
        description: String. Service name used in Manager class and other documentations. If not provided, service name is deduced from `title` configure (from swagger or readme).
      - key: sdk-integration
        type: bool
        description: Boolean. Integrate to [azure-sdk-for-java](https://github.com/Azure/azure-sdk-for-java/). Default is `false`. Provide `output-folder` as absolute path for best performance.
      - key: generate-samples
        type: bool
        description: Boolean. Generate samples from `x-ms-examples` in swagger. Default is `false`.
      - key: add-inner
        type: string
        description: CSV. Treat as inner class (move to `fluent.models` namespace, append `Inner` to class name).
      - key: remove-inner
        type: string
        description: CSV. Exclude from inner classes.
      - key: rename-model
        type: string
        description: CSV. Rename classes. Each item is of pattern `from:to`.
      - key: remove-model
        type: string
        description: CSV. Remove classes.
      - key: preserve-model
        type: string
        description: CSV. Preserve classes from clean-up.
      - key: remove-operation-group
        type: string
        description: CSV. Remove operation groups.
      - key: name-for-ungrouped-operations
        type: string
        description: String. Name for ungrouped operation group. Default to `ResourceProviders` for Lite.
```


gRPC-Java - An RPC library and framework
========================================

    Homepage:
    grpc.io

    Mailing List:
    grpc-io@googlegroups.com

[![Join the chat at https://gitter.im/grpc/grpc](https://badges.gitter.im/grpc/grpc.svg)](https://gitter.im/grpc/grpc?utm_source=badge&utm_medium=badge&utm_campaign=pr-badge&utm_content=badge)
[![GitHub Actions Linux Testing](https://github.com/grpc/grpc-java/actions/workflows/testing.yml/badge.svg?branch=master)](https://github.com/grpc/grpc-java/actions/workflows/testing.yml?branch=master)
[![Line Coverage Status](https://coveralls.io/repos/grpc/grpc-java/badge.svg?branch=master&service=github)](https://coveralls.io/github/grpc/grpc-java?branch=master)
[![Branch-adjusted Line Coverage Status](https://codecov.io/gh/grpc/grpc-java/branch/master/graph/badge.svg)](https://codecov.io/gh/grpc/grpc-java)

Supported Platforms
-------------------

gRPC-Java supports Java 8 and later. Android minSdkVersion 21 (Lollipop) and
later are supported with [Java 8 language desugaring][android-java-8].

TLS usage on Android typically requires Play Services Dynamic Security Provider.
Please see the [Security Readme](SECURITY.md).

Older Java versions are not directly supported, but a branch remains available
for fixes and releases. See [gRFC P5 JDK Version Support
Policy][P5-jdk-version-support].

Java version | gRPC Branch
------------ | -----------
7            | 1.41.x

[android-java-8]: https://developer.android.com/studio/write/java8-support#supported_features
[P5-jdk-version-support]: https://github.com/grpc/proposal/blob/master/P5-jdk-version-support.md#proposal

Getting Started
---------------

For a guided tour, take a look at the [quick start
guide](https://grpc.io/docs/languages/java/quickstart) or the more explanatory [gRPC
basics](https://grpc.io/docs/languages/java/basics).

The [examples](https://github.com/grpc/grpc-java/tree/v1.64.0/examples) and the
[Android example](https://github.com/grpc/grpc-java/tree/v1.64.0/examples/android)
are standalone projects that showcase the usage of gRPC.

Download
--------

Download [the JARs][]. Or for Maven with non-Android, add to your `pom.xml`:
```xml
<dependency>
  <groupId>io.grpc</groupId>
  <artifactId>grpc-netty-shaded</artifactId>
  <version>1.64.0</version>
  <scope>runtime</scope>
</dependency>
<dependency>
  <groupId>io.grpc</groupId>
  <artifactId>grpc-protobuf</artifactId>
  <version>1.64.0</version>
</dependency>
<dependency>
  <groupId>io.grpc</groupId>
  <artifactId>grpc-stub</artifactId>
  <version>1.64.0</version>
</dependency>
<dependency> <!-- necessary for Java 9+ -->
  <groupId>org.apache.tomcat</groupId>
  <artifactId>annotations-api</artifactId>
  <version>6.0.53</version>
  <scope>provided</scope>
</dependency>
```

Or for Gradle with non-Android, add to your dependencies:
```gradle
runtimeOnly 'io.grpc:grpc-netty-shaded:1.64.0'
implementation 'io.grpc:grpc-protobuf:1.64.0'
implementation 'io.grpc:grpc-stub:1.64.0'
compileOnly 'org.apache.tomcat:annotations-api:6.0.53' // necessary for Java 9+
```

For Android client, use `grpc-okhttp` instead of `grpc-netty-shaded` and
`grpc-protobuf-lite` instead of `grpc-protobuf`:
```gradle
implementation 'io.grpc:grpc-okhttp:1.64.0'
implementation 'io.grpc:grpc-protobuf-lite:1.64.0'
implementation 'io.grpc:grpc-stub:1.64.0'
compileOnly 'org.apache.tomcat:annotations-api:6.0.53' // necessary for Java 9+
```

For [Bazel](https://bazel.build), you can either
[use Maven](https://github.com/bazelbuild/rules_jvm_external)
(with the GAVs from above), or use `@io_grpc_grpc_java//api` et al (see below).

[the JARs]:
https://search.maven.org/search?q=g:io.grpc%20AND%20v:1.64.0

Development snapshots are available in [Sonatypes's snapshot
repository](https://oss.sonatype.org/content/repositories/snapshots/).

Generated Code
--------------

For protobuf-based codegen, you can put your proto files in the `src/main/proto`
and `src/test/proto` directories along with an appropriate plugin.

For protobuf-based codegen integrated with the Maven build system, you can use
[protobuf-maven-plugin][] (Eclipse and NetBeans users should also look at
`os-maven-plugin`'s
[IDE documentation](https://github.com/trustin/os-maven-plugin#issues-with-eclipse-m2e-or-other-ides)):
```xml
<build>
  <extensions>
    <extension>
      <groupId>kr.motd.maven</groupId>
      <artifactId>os-maven-plugin</artifactId>
      <version>1.7.1</version>
    </extension>
  </extensions>
  <plugins>
    <plugin>
      <groupId>org.xolstice.maven.plugins</groupId>
      <artifactId>protobuf-maven-plugin</artifactId>
      <version>0.6.1</version>
      <configuration>
        <protocArtifact>com.google.protobuf:protoc:3.25.1:exe:${os.detected.classifier}</protocArtifact>
        <pluginId>grpc-java</pluginId>
        <pluginArtifact>io.grpc:protoc-gen-grpc-java:1.64.0:exe:${os.detected.classifier}</pluginArtifact>
      </configuration>
      <executions>
        <execution>
          <goals>
            <goal>compile</goal>
            <goal>compile-custom</goal>
          </goals>
        </execution>
      </executions>
    </plugin>
  </plugins>
</build>
```

[protobuf-maven-plugin]: https://www.xolstice.org/protobuf-maven-plugin/

For non-Android protobuf-based codegen integrated with the Gradle build system,
you can use [protobuf-gradle-plugin][]:
```gradle
plugins {
    id 'com.google.protobuf' version '0.9.4'
}

protobuf {
  protoc {
    artifact = "com.google.protobuf:protoc:3.25.1"
  }
  plugins {
    grpc {
      artifact = 'io.grpc:protoc-gen-grpc-java:1.64.0'
    }
  }
  generateProtoTasks {
    all()*.plugins {
      grpc {}
    }
  }
}
```

[protobuf-gradle-plugin]: https://github.com/google/protobuf-gradle-plugin

The prebuilt protoc-gen-grpc-java binary uses glibc on Linux. If you are
compiling on Alpine Linux, you may want to use the [Alpine grpc-java package][]
which uses musl instead.

[Alpine grpc-java package]: https://pkgs.alpinelinux.org/package/edge/community/x86_64/grpc-java

For Android protobuf-based codegen integrated with the Gradle build system, also
use protobuf-gradle-plugin but specify the 'lite' options:

```gradle
plugins {
    id 'com.google.protobuf' version '0.9.4'
}

protobuf {
  protoc {
    artifact = "com.google.protobuf:protoc:3.25.1"
  }
  plugins {
    grpc {
      artifact = 'io.grpc:protoc-gen-grpc-java:1.64.0'
    }
  }
  generateProtoTasks {
    all().each { task ->
      task.builtins {
        java { option 'lite' }
      }
      task.plugins {
        grpc { option 'lite' }
      }
    }
  }
}

```

For [Bazel](https://bazel.build), use the [`proto_library`](https://github.com/bazelbuild/rules_proto)
and the [`java_proto_library`](https://bazel.build/reference/be/java#java_proto_library) (no `load()` required)
and `load("@io_grpc_grpc_java//:java_grpc_library.bzl", "java_grpc_library")` (from this project), as in
[this example `BUILD.bazel`](https://github.com/grpc/grpc-java/blob/master/examples/BUILD.bazel).

API Stability
-------------

APIs annotated with `@Internal` are for internal use by the gRPC library and
should not be used by gRPC users. APIs annotated with `@ExperimentalApi` are
subject to change in future releases, and library code that other projects
may depend on should not use these APIs.

We recommend using the
[grpc-java-api-checker](https://github.com/grpc/grpc-java-api-checker)
(an [Error Prone](https://github.com/google/error-prone) plugin)
to check for usages of `@ExperimentalApi` and `@Internal` in any library code
that depends on gRPC. It may also be used to check for `@Internal` usage or
unintended `@ExperimentalApi` consumption in non-library code.

How to Build
------------

If you are making changes to gRPC-Java, see the [compiling
instructions](COMPILING.md).

High-level Components
---------------------

At a high level there are three distinct layers to the library: *Stub*,
*Channel*, and *Transport*.

### Stub

The Stub layer is what is exposed to most developers and provides type-safe
bindings to whatever datamodel/IDL/interface you are adapting. gRPC comes with
a [plugin](https://github.com/google/grpc-java/blob/master/compiler) to the
protocol-buffers compiler that generates Stub interfaces out of `.proto` files,
but bindings to other datamodel/IDL are easy and encouraged.

### Channel

The Channel layer is an abstraction over Transport handling that is suitable for
interception/decoration and exposes more behavior to the application than the
Stub layer. It is intended to be easy for application frameworks to use this
layer to address cross-cutting concerns such as logging, monitoring, auth, etc.

### Transport

The Transport layer does the heavy lifting of putting and taking bytes off the
wire. The interfaces to it are abstract just enough to allow plugging in of
different implementations. Note the transport layer API is considered internal
to gRPC and has weaker API guarantees than the core API under package `io.grpc`.

gRPC comes with multiple Transport implementations:

1. The Netty-based HTTP/2 transport is the main transport implementation based
   on [Netty](https://netty.io). It is not officially supported on Android.
   There is a "grpc-netty-shaded" version of this transport. It is generally
   preferred over using the Netty-based transport directly as it requires less
   dependency management and is easier to upgrade within many applications.
2. The OkHttp-based HTTP/2 transport is a lightweight transport based on
   [Okio](https://square.github.io/okio/) and forked low-level parts of
   [OkHttp](https://square.github.io/okhttp/). It is mainly for use on Android.
3. The in-process transport is for when a server is in the same process as the
   client. It is used frequently for testing, while also being safe for
   production use.
4. The Binder transport is for Android cross-process communication on a single
   device.


# Litho [![GithubCI](https://github.com/facebook/litho/actions/workflows/ci.yml/badge.svg)](https://github.com/facebook/litho/actions/workflows/ci.yml) [![Bintray](https://img.shields.io/maven-metadata/v/https/jcenter.bintray.com/com/facebook/litho/litho-core/maven-metadata.xml.svg?color=orange&label=bintray)](https://bintray.com/facebook/maven/com.facebook.litho%3Alitho-core/_latestVersion) [![Join the chat at https://gitter.im/facebook/litho](https://badges.gitter.im/facebook/litho.svg)](https://gitter.im/facebook/litho?utm_source=badge&utm_medium=badge&utm_campaign=pr-badge&utm_content=badge) [![License](https://img.shields.io/badge/License-Apache%202.0-blue.svg)](https://github.com/facebook/litho/blob/master/LICENSE)

<img src="website/static/logo.png" width=150 align=right>

Litho is a declarative framework for building efficient UIs on Android.

* **Declarative:** Litho uses a declarative API to define UI components. You simply describe the layout for your UI based on a set of immutable inputs and the framework takes care of the rest.
* **Asynchronous layout:** Litho can measure and layout your UI ahead of time without blocking the UI thread.
* **View flattening:** Litho uses [Yoga](https://facebook.github.io/yoga/) for layout and automatically reduces the number of ViewGroups that your UI contains.
* **Fine-grained recycling:** Any component such as a text or image can be recycled and reused anywhere in the UI.

To get started, check out these links:

* [Learn how to use Litho in your project.](http://fblitho.com/docs/getting-started)
* [Get started with our tutorial.](http://fblitho.com/docs/tutorial)
* [Read more about Litho in our docs.](http://fblitho.com/docs/intro)

## Installation
Litho can be integrated either in Gradle or Buck projects. Read our [Getting Started](http://fblitho.com/docs/getting-started) guide for installation instructions.

## Quick start
### 1. Initialize `SoLoader` in your `Application` class.
```java
public class SampleApplication extends Application {
  @Override
  public void onCreate() {
    super.onCreate();
    SoLoader.init(this, false);
  }
}
```
### 2. Create and display a component in your Activity
```java
@Override
public void onCreate(Bundle savedInstanceState) {
    super.onCreate(savedInstanceState);

    final ComponentContext c = new ComponentContext(this);

    final Component component = Text.create(c)
        .text("Hello World")
        .textSizeDip(50)
        .build();

    setContentView(LithoView.create(c, component));
}
```
## Run sample
You can find more examples in our [sample app](https://github.com/facebook/litho/tree/master/sample).

To build and run (on an attached device/emulator) the sample app, execute

    $ buck fetch sample
    $ buck install -r sample

or, if you prefer Gradle,

    $ ./gradlew :sample:installDebug

## Contributing

Before contributing to Litho, please first read the [Code of
Conduct](https://code.facebook.com/codeofconduct) that we expect
project participants to adhere to.

For pull requests, please see our [CONTRIBUTING](CONTRIBUTING.md) guide.

See our [issues](https://github.com/facebook/litho/issues/) page for ideas on how to contribute or to let us know of any problems.

Please also read our [Coding Style](http://fblitho.com/docs/best-practices#coding-style) and [Code of Conduct](https://code.facebook.com/codeofconduct) before you contribute.

## Getting Help

- Post on [StackOverflow](https://stackoverflow.com/questions/tagged/litho)
  using the `#litho` tag.
- Chat with us on [Gitter](https://gitter.im/facebook/litho).
- Join our [Facebook Group](https://www.facebook.com/groups/litho.android/) to
  stay up-to-date with announcements.
- Please open GitHub issues only if you suspect a bug in the framework or have a
  feature request and not for general questions.

## License

Litho is licensed under the [Apache 2.0 License](LICENSE).


# Micronaut Framework

[![Build Status](https://github.com/micronaut-projects/micronaut-core/workflows/Java%20CI/badge.svg)](https://github.com/micronaut-projects/micronaut-core/actions)
[![Revved up by Develocity](https://img.shields.io/badge/Revved%20up%20by-Develocity-06A0CE?logo=Gradle&labelColor=02303A)](https://ge.micronaut.io/scans)
[![Quality Gate Status](https://sonarcloud.io/api/project_badges/measure?project=micronaut-projects_micronaut-core&metric=alert_status)](https://sonarcloud.io/summary/new_code?id=micronaut-projects_micronaut-core)

[Micronaut Framework](https://micronaut.io) is a modern, JVM-based, full stack Java framework designed for building modular, easily testable JVM applications with support for Java, Kotlin and the Groovy language.

The Micronaut framework was originally created by a team who had also worked on the Grails framework. The Micronaut framework takes inspiration from lessons learned over the years building real-world applications from monoliths to microservices using Spring, Spring Boot and the Grails framework. The core team continues to develop and maintain the Micronaut project through the support of the Micronaut Foundation.

Micronaut Framework aims to provide all the tools necessary to build JVM applications including:

* Dependency Injection and Inversion of Control (IoC)
* Aspect Oriented Programming (AOP)
* Sensible Defaults and Auto-Configuration

With Micronaut Framework you can build Message-Driven Applications, Command Line Applications, HTTP Servers and more whilst for Microservices in particular Micronaut Framework also provides:

* Distributed Configuration
* Service Discovery
* HTTP Routing
* Client-Side Load Balancing

At the same time Micronaut Framework aims to avoid the downsides of frameworks like Spring, Spring Boot and Grails by providing:

* Fast startup time
* Reduced memory footprint
* Minimal use of reflection
* Minimal use of proxies
* No runtime bytecode generation
* Easy Unit Testing

This is achieved by pre-computing the framework infrastructure at compilation time which reduces the logic required at runtime for the application to work.

For more information on using Micronaut Framework see the documentation at [micronaut.io](https://micronaut.io)

## Example Applications

Example Micronaut Framework applications can be found in the [Examples repository](https://github.com/micronaut-projects/micronaut-examples)

## Building From Source

To build from source checkout the code and run:

```
./gradlew publishToMavenLocal
```

To build the documentation run `./gradlew docs`. The documentation is built to `build/docs/index.html`.

## Contributing Code

If you wish to contribute to the development of Micronaut Framework please read the [CONTRIBUTING.md](CONTRIBUTING.md)

## Versioning

Micronaut Framework uses Semantic Versioning 2.0.0. To understand what that means, please see the specification [documentation](https://semver.org/). Exclusions to Micronaut Framework's public API include any classes annotated with `@Experimental` or `@Internal`, which reside in the `io.micronaut.core.annotation` package.


[![Build Status](https://github.com/aws/aws-sdk-ruby/workflows/CI/badge.svg)](https://github.com/awsdocs/aws-doc-sdk-examples/actions)
[![GitHub Super-Linter](https://github.com/awsdocs/aws-doc-sdk-examples/actions/workflows/super-linter.yml/badge.svg)](https://github.com/marketplace/actions/super-linter)
![[]](https://img.shields.io/badge/license-MIT%2FApache--2.0-blue)

# AWS SDK Code Examples

This repository contains code examples that demonstrate how to use the [AWSK SDKs](https://aws.amazon.com/developer/tools/) to interact with [AWS services](https://aws.amazon.com/products).

Many examples are injected into the [AWS Documentation](https://docs.aws.amazon.com).

## How this repository is organized

Code examples for each language's SDK can be found within the following subdirectories. The examples here demonstrate the most common uses of the SDK for each language.

|     SDK    |       folder      | SDK version |
|:----------:|:-----------------:|:-----------:|
| .NET       | [dotnetv3/](dotnetv3)         | 3.5+        |
| .NET       | [dotnet/](.dotnet)           | <3.5        |
| C++        | [cpp/](cpp)              | 1           |
| Go         | [gov2/](gov2)            | 2           |
| Go         | [go/](go)               | 1           |
| Java       | [javav2/](javav2)           | 2           |
| Java       | [java/](java)             | 1           |
| JavaScript | [javascriptv3/](javascriptv3)     | 3           |
| JavaScript | [javascript/](javascriptv)       | 2           |
| Kotlin     | [kotlin/](kotlin)           | 1             |
| PHP        | [php/](php)              | 3           |
| Python     | [python/](python)           | 3           |
| Ruby       | [ruby/](ruby)             | 3           |
| Rust       | [rustv1/](rustv1) | 1             |
| Swift      | [swift/](swift)            | preview           |

Within each directory, you will find SDK-specific instructions for understanding and invoking example code.

### NOTE
In alignment with our SDKs and Tools Maintenance Policy, the AWS SDK for Java v1.x will enter maintenance mode on July 31, 2024, and reach end-of-support on December 31, 2025.

For more information, see [Announcing end-of-support for AWS SDK for Java v1.x](https://aws.amazon.com/blogs/developer/announcing-end-of-support-for-aws-sdk-for-java-v1-x-on-december-31-2025/).

### Additional directories

| directory                     | purpose                                                                                | usage                                                                                |
|-------------------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| [/applications](applications) | Contains the non-language-specific components of example applications, which show how the SDKs can be used in the context of a production-like application. | To view the language-specific components for each example application, see the `cross-service` folder in the sub-directory for your desired language (such as `python/cross-service`). |
| [/test](test)                 | Contains all components supporting the custom test automation framework used to routinely test the code in this repository.                                 | Deploys to AWS as a polyglot container-based integration testing solution. WARNING: Still under active construction as of 2023.                                                        |
| [/resources](resources)       | Contains shared components used by many code examples across this repository.                                                                               | Deploys as frontend ([/clients](/resources/clients)) or backend ([/cdk](/resources/cdk) or [/cfn](/resources/cfn)) components.

## Invoke example code

To invoke this example code, you must have an AWS account. For more information about creating an account, see [AWS Free Tier](https://aws.amazon.com/free/).

You must also have AWS credentials configured. For steps on using the AWS Command Line Interface (AWS CLI) to configure credentials, see [CLI Configuration basics](https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-quickstart.html)

## ⚠ Usage disclaimer

These code examples interact with services that may incur charges to your AWS account. For more information, see [AWS Pricing](https://aws.amazon.com/pricing/).

Additionally, example code might theoretically modify or delete existing AWS resources. As a matter of due diligence, do the following:

- Be aware of the resources that these examples create or delete.
- Be aware of the costs that might be charged to your account as a result.
- Back up your important data.

# Contributing

This repository thrives on your contributions! ❤ To get involved, see the [CONTRIBUTING.md](CONTRIBUTING.md). 🙏

## Tributaries

The AWS Documentation includes source code examples from several additional sources. These sources are called tributaries, flowing into the "river" of AWS Documentation.

* [awsdocs/aws-doc-sdk-examples](https://github.com/awsdocs/aws-doc-sdk-examples)
* [aws-samples/serverless-snippets](https://github.com/aws-samples/serverless-snippets)
* [aws/aws-tools-for-powershell](https://github.com/aws/aws-tools-for-powershell)

# Copyright and license

All content in this repository, unless otherwise stated, is
Copyright © Amazon Web Services, Inc. or its affiliates. All rights reserved.

Except where otherwise noted, all examples in this collection are licensed under the [Apache
license, version 2.0](https://www.apache.org/licenses/LICENSE-2.0) (the "License"). The full
license text is provided in the `LICENSE` file accompanying this repository.


<img src="https://geysermc.org/img/geyser-1760-860.png" alt="Geyser" width="600"/>

[![License: MIT](https://img.shields.io/badge/license-MIT-blue.svg)](LICENSE)
[![Discord](https://img.shields.io/discord/613163671870242838.svg?color=%237289da&label=discord)](https://discord.gg/geysermc)
[![Crowdin](https://badges.crowdin.net/e/51361b7f8a01644a238d0fe8f3bddc62/localized.svg)](https://translate.geysermc.org/)

Geyser is a bridge between Minecraft: Bedrock Edition and Minecraft: Java Edition, closing the gap from those wanting to play true cross-platform.

Geyser is an open collaboration project by [CubeCraft Games](https://cubecraft.net).

## What is Geyser?
Geyser is a proxy, bridging the gap between Minecraft: Bedrock Edition and Minecraft: Java Edition servers.
The ultimate goal of this project is to allow Minecraft: Bedrock Edition users to join Minecraft: Java Edition servers as seamlessly as possible. However, due to the nature of Geyser translating packets over the network of two different games, *do not expect everything to work perfectly!*

Special thanks to the DragonProxy project for being a trailblazer in protocol translation and for all the team members who have joined us here!

### Currently supporting Minecraft Bedrock 1.20.80 - 1.21.1 and Minecraft Java 1.21

## Setting Up
Take a look [here](https://wiki.geysermc.org/geyser/setup/) for how to set up Geyser.

[![YouTube Video](https://img.youtube.com/vi/U7dZZ8w7Gi4/0.jpg)](https://www.youtube.com/watch?v=U7dZZ8w7Gi4)

## Links:
- Website: https://geysermc.org
- Docs: https://wiki.geysermc.org/geyser/
- Download: https://geysermc.org/download
- Discord: https://discord.gg/geysermc
- Donate: https://opencollective.com/geysermc
- Test Server: `test.geysermc.org` port `25565` for Java and `19132` for Bedrock

## What's Left to be Added/Fixed
- Near-perfect movement (to the point where anticheat on large servers is unlikely to ban you)
- Some Entity Flags

## What can't be fixed
There are a few things Geyser is unable to support due to various differences between Minecraft Bedrock and Java. For a list of these limitations, see the [Current Limitations](https://wiki.geysermc.org/geyser/current-limitations/) page.

## Compiling
1. Clone the repo to your computer
2. Navigate to the Geyser root directory and run `git submodule update --init --recursive`. This command downloads all the needed submodules for Geyser and is a crucial step in this process.
3. Run `gradlew build` and locate to `bootstrap/build` folder.

## Contributing
Any contributions are appreciated. Please feel free to reach out to us on [Discord](http://discord.geysermc.org/) if
you're interested in helping out with Geyser.

## Libraries Used:
- [Adventure Text Library](https://github.com/KyoriPowered/adventure)
- [NukkitX Bedrock Protocol Library](https://github.com/NukkitX/Protocol)
- [Steveice10's Java Protocol Library](https://github.com/Steveice10/MCProtocolLib)
- [TerminalConsoleAppender](https://github.com/Minecrell/TerminalConsoleAppender)
- [Simple Logging Facade for Java (slf4j)](https://github.com/qos-ch/slf4j)


[![Backers on Open Collective](https://opencollective.com/openvidu/backers/badge.svg)](#backers) [![Sponsors on Open Collective](https://opencollective.com/openvidu/sponsors/badge.svg)](#sponsors)
[![License badge](https://img.shields.io/badge/license-Apache2-orange.svg)](http://www.apache.org/licenses/LICENSE-2.0)
[![OpenVidu Tests](https://github.com/OpenVidu/openvidu/actions/workflows/openvidu-ce-test.yml/badge.svg)](https://github.com/OpenVidu/openvidu/actions/workflows/openvidu-ce-test.yml)
[![Npm version](https://img.shields.io/npm/v/openvidu-browser?label=npm-version)](https://npmjs.org/package/openvidu-browser)
[![Npm downloads](https://img.shields.io/npm/dw/openvidu-browser?label=npm2-downloads)](https://npmjs.org/package/openvidu-browser)

[![Documentation Status](https://readthedocs.org/projects/openvidu/badge/?version=stable)](https://docs.openvidu.io/en/stable/?badge=stable)
[![Docker badge](https://img.shields.io/docker/pulls/openvidu/openvidu-server-kms.svg)](https://hub.docker.com/r/openvidu/openvidu-server-kms)
[![Support badge](https://img.shields.io/badge/support-sof-yellowgreen.svg)](https://openvidu.discourse.group/)
[![Twitter Follow](https://img.shields.io/twitter/follow/openvidu.svg?style=social)](https://twitter.com/openvidu)

[![][OpenViduLogo]](https://openvidu.io)

openvidu
===

Visit [openvidu.io](https://openvidu.io)

## Community Forum

Visit [OpenVidu Community Forum](https://openvidu.discourse.group/)

[OpenViduLogo]: https://secure.gravatar.com/avatar/5daba1d43042f2e4e85849733c8e5702?s=120

## Contributors

This project exists thanks to all the people who contribute.

## Backers

Thank you to all our backers! 🙏 [[Become a backer](https://opencollective.com/openvidu#backer)]

## Acknowledgments

OpenVidu has been supported under project "CPP2021-008720 NewGenVidu: An elastic, user-friendly and privacy-friendly videoconferencing platform", funded by MCIN/AEI/10.13039/501100011033 and by the European Union-NextGenerationEU/PRTR.

## Sponsors

Support this project by becoming a sponsor. Your logo will show up here with a link to your website. [[Become a sponsor](https://opencollective.com/openvidu#sponsor)]


# Copybara

*A tool for transforming and moving code between repositories.*

Copybara is a tool used internally at Google. It transforms and moves code between repositories.

Often, source code needs to exist in multiple repositories, and Copybara allows you to transform
and move source code between these repositories. A common case is a project that involves
maintaining a confidential repository and a public repository in sync.

Copybara requires you to choose one of the repositories to be the authoritative repository, so that
there is always one source of truth. However, the tool allows contributions to any repository, and
any repository can be used to cut a release.

The most common use case involves repetitive movement of code from one repository to another.
Copybara can also be used for moving code once to a new repository.

Examples uses of Copybara include:

  - Importing sections of code from a confidential repository to a public repository.

  - Importing code from a public repository to a confidential repository.

  - Importing a change from a non-authoritative repository into the authoritative repository. When
    a change is made in the non-authoritative repository (for example, a contributor in the public
    repository), Copybara transforms and moves that change into the appropriate place in the
    authoritative repository. Any merge conflicts are dealt with in the same way as an out-of-date
    change within the authoritative repository.

One of the main features of Copybara is that it is stateless, or more specifically, that it stores
the state in the destination repository (As a label in the commit message). This allows several
users (or a service) to use Copybara for the same config/repositories and get the same result.

Currently, the only supported type of repository is Git. Copybara is also able
to read from Mercurial repositories, but the feature is still experimental.
The extensible architecture allows adding bespoke origins and destinations
for almost any use case.
Official support for other repositories types will be added in the future.

## Example

```python
core.workflow(
    name = "default",
    origin = git.github_origin(
      url = "https://github.com/google/copybara.git",
      ref = "master",
    ),
    destination = git.destination(
        url = "file:///tmp/foo",
    ),

    # Copy everything but don't remove a README_INTERNAL.txt file if it exists.
    destination_files = glob(["third_party/copybara/**"], exclude = ["README_INTERNAL.txt"]),

    authoring = authoring.pass_thru("Default email <default@default.com>"),
    transformations = [
        core.replace(
                before = "//third_party/bazel/bashunit",
                after = "//another/path:bashunit",
                paths = glob(["**/BUILD"])),
        core.move("", "third_party/copybara")
    ],
)
```

Run:

```shell
$ (mkdir /tmp/foo ; cd /tmp/foo ; git init --bare)
$ copybara copy.bara.sky
```

## Getting Started using Copybara

Copybara doesn't have a release process yet, so you need to compile from HEAD.
In order to do that, you need to do the following:

  * [Install JDK 11](https://www.oracle.com/java/technologies/downloads/#java11).
  * [Install Bazel](https://bazel.build/install).
  * Clone the copybara source locally:
      * `git clone https://github.com/google/copybara.git`
  * Build:
      * `bazel build //java/com/google/copybara`
      * `bazel build //java/com/google/copybara:copybara_deploy.jar` to create an executable uberjar.
  * Tests: `bazel test //...` if you want to ensure you are not using a broken version. Note that
    certain tests require the underlying tool to be installed(e.g. Mercurial, Quilt, etc.). It is
    fine to skip those tests if your Pull Request is unrelated to those modules (And our CI will
    run all the tests anyway).

### System packages

These packages can be installed using the appropriate package manager for your
system.

#### Arch Linux

  * [`aur/copybara-git`][install/archlinux/aur-git]

[install/archlinux/aur-git]: https://aur.archlinux.org/packages/copybara-git "Copybara on the AUR"

### Using Intellij with Bazel plugin

If you use Intellij and the Bazel plugin, use this project configuration:

```
directories:
  copybara/integration
  java/com/google/copybara
  javatests/com/google/copybara
  third_party

targets:
  //copybara/integration/...
  //java/com/google/copybara/...
  //javatests/com/google/copybara/...
  //third_party/...
```

Note: configuration files can be stored in any place, even in a local folder.
We recommend using a VCS (like git) to store them; treat them as source code.

### Building Copybara in an external Bazel workspace

There are convenience macros defined for all of Copybara's dependencies. Add the
following code to your `WORKSPACE` file, replacing `{{ sha256sum }}` and
`{{ commit }}` as necessary.

```bzl
http_archive(
  name = "com_github_google_copybara",
  sha256 = "{{ sha256sum }}",
  strip_prefix = "copybara-{{ commit }}",
  url = "https://github.com/google/copybara/archive/{{ commit }}.zip",
)

load("@com_github_google_copybara//:repositories.bzl", "copybara_repositories")

copybara_repositories()

load("@com_github_google_copybara//:repositories.maven.bzl", "copybara_maven_repositories")

copybara_maven_repositories()

load("@com_github_google_copybara//:repositories.go.bzl", "copybara_go_repositories")

copybara_go_repositories()
```

You can then build and run the Copybara tool from within your workspace:

```sh
bazel run @com_github_google_copybara//java/com/google/copybara -- <args...>
```

### Using Docker to build and run Copybara

*NOTE: Docker use is currently experimental, and we encourage feedback or contributions.*

You can build copybara using Docker like so

```sh
docker build --rm -t copybara .
```

Once this has finished building, you can run the image like so from the root of
the code you are trying to use Copybara on:

```sh
docker run -it -v "$(pwd)":/usr/src/app copybara help
```

#### Environment variables

In addition to passing cmd args to the container, you can also set the following
environment variables as an alternative:
* `COPYBARA_SUBCOMMAND=migrate`
  * allows you to change the command run, defaults to `migrate`
* `COPYBARA_CONFIG=copy.bara.sky`
  * allows you to specify a path to a config file, defaults to root `copy.bara.sky`
* `COPYBARA_WORKFLOW=default`
  * allows you to specify the workflow to run, defaults to `default`
* `COPYBARA_SOURCEREF=''`
  * allows you to specify the sourceref, defaults to none
* `COPYBARA_OPTIONS=''`
  * allows you to specify options for copybara, defaults to none

```sh
docker run \
    -e COPYBARA_SUBCOMMAND='validate' \
    -e COPYBARA_CONFIG='other.config.sky' \
    -v "$(pwd)":/usr/src/app \
    -it copybara
```

#### Git Config and Credentials

There are a number of ways by which to share your git config and ssh credentials
with the Docker container, an example is below:

```sh
docker run \
    -v ~/.gitconfig:/root/.gitconfig:ro \
    -v ~/.ssh:/root/.ssh \
    -v ${SSH_AUTH_SOCK}:${SSH_AUTH_SOCK} -e SSH_AUTH_SOCK
    -v "$(pwd)":/usr/src/app \
    -it copybara
```

## Documentation

We are still working on the documentation. Here are some resources:

  * [Reference documentation](docs/reference.md)
  * [Examples](docs/examples.md)
  * [Tutorial on how to get started](https://blog.kubesimplify.com/moving-code-between-git-repositories-with-copybara)

## Contact us

If you have any questions about how Copybara works, please contact us at our
[mailing list](https://groups.google.com/forum/#!forum/copybara-discuss).

## Optional tips

* If you want to see the test errors in Bazel, instead of having to `cat` the
  logs, add this line to your `~/.bazelrc`:

  ```
  test --test_output=streamed
  ```


<!--
  - Licensed to the Apache Software Foundation (ASF) under one
  - or more contributor license agreements.  See the NOTICE file
  - distributed with this work for additional information
  - regarding copyright ownership.  The ASF licenses this file
  - to you under the Apache License, Version 2.0 (the
  - "License"); you may not use this file except in compliance
  - with the License.  You may obtain a copy of the License at
  -
  -   http://www.apache.org/licenses/LICENSE-2.0
  -
  - Unless required by applicable law or agreed to in writing,
  - software distributed under the License is distributed on an
  - "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
  - KIND, either express or implied.  See the License for the
  - specific language governing permissions and limitations
  - under the License.
  -->

![Iceberg](https://iceberg.apache.org/assets/images/Iceberg-logo.svg)

[![](https://github.com/apache/iceberg/actions/workflows/java-ci.yml/badge.svg)](https://github.com/apache/iceberg/actions/workflows/java-ci.yml)
[![Slack](https://img.shields.io/badge/chat-on%20Slack-brightgreen.svg)](https://apache-iceberg.slack.com/)

Iceberg is a high-performance format for huge analytic tables. Iceberg brings the reliability and simplicity of SQL tables to big data, while making it possible for engines like Spark, Trino, Flink, Presto, Hive and Impala to safely work with the same tables, at the same time.

Background and documentation is available at <https://iceberg.apache.org>

## Status

Iceberg is under active development at the Apache Software Foundation.

The [Iceberg format specification][iceberg-spec] is stable and new features are added with each version.

The core Java library is located in this repository and is the reference implementation for other libraries.

[Documentation][iceberg-docs] is available for all libraries and integrations.

[iceberg-docs]: https://iceberg.apache.org/docs/latest/
[iceberg-spec]: https://iceberg.apache.org/spec/

## Collaboration

Iceberg tracks issues in GitHub and prefers to receive contributions as pull requests.

Community discussions happen primarily on the [dev mailing list][dev-list] or on specific issues.

[dev-list]: mailto:dev@iceberg.apache.org

### Building

Iceberg is built using Gradle with Java 8, 11, or 17.

* To invoke a build and run tests: `./gradlew build`
* To skip tests: `./gradlew build -x test -x integrationTest`
* To fix code style for default versions: `./gradlew spotlessApply`
* To fix code style for all versions of Spark/Hive/Flink:`./gradlew spotlessApply -DallModules`

Iceberg table support is organized in library modules:

* `iceberg-common` contains utility classes used in other modules
* `iceberg-api` contains the public Iceberg API
* `iceberg-core` contains implementations of the Iceberg API and support for Avro data files, **this is what processing engines should depend on**
* `iceberg-parquet` is an optional module for working with tables backed by Parquet files
* `iceberg-arrow` is an optional module for reading Parquet into Arrow memory
* `iceberg-orc` is an optional module for working with tables backed by ORC files
* `iceberg-hive-metastore` is an implementation of Iceberg tables backed by the Hive metastore Thrift client
* `iceberg-data` is an optional module for working with tables directly from JVM applications

Iceberg also has modules for adding Iceberg support to processing engines:

* `iceberg-spark` is an implementation of Spark's Datasource V2 API for Iceberg with submodules for each spark versions (use runtime jars for a shaded version)
* `iceberg-flink` contains classes for integrating with Apache Flink (use iceberg-flink-runtime for a shaded version)
* `iceberg-mr` contains an InputFormat and other classes for integrating with Apache Hive
* `iceberg-pig` is an implementation of Pig's LoadFunc API for Iceberg

---
**NOTE**

The tests require Docker to execute. On MacOS (with Docker Desktop), you might need to create a symbolic name to the docker socket in order to be detected by the tests:

```
sudo ln -s $HOME/.docker/run/docker.sock /var/run/docker.sock
```
---

### Engine Compatibility

See the [Multi-Engine Support](https://iceberg.apache.org/multi-engine-support/) page to know about Iceberg compatibility with different Spark, Flink and Hive versions.
For other engines such as Presto or Trino, please visit their websites for Iceberg integration details.

### Implementations

This repository contains the Java implementation of Iceberg. Other implementations can be found at:

* **Go**: [iceberg-go](https://github.com/apache/iceberg-go)
* **PyIceberg** (Python): [iceberg-python](https://github.com/apache/iceberg-python)
* **Rust**: [iceberg-rust](https://github.com/apache/iceberg-rust)


= ServiceTalk

image:https://img.shields.io/maven-central/v/io.servicetalk/servicetalk-annotations?color=blue[Maven Central]
image:https://github.com/apple/servicetalk/actions/workflows/ci-snapshot.yml/badge.svg[Snapshot Publisher]
image:https://img.shields.io/nexus/s/io.servicetalk/servicetalk-annotations?color=blue&server=https%3A%2F%2Foss.sonatype.org[Sonatype Snapshot]

ServiceTalk is a JVM network application framework with APIs tailored to specific protocols (e.g. HTTP/1.x,
HTTP/2.x, etc...) and supports multiple programming paradigms.

It is built on link:https://netty.io[Netty] and is designed to provide most of the performance/scalability benefits of
Netty for common networking protocols used in service to service communication. ServiceTalk provides server support and
"smart client" like features such as client-side load balancing and service discovery integration.

See the link:https://docs.servicetalk.io/[ServiceTalk docs] for more information.

== Getting Started

ServiceTalk releases are available on link:https://repo1.maven.org/maven2/io/servicetalk/[Maven Central].

Refer to the link:https://docs.servicetalk.io/[ServiceTalk docs] for various examples that will get you started with the
different features of ServiceTalk.

NOTE: Builds of the development version are available in
link:https://oss.sonatype.org/content/repositories/snapshots/io/servicetalk/[Sonatype's snapshots Maven repository].

== Supported JVM
The minimum supported JDK version is 1.8.

== Compatibility
ServiceTalk follows link:https://semver.org/#semantic-versioning-200[SemVer 2.0.0]. API/ABI breaking changes will
require package renaming for that module to avoid runtime classpath conflicts.

NOTE: `0.x.y` releases are not stable and are permitted to break API/ABI.

== Contributor Setup

IMPORTANT: If you're intending to contribute to ServiceTalk,
           make sure to first read the xref:CONTRIBUTING.adoc[contribution guidelines].

ServiceTalk uses link:https://gradle.org[Gradle] as its build tool and only requires JDK 8 or higher to be
pre-installed. ServiceTalk ships with the Gradle Wrapper, which means that there is no need to install Gradle on your
machine beforehand.

=== File Encoding

ServiceTalk's source code is UTF-8 encoded: make sure your filesystem supports it before attempting to build
the project. Setting the `JAVA_TOOL_OPTIONS` env var to `-Dfile.encoding=UTF-8` should help building the project in
non-UTF-8 environments. Editors and IDEs must also support UTF-8 in order to successfully edit ServiceTalk's source
code.

=== Build Commands

You should be able to run the following command to build ServiceTalk and verify that all
tests and code quality checks pass:

[source,shell]
----
$ ./gradlew build
----

The supported IDE is link:https://www.jetbrains.com/idea[IntelliJ IDEA].
In order to generate IntelliJ IDEA project files for ServiceTalk,
you can run the following command:

[source,shell]
----
$ ./gradlew idea
----

When done, running one of following commands would open ServiceTalk in IntelliJ:

.Generic
[source,shell]
----
$ idea .
----

.macOS
[source,shell]
----
$ open servicetalk.ipr
----

== Project Communication
We encourage your participation asking questions and helping improve the ServiceTalk project.
link:https://github.com/apple/servicetalk/issues[Github issues] and
link:https://github.com/apple/servicetalk/pulls[pull requests] are the primary mechanisms of
participation and communication for ServiceTalk.


# Google Cloud Dataflow Template Pipelines

These Dataflow templates are an effort to solve simple, but large, in-Cloud data
tasks, including data import/export/backup/restore and bulk API operations,
without a development environment. The technology under the hood which makes
these operations possible is the
[Google Cloud Dataflow](https://cloud.google.com/dataflow/) service combined
with a set of [Apache Beam](https://beam.apache.org/) SDK templated pipelines.

Google is providing this collection of pre-implemented Dataflow templates as a
reference and to provide easy customization for developers wanting to extend
their functionality.

[![Open in Cloud Shell](http://gstatic.com/cloudssh/images/open-btn.svg)](https://console.cloud.google.com/cloudshell/editor?cloudshell_git_repo=https%3A%2F%2Fgithub.com%2FGoogleCloudPlatform%2FDataflowTemplates.git)

## Note on Default Branch

As of November 18, 2021, our default branch is now named "main". This does not
affect forks. If you would like your fork and its local clone to reflect these
changes you can
follow [GitHub's branch renaming guide](https://docs.github.com/en/repositories/configuring-branches-and-merges-in-your-repository/managing-branches-in-your-repository/renaming-a-branch).

## Template Pipelines

- Get Started
    - [Word Count](https://github.com/search?q=repo%3AGoogleCloudPlatform%2FDataflowTemplates%20Word_Count&type=code)
- Process Data Continuously (stream)
    - [Azure Eventhub to Pubsub](https://github.com/search?q=repo%3AGoogleCloudPlatform%2FDataflowTemplates%20Azure_Eventhub_to_PubSub&type=code)
    - [Bigtable Change Streams to HBase Replicator](https://github.com/search?q=repo%3AGoogleCloudPlatform%2FDataflowTemplates%20Bigtable_Change_Streams_to_HBase&type=code)
    - [Cloud Bigtable change streams to BigQuery](https://github.com/search?q=repo%3AGoogleCloudPlatform%2FDataflowTemplates%20Bigtable_Change_Streams_to_BigQuery&type=code)
    - [Cloud Bigtable change streams to Cloud Storage](https://github.com/search?q=repo%3AGoogleCloudPlatform%2FDataflowTemplates%20Bigtable_Change_Streams_to_Google_Cloud_Storage&type=code)
    - [Cloud Spanner change streams to BigQuery](https://github.com/search?q=repo%3AGoogleCloudPlatform%2FDataflowTemplates%20Spanner_Change_Streams_to_BigQuery&type=code)
    - [Cloud Spanner change streams to Cloud Storage](https://github.com/search?q=repo%3AGoogleCloudPlatform%2FDataflowTemplates%20Spanner_Change_Streams_to_Google_Cloud_Storage&type=code)
    - [Cloud Spanner change streams to Pub/Sub](https://github.com/search?q=repo%3AGoogleCloudPlatform%2FDataflowTemplates%20Spanner_Change_Streams_to_PubSub&type=code)
    - [Cloud Storage Text to BigQuery (Stream)](https://github.com/search?q=repo%3AGoogleCloudPlatform%2FDataflowTemplates%20Stream_GCS_Text_to_BigQuery_Flex&type=code)
    - [Data Masking/Tokenization from Cloud Storage to BigQuery (using Cloud DLP)](https://github.com/search?q=repo%3AGoogleCloudPlatform%2FDataflowTemplates%20Stream_DLP_GCS_Text_to_BigQuery_Flex&type=code)
    - [Datastream to BigQuery](https://github.com/search?q=repo%3AGoogleCloudPlatform%2FDataflowTemplates%20Cloud_Datastream_to_BigQuery&type=code)
    - [Datastream to Cloud Spanner](https://github.com/search?q=repo%3AGoogleCloudPlatform%2FDataflowTemplates%20Cloud_Datastream_to_Spanner&type=code)
    - [Datastream to SQL](https://github.com/search?q=repo%3AGoogleCloudPlatform%2FDataflowTemplates%20Cloud_Datastream_to_SQL&type=code)
    - [JMS to Pubsub](https://github.com/search?q=repo%3AGoogleCloudPlatform%2FDataflowTemplates%20Jms_to_PubSub&type=code)
    - [Kafka to BigQuery](https://github.com/search?q=repo%3AGoogleCloudPlatform%2FDataflowTemplates%20Kafka_to_BigQuery&type=code)
    - [Kafka to Cloud Storage](https://github.com/search?q=repo%3AGoogleCloudPlatform%2FDataflowTemplates%20Kafka_to_GCS&type=code)
    - [Kinesis To Pubsub](https://github.com/search?q=repo%3AGoogleCloudPlatform%2FDataflowTemplates%20Kinesis_To_Pubsub&type=code)
    - [MongoDB to BigQuery (CDC)](https://github.com/search?q=repo%3AGoogleCloudPlatform%2FDataflowTemplates%20MongoDB_to_BigQuery_CDC&type=code)
    - [Mqtt to Pubsub](https://github.com/search?q=repo%3AGoogleCloudPlatform%2FDataflowTemplates%20Mqtt_to_PubSub&type=code)
    - [Ordered change stream buffer to Source DB](https://github.com/search?q=repo%3AGoogleCloudPlatform%2FDataflowTemplates%20Ordered_Changestream_Buffer_to_Sourcedb&type=code)
    - [Pub/Sub Avro to BigQuery](https://github.com/search?q=repo%3AGoogleCloudPlatform%2FDataflowTemplates%20PubSub_Avro_to_BigQuery&type=code)
    - [Pub/Sub CDC to Bigquery](https://github.com/search?q=repo%3AGoogleCloudPlatform%2FDataflowTemplates%20PubSub_CDC_to_BigQuery&type=code)
    - [Pub/Sub Proto to BigQuery](https://github.com/search?q=repo%3AGoogleCloudPlatform%2FDataflowTemplates%20PubSub_Proto_to_BigQuery&type=code)
    - [Pub/Sub Subscription or Topic to Text Files on Cloud Storage](https://github.com/search?q=repo%3AGoogleCloudPlatform%2FDataflowTemplates%20Cloud_PubSub_to_GCS_Text_Flex&type=code)
    - [Pub/Sub Subscription to BigQuery](https://github.com/search?q=repo%3AGoogleCloudPlatform%2FDataflowTemplates%20PubSub_to_BigQuery_Flex&type=code)
    - [Pub/Sub Topic to BigQuery](https://github.com/search?q=repo%3AGoogleCloudPlatform%2FDataflowTemplates%20PubSub_to_BigQuery&type=code)
    - [Pub/Sub to Avro Files on Cloud Storage](https://github.com/search?q=repo%3AGoogleCloudPlatform%2FDataflowTemplates%20Cloud_PubSub_to_Avro_Flex&type=code)
    - [Pub/Sub to Datadog](https://github.com/search?q=repo%3AGoogleCloudPlatform%2FDataflowTemplates%20Cloud_PubSub_to_Datadog&type=code)
    - [Pub/Sub to Elasticsearch](https://github.com/search?q=repo%3AGoogleCloudPlatform%2FDataflowTemplates%20PubSub_to_Elasticsearch&type=code)
    - [Pub/Sub to JDBC](https://github.com/search?q=repo%3AGoogleCloudPlatform%2FDataflowTemplates%20Pubsub_to_Jdbc&type=code)
    - [Pub/Sub to Kafka](https://github.com/search?q=repo%3AGoogleCloudPlatform%2FDataflowTemplates%20PubSub_to_Kafka&type=code)
    - [Pub/Sub to MongoDB](https://github.com/search?q=repo%3AGoogleCloudPlatform%2FDataflowTemplates%20Cloud_PubSub_to_MongoDB&type=code)
    - [Pub/Sub to Pub/Sub](https://github.com/search?q=repo%3AGoogleCloudPlatform%2FDataflowTemplates%20Cloud_PubSub_to_Cloud_PubSub&type=code)
    - [Pub/Sub to Redis](https://github.com/search?q=repo%3AGoogleCloudPlatform%2FDataflowTemplates%20Cloud_PubSub_to_Redis&type=code)
    - [Pub/Sub to Splunk](https://github.com/search?q=repo%3AGoogleCloudPlatform%2FDataflowTemplates%20Cloud_PubSub_to_Splunk&type=code)
    - [Pub/Sub to Text Files on Cloud Storage](https://github.com/search?q=repo%3AGoogleCloudPlatform%2FDataflowTemplates%20Cloud_PubSub_to_GCS_Text&type=code)
    - [Pubsub to JMS](https://github.com/search?q=repo%3AGoogleCloudPlatform%2FDataflowTemplates%20Pubsub_to_Jms&type=code)
    - [Spanner Change Streams to Sink](https://github.com/search?q=repo%3AGoogleCloudPlatform%2FDataflowTemplates%20Spanner_Change_Streams_to_Sink&type=code)
    - [Synchronizing CDC data to BigQuery](https://github.com/search?q=repo%3AGoogleCloudPlatform%2FDataflowTemplates%20Cdc_To_BigQuery_Template&type=code)
    - [Text Files on Cloud Storage to Pub/Sub](https://github.com/search?q=repo%3AGoogleCloudPlatform%2FDataflowTemplates%20Stream_GCS_Text_to_Cloud_PubSub&type=code)
- Process Data in Bulk (batch)
    - [Any SourceDB to Cloud Spanner](https://github.com/search?q=repo%3AGoogleCloudPlatform%2FDataflowTemplates%20Sourcedb_To_Spanner&type=code)
    - [AstraDB to BigQuery](https://github.com/search?q=repo%3AGoogleCloudPlatform%2FDataflowTemplates%20AstraDB_To_BigQuery&type=code)
    - [Avro Files on Cloud Storage to Cloud Bigtable](https://github.com/search?q=repo%3AGoogleCloudPlatform%2FDataflowTemplates%20GCS_Avro_to_Cloud_Bigtable&type=code)
    - [Avro Files on Cloud Storage to Cloud Spanner](https://github.com/search?q=repo%3AGoogleCloudPlatform%2FDataflowTemplates%20GCS_Avro_to_Cloud_Spanner&type=code)
    - [BigQuery export to Parquet (via Storage API)](https://github.com/search?q=repo%3AGoogleCloudPlatform%2FDataflowTemplates%20BigQuery_to_Parquet&type=code)
    - [BigQuery to Bigtable](https://github.com/search?q=repo%3AGoogleCloudPlatform%2FDataflowTemplates%20BigQuery_to_Bigtable&type=code)
    - [BigQuery to Datastore](https://github.com/search?q=repo%3AGoogleCloudPlatform%2FDataflowTemplates%20Cloud_BigQuery_to_Cloud_Datastore&type=code)
    - [BigQuery to Elasticsearch](https://github.com/search?q=repo%3AGoogleCloudPlatform%2FDataflowTemplates%20BigQuery_to_Elasticsearch&type=code)
    - [BigQuery to MongoDB](https://github.com/search?q=repo%3AGoogleCloudPlatform%2FDataflowTemplates%20BigQuery_to_MongoDB&type=code)
    - [BigQuery to TensorFlow Records](https://github.com/search?q=repo%3AGoogleCloudPlatform%2FDataflowTemplates%20Cloud_BigQuery_to_GCS_TensorFlow_Records&type=code)
    - [Cassandra to Cloud Bigtable](https://github.com/search?q=repo%3AGoogleCloudPlatform%2FDataflowTemplates%20Cassandra_To_Cloud_Bigtable&type=code)
    - [Cloud Bigtable to Avro Files in Cloud Storage](https://github.com/search?q=repo%3AGoogleCloudPlatform%2FDataflowTemplates%20Cloud_Bigtable_to_GCS_Avro&type=code)
    - [Cloud Bigtable to Parquet Files on Cloud Storage](https://github.com/search?q=repo%3AGoogleCloudPlatform%2FDataflowTemplates%20Cloud_Bigtable_to_GCS_Parquet&type=code)
    - [Cloud Bigtable to SequenceFile Files on Cloud Storage](https://github.com/search?q=repo%3AGoogleCloudPlatform%2FDataflowTemplates%20Cloud_Bigtable_to_GCS_SequenceFile&type=code)
    - [Cloud Spanner to Avro Files on Cloud Storage](https://github.com/search?q=repo%3AGoogleCloudPlatform%2FDataflowTemplates%20Cloud_Spanner_to_GCS_Avro&type=code)
    - [Cloud Spanner to Text Files on Cloud Storage](https://github.com/search?q=repo%3AGoogleCloudPlatform%2FDataflowTemplates%20Spanner_to_GCS_Text&type=code)
    - [Cloud Storage To Splunk](https://github.com/search?q=repo%3AGoogleCloudPlatform%2FDataflowTemplates%20GCS_To_Splunk&type=code)
    - [Cloud Storage to Elasticsearch](https://github.com/search?q=repo%3AGoogleCloudPlatform%2FDataflowTemplates%20GCS_to_Elasticsearch&type=code)
    - [Dataplex JDBC Ingestion](https://github.com/search?q=repo%3AGoogleCloudPlatform%2FDataflowTemplates%20Dataplex_JDBC_Ingestion&type=code)
    - [Dataplex: Convert Cloud Storage File Format](https://github.com/search?q=repo%3AGoogleCloudPlatform%2FDataflowTemplates%20Dataplex_File_Format_Conversion&type=code)
    - [Dataplex: Tier Data from BigQuery to Cloud Storage](https://github.com/search?q=repo%3AGoogleCloudPlatform%2FDataflowTemplates%20Dataplex_BigQuery_to_GCS&type=code)
    - [Firestore (Datastore mode) to BigQuery](https://github.com/search?q=repo%3AGoogleCloudPlatform%2FDataflowTemplates%20Firestore_to_BigQuery_Flex&type=code)
    - [Firestore (Datastore mode) to Text Files on Cloud Storage](https://github.com/search?q=repo%3AGoogleCloudPlatform%2FDataflowTemplates%20Firestore_to_GCS_Text&type=code)
    - [Google Ads to BigQuery](https://github.com/search?q=repo%3AGoogleCloudPlatform%2FDataflowTemplates%20Google_Ads_to_BigQuery&type=code)
    - [Google Cloud to Neo4j](https://github.com/search?q=repo%3AGoogleCloudPlatform%2FDataflowTemplates%20Google_Cloud_to_Neo4j&type=code)
    - [JDBC to BigQuery](https://github.com/search?q=repo%3AGoogleCloudPlatform%2FDataflowTemplates%20Jdbc_to_BigQuery&type=code)
    - [JDBC to BigQuery with BigQuery Storage API support](https://github.com/search?q=repo%3AGoogleCloudPlatform%2FDataflowTemplates%20Jdbc_to_BigQuery_Flex&type=code)
    - [JDBC to Pub/Sub](https://github.com/search?q=repo%3AGoogleCloudPlatform%2FDataflowTemplates%20Jdbc_to_PubSub&type=code)
    - [MongoDB to BigQuery](https://github.com/search?q=repo%3AGoogleCloudPlatform%2FDataflowTemplates%20MongoDB_to_BigQuery&type=code)
    - [MySQL to BigQuery](https://github.com/search?q=repo%3AGoogleCloudPlatform%2FDataflowTemplates%20MySQL_to_BigQuery&type=code)
    - [Parquet Files on Cloud Storage to Cloud Bigtable](https://github.com/search?q=repo%3AGoogleCloudPlatform%2FDataflowTemplates%20GCS_Parquet_to_Cloud_Bigtable&type=code)
    - [PostgreSQL to BigQuery](https://github.com/search?q=repo%3AGoogleCloudPlatform%2FDataflowTemplates%20PostgreSQL_to_BigQuery&type=code)
    - [SQLServer to BigQuery](https://github.com/search?q=repo%3AGoogleCloudPlatform%2FDataflowTemplates%20SQLServer_to_BigQuery&type=code)
    - [SequenceFile Files on Cloud Storage to Cloud Bigtable](https://github.com/search?q=repo%3AGoogleCloudPlatform%2FDataflowTemplates%20GCS_SequenceFile_to_Cloud_Bigtable&type=code)
    - [Text Files on Cloud Storage to BigQuery](https://github.com/search?q=repo%3AGoogleCloudPlatform%2FDataflowTemplates%20GCS_Text_to_BigQuery&type=code)
    - [Text Files on Cloud Storage to BigQuery with BigQuery Storage API support](https://github.com/search?q=repo%3AGoogleCloudPlatform%2FDataflowTemplates%20GCS_Text_to_BigQuery_Flex&type=code)
    - [Text Files on Cloud Storage to Cloud Spanner](https://github.com/search?q=repo%3AGoogleCloudPlatform%2FDataflowTemplates%20GCS_Text_to_Cloud_Spanner&type=code)
    - [Text Files on Cloud Storage to Firestore (Datastore mode)](https://github.com/search?q=repo%3AGoogleCloudPlatform%2FDataflowTemplates%20GCS_Text_to_Firestore&type=code)
- Utilities
    - [Bulk Compress Files on Cloud Storage](https://github.com/search?q=repo%3AGoogleCloudPlatform%2FDataflowTemplates%20Bulk_Compress_GCS_Files&type=code)
    - [Bulk Decompress Files on Cloud Storage](https://github.com/search?q=repo%3AGoogleCloudPlatform%2FDataflowTemplates%20Bulk_Decompress_GCS_Files&type=code)
    - [Bulk Delete Entities in Firestore (Datastore mode)](https://github.com/search?q=repo%3AGoogleCloudPlatform%2FDataflowTemplates%20Firestore_to_Firestore_Delete&type=code)
    - [Convert file formats between Avro, Parquet & CSV](https://github.com/search?q=repo%3AGoogleCloudPlatform%2FDataflowTemplates%20File_Format_Conversion&type=code)
    - [Streaming Data Generator](https://github.com/search?q=repo%3AGoogleCloudPlatform%2FDataflowTemplates%20Streaming_Data_Generator&type=code)
- Legacy Templates
    - [Bulk Delete Entities in Datastore [Deprecated]](https://github.com/search?q=repo%3AGoogleCloudPlatform%2FDataflowTemplates%20Datastore_to_Datastore_Delete&type=code)
    - [Datastore to Text Files on Cloud Storage [Deprecated]](https://github.com/search?q=repo%3AGoogleCloudPlatform%2FDataflowTemplates%20Datastore_to_GCS_Text&type=code)
    - [Text Files on Cloud Storage to Datastore [Deprecated]](https://github.com/search?q=repo%3AGoogleCloudPlatform%2FDataflowTemplates%20GCS_Text_to_Datastore&type=code)

For documentation on each template's usage and parameters, please see the
official [docs](https://cloud.google.com/dataflow/docs/templates/provided-templates).

## Contributing

To contribute to the repository, see [CONTRIBUTING.md](./CONTRIBUTING.md).

## Release Process

Templates are released in a weekly basis (best-effort) as part of the efforts to
keep [Google-provided Templates](https://cloud.google.com/dataflow/docs/guides/templates/provided-templates)
updated with latest fixes and improvements.

To learn more about this process, or how you can stage your own changes, see [Release Process](./release-process.md).

## More Information

* [Dataflow](https://cloud.google.com/dataflow/docs/overview) - general Dataflow documentation.
* [Dataflow Templates](https://cloud.google.com/dataflow/docs/concepts/dataflow-templates) - basic template concepts.
* [Google-provided Templates](https://cloud.google.com/dataflow/docs/guides/templates/provided-templates) - official documentation for templates provided by Google (the source code is in this repository).
* Dataflow Cookbook: [Blog](https://cloud.google.com/blog/products/data-analytics/introducing-dataflow-cookbook), [GitHub Repository](https://github.com/GoogleCloudPlatform/dataflow-cookbook) - pipeline examples and practical solutions to common data processing challenges.
* [Dataflow Metrics Collector](https://github.com/GoogleCloudPlatform/dataflow-metrics-exporter) -  CLI tool to collect dataflow resource & execution metrics and export to either BigQuery or Google Cloud Storage. Useful for comparison and visualization of the metrics while benchmarking the dataflow pipelines using various data formats, resource configurations etc
* [Apache Beam](https://beam.apache.org)
  - [Overview](https://beam.apache.org/use/beam-overview/)
  - Quickstart: [Java](https://beam.apache.org/get-started/quickstart-java), [Python](https://beam.apache.org/get-started/quickstart-py), [Go](https://beam.apache.org/get-started/quickstart-go)
  - [Tour of Beam](https://tour.beam.apache.org/) - an interactive tour with learning topics covering core Beam concepts from simple ones to more advanced ones.
  - [Beam Playground](https://beam.apache.org/get-started/try-beam-playground/) -  an interactive environment to try out Beam transforms and examples without having to install Apache Beam.
  - [Beam College](https://beamcollege.dev/) - hands-on training and practical tips, including video recordings of Apache Beam and Dataflow Templates lessons.
  - [Getting Started with Apache Beam - Quest](https://www.cloudskillsboost.google/course_templates/724) - A 5 lab series that provides a Google Cloud certified badge upon completion.


**Got questions, feedback, or feature requests? Join our community on [Slack!](https://join.slack.com/t/batfish-org/shared_invite/enQtMzA0Nzg2OTAzNzQ1LTcyYzY3M2Q0NWUyYTRhYjdlM2IzYzRhZGU1NWFlNGU2MzlhNDY3OTJmMDIyMjQzYmRlNjhkMTRjNWIwNTUwNTQ)**

[![codecov](https://codecov.io/gh/batfish/batfish/branch/master/graph/badge.svg)](https://codecov.io/gh/batfish/batfish)

## What is Batfish?

Batfish is a network validation tool that provides correctness guarantees for security, reliability, and compliance by analyzing the configuration of network devices. It builds complete models of network behavior from device configurations and finds violations of network policies (built-in, user-defined, and best-practices).

A primary use case for Batfish is to validate configuration changes *before* deployment (though it can be used to validate deployed configurations as well). Pre-deployment validation is a critical gap in existing network automation workflows. By including Batfish in automation workflows, network engineers can close this gap and ensure that only correct changes are deployed.

**Batfish does NOT require direct access to network devices.** The core analysis requires only the configuration of network devices. This analysis may be enhanced using additional information from the network such as:
* BGP routes received from external peers
* Topology information represented by LLDP/CDP

See [www.batfish.org](http://www.batfish.org) for technical information on how it works.

## What kinds of correctness checks does Batfish support?

[](https://www.youtube.com/channel/UCA-OUW_3IOt9U_s60KvmJYA/videos)
[](https://github.com/batfish/pybatfish/tree/master/jupyter_notebooks)

The [Batfish YouTube channel](https://www.youtube.com/channel/UCA-OUW_3IOt9U_s60KvmJYA/videos) (subscribe!) and [Python notebooks](https://github.com/batfish/pybatfish/tree/master/jupyter_notebooks) illustrate many checks. Batfish checks span a range of network behaviors.
#### Configuration Compliance
* Flag undefined-but-referenced or defined-but-unreferenced structures (e.g., ACLs, route maps)
* Configuration settings for MTUs, AAA, NTP, logging, etc. match templates
* Devices can only be accessed using SSHv2 and password is not null
#### Reliability
* End-to-end reachability is not impacted for any flow after any single-link or single-device failure
* Certain services (e.g., DNS) are globally reachable
#### Security
* Sensitive services can be reached only from specific subnets or devices
* Paths between endpoints are as expected (e.g., traverse a firewall, have at least 2 way ECMP, etc...)
#### Change Analysis
* End-to-end reachability is identical across the current and a planned configuration
* Planned ACL or firewall changes are provably correct and causes no collateral damage for other traffic
* Two configurations, potentially from different vendors, are functionally equivalent

## How do I get started?

#### 1. Run the Batfish service
Getting started with Batfish is easy. Just pull and run the latest `allinone` Docker container that includes Batfish as well as example Jupyter notebooks.

    docker pull batfish/allinone

    docker run --name batfish -v batfish-data:/data -p 8888:8888 -p 9997:9997 -p 9996:9996 batfish/allinone

The second command starts the Batfish service and maps the necessary TCP ports.

##### Advanced Docker configuration:
The amount of memory available to Batfish is determined by the Docker configuration. You may wish to supply the [`--memory` command-line argument](https://docs.docker.com/config/containers/resource_constraints/#limit-a-containers-access-to-memory) to explicitly set this value.

On Linux systems that run the OOM Killer, you may also wish to supply the `--oom-kill-disable` argument, which runs in conjunction with the `--memory` argument to prevent Linux from killing Batfish when there is memory pressure on the system.

#### 2. Browse example notebooks (optional)

If you are new to Batfish, consider walking through our notebooks which highlight different capabilities and use cases of Batfish. Point your browser to [http://localhost:8888](http://localhost:8888), and in the `Password or token:` prompt, enter the token that Jupyter showed when you ran the container (e.g. **token=abcdef123456...**).

Jupyter will show you the list of available notebooks. "Getting Started with Batfish" is a good one to start with. This [README](https://github.com/batfish/pybatfish/tree/master/jupyter_notebooks) explains what each notebook does.

#### 3. Install Pybatfish

To analyze your network configurations, you also need [Pybatfish](https://www.github.com/batfish/pybatfish), a Python 3 SDK to interact with the Batfish service. Though not strictly necessary, we recommend that you install Pybatfish in a [virtual environment](https://docs.python.org/3/library/venv.html).

To install Pybatfish run the following commands (in a virtual environment if applicable):

    python3 -m pip install --upgrade pybatfish

#### 4. Develop your analysis

After installing Pybatfish, use your Python environment of choice (e.g., PyCharm, interactive Python shell, Jupyter, ..) to interact with Batfish. The [notebooks](https://github.com/batfish/pybatfish/tree/master/jupyter_notebooks) provide examples of such scripts.

See complete documentation of Pybatfish on [readthedocs](https://pybatfish.readthedocs.io/en/latest/).

## System Requirements for running Batfish

Batfish can be run on any operating system that supports Docker. The containers are actively tested on Mac OS X and Ubuntu 16.04 LTS.

To get started with the example Jupyter notebooks, all you need is a reasonably capable laptop:

* Dual core CPU
* 8 GB RAM
* 256 GB hard-drive

When you transition to running Batfish on your own network, we recommend a server that at least has:

* Quad-core CPU with 2 threads per CPU
* 32 GB RAM
* 256 GB hard-drive

## Supported Network Device and Operating System List

Batfish supports configurations for a large and growing set of (physical and virtual) devices, including:

* A10 Networks
* Arista
* AWS (VPCs, Network ACLs, VPN GW, NAT GW, Internet GW, Security Groups, etc…)
* Cisco (All Cisco NX-OS, IOS, IOS-XE, IOS-XR and ASA devices)
* Check Point
* [Cumulus](https://github.com/batfish/batfish/wiki/Packaging-snapshots-for-analysis#format-for-cumulus-configuration-files)
* [F5 BIG-IP](https://github.com/batfish/batfish/wiki/Packaging-snapshots-for-analysis#format-for-f5-big-ip-configuration-files)
* Fortinet
* Free-Range Routing (FRR)
* [iptables (on hosts)](https://github.com/batfish/batfish/wiki/Packaging-snapshots-for-analysis#format-for-host-json-files)
* Juniper (All JunOS platforms: MX, EX, QFX, SRX, T-series, PTX)
* Palo Alto Networks
* SONiC

Batfish has limited support for the following platforms:

* Aruba
* Dell Force10
* Foundry

If you'd like support for additional vendors or currently-unsupported configuration features, let us know via [Slack](https://join.slack.com/t/batfish-org/shared_invite/enQtMzA0Nzg2OTAzNzQ1LTcyYzY3M2Q0NWUyYTRhYjdlM2IzYzRhZGU1NWFlNGU2MzlhNDY3OTJmMDIyMjQzYmRlNjhkMTRjNWIwNTUwNTQ) or [GitHub](https://github.com/batfish/batfish/issues/new). We'll try to add support. Or, you can — we welcome pull requests! :)

## License and Dependencies

Batfish is released under The Apache Software License, Version 2.0. All
third-party dependencies are compatible with this licensing.