[![stable](http://badges.github.io/stability-badges/dist/stable.svg)](http://github.com/badges/stability-badges)

**Deprecation Note**: Cloud Tools for Eclipse is only compatible with the App Engine Java 8 runtime, which will reach the [end of support](/appengine/docs/standard/lifecycle/runtime-lifecycle#end_of_support)
on January 31, 2024. After this date, your existing Java 8 applications will continue
to run and receive traffic. However, you cannot deploy new or update existing applications
that use runtimes [after their end of support date](/appengine/docs/standard/lifecycle/support-schedule#java).
We recommend that you [migrate to the latest supported version of Java](/appengine/migration-center/standard/migrate-to-second-gen/java-differences).

This project provides an Eclipse plugin for building, debugging, and deploying Google Cloud Platform applications.

[End user documentation and installation instructions can be found on cloud.google.com.](https://cloud.google.com/eclipse/docs/?utm_source=github&utm_medium=google-cloud-eclipse&utm_campaign=ToolsforEclipse)

_TL;DR_: `mvn package` should
generate a p2-accessible repository in `gcp-repo/target/repository`.

# Development

This project is built using _Maven Tycho_, a set of extensions to
Maven for building Eclipse bundles and features.

## Requirements

1. The [Google Cloud SDK](https://cloud.google.com/sdk/); install this somewhere on your file system and add it to your path.

2. Install the App Engine Java components:

   `$ gcloud components install app-engine-java`

3. JDK 8

4. The [Eclipse IDE](https://www.eclipse.org/downloads/eclipse-packages/).
   It's easiest to use the _Eclipse IDE for Java EE Developers_ package. You must use
   Eclipse 4.8 - 4.21 (versions since 4.22 are currently not supported).  We use _target platforms_ to support building
   for earlier versions of Eclipse.  You also need the following:

   1. The [M2Eclipse plugin](http://www.eclipse.org/m2e/) (also called m2e) is
      required to import the projects into Eclipse.  M2Eclipse is included in
      [several packages](https://www.eclipse.org/downloads/compare.php?release=photon),
      such as the _Eclipse IDE for Java EE Developers_ package.

   2. The [m2e connector for maven-dependency-plugin](https://github.com/ianbrandt/m2e-maven-dependency-plugin)
      should be installed from `http://ianbrandt.github.io/m2e-maven-dependency-plugin/`.
      This connector should be prompted for by M2Eclipse.  If not, use
      _Preferences > Maven > Discovery > Open Catalog_ and search
      for _Dependency_ and install.

   3. The [Google Java Format plugin for Eclipse](https://github.com/google/google-java-format/).
      Download the [latest version](https://github.com/google/google-java-format/releases/)
      and place the jar into your Eclipse installation's `dropins/` directory
      (on MacOS this may be in `Eclipse.app/Contents/Eclipse/dropins/`).

5. Maven 3.5.0 or later.  Although m2eclipse is bundled with its own Maven install,
   Maven is necessary to test command-line builds.

6. git (optional: you can use EGit from within Eclipse instead)

7. Clone the project to a local directory using `git clone
   https://github.com/GoogleCloudPlatform/google-cloud-eclipse.git`.

## Running Tests

If you're using Goobuntu, first point JAVA_HOME at a stock OpenJDK, not at the Google JDK:

```sh
$ export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
```

If all you want is to run tests, just do `mvn verify`.

However, the integration tests will interfere with your display, disrupt your using your computer, and likely fail if you touch the screen while the integration tests are running. A nice solution on Unix-like systems is to do `xvfb-run mvn verify`. (Install [Xvfb](https://linux.die.net/man/1/xvfb) if necessary.) `xvfb-run` will use a virtual display, and you and the tests will not be disrupted.

## Configuring Maven/Tycho Builds

The plugin is built using Maven/Tycho and targeted to Java 8.

The tests need to find the Google Cloud SDK.  You can either:

  1. Place the _SDK_`/bin` directory on your `PATH`
  2. Set `GOOGLE_CLOUD_SDK_HOME` to point to your SDK

### Changing the Eclipse Platform compilation and testing target

By default, the build is targeted against Eclipse `2018-09` (4.9).
You can explicitly set the `eclipse.target` property to `2018-12` (4.10).
```
$ mvn -Declipse.target=2018-12 package
```

### Adding a new bundle/fragment

We normally put production code into a bundle and tests as a fragment hosted
by that bundle, put under the `plugins/` directory.
For now we have been committing both the `pom.xml` and Eclipse's
`.project`, `.classpath`, and `.settings/` files.  We have a master set
of project settings in [`eclipse/settings`](eclipse/settings/); see the
[`README.md`](eclipse/settings/README.md) for more details.

Our CI process is configured to run our tests with JaCoCo, which requires
some additional configuration to add new bundles and fragments
in `build/jacoco/`.

## Import into Eclipse

We pull in some dependencies directly from Maven-style repositories,
such as Maven Central and the Sonatype staging repository, which isn't
directly supported within Eclipse.  We have a few hoops to jump
through to set up a working development environment.

### Assemble the IDE Target Platform

The Eclipse IDE and Tycho both use a _Target Platform_ to manage
the dependencies for the source bundles and features under development.
Although Tycho can pull dependencies directly from Maven-style
repositories (like [Maven Central](https://search.maven.org)), Eclipse
cannot.  So we use Tycho to cobble together
a target platform suitable for the Eclipse IDE with the following command.
```
$ (cd eclipse; mvn -Pide-target-platform package)        # may want -Declipse.target=XXX
```
This command creates a local copy of the
target platform, including any Maven dependencies, into
[`eclipse/ide-target-platform/target/repository`](eclipse/ide-target-platform/target/repository).
You will use this repository to create a target platform within the IDE,
as described below.

The Eclipse version used for the target platform is affected by the
`eclipse.target` property, described below.

You must regenerate the target platform and reconfigure the IDE's
target platform whenever dependencies are updated.

### Steps to import into the Eclipse IDE

1. Setup JDK 8 in Eclipse (this may already be set up by Eclipse's JRE/JDK auto-discovery)

   1. Select `Window/Preferences` (on Mac `Eclipse/Preferences`).

   2. Under `Java/Installed JREs` click `Add`.

   3. Select Standard VM and click `Next`.

   4. Select the folder that contains the JDK 8 installation by clicking
      `Directory`.

   5. Click `Finish`.

   6. Select `Java/Installed JREs/Execution Environments` page.

   7. Click on `JavaSE-1.8` in the list on the left under `Execution
      Environments:`.

   8. The JDK just added should show up in the list on the right along with other
      installed JDKs/JREs. Set the checkbox next to the JDK 8 added in the
      previous steps to mark it as compatible with the `JavaSE-1.8` execution
      environment.

   9. Click `OK`.

2. Set up the Target Platform: you will need to repeat this process whenever
   items are changed in the target platform, such as a new release of the
   `appengine-plugins-core`.

   1. As described above, you must first build the target platform with Maven:

      `$ (cd eclipse; mvn -Pide-target-platform package)`

   2. Open the `Preferences` dialog, go to `Plug-in Development` > `Target Platform`.

   3. Click `Add...` > `Nothing` to create a new Target Platform.

   4. Name it `GCP IDE Target Platform`.

   5. Select `Add` > `Software Site`.

   6. Select the `Add...` button (found beside the `Work with:` field) and then select `Local`
      to find a local repository. Navigate to `.../eclipse/ide-target-platform/target/repository`,
      and click `OK`.

   7. Once the main content populates, check the `Uncategorized` item to pull in all items. Click `Finish`.

   8. Click `Finish` to complete the new target platform definition.

   9. Select your new target platform (instead of Running Platform) in the `Target Platform` preferences.

   10. Click `Apply and Finish` to load this new target platform.

   11. Eclipse will load the target.

3. Import the projects.

   1. Select `File/Import...` menu in Eclipse.

   2. Select `Existing Maven Projects` from the list.

   3. Click `Browse...` and select the directory that
      contains the project.

   4. Under `Projects:` the `pom.xml` files representing modules should be
      displayed. Make sure that all of them are selected _except_
      `verify_feature_completeness` and the sub-directories under
      `eclipse`. You probably need to explicitly unselect these
      projects. Click `Finish`.

      - The subprojects under the `eclipse` directory define target platforms for the Tycho build. It's easier to edit the files from the `eclipse-setup` project.

   5. Maven may prompt you to install several additional plugin connector plugins from
      [Tycho](https://eclipse.org/tycho/) if they are not already installed. Click
      `Finish` to install them. If Eclipse prompts you to install any other
      plugins, do so.

   6. Restart Eclipse when prompted.

4. Check the imported projects.

   - There should be no errors in the `Markers` or `Problems` views in Eclipse.
      However you may see several low-priority warnings.

      - You may see Maven-related errors like _"plugin execution not
        covered by lifecycle configuration"_.
        If so, right-click on the problem and select
        _Quick Fix_ > _Discover new m2e connectors_
	and follow the process to install the recommended plugin
	connectors.

5. Create and initialize a launch configuration:

   1. Right-click the `gcloud-eclipse-tools.launch` file under the
      `google-cloud-eclipse` module in the `Package Explorer`.

   2. Select `Run As` > `Run Configurations...`

   3. Set variables required for launch:

      1. Go to the second tab for `Arguments`

      2. Click the `Variables...` button for `VM argument:`

      3. Click the `Edit variables...` button

      4. Click `New...`

      5. Set the name to `oauth_id`, and the value to the value you want to use
        (description optional)

      6. Click `OK`, the variable will appear in the list

      7. Repeat steps 6-8 but use `oauth_secret` as the name and use the
         corresponding value

      8. Click `OK` to close the edit variables dialog

      9. Click `Cancel` to close the variable selection dialog

      10. Click `Apply` to apply the changes to the run config

   4. From the `Run` menu, select `Run History > gcloud-eclipse-tools`. A new instance of Eclipse launches with the plugin installed.

# Updating Target Platforms

### Updating the `.target` files

We use _Target Platform_ files (`.target`) to collect the dependencies used
for the build.  These targets specify exact versions of the bundles and
features being built against. We currently maintain three target platforms,
targeting the latest version of the current, previous, and next releases.
This is currently:

  - Eclipse 2018-09 (4.9): [`eclipse/eclipse-2018-09/gcp-eclipse-2018-09.target`](eclipse/eclipse-2018-09/gcp-eclipse-2018-09.target)
  - Eclipse 2018-12 (4.10): [`eclipse/eclipse-2018-12/gcp-eclipse-2018-12.target`](eclipse/eclipse-2018-12/gcp-eclipse-2018-12.target)

These `.target` files are generated and *should not be manually updated*.
Updating `.target` files directly becomes a chore once it has more than a
couple of dependencies.  We instead generate these `.target`s from
_Target Platform Definition_ `.tpd` files.
The `.tpd` files use a simple DSL to specify the bundles and features,
and the location of the repositories containing them.
The `.tpd` files are processed using the [TPD Editor](https://github.com/eclipse-cbi/targetplatform-dsl)
which resolves the specified dependencies and creates a `.target`.
The process is:

   1. Install the TPD Editor, if necessary
      - Use _Help > Install New Software_ and specify `https://download.eclipse.org/cbi/tpd/3.0.0-SNAPSHOT/`
        as the location.
      - Restart Eclipse when prompted
   2. Open the `.tpd` file in Eclipse.
   3. Make any necessary changes and save.
      - Note that the TPDs specify artifacts using their _p2 identifiers_.
        Bundles are specified using their OSGi Bundle Symbolic Name (e.g.,
        `org.eclipse.core.runtime`).
        Features are specified using their Feature ID suffixed with `.feature.group`
        (e.g., `org.eclipse.rcp.feature.group`).
   4. Select the file in the Package Explorer, right-click, and choose _Create Target Definition File_
      to update the corresponding .target file.

Both the `.tpd` and `.target` files should be committed.

### Updating Dependencies

The IDE Target Platform needs to be rebuilt at the command line
and reimported into Eclipse when dependency versions are changed:

1. `(cd eclipse; mvn -Pide-target-platform package)`
2. Preferences > Plug-in Development > Target Platforms
3. Select your target ("GCP IDE Target Platform") and click Edit
4. Select the location and click Reload to cause any cached info to be discarded.
5. Click Edit and then select Uncategorized.
6. Finish / OK until done.

### Updating the Eclipse IDE Target Platforms

The IDE Target Platform, defined in `eclipse/ide-target-platform`,
may need to be updated when dependencies are added or removed.  The
contents are defined in the `category.xml` file, which specifies
the list of features and bundles that should be included.  This
file can be edited using the Category editor in Eclipse.  Ideally
the version should be specified as `"0.0.0"` to indicate that the
current version found should be used.  Unlike the `.tpd` file,
the identifiers are not p2 identifiers, and so features do not
require the `.feature.group` suffix.

## Other Miscellaneous Dev Tasks

### Updating IDE settings

See [`eclipse/settings/`](eclipse/settings/README.md) for details.

### Configuring Maven/Tycho Toolchains for new JDK releases

Now that OpenJDK is moving to a 6-month release cycle...

We use Tycho's support for Maven Toolchains to ensure that new
language features do not creep into the code.  Tycho's support is
automatically enabled in the build when compiling with a newer JDK
than our minimium supported platform.  When using such a JDK, currently
anything later than JDK 8, we configure the Tycho compiler plugin to
use the [`useJDK=BREE`](https://eclipse.org/tycho/sitedocs/tycho-compiler-plugin/compile-mojo.html)
setting to ensure bundles are *compiled* with a JDK that matches
the bundle's `Bundle-RequiredExecutionEnvironment`.  However we leave
`tycho-surefire` to run the tests using the current toolchain
(the default for
[`useJDK=SYSTEM`](https://eclipse.org/tycho/sitedocs/tycho-surefire/tycho-surefire-plugin/test-mojo.html#useJDK))
so as to catch any non-backwards compatible changes.

These settings require configuring
[Maven's toolchains](https://maven.apache.org/guides/mini/guide-using-toolchains.html)
to point to appropriate JRE installations.  Tycho further requires
that a toolchain defines an `id` for the specified _Execution
Environment_ identifier.  For example, a `~/.m2/toolchains.xml` to
configure Maven on macOS for 8, and 11 toolchains might be:

```xml
<?xml version="1.0" encoding="UTF8"?>
<toolchains>
  <toolchain>
    <type>jdk</type>
    <provides>
      <id>JavaSE-11</id>
      <version>11</version>
      <vendor>openjdk</vendor>
    </provides>
    <configuration>
      <jdkHome>/Library/Java/JavaVirtualMachines/jdk-11.jdk/Contents/Home</jdkHome>
    </configuration>
  </toolchain>
  <toolchain>
    <type>jdk</type>
    <provides>
      <id>JavaSE-1.8</id>
      <version>1.8</version>
      <vendor>oracle</vendor>
    </provides>
    <configuration>
      <jdkHome>/Library/Java/JavaVirtualMachines/jdk1.8.0_181.jdk/Contents/Home/jre</jdkHome>
    </configuration>
  </toolchain>
</toolchains>
```

Note that _jdkHome_ for `JavaSE-1.8` specifies the
`jre/` directory: Tycho sets the default boot classpath to
_jdkHome_`/lib/*`, _jdkHome_`/lib/ext/*`, and _jdkHome_`/lib/endorsed/*`.
For many JDKs, including Oracle's JDK and the OpenJDK *prior to Java 9*, those
directories are actually found in the `jre/` directory.  Compilation
errors such as `java.lang.String` not found and `java.lang.Exception`
not found indicate a misconfigured _jdkHome_.  With the introduction of
_Java modules_ with Java 9, there is no longer a separate JRE distribution.


# Kafka Connect for Azure Cosmos DB (SQL API)

[![Open Source Love svg2](https://badges.frapsoft.com/os/v2/open-source.svg?v=103)](https://github.com/microsoft/kafka-connect-cosmosdb/blob/dev/CODE_OF_CONDUCT.MD) [![PRs Welcome](https://img.shields.io/badge/PRs-welcome-brightgreen.svg?style=flat-square)](https://github.com/microsoft/kafka-connect-cosmosdb/blob/dev/CONTRIBUTING.md) [![Maintenance](https://img.shields.io/badge/Maintained%3F-yes-green.svg)](https://github.com/microsoft/kafka-connect-cosmosdb/pulse)

[![Java CI with Maven](https://github.com/microsoft/kafka-connect-cosmosdb/workflows/Java%20CI%20with%20Maven/badge.svg)](https://github.com/microsoft/kafka-connect-cosmosdb/actions?query=workflow%3A%22Java+CI+with+Maven%22) [![Release](https://img.shields.io/github/v/release/microsoft/kafka-connect-cosmosdb?include_prereleases&label=Latest%20Release)](https://github.com/microsoft/kafka-connect-cosmosdb/releases/)

## Introduction

File any issues / feature requests / questions etc. you may have in the [Issues](https://github.com/microsoft/kafka-connect-cosmosdb/issues) for this repo.

This project provides connectors for [Kafka Connect](http://kafka.apache.org/documentation.html#connect) to read from and write data to [Azure Cosmos DB](https://azure.microsoft.com/en-us/services/cosmos-db/)(SQL API).

The connectors in this repository are specifically for the Cosmos DB **SQL API**.
If you are using Cosmos DB with other APIs then there is likely a specific connector for that API, but it's not this one.
- Mongo API - [MongoDB Kafka Connector](https://www.mongodb.com/docs/kafka-connector/current/)
- Cassandra API - [Cassandra Sink Connector](https://docs.confluent.io/kafka-connect-cassandra/current/overview.html)
- Gremlin API - [Kafka Connect connector for Cosmos DB Gremlin API](https://github.com/Azure/kafka-connect-cosmosdb-graph)

## Exactly-Once Support

- **Source** Connector
  - For the time being, this connector supports at-least once with multiple tasks and exactly-once for single tasks.
- **Sink** Connector
  - The sink connector fully supports exactly-once semantics.

## Supported Data Formats

The sink & source connectors are configurable in order to support:

| Format Name  | Description |
| :----------- | :---------- |
| JSON (Plain) | JSON record structure without any attached schema. |
| JSON with Schema | JSON record structure with explicit schema information to ensure the data matches the expected format. |
| AVRO | A row-oriented remote procedure call and data serialization framework developed within Apache's Hadoop project. It uses JSON for defining data types and protocols, and serializes data in a compact binary format.

Since key and value settings, including the format and serialization, can be independently configured in Kafka, it is possible to work with different data formats for records' keys and values respectively.

To cater for this there is converter configuration for both *key.converter* and *value.converter*.

### Converter Configuration Examples

#### JSON (Plain)

- If you need to use JSON without Schema Registry for Connect data, you can use the JsonConverter supported with Kafka. The example below shows the JsonConverter key and value properties that are added to the configuration:

  ```properties
  key.converter=org.apache.kafka.connect.json.JsonConverter
  key.converter.schemas.enable=false
  value.converter=org.apache.kafka.connect.json.JsonConverter
  value.converter.schemas.enable=false
  ```

#### JSON with Schema

- When the properties `key.converter.schemas.enable` and `value.converter.schemas.enable` are set to true, the key or value is not treated as plain JSON, but rather as a composite JSON object containing both an internal schema and the data.

  ```properties
  key.converter=org.apache.kafka.connect.json.JsonConverter
  key.converter.schemas.enable=true
  value.converter=org.apache.kafka.connect.json.JsonConverter
  value.converter.schemas.enable=true
  ```

- The resulting message to Kafka would look like the example below, with schema and payload top-level elements in the JSON:

  ```json
  {
    "schema": {
      "type": "struct",
      "fields": [
        {
          "type": "int32",
          "optional": false,
          "field": "userid"
        },
        {
          "type": "string",
          "optional": false,
          "field": "name"
        }
      ],
      "optional": false,
      "name": "ksql.users"
    },
    "payload": {
      "userid": 123,
      "name": "user's name"
    }
  }
  ```

> NOTE: The message written is made up of the schema + payload. Notice the size of the message, as well as the proportion of it that is made up of the payload vs. the schema. This is repeated in every message you write to Kafka. In scenarios like this, you may want to use a serialisation format like JSON Schema or Avro, where the schema is stored separately and the message holds just the payload.

#### AVRO

- This connector supports AVRO. To use AVRO you need to configure a AvroConverter so that Kafka Connect knows how to work with AVRO data. This connector has been tested with the [AvroConverter](https://www.confluent.io/hub/confluentinc/kafka-connect-avro-converter) supplied by Confluent, under Apache 2.0 license, but another custom converter can be used in its place instead if you prefer.

- Kafka deals with keys and values independently, you need to specify the `key.converter` and `value.converter` properties as required in the worker configuration.

- An additional converter property must also be added, when using  AvroConverter, that provides the URL for the Schema Registry.

The example below shows the AvroConverter key and value properties that are added to the configuration:

  ```properties
  key.converter=io.confluent.connect.avro.AvroConverter
  key.converter.schema.registry.url=http://schema-registry:8081
  value.converter=io.confluent.connect.avro.AvroConverter
  value.converter.schema.registry.url=http://schema-registry:8081
  ```

### Choosing a conversion format

- If you're configuring a **Source connector** and
  - If you want Kafka Connect to include plain JSON in the message it writes to Kafka, you'd set [JSON (Plain)](#json-plain) configuration.
  - If you want Kafka Connect to include the schema in the message it writes to Kafka, you’d set [JSON with Schema](#json-with-schema) configuration.
  - If you want Kafka Connect to include AVRO format in the message it writes to Kafka, you'd set [AVRO](#avro) configuration.

- If you’re consuming JSON data from a Kafka topic in to a **Sink connector**, you need to understand how the JSON was serialised when it was written to the Kafka topic:
  - If it was with JSON serialiser, then you need to set Kafka Connect to use the JSON converter `(org.apache.kafka.connect.json.JsonConverter)`.
    - If the JSON data was written as a plain string, then you need to determine if the data includes a nested schema/payload. If it does,then you would set, [JSON with Schema](#json-with-schema) configuration.
    - However, if you’re consuming JSON data and it doesn’t have the schema/payload construct, then you must tell Kafka Connect **not** to look for a schema by setting `schemas.enable=false` as per [JSON (Plain)](#json-plain) configuration.
  - If it was with AVRO serialiser, then you need to set Kafka Connect to use the AVRO converter `(io.confluent.connect.avro.AvroConverter)` as per [AVRO](#avro) configuration.

#### Common Errors

Some of the [common errors](https://www.confluent.io/blog/kafka-connect-deep-dive-converters-serialization-explained/#common-errors) you can get if you misconfigure the converters in Kafka Connect. These will show up in the sinks you configure for Kafka Connect, as it’s this point at which you’ll be trying to deserialize the messages already stored in Kafka. Converter problems tend not to occur in sources because it’s in the source that the serialization is set.

![Converter Configuration Errors](doc/images/converter-misconfigurations.png "CosmosDB Converter Configurations")

## Configuration

### Common Configuration Properties

The Sink and Source connectors share the following common configuration properties

| Name | Type | Description | Required/Optional |
| :--- | :--- | :--- | :--- |
| connect.cosmos.connection.endpoint | uri | Cosmos endpoint URI string | Required |
| connect.cosmos.master.key | string | The Cosmos primary key that the sink connects with | Required |
| connect.cosmos.databasename | string | The name of the Cosmos database the sink writes to | Required |
| connect.cosmos.containers.topicmap | string | Mapping between Kafka Topics and Cosmos Containers, formatted using CSV as shown: `topic#container,topic2#container2` | Required |

For Sink connector specific configuration, please refer to the [Sink Connector Documentation](./doc/README_Sink.md)

For Source connector specific configuration, please refer to the [Source Connector Documentation](./doc/README_Source.md)

## Project Setup

Please refer [Developer Walkthrough and Project Setup](./doc/Developer_Walkthrough.md) for initial setup instructions.

## Performance Testing

For more information on the performance tests run for the Sink and Source connectors, refer to the [Performance testing document](./doc/Performance_Testing.md).

Refer to the [Performance Environment Setup](./src/perf/README.md) for exact steps on deploying the performance test environment for the Connectors.

## Dead Letter Queue
We introduced the standard dead level queue from Kafka. For more info see: https://www.confluent.io/blog/kafka-connect-deep-dive-error-handling-dead-letter-queues

## Resources

- [Kafka Connect](http://kafka.apache.org/documentation.html#connect)
- [Kafka Connect Deep Dive – Converters and Serialization Explained](https://www.confluent.io/blog/kafka-connect-deep-dive-converters-serialization-explained/)


![Cloudogu logo](https://cloudogu.com/images/logo.png)

[![GitHub license](https://img.shields.io/github/license/cloudogu/smeagol.svg)](https://github.com/cloudogu/smeagol/blob/master/LICENSE)
[![GitHub release](https://img.shields.io/github/release/cloudogu/smeagol.svg)](https://github.com/cloudogu/smeagol/releases)

# Smeagol Dogu
[https://cloudogu.com](https://cloudogu.com/?mtm_campaign=smeagol&mtm_kwd=website&mtm_source=github&mtm_medium=link)

This repository contains smeagol, an wiki server which stores its files in a git repository.

## Development

### Environment
> **Note**: If you intend to use your local ecosystem, which will also reside on `192.168.56.2` by default, you have to set an etcd key first.
> This can be done in vagrant with the command `etcdctl set /config/_global/stage "development"` and restarting CAS in vagrant via `docker restart cas`.

#### Setting up a development ecosystem via groovy
Smeagol requires a proper configured Cloudogu EcoSystem in order to work properly.
This can be achieved by using the provided groovy execution:

```bash
./mvnw groovy:execute@up
```

This command will clone the ecosystem repository into the `.workspace` folder, creates a `setup.json` with a proper
configuration and starts the vm. After the vm has started, open the setup in your browser at
`http://192.168.56.2:8080`, register the instance and finish the setup.

Then you can access SCM-Manager at: `https://192.168.56.2/scm/`.

Username: `admin`, Password: `adminpw`

With those credentials you can also enter the VM via SSH:

* `ssh admin@192.168.56.2`
* or `cd .workspace/ecosystem; vagrant ssh`

Hint: Make sure your SCM-Manager has the Smeagol Plugin installed!

To stop the vm use the down execution:

```bash
./mvnw groovy:execute@down
```

If the vm is not longer required, it can be removed with the destroy execution:

```bash
./mvnw groovy:execute@destroy
```

### REST API

The server for the REST API can be started with the run goal of the spring-boot maven plugin:
```bash
./mvnw clean spring-boot:run
```
To access the development instance open `http://192.168.56.1:8080` in your browser.
For example, you can get all repositories from `http://192.168.56.1:8080/smeagol/api/v1/repositories`.

### UI

Make sure your SCM-Manager has the Smeagol Plugin installed!
- https://192.168.56.2/scm/admin/plugins/available/

If Smeagol runs on your local host and you use CAS and SCM-Manger inside a CES instance please make sure the SCM-Manager CAS setting accepts any proxy like so:
- Administration -> Settings -> CAS -> Accept any proxy (enabled)

To start the UI development server, you should use yarn. First install the required dependencies:

```bash
yarn install
```

After the dependency installation has finished, start the development server:

```bash
yarn run start
```

After the RESTApi and the UI are successfully started, we can open Smeagol in our browser at `http://192.168.56.1:8080`.
The spring-boot server from the RESTApi will proxy every ui request to the UI server.

### Hot Reload

Hot reload should work out of the box for the ui.

To enable hot reload for java classes, the application must be started with the maven goal "spring-boot:run". For
Intellij we have to recompile the project or enable "Make project automatically", but this requires some sort of hack.
Please have a look at the description of the issue [IDEA-141638](https://youtrack.jetbrains.com/issue/IDEA-141638).

---
### What is the Cloudogu EcoSystem?
The Cloudogu EcoSystem is an open platform, which lets you choose how and where your team creates great software. Each service or tool is delivered as a Dogu, a Docker container. Each Dogu can easily be integrated in your environment just by pulling it from our registry. We have a growing number of ready-to-use Dogus, e.g. SCM-Manager, Jenkins, Nexus, SonarQube, Redmine and many more. Every Dogu can be tailored to your specific needs. Take advantage of a central authentication service, a dynamic navigation, that lets you easily switch between the web UIs and a smart configuration magic, which automatically detects and responds to dependencies between Dogus. The Cloudogu EcoSystem is open source and it runs either on-premises or in the cloud. The Cloudogu EcoSystem is developed by Cloudogu GmbH under [MIT License](https://cloudogu.com/en/license/?mtm_campaign=smeagol&mtm_kwd=license&mtm_source=github&mtm_medium=link).

### How to get in touch?
Want to talk to the Cloudogu team? Need help or support? There are several ways to get in touch with us:

* [Website](https://cloudogu.com/?mtm_campaign=smeagol&mtm_kwd=website&mtm_source=github&mtm_medium=link)
* [myCloudogu-Forum](https://forum.cloudogu.com/topic/34?ctx=1)
* [Email hello@cloudogu.com](mailto:hello@cloudogu.com)

---
© 2020 Cloudogu GmbH - MADE WITH :heart: FOR DEV ADDICTS. [Legal notice / Impressum](https://cloudogu.com/en/imprint/?mtm_campaign=smeagol&mtm_kwd=imprint&mtm_source=github&mtm_medium=link)


# gitflow-incremental-builder (GIB)
[![GitHub license](https://img.shields.io/github/license/gitflow-incremental-builder/gitflow-incremental-builder.svg)](https://github.com/gitflow-incremental-builder/gitflow-incremental-builder/blob/main/LICENSE)
[![Maven Central](https://img.shields.io/maven-central/v/io.github.gitflow-incremental-builder/gitflow-incremental-builder)](https://maven-badges.herokuapp.com/maven-central/io.github.gitflow-incremental-builder/gitflow-incremental-builder)
[![CI](https://github.com/gitflow-incremental-builder/gitflow-incremental-builder/workflows/CI/badge.svg)](https://github.com/gitflow-incremental-builder/gitflow-incremental-builder/actions?query=workflow%3A%22CI%22+branch%3Amain)
[![Codacy Badge](https://app.codacy.com/project/badge/Grade/2a74eedd5e0c4694ac8cf44b315cfb5a)](https://app.codacy.com/gh/gitflow-incremental-builder/gitflow-incremental-builder/dashboard?utm_source=github.com&utm_medium=referral&utm_content=gitflow-incremental-builder/gitflow-incremental-builder&utm_campaign=Badge_Grade)
[![Codacy Badge](https://app.codacy.com/project/badge/Coverage/2a74eedd5e0c4694ac8cf44b315cfb5a)](https://app.codacy.com/gh/gitflow-incremental-builder/gitflow-incremental-builder/dashboard?utm_source=github.com&utm_medium=referral&utm_content=gitflow-incremental-builder/gitflow-incremental-builder&utm_campaign=Badge_Coverage)
[![Supported JVM Versions](https://img.shields.io/badge/JVM-11_17_21_23--ea-brightgreen.svg?logo=Java)](https://github.com/gitflow-incremental-builder/gitflow-incremental-builder/actions?query=workflow%3A%22CI%22+branch%3Amain)
[![GitHub Discussions](https://img.shields.io/github/discussions/gitflow-incremental-builder/gitflow-incremental-builder)](https://github.com/gitflow-incremental-builder/gitflow-incremental-builder/discussions)

A maven extension for incremental building of multi-module projects when using [feature branches (Git Flow)](http://nvie.com/posts/a-successful-git-branching-model/).
Builds or tests only changed maven modules compared to reference branch in Git (e.g. origin/develop) and all their dependents.

Powered by [JGit](https://www.eclipse.org/jgit/).

This extension is **not limited to Git Flow setups!** The [extensive configuration options](#configuration) provide support for many other branch setups and/or use cases.

## Table of Contents

- [Requirements](#requirements)

- [Usage](#usage)
  - [Usage as a Maven plugin](#usage-as-a-maven-plugin)
  - [Usage as a Maven extension](#usage-as-a-maven-extension)
  - [Disable in IDE](#disable-in-ide)

- [Example](#example)

- [Configuration](#configuration)
  - [gib.help](#gibhelp)
  - [gib.disable](#gibdisable)
  - [gib.disableIfBranchMatches](#gibdisableifbranchmatches)
  - [gib.disableIfReferenceBranchMatches](#gibdisableifreferencebranchmatches)
  - [gib.disableBranchComparison](#gibdisablebranchcomparison)
  - [gib.referenceBranch](#gibreferencebranch)
  - [gib.fetchReferenceBranch](#gibfetchreferencebranch)
  - [gib.baseBranch](#gibbasebranch)
  - [gib.fetchBaseBranch](#gibfetchbasebranch)
  - [gib.compareToMergeBase](#gibcomparetomergebase)
  - [gib.uncommitted](#gibuncommitted)
  - [gib.untracked](#gibuntracked)
  - [gib.skipIfPathMatches](#gibskipifpathmatches)
  - [gib.excludePathsMatching](#gibexcludepathsmatching)
  - [gib.includePathsMatching](#gibincludepathsmatching)
  - [gib.buildAll](#gibbuildall)
  - [gib.buildAllIfNoChanges](#gibbuildallifnochanges)
  - [gib.buildDownstream](#gibbuilddownstream)
  - [gib.buildUpstream](#gibbuildupstream)
  - [gib.buildUpstreamMode](#gibbuildupstreammode)
  - [gib.skipTestsForUpstreamModules](#gibskiptestsforupstreammodules)
  - [gib.argsForUpstreamModules](#gibargsforupstreammodules)
  - [gib.argsForDownstreamModules](#gibargsfordownstreammodules)
  - [gib.forceBuildModules](#gibforcebuildmodules)
  - [gib.excludeDownstreamModulesPackagedAs](#gibexcludedownstreammodulespackagedas)
  - [gib.disableSelectedProjectsHandling](#gibdisableselectedprojectshandling)
  - [gib.failOnMissingGitDir](#failonmissinggitdir)
  - [gib.failOnError](#gibfailonerror)
  - [gib.logImpactedTo](#giblogimpactedto)
  - [gib.logProjectsMode](#giblogprojectsmode)

- [Explicitly selected projects](#explicitly-selected-projects)
  - [mvn -pl](#mvn--pl)
  - [mvn -f and others](#mvn--f-and-others)
  - [mvn -N](#mvn--N)

- [BOM support](#bom-support)

- [Test only changes](#test-only-changes)

- [Authentication](#authentication)
  - [HTTP](#http)
  - [SSH](#ssh)

## Requirements

To be able to use GIB, your project must use:

- **Apache Maven** build tool, version 3.9.8 is recommended
  - The minimum Maven version is 3.6.3 due to [MNG-6580](https://issues.apache.org/jira/browse/MNG-6580)

- **Git** version control

- **Java 11** (or higher)

ℹ If you need support for Maven <= 3.6.2 and/or Java < 11, you might want to consider using [version 3 of GIB](https://github.com/gitflow-incremental-builder/gitflow-incremental-builder/tree/3.x).

## Usage

There are two usage scenarios of GIB:

### Usage as a Maven plugin

ℹ This is the recommended way to configure GIB!

Add to (root) `pom.xml`:
```xml
<build>
    <plugins>
        <plugin>
            <groupId>io.github.gitflow-incremental-builder</groupId>
            <artifactId>gitflow-incremental-builder</artifactId>
            <version>4.5.4</version>
            <extensions>true</extensions>
            <configuration>
                <!-- ... -->
            </configuration>
        </plugin>
        <!-- ... -->
    </plugins>
    <!-- ... -->
</build>
```

The [Configuration](#configuration) can then be added to the `configuration` block of the plugin (and/or via project or system properties).

When defined as a plugin, GIB will still do its work as an extension (see `<extensions>true</extensions>`).

The plugin definition is merely a "shell" to provide `<profile>`-support, better visibility of parameters and works around 3rd-party issues like
[`versions-maven-plugin`: Support for reporting new extenions versions](https://github.com/mojohaus/versions-maven-plugin/issues/74).

:warning: GIB comes with a "fake" mojo (or goal). **Do _not_ try to execute this fake mojo!**

It only exists to generate a plugin descriptor so that you can see the parameters nicely in your IDE or via `maven-help-plugin`.

Because of this limitation, you have to use the general `<configuration>` section of the plugin instead of any `<execution>` block.

### Usage as a Maven extension

Add to (root) `pom.xml`:
```xml
<build>
    <extensions>
        <extension>
            <groupId>io.github.gitflow-incremental-builder</groupId>
            <artifactId>gitflow-incremental-builder</artifactId>
            <version>4.5.4</version>
        </extension>
    </extensions>
    <!-- ... -->
</build>
```
Or use [`.mvn/extensions.xml`](https://maven.apache.org/ref/3.8.8/maven-embedder/core-extensions.html).
If using `.mvn/extensions.xml` and Maven 3.8.8 or higher, it's strongly recommended to add `<classLoadingStrategy>plugin</classLoadingStrategy>` to the `extension` element. See [MNG-7160](https://issues.apache.org/jira/browse/MNG-7160) for more details.

In both cases the [Configuration](#configuration) can then be added via project or system properties.

### Disable in IDE

As IDEs like IntelliJ IDEA or Eclipse usually apply their own custom strategy to building changed modules,
this extension should be disabled in such environments to avoid inconsistencies or errors (e.g. see [issue 49](../../issues/49)).

See [gib.disable](#gibdisable) in the configuration section.

:information_source: If using IntelliJ IDEA, version **2019.3.1** or higher is required for GIB 3.8+ (even if disabled).
See [IDEA-200272](https://youtrack.jetbrains.com/issue/IDEA-200272) and [issue 91](../../issues/91) for more details.

## Example

Maven project `parent` has two submodules `child1` and `child2`:
```
$ ls *
pom.xml

child1:
pom.xml  src

child2:
pom.xml  src
```
`parent` has gitflow-incremental-builder configured:
```
$ cat pom.xml
<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>

    <groupId>parent</groupId>
    <artifactId>parent</artifactId>
    <version>1.0-SNAPSHOT</version>
    <packaging>pom</packaging>

    <properties>
        <gib.referenceBranch>refs/heads/develop</gib.referenceBranch>
    </properties>

    <modules>
        <module>child1</module>
        <module>child2</module>
    </modules>

    <build>
        <extensions>
            <extension>
                <groupId>io.github.gitflow-incremental-builder</groupId>
                <artifactId>gitflow-incremental-builder</artifactId>
                <version>4.5.4</version>
            </extension>
        </extensions>
    </build>

</project>
```
Currently checked out branch is `feature`. There is also a branch `develop`:
```
$ git branch
  develop
* feature
```
Branches are the same now:
```
$ git diff develop
```
Thus incremental does not build and test anything (see [gib.buildAllIfNoChanges](#gibbuildallifnochanges) for an alternate mode):
```
$ mvn test
[INFO] Scanning for projects...
[INFO] gitflow-incremental-builder starting...
[INFO] Git dir is: C:\Users\Vaclav\IdeaProjects\gitflow-incremental-builder\tmp\example\.git
[INFO] Head of branch HEAD is commit of id: commit 17b09f1f9ee9d2a56b7d7bf43d319db529cade9e 1483112738 -----p
[INFO] Head of branch refs/heads/develop is commit of id: commit 17b09f1f9ee9d2a56b7d7bf43d319db529cade9e 1483112738 -----p
[INFO] Using merge base of id: commit 17b09f1f9ee9d2a56b7d7bf43d319db529cade9e 1483112738 -tr-sp
[INFO] ------------------------------------------------------------------------
[INFO] Changed Artifacts:
[INFO]
[INFO]
[INFO] No changed artifacts to build. Executing validate goal only.
[INFO] gitflow-incremental-builder exiting...
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Build Order:
[INFO]
[INFO] child1
[INFO] child2
[INFO] parent
[INFO]
[INFO] ------------------------------------------------------------------------
[INFO] Building child1 1.0-SNAPSHOT
[INFO] ------------------------------------------------------------------------
[INFO]
[INFO] ------------------------------------------------------------------------
[INFO] Building child2 1.0-SNAPSHOT
[INFO] ------------------------------------------------------------------------
[INFO]
[INFO] ------------------------------------------------------------------------
[INFO] Building parent 1.0-SNAPSHOT
[INFO] ------------------------------------------------------------------------
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary:
[INFO]
[INFO] child1 ............................................. SUCCESS [  0.002 s]
[INFO] child2 ............................................. SUCCESS [  0.001 s]
[INFO] parent ............................................. SUCCESS [  0.000 s]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 0.953 s
[INFO] Finished at: 2016-12-30T16:56:35+01:00
[INFO] Final Memory: 9M/162M
[INFO] ------------------------------------------------------------------------
```
Now a file is written and added to git control making branches different:
```
$ echo "text" > child1/src/main/java/file1.txt
$ git add child1/src/main/java/file1.txt
$ git diff develop
diff --git a/child1/src/main/java/file1.txt b/child1/src/main/java/file1.txt
new file mode 100644
index 0000000..8e27be7
--- /dev/null
+++ b/child1/src/main/java/file1.txt
@@ -0,0 +1 @@
+text
warning: LF will be replaced by CRLF in child1/src/main/java/file1.txt.
The file will have its original line endings in your working directory.
```

Thus incremental build builds and tests only child1:
```
$ mvn test
[INFO] Scanning for projects...
[INFO] gitflow-incremental-builder starting...
[INFO] Git dir is: C:\Users\Vaclav\IdeaProjects\gitflow-incremental-builder\tmp\example\.git
[INFO] Head of branch HEAD is commit of id: commit 17b09f1f9ee9d2a56b7d7bf43d319db529cade9e 1483112738 -----p
[INFO] Head of branch refs/heads/develop is commit of id: commit 17b09f1f9ee9d2a56b7d7bf43d319db529cade9e 1483112738 -----p
[INFO] Using merge base of id: commit 17b09f1f9ee9d2a56b7d7bf43d319db529cade9e 1483112738 -tr-sp
[INFO] ------------------------------------------------------------------------
[INFO] Changed Artifacts:
[INFO]
[INFO] child1
[INFO]
[INFO] gitflow-incremental-builder exiting...
[INFO]
[INFO] ------------------------------------------------------------------------
[INFO] Building child1 1.0-SNAPSHOT
[INFO] ------------------------------------------------------------------------
[INFO]
[INFO] --- maven-resources-plugin:2.6:resources (default-resources) @ child1 ---
[WARNING] Using platform encoding (Cp1250 actually) to copy filtered resources, i.e. build is platform dependent!
[INFO] skip non existing resourceDirectory C:\Users\Vaclav\IdeaProjects\gitflow-incremental-builder\tmp\example\child1\src\main\resources
[INFO]
[INFO] --- maven-compiler-plugin:3.1:compile (default-compile) @ child1 ---
[INFO] Changes detected - recompiling the module!
[WARNING] File encoding has not been set, using platform encoding Cp1250, i.e. build is platform dependent!
[INFO] Compiling 1 source file to C:\Users\Vaclav\IdeaProjects\gitflow-incremental-builder\tmp\example\child1\target\classes
[INFO]
[INFO] --- maven-resources-plugin:2.6:testResources (default-testResources) @ child1 ---
[WARNING] Using platform encoding (Cp1250 actually) to copy filtered resources, i.e. build is platform dependent!
[INFO] skip non existing resourceDirectory C:\Users\Vaclav\IdeaProjects\gitflow-incremental-builder\tmp\example\child1\src\test\resources
[INFO]
[INFO] --- maven-compiler-plugin:3.1:testCompile (default-testCompile) @ child1 ---
[INFO] Changes detected - recompiling the module!
[WARNING] File encoding has not been set, using platform encoding Cp1250, i.e. build is platform dependent!
[INFO] Compiling 1 source file to C:\Users\Vaclav\IdeaProjects\gitflow-incremental-builder\tmp\example\child1\target\test-classes
[INFO]
[INFO] --- maven-surefire-plugin:2.12.4:test (default-test) @ child1 ---
[INFO] Surefire report directory: C:\Users\Vaclav\IdeaProjects\gitflow-incremental-builder\tmp\example\child1\target\surefire-reports

-------------------------------------------------------
 T E S T S
-------------------------------------------------------
Running ATest
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.041 sec

Results :

Tests run: 1, Failures: 0, Errors: 0, Skipped: 0

[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 2.043 s
[INFO] Finished at: 2016-12-30T17:01:56+01:00
[INFO] Final Memory: 18M/167M
[INFO] ------------------------------------------------------------------------
```

## Configuration

Maven pom properties configuration with default values is below:
```xml
<properties>
    <gib.help>false</gib.help>                                                         <!-- or -Dgib.h=...     -->
    <gib.disable>false</gib.disable>                                                   <!-- or -Dgib.d=...     -->
    <gib.disableIfBranchMatches></gib.disableIfBranchMatches>                          <!-- or -Dgib.dibm=...  -->
    <gib.disableIfReferenceBranchMatches></gib.disableIfReferenceBranchMatches>        <!-- or -Dgib.dirbm=... -->
    <gib.disableBranchComparison>false</gib.disableBranchComparison>                   <!-- or -Dgib.dbc=...   -->
    <gib.referenceBranch>refs/remotes/origin/develop</gib.referenceBranch>             <!-- or -Dgib.rb=...    -->
    <gib.fetchReferenceBranch>false</gib.fetchReferenceBranch>                         <!-- or -Dgib.frb=...   -->
    <gib.baseBranch>HEAD</gib.baseBranch>                                              <!-- or -Dgib.bb=...    -->
    <gib.fetchBaseBranch>false</gib.fetchBaseBranch>                                   <!-- or -Dgib.fbb=...   -->
    <gib.compareToMergeBase>true</gib.compareToMergeBase>                              <!-- or -Dgib.ctmb=...  -->
    <gib.uncommitted>true</gib.uncommitted>                                            <!-- or -Dgib.uc=...    -->
    <gib.untracked>true</gib.untracked>                                                <!-- or -Dgib.ut=...    -->
    <gib.skipIfPathMatches></gib.skipIfPathMatches>                                    <!-- or -Dgib.sipm=...  -->
    <gib.excludePathsMatching></gib.excludePathsMatching>                              <!-- or -Dgib.epm=...   -->
    <gib.includePathsMatching></gib.includePathsMatching>                              <!-- or -Dgib.ipm=...   -->
    <gib.buildAll>false</gib.buildAll>                                                 <!-- or -Dgib.ba=...    -->
    <gib.buildAllIfNoChanges>false</gib.buildAllIfNoChanges>                           <!-- or -Dgib.bainc=... -->
    <gib.buildDownstream>always</gib.buildDownstream>                                  <!-- or -Dgib.bd=...    -->
    <gib.buildUpstream>derived</gib.buildUpstream>                                     <!-- or -Dgib.bu=...    -->
    <gib.buildUpstreamMode>changed</gib.buildUpstreamMode>                             <!-- or -Dgib.bum=...   -->
    <gib.skipTestsForUpstreamModules>false</gib.skipTestsForUpstreamModules>           <!-- or -Dgib.stfum=... -->
    <gib.argsForUpstreamModules></gib.argsForUpstreamModules>                          <!-- or -Dgib.afum=...  -->
    <gib.argsForDownstreamModules></gib.argsForDownstreamModules>                      <!-- or -Dgib.afdm=...  -->
    <gib.forceBuildModules></gib.forceBuildModules>                                    <!-- or -Dgib.fbm=...   -->
    <gib.excludeDownstreamModulesPackagedAs></gib.excludeDownstreamModulesPackagedAs>  <!-- or -Dgib.edmpa=... -->
    <gib.disableSelectedProjectsHandling>false</gib.disableSelectedProjectsHandling>   <!-- or -Dgib.dsph=...  -->
    <gib.failOnMissingGitDir>true</gib.failOnMissingGitDir>                            <!-- or -Dgib.fomgd=... -->
    <gib.failOnError>true</gib.failOnError>                                            <!-- or -Dgib.foe=...   -->
    <gib.logImpactedTo></gib.logImpactedTo>                                            <!-- or -Dgib.lit=...   -->
    <gib.logProjectsMode>changed</gib.logProjectsMode>                                 <!-- or -Dgib.lpm=...   -->
</properties>
```

Each property can also be set via `-D` on the command line and to reduce typing to a minimum, each property has a short name (see code block above).

E.g. `-Dgib.d=true` yields the same result as `-Dgib.disable=true`.

Properties that support the value `true` can be specified _without_ a value, e.g. `-Dgib.disable` is the same as `-Dgib.disable=true`.

System properties (`-D`) take precedence over project properties from the POM and secondarily to that a full name takes precedence over the respective short name.

Short names can only be used as system properties.

If GIB is [used as a **plugin**](#usage-as-a-maven-plugin) (instead of an extension), the same properties _can_ still be used but it is recommended
to put the properties into the plugin `<configuration>`-section, but _without_ the `gib.`-prefix.

E.g. instead of:
```xml
<properties>
    <gib.uncommitted>true</gib.uncommitted>
</properties>
```
use:
```xml
<configuration>
    <uncommitted>true</uncommitted>
</configuration>
```
Just like for regular plugins, a plugin config property that is set to a _fixed_ value **cannot be changed via the respective system property** (here: `-Dgib.uncommitted=false`).

### gib.help

Logs the available properties etc.

Note: Independent of `gib.disable`.

Since: 3.9.0

### gib.disable

Can be used to disable this extension temporarily or permanently (e.g. to avoid clashes with IDE building strategy).

Since: 3.11.2 (replaces previously used `enabled` property)

### gib.disableIfBranchMatches

Can be used to disable this extension on certain branches (e.g. `main|develop`). By default, GIB runs on all branches.

Since: 3.11.0

Was renamed from `disableIfBranchRegex` in 3.13.0.

### gib.disableIfReferenceBranchMatches

Can be used to disable this extension on certain _reference_ branches (e.g. `origin/(main|develop)`). By default, GIB runs for all reference branches.

See also: [gib.referenceBranch](#gibreferencebranch)

Since: 3.14.2

### gib.disableBranchComparison

Disables the comparison between `baseBranch` and `referenceBranch`. This property should be enabled if _only_ uncommitted and/or untracked files shall be detected to only build projects that have been changed since the last commit in the current branch (see `gib.uncommitted` and `gib.untracked`).
The following properties are _not_ evaluated when `gib.disableBranchComparison` is enabled:
- `gib.referenceBranch`
- `gib.compareToMergeBase`
- `gib.fetchReferenceBranch`

### gib.referenceBranch

The branch to compare `baseBranch` to. You can use the simple branch name to compare against a local branch, e.g. `develop`.

If you want to compare to a _remote_ branch you need to use the prefix `refs/remotes/<remoteName>/`, e.g. `refs/remotes/origin/develop`.

You can also use a tag name, e.g. `refs/tags/myTag` or just `myTag`.

### gib.fetchReferenceBranch

Fetches the `referenceBranch` from the remote repository.

You _must_ use the prefix `refs/remotes/<remoteName>/`, e.g. `refs/remotes/origin/develop`, so that GIB can determine the remote repo name.

Alternatively, since 3.15.0, you can also fetch tags via `refs/tags/<tagName>`.

See also: [Authentication](#authentication)

### gib.baseBranch

The branch that is compared to `referenceBranch`. Usually just the current `HEAD`.

See also: [gib.referenceBranch](#gibreferencebranch)

### gib.fetchBaseBranch

Fetches the `baseBranch` from the remote repository.

See also:

- [Authentication](#authentication)
- [gib.fetchReferenceBranch](#gibfetchreferencebranch)

### gib.compareToMergeBase

Controls whether or not to the [merge-base](https://git-scm.com/docs/git-merge-base) mechanism to compare the branches.

### gib.uncommitted

Detects changed files that have not yet been committed. This does **not** include _untracked_ files (see `git status` manual).

### gib.untracked

Detects files that are not yet tracked by git (see `git status` manual). This does **not** include _uncommitted_ files. A new file is not _untracked_ anymore after it is added to the index.

### gib.skipIfPathMatches

Can be used to skip this extension on certain changes. By default, no changed file results in GIB being skipped.

This property is helpful for more complex project setups in which not all submodules depend on the root module (= are not direct or indirect children of the root module).

In such a case, GIB will assign a change to e.g. `.github/workflows/ci.yml` to the root module, building all modules that depend on the root module.

Consequently, this will **not** include the other modules that are referenced top-down from root, but not bottom-up via `<parent>...</parent>`.

One example of such modules are [the independent-projects of Quarkus](https://github.com/quarkusio/quarkus/tree/1.11.3.Final/independent-projects).

By setting this property to e.g. `\.github[/\\].*` you can tell GIB that this path has a _global_ effect and that it's pointless to attempt an incremental build.

:information_source: Use `[/\\]` instead of just `/` to also cover Windows path separators.

Since: 3.12.2

### gib.excludePathsMatching

Can be used to exclude certain changed files from being detected as changed, reducing the number of modules to build. By default, nothing is excluded.

The regular expression does _not_ need to describe the entire (absolute) path, but only the relevant part _inside_ the git repository context. Example:
```
/tmp/repo/excluded/some-file.txt
```
will be excluded when using `-Dgib.excludePathsMatching=.*excluded.*` or `-Dgib.excludePathsMatching=.*some-file\..*` etc., but is _not_ excluded when adding to the regular expression anything _outside_ of the git repository context like `/tmp/repo` or `repo`.

This the opposite of [gib.includePathsMatching](#gibincludepathsmatching) which can be combined with this property, but `gib.excludePathsMatching` will take precedence.

:information_source: Use `[/\\]` instead of just `/` to also cover Windows path separators.

Was renamed from `excludePathRegex` in 3.13.0.

### gib.includePathsMatching

Can be used to include only certain changed files from being detected as changed, reducing the number of modules to build. By default, everything is included.

This the opposite of [gib.excludePathsMatching](#gibexcludepathsmatching) which can be combined with this property, but `gib.excludePathsMatching` will take precedence.

See [gib.excludePathsMatching](#gibexcludepathsmatching) for general path matching rules.

:information_source: Use `[/\\]` instead of just `/` to also cover Windows path separators.

Since: 3.10.0

Was renamed from `includePathRegex` in 3.13.0.

### gib.buildAll

Builds all modules, including upstream modules (see also `gib.buildUpstream`). Can be used to (temporarily) override the reduction of modules to build.

Can be combined/useful with `gib.skipTestsForUpstreamModules` and/or `gib.argsForUpstreamModules`.

### gib.buildAllIfNoChanges

In case no changes are detected, GIB will (by default) just invoke goal `validate` in the "current project" (usually the root module), skipping any submodules.

If this property is enabled, GIB will instead auto-activate [gib.buildAll](#gibbuildall) for all modules and will leave the goals untouched.

This property is ignored when [explicitly selected projects](#explicitly-selected-projects) are involved.

Since: 3.9.2

### gib.buildDownstream

Controls whether or not to build downstream modules (= modules that depend on the modules GIB detected as changed):

- `always` or `true` (default value): always build downstream modules (depends on `-amd` when `-pl` is used)
- `derived`: only build downstream modules if `mvn -amd` is called
- `never` or `false`: never build downstream modules

Since: 3.8

### gib.buildUpstream

Controls whether or not to build upstream modules (= dependencies and parents of the modules GIB has determined to build):

- `always` or `true`: always build upstream modules (depends on `-am` when `-pl` is used)
- `derived` (default value): only build upstream modules if `mvn -am` is called
- `never` or `false`: never build upstream modules

See also `gib.buildUpstreamMode`.

Since: 3.8

### gib.buildUpstreamMode

This property controls which upstream modules to build (_if_ at all building upstream modules, see `gib.buildUpstream`):

- `changed` (default value): selects only upstream modules of the _directly changed_ modules
- `impacted`: like `changed` but also selects upstream modules of modules _that depend on_ the directly changed modules (in other words: upstream modules of the downstream modules of the changed modules)

`changed` is a subset of `impacted`.

`impacted` may seem odd at first, but it does come in handy in certain scenarios, e.g. a Jenkins PR job that locally merges target branch into the PR branch before building.
Here it might be required to freshly compile upstream modules of not directly changed modules to avoid compile errors or test failures which originate from the target branch.

Both strategies can and usually should be combined with `gib.skipTestsForUpstreamModules` and/or `gib.argsForUpstreamModules`.

This property is ignored when [explicitly selected projects](#explicitly-selected-projects) are involved.

Note: _Before_ 3.8, GIB did non have this property and was implicitly applying the `impacted` strategy, see also [issue 44](../../issues/44).

Since: 3.8

### gib.skipTestsForUpstreamModules

This property disables the compilation/execution of tests for upstream modules by adding `maven.test.skip=true`. In case an upstream module produces a test jar just the test _execution_ is disabled via `skipTests=true`.

See `gib.buildUpstream` or `gib.buildAll` to learn when upstream modules are built.

Can be combined with `gib.argsForUpstreamModules`.

### gib.argsForUpstreamModules

This property allows adding arbitrary arguments/properties for upstream modules to further reduce overhead, e.g. skip Checkstyle or Enforcer plugin.
Arguments have to be sparated with a single space character and values are optional. Example:

```
mvn clean install -am -Dgib.argsForUpstreamModules='enforcer.skip checkstyle.skip=true'
```
(notice the missing `-D`)

See `gib.buildUpstream` or `gib.buildAll` to learn when upstream modules are built.

Can be combined with `gib.skipTestsForUpstreamModules`.

### gib.argsForDownstreamModules

This property allows adding arbitrary arguments/properties for downstream modules to e.g. run them with a smaller testset than the directly changed modules. Only _unchanged_ downstream modules are affected by this setting.
Arguments have to be sparated with a single space character and values are optional. Example:

```
mvn clean install -am -Dgib.argsForDownstreamModules='testprofile=small'
```
(notice the missing `-D`)

See `gib.buildDownstream` or `gib.buildAll` to learn when upstream modules are built.

Since: 4.2.0

### gib.forceBuildModules

Defines artifact ids of modules to build forcibly, even if these modules have not been changed and/or do not depend on changed modules. Example:

```
mvn clean install -Dgib.forceBuildModules=unchanged-module-1,unchanged-module-2
```

Regular expressions are also supported in each comma separated part, e.g.:

```
mvn clean install -Dgib.forceBuildModules=unchanged-module-.*,another-module
```

Each of these modules is subject to `argsForUpstreamModules` and `skipTestsForUpstreamModules`.

This property has no effect in case `buildAll` is enabled.

Additionally (since 3.14.4), forced modules can be defined _conditionally_, e.g.:

```
changed-module=unchanged-module-1|unchanged-module-2
```

will build `unchanged-module-1` _and_ `unchanged-module-2` only if `changed-module` was changed (or depends on changed modules).

There can be multiple such key=value pairs, fully supporting regular expressions and you can mix both flavors, e.g.:

```
another-module, changed-module=unchanged-module-1|unchanged-module-2, .*-core-module=lib-.*-module, test-.*
```

Note: `,` and `=` are reserved delimiters and can therefore _not_ be used in artifact id strings/patterns.

### gib.excludeDownstreamModulesPackagedAs

Defines the packaging (e.g. `jar`) of modules that depend on changed modules but shall not be built.

One possible use case for this is mainly working in an IDE, fixing all compile errors etc. and then just quickly building the least possible amount of modules
which are needed to (hot-)deploy the changes via `mvn` on the command line.
In this scenario, by defining `-Dgib.excludeDownstreamModulesPackagedAs=jar,pom`, only the directly changed `jar` modules and the dependent `war` and/or `ear`
deployment modules will be built.

This property has no effect in case `buildAll` is enabled and an exclusion might be overridden by `gib.forceBuildModules`.

### gib.disableSelectedProjectsHandling

Disables special handling of [explicitly selected projects](#explicitly-selected-projects) (-pl, -f etc.).

This can come in handy if you select just one module with `-pl` but you only want to have it built _fully_ (with tests etc.)
if the selected module itself is changed or one of its (non-selected) upstream modules.

Since: 3.12.0

### gib.failOnMissingGitDir

Controls whether or not to fail on missing `.git` directory.

### gib.failOnError

Controls whether or not to fail on any error.

### gib.logImpactedTo

Defines an optional logfile which GIB shall write all "impacted" modules to. Each line represents the base directory of a changed module
or a downstream module of a changed module.

GIB overwrites the file if it already exists and will create an empty file in case no changes are detected.

Starting with 4.5.0, GIB does also write the actually built modules to this file:

- if only [explicitly selected projects](#explicitly-selected-projects) are present
- or if Maven is building non-recursively (`-N`)
- or if Maven is building only a single leaf module

Before 4.5.0 GIB only created an empty file in those cases.

Starting with 4.5.0, GIB will always remove the file first (unless disabled via any of the `disabled*` properties), so that cases like [skipIfPathMatches](#gibskipifpathmatches) don't leave behind an empty file which could be misinterpreted as "no changes are detected".

Since: 3.10.1

### gib.logProjectsMode

Controls which projects to log to the console. Default is `changed`, other options are `none`, `impacted` (changed + downstream) and `all` (including upstream).

Since: 4.5.0

## Explicitly selected projects

By default, GIB tries not to interfere with any projects/modules that have been selected explicitly by the user.

The details are described in the following subsections. This special handling can be disabled via [gib.disableSelectedProjectsHandling](#gibdisableselectedprojectshandling).

### mvn -pl

Since 3.10.0, special rules apply when `mvn -pl ...` (or `--projects ...`) is used:

- _Every_ such "preselected" project is _always_ built, including tests etc., **regardless of being changed or not!**

- Downstream projects of these selected projects are built if:
  - `-amd` (`--also-make-dependents`) is used
  - _and_:
    - [`buildDownstream`](#gibbuilddownstream) is _not_ `never` or `false`
    - _or_ [`buildAll`](#gibbuildall) is enabled

- Upstream projects of these selected projects are built if:
  - `-am` (`--also-make`) is used
  - _and_:
    - [`buildUpstream`](#gibbuildUpstream) is _not_ `never` or `false`
    - _or_ [`buildAll`](#gibbuildall) is enabled

Other properties/features are applied as usual to the resulting subset of modules/projects.

Since 3.10.1, "deselected" projects (`mvn -pl !...`) that contain changes will _not_ be built, but their up- and downstream projects will be built
(if not also deselected).

### mvn -f and others

Since 3.10.0, GIB will _always_ build a "leaf module" that is selected via `mvn -f ...` (or `--file ...`), **regardless of being changed or not!**

A "leaf module" is a module without submodules.

The same applies if the current directory is changed from the root directory of the multi-module-project to the leaf module subdirectory via `cd`.

In contrast, a module _with_ submodules that is selected via `-f` or via `cd` is subject to the regular change detection rules (unless [`-pl`](#mvn--pl) or [`-N`](#mvn--N) is added).

### mvn -N

Since 3.10.2, GIB will _always_ build a "node module" when non-recursive build is activated via `mvn -N` (or `--non-recursive`), **regardless of being changed or not!**

A "node module" is a module _with_ submodules.

## BOM support

Maven does not consider modules importing a BOM (bill of material) via `dependencyManagement` as downstream modules of that BOM module.

This effectively means that (without GIB) you **cannot** just change e.g. a dependency version in the BOM and run `mvn -pl bom -amd` as this will only build `bom`.

To close this "gap", GIB adds downstream detection for BOM modules so that your Dependabot PRs or similar can trigger a proper incremental builds.

## Test only changes

If a module contains changes only within `src/test`, GIB will **not** build any of its downstream modules _unless_ that module defines a
[test-jar goal and those downstream modules are depending on that jar](https://maven.apache.org/guides/mini/guide-attached-tests.html#guide-to-using-attached-tests).

## Authentication

When using `gib.fetchBaseBranch` or `gib.fetchReferenceBranch`, GIB provides basic support to authenticate against a possibly protected remote repository.

### HTTP

For HTTP(S), GIB will query the credentials from the local native Git executable via [`git credential fill`](https://git-scm.com/docs/git-credential).

These credentials are then forwarded to JGit and are not persisted in any way. GIB will only cache the credentials _transiently_ for a very short time and will actively remove them as soon as possible.

See also [HttpDelegatingCredentialsProvider](../main/src/main/java/io/github/gitflowincrementalbuilder/jgit/HttpDelegatingCredentialsProvider.java).

Since `git credential fill` will trigger all configured [credential helpers](https://git-scm.com/docs/gitcredentials) (if any), you _might_ see a popup dialog box asking for credentials.

This only happens in case the respective helper was _not_ able to provide the credentials. Such a dialog box is _not_ created by GIB, instead it is spawned by a configured credential helper!

As GIB does _not_ (yet) provide a console input passthrough mechanism to native Git, console input queries by native Git are disabled. This means that if _no_ credential helper is configured, GIB will _not_ be able to fetch via HTTP(S) from a remote repo that requires authentication.

### SSH

For SSH, GIB pretty much relies on [JGit SSH agent support via Apache MINA sshd](https://git.eclipse.org/r/plugins/gitiles/jgit/jgit/+/master/org.eclipse.jgit.ssh.apache/README.md#support-for-ssh-agents).

Your private key will be picked up automatically in case it is located in `~/.ssh/` (as `identity`, `id_rsa` or `id_dsa`).

To use a custom key, create a [SSH config](https://www.cyberciti.biz/faq/create-ssh-config-file-on-linux-unix/) in `~/.ssh/config` like the following:

```
Host git.somedomain.org
  IdentityFile ~/.ssh/my_key
```

If your key is protected by a **passphrase**, you will have to use a SSH agent (`ssh-agent` on Linux or `pageant` from `PuTTY` on Windows) and add your key(s) to it.

GIB relies on [SSH Agent Support as provided by JGit 6.0+](https://wiki.eclipse.org/JGit/New_and_Noteworthy/6.0#SSH_Agent_Support).

Hint: When using an agent, you don't need to put your key in a standard location, you don't need `~/.ssh/config` and your key is also _not required_ to be passphrase protected.


![Logo](https://github.com/openrewrite/rewrite/raw/main/doc/logo-oss.png)
### Eliminate Tech-Debt. At build time.

[![ci](https://github.com/openrewrite/rewrite-maven-plugin/actions/workflows/ci.yml/badge.svg)](https://github.com/openrewrite/rewrite-maven-plugin/actions/workflows/ci.yml)
[![Apache 2.0](https://img.shields.io/github/license/openrewrite/rewrite-maven-plugin.svg)](https://www.apache.org/licenses/LICENSE-2.0)
[![Maven Central](https://img.shields.io/maven-central/v/org.openrewrite.maven/rewrite-maven-plugin.svg)](https://mvnrepository.com/artifact/org.openrewrite.maven/rewrite-maven-plugin)
[![Contributing Guide](https://img.shields.io/badge/Contributing-Guide-informational)](https://github.com/openrewrite/.github/blob/main/CONTRIBUTING.md)

## What is this?

This project provides a Maven plugin that applies [Rewrite](https://github.com/openrewrite/rewrite) checking and fixing tasks as build tasks, one of several possible workflows for propagating change across an organization's source code.

## Getting started

This `README` may not have the most up-to-date documentation. For the most up-to-date documentation and reference guides, see:

- [Auto-generated maven plugin documentation](https://openrewrite.github.io/rewrite-maven-plugin/plugin-info.html)
- [Maven Plugin Configuration](https://docs.openrewrite.org/reference/rewrite-maven-plugin)
- [OpenRewrite Quickstart Guide](https://docs.openrewrite.org/running-recipes/getting-started)

To configure, add the plugin to your POM:

```xml
<?xml version="1.0" encoding="UTF-8"?>
<project>
    ...
    <build>
        <plugins>
            <plugin>
                <groupId>org.openrewrite.maven</groupId>
                <artifactId>rewrite-maven-plugin</artifactId>
                <version><!-- latest version here --></version>
                <configuration>
                    <activeRecipes>
                        <recipe>org.openrewrite.java.format.AutoFormat</recipe>
                    </activeRecipes>
                </configuration>
            </plugin>
        </plugins>
    </build>
</project>
```

If wanting to leverage recipes from other dependencies:

```xml
<?xml version="1.0" encoding="UTF-8"?>
<project>
    ...
    <build>
        <plugins>
            <plugin>
                <groupId>org.openrewrite.maven</groupId>
                <artifactId>rewrite-maven-plugin</artifactId>
                <version><!-- latest version here --></version>
                <configuration>
                    <activeRecipes>
                        <recipe>org.openrewrite.java.testing.junit5.JUnit5BestPractices</recipe>
                        <recipe>org.openrewrite.github.ActionsSetupJavaAdoptOpenJDKToTemurin</recipe>
                    </activeRecipes>
                </configuration>
                <dependencies>
                    <dependency>
                        <groupId>org.openrewrite.recipe</groupId>
                        <artifactId>rewrite-testing-frameworks</artifactId>
                        <version><!-- latest dependency version here --></version>
                    </dependency>
                    <dependency>
                        <groupId>org.openrewrite.recipe</groupId>
                        <artifactId>rewrite-github-actions</artifactId>
                        <version><!-- latest dependency version here --></version>
                    </dependency>
                </dependencies>
            </plugin>
        </plugins>
    </build>
</project>
```

To get started, try `mvn rewrite:help`, `mvn rewrite:discover`, `mvn rewrite:dryRun`, `mvn rewrite:run`, among other plugin goals.

See the [Maven Plugin Configuration](https://docs.openrewrite.org/reference/rewrite-maven-plugin) documentation for full configuration and usage options.

### Snapshots

To use the latest `-SNAPSHOT` version, add a `<pluginRepositories>` entry for OSSRH snapshots. For example:

```xml
<?xml version="1.0" encoding="UTF-8"?>
<project>
    ...
    <build>
        <plugins>
            <plugin>
                <groupId>org.openrewrite.maven</groupId>
                <artifactId>rewrite-maven-plugin</artifactId>
                <!-- Use whichever version is latest at the time of reading. This number is a placeholder. -->
                <version>4.17.0-SNAPSHOT</version>
                <configuration>
                    <activeRecipes>
                        <recipe>org.openrewrite.java.logging.slf4j.Log4j2ToSlf4j</recipe>
                    </activeRecipes>
                </configuration>
                <dependencies>
                    <dependency>
                        <groupId>org.openrewrite.recipe</groupId>
                        <artifactId>rewrite-testing-frameworks</artifactId>
                        <!-- Use whichever version is latest at the time of reading. This number is a placeholder. -->
                        <version>1.1.0-SNAPSHOT</version>
                    </dependency>
                </dependencies>
            </plugin>
        </plugins>
    </build>

    <pluginRepositories>
        <pluginRepository>
            <id>ossrh-snapshots</id>
            <url>https://oss.sonatype.org/content/repositories/snapshots</url>
        </pluginRepository>
    </pluginRepositories>

</project>
```

## Notes for developing and testing this plugin

This plugin uses the [`Maven Integration Testing Framework Extension`](https://github.com/khmarbaise/maven-it-extension) for tests.

All tests can be run from the command line using:

```sh
./mvnw verify
```

If you're looking for more information on the output from a test, try checking the `target/maven-it/**/*IT/**` directory contents after running the tests. It will contain the project state output, including maven logs, etc. Check the [`Integration Testing Framework Users Guide`](https://khmarbaise.github.io/maven-it-extension/itf-documentation/usersguide/usersguide.html) for information, too. It's good.

## Contributing

We appreciate all types of contributions. See the [contributing guide](https://github.com/openrewrite/.github/blob/main/CONTRIBUTING.md) for detailed instructions on how to get started.

### Resource guides

- https://blog.soebes.io/posts/2020/08/2020-08-17-itf-part-i/
- https://carlosvin.github.io/posts/creating-custom-maven-plugin/en/#_dependency_injection
- https://developer.okta.com/blog/2019/09/23/tutorial-build-a-maven-plugin
- https://medium.com/swlh/step-by-step-guide-to-developing-a-custom-maven-plugin-b6e3a0e09966


![Athenz](docs/images/athenz-logo.png)

# Athenz

[![Pipeline Status][status-image]][status-url]
[![Publish Status][publish-status-image]][status-url]
[![SourceSpy Dashboard](https://sourcespy.com/shield.svg)](https://sourcespy.com/github/athenzathenz/)
[![CII Best Practices](https://bestpractices.coreinfrastructure.org/projects/4681/badge)](https://bestpractices.coreinfrastructure.org/projects/4681)
[![Licenses](https://app.fossa.io/api/projects/git%2Bhttps%3A%2F%2Fgithub.com%2FAthenZ%2Fathenz.svg?type=shield)](https://app.fossa.io/projects/git%2Bhttps%3A%2F%2Fgithub.com%2FAthenZ%2Fathenz?ref=badge_shield)

[status-image]: https://cd.screwdriver.cd/pipelines/6606/badge
[publish-status-image]: https://cd.screwdriver.cd/pipelines/6606/publish/badge
[status-url]: https://cd.screwdriver.cd/pipelines/6606

> Athenz is an open source platform for X.509 certificate based service authentication and fine-grained
> access control in dynamic infrastructures. It supports provisioning and configuration (centralized
> authorization) use cases as well as serving/runtime (decentralized authorization) use cases. Athenz
> authorization system utilizes x.509 certificates and industry standard mutual TLS bound oauth2 access
> tokens. The name “Athenz” is derived from “AuthNZ” (N for authentication and Z for authorization).

## Table of Contents

* [Background](#background)
* [Install](#install)
* [Usage](#usage)
* [Contribute](#contribute)
* [License](#license)

## Background

Athenz is an open source platform for X.509 certificate based service authentication
and fine-grained role based access control in dynamic infrastructures. It provides
support for the following three major functional areas.

### Service Authentication

Athenz provides secure identity in the form of short lived X.509 certificate
for every workload or service deployed in private (e.g. Openstack, K8S, Screwdriver)
or public cloud (e.g. AWS EC2, ECS, Fargate, Lambda). Using these X.509 certificates
clients and services establish secure connections and through mutual TLS authentication verify
each other's identity. The service identity certificates are valid for 30 days only,
and the service identity agents (SIA) part of those frameworks automatically refresh
them daily. The term service within Athenz is more generic than a traditional service.
A service identity could represent a command, job, daemon, workflow, as well as both an
application client, and an application service.

Since Athenz service authentication is based on
[X.509 certificates](https://en.wikipedia.org/wiki/X.509), it is
important that you have a good understanding of what X.509 certificates are
and how they're used to establish secure connections in Internet protocols
such as [TLS](https://en.wikipedia.org/wiki/Transport_Layer_Security).

### Role-Based Authorization (RBAC)

Once the client is authenticated with its x.509 certificate, the service
can then check if the given client is authorized to carry out the requested
action. Athenz provides fine-grained role-based access control (RBAC) support
for a centralized management system with support for control-plane access control
decisions and a decentralized enforcement mechanism suitable for data-plane
access control decisions. It also provides a delegated management model that
supports multi-tenant and self-service concepts.

### AWS Temporary Credentials Support

When working with AWS, Athenz provides support to access AWS services
from on-prem services with using AWS temporary credentials rather than
static credentials. Athenz ZTS server can be used to request AWS temporary
credentials for configured AWS IAM roles.

## Install

* [Development Environment](docs/dev_environment.md)
* Local/Development/Production Environment Setup
    * [ZMS Server](docs/setup_zms.md)
    * [ZTS Server](docs/setup_zts.md)
    * [UI Server](docs/setup_ui.md)
* AWS Production Environment Setup
    * [Introduction](docs/aws_athenz_setup.md)

## Usage

* Architecture
    * [Data Model](docs/data_model.md)
    * [System View](docs/system_view.md)
    * [Authorization Flow](docs/auth_flow.md)
* Features
    * [Service Identity X.509 Certificates - Copper Argos](docs/copper_argos.md)
* Developer Guide
    * [Centralized Access Control](docs/cent_authz_flow.md)
        * [Java Client/Servlet Example](docs/example_java_centralized_access.md)
        * [Go Client/Server Example](docs/example_go_centralized_access.md)
    * [Decentralized Access Control](docs/decent_authz_flow.md)
        * [Java Client/Servlet Example](docs/example_java_decentralized_access.md)
* Customizing Athenz
    * [Principal Authentication](docs/principal_authentication.md)
    * [Private Key Store](docs/private_key_store.md)
    * [Certificate Signer](docs/cert_signer.md)
    * [Service Identity X.509 Certificate Support Requirements - Copper Argos](docs/copper_argos_dev.md)
    * [OIDC Authentication Provider Support for AWS EKS](docs/oidc_aws_eks.md)
* User Guide
    * [ZMS Client Utility](docs/zms_client.md)
    * [ZPU Utility](docs/setup_zpu.md)
    * [Registering ZMS Service Identity](docs/reg_service_guide.md)
    * [ZMS API](docs/zms_api.md)
    * [ZTS API](docs/zts_api.md)

## Contribute

Please refer to the [contributing file](CONTRIBUTING.md) for information about how to get involved. We welcome issues, questions, and pull requests.

You can also contact us for any user and development discussions through our groups:

* [Athenz-Dev](https://groups.google.com/d/forum/athenz-dev) for development discussions
* [Athenz-Users](https://groups.google.com/d/forum/athenz-users) for users questions

The [sourcespy dashboard](https://sourcespy.com/github/yahooathenz/) provides a high level overview of the repository including [module dependencies](https://sourcespy.com/github/yahooathenz/xx-omodulesc-.html), [module hierarchy](https://sourcespy.com/github/yahooathenz/xx-omodules-.html), [external libraries](https://sourcespy.com/github/yahooathenz/xx-ojavalibs-.html), [web services](https://sourcespy.com/github/yahooathenz/xx-owebservices-.html), and other components of the system.

## License

Licensed under the Apache License, Version 2.0: [http://www.apache.org/licenses/LICENSE-2.0](http://www.apache.org/licenses/LICENSE-2.0)


## Badges

[![CircleCI](https://circleci.com/gh/OpenLineage/OpenLineage/tree/main.svg?style=shield)](https://circleci.com/gh/OpenLineage/OpenLineage/tree/main)
[![status](https://img.shields.io/badge/status-active-brightgreen.svg)](#status)
[![Slack](https://img.shields.io/badge/slack-chat-blue.svg)](http://bit.ly/OpenLineageSlack)
[![license](https://img.shields.io/badge/license-Apache_2.0-blue.svg)](https://github.com/OpenLineage/OpenLineage/blob/main/LICENSE)
[![maven](https://img.shields.io/maven-central/v/io.openlineage/openlineage-java.svg)](https://search.maven.org/search?q=g:io.openlineage)
[![CII Best Practices](https://bestpractices.coreinfrastructure.org/projects/4888/badge)](https://bestpractices.coreinfrastructure.org/projects/4888)

## Overview
OpenLineage is an Open standard for metadata and lineage collection designed to instrument jobs as they are running.
It defines a generic model of run, job, and dataset entities identified using consistent naming strategies.
The core lineage model is extensible by defining specific facets to enrich those entities.

## Status

OpenLineage is an [LF AI & Data Foundation](https://lfaidata.foundation/projects/openlineage) incubation project under active development, and we'd love your help!

## Problem

### Before

- Duplication of effort: each project has to instrument all jobs
- Integrations are external and can break with new versions

![Before OpenLineage](doc/before-ol.svg)

### With OpenLineage

- Effort of integration is shared
- Integration can be pushed in each project: no need to play catch up

![With OpenLineage](doc/with-ol.svg)

## Scope
OpenLineage defines the metadata for running jobs and the corresponding events.
A configurable backend allows the user to choose what protocol to send the events to.
 ![Scope](doc/scope.svg)

## Core model

 ![Model](doc/datamodel.svg)

 A facet is an atomic piece of metadata attached to one of the core entities.
 See the spec for more details.

## Spec
The [specification](spec/OpenLineage.md) is defined using OpenAPI and allows extension through custom facets.

## Integration matrix

The OpenLineage repository contains integrations with several systems.

| Name| Table-level lineage| Column-level lineage |
| ----| ------------------ | -------------------- |
|[Apache Spark](https://github.com/OpenLineage/OpenLineage/tree/main/integration/spark)| :white_check_mark: | :white_check_mark:1 |
|[Apache Airflow](https://github.com/OpenLineage/OpenLineage/tree/main/integration/airflow)| :white_check_mark: | :white_check_mark:2 |
|[Dagster](https://github.com/OpenLineage/OpenLineage/tree/main/integration/dagster)| :white_check_mark: | :x: |
|[dbt](https://github.com/OpenLineage/OpenLineage/tree/main/integration/dbt) |:white_check_mark: | :x: |
|[Flink](https://github.com/OpenLineage/OpenLineage/tree/main/integration/flink)|:white_check_mark: | :x: |

1. Does not support `SELECT *` queries with JDBC.
2. Supports SQL-based operators other than BigQuery.

## Related projects
- [Marquez](https://marquezproject.ai/): Marquez is an [LF AI & DATA](https://lfaidata.foundation/) project to collect, aggregate, and visualize a data ecosystem's metadata. It is the reference implementation of the OpenLineage API.
  - [OpenLineage collection implementation](https://github.com/MarquezProject/marquez/blob/main/api/src/main/java/marquez/api/OpenLineageResource.java)
- [Egeria](https://egeria.odpi.org/): Egeria offers open metadata and governance for enterprises - automatically capturing, managing and exchanging metadata between tools and platforms, no matter the vendor.

## Community
- Website: [openlineage.io](http://openlineage.io)
- Slack: [OpenLineage.slack.com](http://bit.ly/OpenLineageSlack)
  - Not a member? Join [here](https://bit.ly/lineageslack).
- Twitter: [@OpenLineage](https://twitter.com/OpenLineage)
- Mailing list: [openlineage-tsc](https://lists.lfaidata.foundation/g/openlineage-tsc)
- Wiki: [OpenLineage+Home](https://wiki.lfaidata.foundation/display/OpenLineage/OpenLineage+Home)
- LinkedIn: [13927795](https://www.linkedin.com/groups/13927795/)
- YouTube: [channel](https://www.youtube.com/channel/UCRMLy4AaSw_ka-gNV9nl7VQ)
- Mastodon: [@openlineage@fostodon.org](openlineage@fosstodon.org)

## Talks
- [Data+AI Summit June 2023. Cross-Platform Data Lineage with OpenLineage](https://www.databricks.com/dataaisummit/session/cross-platform-data-lineage-openlineage/)
- [Berlin Buzzwords June 2023. Column-Level Lineage is Coming to the Rescue](https://youtu.be/xFVSZCCbZlY)
- [Berlin Buzzwords June 2022. Cross-Platform Data Lineage with OpenLineage](https://www.youtube.com/watch?v=pLBVGIPuwEo)
- [Berlin Buzzwords June 2021. Observability for Data Pipelines with OpenLineage](https://2021.berlinbuzzwords.de/member/julien-le-dem)
- [Data Driven NYC February 2021. Data Observability and Pipelines: OpenLineage and Marquez](https://mattturck.com/datakin/)
- [Big Data Technology Warsaw Summit February 2021. Data lineage and Observability with Marquez and OpenLineage](https://bigdatatechwarsaw.eu/edition-2021/)
- [Metadata Day 2020. OpenLineage Lightning Talk](https://www.youtube.com/watch?v=anlV5Er_BpM)
- [Open Core Summit 2020. Observability for Data Pipelines: OpenLineage Project Launch](https://www.coss.community/coss/ocs-2020-breakout-julien-le-dem-3eh4)

## Contributing

See [CONTRIBUTING.md](https://github.com/OpenLineage/OpenLineage/blob/main/CONTRIBUTING.md) for more details about how to contribute.

## Report a Vulnerability

If you discover a vulnerability in the project, please [open an issue](https://github.com/OpenLineage/OpenLineage/issues/new/choose) and attach the "security" label.

----
SPDX-License-Identifier: Apache-2.0\
Copyright 2018-2024 contributors to the OpenLineage project