Sleeper
=======

## Introduction

Sleeper is a serverless, cloud-native, log-structured merge tree based, scalable key-value store. It is designed to
allow the ingest of very large volumes of data at low cost. Data is stored in rows in tables. Each row has a key field,
and an optional sort field, and some value fields. Queries for rows where the key takes a given value takes around
1-2 seconds, but many thousands can be run in parallel. Each individual query has a negligible cost.

Sleeper can be thought of as a cloud-native reimagining of systems such as Hbase and Accumulo. The architecture is
very different to those systems. Sleeper has no long running servers. This means that if there is no work to be done,
i.e. no data is being ingested and no background operations such as compactions are in progress, then the only cost
is the cost of the storage. There are no wasted compute cycles, i.e. it is "serverless".

The current codebase can only be deployed to AWS, but there is nothing in the design that limits it to AWS. In time
we would like to be able to deploy Sleeper to other public cloud environments such as Microsoft Azure
or to a Kubernetes cluster.

Note that Sleeper is currently a prototype. Further development and testing is needed before it can be considered
to be ready for production use.

## Functionality

Sleeper stores records in tables. A table is a collection of records that conform to a schema. A record is a map
from a field name to value. For example, a schema might have a row key field called 'id' of type string, a sort
field called 'timestamp' of type long, and a value field called 'name' of type string. Each record in a table
with that schema is a map with keys of id, timestamp and name. Data in the table is stored range-partitioned by
the key field. Within partitions, records are stored in Parquet files in S3. These files contain records in sorted
order (sorted by the key field and then by the sort field).

Sleeper is deployed using CDK. Each bit of functionality is deployed using a separate CDK substack of one main
stack.

- State store stacks: Each table has a state store that stores metadata about the table such as files
  that are in the table, the partitions they are in and information about the partitions themselves.
- Compaction stack: As the number of files in a partition increases, their contents must be merged
  ("compacted") into a single sorted file. The compaction stack performs this task using lambda and SQS for job
  creation, queueing and Fargate/EC2 for execution of the tasks.
- Garbage collector stack: After compaction jobs have completed, the input files are deleted (after a user
  configurable delay to ensure the files are not in use by queries). The garbage collector stack performs this
  task using a lambda function.
- Partition splitting stack: Partitions need to be split once they reach a user configurable size. This only
  affects the metadata about partitions in the state store. This task is performed using a lambda.
- Ingest stack: This allows the ingestion of data from Parquet files. These files need to consist of data that
  matches the schema of the table. They do not need to be sorted or partitioned in any way. Ingest job requests
  are submitted to an SQS queue. They are then executed by tasks running on an ECS cluster. These tasks are scaled
  up using a lambda that periodically monitors the number of items on the queue. The tasks scale down naturally
  as if there are no jobs on the queue a task will terminate.
- Query stack: This stack allows queries to be executed in lambdas. Queries are submitted to an SQS queue. This
  triggers a lambda that executes the queries. It can then write the results to files in an S3 bucket, or send the
  results to an SQS queue. A query can also be executed from a websocket client. In this case the results will be
  returned directly to a client. Note that queries can also be submitted directly from a Java client, and this does
  not require the query stack.
- Topic stack: Notifications of errors and failures will appear on an SNS topic.
- EMR bulk import stack: This allows large volumes of data in Parquet files to be ingested. A bulk import job
  request is sent to an SQS queue. This triggers a lambda that creates an EMR cluster. This cluster runs a Spark
  job to import the data. Once the job is finished the EMR cluster shuts down.
- Persistent EMR bulk import stack: Similar to the above stack, but the EMR cluster is persistent, i.e. it never
  shuts down. This is appropriate if there is a steady stream of import jobs. The cluster can either be of fixed
  size or it can use EMR managed scaling.
- Dashboard stack: This displays properties of the system in a Cloudwatch dashboard.

The following functionality is experimental:

- Athena: There is experimental integration with Athena to allow SQL analytics to be run against a table.
- Trino: There is a plugin for Trino that allows SQL select, join and insert statements to be run against multiple
  Sleeper tables. It returns results quickly and it is particularly suitable for interactive queries which retrieve just
  a few rows from the underlying Sleeper instance, such as SELECT * FROM t1 INNER JOIN t2 USING (keycol) WHERE t1.name =
  "Fred Jones"
- Bulk import using Spark running on EKS: This stack allows data to be ingested by running Spark on EKS. Currently
  this runs the Spark executors in either Fargate or EC2.

Sleeper provides the tools to implement fine-grained security on the data, although further work is needed to make
these easier to use. Briefly, the following steps are required:

- Decide how to store the security information in the table, e.g. there might be one security label per row,
  or two per row, or one per cell. These fields must be added to the schema.
- Write an iterator that will run on the results of every query to filter out records that a user is not permitted
  to see. This takes as input a user's authorisations and uses those to make a decision as to whether the user can see
  the data.
- Ensure the Sleeper instance is deployed such that the boundary of the system is protected.
- Ensure that queries are submitted to the query queue via a service that authenticates users, and passes their
  authorisations into the query time iterator configuration.

## License

Sleeper is licensed under the [Apache 2](http://www.apache.org/licenses/LICENSE-2.0) license.

## Documentation

See the documentation contained in the docs folder:

1. [Getting started](docs/01-getting-started.md)
2. [Deployment guide](docs/02-deployment-guide.md)
3. [Creating a schema](docs/03-schema.md)
4. [Tables](docs/04-tables.md)
5. [Ingesting data](docs/05-ingest.md)
6. [Checking the status of the system](docs/06-status.md)
7. [Retrieving data](docs/07-data-retrieval.md)
8. [Python API](docs/08-python-api.md)
9. [Trino](docs/09-trino.md)
10. [Deploying to LocalStack](docs/10-deploy-to-localstack.md)
11. [Developer guide](docs/11-dev-guide.md)
12. [Design](docs/12-design.md)
13. [System tests](docs/13-system-tests.md)
14. [Release process](docs/14-release-process.md)
15. [Common problems and their solutions](docs/15-common-problems-and-their-solutions.md)
16. [Roadmap](docs/16-roadmap.md)


<!--

    Licensed to the Apache Software Foundation (ASF) under one
    or more contributor license agreements.  See the NOTICE file
    distributed with this work for additional information
    regarding copyright ownership.  The ASF licenses this file
    to you under the Apache License, Version 2.0 (the
    "License"); you may not use this file except in compliance
    with the License.  You may obtain a copy of the License at

      http://www.apache.org/licenses/LICENSE-2.0

    Unless required by applicable law or agreed to in writing,
    software distributed under the License is distributed on an
    "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
    KIND, either express or implied.  See the License for the
    specific language governing permissions and limitations
    under the License.

-->

# [A one-stop, full-scenario integration framework for massive data](https://inlong.apache.org/)
[![GitHub Actions](https://github.com/apache/inlong/actions/workflows/ci_build.yml/badge.svg)](https://github.com/apache/inlong/actions)
[![CodeCov](https://codecov.io/gh/apache/inlong/branch/master/graph/badge.svg)](https://codecov.io/gh/apache/inlong)
[![Maven Central](https://maven-badges.herokuapp.com/maven-central/org.apache.inlong/inlong/badge.svg)](http://search.maven.org/#search%7Cga%7C1%7Corg.apache.inlong)
[![GitHub release](https://img.shields.io/badge/release-download-orange.svg)](https://inlong.apache.org/download/)
[![License](https://img.shields.io/badge/license-Apache%202-4EB1BA.svg)](https://www.apache.org/licenses/LICENSE-2.0.html)
[![Twitter](https://img.shields.io/twitter/url?label=Follow%20ApacheInlong&style=social&url=https%3A%2F%2Ftwitter.com%2FApacheInlong)](https://twitter.com/ApacheInlong)
[![Slack](https://img.shields.io/badge/%20Slack-InLong%20Channel-blueviolet)](https://the-asf.slack.com/archives/C01QAG6U00L)

- [What is Apache InLong?](#what-is-apache-inlong)
- [Features](#features)
- [When should I use InLong?](#when-should-i-use-inlong)
- [Build InLong](#build-inlong)
- [Deploy InLong](#deploy-inlong)
- [Contribute to InLong](#contribute-to-inlong)
- [Contact Us](#contact-us)
- [Documentation](#documentation)
- [License](#license)

# What is Apache InLong?
|                                       **Stargazers Over Time**                                        |                                                                                **Contributors Over Time**                                                                                |
|:-----------------------------------------------------------------------------------------------------:|:------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------:|
| [![Stargazers over time](https://starchart.cc/apache/inlong.svg)](https://starchart.cc/apache/inlong) | [![Contributor Over Time](https://contributor-overtime-api.git-contributor.com/contributors-svg?chart=contributorOverTime&repo=apache/inlong)](https://git-contributor.com?chart=contributorOverTime&repo=apache/inlong) |

[Apache InLong](https://inlong.apache.org) is a one-stop, full-scenario integration framework for massive data that supports `Data Ingestion`, `Data Synchronization` and `Data Subscription`, and it provides automatic, secure and reliable data transmission capabilities. InLong also supports both batch and stream data processing at the same time, which offers great power to build data analysis, modeling and other real-time  applications based on streaming data.

InLong (应龙) is a divine beast in Chinese mythology who guides the river into the sea, and it is regarded as a metaphor of the InLong system for reporting data streams.

InLong was originally built at Tencent, which has served online businesses for more than 8 years, to support massive data (data scale of more than 80 trillion pieces of data per day) reporting services in big data scenarios. The entire platform has integrated 5 modules:  Ingestion, Convergence, Caching, Sorting, and Management, so that the business only needs to provide data sources, data service quality, data landing clusters and data landing formats, that is, the data can be continuously pushed from the source to the target cluster, which greatly meets the data reporting service requirements in the business big data scenario.

For getting more information, please visit our project documentation at https://inlong.apache.org/.
![inlong-structure-en.png](https://github.com/apache/inlong-website/blob/master/static/img/inlong-structure-en.png)

## Features
Apache InLong offers a variety of features:
* **Ease of Use**: a SaaS-based service platform. Users can easily and quickly report, transfer, and distribute data by publishing and subscribing to data based on topics.
* **Stability & Reliability**: derived from the actual online production environment. It delivers high-performance processing capabilities for 10 trillion-level data streams and highly reliable services for 100 billion-level data streams.
* **Comprehensive Features**: supports various types of data access methods and can be integrated with different types of Message Queue (MQ). It also provides real-time data extract, transform, and load (ETL) and sorting capabilities based on rules. InLong also allows users to plug features to extend system capabilities.
* **Service Integration**: provides unified system monitoring and alert services. It provides fine-grained metrics to facilitate data visualization. Users can view the running status of queues and topic-based data statistics in a unified data metric platform. Users can also configure the alert service based on their business requirements so that users can be alerted when errors occur.
* **Scalability**: adopts a pluggable architecture that allows you to plug modules into the system based on specific protocols. Users can replace components and add features based on their business requirements.

## When should I use InLong?
InLong aims to provide a one-stop, full-scenario integration framework for massive data, users can easily build stream-based data applications. It supports `Data Ingestion`, `Data Synchronization` and `Data Subscription` at the same time, and is suitable for environments that need to quickly build a data reporting platform, as well as an ultra-large-scale data reporting environment that InLong is very suitable for, and an environment that needs to automatically sort and land the reported data.

You can use InLong in the following ways：
- Integrate InLong, manage data streams through [SDK](https://inlong.apache.org/docs/next/sdk/manager-sdk/example).
- Use [the InLong command-line tool](https://inlong.apache.org/docs/next/user_guide/command_line_tools) to view and create data streams.
- Visualize your operations on [InLong dashboard](https://inlong.apache.org/docs/next/user_guide/dashboard_usage).

## Supported Data Nodes (Updating)
| Type         | Name              | Version                      |
|--------------|-------------------|------------------------------|
| Extract Node | Auto Push         | None                         |
|              | File              | None                         |
|              | Kafka             | 2.x                          |
|              | MongoDB           | >= 3.6                       |
|              | MQTT              | >= 3.1                       |
|              | MySQL             | 5.6, 5.7, 8.0.x              |
|              | Oracle            | 11,12,19                     |
|              | PostgreSQL        | 9.6, 10, 11, 12              |
|              | Pulsar            | 2.8.x                        |
|              | Redis             | 2.6.x                        |
|              | SQLServer         | 2012, 2014, 2016, 2017, 2019 |
| Load Node    | Auto Consumption  | None                         |
|              | ClickHouse        | 20.7+                        |
|              | Elasticsearch     | 6.x, 7.x                     |
|              | Greenplum         | 4.x, 5.x, 6.x                |
|              | HBase             | 2.2.x                        |
|              | HDFS              | 2.x, 3.x                     |
|              | Hive              | 1.x, 2.x, 3.x                |
|              | Iceberg           | 0.12.x                       |
|              | Hudi              | 0.12.x                       |
|              | Kafka             | 2.x                          |
|              | MySQL             | 5.6, 5.7, 8.0.x              |
|              | Oracle            | 11, 12, 19                   |
|              | PostgreSQL        | 9.6, 10, 11, 12              |
|              | SQLServer         | 2012, 2014, 2016, 2017, 2019 |
|              | TDSQL-PostgreSQL  | 10.17                        |
|              | Doris             | >= 0.13                      |
|              | StarRocks         | >= 2.0                       |
|              | Kudu              | >= 1.12.0                    |
|              | Redis             | >= 3.0                       |

## Build InLong
More detailed instructions can be found at [Quick Start](https://inlong.apache.org/docs/next/quick_start/how_to_build) section in the documentation.

Requirements:
- Java [JDK 8](https://adoptopenjdk.net/?variant=openjdk8)
- Maven 3.6.1+
- [Docker](https://docs.docker.com/engine/install/) 19.03.1+

CodeStyle:

```
mvn spotless:apply
```

Compile and install:
```
mvn clean install -DskipTests
```
(Optional) Compile using docker image:
```
docker pull maven:3.6-openjdk-8
docker run -v `pwd`:/inlong  -w /inlong maven:3.6-openjdk-8 mvn clean install -DskipTests
```
after compile successfully, you could find distribution file at `inlong-distribution/target`.

## Deploy InLong
- [Standalone for InLong](https://inlong.apache.org/docs/next/deployment/standalone)
- [Docker Compose](https://inlong.apache.org/docs/next/deployment/docker)
- [InLong on Kubernetes](https://inlong.apache.org/docs/next/deployment/k8s)
- [Bare Metal](https://inlong.apache.org/docs/next/deployment/bare_metal)

## Develop InLong
- [Agent Plugin extends a Extract Data Node](https://inlong.apache.org/docs/next/design_and_concept/how_to_write_plugin_agent)
- [Sort Plugin extends a Data Node](https://inlong.apache.org/docs/next/design_and_concept/how_to_extend_data_node_for_sort)
- [Manager Plugin extends a Data Node](https://inlong.apache.org/docs/next/design_and_concept/how_to_extend_data_node_for_manager)
- [Dashboard Plugin extends a Data Node page](https://inlong.apache.org/docs/next/design_and_concept/how_to_write_plugin_dashboard)

## Contribute to InLong
- Report any issue on [GitHub Issue](https://github.com/apache/inlong/issues)
- Code pull request according to [How to contribute](https://inlong.apache.org/community/how-to-contribute).

## Contact Us
- Join Apache InLong mailing lists:
    | Name                                                                          | Scope                           |                                                                 |                                                                     |                                                                              |
    |:------------------------------------------------------------------------------|:--------------------------------|:----------------------------------------------------------------|:--------------------------------------------------------------------|:-----------------------------------------------------------------------------|
    | [dev@inlong.apache.org](mailto:dev@inlong.apache.org)     | Development-related discussions | [Subscribe](mailto:dev-subscribe@inlong.apache.org)   | [Unsubscribe](mailto:dev-unsubscribe@inlong.apache.org)   | [Archives](http://mail-archives.apache.org/mod_mbox/inlong-dev/)   |
- Ask questions on [Apache InLong Slack](https://the-asf.slack.com/archives/C01QAG6U00L)

## Documentation
- Home page: https://inlong.apache.org/
- Issues: https://github.com/apache/inlong/issues

## License
© Contributors Licensed under an [Apache-2.0](LICENSE) license.


[![License](https://img.shields.io/badge/License-Apache%202.0-blue.svg)](https://opensource.org/licenses/Apache-2.0)
# Welcome to primihub-platform
primihub-platform is a Muti-Party Computation and Muti-Party federated task security scheduling platform for mpc and fl point to point service.

## Features
Providing production-level service capabilities:
- Data access
- Multi-Party Resource Fusion
- Task Scheduling
- Multi-Party Federated Model Registry
- Multi-Party Cooperation Authority Management

and have a clear directory:

    ├─primihub-platform
        ├─primihub-service
        │   ├─application
        │   ├─biz
        │   ├─gateway
        │   └─script
        └─primihub-webconsole

## Getting Started
Before we start, please refer to [primihub](https://github.com/primihub/primihub) and start node.

then we can get started from those projects.

- [primihub-meta](https://github.com/primihub/primihub-meta) : connect primihub-service and have the right data access.
- [primihub-service](./primihub-service/README.md) : provide all most of service capabilities and api.
- [primihub-webconsole](./primihub-webconsole/README.md) : you can operate specific functions and have a clear view.

## License
[Apache License 2.0](./LICENSE)

## Contact Us

It's pleasure to offer a primihub assistant to a contract list by scanning the QR code. You can get support on technique,business and the chance to community with us util the assistant invite you to the open source community group.

![assitant](./assitant.png)

And also welcome to follow our official account and slack home page([primihub slack](https://primihub.slack.com/join/shared_invite/zt-1af0l22ar-jmTI2C_DPUd3QSuPuOsYdA#/shared-invite/email)).

![offical](./offical.JPEG)


**严肃声明：现在、未来都不会有商业版本，所有代码全部开源！**

**「我喜欢写代码，乐此不疲」**
**「我喜欢做开源，以此为乐」**

我 🐶 在上海艰苦奋斗，早中晚在 top3 大厂认真搬砖，夜里为开源做贡献。

如果这个项目让你有所收获，记得 Star 关注哦，这对我是非常不错的鼓励与支持。

## 🐰 版本说明

| 版本                                                                    | JDK 8 + Spring Boot 2.7                                                  | JDK 17/21 + Spring Boot 3.2                                                          |
|-----------------------------------------------------------------------|--------------------------------------------------------------------------|--------------------------------------------------------------------------------------|
| 【完整版】[yudao-cloud](https://gitee.com/zhijiantianya/yudao-cloud)       | [`master`](https://gitee.com/zhijiantianya/yudao-cloud/tree/master/) 分支  | [`master-jdk17`](https://gitee.com/zhijiantianya/yudao-cloud/tree/master-jdk17/) 分支  |
| 【精简版】[yudao-cloud-mini](https://gitee.com/yudaocode/yudao-cloud-mini) | [`master`](https://gitee.com/yudaocode/yudao-cloud-mini/tree/master/) 分支 | [`master-jdk17`](https://gitee.com/yudaocode/yudao-cloud-mini/tree/master-jdk17/) 分支 |

* 【完整版】：包括系统功能、基础设施、会员中心、数据报表、工作流程、商城系统、微信公众号、CRM、ERP 等功能
* 【精简版】：只包括系统功能、基础设施功能，不包括会员中心、数据报表、工作流程、商城系统、微信公众号、CRM、ERP 等功能

可参考 [《迁移文档》](https://cloud.iocoder.cn/migrate-module/) ，只需要 5-10 分钟，即可将【完整版】按需迁移到【精简版】

## 🐶 新手必读

* 演示地址【Vue3 + element-plus】：<http://dashboard-vue3.yudao.iocoder.cn>
* 演示地址【Vue3 + vben(ant-design-vue)】：<http://dashboard-vben.yudao.iocoder.cn>
* 演示地址【Vue2 + element-ui】：<http://dashboard.yudao.iocoder.cn>
* 启动文档：<https://cloud.iocoder.cn/quick-start/>
* 视频教程：<https://cloud.iocoder.cn/video/>

## 🐯 平台简介

**芋道**，以开发者为中心，打造中国第一流的快速开发平台，全部开源，个人与企业可 100% 免费使用。

> 有任何问题，或者想要的功能，可以在 _Issues_ 中提给艿艿。
>
> 😜 给项目点点 Star 吧，这对我们真的很重要！

![架构图](/.image/common/yudao-cloud-architecture.png)

* Java 后端：`master` 分支为 JDK 8 + Spring Boot 2.7，`master-jdk17` 分支为 JDK 17/21 + Spring Boot 3.2
* 管理后台的电脑端：Vue3 提供 [element-plus](https://gitee.com/yudaocode/yudao-ui-admin-vue3)、[vben(ant-design-vue)](https://gitee.com/yudaocode/yudao-ui-admin-vben) 两个版本，Vue2 提供 [element-ui](https://gitee.com/zhijiantianya/ruoyi-vue-pro/tree/master/yudao-ui-admin) 版本
* 管理后台的移动端：采用 [uni-app](https://github.com/dcloudio/uni-app) 方案，一份代码多终端适配，同时支持 APP、小程序、H5！
* 后端采用 Spring Cloud Alibaba 微服务架构，注册中心 + 配置中心 Nacos，定时任务 XXL-Job，服务保障 Sentinel，服务网关 Gateway，分布式事务 Seata
* 数据库可使用 MySQL、Oracle、PostgreSQL、SQL Server、MariaDB、国产达梦 DM、TiDB 等，基于 MyBatis Plus、Redis + Redisson 操作
* 消息队列可使用 Event、Redis、RabbitMQ、Kafka、RocketMQ 等
* 权限认证使用 Spring Security & Token & Redis，支持多终端、多种用户的认证系统，支持 SSO 单点登录
* 支持加载动态权限菜单，按钮级别权限控制，Redis 缓存提升性能
* 支持 SaaS 多租户系统，可自定义每个租户的权限，提供透明化的多租户底层封装
* 高效率开发，使用代码生成器可以一键生成 Java、Vue 前后端代码、SQL 脚本、接口文档，支持单表、树表、主子表
* 实时通信，采用 Spring WebSocket 实现，内置 Token 身份校验，支持 WebSocket 集群
* 集成微信小程序、微信公众号、企业微信、钉钉等三方登陆，集成支付宝、微信等支付与退款
* 集成阿里云、腾讯云等短信渠道，集成 MinIO、阿里云、腾讯云、七牛云等云存储服务
* 集成报表设计器、大屏设计器，通过拖拽即可生成酷炫的报表与大屏

##  🐳 项目关系

![架构演进](/.image/common/yudao-roadmap.png)

三个项目的功能对比，可见社区共同整理的 [国产开源项目对比](https://www.yuque.com/xiatian-bsgny/lm0ec1/wqf8mn) 表格。

### 后端项目

| 项目                                                              | Star                                                                                | 简介                          |
|-----------------------------------------------------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|-----------------------------|
| [ruoyi-vue-pro](https://gitee.com/zhijiantianya/ruoyi-vue-pro)  | [![Gitee star](https://gitee.com/zhijiantianya/ruoyi-vue-pro/badge/star.svg?theme=white)](https://gitee.com/zhijiantianya/ruoyi-vue-pro) [![GitHub stars](https://img.shields.io/github/stars/YunaiV/ruoyi-vue-pro.svg?style=social&label=Stars)](https://github.com/YunaiV/ruoyi-vue-pro)       | 基于 Spring Boot 多模块架构        |
| [yudao-cloud](https://gitee.com/zhijiantianya/yudao-cloud)      | [![Gitee star](https://gitee.com/zhijiantianya/yudao-cloud/badge/star.svg?theme=white)](https://gitee.com/zhijiantianya/yudao-cloud) [![GitHub stars](https://img.shields.io/github/stars/YunaiV/yudao-cloud.svg?style=social&label=Stars)](https://github.com/YunaiV/yudao-cloud)               | 基于 Spring Cloud 微服务架构       |
| [Spring-Boot-Labs](https://gitee.com/yudaocode/SpringBoot-Labs) | [![Gitee star](https://gitee.com/yudaocode/SpringBoot-Labs/badge/star.svg?theme=white)](https://gitee.com/zhijiantianya/yudao-cloud) [![GitHub stars](https://img.shields.io/github/stars/yudaocode/SpringBoot-Labs.svg?style=social&label=Stars)](https://github.com/yudaocode/SpringBoot-Labs) | 系统学习 Spring Boot & Cloud 专栏 |

### 前端项目

| 项目                                                                         | Star                                                                                | 简介                                     |
|----------------------------------------------------------------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|----------------------------------------|
| [yudao-ui-admin-vue3](https://gitee.com/yudaocode/yudao-ui-admin-vue3)     | [![Gitee star](https://gitee.com/yudaocode/yudao-ui-admin-vue3/badge/star.svg?theme=white)](https://gitee.com/yudaocode/yudao-ui-admin-vue3) [![GitHub stars](https://img.shields.io/github/stars/yudaocode/yudao-ui-admin-vue3.svg?style=social&label=Stars)](https://github.com/yudaocode/yudao-ui-admin-vue3)         | 基于 Vue3 + element-plus 实现的管理后台         |
| [yudao-ui-admin-vben](https://gitee.com/yudaocode/yudao-ui-admin-vben)     | [![Gitee star](https://gitee.com/yudaocode/yudao-ui-admin-vben/badge/star.svg?theme=white)](https://gitee.com/yudaocode/yudao-ui-admin-vben) [![GitHub stars](https://img.shields.io/github/stars/yudaocode/yudao-ui-admin-vben.svg?style=social&label=Stars)](https://github.com/yudaocode/yudao-ui-admin-vben)         | 基于 Vue3 + vben(ant-design-vue) 实现的管理后台 |
| [yudao-mall-uniapp](https://gitee.com/yudaocode/yudao-mall-uniapp)         | [![Gitee star](https://gitee.com/yudaocode/yudao-mall-uniapp/badge/star.svg?theme=white)](https://gitee.com/yudaocode/yudao-mall-uniapp) [![GitHub stars](https://img.shields.io/github/stars/yudaocode/yudao-mall-uniapp.svg?style=social&label=Stars)](https://github.com/yudaocode/yudao-mall-uniapp)                 | 基于 uni-app 实现的商城小程序                    |
| [yudao-ui-admin-vue2](https://gitee.com/yudaocode/yudao-ui-admin-vue2)     | [![Gitee star](https://gitee.com/yudaocode/yudao-ui-admin-vue2/badge/star.svg?theme=white)](https://gitee.com/yudaocode/yudao-ui-admin-vue2) [![GitHub stars](https://img.shields.io/github/stars/yudaocode/yudao-ui-admin-vue2.svg?style=social&label=Stars)](https://github.com/yudaocode/yudao-ui-admin-vue2)         | 基于 Vue2 + element-ui 实现的管理后台           |
| [yudao-ui-admin-uniapp](https://gitee.com/yudaocode/yudao-ui-admin-uniapp) | [![Gitee star](https://gitee.com/yudaocode/yudao-ui-admin-uniapp/badge/star.svg?theme=white)](https://gitee.com/yudaocode/yudao-ui-admin-uniapp) [![GitHub stars](https://img.shields.io/github/stars/yudaocode/yudao-ui-admin-uniapp.svg?style=social&label=Stars)](https://github.com/yudaocode/yudao-ui-admin-uniapp) | 基于 Vue2 + element-ui 实现的管理后台           |
| [yudao-ui-go-view](https://gitee.com/yudaocode/yudao-ui-go-view)           | [![Gitee star](https://gitee.com/yudaocode/yudao-ui-go-view/badge/star.svg?theme=white)](https://gitee.com/yudaocode/yudao-ui-go-view) [![GitHub stars](https://img.shields.io/github/stars/yudaocode/yudao-ui-go-view.svg?style=social&label=Stars)](https://github.com/yudaocode/yudao-ui-go-view)                     | 基于 Vue3 + naive-ui 实现的大屏报表             |

## 😎 开源协议

**为什么推荐使用本项目？**

① 本项目采用比 Apache 2.0 更宽松的 [MIT License](https://gitee.com/zhijiantianya/ruoyi-vue-pro/blob/master/LICENSE) 开源协议，个人与企业可 100% 免费使用，不用保留类作者、Copyright 信息。

② 代码全部开源，不会像其他项目一样，只开源部分代码，让你无法了解整个项目的架构设计。[国产开源项目对比](https://www.yuque.com/xiatian-bsgny/lm0ec1/wqf8mn)

![开源项目对比](/.image/common/project-vs.png)

③ 代码整洁、架构整洁，遵循《阿里巴巴 Java 开发手册》规范，代码注释详细，57000 行 Java 代码，22000 行代码注释。

## 🤝 项目外包

我们也是接外包滴，如果你有项目想要外包，可以微信联系【**Aix9975**】。

团队包含专业的项目经理、架构师、前端工程师、后端工程师、测试工程师、运维工程师，可以提供全流程的外包服务。

项目可以是商城、SCRM 系统、OA 系统、物流系统、ERP 系统、CMS 系统、HIS 系统、支付系统、IM 聊天、微信公众号、微信小程序等等。

## 🐼 内置功能

系统内置多种多种业务功能，可以用于快速你的业务系统：

![功能分层](/.image/common/ruoyi-vue-pro-biz.png)

* 通用模块（必选）：系统功能、基础设施
* 通用模块（可选）：工作流程、支付系统、数据报表、会员中心
* 业务系统（按需）：ERP 系统、CRM 系统、商城系统、微信公众号

> 友情提示：本项目基于 RuoYi-Vue 修改，**重构优化**后端的代码，**美化**前端的界面。
>
> * 额外新增的功能，我们使用 🚀 标记。
> * 重新实现的功能，我们使用 ⭐ 标记。

🙂 所有功能，都通过 **单元测试** 保证高质量。

### 系统功能

|     | 功能    | 描述                              |
|-----|-------|---------------------------------|
|     | 用户管理  | 用户是系统操作者，该功能主要完成系统用户配置          |
| ⭐  | 在线用户  | 当前系统中活跃用户状态监控，支持手动踢下线           |
|     | 角色管理  | 角色菜单权限分配、设置角色按机构进行数据范围权限划分      |
|     | 菜单管理  | 配置系统菜单、操作权限、按钮权限标识等，本地缓存提供性能    |
|     | 部门管理  | 配置系统组织机构（公司、部门、小组），树结构展现支持数据权限  |
|     | 岗位管理  | 配置系统用户所属担任职务                    |
| 🚀  | 租户管理  | 配置系统租户，支持 SaaS 场景下的多租户功能        |
| 🚀  | 租户套餐  | 配置租户套餐，自定每个租户的菜单、操作、按钮的权限       |
|     | 字典管理  | 对系统中经常使用的一些较为固定的数据进行维护          |
| 🚀  | 短信管理  | 短信渠道、短息模板、短信日志，对接阿里云、腾讯云等主流短信平台 |
| 🚀  | 邮件管理  | 邮箱账号、邮件模版、邮件发送日志，支持所有邮件平台       |
| 🚀  | 站内信   | 系统内的消息通知，提供站内信模版、站内信消息          |
| 🚀  | 操作日志  | 系统正常操作日志记录和查询，集成 Swagger 生成日志内容 |
| ⭐  | 登录日志  | 系统登录日志记录查询，包含登录异常               |
| 🚀  | 错误码管理 | 系统所有错误码的管理，可在线修改错误提示，无需重启服务     |
|     | 通知公告  | 系统通知公告信息发布维护                    |
| 🚀  | 敏感词   | 配置系统敏感词，支持标签分组                  |
| 🚀  | 应用管理  | 管理 SSO 单点登录的应用，支持多种 OAuth2 授权方式 |
| 🚀  | 地区管理  | 展示省份、城市、区镇等城市信息，支持 IP 对应城市      |

![功能图](/.image/common/system-feature.png)

### 工作流程

|     | 功能    | 描述                                     |
|-----|-------|----------------------------------------|
| 🚀  | 流程模型  | 配置工作流的流程模型，支持文件导入与在线设计流程图，提供 7 种任务分配规则 |
| 🚀  | 流程表单  | 拖动表单元素生成相应的工作流表单，覆盖 Element UI 所有的表单组件 |
| 🚀  | 用户分组  | 自定义用户分组，可用于工作流的审批分组                    |
| 🚀  | 我的流程  | 查看我发起的工作流程，支持新建、取消流程等操作，高亮流程图、审批时间线    |
| 🚀  | 待办任务  | 查看自己【未】审批的工作任务，支持通过、不通过、转发、委派、退回等操作    |
| 🚀  | 已办任务  | 查看自己【已】审批的工作任务，未来会支持回退操作               |
| 🚀  | OA 请假 | 作为业务自定义接入工作流的使用示例，只需创建请求对应的工作流程，即可进行审批 |

![功能图](/.image/common/bpm-feature.png)

### 支付系统

|     | 功能   | 描述                        |
|-----|------|---------------------------|
| 🚀  | 应用信息 | 配置商户的应用信息，对接支付宝、微信等多个支付渠道 |
| 🚀  | 支付订单 | 查看用户发起的支付宝、微信等的【支付】订单     |
| 🚀  | 退款订单 | 查看用户发起的支付宝、微信等的【退款】订单     |
| 🚀  | 回调通知 | 查看支付回调业务的【支付】【退款】的通知结果    |
| 🚀  | 接入示例 | 提供接入支付系统的【支付】【退款】的功能实战    |

### 基础设施

|    | 功能        | 描述                                           |
|----|-----------|----------------------------------------------|
| 🚀 | 代码生成      | 前后端代码的生成（Java、Vue、SQL、单元测试），支持 CRUD 下载       |
| 🚀 | 系统接口      | 基于 Swagger 自动生成相关的 RESTful API 接口文档          |
| 🚀 | 数据库文档     | 基于 Screw 自动生成数据库文档，支持导出 Word、HTML、MD 格式      |
|    | 表单构建      | 拖动表单元素生成相应的 HTML 代码，支持导出 JSON、Vue 文件         |
| 🚀 | 配置管理      | 对系统动态配置常用参数，支持 SpringBoot 加载                 |
| ⭐ | 定时任务      | 在线（添加、修改、删除)任务调度包含执行结果日志                     |
| 🚀 | 文件服务      | 支持将文件存储到 S3（MinIO、阿里云、腾讯云、七牛云）、本地、FTP、数据库等   |
| 🚀 | WebSocket | 提供 WebSocket 接入示例，支持一对一、一对多发送方式              |
| 🚀 | API 日志    | 包括 RESTful API 访问日志、异常日志两部分，方便排查 API 相关的问题   |
|    | MySQL 监控  | 监视当前系统数据库连接池状态，可进行分析SQL找出系统性能瓶颈              |
|    | Redis 监控  | 监控 Redis 数据库的使用情况，使用的 Redis Key 管理           |
| 🚀 | 消息队列      | 基于 Redis 实现消息队列，Stream 提供集群消费，Pub/Sub 提供广播消费 |
| 🚀 | Java 监控   | 基于 Spring Boot Admin 实现 Java 应用的监控           |
| 🚀 | 链路追踪      | 接入 SkyWalking 组件，实现链路追踪                      |
| 🚀 | 日志中心      | 接入 SkyWalking 组件，实现日志中心                      |
| 🚀 | 服务保障      | 基于 Redis 实现分布式锁、幂等、限流功能，满足高并发场景              |
| 🚀 | 日志服务      | 轻量级日志中心，查看远程服务器的日志                           |
| 🚀 | 单元测试      | 基于 JUnit + Mockito 实现单元测试，保证功能的正确性、代码的质量等    |

![功能图](/.image/common/infra-feature.png)

### 数据报表

|     | 功能    | 描述                 |
|-----|-------|--------------------|
| 🚀  | 报表设计器 | 支持数据报表、图形报表、打印设计等  |
| 🚀  | 大屏设计器 | 拖拽生成数据大屏，内置几十种图表组件 |

### 微信公众号

|     | 功能     | 描述                            |
|-----|--------|-------------------------------|
| 🚀  | 账号管理   | 配置接入的微信公众号，可支持多个公众号           |
| 🚀  | 数据统计   | 统计公众号的用户增减、累计用户、消息概况、接口分析等数据  |
| 🚀  | 粉丝管理   | 查看已关注、取关的粉丝列表，可对粉丝进行同步、打标签等操作 |
| 🚀  | 消息管理   | 查看粉丝发送的消息列表，可主动回复粉丝消息         |
| 🚀  | 自动回复   | 自动回复粉丝发送的消息，支持关注回复、消息回复、关键字回复 |
| 🚀  | 标签管理   | 对公众号的标签进行创建、查询、修改、删除等操作       |
| 🚀  | 菜单管理   | 自定义公众号的菜单，也可以从公众号同步菜单         |
| 🚀  | 素材管理   | 管理公众号的图片、语音、视频等素材，支持在线播放语音、视频 |
| 🚀  | 图文草稿箱  | 新增常用的图文素材到草稿箱，可发布到公众号         |
| 🚀  | 图文发表记录 | 查看已发布成功的图文素材，支持删除操作           |

### 商城系统

![功能图](/.image/common/mall-feature.png)

![功能图](/.image/common/mall-preview.png)

演示地址：<https://cloud.iocoder.cn/mall-preview/>

### 会员中心

|     | 功能   | 描述                               |
|-----|------|----------------------------------|
| 🚀  | 会员管理 | 会员是 C 端的消费者，该功能用于会员的搜索与管理        |
| 🚀  | 会员标签 | 对会员的标签进行创建、查询、修改、删除等操作           |
| 🚀  | 会员等级 | 对会员的等级、成长值进行管理，可用于订单折扣等会员权益      |
| 🚀  | 会员分组 | 对会员进行分组，用于用户画像、内容推送等运营手段         |
| 🚀  | 积分签到 | 回馈给签到、消费等行为的积分，会员可订单抵现、积分兑换等途径消耗 |

### ERP 系统

![功能图](/.image/common/erp-feature.png)

演示地址：<https://cloud.iocoder.cn/erp-preview/>

### ERP 系统

![功能图](/.image/common/crm-feature.png)

演示地址：<https://cloud.iocoder.cn/crm-preview/>

## 🐨 技术栈

### 微服务

| 项目                                                                       | 说明                 |
|--------------------------------------------------------------------------|--------------------|
| `yudao-dependencies`                                                     | Maven 依赖版本管理       |
| `yudao-framework`                                                        | Java 框架拓展          |
| `yudao-server`                                                           | 管理后台 + 用户 APP 的服务端 |
| `yudao-module-system`                                                    | 系统功能的 Module 模块    |
| `yudao-module-member`                                                    | 会员中心的 Module 模块    |
| `yudao-module-infra`                                                     | 基础设施的 Module 模块    |
| `yudao-module-bpm`                                                       | 工作流程的 Module 模块    |
| `yudao-module-pay`                                                       | 支付系统的 Module 模块    |
| `yudao-module-mall`                                                      | 商城系统的 Module 模块    |
| `yudao-module-mp`                                                        | 微信公众号的 Module 模块   |
| `yudao-module-report`                                                    | 大屏报表 Module 模块     |

### 框架

| 框架                                                                                | 说明               | 版本          | 学习指南                                                                |
|---------------------------------------------------------------------------------------------|------------------|-------------|---------------------------------------------------------------------|
| [Spring Cloud Alibaba](https://github.com/alibaba/spring-cloud-alibaba)                     | 微服务框架            | 2021.0.4.0  | [文档](https://github.com/YunaiV/SpringBoot-Labs)                     |
| [Nacos](https://github.com/alibaba/nacos)                                                   | 配置中心 & 注册中心      | 2.0.4       | [文档](https://www.iocoder.cn/categories/Nacos/?yudao)                |
| [RocketMQ](https://github.com/apache/rocketmq)                                              | 消息队列             | 4.9.4       | [文档](https://www.iocoder.cn/categories/RocketMQ/?yudao)             |
| [Sentinel](https://github.com/alibaba/sentinel)                                             | 服务保障             | 1.8.6       | [文档](https://www.iocoder.cn/categories/Sentinel/?yudao)             |
| [XXL Job](https://github.com/xuxueli/xxl-job)                                               | 定时任务             | 2.3.1       | [文档](https://www.iocoder.cn/XXL-JOB/good-collection/?yudao)         |
| [Spring Cloud Gateway](https://github.com/spring-cloud/spring-cloud-gateway)                | 服务网关             | 3.4.1       | [文档](https://www.iocoder.cn/categories/Spring-Cloud-Gateway/?yudao) |
| [Seata](https://github.com/seata/seata)                                                     | 分布式事务            | 1.6.1       | [文档](https://www.iocoder.cn/categories/Seata/?yudao)                |
| [MySQL](https://www.mysql.com/cn/)                                                          | 数据库服务器           | 5.7 / 8.0+  |                                                                     |
| [Druid](https://github.com/alibaba/druid)                                                   | JDBC 连接池、监控组件    | 1.2.19      | [文档](http://www.iocoder.cn/Spring-Boot/datasource-pool/?yudao)      |
| [MyBatis Plus](https://mp.baomidou.com/)                                                    | MyBatis 增强工具包    | 3.5.3.1     | [文档](http://www.iocoder.cn/Spring-Boot/MyBatis/?yudao)              |
| [Dynamic Datasource](https://dynamic-datasource.com/)                                       | 动态数据源            | 3.6.0       | [文档](http://www.iocoder.cn/Spring-Boot/datasource-pool/?yudao)      |
| [Redis](https://redis.io/)                                                                  | key-value 数据库    | 5.0 / 6.0   |                                                                     |
| [Redisson](https://github.com/redisson/redisson)                                            | Redis 客户端        | 3.18.0      | [文档](http://www.iocoder.cn/Spring-Boot/Redis/?yudao)                |
| [Spring MVC](https://github.com/spring-projects/spring-framework/tree/master/spring-webmvc) | MVC 框架           | 5.3.24      | [文档](http://www.iocoder.cn/SpringMVC/MVC/?yudao)                    |
| [Spring Security](https://github.com/spring-projects/spring-security)                       | Spring 安全框架      | 5.7.5       | [文档](http://www.iocoder.cn/Spring-Boot/Spring-Security/?yudao)      |
| [Hibernate Validator](https://github.com/hibernate/hibernate-validator)                     | 参数校验组件           | 6.2.5       | [文档](http://www.iocoder.cn/Spring-Boot/Validation/?yudao)           |
| [Flowable](https://github.com/flowable/flowable-engine)                                     | 工作流引擎            | 6.7.2       | [文档](https://doc.iocoder.cn/bpm/)                                   |
| [Knife4j](https://gitee.com/xiaoym/knife4j)                                                 | Swagger 增强 UI 实现 | 4.0.0       | [文档](http://www.iocoder.cn/Spring-Boot/Swagger/?yudao)              |
| [SkyWalking](https://skywalking.apache.org/)                                                | 分布式应用追踪系统        | 8.12.0      | [文档](http://www.iocoder.cn/Spring-Boot/SkyWalking/?yudao)           |
| [Spring Boot Admin](https://github.com/codecentric/spring-boot-admin)                       | Spring Boot 监控平台 | 2.7.10      | [文档](http://www.iocoder.cn/Spring-Boot/Admin/?yudao)                |
| [Jackson](https://github.com/FasterXML/jackson)                                             | JSON 工具库         | 2.13.3      |                                                                     |
| [MapStruct](https://mapstruct.org/)                                                         | Java Bean 转换     | 1.5.5.Final | [文档](http://www.iocoder.cn/Spring-Boot/MapStruct/?yudao)            |
| [Lombok](https://projectlombok.org/)                                                        | 消除冗长的 Java 代码    | 1.18.26     | [文档](http://www.iocoder.cn/Spring-Boot/Lombok/?yudao)               |
| [JUnit](https://junit.org/junit5/)                                                          | Java 单元测试框架      | 5.8.2       | -                                                                   |
| [Mockito](https://github.com/mockito/mockito)                                               | Java Mock 框架     | 4.8.0       | -                                                                   |

## 🐷 演示图

### 系统功能

| 模块       | biu                         | biu                       | biu                      |
|----------|-----------------------------|---------------------------|--------------------------|
| 登录 & 首页  | ![登录](/.image/登录.jpg)       | ![首页](/.image/首页.jpg)     | ![个人中心](/.image/个人中心.jpg) |
| 用户 & 应用  | ![用户管理](/.image/用户管理.jpg)   | ![令牌管理](/.image/令牌管理.jpg) | ![应用管理](/.image/应用管理.jpg) |
| 租户 & 套餐  | ![租户管理](/.image/租户管理.jpg)   | ![租户套餐](/.image/租户套餐.png) | -                        |
| 部门 & 岗位  | ![部门管理](/.image/部门管理.jpg)   | ![岗位管理](/.image/岗位管理.jpg) | -                        |
| 菜单 & 角色  | ![菜单管理](/.image/菜单管理.jpg)   | ![角色管理](/.image/角色管理.jpg) | -                        |
| 审计日志     | ![操作日志](/.image/操作日志.jpg)   | ![登录日志](/.image/登录日志.jpg) | -                        |
| 短信       | ![短信渠道](/.image/短信渠道.jpg)   | ![短信模板](/.image/短信模板.jpg) | ![短信日志](/.image/短信日志.jpg) |
| 字典 & 敏感词 | ![字典类型](/.image/字典类型.jpg)   | ![字典数据](/.image/字典数据.jpg) | ![敏感词](/.image/敏感词.jpg)  |
| 错误码 & 通知 | ![错误码管理](/.image/错误码管理.jpg) | ![通知公告](/.image/通知公告.jpg) | -                        |

### 工作流程

| 模块      | biu                             | biu                             | biu                             |
|---------|---------------------------------|---------------------------------|---------------------------------|
| 流程模型    | ![流程模型-列表](/.image/流程模型-列表.jpg) | ![流程模型-设计](/.image/流程模型-设计.jpg) | ![流程模型-定义](/.image/流程模型-定义.jpg) |
| 表单 & 分组 | ![流程表单](/.image/流程表单.jpg)       | ![用户分组](/.image/用户分组.jpg)       | -                               |
| 我的流程    | ![我的流程-列表](/.image/我的流程-列表.jpg) | ![我的流程-发起](/.image/我的流程-发起.jpg) | ![我的流程-详情](/.image/我的流程-详情.jpg) |
| 待办 & 已办 | ![任务列表-审批](/.image/任务列表-审批.jpg) | ![任务列表-待办](/.image/任务列表-待办.jpg) | ![任务列表-已办](/.image/任务列表-已办.jpg) |
| OA 请假   | ![OA请假-列表](/.image/OA请假-列表.jpg) | ![OA请假-发起](/.image/OA请假-发起.jpg) | ![OA请假-详情](/.image/OA请假-详情.jpg) |

### 基础设施

| 模块            | biu                           | biu                         | biu                       |
|---------------|-------------------------------|-----------------------------|---------------------------|
| 代码生成          | ![代码生成](/.image/代码生成.jpg)     | ![生成效果](/.image/生成效果.jpg)   | -                         |
| 文档            | ![系统接口](/.image/系统接口.jpg)     | ![数据库文档](/.image/数据库文档.jpg) | -                         |
| 文件 & 配置       | ![文件配置](/.image/文件配置.jpg)     | ![文件管理](/.image/文件管理2.jpg)  | ![配置管理](/.image/配置管理.jpg) |
| 定时任务          | ![定时任务](/.image/定时任务.jpg)     | ![任务日志](/.image/任务日志.jpg)   | -                         |
| API 日志        | ![访问日志](/.image/访问日志.jpg)     | ![错误日志](/.image/错误日志.jpg)   | -                         |
| MySQL & Redis | ![MySQL](/.image/MySQL.jpg)   | ![Redis](/.image/Redis.jpg) | -                         |
| 监控平台          | ![Java监控](/.image/Java监控.jpg) | ![链路追踪](/.image/链路追踪.jpg)   | ![日志中心](/.image/日志中心.jpg) |

### 支付系统

| 模块      | biu                       | biu                             | biu                             |
|---------|---------------------------|---------------------------------|---------------------------------|
| 商家 & 应用 | ![商户信息](/.image/商户信息.jpg) | ![应用信息-列表](/.image/应用信息-列表.jpg) | ![应用信息-编辑](/.image/应用信息-编辑.jpg) |
| 支付 & 退款 | ![支付订单](/.image/支付订单.jpg) | ![退款订单](/.image/退款订单.jpg)       | ---                             |
### 数据报表

| 模块    | biu                             | biu                             | biu                                   |
|-------|---------------------------------|---------------------------------|---------------------------------------|
| 报表设计器 | ![数据报表](/.image/报表设计器-数据报表.jpg) | ![图形报表](/.image/报表设计器-图形报表.jpg) | ![报表设计器-打印设计](/.image/报表设计器-打印设计.jpg) |
| 大屏设计器 | ![大屏列表](/.image/大屏设计器-列表.jpg)   | ![大屏预览](/.image/大屏设计器-预览.jpg)   | ![大屏编辑](/.image/大屏设计器-编辑.jpg)         |

### 移动端（管理后台）

| biu                              | biu                              | biu                              |
|----------------------------------|----------------------------------|----------------------------------|
| ![](/.image/admin-uniapp/01.png) | ![](/.image/admin-uniapp/02.png) | ![](/.image/admin-uniapp/03.png) |
| ![](/.image/admin-uniapp/04.png) | ![](/.image/admin-uniapp/05.png) | ![](/.image/admin-uniapp/06.png) |
| ![](/.image/admin-uniapp/07.png) | ![](/.image/admin-uniapp/08.png) | ![](/.image/admin-uniapp/09.png) |

目前已经实现登录、我的、工作台、编辑资料、头像修改、密码修改、常见问题、关于我们等基础功能。


# StarExec

StarExec is a cross community logic solving service developed at the University
of Iowa under the direction of principal investigators Aaron Stump (Iowa), Geoff
Sutcliffe (University of Miami), and Cesare Tinelli (Iowa).

Its main goal is to facilitate the experimental evaluation of logic solvers,
broadly understood as automated tools based on formal reasoning. The service is
designed to provide a single piece of storage and computing infrastructure to
logic solving communities and their members. It aims at reducing duplication of
effort and resources as well as enabling individual researchers or groups with
no access to comparable infrastructure.

StarExec allows

 * community organizers to store, manage and make available benchmark libraries;
 * competition organizers to run logic solver competitions; and
 * community members to perform comparative evaluations of logic solvers on
   public or private benchmark problems.

Development details can be found on our
[public development wiki](http://wiki.uiowa.edu/display/stardev/Home).

## Dependencies

### Java

StarExec requires Ant to build, and has been tested with version 1.9.2.
An installation guide for Ant is below, or you may use any applicable
package manager.

http://ant.apache.org/manual/install.html

### SASS

StarExec requires [Sass](https://sass-lang.com) at build time to compile `.scss`
stylesheets to `.css`. Sass depends on [Ruby](https://www.ruby-lang.org/en/).

We are currently using Ruby Sass 3.4.24

https://sass-lang.com/install

### Apache Tomcat

StarExec depends on Apache Tomcat 7.0.64. While newer versions may work, we have
frequently seen that even minor version releases of Tomcat can have breaking
changes for StarExec, so using a different version of Tomcat is not recommended.

A full release of Tomcat is included in the starexec package under the
`distribution/` directory. This is identical to a release that you can download
from Apache, with the exception that the `mysql-connector-java-5.1.22-bin.jar`
and file is included in the `lib/` directory. This `.jar` file
is required for StarExec to connect to its database, and as such we
recommend that you install Tomcat using the provided archive. If you would like
to install a clean copy of Tomcat, you will need to copy MySQL connector to the new lib directory.

PLEASE NOTE: We recently migrated our servers from centOS to Rocky8. Due to this, DRMAA is no longer
a required dependency. Instead, we will be using qsub. For more information about this command, please
see [https://www.jlab.org/hpc/PBS/qsub.html](https://www.jlab.org/hpc/PBS/qsub.html)

If you install Tomcat using the provided archive, you may need to update
permissions on the install directory to make Tomcat's scripts executable. This
can be done, for example, by using `chmod 700 -R tomcat_directory`

### MySQL and MariaDB

StarExec depends on MariaDB 5.5.56.

https://downloads.mariadb.org/

### Backend

StarExec's backend refers to the utility that is responsible for accepting new
jobs from the web app and distributing them over the available compute nodes.
StarExec supports 3 different backend implementations:
[SGE](https://arc.liv.ac.uk/trac/SGE),
[OAR](https://oar.imag.fr/),
or a simple local backend implemented in StarExec itself.

To install SGE or OAR, you will need to refer to their documentation. In the
case of OAR, the document
[`distribution/OAR installation notes.txt`](distribution/OAR%20installation%20notes.txt)
describes extra installation steps you should take to configure OAR for use with
StarExec.

The local backend is a primitive solution if you only want to run jobs on the
same machine that StarExec is running off of. The local backend does not support
multiple queues or nodes, and only a single job will run at a time.

## Configuration

StarExec is configured by Ant at build time.

StarExec's default configuration is specified by
[`build/default.properties`](build/default.properties), but several properties
will need to be overridden for a particular StarExec instance. These properties
may be overridden in three ways:

1) If `build/overrides.properties` exists, any properties specified in this file
   will override the defaults
2) Ant can be passed a `.properties` file if invoked with the `-propertyfile`
   option. Any properties specified in this file will override the defaults
   _and_ any properties that exist in `build/overrides.properties`
3) Individual properties can be set by invoking Ant with the
   `-D<property>=<value>` option. Any properties set this way will take
   precidence.

An empty configuration file is provided as
[`example.properties`](example.properties).
This file also explains the properties that must be set for a particular
StarExec instance.

### Database

`DB.User` must be set to the username of a MariaDB user that has full
permissions for the database. `DB.Pass` must be set to the password for that
user. This user will require _all_ permissions in the StarExec database,
excluding server administration permissions.
If desired, a user with fewer permissions may be used by the compute nodes to
report results to the database. This user _only_ needs `EXECUTE` permission, and
is configured via `Cluster.DB.User` and `Cluster.DB.Pass`. If unspecified, these
will default to the values of `DB.User` and `DB.Pass`.

### Email

StarExec sends automated emails for several purposes, such as sending
notifications when new users are registered or sending weekly status updates.
To do this, StarExec requires an email account that it can send emails from.
`Email.User` should be set to the username of the account to send from, and
`Email.Pass`, `Email.Smtp` and `Email.Port` should be set as decribed.

StarExec is also configured to use a `Email.Contact`, which is intended to
receive emails directed at StarExec admins. This email address will appear
on the site for users who want to send bug reports or ask questions.

### Backend

You will need to make sure that you have mapped the StarExec data directory,
(`data_dir`), to a matching path on each compute node, as your compute nodes
will need access to the StarExec data directory that exists on the head node.

`Cluster.UserOne` and `Cluster.UserTwo` (by default, `sandbox` and `sandbox2`
respectively) refer to users that will execute jobs on compute nodes.
Ensure that these accounts exist on the head node and all compute nodes, and
have appropriate permissions.

Create the `star-web` group.

Create the user `tomcat` and add this user to the `star-web` group, and change
the primary group for `tomcat` to `star-web`.
This is the user that you will need to use when starting up **Tomcat** using
`startup.sh` in the **Tomcat** `bin/` folder.
You should also ensure that `tomcat` is the owner of the entire
**Tomcat** installation directory.

Finally, any users that are going to be administering StarExec should also be
added to the `star-web` group. Being a member of `star-web` will be necessary
for correctly executing the StarExec deploy scripts.

Create the `sandbox` group, and add `Cluster.UserOne` to this group.
Create another group `sandbox2` and add `Cluster.UserTwo` to this group.
Add the `tomcat` user to both of these groups.

If you are using SGE as a backend, you need to create the user `sgeadmin` and
ensure this user does have administrator privileges in SGE.

A sandbox directory will need to be created on the StarExec head node.
This directory is used to execute user-provided scripts in a sandboxed
environment, preventing them from affecting other parts of the system.
You should create a directory at the location specified by `sandbox_dir`.
Make the owner `Cluster.UserOne`, and make the group `sandbox`.
Use `chmod` on the directory to make permissions `770`.
Additionally, use `chmod g+s` to set the GID for the directory.
Finally, use the following command to ensure that new directories in the sandbox
have `g+rwx` permissions.

    setfacl -d -m g::rwx sandbox

The directory configured as `Backend.WorkingDir` needs to be created.
`tomcat` should be the owner and `star-web` should be the group.
Under this directory, create two directories named `sandbox/` and `sandbox2/`.
These should also use the `tomcat` user and the `star-web` group.

Sudo permissions need to be configured.
StarExec uses `sudo` in several locations to execute commands as other users,
most often to execute commands using the `Cluster.UserOne` and `Cluster.UserTwo`
users. The `tomcat` user will need all of the following `sudo` permissions.

#### HEAD NODE

User `tomcat` may run the following commands on this host:

    (SANDBOX_USER_ONE) NOPASSWD: ALL
    (SANDBOX_USER_TWO) NOPASSWD: ALL
    (root) NOPASSWD: /sbin/service tomcat7 restart, /sbin/service tomcat7 stop, /sbin/service tomcat7 start

The following entries are needed only if you are using an SGE backend.
Replace `/cluster/gridengine-8.1.8/bin/lx-amd64/` in each path with your install directory

    (sgeadmin) NOPASSWD: /cluster/gridengine-8.1.8/bin/lx-amd64/qconf, /cluster/gridengine-8.1.8/bin/lx-amd64/qmod

####  COMPUTE NODE (or head node if you are using a local backend)

User `tomcat` may run the following commands on this host:

For all of the following, the prefix `/export/starexec` should be replaced with
your configured value of `Backend.WorkingDir`, and `UserOne` and `UserTwo`
should be replaced with the values of `Cluster.UserOne` and `Cluster.UserTwo`
respectively.

    (UserOne) NOPASSWD: ALL
    (root) NOPASSWD: /bin/chown -R UserOne /export/starexec/sandbox, /bin/chown -R tomcat /export/starexec/sandbox, /bin/chown -R tomcat /export/starexec/sandbox/benchmark, /bin/chown tomcat
    (UserTwo) NOPASSWD: ALL
    (root) NOPASSWD: /bin/chown -R UserTwo /export/starexec/sandbox2, /bin/chown -R tomcat /export/starexec/sandbox2, /bin/chown -R tomcat /export/starexec/sandbox2/benchmark, /bin/chown tomcat

The same applies as on the head node for the following commands

    (sgeadmin) NOPASSWD: /cluster/gridengine-8.1.8/bin/lx-amd64/qconf, /cluster/gridengine-8.1.8/bin/lx-amd64/qmod


[![LICENSE](https://img.shields.io/github/license/tapdata/tapdata.svg)](https://github.com/tapdata/tapdata/blob/main/LICENSE)
[![Contributors](https://img.shields.io/github/contributors/tapdata/tapdata)](https://github.com/tapdata/tapdata/graphs/contributors)
[![Activity](https://img.shields.io/github/commit-activity/m/tapdata/tapdata)](https://github.com/tapdata/tapdata/pulse)
[![Release](https://img.shields.io/github/v/tag/tapdata/tapdata.svg?sort=semver)](https://github.com/tapdata/tapdata/releases)

---

[![Try Cloud Service](<https://img.shields.io/badge/-Try%20Cloud%20Service%20%E2%86%92-rgb(255,140,0)?style=for-the-badge>)](https://cloud.tapdata.net)
[![Official Website](<https://img.shields.io/badge/-Official%20Website%20%E2%86%92-rgb(59,71,229)?style=for-the-badge>)](https://tapdata.io)
[![Docs](<https://img.shields.io/badge/-Online%20Documentation%20%E2%86%92-rgb(0,205,102)?style=for-the-badge>)](https://docs.tapdata.io)

## What is Tapdata ?
Tapdata is a Change Data Capture(CDC) based, Real-Time data integration platform that enables data to be synchronized in real-time among various systems such as databases, SaaS services, applications, and files.
The synchronization tasks can be easily built through drag-and-drop operations, from table creation to full and incremental synchronization, all processes are fully automated.

1. [Supported Connectors](https://docs.tapdata.io/introduction/supported-databases)
2. [FAQ](https://docs.tapdata.io/faq/)

For more details, please read [docs](https://docs.tapdata.io/)

Based on different distribution strategies and functionalities, Tapdata has three distribution versions: Community Edition, Cloud Service, and Enterprise Edition. All content under this open-source project corresponds to the Community Edition.

For the differences between the three versions, you can refer to: https://tapdata.io/

## Table of contents

  1. Key Features

  2. Use Cases

  3. Quick Start

  4. Examples

  5. License

  6. Contact Us

  7. Build From Source

  8. Contributing

## Key Features
### End-To-End Visual UI

|                                                                                |                                       |
| ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------- |
| Full Process Automation
Automatic table creation, and Automatic switching between Full and Incremental synchronization.

Drag-And-Drop
Operate through a visual interface, Easy to use

Monitor
sync performance, task progress, key events, and logs.                                                                    |      |

### Lightweight ETL Solution

|                                                                                |                                       |
| ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------- |
| Table Filter and Mapping
Rich visual processors that allow table selection, renaming, adding or deleting fields, and renaming.

Record Transformation
Easily use JS/Python code to do transformation

Union
Union multi tables to one target.                                                   |  |

### Build materialized views in MongoDB

|                                                                                |                                        |
| ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------- |
| Document-style view
Support Embedded documents and arrays, deeply integrated with MongoDB.

Multi-table association
hierarchical nesting, and convenient construction of 1:1 and 1:N models.

Multi-table stream merging
Unified batch and stream processing, multi-table updates in any order. |  |

## Primary Use Cases
1. Synchronize data from traditional RDBMS to modern databases such as MongoDB, Elasticsearch (ES), or Redis to support new business uses.
2. Consolidate data from various databases into a unified data warehouse.
3. Stream database changes to Kafka.
4. Perform data synchronization between heterogeneous databases.
5. Use MongoDB to build your unified data hub.

## Quick Start
### Start with docker
To run the complete service, the basic resource requirements are:
1. At least 5GB of available memory
2. At least 20GB of available disk space
3. At least 1 free CPU core

RUN `docker run -d -p 3030:3030 ghcr.io/tapdata/tapdata:latest`, wait for 3 minutes, then you can get it from http://localhost:3030/, if everything is ok, you can see login page like this:

default username is: admin@admin.com, default password is admin

### Start with cloud service
Tapdata service is available in cloud service, you can use fully-managed service, or deploy engine to your private network

Try on https://cloud.tapdata.io/, support google and github account login, free trial, NO credit card needed, start your real-time data journey immediately.

## Examples

🗂 Create Datasource and Test it

1. Login tapdata platform

2. In the left navigation panel, click Connections

3. On the right side of the page, click Create

4. In the pop-up dialog, search and select MySQL

5. On the page that you are redirected to, follow the instructions below to fill in the connection information for MySQL

6. Click Test, make sure all test pass, then click Save

🗂 Sync Data From MySQL To MongoDB

1. Create MySQL and MongoDB data source

2. In the left navigation panel, click Data Pipelines -> Data Replications

3. On the right side of the page, click Create

4. Drag and drop MySQL and MongoDB data sources onto the canvas

5. Drag a line from the MySQL data source to MongoDB

6. Configure the MySQL data source and select the data tables you want to synchronize

7. Click the Save button in the upper right corner, then click the Start button

8. Observe the indicators and events on the task page until data is in sync

🗂 MySQL To PostgreSQL with Simple ETL

1. Create MySQL and PostgreSQL data source

2. In the left navigation panel, click Data Pipelines -> Data Transformation

3. On the right side of the page, click Create

4. Drag and drop MySQL and PostgreSQL data sources onto the canvas

5. Drag a line from the MySQL data source to PostgreSQL

6. Click the plus sign on the connection line and select Field Rename

7. Click Field Rename node, change i_price to price, i_data to data in config form

8. Click the Save button in the upper right corner, then click the Start button

9. Observe the indicators and events on the task page until data is in sync

🗂 Making materialized views in MongoDB

Materialized view is a special feature of tapdata, You can give full play to the characteristics of MongoDB document database and create the data model you need, try enjoy it !

In this example, I will make a view using 2 tables in MySQL: order and product, make product as a embedded document of order, here is the step:

1. Create MySQL and MongoDB data source

2. In the left navigation panel, click Data Pipelines -> Data Transformation

3. On the right side of the page, click Create

4. Click mysql data source in left up side, then drag and drop order table and product table onto the canvas

5. Drag and drop "Master-slave merge" node in left bottom side onto the canvas

6. Drag a line from the order table to Master-slave merge

7. Drag a line from the product table to Master-slave merge

8. Drag and drop MongoDB data source onto the canvas, and drag a line from the "Master-slave merge" node to MongoDB node

9. Click "Master-slave merge" node, then drag product table into order table in the right side in "Table Name"

10. Click "Master-slave merge" node, then click product table, config Data write model to "Match and Merge", Field write path to "product", Association Conditions to "order_id" => "order_id", then you can see Schema in bottom changed

11. Click MongoDB node, and config target table name as order_with_product, update condition field config as "order_id"

12. Click the Save button in the upper right corner, then click the Start button

13. Observe the indicators and events on the task page until data is in sync

14. Check collection order_with_product in MongoDB, and you will see the data model

🗂 Data consistency check

Using the data verification feature, you can quickly check whether the synchronized data is consistent and accurate

1. In the left navigation panel, click Data Pipelines -> Data Validation

2. On the right side of the page, click Task Consistency Validation

3. Choose 1 task, and valid type choose "All Fields Validation", it means system will check all fields for all record

4. Click Save, then click Execute in the task list

5. Wait validation task finished, click Result in the task list, and check the validation result

## Architecture
![Alt Text](./assets/architecture.jpg)

## License
Tapdata Community is under the Apache 2.0 license. See the [LICENSE](https://github.com/tapdata/tapdata/blob/main/LICENSE) file for details.

## Contact Us
- [Send Email](mailto:team@tapdata.io)
- [Slack channel](https://join.slack.com/t/tapdatacommunity/shared_invite/zt-1biraoxpf-NRTsap0YLlAp99PHIVC9eA)


= Testcontainers Spring Boot

https://codecov.io/gh/PlaytikaOSS/testcontainers-spring-boot[image:https://codecov.io/gh/PlaytikaOSS/testcontainers-spring-boot/graph/badge.svg?token=9E0VBFIB5B[codecov]]
https://maven-badges.herokuapp.com/maven-central/com.playtika.testcontainers/testcontainers-spring-boot[image:https://maven-badges.herokuapp.com/maven-central/com.playtika.testcontainers/testcontainers-spring-boot/badge.svg[Maven Central]]

If you develop services using Spring Boot and maybe Spring Cloud and you do
https://testing.googleblog.com/2010/12/test-sizes.html[medium sized] tests during build process, then this set of
Spring Boot auto-configurations might be handy. By adding module into classpath, you will get stateful service,
like Couchbase or Kafka, auto-started and available for connection from your application service w/o wiring any
additional code. https://www.docker.com/[Docker] and https://www.testcontainers.org/[TestContainers] are used to
bootstrap stateful service using Spring Cloud https://cloud.spring.io/spring-cloud-static/spring-cloud.html#_the_bootstrap_application_context[bootstrap phase].
Usage of Spring Cloud in your production code is optional, but __you will need it in tests__. See <<how-to-use, How to use>> below.

== Versions compatibility

|===
| Spring Boot | Test Containers Spring Boot

|2.5.X, 2.6.X, 2.7.X
|2.X.X

|3.0.X, 3.1.X
|3.0.X

|3.2.X
|3.1.X
|===

[[how-to-use]]
== How to use

. https://docs.docker.com/install/[Install Docker] on your machine
. Make sure you have http://projects.spring.io/spring-cloud/#quick-start[Spring Boot and Spring Cloud] in classpath of your tests.
In case if you need to https://mvnrepository.com/artifact/org.springframework.cloud/spring-cloud-starter-bootstrap[pick version].
+
.pom.xml
[source,xml]
----
<project>
...
      <dependency>
            <groupId>org.springframework.cloud</groupId>
            <artifactId>spring-cloud-starter-bootstrap</artifactId>
            <!-- https://mvnrepository.com/artifact/org.springframework.cloud/spring-cloud-starter-bootstrap -->
            <version>[pick version]</version>
        </dependency>
...
</project>
----
+
NOTE: `testcontainers-spring-boot` project migrated to Spring Boot 2.4 in 2.0.0 version.
Please note, that in order to use this project with Spring Boot 2.4, you need to use `spring-cloud-starter-bootstrap` dependency.
For earlier Spring Boot (prior to 2.4) users -- you need to use `spring-cloud-starter` dependency instead.

. If you do not use Spring Cloud - make it work for tests only:
+
.pom.xml
[source,xml]
----
<project>
...
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter</artifactId>
            <!-- https://mvnrepository.com/artifact/org.springframework.boot/spring-boot-starter -->
            <version>[pick version]</version>
        </dependency>
        <dependency>
            <groupId>org.springframework.cloud</groupId>
            <artifactId>spring-cloud-starter-bootstrap</artifactId>
            <!-- https://mvnrepository.com/artifact/org.springframework.cloud/spring-cloud-starter-bootstrap -->
            <version>[pick version]</version>
            <scope>test</scope>
        </dependency>
...
</project>
----

. Add data service library:
+
.pom.xml
[source,xml]
----
<project>
...
        <dependency>
            <groupId>com.playtika.testcontainers</groupId>
            <artifactId>embedded-kafka</artifactId>
            <!-- https://mvnrepository.com/artifact/com.playtika.testcontainers/ -->
            <version>[pick version]</version>
            <scope>test</scope>
        </dependency>
...
</project>
----

. Use produced properties in your configuration.
+
Example:
+
./src/test/resources/application.properties
[source,properties]
----
spring.kafka.bootstrap-servers=${embedded.kafka.brokerList}
----
+
./src/test/resources/bootstrap.properties
[source,properties]
----
embedded.kafka.topicsToCreate=some_topic
----

== In-depth guides, how-tos and samples

- https://martinfowler.com/articles/microservice-testing/[Testing Strategies in a Microservice Architecture]
- https://dzone.com/articles/advanced-functional-testing-in-spring-boot-by-usin[Functional Testing in Spring Boot Using Docker in Tests]
- https://github.com/tdanylchuk/functional-tests-best-practices[Functional tests best practices sample repo]
- https://medium.com/@isadounikau/microservices-integration-testing-spring-boot-404b6f8617d1[Spring Boot Microservices Integration Testing]
- https://mydeveloperplanet.com/2020/05/05/easy-integration-testing-with-testcontainers[Easy Integration Testing With Testcontainers]
- https://dev.to/sivalabs/springboot-integration-testing-using-testcontainers-starter-13h2[SpringBoot Integration Testing using TestContainers Starter]
- https://alexromanov.github.io/2019/04/02/spring-boot-docker-containers/[Easy way to test Spring Boot microservices]

== Common configuration options
=== Shutdown of embedded containers on spring application shutdown immediately
./src/test/resources/application.properties
[source,properties]
----
embedded.containers.forceShutdown=true #default is false
----
NOTE: Otherwise, it will be shutdown with delay, see https://github.com/testcontainers/moby-ryuk

=== Disable all embedded containers

./src/test/resources/bootstrap.properties
[source,properties]
----
embedded.containers.enabled=true #default is true
----
NOTE: If you setup, for example  embedded.kafka.enabled + embedded.containers.enabled, result will be same as using AND between two booleans.

NOTE: embedded.kafka.enabled=false will cause DockerNotPresentException if you don't have docker installed. But embedded.containers.enabled=false won't cause any exceptions in this case.

|===
|Setting1 |Setting2 |Outcome

|embedded.containers.enabled=false
|embedded.memsql.enabled=true
|Memsql will not start

|embedded.containers.enabled=true
|embedded.memsql.enabled=false
|Memsql will not start

|embedded.containers.enabled=true
|embedded.memsql.enabled=true
|Memsql will start

|embedded.containers.enabled is missing
|embedded.memsql.enabled is missing
|Memsql will start
|===

=== Other specific container related properties
[cols="a,a,a"]
|===
|Setting name | Default value |Description

|embedded.{module-name}.dockerImage
|Depends on module
|Full Docker image name for container setup. Most of the modules have default value already setup.

|embedded.{module-name}.dockerImageVersion
|N/A
|Use this property if you want to override only Docker image's version.

|embedded.{module-name}.waitTimeoutInSeconds
|60
|Waiting time for a container to start in seconds

|embedded.{module-name}.enabled
|true
|Enables a container to be started on startup

|embedded.{module-name}.reuseContainer
|false
|Enables a reuse container Testcontainers feature. For more info please refer to: https://github.com/testcontainers/testcontainers-java/pull/2555 and https://github.com/testcontainers/testcontainers-java/pull/1781.

|embedded.{module-name}.command
|null
|List of keywords which combines into command for container startup. Some modules ship container's commands by default, so resetting this value may lead to incorrect work of container.

|embedded.{module-name}.attachContainerLog
|false
|Attach embedded container output log.

|embedded.{module-name}.env
|null
|key-value map of additional environment variables. Where key is name of variable and value is actual value of it.

|embedded.{module-name}.label
|null
|key-value map of additional labels to the container. Where key is name of label and value is actual value of label.

|embedded.{module-name}.filesToInclude
| empty list
|List of files to include objects.
Each object should have two parameters:

 * `classpathResource` (path to local file)
 * `containerPath` (path in a container to where file needs to be copied)

Example:
[source,yaml]
----
embedded.redis.filesToInclude:
  classpathResource: "/my_local_file.txt"
  containerPath: "/etc/path_in_container.txt"
----

|embedded.{module-name}.mountVolumes
| empty list
|List of mount volumes to persist between container restarts.
Each object should have three parameters:

 * `hostPath` (path to local file/directory)
 * `containerPath` (path in container to mount file/directory onto)
 * `mode` (access mode default *READ_ONLY*, or *READ_WRITE*)

Example:
[source,yaml]
----
embedded.postgresql.mountVolumes:
  hostPath: "pgdata"
  containerPath: "/var/lib/postgresql/data"
  mode: READ_WRITE
----

|embedded.{module-name}.capabilities
| empty list. `NET_ADMIN` is set for Aerospike, Couchbase, Elasticsearch, Kafka, Mariadb, Memsql, Minio, Mongodb, Mysql, Neo4j, Redis containers.
|The Linux capabilities that should be enabled. You can disable all capabilities by providing empty value for this property.
See: https://man7.org/linux/man-pages/man7/capabilities.7.html.
Available values can be taken from `com.github.dockerjava.api.model.Capability` class.

|embedded.{module-name}.tmpFs.mounts
| empty list
| A list of container directories which should be replaced by tmpfs mounts, and their corresponding mount options. Check https://docs.docker.com/storage/tmpfs/[TmpFs mount docs].

For example, for MariaDb:
[source,yaml]
----
embedded:
  mariadb:
    tmp-fs:
      mounts:
        - folder: /var/lib/mysql
          options: rw
----

|===

== Supported services

=== link:embedded-mariadb/README.adoc[embedded-mariadb]

=== link:embedded-couchbase/README.adoc[embedded-couchbase]

=== link:embedded-kafka/README.adoc[embedded-kafka]

=== link:embedded-rabbitmq/README.adoc[embedded-rabbitmq]

=== link:embedded-aerospike/README.adoc[embedded-aerospike]

=== link:embedded-memsql/README.adoc[embedded-memsql]

=== link:embedded-redis/README.adoc[embedded-redis]

=== link:embedded-neo4j/README.adoc[embedded-neo4j]

=== link:embedded-postgresql/README.adoc[embedded-postgresql]

=== link:embedded-elasticsearch/README.adoc[embedded-elasticsearch]

=== link:embedded-opensearch/README.adoc[embedded-opensearch]

=== link:embedded-dynamodb/README.adoc[embedded-dynamodb]

=== link:embedded-voltdb/README.adoc[embedded-voltdb]

=== link:embedded-minio/README.adoc[embedded-minio]

=== link:embedded-mongodb/README.adoc[embedded-mongodb]

=== link:embedded-google-pubsub/README.adoc[embedded-google-pubsub]

=== link:embedded-google-storage/README.adoc[embedded-google-storage]

=== link:embedded-keycloak/README.adoc[embedded-keycloak]

=== link:embedded-keydb/README.adoc[embedded-keydb]

=== link:embedded-influxdb/README.adoc[embedded-influxdb]

=== link:embedded-vault/README.adoc[embedded-vault]

=== link:embedded-oracle-xe/README.adoc[embedded-oracle-xe]

=== link:embedded-mysql/README.adoc[embedded-mysql]

=== link:embedded-localstack/README.adoc[embedded-localstack]

=== link:embedded-cassandra/README.adoc[embedded-cassandra]

=== link:embedded-clickhouse/README.adoc[embedded-clickhouse]

=== link:embedded-pulsar/README.adoc[embedded-pulsar]

=== link:embedded-vertica/README.adoc[embedded-vertica]

=== link:embedded-prometheus/README.adoc[embedded-prometheus]

=== link:embedded-grafana/README.adoc[embedded-grafana]

=== link:embedded-consul/README.adoc[embedded-consul]

=== link:embedded-artifactory/README.adoc[embedded-artifactory]

=== link:embedded-azurite/README.adoc[embedded-azurite]

=== link:embedded-toxiproxy/README.adoc[embedded-toxiproxy]

=== link:embedded-nats/README.adoc[embedded-nats]

=== link:embedded-k3s/README.adoc[embedded-k3s]

=== link:embedded-mockserver/README.adoc[embedded-mockserver]

=== link:embedded-solr/README.adoc[embedded-solr]

=== link:embedded-cockroachdb/README.adoc[embedded-cockroachdb]

=== link:embedded-git/README.adoc[embedded-git]

=== link:embedded-wiremock/README.adoc[embedded-wiremock]

=== link:embedded-mailhog/README.adoc[embedded-mailhog]

=== link:embedded-spicedb/README.adoc[embedded-spicedb]

== How to contribute

=== Flow

* You need to fork project and create branch from `develop`
* You do not need to update project version in `pom.xml` files, this will be done by release job
* Once finished - create pull request to `develop` from your fork, pass review and wait for merge
* On release, ci job will update to next release version + publish artifacts to the Maven Central

=== Checklist for contributing new module

* Naming/formatting patterns match existing code
* Test for success scenario
* Test for negative scenario (autoconfiguration is disabled via properties). https://spring.io/blog/2018/03/07/testing-auto-configurations-with-spring-boot-2-0[How to test autoconfiguration]
* Add new module to `testcontainers-spring-boot-bom`
* Module provides documentation in `README.adoc` and this documentation is included in parent `README.adoc` (see an example in already existing modules). Documentation should include:
** maven module declaration
** consumed properties
** produced properties
** notes (if applicable)
** example of usage

== Release
//* Release build is done using https://github.com/aleksandr-m/gitflow-maven-plugin[gitflow-maven-plugin]
* Release is done per each major change, critical bug
* Release can be done by contributor request
* Contacts to start release:
** mailto:sstus@playtika.com[sstus@playtika.com]
** mailto:iyova@playtika.com[iyova@playtika.com]
** mailto:admitrov@playtika.com[admitrov@playtika.com]


<!--
  ~ Licensed to the Apache Software Foundation (ASF) under one or more
  ~ contributor license agreements.  See the NOTICE file distributed with
  ~ this work for additional information regarding copyright ownership.
  ~ The ASF licenses this file to You under the Apache License, Version 2.0
  ~ (the "License"); you may not use this file except in compliance with
  ~ the License.  You may obtain a copy of the License at
  ~
  ~    http://www.apache.org/licenses/LICENSE-2.0
  ~
  ~ Unless required by applicable law or agreed to in writing, software
  ~ distributed under the License is distributed on an "AS IS" BASIS,
  ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
  ~ See the License for the specific language governing permissions and
  ~ limitations under the License.
  -->

# Apache Uniffle (Incubating)

Uniffle is a high performance, general purpose remote shuffle service for distributed computing engines.
It provides the ability to push shuffle data into centralized storage service,
changing the shuffle style from "local file pull-like style" to "remote block push-like style".
It brings in several advantages like supporting disaggregated storage deployment,
super large shuffle jobs, and high elasticity.
Currently it supports [Apache Spark][1], [Apache Hadoop MapReduce][2] and [Apache Tez][3].

[1]: https://spark.apache.org
[2]: https://hadoop.apache.org
[3]: https://tez.apache.org

[![Build](https://github.com/apache/incubator-uniffle/actions/workflows/build.yml/badge.svg?branch=master&event=push)](https://github.com/apache/incubator-uniffle/actions/workflows/build.yml)
[![Codecov](https://codecov.io/gh/apache/incubator-uniffle/branch/master/graph/badge.svg)](https://codecov.io/gh/apache/incubator-uniffle)
[![License](https://img.shields.io/github/license/apache/incubator-uniffle)](https://github.com/apache/incubator-uniffle/blob/master/LICENSE)
[![Release](https://img.shields.io/github/v/release/apache/incubator-uniffle)](https://github.com/apache/incubator-uniffle/releases)
[![Slack](https://img.shields.io/badge/chat-on%20Slack-brightgreen.svg)](https://join.slack.com/t/the-asf/shared_invite/zt-1fm9561yr-uzTpjqg3jf5nxSJV5AE3KQ)

## Architecture
![Rss Architecture](docs/asset/rss_architecture.png)
Uniffle cluster consists of three components, a coordinator cluster, a shuffle server cluster and an optional remote storage (e.g., HDFS).

Coordinator will collect the status of shuffle servers and assign jobs based on some strategy.

Shuffle server will receive the shuffle data, merge them and write to storage.

Depending on different situations, Uniffle supports Memory & Local, Memory & Remote Storage(e.g., HDFS), Memory & Local & Remote Storage(recommendation for production environment).

## Shuffle Process with Uniffle

* Spark driver ask coordinator to get shuffle server for shuffle process
* Spark task write shuffle data to shuffle server with following step:
![Rss Shuffle_Write](docs/asset/rss_shuffle_write.png)
 1. Send KV data to buffer
 2. Flush buffer to queue when buffer is full or buffer manager is full
 3. Thread pool get data from queue
 4. Request memory from shuffle server first and send the shuffle data
 5. Shuffle server cache data in memory first and flush to queue when buffer manager is full
 6. Thread pool get data from queue
 7. Write data to storage with index file and data file
 8. After write data, task report all blockId to shuffle server, this step is used for data validation later
 9. Store taskAttemptId in MapStatus to support Spark speculation

* Depending on different storage types, the spark task will read shuffle data from shuffle server or remote storage or both of them.

## Shuffle file format
The shuffle data is stored with index file and data file. Data file has all blocks for a specific partition and the index file has metadata for every block.

![Rss Shuffle_Write](docs/asset/rss_data_format.png)

## Supported Spark Version
Currently supports Spark 2.3.x, Spark 2.4.x, Spark 3.0.x, Spark 3.1.x, Spark 3.2.x, Spark 3.3.x, Spark 3.4.x, Spark 3.5.x

Note: To support dynamic allocation, the patch(which is included in patch/spark folder) should be applied to Spark

## Supported MapReduce Version
Currently supports the MapReduce framework of Hadoop 2.8.5, Hadoop 3.2.1

## Building Uniffle
> note: currently Uniffle requires JDK 1.8 to build, adding later JDK support is on our roadmap.

Uniffle is built using [Apache Maven](https://maven.apache.org/).
To build it, run:

    ./mvnw -DskipTests clean package

To fix code style issues, run:

    ./mvnw spotless:apply -Pspark3 -Pspark2 -Ptez -Pmr -Phadoop2.8 -Pdashboard

Build against profile Spark 2 (2.4.6)

    ./mvnw -DskipTests clean package -Pspark2

Build against profile Spark 3 (3.1.2)

    ./mvnw -DskipTests clean package -Pspark3

Build against Spark 3.2.x, Except 3.2.0

    ./mvnw -DskipTests clean package -Pspark3.2

Build against Spark 3.2.0

    ./mvnw -DskipTests clean package -Pspark3.2.0

Build against Hadoop MapReduce 2.8.5

    ./mvnw -DskipTests clean package -Pmr,hadoop2.8

Build against Hadoop MapReduce 3.2.1

    ./mvnw -DskipTests clean package -Pmr,hadoop3.2

Build against Tez 0.9.1

    ./mvnw -DskipTests clean package -Ptez

Build against Tez 0.9.1 and Hadoop 3.2.1

    ./mvnw -DskipTests clean package -Ptez,hadoop3.2

Build with dashboard

    ./mvnw -DskipTests clean package -Pdashboard

To package the Uniffle, run:

    ./build_distribution.sh

Package against Spark 3.2.x, Except 3.2.0, run:

    ./build_distribution.sh --spark3-profile 'spark3.2'

Package against Spark 3.2.0, run:

    ./build_distribution.sh --spark3-profile 'spark3.2.0'

Package will build against Hadoop 2.8.5 in default. If you want to build package against Hadoop 3.2.1, run:

    ./build_distribution.sh --hadoop-profile 'hadoop3.2'

Package with hadoop jars, If you want to build package against Hadoop 3.2.1, run:

    ./build_distribution.sh --hadoop-profile 'hadoop3.2' -Phadoop-dependencies-included

rss-xxx.tgz will be generated for deployment

## Deploy

If you have packaged tgz with hadoop jars, the env of `HADOOP_HOME` is needn't specified in `rss-env.sh`.

### Deploy Coordinator

1. unzip package to RSS_HOME
2. update RSS_HOME/bin/rss-env.sh, e.g.,
   ```
     JAVA_HOME=<java_home>
     HADOOP_HOME=<hadoop home>
     XMX_SIZE="16g"
   ```
3. update RSS_HOME/conf/coordinator.conf, e.g.,
   ```
     rss.rpc.server.port 19999
     rss.jetty.http.port 19998
     rss.coordinator.server.heartbeat.timeout 30000
     rss.coordinator.app.expired 60000
     rss.coordinator.shuffle.nodes.max 5
     # enable dynamicClientConf, and coordinator will be responsible for most of client conf
     rss.coordinator.dynamicClientConf.enabled true
     # config the path of client conf
     rss.coordinator.dynamicClientConf.path <RSS_HOME>/conf/dynamic_client.conf
     # config the path of excluded shuffle server
     rss.coordinator.exclude.nodes.file.path <RSS_HOME>/conf/exclude_nodes
   ```
4. update <RSS_HOME>/conf/dynamic_client.conf, rss client will get default conf from coordinator e.g.,
   ```
    # MEMORY_LOCALFILE_HDFS is recommended for production environment
    rss.storage.type MEMORY_LOCALFILE_HDFS
    # multiple remote storages are supported, and client will get assignment from coordinator
    rss.coordinator.remote.storage.path hdfs://cluster1/path,hdfs://cluster2/path
    rss.writer.require.memory.retryMax 1200
    rss.client.retry.max 50
    rss.client.send.check.timeout.ms 600000
    rss.client.read.buffer.size 14m
   ```
5. start Coordinator
   ```
    bash RSS_HOME/bin/start-coordnator.sh
   ```

### Deploy Shuffle Server
We recommend to use JDK 11+ if we want to have better performance when we deploy the shuffle server.
Some benchmark tests among different JDK is as below:
(using spark to write shuffle data with 20 executors. Single executor will total write 1G, and each time write 14M.
Shuffle Server use GRPC to transfer data)

| Java version | ShuffleServer GC  | Max pause time | ThroughOutput |
| ------------- | ------------- | ------------- | ------------- |
| 8  | G1  | 30s | 0.3 |
| 11  | G1  | 2.5s | 0.8 |
| 18  | G1  | 2.5s | 0.8 |
| 18  | ZGC  | 0.2ms | 0.99997 |

Deploy Steps:
1. unzip package to RSS_HOME
2. update RSS_HOME/bin/rss-env.sh, e.g.,
   ```
     JAVA_HOME=<java_home>
     HADOOP_HOME=<hadoop home>
     XMX_SIZE="80g"
   ```
3. update RSS_HOME/conf/server.conf, e.g.,
   ```
     rss.rpc.server.port 19999
     rss.jetty.http.port 19998
     rss.rpc.executor.size 2000
     # it should be configured the same as in coordinator
     rss.storage.type MEMORY_LOCALFILE_HDFS
     rss.coordinator.quorum <coordinatorIp1>:19999,<coordinatorIp2>:19999
     # local storage path for shuffle server
     rss.storage.basePath /data1/rssdata,/data2/rssdata....
     # it's better to config thread num according to local disk num
     rss.server.flush.thread.alive 5
     rss.server.flush.localfile.threadPool.size 10
     rss.server.flush.hadoop.threadPool.size 60
     rss.server.buffer.capacity 40g
     rss.server.read.buffer.capacity 20g
     rss.server.heartbeat.interval 10000
     rss.rpc.message.max.size 1073741824
     rss.server.preAllocation.expired 120000
     rss.server.commit.timeout 600000
     rss.server.app.expired.withoutHeartbeat 120000
     # note: the default value of rss.server.flush.cold.storage.threshold.size is 64m
     # there will be no data written to DFS if set it as 100g even rss.storage.type=MEMORY_LOCALFILE_HDFS
     # please set a proper value if DFS is used, e.g., 64m, 128m.
     rss.server.flush.cold.storage.threshold.size 100g
   ```
4. start Shuffle Server
   ```
    bash RSS_HOME/bin/start-shuffle-server.sh
   ```

### Deploy Spark Client
1. Add client jar to Spark classpath, e.g., SPARK_HOME/jars/

   The jar for Spark2 is located in <RSS_HOME>/jars/client/spark2/rss-client-spark2-shaded-${version}.jar

   The jar for Spark3 is located in <RSS_HOME>/jars/client/spark3/rss-client-spark3-shaded-${version}.jar

2. Update Spark conf to enable Uniffle, e.g.,

   ```
   # Uniffle transmits serialized shuffle data over network, therefore a serializer that supports relocation of
   # serialized object should be used.
   spark.serializer org.apache.spark.serializer.KryoSerializer # this could also be in the spark-defaults.conf
   spark.shuffle.manager org.apache.spark.shuffle.RssShuffleManager
   spark.rss.coordinator.quorum <coordinatorIp1>:19999,<coordinatorIp2>:19999
   # Note: For Spark2, spark.sql.adaptive.enabled should be false because Spark2 doesn't support AQE.
   ```

### Support Spark dynamic allocation

To support spark dynamic allocation with Uniffle, spark code should be updated.
There are 7 patches for spark (2.3.4/2.4.6/3.0.1/3.1.2/3.2.1/3.3.1/3.4.1) in patch/spark folder for reference.

After apply the patch and rebuild spark, add following configuration in spark conf to enable dynamic allocation:
  ```
  spark.shuffle.service.enabled false
  spark.dynamicAllocation.enabled true
  ```
For spark3.5 or above just add one more configuration:
  ```
  spark.shuffle.sort.io.plugin.class org.apache.spark.shuffle.RssShuffleDataIo
  ```

### Deploy MapReduce Client

1. Add client jar to the classpath of each NodeManager, e.g., <HADOOP>/share/hadoop/mapreduce/

The jar for MapReduce is located in <RSS_HOME>/jars/client/mr/rss-client-mr-XXXXX-shaded.jar

2. Update MapReduce conf to enable Uniffle, e.g.,

   ```
   -Dmapreduce.rss.coordinator.quorum=<coordinatorIp1>:19999,<coordinatorIp2>:19999
   -Dyarn.app.mapreduce.am.command-opts=org.apache.hadoop.mapreduce.v2.app.RssMRAppMaster
   -Dmapreduce.job.map.output.collector.class=org.apache.hadoop.mapred.RssMapOutputCollector
   -Dmapreduce.job.reduce.shuffle.consumer.plugin.class=org.apache.hadoop.mapreduce.task.reduce.RssShuffle
   ```
Note that the RssMRAppMaster will automatically disable slow start (i.e., `mapreduce.job.reduce.slowstart.completedmaps=1`)
and job recovery (i.e., `yarn.app.mapreduce.am.job.recovery.enable=false`)

### Deploy Tez Client

1. Append client jar to pacakge which is set by 'tez.lib.uris'.

In production mode, you can append client jar (rss-client-tez-XXXXX-shaded.jar) to package which is set by 'tez.lib.uris'.

In development mode, you can append client jar (rss-client-tez-XXXXX-shaded.jar) to HADOOP_CLASSPATH.

2. Update tez-site.xml to enable Uniffle.

| Property Name              |Default| Description                  |
|----------------------------|---|------------------------------|
| tez.am.launch.cmd-opts     |-XX:+PrintGCDetails -verbose:gc -XX:+PrintGCTimeStamps -XX:+UseNUMA -XX:+UseParallelGC org.apache.tez.dag.app.RssDAGAppMaster| enable remote shuffle service |
| tez.rss.coordinator.quorum |coordinatorIp1:19999,coordinatorIp2:19999|coordinator address|

Note that the RssDAGAppMaster will automatically disable slow start (i.e., `tez.shuffle-vertex-manager.min-src-fraction=1`, `tez.shuffle-vertex-manager.max-src-fraction=1`).

### Deploy In Kubernetes

We have provided an operator for deploying uniffle in kubernetes environments.

For details, see the following document:

[operator docs](docs/operator)

## Configuration

The important configuration is listed as follows.

|Role|Link|
|---|---|
|coordinator|[Uniffle Coordinator Guide](https://github.com/apache/incubator-uniffle/blob/master/docs/coordinator_guide.md)|
|shuffle server|[Uniffle Shuffle Server Guide](https://github.com/apache/incubator-uniffle/blob/master/docs/server_guide.md)|
|client|[Uniffle Shuffle Client Guide](https://github.com/apache/incubator-uniffle/blob/master/docs/client_guide/client_guide.md)|

## Security: Hadoop kerberos authentication
The primary goals of the Uniffle Kerberos security are:
1. to enable secure data access for coordinator/shuffle-servers, like dynamic conf/exclude-node files stored in secured dfs cluster
2. to write shuffle data to kerberos secured dfs cluster for shuffle-servers.

The following security configurations are introduced.

|Property Name|Default|Description|
|---|---|---|
|rss.security.hadoop.kerberos.enable|false|Whether enable access secured hadoop cluster|
|rss.security.hadoop.kerberos.krb5-conf.file|-|The file path of krb5.conf. And only when rss.security.hadoop.kerberos.enable is enabled, the option will be valid|
|rss.security.hadoop.kerberos.keytab.file|-|The kerberos keytab file path. And only when rss.security.hadoop.kerberos.enable is enabled, the option will be valid|
|rss.security.hadoop.kerberos.principal|-|The kerberos keytab principal. And only when rss.security.hadoop.kerberos.enable is enabled, the option will be valid|
|rss.security.hadoop.kerberos.relogin.interval.sec|60|The kerberos authentication relogin interval. unit: sec|
|rss.security.hadoop.kerberos.proxy.user.enable|true|Whether using proxy user for job user to access secured Hadoop cluster.|

* The proxy user mechanism is used to keep the data isolation in uniffle, which means the shuffle-data written by
  shuffle-servers is owned by spark app's user. To achieve the this, the login user specified by above config should
  be as the superuser for HDFS. For more details of related sections,
  please see [Proxy user - Superusers Acting On Behalf Of Other Users](https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-common/Superusers.html)

## Benchmark
We provide some benchmark tests for Uniffle. For details, you can see [Uniffle 0.2.0 Benchmark](docs/benchmark.md), [Uniffle 0.9.0 Benchmark](docs/benchmark_netty_case_report.md).

## LICENSE

Uniffle is under the Apache License Version 2.0. See the [LICENSE](https://github.com/apache/incubator-uniffle/blob/master/LICENSE) file for details.

## Contributing
For more information about contributing issues or pull requests, see [Uniffle Contributing Guide](https://github.com/apache/incubator-uniffle/blob/master/CONTRIBUTING.md).