[KYUUBI #3473] Add Docker Compose based Kyuubi Playground

### _Why are the changes needed?_ Add [Docker Compose](https://docs.docker.com/compose/)-based Kyuubi Playground, w/ basic docs. - Only support Spark engine for now. - The images are not part of the official apache binary release, so we don't have license issue. - The images will be published to the docker hub under `nekyuubi` namespace, which is managed by one of the PPMC members, but not the whole PPMC. - The images support both x86 & arm platforms. ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [x] Add screenshots for manual tests if appropriate <img width="1408" alt="image" src="https://user-images.githubusercontent.com/26535726/189531044-50812bc6-43d1-43ab-8ec7-fb160ad805b3.png"> - [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #3473 from pan3793/play. Closes #3473 47b6c753 [Cheng Pan] doc 0325b646 [Cheng Pan] license header 89b07907 [Cheng Pan] incremental 76dc08ec [Cheng Pan] nit a0a58915 [Cheng Pan] docs 763a2240 [Cheng Pan] arg 25a9bfd1 [Cheng Pan] nit 11d92c16 [Cheng Pan] Docker Compose-based Kyuubi Playground Authored-by: Cheng Pan <chengpan@apache.org> Signed-off-by: Cheng Pan <chengpan@apache.org>
2022-09-13 15:00:29 +00:00 · 2022-09-13 15:00:29 +00:00 · 5dd245dfa1
commit 5dd245dfa1
parent e8622e6bd7
15 changed files with 2214 additions and 0 deletions
--- a/docker/playground/.env
+++ b/docker/playground/.env
@ -0,0 +1,30 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+AWS_JAVA_SDK_VERSION=1.12.239
+HADOOP_VERSION=3.3.1
+HIVE_VERSION=2.3.9
+ICEBERG_VERSION=0.14.1
+KYUUBI_VERSION=1.6.0-incubating
+KYUUBI_HADOOP_VERSION=3.3.4
+POSTGRES_VERSION=12
+POSTGRES_JDBC_VERSION=42.3.4
+SCALA_BINARY_VERSION=2.12
+SPARK_VERSION=3.3.0
+SPARK_BINARY_VERSION=3.3
+SPARK_HADOOP_VERSION=3.3.2
+ZOOKEEPER_VERSION=3.6.3
--- a/docker/playground/README.md
+++ b/docker/playground/README.md
@ -0,0 +1,41 @@
+Playground
+===
+
+## For Users
+
+### Setup
+
+1. Install [Docker](https://docs.docker.com/get-docker/) and [Docker Compose](https://docs.docker.com/compose/);
+2. Go to `docker/playground`, and use `docker compose up` to start compose services in the foreground, or use `docker compose up -d` to run compose services as daemon;
+
+### Play
+
+1. Connect using `beeline`
+
+`docker exec -it kyuubi /opt/kyuubi/bin/beeline -u 'jdbc:hive2://0.0.0.0:10009/'`;
+
+2. Connect using DBeaver
+
+Add a Kyuubi datasource with
+
+- connection url `jdbc:hive2://0.0.0.0:10009/`
+- username: `anonymous`
+- password: `<empty>`
+
+### Access Service
+
+- MinIO: http://localhost:9001
+- PostgreSQL localhost:5432 (username: postgres, password: postgres)
+- Spark UI: http://localhost:4040 (available after Spark application launching by Kyuubi, port may be 4041, 4042... if you launch more than one Spark applications)
+
+### Shutdown
+
+1. Stop compose services by pressing `CTRL+C` if they are running on the foreground, or by `docker compose down` if they are running as daemon;
+2. Remove the stopped containers `docker compose rm`;
+
+## For Maintainers
+
+### Build
+
+1. Build images `docker/playground/build-image.sh`;
+2. Optional to use `buildx` to build and publish cross-platform images `BUILDX=1 docker/playground/build-image.sh`;
--- a/docker/playground/build-image.sh
+++ b/docker/playground/build-image.sh
@ -0,0 +1,87 @@
+#!/usr/bin/env bash
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+# Set Mirror:
+# APACHE_MIRROR=mirrors.cloud.tencent.com/apache
+# MAVEN_MIRROR=mirrors.cloud.tencent.com/maven
+
+set -e
+
+
+APACHE_MIRROR=${APACHE_MIRROR:-https://dlcdn.apache.org}
+MAVEN_MIRROR=${MAVEN_MIRROR:-https://repo1.maven.org/maven2}
+BUILD_CMD="docker build"
+
+if [ $BUILDX ]; then
+  echo "Using buildx to build cross-platform images"
+  BUILD_CMD="docker buildx build --platform=linux/amd64,linux/arm64 --push"
+fi
+
+SELF_DIR="$(cd "$(dirname "$0")"; pwd)"
+
+source "${SELF_DIR}/.env"
+
+${BUILD_CMD} \
+  --build-arg APACHE_MIRROR=${APACHE_MIRROR} \
+  --build-arg MAVEN_MIRROR=${MAVEN_MIRROR} \
+  --build-arg KYUUBI_VERSION=${KYUUBI_VERSION} \
+  --file "${SELF_DIR}/image/kyuubi-playground-base.Dockerfile" \
+  --tag nekyuubi/kyuubi-playground-base:${KYUUBI_VERSION} \
+  "${SELF_DIR}/image" $@
+
+${BUILD_CMD} \
+  --build-arg APACHE_MIRROR=${APACHE_MIRROR} \
+  --build-arg MAVEN_MIRROR=${MAVEN_MIRROR} \
+  --build-arg KYUUBI_VERSION=${KYUUBI_VERSION} \
+  --build-arg AWS_JAVA_SDK_VERSION=${AWS_JAVA_SDK_VERSION} \
+  --build-arg HADOOP_VERSION=${HADOOP_VERSION} \
+  --file "${SELF_DIR}/image/kyuubi-playground-hadoop.Dockerfile" \
+  --tag nekyuubi/kyuubi-playground-hadoop:${KYUUBI_VERSION} \
+  "${SELF_DIR}/image" $@
+
+${BUILD_CMD} \
+  --build-arg APACHE_MIRROR=${APACHE_MIRROR} \
+  --build-arg MAVEN_MIRROR=${MAVEN_MIRROR} \
+  --build-arg KYUUBI_VERSION=${KYUUBI_VERSION} \
+  --build-arg HIVE_VERSION=${HIVE_VERSION} \
+  --file "${SELF_DIR}/image/kyuubi-playground-metastore.Dockerfile" \
+  --tag nekyuubi/kyuubi-playground-metastore:${KYUUBI_VERSION} \
+  "${SELF_DIR}/image" $@
+
+${BUILD_CMD} \
+  --build-arg APACHE_MIRROR=${APACHE_MIRROR} \
+  --build-arg MAVEN_MIRROR=${MAVEN_MIRROR} \
+  --build-arg KYUUBI_VERSION=${KYUUBI_VERSION} \
+  --build-arg AWS_JAVA_SDK_VERSION=${AWS_JAVA_SDK_VERSION} \
+  --build-arg CLICKHOUSE_JDBC_VERSION=${CLICKHOUSE_JDBC_VERSION} \
+  --build-arg SPARK_HADOOP_VERSION=${SPARK_HADOOP_VERSION} \
+  --build-arg ICEBERG_VERSION=${ICEBERG_VERSION} \
+  --build-arg POSTGRES_JDBC_VERSION=${POSTGRES_JDBC_VERSION} \
+  --build-arg SCALA_BINARY_VERSION=${SCALA_BINARY_VERSION} \
+  --build-arg SPARK_VERSION=${SPARK_VERSION} \
+  --build-arg SPARK_BINARY_VERSION=${SPARK_BINARY_VERSION} \
+  --file "${SELF_DIR}/image/kyuubi-playground-spark.Dockerfile" \
+  --tag nekyuubi/kyuubi-playground-spark:${KYUUBI_VERSION} \
+  "${SELF_DIR}/image" $@
+
+${BUILD_CMD} \
+  --build-arg APACHE_MIRROR=${APACHE_MIRROR} \
+  --build-arg MAVEN_MIRROR=${MAVEN_MIRROR} \
+  --build-arg KYUUBI_VERSION=${KYUUBI_VERSION} \
+  --build-arg AWS_JAVA_SDK_VERSION=${AWS_JAVA_SDK_VERSION} \
+  --build-arg KYUUBI_HADOOP_VERSION=${KYUUBI_HADOOP_VERSION} \
+  --file "${SELF_DIR}/image/kyuubi-playground-kyuubi.Dockerfile" \
+  --tag nekyuubi/kyuubi-playground-kyuubi:${KYUUBI_VERSION} \
+  "${SELF_DIR}/image" $@
--- a/docker/playground/compose.yml
+++ b/docker/playground/compose.yml
@ -0,0 +1,79 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+services:
+  minio:
+    image: alekcander/bitnami-minio-multiarch:RELEASE.2022-05-26T05-48-41Z
+    environment:
+      MINIO_ROOT_USER: minio
+      MINIO_ROOT_PASSWORD: minio_minio
+      MINIO_DEFAULT_BUCKETS: spark-bucket,iceberg-bucket
+    container_name: minio
+    hostname: minio
+    ports:
+      - 9000
+      - 9001:9001
+
+  postgres:
+    image: postgres:${POSTGRES_VERSION}
+    environment:
+      POSTGRES_USER: postgres
+      POSTGRES_PASSWORD: postgres
+      POSTGRES_DB: metastore
+    container_name: postgres
+    hostname: postgres
+    ports:
+      - 5432:5432
+    volumes:
+      - ./image/hive-schema-2.3.0.postgres.sql:/docker-entrypoint-initdb.d/hive-schema-2.3.0.postgres.sql
+      - ./image/hive-txn-schema-2.3.0.postgres.sql:/docker-entrypoint-initdb.d/hive-txn-schema-2.3.0.postgres.sql
+
+  zookeeper:
+    image: zookeeper:${ZOOKEEPER_VERSION}
+    ports:
+      - 2181
+    container_name: zookeeper
+    hostname: zookeeper
+
+  metastore:
+    image: nekyuubi/kyuubi-playground-metastore:${KYUUBI_VERSION}
+    container_name: metastore
+    hostname: metastore
+    ports:
+      - 9083
+    volumes:
+      - ./image/core-site.xml:/etc/hadoop/conf/core-site.xml
+      - ./image/hive-site.xml:/etc/hive/conf/hive-site.xml
+    depends_on:
+      - postgres
+
+  kyuubi:
+    image: nekyuubi/kyuubi-playground-kyuubi:${KYUUBI_VERSION}
+    container_name: kyuubi
+    hostname: kyuubi
+    ports:
+      - 4040-4050:4040-4050
+      - 10009:10009
+    volumes:
+      - ./image/core-site.xml:/etc/hadoop/conf/core-site.xml
+      - ./image/hive-site.xml:/etc/hive/conf/hive-site.xml
+      - ./image/spark-defaults.conf:/etc/spark/conf/spark-defaults.conf
+      - ./image/kyuubi-defaults.conf:/etc/kyuubi/conf/kyuubi-defaults.conf
+    depends_on:
+      - metastore
+      - minio
+      - zookeeper
--- a/docker/playground/image/core-site.xml
+++ b/docker/playground/image/core-site.xml
@ -0,0 +1,48 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!--
+  ~ Licensed to the Apache Software Foundation (ASF) under one or more
+  ~ contributor license agreements.  See the NOTICE file distributed with
+  ~ this work for additional information regarding copyright ownership.
+  ~ The ASF licenses this file to You under the Apache License, Version 2.0
+  ~ (the "License"); you may not use this file except in compliance with
+  ~ the License.  You may obtain a copy of the License at
+  ~
+  ~     http://www.apache.org/licenses/LICENSE-2.0
+  ~
+  ~ Unless required by applicable law or agreed to in writing, software
+  ~ distributed under the License is distributed on an "AS IS" BASIS,
+  ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+  ~ See the License for the specific language governing permissions and
+  ~ limitations under the License.
+  -->
+<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
+<configuration>
+    <property>
+        <name>fs.s3a.access.key</name>
+        <value>minio</value>
+    </property>
+    <property>
+        <name>fs.s3a.secret.key</name>
+        <value>minio_minio</value>
+    </property>
+    <property>
+        <name>fs.s3a.connection.ssl.enabled</name>
+        <value>false</value>
+    </property>
+    <property>
+        <name>fs.s3a.path.style.access</name>
+        <value>true</value>
+    </property>
+    <property>
+        <name>fs.s3a.endpoint</name>
+        <value>http://minio:9000</value>
+    </property>
+    <property>
+        <name>fs.s3a.committer.name</name>
+        <value>magic</value>
+    </property>
+    <property>
+        <name>fs.defaultFS</name>
+        <value>s3a://spark-bucket/</value>
+    </property>
+</configuration>
--- a/docker/playground/image/hive-schema-2.3.0.postgres.sql
+++ b/docker/playground/image/hive-schema-2.3.0.postgres.sql
--- a/docker/playground/image/hive-site.xml
+++ b/docker/playground/image/hive-site.xml
@ -0,0 +1,45 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!--
+  ~ Licensed to the Apache Software Foundation (ASF) under one or more
+  ~ contributor license agreements.  See the NOTICE file distributed with
+  ~ this work for additional information regarding copyright ownership.
+  ~ The ASF licenses this file to You under the Apache License, Version 2.0
+  ~ (the "License"); you may not use this file except in compliance with
+  ~ the License.  You may obtain a copy of the License at
+  ~
+  ~     http://www.apache.org/licenses/LICENSE-2.0
+  ~
+  ~ Unless required by applicable law or agreed to in writing, software
+  ~ distributed under the License is distributed on an "AS IS" BASIS,
+  ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+  ~ See the License for the specific language governing permissions and
+  ~ limitations under the License.
+  -->
+<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
+<configuration>
+    <property>
+        <name>javax.jdo.option.ConnectionURL</name>
+        <value>jdbc:postgresql://postgres:5432/metastore</value>
+    </property>
+    <property>
+        <name>javax.jdo.option.ConnectionDriverName</name>
+        <value>org.postgresql.Driver</value>
+    </property>
+    <property>
+        <name>javax.jdo.option.ConnectionUserName</name>
+        <value>postgres</value>
+    </property>
+    <property>
+        <name>javax.jdo.option.ConnectionPassword</name>
+        <value>postgres</value>
+    </property>
+    <property>
+        <name>hive.metastore.warehouse.dir</name>
+        <value>s3a://spark-bucket/warehouse</value>
+        <description>location of default database for the warehouse</description>
+    </property>
+    <property>
+        <name>hive.metastore.uris</name>
+        <value>thrift://metastore:9083</value>
+    </property>
+</configuration>
--- a/docker/playground/image/hive-txn-schema-2.3.0.postgres.sql
+++ b/docker/playground/image/hive-txn-schema-2.3.0.postgres.sql
@ -0,0 +1,133 @@
+-- Licensed to the Apache Software Foundation (ASF) under one or more
+-- contributor license agreements.  See the NOTICE file distributed with
+-- this work for additional information regarding copyright ownership.
+-- The ASF licenses this file to You under the Apache License, Version 2.0
+-- (the "License"); you may not use this file except in compliance with
+-- the License.  You may obtain a copy of the License at
+--
+--     http://www.apache.org/licenses/LICENSE-2.0
+--
+-- Unless required by applicable law or agreed to in writing, software
+-- distributed under the License is distributed on an "AS IS" BASIS,
+-- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+-- See the License for the specific language governing permissions and
+-- limitations under the License.
+
+--
+-- Tables for transaction management
+-- 
+
+CREATE TABLE TXNS (
+  TXN_ID bigint PRIMARY KEY,
+  TXN_STATE char(1) NOT NULL,
+  TXN_STARTED bigint NOT NULL,
+  TXN_LAST_HEARTBEAT bigint NOT NULL,
+  TXN_USER varchar(128) NOT NULL,
+  TXN_HOST varchar(128) NOT NULL,
+  TXN_AGENT_INFO varchar(128),
+  TXN_META_INFO varchar(128),
+  TXN_HEARTBEAT_COUNT integer
+);
+
+CREATE TABLE TXN_COMPONENTS (
+  TC_TXNID bigint REFERENCES TXNS (TXN_ID),
+  TC_DATABASE varchar(128) NOT NULL,
+  TC_TABLE varchar(128),
+  TC_PARTITION varchar(767) DEFAULT NULL,
+  TC_OPERATION_TYPE char(1) NOT NULL
+);
+
+CREATE INDEX TC_TXNID_INDEX ON TXN_COMPONENTS USING hash (TC_TXNID);
+
+CREATE TABLE COMPLETED_TXN_COMPONENTS (
+  CTC_TXNID bigint,
+  CTC_DATABASE varchar(128) NOT NULL,
+  CTC_TABLE varchar(256),
+  CTC_PARTITION varchar(767)
+);
+
+CREATE TABLE NEXT_TXN_ID (
+  NTXN_NEXT bigint NOT NULL
+);
+INSERT INTO NEXT_TXN_ID VALUES(1);
+
+CREATE TABLE HIVE_LOCKS (
+  HL_LOCK_EXT_ID bigint NOT NULL,
+  HL_LOCK_INT_ID bigint NOT NULL,
+  HL_TXNID bigint,
+  HL_DB varchar(128) NOT NULL,
+  HL_TABLE varchar(128),
+  HL_PARTITION varchar(767) DEFAULT NULL,
+  HL_LOCK_STATE char(1) NOT NULL,
+  HL_LOCK_TYPE char(1) NOT NULL,
+  HL_LAST_HEARTBEAT bigint NOT NULL,
+  HL_ACQUIRED_AT bigint,
+  HL_USER varchar(128) NOT NULL,
+  HL_HOST varchar(128) NOT NULL,
+  HL_HEARTBEAT_COUNT integer,
+  HL_AGENT_INFO varchar(128),
+  HL_BLOCKEDBY_EXT_ID bigint,
+  HL_BLOCKEDBY_INT_ID bigint,
+  PRIMARY KEY(HL_LOCK_EXT_ID, HL_LOCK_INT_ID)
+); 
+
+CREATE INDEX HL_TXNID_INDEX ON HIVE_LOCKS USING hash (HL_TXNID);
+
+CREATE TABLE NEXT_LOCK_ID (
+  NL_NEXT bigint NOT NULL
+);
+INSERT INTO NEXT_LOCK_ID VALUES(1);
+
+CREATE TABLE COMPACTION_QUEUE (
+  CQ_ID bigint PRIMARY KEY,
+  CQ_DATABASE varchar(128) NOT NULL,
+  CQ_TABLE varchar(128) NOT NULL,
+  CQ_PARTITION varchar(767),
+  CQ_STATE char(1) NOT NULL,
+  CQ_TYPE char(1) NOT NULL,
+  CQ_TBLPROPERTIES varchar(2048),
+  CQ_WORKER_ID varchar(128),
+  CQ_START bigint,
+  CQ_RUN_AS varchar(128),
+  CQ_HIGHEST_TXN_ID bigint,
+  CQ_META_INFO bytea,
+  CQ_HADOOP_JOB_ID varchar(32)
+);
+
+CREATE TABLE NEXT_COMPACTION_QUEUE_ID (
+  NCQ_NEXT bigint NOT NULL
+);
+INSERT INTO NEXT_COMPACTION_QUEUE_ID VALUES(1);
+
+CREATE TABLE COMPLETED_COMPACTIONS (
+  CC_ID bigint PRIMARY KEY,
+  CC_DATABASE varchar(128) NOT NULL,
+  CC_TABLE varchar(128) NOT NULL,
+  CC_PARTITION varchar(767),
+  CC_STATE char(1) NOT NULL,
+  CC_TYPE char(1) NOT NULL,
+  CC_TBLPROPERTIES varchar(2048),
+  CC_WORKER_ID varchar(128),
+  CC_START bigint,
+  CC_END bigint,
+  CC_RUN_AS varchar(128),
+  CC_HIGHEST_TXN_ID bigint,
+  CC_META_INFO bytea,
+  CC_HADOOP_JOB_ID varchar(32)
+);
+
+CREATE TABLE AUX_TABLE (
+  MT_KEY1 varchar(128) NOT NULL,
+  MT_KEY2 bigint NOT NULL,
+  MT_COMMENT varchar(255),
+  PRIMARY KEY(MT_KEY1, MT_KEY2)
+);
+
+CREATE TABLE WRITE_SET (
+  WS_DATABASE varchar(128) NOT NULL,
+  WS_TABLE varchar(128) NOT NULL,
+  WS_PARTITION varchar(767),
+  WS_TXNID bigint NOT NULL,
+  WS_COMMIT_ID bigint NOT NULL,
+  WS_OPERATION_TYPE char(1) NOT NULL
+);
--- a/docker/playground/image/kyuubi-defaults.conf
+++ b/docker/playground/image/kyuubi-defaults.conf
@ -0,0 +1,31 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+## Kyuubi Configurations
+
+kyuubi.authentication=NONE
+kyuubi.frontend.bind.host=0.0.0.0
+kyuubi.frontend.bind.port=10009
+kyuubi.ha.zookeeper.quorum=zookeeper:2181
+kyuubi.session.engine.idle.timeout=PT5M
+kyuubi.operation.incremental.collect=true
+kyuubi.operation.progress.enabled=true
+
+kyuubi.engine.session.initialize.sql \
+      show namespaces in tpcds; \
+      show namespaces in tpch; \
+      show namespaces in postgres;
--- a/docker/playground/image/kyuubi-playground-base.Dockerfile
+++ b/docker/playground/image/kyuubi-playground-base.Dockerfile
@ -0,0 +1,23 @@
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     https://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+FROM eclipse-temurin:8-focal
+
+RUN set -x && \
+    ln -snf /usr/bin/bash /usr/bin/sh && \
+    apt-get update -q && \
+    apt-get install -yq retry busybox && \
+    rm -rf /var/lib/apt/lists/* && \
+    mkdir /opt/busybox && \
+    busybox --install /opt/busybox
+
+ENV PATH=/opt/java/openjdk/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/opt/busybox
--- a/docker/playground/image/kyuubi-playground-hadoop.Dockerfile
+++ b/docker/playground/image/kyuubi-playground-hadoop.Dockerfile
@ -0,0 +1,37 @@
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     https://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+ARG KYUUBI_VERSION
+
+FROM nekyuubi/kyuubi-playground-base:${KYUUBI_VERSION}
+
+ARG AWS_JAVA_SDK_VERSION
+ARG HADOOP_VERSION
+
+ARG APACHE_MIRROR
+ARG MAVEN_MIRROR
+
+ENV HADOOP_HOME=/opt/hadoop
+ENV HADOOP_CONF_DIR=/etc/hadoop/conf
+
+RUN set -x && \
+    if [ $(uname -m) = "aarch64" ]; then HADOOP_TAR_NAME=hadoop-${HADOOP_VERSION}-aarch64; else HADOOP_TAR_NAME=hadoop-${HADOOP_VERSION}; fi && \
+    wget -q ${APACHE_MIRROR}/hadoop/core/hadoop-${HADOOP_VERSION}/${HADOOP_TAR_NAME}.tar.gz && \
+    tar -xzf ${HADOOP_TAR_NAME}.tar.gz -C /opt && \
+    ln -s /opt/hadoop-${HADOOP_VERSION} ${HADOOP_HOME} && \
+    rm ${HADOOP_TAR_NAME}.tar.gz && \
+    HADOOP_CLOUD_STORAGE_JAR_NAME=hadoop-cloud-storage && \
+    wget -q ${MAVEN_MIRROR}/org/apache/hadoop/${HADOOP_CLOUD_STORAGE_JAR_NAME}/${HADOOP_VERSION}/${HADOOP_CLOUD_STORAGE_JAR_NAME}-${HADOOP_VERSION}.jar -P ${HADOOP_HOME}/share/hadoop/hdfs/lib && \
+    HADOOP_AWS_JAR_NAME=hadoop-aws && \
+    wget -q ${MAVEN_MIRROR}/org/apache/hadoop/${HADOOP_AWS_JAR_NAME}/${HADOOP_VERSION}/${HADOOP_AWS_JAR_NAME}-${HADOOP_VERSION}.jar -P ${HADOOP_HOME}/share/hadoop/hdfs/lib && \
+    AWS_JAVA_SDK_BUNDLE_JAR_NAME=aws-java-sdk-bundle && \
+    wget -q ${MAVEN_MIRROR}/com/amazonaws/${AWS_JAVA_SDK_BUNDLE_JAR_NAME}/${AWS_JAVA_SDK_VERSION}/${AWS_JAVA_SDK_BUNDLE_JAR_NAME}-${AWS_JAVA_SDK_VERSION}.jar -P ${HADOOP_HOME}/share/hadoop/hdfs/lib
--- a/docker/playground/image/kyuubi-playground-kyuubi.Dockerfile
+++ b/docker/playground/image/kyuubi-playground-kyuubi.Dockerfile
@ -0,0 +1,40 @@
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     https://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+ARG KYUUBI_VERSION
+
+FROM nekyuubi/kyuubi-playground-spark:${KYUUBI_VERSION}
+
+ARG AWS_JAVA_SDK_VERSION
+ARG KYUUBI_VERSION
+ARG KYUUBI_HADOOP_VERSION
+
+ARG APACHE_MIRROR
+ARG MAVEN_MIRROR
+
+ENV KYUUBI_HOME=/opt/kyuubi
+ENV KYUUBI_CONF_DIR=/etc/kyuubi/conf
+
+RUN set -x && \
+    wget -q ${APACHE_MIRROR}/incubator/kyuubi/kyuubi-${KYUUBI_VERSION}/apache-kyuubi-${KYUUBI_VERSION}-bin.tgz && \
+    tar -xzf apache-kyuubi-${KYUUBI_VERSION}-bin.tgz -C /opt && \
+    ln -s /opt/apache-kyuubi-${KYUUBI_VERSION}-bin ${KYUUBI_HOME} && \
+    rm apache-kyuubi-${KYUUBI_VERSION}-bin.tgz && \
+    HADOOP_CLOUD_STORAGE_JAR_NAME=hadoop-cloud-storage && \
+    wget -q ${MAVEN_MIRROR}/org/apache/hadoop/${HADOOP_CLOUD_STORAGE_JAR_NAME}/${KYUUBI_HADOOP_VERSION}/${HADOOP_CLOUD_STORAGE_JAR_NAME}-${KYUUBI_HADOOP_VERSION}.jar -P ${KYUUBI_HOME}/jars && \
+    HADOOP_AWS_JAR_NAME=hadoop-aws && \
+    wget -q ${MAVEN_MIRROR}/org/apache/hadoop/${HADOOP_AWS_JAR_NAME}/${KYUUBI_HADOOP_VERSION}/${HADOOP_AWS_JAR_NAME}-${KYUUBI_HADOOP_VERSION}.jar -P ${KYUUBI_HOME}/jars && \
+    AWS_JAVA_SDK_BUNDLE_JAR_NAME=aws-java-sdk-bundle && \
+    wget -q ${MAVEN_MIRROR}/com/amazonaws/${AWS_JAVA_SDK_BUNDLE_JAR_NAME}/${AWS_JAVA_SDK_VERSION}/${AWS_JAVA_SDK_BUNDLE_JAR_NAME}-${AWS_JAVA_SDK_VERSION}.jar -P ${KYUUBI_HOME}/jars && \
+    useradd anonymous
+
+ENTRYPOINT ["/opt/kyuubi/bin/kyuubi", "run"]
--- a/docker/playground/image/kyuubi-playground-metastore.Dockerfile
+++ b/docker/playground/image/kyuubi-playground-metastore.Dockerfile
@ -0,0 +1,30 @@
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     https://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+ARG KYUUBI_VERSION
+
+FROM nekyuubi/kyuubi-playground-hadoop:${KYUUBI_VERSION}
+
+ARG HIVE_VERSION
+
+ARG APACHE_MIRROR
+
+ENV HIVE_HOME=/opt/hive
+ENV HIVE_CONF_DIR=/etc/hive/conf
+
+RUN set -x && \
+    wget -q ${APACHE_MIRROR}/hive/hive-${HIVE_VERSION}/apache-hive-${HIVE_VERSION}-bin.tar.gz && \
+    tar -xzf apache-hive-${HIVE_VERSION}-bin.tar.gz -C /opt && \
+    ln -s /opt/apache-hive-${HIVE_VERSION}-bin ${HIVE_HOME} && \
+    rm apache-hive-${HIVE_VERSION}-bin.tar.gz
+
+ENTRYPOINT ["/opt/hive/bin/hive", "--service", "metastore"]
--- a/docker/playground/image/kyuubi-playground-spark.Dockerfile
+++ b/docker/playground/image/kyuubi-playground-spark.Dockerfile
@ -0,0 +1,54 @@
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     https://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+ARG KYUUBI_VERSION
+
+FROM nekyuubi/kyuubi-playground-base:${KYUUBI_VERSION}
+
+ARG AWS_JAVA_SDK_VERSION
+ARG ICEBERG_VERSION
+ARG KYUUBI_VERSION
+ARG SPARK_HADOOP_VERSION
+ARG POSTGRES_JDBC_VERSION
+ARG SCALA_BINARY_VERSION
+ARG SPARK_VERSION
+ARG SPARK_BINARY_VERSION
+
+ARG APACHE_MIRROR
+ARG MAVEN_MIRROR
+
+ENV SPARK_HOME=/opt/spark
+ENV HADOOP_CONF_DIR=/etc/hadoop/conf
+ENV HIVE_CONF_DIR=/etc/hive/conf
+ENV SPARK_CONF_DIR=/etc/spark/conf
+
+RUN set -x && \
+    wget -q ${APACHE_MIRROR}/spark/spark-${SPARK_VERSION}/spark-${SPARK_VERSION}-bin-hadoop3.tgz && \
+    tar -xzf spark-${SPARK_VERSION}-bin-hadoop3.tgz -C /opt && \
+    ln -s /opt/spark-${SPARK_VERSION}-bin-hadoop3 ${SPARK_HOME} && \
+    rm spark-${SPARK_VERSION}-bin-hadoop3.tgz && \
+    ICEBERG_SPARK_JAR_NAME=iceberg-spark-runtime-${SPARK_BINARY_VERSION}_${SCALA_BINARY_VERSION} && \
+    wget -q ${MAVEN_MIRROR}/org/apache/iceberg/${ICEBERG_SPARK_JAR_NAME}/${ICEBERG_VERSION}/${ICEBERG_SPARK_JAR_NAME}-${ICEBERG_VERSION}.jar -P ${SPARK_HOME}/jars && \
+    SPARK_HADOOP_CLOUD_JAR_NAME=spark-hadoop-cloud_${SCALA_BINARY_VERSION} && \
+    wget -q ${MAVEN_MIRROR}/org/apache/spark/${SPARK_HADOOP_CLOUD_JAR_NAME}/${SPARK_VERSION}/${SPARK_HADOOP_CLOUD_JAR_NAME}-${SPARK_VERSION}.jar -P ${SPARK_HOME}/jars && \
+    HADOOP_CLOUD_STORAGE_JAR_NAME=hadoop-cloud-storage && \
+    wget -q ${MAVEN_MIRROR}/org/apache/hadoop/${HADOOP_CLOUD_STORAGE_JAR_NAME}/${SPARK_HADOOP_VERSION}/${HADOOP_CLOUD_STORAGE_JAR_NAME}-${SPARK_HADOOP_VERSION}.jar -P ${SPARK_HOME}/jars && \
+    HADOOP_AWS_JAR_NAME=hadoop-aws && \
+    wget -q ${MAVEN_MIRROR}/org/apache/hadoop/${HADOOP_AWS_JAR_NAME}/${SPARK_HADOOP_VERSION}/${HADOOP_AWS_JAR_NAME}-${SPARK_HADOOP_VERSION}.jar -P ${SPARK_HOME}/jars && \
+    AWS_JAVA_SDK_BUNDLE_JAR_NAME=aws-java-sdk-bundle && \
+    wget -q ${MAVEN_MIRROR}/com/amazonaws/${AWS_JAVA_SDK_BUNDLE_JAR_NAME}/${AWS_JAVA_SDK_VERSION}/${AWS_JAVA_SDK_BUNDLE_JAR_NAME}-${AWS_JAVA_SDK_VERSION}.jar -P ${SPARK_HOME}/jars && \
+    POSTGRES_JDBC_JAR_NAME=postgresql && \
+    wget -q ${MAVEN_MIRROR}/org/postgresql/${POSTGRES_JDBC_JAR_NAME}/${POSTGRES_JDBC_VERSION}/${POSTGRES_JDBC_JAR_NAME}-${POSTGRES_JDBC_VERSION}.jar -P ${SPARK_HOME}/jars && \
+    TPCDS_CONNECTOR_JAR_NAME=kyuubi-spark-connector-tpcds_${SCALA_BINARY_VERSION} && \
+    wget -q ${MAVEN_MIRROR}/org/apache/kyuubi/${TPCDS_CONNECTOR_JAR_NAME}/${KYUUBI_VERSION}/${TPCDS_CONNECTOR_JAR_NAME}-${KYUUBI_VERSION}.jar -P ${SPARK_HOME}/jars && \
+    TPCH_CONNECTOR_JAR_NAME=kyuubi-spark-connector-tpch_${SCALA_BINARY_VERSION} && \
+    wget -q ${MAVEN_MIRROR}/org/apache/kyuubi/${TPCH_CONNECTOR_JAR_NAME}/${KYUUBI_VERSION}/${TPCH_CONNECTOR_JAR_NAME}-${KYUUBI_VERSION}.jar -P ${SPARK_HOME}/jars
--- a/docker/playground/image/spark-defaults.conf
+++ b/docker/playground/image/spark-defaults.conf
@ -0,0 +1,44 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+# Default system properties included when running spark-submit.
+# This is useful for setting default environmental settings.
+
+spark.driver.host=0.0.0.0
+spark.ui.port=4040
+spark.sql.shuffle.partitions=16
+spark.sql.warehouse.dir=s3a://spark-bucket/warehouse
+spark.sql.defaultCatalog=spark_catalog
+spark.sql.extensions=org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions
+spark.hadoop.fs.s3a.bucket.spark-bucket.committer.magic.enabled=true
+spark.hadoop.fs.s3a.bucket.iceberg-bucket.committer.magic.enabled=true
+
+spark.sql.catalog.spark_catalog=org.apache.iceberg.spark.SparkSessionCatalog
+spark.sql.catalog.spark_catalog.type=hive
+
+spark.sql.catalog.tpcds=org.apache.kyuubi.spark.connector.tpcds.TPCDSCatalog
+spark.sql.catalog.tpch=org.apache.kyuubi.spark.connector.tpch.TPCHCatalog
+
+spark.sql.catalog.postgres=org.apache.spark.sql.execution.datasources.v2.jdbc.JDBCTableCatalog
+spark.sql.catalog.postgres.url=jdbc:postgresql://postgres:5432/metastore
+spark.sql.catalog.postgres.driver=org.postgresql.Driver
+spark.sql.catalog.postgres.user=postgres
+spark.sql.catalog.postgres.password=postgres
+
+spark.sql.catalog.iceberg=org.apache.iceberg.spark.SparkCatalog
+spark.sql.catalog.iceberg.type=hadoop
+spark.sql.catalog.iceberg.warehouse=s3a://iceberg-bucket/iceberg-warehouse