6297651d83
# 🔍 Description ## Issue References 🔗 Kyuubi fully supports Spark 3.5 now, this pull request aims to set the default Spark to 3.5 in Kyuubi 1.9 ## Types of changes 🔖 - [ ] Bugfix (non-breaking change which fixes an issue) - [ ] New feature (non-breaking change which adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) ## Test Plan 🧪 Pass GA. --- # Checklist 📝 - [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html) **Be nice. Be informative.** Closes #6163 from pan3793/spark-3.5-default. Closes #6163 f386aeb7a [Cheng Pan] Set default Spark version to 3.5 Authored-by: Cheng Pan <chengpan@apache.org> Signed-off-by: Cheng Pan <chengpan@apache.org>
43 lines
1.4 KiB
Markdown
43 lines
1.4 KiB
Markdown
<!--
|
|
- Licensed to the Apache Software Foundation (ASF) under one or more
|
|
- contributor license agreements. See the NOTICE file distributed with
|
|
- this work for additional information regarding copyright ownership.
|
|
- The ASF licenses this file to You under the Apache License, Version 2.0
|
|
- (the "License"); you may not use this file except in compliance with
|
|
- the License. You may obtain a copy of the License at
|
|
-
|
|
- http://www.apache.org/licenses/LICENSE-2.0
|
|
-
|
|
- Unless required by applicable law or agreed to in writing, software
|
|
- distributed under the License is distributed on an "AS IS" BASIS,
|
|
- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
|
- See the License for the specific language governing permissions and
|
|
- limitations under the License.
|
|
-->
|
|
|
|
# Kyuubi Spark Listener Extension
|
|
|
|
## Functions
|
|
|
|
- [x] All `listener` extensions can be implemented in this module, like `QueryExecutionListener` and `ExtraListener`
|
|
- [x] Add `SparkOperationLineageQueryExecutionListener` to extends spark `QueryExecutionListener`
|
|
- [x] SQL lineage parsing will be triggered after SQL execution and will be written to the json logger file
|
|
|
|
## Build
|
|
|
|
```shell
|
|
build/mvn clean package -DskipTests -pl :kyuubi-spark-lineage_2.12 -am -Dspark.version=3.2.1
|
|
```
|
|
|
|
### Supported Apache Spark Versions
|
|
|
|
`-Dspark.version=`
|
|
|
|
- [x] master
|
|
- [x] 3.5.x (default)
|
|
- [x] 3.4.x
|
|
- [x] 3.3.x
|
|
- [x] 3.2.x
|
|
- [x] 3.1.x
|
|
|