[KYUUBI #6099] Remove support of Derby for Kyuubi metastore
# 🔍 Description ## Issue References 🔗 This pull request removes support of Derby for Kyuubi metastore. ## Describe Your Solution 🔧 Previously, we migrated the embedded DB of Kyuubi metastore from Derby to SQLite, and also marked Derby as deprecated (#4950), now, I propose to remove support of Derby for Kyuubi metastore. Note, that both Derby and SQLite are mainly for testing purposes, and they're not supposed to be used in production. Users should not be surprised by this removal. Also, the Derby we used suffers CVE-2022-46337 > Mitigation: > > Users should upgrade to Java 21 and Derby 10.17.1.0. > > Alternatively, users who wish to remain on older Java versions should build their own Derby distribution from one of the release families to which the fix was backported: 10.16, 10.15, and 10.14. Those are the releases which correspond, respectively, with Java LTS versions 17, 11, and 8. ## Types of changes 🔖 - [ ] Bugfix (non-breaking change which fixes an issue) - [ ] New feature (non-breaking change which adds functionality) - [x] Breaking change (fix or feature that would cause existing functionality to change) ## Test Plan 🧪 Some tests are modified accordingly. Pass GA. --- # Checklist 📝 - [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html) **Be nice. Be informative.** Closes #6099 from pan3793/remove-derby. Closes #6099 7e9dfd692 [Cheng Pan] Remove support of Derby for Kyuubi metastore Authored-by: Cheng Pan <chengpan@apache.org> Signed-off-by: Cheng Pan <chengpan@apache.org>
This commit is contained in:
Cheng Pan
parent
c6962beb65
commit
d007d7af10
@ -212,7 +212,6 @@ com.google.android:annotations
|
||||
commons-lang:commons-lang
|
||||
commons-logging:commons-logging
|
||||
org.apache.commons:commons-lang3
|
||||
org.apache.derby:derby
|
||||
com.google.errorprone:error_prone_annotations
|
||||
dev.failsafe:failsafe
|
||||
com.jakewharton.fliptables:fliptables
|
||||
|
||||
281
NOTICE-binary
281
NOTICE-binary
@ -118,287 +118,6 @@ Copyright 2001-2020 The Apache Software Foundation
|
||||
Apache Commons Logging
|
||||
Copyright 2003-2013 The Apache Software Foundation
|
||||
|
||||
=========================================================================
|
||||
== NOTICE file corresponding to section 4(d) of the Apache License,
|
||||
== Version 2.0, in this case for the Apache Derby distribution.
|
||||
==
|
||||
== DO NOT EDIT THIS FILE DIRECTLY. IT IS GENERATED
|
||||
== BY THE buildnotice TARGET IN THE TOP LEVEL build.xml FILE.
|
||||
==
|
||||
=========================================================================
|
||||
|
||||
Apache Derby
|
||||
Copyright 2004-2018 The Apache Software Foundation
|
||||
|
||||
|
||||
=========================================================================
|
||||
|
||||
Portions of Derby were originally developed by
|
||||
International Business Machines Corporation and are
|
||||
licensed to the Apache Software Foundation under the
|
||||
"Software Grant and Corporate Contribution License Agreement",
|
||||
informally known as the "Derby CLA".
|
||||
The following copyright notice(s) were affixed to portions of the code
|
||||
with which this file is now or was at one time distributed
|
||||
and are placed here unaltered.
|
||||
|
||||
(C) Copyright 1997,2004 International Business Machines Corporation. All rights reserved.
|
||||
|
||||
(C) Copyright IBM Corp. 2003.
|
||||
|
||||
|
||||
=========================================================================
|
||||
|
||||
|
||||
The portion of the functionTests under 'nist' was originally
|
||||
developed by the National Institute of Standards and Technology (NIST),
|
||||
an agency of the United States Department of Commerce, and adapted by
|
||||
International Business Machines Corporation in accordance with the NIST
|
||||
Software Acknowledgment and Redistribution document at
|
||||
http://www.itl.nist.gov/div897/ctg/sql_form.htm
|
||||
|
||||
|
||||
|
||||
=========================================================================
|
||||
|
||||
|
||||
The Derby build relies on source files supplied by the Apache Felix
|
||||
project. The following notice covers the Felix files:
|
||||
|
||||
Apache Felix Main
|
||||
Copyright 2008 The Apache Software Foundation
|
||||
|
||||
|
||||
I. Included Software
|
||||
|
||||
This product includes software developed at
|
||||
The Apache Software Foundation (http://www.apache.org/).
|
||||
Licensed under the Apache License 2.0.
|
||||
|
||||
This product includes software developed at
|
||||
The OSGi Alliance (http://www.osgi.org/).
|
||||
Copyright (c) OSGi Alliance (2000, 2007).
|
||||
Licensed under the Apache License 2.0.
|
||||
|
||||
This product includes software from http://kxml.sourceforge.net.
|
||||
Copyright (c) 2002,2003, Stefan Haustein, Oberhausen, Rhld., Germany.
|
||||
Licensed under BSD License.
|
||||
|
||||
II. Used Software
|
||||
|
||||
This product uses software developed at
|
||||
The OSGi Alliance (http://www.osgi.org/).
|
||||
Copyright (c) OSGi Alliance (2000, 2007).
|
||||
Licensed under the Apache License 2.0.
|
||||
|
||||
|
||||
III. License Summary
|
||||
- Apache License 2.0
|
||||
- BSD License
|
||||
|
||||
|
||||
=========================================================================
|
||||
|
||||
|
||||
The Derby build relies on jar files supplied by the Apache Lucene
|
||||
project. The following notice covers the Lucene files:
|
||||
|
||||
Apache Lucene
|
||||
Copyright 2013 The Apache Software Foundation
|
||||
|
||||
Includes software from other Apache Software Foundation projects,
|
||||
including, but not limited to:
|
||||
- Apache Ant
|
||||
- Apache Jakarta Regexp
|
||||
- Apache Commons
|
||||
- Apache Xerces
|
||||
|
||||
ICU4J, (under analysis/icu) is licensed under an MIT styles license
|
||||
and Copyright (c) 1995-2008 International Business Machines Corporation and others
|
||||
|
||||
Some data files (under analysis/icu/src/data) are derived from Unicode data such
|
||||
as the Unicode Character Database. See http://unicode.org/copyright.html for more
|
||||
details.
|
||||
|
||||
Brics Automaton (under core/src/java/org/apache/lucene/util/automaton) is
|
||||
BSD-licensed, created by Anders Møller. See http://www.brics.dk/automaton/
|
||||
|
||||
The levenshtein automata tables (under core/src/java/org/apache/lucene/util/automaton) were
|
||||
automatically generated with the moman/finenight FSA library, created by
|
||||
Jean-Philippe Barrette-LaPierre. This library is available under an MIT license,
|
||||
see http://sites.google.com/site/rrettesite/moman and
|
||||
http://bitbucket.org/jpbarrette/moman/overview/
|
||||
|
||||
The class org.apache.lucene.util.WeakIdentityMap was derived from
|
||||
the Apache CXF project and is Apache License 2.0.
|
||||
|
||||
The Google Code Prettify is Apache License 2.0.
|
||||
See http://code.google.com/p/google-code-prettify/
|
||||
|
||||
JUnit (junit-4.10) is licensed under the Common Public License v. 1.0
|
||||
See http://junit.sourceforge.net/cpl-v10.html
|
||||
|
||||
This product includes code (JaspellTernarySearchTrie) from Java Spelling Checkin
|
||||
g Package (jaspell): http://jaspell.sourceforge.net/
|
||||
License: The BSD License (http://www.opensource.org/licenses/bsd-license.php)
|
||||
|
||||
The snowball stemmers in
|
||||
analysis/common/src/java/net/sf/snowball
|
||||
were developed by Martin Porter and Richard Boulton.
|
||||
The snowball stopword lists in
|
||||
analysis/common/src/resources/org/apache/lucene/analysis/snowball
|
||||
were developed by Martin Porter and Richard Boulton.
|
||||
The full snowball package is available from
|
||||
http://snowball.tartarus.org/
|
||||
|
||||
The KStem stemmer in
|
||||
analysis/common/src/org/apache/lucene/analysis/en
|
||||
was developed by Bob Krovetz and Sergio Guzman-Lara (CIIR-UMass Amherst)
|
||||
under the BSD-license.
|
||||
|
||||
The Arabic,Persian,Romanian,Bulgarian, and Hindi analyzers (common) come with a default
|
||||
stopword list that is BSD-licensed created by Jacques Savoy. These files reside in:
|
||||
analysis/common/src/resources/org/apache/lucene/analysis/ar/stopwords.txt,
|
||||
analysis/common/src/resources/org/apache/lucene/analysis/fa/stopwords.txt,
|
||||
analysis/common/src/resources/org/apache/lucene/analysis/ro/stopwords.txt,
|
||||
analysis/common/src/resources/org/apache/lucene/analysis/bg/stopwords.txt,
|
||||
analysis/common/src/resources/org/apache/lucene/analysis/hi/stopwords.txt
|
||||
See http://members.unine.ch/jacques.savoy/clef/index.html.
|
||||
|
||||
The German,Spanish,Finnish,French,Hungarian,Italian,Portuguese,Russian and Swedish light stemmers
|
||||
(common) are based on BSD-licensed reference implementations created by Jacques Savoy and
|
||||
Ljiljana Dolamic. These files reside in:
|
||||
analysis/common/src/java/org/apache/lucene/analysis/de/GermanLightStemmer.java
|
||||
analysis/common/src/java/org/apache/lucene/analysis/de/GermanMinimalStemmer.java
|
||||
analysis/common/src/java/org/apache/lucene/analysis/es/SpanishLightStemmer.java
|
||||
analysis/common/src/java/org/apache/lucene/analysis/fi/FinnishLightStemmer.java
|
||||
analysis/common/src/java/org/apache/lucene/analysis/fr/FrenchLightStemmer.java
|
||||
analysis/common/src/java/org/apache/lucene/analysis/fr/FrenchMinimalStemmer.java
|
||||
analysis/common/src/java/org/apache/lucene/analysis/hu/HungarianLightStemmer.java
|
||||
analysis/common/src/java/org/apache/lucene/analysis/it/ItalianLightStemmer.java
|
||||
analysis/common/src/java/org/apache/lucene/analysis/pt/PortugueseLightStemmer.java
|
||||
analysis/common/src/java/org/apache/lucene/analysis/ru/RussianLightStemmer.java
|
||||
analysis/common/src/java/org/apache/lucene/analysis/sv/SwedishLightStemmer.java
|
||||
|
||||
The Stempel analyzer (stempel) includes BSD-licensed software developed
|
||||
by the Egothor project http://egothor.sf.net/, created by Leo Galambos, Martin Kvapil,
|
||||
and Edmond Nolan.
|
||||
|
||||
The Polish analyzer (stempel) comes with a default
|
||||
stopword list that is BSD-licensed created by the Carrot2 project. The file resides
|
||||
in stempel/src/resources/org/apache/lucene/analysis/pl/stopwords.txt.
|
||||
See http://project.carrot2.org/license.html.
|
||||
|
||||
The SmartChineseAnalyzer source code (smartcn) was
|
||||
provided by Xiaoping Gao and copyright 2009 by www.imdict.net.
|
||||
|
||||
WordBreakTestUnicode_*.java (under modules/analysis/common/src/test/)
|
||||
is derived from Unicode data such as the Unicode Character Database.
|
||||
See http://unicode.org/copyright.html for more details.
|
||||
|
||||
The Morfologik analyzer (morfologik) includes BSD-licensed software
|
||||
developed by Dawid Weiss and Marcin Miłkowski (http://morfologik.blogspot.com/).
|
||||
|
||||
Morfologik uses data from Polish ispell/myspell dictionary
|
||||
(http://www.sjp.pl/slownik/en/) licenced on the terms of (inter alia)
|
||||
LGPL and Creative Commons ShareAlike.
|
||||
|
||||
Morfologic includes data from BSD-licensed dictionary of Polish (SGJP)
|
||||
(http://sgjp.pl/morfeusz/)
|
||||
|
||||
Servlet-api.jar and javax.servlet-*.jar are under the CDDL license, the original
|
||||
source code for this can be found at http://www.eclipse.org/jetty/downloads.php
|
||||
|
||||
===========================================================================
|
||||
Kuromoji Japanese Morphological Analyzer - Apache Lucene Integration
|
||||
===========================================================================
|
||||
|
||||
This software includes a binary and/or source version of data from
|
||||
|
||||
mecab-ipadic-2.7.0-20070801
|
||||
|
||||
which can be obtained from
|
||||
|
||||
http://atilika.com/releases/mecab-ipadic/mecab-ipadic-2.7.0-20070801.tar.gz
|
||||
|
||||
or
|
||||
|
||||
http://jaist.dl.sourceforge.net/project/mecab/mecab-ipadic/2.7.0-20070801/mecab-ipadic-2.7.0-20070801.tar.gz
|
||||
|
||||
===========================================================================
|
||||
mecab-ipadic-2.7.0-20070801 Notice
|
||||
===========================================================================
|
||||
|
||||
Nara Institute of Science and Technology (NAIST),
|
||||
the copyright holders, disclaims all warranties with regard to this
|
||||
software, including all implied warranties of merchantability and
|
||||
fitness, in no event shall NAIST be liable for
|
||||
any special, indirect or consequential damages or any damages
|
||||
whatsoever resulting from loss of use, data or profits, whether in an
|
||||
action of contract, negligence or other tortuous action, arising out
|
||||
of or in connection with the use or performance of this software.
|
||||
|
||||
A large portion of the dictionary entries
|
||||
originate from ICOT Free Software. The following conditions for ICOT
|
||||
Free Software applies to the current dictionary as well.
|
||||
|
||||
Each User may also freely distribute the Program, whether in its
|
||||
original form or modified, to any third party or parties, PROVIDED
|
||||
that the provisions of Section 3 ("NO WARRANTY") will ALWAYS appear
|
||||
on, or be attached to, the Program, which is distributed substantially
|
||||
in the same form as set out herein and that such intended
|
||||
distribution, if actually made, will neither violate or otherwise
|
||||
contravene any of the laws and regulations of the countries having
|
||||
jurisdiction over the User or the intended distribution itself.
|
||||
|
||||
NO WARRANTY
|
||||
|
||||
The program was produced on an experimental basis in the course of the
|
||||
research and development conducted during the project and is provided
|
||||
to users as so produced on an experimental basis. Accordingly, the
|
||||
program is provided without any warranty whatsoever, whether express,
|
||||
implied, statutory or otherwise. The term "warranty" used herein
|
||||
includes, but is not limited to, any warranty of the quality,
|
||||
performance, merchantability and fitness for a particular purpose of
|
||||
the program and the nonexistence of any infringement or violation of
|
||||
any right of any third party.
|
||||
|
||||
Each user of the program will agree and understand, and be deemed to
|
||||
have agreed and understood, that there is no warranty whatsoever for
|
||||
the program and, accordingly, the entire risk arising from or
|
||||
otherwise connected with the program is assumed by the user.
|
||||
|
||||
Therefore, neither ICOT, the copyright holder, or any other
|
||||
organization that participated in or was otherwise related to the
|
||||
development of the program and their respective officials, directors,
|
||||
officers and other employees shall be held liable for any and all
|
||||
damages, including, without limitation, general, special, incidental
|
||||
and consequential damages, arising out of or otherwise in connection
|
||||
with the use or inability to use the program or any product, material
|
||||
or result produced or otherwise obtained by using the program,
|
||||
regardless of whether they have been advised of, or otherwise had
|
||||
knowledge of, the possibility of such damages at any time during the
|
||||
project or thereafter. Each user will be deemed to have agreed to the
|
||||
foregoing by his or her commencement of use of the program. The term
|
||||
"use" as used herein includes, but is not limited to, the use,
|
||||
modification, copying and distribution of the program and the
|
||||
production of secondary products from the program.
|
||||
|
||||
In the case where the program, whether in its original form or
|
||||
modified, was distributed or delivered to or received by a user from
|
||||
any person, organization or entity other than ICOT, unless it makes or
|
||||
grants independently of ICOT any specific warranty to the user in
|
||||
writing, such person, organization or entity, will also be exempted
|
||||
from and not be held liable to the user for any such damages as noted
|
||||
above as far as the program is concerned.
|
||||
|
||||
=========================================================================
|
||||
|
||||
The Derby build relies on a jar file supplied by the JSON Simple
|
||||
project, hosted at https://code.google.com/p/json-simple/.
|
||||
The JSON simple jar file is licensed under the Apache 2.0 License.
|
||||
No other notice covers that jar file.
|
||||
|
||||
Apache Hadoop
|
||||
Copyright 2006 and onwards The Apache Software Foundation.
|
||||
|
||||
|
||||
@ -33,7 +33,6 @@ commons-collections/3.2.2//commons-collections-3.2.2.jar
|
||||
commons-lang/2.6//commons-lang-2.6.jar
|
||||
commons-lang3/3.13.0//commons-lang3-3.13.0.jar
|
||||
commons-logging/1.1.3//commons-logging-1.1.3.jar
|
||||
derby/10.14.2.0//derby-10.14.2.0.jar
|
||||
error_prone_annotations/2.20.0//error_prone_annotations-2.20.0.jar
|
||||
failsafe/3.3.2//failsafe-3.3.2.jar
|
||||
failureaccess/1.0.1//failureaccess-1.0.1.jar
|
||||
|
||||
@ -359,24 +359,24 @@ You can configure the Kyuubi properties in `$KYUUBI_HOME/conf/kyuubi-defaults.co
|
||||
|
||||
### Metadata
|
||||
|
||||
| Key | Default | Meaning | Type | Since |
|
||||
|-------------------------------------------------|----------------------------------------------------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|----------|-------|
|
||||
| kyuubi.metadata.cleaner.enabled | true | Whether to clean the metadata periodically. If it is enabled, Kyuubi will clean the metadata that is in the terminate state with max age limitation. | boolean | 1.6.0 |
|
||||
| kyuubi.metadata.cleaner.interval | PT30M | The interval to check and clean expired metadata. | duration | 1.6.0 |
|
||||
| kyuubi.metadata.max.age | PT72H | The maximum age of metadata, the metadata exceeding the age will be cleaned. | duration | 1.6.0 |
|
||||
| kyuubi.metadata.recovery.threads | 10 | The number of threads for recovery from the metadata store when the Kyuubi server restarts. | int | 1.6.0 |
|
||||
| kyuubi.metadata.request.async.retry.enabled | true | Whether to retry in async when metadata request failed. When true, return success response immediately even the metadata request failed, and schedule it in background until success, to tolerate long-time metadata store outages w/o blocking the submission request. | boolean | 1.7.0 |
|
||||
| kyuubi.metadata.request.async.retry.queue.size | 65536 | The maximum queue size for buffering metadata requests in memory when the external metadata storage is down. Requests will be dropped if the queue exceeds. Only take affect when kyuubi.metadata.request.async.retry.enabled is `true`. | int | 1.6.0 |
|
||||
| kyuubi.metadata.request.async.retry.threads | 10 | Number of threads in the metadata request async retry manager thread pool. Only take affect when kyuubi.metadata.request.async.retry.enabled is `true`. | int | 1.6.0 |
|
||||
| kyuubi.metadata.request.retry.interval | PT5S | The interval to check and trigger the metadata request retry tasks. | duration | 1.6.0 |
|
||||
| kyuubi.metadata.store.class | org.apache.kyuubi.server.metadata.jdbc.JDBCMetadataStore | Fully qualified class name for server metadata store. | string | 1.6.0 |
|
||||
| kyuubi.metadata.store.jdbc.database.schema.init | true | Whether to init the JDBC metadata store database schema. | boolean | 1.6.0 |
|
||||
| kyuubi.metadata.store.jdbc.database.type | SQLITE | The database type for server jdbc metadata store.<ul> <li>(Deprecated) DERBY: Apache Derby, JDBC driver `org.apache.derby.jdbc.AutoloadedDriver`.</li> <li>SQLITE: SQLite3, JDBC driver `org.sqlite.JDBC`.</li> <li>MYSQL: MySQL, JDBC driver `com.mysql.cj.jdbc.Driver` (fallback `com.mysql.jdbc.Driver`).</li> <li>POSTGRESQL: PostgreSQL, JDBC driver `org.postgresql.Driver`.</li> <li>CUSTOM: User-defined database type, need to specify corresponding JDBC driver.</li> Note that: The JDBC datasource is powered by HiKariCP, for datasource properties, please specify them with the prefix: kyuubi.metadata.store.jdbc.datasource. For example, kyuubi.metadata.store.jdbc.datasource.connectionTimeout=10000. | string | 1.6.0 |
|
||||
| kyuubi.metadata.store.jdbc.driver | <undefined> | JDBC driver class name for server jdbc metadata store. | string | 1.6.0 |
|
||||
| kyuubi.metadata.store.jdbc.password || The password for server JDBC metadata store. | string | 1.6.0 |
|
||||
| kyuubi.metadata.store.jdbc.priority.enabled | false | Whether to enable the priority scheduling for batch impl v2. When false, ignore kyuubi.batch.priority and use the FIFO ordering strategy for batch job scheduling. Note: this feature may cause significant performance issues when using MySQL 5.7 as the metastore backend due to the lack of support for mixed order index. See more details at KYUUBI #5329. | boolean | 1.8.0 |
|
||||
| kyuubi.metadata.store.jdbc.url | jdbc:sqlite:<KYUUBI_HOME>/kyuubi_state_store.db | The JDBC url for server JDBC metadata store. By default, it is a SQLite database url, and the state information is not shared across Kyuubi instances. To enable high availability for multiple kyuubi instances, please specify a production JDBC url. Note: this value support the variables substitution: `<KYUUBI_HOME>`. | string | 1.6.0 |
|
||||
| kyuubi.metadata.store.jdbc.user || The username for server JDBC metadata store. | string | 1.6.0 |
|
||||
| Key | Default | Meaning | Type | Since |
|
||||
|-------------------------------------------------|----------------------------------------------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|----------|-------|
|
||||
| kyuubi.metadata.cleaner.enabled | true | Whether to clean the metadata periodically. If it is enabled, Kyuubi will clean the metadata that is in the terminate state with max age limitation. | boolean | 1.6.0 |
|
||||
| kyuubi.metadata.cleaner.interval | PT30M | The interval to check and clean expired metadata. | duration | 1.6.0 |
|
||||
| kyuubi.metadata.max.age | PT72H | The maximum age of metadata, the metadata exceeding the age will be cleaned. | duration | 1.6.0 |
|
||||
| kyuubi.metadata.recovery.threads | 10 | The number of threads for recovery from the metadata store when the Kyuubi server restarts. | int | 1.6.0 |
|
||||
| kyuubi.metadata.request.async.retry.enabled | true | Whether to retry in async when metadata request failed. When true, return success response immediately even the metadata request failed, and schedule it in background until success, to tolerate long-time metadata store outages w/o blocking the submission request. | boolean | 1.7.0 |
|
||||
| kyuubi.metadata.request.async.retry.queue.size | 65536 | The maximum queue size for buffering metadata requests in memory when the external metadata storage is down. Requests will be dropped if the queue exceeds. Only take affect when kyuubi.metadata.request.async.retry.enabled is `true`. | int | 1.6.0 |
|
||||
| kyuubi.metadata.request.async.retry.threads | 10 | Number of threads in the metadata request async retry manager thread pool. Only take affect when kyuubi.metadata.request.async.retry.enabled is `true`. | int | 1.6.0 |
|
||||
| kyuubi.metadata.request.retry.interval | PT5S | The interval to check and trigger the metadata request retry tasks. | duration | 1.6.0 |
|
||||
| kyuubi.metadata.store.class | org.apache.kyuubi.server.metadata.jdbc.JDBCMetadataStore | Fully qualified class name for server metadata store. | string | 1.6.0 |
|
||||
| kyuubi.metadata.store.jdbc.database.schema.init | true | Whether to init the JDBC metadata store database schema. | boolean | 1.6.0 |
|
||||
| kyuubi.metadata.store.jdbc.database.type | SQLITE | The database type for server jdbc metadata store.<ul> <li>SQLITE: SQLite3, JDBC driver `org.sqlite.JDBC`.</li> <li>MYSQL: MySQL, JDBC driver `com.mysql.cj.jdbc.Driver` (fallback `com.mysql.jdbc.Driver`).</li> <li>POSTGRESQL: PostgreSQL, JDBC driver `org.postgresql.Driver`.</li> <li>CUSTOM: User-defined database type, need to specify corresponding JDBC driver.</li> Note that: The JDBC datasource is powered by HiKariCP, for datasource properties, please specify them with the prefix: kyuubi.metadata.store.jdbc.datasource. For example, kyuubi.metadata.store.jdbc.datasource.connectionTimeout=10000. | string | 1.6.0 |
|
||||
| kyuubi.metadata.store.jdbc.driver | <undefined> | JDBC driver class name for server jdbc metadata store. | string | 1.6.0 |
|
||||
| kyuubi.metadata.store.jdbc.password || The password for server JDBC metadata store. | string | 1.6.0 |
|
||||
| kyuubi.metadata.store.jdbc.priority.enabled | false | Whether to enable the priority scheduling for batch impl v2. When false, ignore kyuubi.batch.priority and use the FIFO ordering strategy for batch job scheduling. Note: this feature may cause significant performance issues when using MySQL 5.7 as the metastore backend due to the lack of support for mixed order index. See more details at KYUUBI #5329. | boolean | 1.8.0 |
|
||||
| kyuubi.metadata.store.jdbc.url | jdbc:sqlite:<KYUUBI_HOME>/kyuubi_state_store.db | The JDBC url for server JDBC metadata store. By default, it is a SQLite database url, and the state information is not shared across Kyuubi instances. To enable high availability for multiple kyuubi instances, please specify a production JDBC url. Note: this value support the variables substitution: `<KYUUBI_HOME>`. | string | 1.6.0 |
|
||||
| kyuubi.metadata.store.jdbc.user || The username for server JDBC metadata store. | string | 1.6.0 |
|
||||
|
||||
### Metrics
|
||||
|
||||
|
||||
@ -20,6 +20,7 @@
|
||||
## Upgrading from Kyuubi 1.8 to 1.9
|
||||
|
||||
* Since Kyuubi 1.9.0, `kyuubi.session.conf.advisor` can be set as a sequence, Kyuubi supported chaining SessionConfAdvisors.
|
||||
* Since Kyuubi 1.9.0, the support of Derby is removal for Kyuubi metastore.
|
||||
|
||||
## Upgrading from Kyuubi 1.8.0 to 1.8.1
|
||||
|
||||
|
||||
@ -155,12 +155,6 @@
|
||||
<scope>test</scope>
|
||||
</dependency>
|
||||
|
||||
<dependency>
|
||||
<groupId>org.apache.derby</groupId>
|
||||
<artifactId>derby</artifactId>
|
||||
<scope>test</scope>
|
||||
</dependency>
|
||||
|
||||
<dependency>
|
||||
<groupId>org.xerial</groupId>
|
||||
<artifactId>sqlite-jdbc</artifactId>
|
||||
|
||||
@ -113,7 +113,6 @@ object JdbcUtils extends Logging {
|
||||
|
||||
def isDuplicatedKeyDBErr(cause: Throwable): Boolean = {
|
||||
val duplicatedKeyKeywords = Seq(
|
||||
"duplicate key value in a unique or primary key constraint or unique index", // Derby
|
||||
"Duplicate entry", // MySQL
|
||||
"duplicate key value violates unique constraint", // PostgreSQL
|
||||
"A UNIQUE constraint failed" // SQLite
|
||||
|
||||
@ -26,10 +26,8 @@ import org.apache.kyuubi.KyuubiFunSuite
|
||||
|
||||
class JdbcUtilsSuite extends KyuubiFunSuite {
|
||||
|
||||
private val dbUrl = s"jdbc:derby:memory:jdbc_utils_test"
|
||||
private val connUrl = s"$dbUrl;create=true"
|
||||
private val shutdownUrl = s"$dbUrl;shutdown=true"
|
||||
private val driverClz = "org.apache.derby.jdbc.AutoloadedDriver"
|
||||
private val connUrl = "jdbc:sqlite:file:test_jdbc.db"
|
||||
private val driverClz = "org.sqlite.JDBC"
|
||||
|
||||
implicit private val ds: DataSource =
|
||||
new DriverDataSource(connUrl, driverClz, new Properties, "test", "test")
|
||||
@ -37,6 +35,7 @@ class JdbcUtilsSuite extends KyuubiFunSuite {
|
||||
case class Person(id: Int, name: String)
|
||||
|
||||
test("JdbcUtils methods") {
|
||||
JdbcUtils.execute("DROP TABLE IF EXISTS person")()
|
||||
JdbcUtils.execute(
|
||||
"""CREATE TABLE person(
|
||||
| id INT NOT NULL PRIMARY KEY,
|
||||
@ -78,5 +77,6 @@ class JdbcUtilsSuite extends KyuubiFunSuite {
|
||||
JdbcUtils.redactPassword(None)
|
||||
}
|
||||
|
||||
JdbcUtils.execute("DROP TABLE IF EXISTS person")()
|
||||
}
|
||||
}
|
||||
|
||||
@ -262,11 +262,6 @@
|
||||
<artifactId>HikariCP</artifactId>
|
||||
</dependency>
|
||||
|
||||
<dependency>
|
||||
<groupId>org.apache.derby</groupId>
|
||||
<artifactId>derby</artifactId>
|
||||
</dependency>
|
||||
|
||||
<dependency>
|
||||
<groupId>org.xerial</groupId>
|
||||
<artifactId>sqlite-jdbc</artifactId>
|
||||
@ -410,6 +405,12 @@
|
||||
</exclusions>
|
||||
</dependency>
|
||||
|
||||
<dependency>
|
||||
<groupId>org.apache.derby</groupId>
|
||||
<artifactId>derby</artifactId>
|
||||
<scope>test</scope>
|
||||
</dependency>
|
||||
|
||||
<dependency>
|
||||
<groupId>org.apache.kafka</groupId>
|
||||
<artifactId>kafka-clients</artifactId>
|
||||
|
||||
@ -1,8 +0,0 @@
|
||||
ALTER TABLE metadata ALTER COLUMN session_type varchar(32) NOT NULL;
|
||||
ALTER TABLE metadata ALTER COLUMN real_user varchar(255) NOT NULL;
|
||||
ALTER TABLE metadata ALTER COLUMN user_name varchar(255) NOT NULL;
|
||||
ALTER TABLE metadata ALTER COLUMN ip_address varchar(128);
|
||||
ALTER TABLE metadata ALTER COLUMN engine_type varchar(32) NOT NULL;
|
||||
ALTER TABLE metadata ALTER COLUMN engine_state varchar(32);
|
||||
|
||||
DROP INDEX metadata_kyuubi_instance_index;
|
||||
@ -1 +0,0 @@
|
||||
ALTER TABLE metadata ADD COLUMN engine_open_time bigint;
|
||||
@ -1 +0,0 @@
|
||||
ALTER TABLE metadata ALTER COLUMN kyuubi_instance DROP NOT NULL;
|
||||
@ -1 +0,0 @@
|
||||
CREATE INDEX metadata_create_time_index ON metadata(create_time);
|
||||
@ -1,3 +0,0 @@
|
||||
ALTER TABLE metadata ADD COLUMN priority int NOT NULL DEFAULT 10;
|
||||
|
||||
CREATE INDEX metadata_priority_create_time_index ON metadata(priority, create_time);
|
||||
@ -1,98 +0,0 @@
|
||||
#
|
||||
# Licensed to the Apache Software Foundation (ASF) under one or more
|
||||
# contributor license agreements. See the NOTICE file distributed with
|
||||
# this work for additional information regarding copyright ownership.
|
||||
# The ASF licenses this file to You under the Apache License, Version 2.0
|
||||
# (the "License"); you may not use this file except in compliance with
|
||||
# the License. You may obtain a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
#
|
||||
|
||||
Kyuubi MetaStore Upgrade HowTo
|
||||
============================
|
||||
|
||||
This document describes how to upgrade the schema of a Derby backed
|
||||
Kyuubi MetaStore instance from one release version of Kyuubi to another
|
||||
release version of Kyuubi. For example, by following the steps listed
|
||||
below it is possible to upgrade a Kyuubi 1.6.0 MetaStore schema to a
|
||||
Kyuubi 1.7.0 MetaStore schema. Before attempting this project we
|
||||
strongly recommend that you read through all of the steps in this
|
||||
document and familiarize yourself with the required tools.
|
||||
|
||||
MetaStore Upgrade Steps
|
||||
=======================
|
||||
|
||||
1) Shutdown your MetaStore instance and restrict access to the
|
||||
MetaStore's Derby database. It is very important that no one else
|
||||
accesses or modifies the contents of database while you are
|
||||
performing the schema upgrade.
|
||||
|
||||
2) Create a backup of your Derby metastore database. This will allow
|
||||
you to revert any changes made during the upgrade process if
|
||||
something goes wrong. The easiest way of accomplishing this task is
|
||||
by creating a copy of the directory containing your Derby database.
|
||||
|
||||
3) Dump your MetaStore database schema to a file using Derby's dblook utility:
|
||||
|
||||
% dblook -d <derby_db_url> -z "APP" > my-schema-x.y.z.derby.sql
|
||||
|
||||
Note that "APP" is Derby's default schema for user-created catalog
|
||||
objects.
|
||||
|
||||
|
||||
4) The schema upgrade scripts assume that the schema you are upgrading
|
||||
closely matches the official schema for your particular version of
|
||||
Kyuubi. The files in this directory with names like
|
||||
"metadata-store-schema-x.y.z.derby.sql" contain dumps of the official schemas
|
||||
corresponding to each of the released versions of Kyuubi. You can
|
||||
determine differences between your schema and the official schema
|
||||
by comparing the contents of the official dump with the schema dump
|
||||
you created in the previous step. Note that due to a bug in Derby
|
||||
the order in which the DDL statements appear is non-deterministic,
|
||||
so simply diffing the two dumps is unlikely to result in useable
|
||||
results. A simple workaround for this problem is to compare sorted
|
||||
versions of the two schema dumps, e.g:
|
||||
|
||||
% sort metadata-store-schema-x.y.z.derby.sql > metadata-store-schema-x.y.z.derby.sql.sorted
|
||||
% sort my-schema-x.y.z.derby.sql > my-schema-x.y.z.derby.sql.sorted
|
||||
% diff metadata-store-schema-x.y.z.derby.sql.sorted my-schema-x.y.z.derby.sql.sorted
|
||||
|
||||
Some differences are acceptable and will not interfere
|
||||
with the upgrade process, but others need to be resolved manually
|
||||
or the upgrade scripts will fail to complete.
|
||||
|
||||
5) You are now ready to run the schema upgrade scripts. If you are
|
||||
upgrading from Kyuubi 1.6.0 to Kyuubi 1.7.0 you need to run the
|
||||
upgrade-1.6.0-to-1.7.0.derby.sql script, but if you are upgrading
|
||||
from 1.6.0 to 1.8.0 you will need to run the 1.6.0 to 1.7.0 upgrade
|
||||
script followed by the 1.7.0 to 1.8.0 upgrade script.
|
||||
|
||||
NOTE: You may need to install the Derby 'ij' utility.
|
||||
Look here for installation instructions:
|
||||
http://db.apache.org/derby/docs/10.4/getstart/
|
||||
|
||||
% ij
|
||||
ij version 10.4
|
||||
ij> CONNECT 'jdbc:derby:/Users/bob/hive/metastore_db';
|
||||
ij> RUN 'upgrade-1.6.0-to-1.7.0.derby.sql';
|
||||
ij> RUN 'upgrade-1.7.0-to-1.8.0.derby.sql';
|
||||
ij> quit;
|
||||
|
||||
These scripts should run to completion without any errors. If you
|
||||
do encounter errors you need to analyze the cause and attempt to
|
||||
trace it back to one of the preceding steps.
|
||||
|
||||
6) The final step of the upgrade process is validating your freshly
|
||||
upgraded schema against the official schema for your particular
|
||||
version of Kyuubi. This is accomplished by repeating steps (3) and
|
||||
(4), but this time comparing against the official version of the
|
||||
upgraded schema, e.g. if you upgraded the schema to Kyuubi 1.7.0 then
|
||||
you will want to compare your schema dump against the contents of
|
||||
metadata-store-schema-1.7.0.derby.sql
|
||||
@ -1,37 +0,0 @@
|
||||
-- Derby does not support `CREATE TABLE IF NOT EXISTS`
|
||||
|
||||
-- the metadata table ddl
|
||||
|
||||
CREATE TABLE metadata(
|
||||
key_id bigint PRIMARY KEY GENERATED BY DEFAULT AS IDENTITY, -- the auto increment key id
|
||||
identifier varchar(36) NOT NULL, -- the identifier id, which is an UUID
|
||||
session_type varchar(128) NOT NULL, -- the session type, SQL or BATCH
|
||||
real_user varchar(1024) NOT NULL, -- the real user
|
||||
user_name varchar(1024) NOT NULL, -- the user name, might be a proxy user
|
||||
ip_address varchar(512), -- the client ip address
|
||||
kyuubi_instance varchar(1024) NOT NULL, -- the kyuubi instance that creates this
|
||||
state varchar(128) NOT NULL, -- the session state
|
||||
resource varchar(1024), -- the main resource
|
||||
class_name varchar(1024), -- the main class name
|
||||
request_name varchar(1024), -- the request name
|
||||
request_conf clob, -- the request config map
|
||||
request_args clob, -- the request arguments
|
||||
create_time BIGINT NOT NULL, -- the metadata create time
|
||||
engine_type varchar(1024) NOT NULL, -- the engine type
|
||||
cluster_manager varchar(128), -- the engine cluster manager
|
||||
engine_id varchar(128), -- the engine application id
|
||||
engine_name clob, -- the engine application name
|
||||
engine_url varchar(1024), -- the engine tracking url
|
||||
engine_state varchar(128), -- the engine application state
|
||||
engine_error clob, -- the engine application diagnose
|
||||
end_time bigint, -- the metadata end time
|
||||
peer_instance_closed boolean default FALSE -- closed by peer kyuubi instance
|
||||
);
|
||||
|
||||
CREATE INDEX metadata_kyuubi_instance_index ON metadata(kyuubi_instance);
|
||||
|
||||
CREATE UNIQUE INDEX metadata_unique_identifier_index ON metadata(identifier);
|
||||
|
||||
CREATE INDEX metadata_user_name_index ON metadata(user_name);
|
||||
|
||||
CREATE INDEX metadata_engine_type_index ON metadata(engine_type);
|
||||
@ -1,36 +0,0 @@
|
||||
-- Derby does not support `CREATE TABLE IF NOT EXISTS`
|
||||
|
||||
-- the metadata table ddl
|
||||
|
||||
CREATE TABLE metadata(
|
||||
key_id bigint PRIMARY KEY GENERATED BY DEFAULT AS IDENTITY, -- the auto increment key id
|
||||
identifier varchar(36) NOT NULL, -- the identifier id, which is an UUID
|
||||
session_type varchar(32) NOT NULL, -- the session type, SQL or BATCH
|
||||
real_user varchar(255) NOT NULL, -- the real user
|
||||
user_name varchar(255) NOT NULL, -- the user name, might be a proxy user
|
||||
ip_address varchar(128), -- the client ip address
|
||||
kyuubi_instance varchar(1024) NOT NULL, -- the kyuubi instance that creates this
|
||||
state varchar(128) NOT NULL, -- the session state
|
||||
resource varchar(1024), -- the main resource
|
||||
class_name varchar(1024), -- the main class name
|
||||
request_name varchar(1024), -- the request name
|
||||
request_conf clob, -- the request config map
|
||||
request_args clob, -- the request arguments
|
||||
create_time BIGINT NOT NULL, -- the metadata create time
|
||||
engine_type varchar(32) NOT NULL, -- the engine type
|
||||
cluster_manager varchar(128), -- the engine cluster manager
|
||||
engine_open_time bigint, -- the engine open time
|
||||
engine_id varchar(128), -- the engine application id
|
||||
engine_name clob, -- the engine application name
|
||||
engine_url varchar(1024), -- the engine tracking url
|
||||
engine_state varchar(32), -- the engine application state
|
||||
engine_error clob, -- the engine application diagnose
|
||||
end_time bigint, -- the metadata end time
|
||||
peer_instance_closed boolean default FALSE -- closed by peer kyuubi instance
|
||||
);
|
||||
|
||||
CREATE UNIQUE INDEX metadata_unique_identifier_index ON metadata(identifier);
|
||||
|
||||
CREATE INDEX metadata_user_name_index ON metadata(user_name);
|
||||
|
||||
CREATE INDEX metadata_engine_type_index ON metadata(engine_type);
|
||||
@ -1,41 +0,0 @@
|
||||
-- Derby does not support `CREATE TABLE IF NOT EXISTS`
|
||||
|
||||
-- the metadata table ddl
|
||||
|
||||
CREATE TABLE metadata(
|
||||
key_id bigint PRIMARY KEY GENERATED BY DEFAULT AS IDENTITY, -- the auto increment key id
|
||||
identifier varchar(36) NOT NULL, -- the identifier id, which is an UUID
|
||||
session_type varchar(32) NOT NULL, -- the session type, SQL or BATCH
|
||||
real_user varchar(255) NOT NULL, -- the real user
|
||||
user_name varchar(255) NOT NULL, -- the user name, might be a proxy user
|
||||
ip_address varchar(128), -- the client ip address
|
||||
kyuubi_instance varchar(1024), -- the kyuubi instance that creates this
|
||||
state varchar(128) NOT NULL, -- the session state
|
||||
resource varchar(1024), -- the main resource
|
||||
class_name varchar(1024), -- the main class name
|
||||
request_name varchar(1024), -- the request name
|
||||
request_conf clob, -- the request config map
|
||||
request_args clob, -- the request arguments
|
||||
create_time BIGINT NOT NULL, -- the metadata create time
|
||||
engine_type varchar(32) NOT NULL, -- the engine type
|
||||
cluster_manager varchar(128), -- the engine cluster manager
|
||||
engine_open_time bigint, -- the engine open time
|
||||
engine_id varchar(128), -- the engine application id
|
||||
engine_name clob, -- the engine application name
|
||||
engine_url varchar(1024), -- the engine tracking url
|
||||
engine_state varchar(32), -- the engine application state
|
||||
engine_error clob, -- the engine application diagnose
|
||||
end_time bigint, -- the metadata end time
|
||||
priority int NOT NULL DEFAULT 10, -- the application priority, high value means high priority
|
||||
peer_instance_closed boolean default FALSE -- closed by peer kyuubi instance
|
||||
);
|
||||
|
||||
CREATE UNIQUE INDEX metadata_unique_identifier_index ON metadata(identifier);
|
||||
|
||||
CREATE INDEX metadata_user_name_index ON metadata(user_name);
|
||||
|
||||
CREATE INDEX metadata_engine_type_index ON metadata(engine_type);
|
||||
|
||||
CREATE INDEX metadata_create_time_index ON metadata(create_time);
|
||||
|
||||
CREATE INDEX metadata_priority_create_time_index ON metadata(priority, create_time);
|
||||
@ -1,2 +0,0 @@
|
||||
RUN '001-KYUUBI-3967.derby.sql';
|
||||
RUN '002-KYUUBI-4119.derby.sql';
|
||||
@ -1,2 +0,0 @@
|
||||
RUN '003-KYUUBI-5078.derby.sql';
|
||||
RUN '004-KYUUBI-5131.derby.sql';
|
||||
@ -20,5 +20,5 @@ package org.apache.kyuubi.server.metadata.jdbc
|
||||
object DatabaseType extends Enumeration {
|
||||
type DatabaseType = Value
|
||||
|
||||
val DERBY, MYSQL, CUSTOM, SQLITE, POSTGRESQL = Value
|
||||
val MYSQL, CUSTOM, SQLITE, POSTGRESQL = Value
|
||||
}
|
||||
|
||||
@ -55,7 +55,6 @@ class JDBCMetadataStore(conf: KyuubiConf) extends MetadataStore with Logging {
|
||||
}
|
||||
private val driverClass = dbType match {
|
||||
case SQLITE => driverClassOpt.getOrElse("org.sqlite.JDBC")
|
||||
case DERBY => driverClassOpt.getOrElse("org.apache.derby.jdbc.AutoloadedDriver")
|
||||
case MYSQL => driverClassOpt.getOrElse(mysqlDriverClass)
|
||||
case POSTGRESQL => driverClassOpt.getOrElse("org.postgresql.Driver")
|
||||
case CUSTOM => driverClassOpt.getOrElse(
|
||||
@ -63,7 +62,6 @@ class JDBCMetadataStore(conf: KyuubiConf) extends MetadataStore with Logging {
|
||||
}
|
||||
|
||||
private val dialect = dbType match {
|
||||
case DERBY => new DerbyDatabaseDialect
|
||||
case SQLITE => new SQLiteDatabaseDialect
|
||||
case MYSQL => new MySQLDatabaseDialect
|
||||
case POSTGRESQL => new PostgreSQLDatabaseDialect
|
||||
|
||||
@ -51,8 +51,6 @@ object JDBCMetadataStoreConf {
|
||||
val METADATA_STORE_JDBC_DATABASE_TYPE: ConfigEntry[String] =
|
||||
buildConf("kyuubi.metadata.store.jdbc.database.type")
|
||||
.doc("The database type for server jdbc metadata store.<ul>" +
|
||||
" <li>(Deprecated) DERBY: Apache Derby, JDBC driver " +
|
||||
"`org.apache.derby.jdbc.AutoloadedDriver`.</li>" +
|
||||
" <li>SQLITE: SQLite3, JDBC driver `org.sqlite.JDBC`.</li>" +
|
||||
" <li>MYSQL: MySQL, JDBC driver `com.mysql.cj.jdbc.Driver` " +
|
||||
"(fallback `com.mysql.jdbc.Driver`).</li>" +
|
||||
|
||||
@ -21,12 +21,6 @@ trait JdbcDatabaseDialect {
|
||||
def limitClause(limit: Int, offset: Int): String
|
||||
}
|
||||
|
||||
class DerbyDatabaseDialect extends JdbcDatabaseDialect {
|
||||
override def limitClause(limit: Int, offset: Int): String = {
|
||||
s"OFFSET $offset ROWS FETCH NEXT $limit ROWS ONLY"
|
||||
}
|
||||
}
|
||||
|
||||
class GenericDatabaseDialect extends JdbcDatabaseDialect {
|
||||
override def limitClause(limit: Int, offset: Int): String = {
|
||||
s"LIMIT $limit OFFSET $offset"
|
||||
|
||||
@ -53,7 +53,6 @@ class JDBCMetadataStoreSuite extends KyuubiFunSuite {
|
||||
}
|
||||
|
||||
test("test get init schema stream") {
|
||||
assert(jdbcMetadataStore.getInitSchema(DatabaseType.DERBY).isDefined)
|
||||
assert(jdbcMetadataStore.getInitSchema(DatabaseType.MYSQL).isDefined)
|
||||
assert(jdbcMetadataStore.getInitSchema(DatabaseType.POSTGRESQL).isDefined)
|
||||
assert(jdbcMetadataStore.getInitSchema(DatabaseType.CUSTOM).isEmpty)
|
||||
|
||||
Loading…
Reference in New Issue
Block a user