apache/kyuubi
1
0
Fork 0
Code Issues Pull Requests Actions 5 Packages Projects Releases Wiki Activity
f88c4cbbd5
kyuubi/docs
History
odone 9716548380
[KYUUBI #2282] Add lineage records for sql statement execution in Kyuubi engine logs 
### _Why are the changes needed?_

Lineage information:
```
col0 -> (table.a, table.b)
col1 -> (table.c, table.a)
```

SQL lineage logger JSON format example.
**SQL:**
```
select a as col0, b as col1 from test_table0
```
**Lineage:**
```
{
   "inputTables": ["default.test_table0"],
   "outputTables": [],
   "columnLineage": [{
      "column": "col0",
      "originalColumns": ["default.test_table0.a"]
   }, {
      "column": "col1",
      "originalColumns": ["default.test_table0.b"]
   }]
}
```

Currently supported column lineage for spark `Command` and `Query` `TreeNode`:

### Query
- `Select`

### Command
- `CreateDataSourceTableAsSelectCommand`
- `CreateHiveTableAsSelectCommand`
- `OptimizedCreateHiveTableAsSelectCommand`
- `CreateTableAsSelect`
- `ReplaceTableAsSelect`
- `InsertIntoDataSourceCommand`
- `InsertIntoHadoopFsRelationCommand`
- `InsertIntoDataSourceDirCommand`
- `InsertIntoHiveDirCommand`
- `InsertIntoHiveTable`

### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #3185 from iodone/kyuubi-2282.

Closes #2282

002c6d61 [odone] delete spark-sql-engine test for lineage
e1728a79 [odone] update lineage entity schema
de2a3e9a [odone] change kyuubi-spark-listener module to kyuubi-spark-lineage module
9258125e [odone] optimize lineage output
834669ed [odone] delete engine lineage parse
d9c7a3dc [odone] add spark listener to support lineage
4bae8c2f [odone] update for code cleaning
32b3392b [odone] update for review
fe09e478 [odone] add some test

Authored-by: odone <odone.zhang@gmail.com>
Signed-off-by: Kent Yao <yao@apache.org>
2022-08-24 14:11:35 +08:00
..
_static/css [KYUUBI #2764] [DOCS] Fix tables in docs being coverd by right toc sidebar 2022-05-27 17:21:42 +08:00
appendix [KYUUBI #2751] [DOC] Replace sphinx_rtd_theme with sphinx_book_theme 2022-05-27 02:22:36 +08:00
changelog [KYUUBI #2846] Add v1.5.2-incubating changelog 2022-06-10 10:53:52 +08:00
client [KYUUBI #3101] [Subtask][#3100] Build the content for extension points documentation 2022-07-21 15:37:19 +08:00
community [KYUUBI #3268] Update release docs for status file news 2022-08-18 17:55:13 +08:00
connector [KYUUBI #3252] Fix the problem that |release| in the document was not replaced correctly 2022-08-17 13:44:40 +08:00
deployment [KYUUBI #3128] Support CostMode for PlanOnlyStatement 2022-08-24 14:06:40 +08:00
develop_tools [KYUUBI #3241][DOCS] Update Develop Tools / Building a Runnable Distribution 2022-08-18 15:38:09 +08:00
extensions [KYUUBI #2282] Add lineage records for sql statement execution in Kyuubi engine logs 2022-08-24 14:11:35 +08:00
imgs [KYUUBI #3060] [Subtask][#3059] Build content of the connector document section 2022-07-14 14:19:13 +08:00
monitor [KYUUBI #3157][DOC] Modify logging doc due to using log4j2 instead of log4j 2022-08-15 21:15:56 +08:00
overview [KYUUBI #2751] [DOC] Replace sphinx_rtd_theme with sphinx_book_theme 2022-05-27 02:22:36 +08:00
quick_start [KYUUBI #3206] Change Flink default version to 1.15 2022-08-11 15:31:59 +08:00
security [KYUUBI #3226][DOC] [FOLLOWUP] Kyuubi authZ only support spark 3.x 2022-08-24 14:05:02 +08:00
tools [KYUUBI #3045][FOLLOWUP] Correct the common options and add docs for kyuubi-admin command 2022-07-15 10:47:49 +08:00
conf.py [KYUUBI #2764] [DOCS] Fix tables in docs being coverd by right toc sidebar 2022-05-27 17:21:42 +08:00
index.rst [KYUUBI #3101] [Subtask][#3100] Build the content for extension points documentation 2022-07-21 15:37:19 +08:00
make.bat [KYUUBI #874] [ASF] ASF Publish 2021-08-16 11:48:21 +08:00
Makefile [KYUUBI #874] [ASF] ASF Publish 2021-08-16 11:48:21 +08:00
requirements.txt [KYUUBI #3126] Using markdown 3.3.7 for kyuubi document build 2022-07-25 14:46:20 +08:00