### _Why are the changes needed?_
Store all jobInfos for each statement in mem.
- First, according to jobInfo, you can get how long did it run, and get which stage took the longest time by stageIds.
- Second, if this job failed that you can get from jobResult field, you can look up which stage cause this situation by stageIds.
Some interfaces:
- KyuubiJobInfo: job's summary info, contains statementId, startTime, endTime, jobResult, stageIds.
- KyuubiStatementListener: singleton pattern, used for getting metrics about job, stage, executor and so on.
- KyuubiStatementMonitor: for storing data in mem and dumpping them to a file when reached threshold.
Test case:
externals/kyuubi-spark-sql-engine/src/test/scala/org/apache/kyuubi/engine/spark/KyuubiStatementMonitorSuite.scala
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request
Closes#814 from zhang1002/branch-1.2_add-jobinfo.
Closes#814
7d8ece23 [张宇翔] 1. Change the log level from INFO to DEBUG 2. Code modify: avoid producing NullPointerException
0ddd83a7 [张宇翔] 1. Remove waitForOperationToComplete(client, opHandle)
bbc36bb3 [张宇翔] 1. Merge statementListener to sqlEngineListener
741fcd0f [张宇翔] Add interval
b61b9406 [张宇翔] Code changes
cb6f92c4 [张宇翔] 1. Remove object-lock 2. Add eventually code
c1a1d732 [张宇翔] 1. Remove volatile for jobInfo
9547d49e [张宇翔] 1. Remove volatile for jobInfo
970c0002 [张宇翔] 1. Add volatile for jobInfo
62ecb53d [张宇翔] 1. Remove kyuubiJobInfoQueue 2. Modify some unit test 3. Add some code annotations
db1de381 [张宇翔] 1. Add some log 2. Add some annotations
8deb61f2 [张宇翔] change some test
87d9c102 [张宇翔] 1. Event tracking: for jonInfo 2. Add unit test
86c49ebc [张宇翔] Merge master branch
a0a99b3a [张宇翔] Merge master branch
f248bef7 [张宇翔] Merge remote-tracking branch 'upstream/master'
5d3b9afb [张宇翔] Code optimization
db3a0b6f [张宇翔] Format changes
08d6d1fc [张宇翔] Modify some annotations
f62f00e9 [张宇翔] Event Tracking: For job
85025193 [张宇翔] Merge branch 'branch-1.2_spark-monitor' into branch-1.2_add-jobinfo
3c8d9af1 [张宇翔] 1. change directory structure 2. code optimizing
b4290bfa [张宇翔] Event tracking: For jobInfo
fe1f7cce [张宇翔] Change directory structure
5ffb54f3 [张宇翔] Add kyuubi-spark-monitor module for nightly.yml
71d33b9a [张宇翔] Add some comment
6b87dff0 [张宇翔] 1. Remove some unused code 2. Add some comment
f364d433 [张宇翔] Format change: Add newline in KyuubiStatementInfo
8bddb202 [张宇翔] 1. Remove the relationship between executionId and operationId 2. Get each state by the function: setState 3. Get this statement's physicalPlan in ExecuteStatement 4. Add sparkUser item 5. Remove java code
f43b3c8f [张宇翔] merge master branch
55522613 [张宇翔] Merge branch 'master' into branch-1.2_spark-monitor
6f0be547 [张宇翔] format change
bbfba274 [张宇翔] Even tracking
5a62f586 [张宇翔] remove some unused conf
b26345ac [张宇翔] Merge master branch and resolve conflict
fdc12bbe [张宇翔] Event tracking: for kStatement 1. Store the relationship between executionId and operationId 2. Store the relationship between operationId and statement 3. Store the relationship between executionId and physicalPlan 4. Store each state and its happen time for this statement: initialized, running, finished
d1676268 [张宇翔] event tracking: for SQLExecutionStart
Authored-by: 张宇翔 <zhang1002@126.com>
Signed-off-by: Kent Yao <yao@apache.org>
e7e77f9277