From 7b8ca1235c842accb81f195bc0dc9db006cbbf8a Mon Sep 17 00:00:00 2001 From: Cheng Pan Date: Wed, 15 Mar 2023 15:08:11 +0800 Subject: [PATCH] [KYUUBI #4519] Update metadata when batch application state changed ### _Why are the changes needed?_ Kyuubi Server should update the metadata store when the batch application state changes, otherwise the peers see the outdated application state. The following inconsistent happened on a dual Kyuubi Server deployment ``` 23/03/15 11:15:36 batchId: e0b06788-46c0-484e-8648-deb08fc5dbd2, appId: null 23/03/15 11:15:41 batchId: e0b06788-46c0-484e-8648-deb08fc5dbd2, appId: null, appStatus: null 23/03/15 11:15:46 batchId: e0b06788-46c0-484e-8648-deb08fc5dbd2, appId: spark-934bd65c1de14fceb22411986dc5fe99, appStatus: PENDING 23/03/15 11:15:51 batchId: e0b06788-46c0-484e-8648-deb08fc5dbd2, appId: spark-934bd65c1de14fceb22411986dc5fe99, appStatus: PENDING 23/03/15 11:15:56 batchId: e0b06788-46c0-484e-8648-deb08fc5dbd2, appId: spark-934bd65c1de14fceb22411986dc5fe99, appStatus: PENDING 23/03/15 11:16:01 batchId: e0b06788-46c0-484e-8648-deb08fc5dbd2, appId: spark-934bd65c1de14fceb22411986dc5fe99, appStatus: PENDING 23/03/15 11:16:06 batchId: e0b06788-46c0-484e-8648-deb08fc5dbd2, appId: spark-934bd65c1de14fceb22411986dc5fe99, appStatus: PENDING 23/03/15 11:16:11 batchId: e0b06788-46c0-484e-8648-deb08fc5dbd2, appId: spark-934bd65c1de14fceb22411986dc5fe99, appStatus: PENDING 23/03/15 11:16:16 batchId: e0b06788-46c0-484e-8648-deb08fc5dbd2, appId: spark-934bd65c1de14fceb22411986dc5fe99, appStatus: PENDING 23/03/15 11:16:21 batchId: e0b06788-46c0-484e-8648-deb08fc5dbd2, appId: spark-934bd65c1de14fceb22411986dc5fe99, appStatus: RUNNING 23/03/15 11:16:26 batchId: e0b06788-46c0-484e-8648-deb08fc5dbd2, appId: spark-934bd65c1de14fceb22411986dc5fe99, appStatus: PENDING 23/03/15 11:16:31 batchId: e0b06788-46c0-484e-8648-deb08fc5dbd2, appId: spark-934bd65c1de14fceb22411986dc5fe99, appStatus: RUNNING 23/03/15 11:16:37 batchId: e0b06788-46c0-484e-8648-deb08fc5dbd2, appId: spark-934bd65c1de14fceb22411986dc5fe99, appStatus: PENDING 23/03/15 11:16:42 batchId: e0b06788-46c0-484e-8648-deb08fc5dbd2, appId: spark-934bd65c1de14fceb22411986dc5fe99, appStatus: RUNNING 23/03/15 11:16:47 batchId: e0b06788-46c0-484e-8648-deb08fc5dbd2, appId: spark-934bd65c1de14fceb22411986dc5fe99, appStatus: PENDING 23/03/15 11:16:52 batchId: e0b06788-46c0-484e-8648-deb08fc5dbd2, appId: spark-934bd65c1de14fceb22411986dc5fe99, appStatus: RUNNING ``` ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request Closes #4519 from pan3793/batch-metadata. Closes #4519 6b9b3e706 [Cheng Pan] Update metadata when batch application state changed Authored-by: Cheng Pan Signed-off-by: Cheng Pan --- .../scala/org/apache/kyuubi/operation/BatchJobSubmission.scala | 1 + 1 file changed, 1 insertion(+) diff --git a/kyuubi-server/src/main/scala/org/apache/kyuubi/operation/BatchJobSubmission.scala b/kyuubi-server/src/main/scala/org/apache/kyuubi/operation/BatchJobSubmission.scala index 883afcf58..3cbb16907 100644 --- a/kyuubi-server/src/main/scala/org/apache/kyuubi/operation/BatchJobSubmission.scala +++ b/kyuubi-server/src/main/scala/org/apache/kyuubi/operation/BatchJobSubmission.scala @@ -300,6 +300,7 @@ class BatchJobSubmission( val newApplicationStatus = currentApplicationInfo if (newApplicationStatus.map(_.state) != _applicationInfo.map(_.state)) { _applicationInfo = newApplicationStatus + updateBatchMetadata() info(s"Batch report for $batchId, ${_applicationInfo}") } }