apache/kyuubi
1
0
Fork 0
Code Issues Pull Requests Actions 5 Packages Projects Releases Wiki Activity

[KYUUBI #940] Fix process-logger-capture thread consumes high usage CPU

Browse Source 
<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->

After several benchmarks in the offline environment, kyuubi consumes a lot of CPU.

```
[bigdata]$ jps
3188 Main
6036 Jps
4189 KyuubiServer

[bigdata]$ top
top - 09:34:21 up 23 min,  1 user,  load average: 10.67, 7.85, 4.98
Tasks: 320 total,   1 running, 319 sleeping,   0 stopped,   0 zombie
%Cpu(s): 48.2 us, 23.9 sy,  0.0 ni, 27.4 id,  0.0 wa,  0.5 hi,  0.0 si,  0.0 st
MiB Mem :  15806.0 total,   4738.4 free,   6085.0 used,   4982.5 buff/cache
MiB Swap:      0.0 total,      0.0 free,      0.0 used.   9306.5 avail Mem

    PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND
   4189 bigdata   20   0   10.1g 311272  22484 S 633.3   1.9  48:12.21 java
```
```
Threads:  53 total,   6 running,  47 sleeping,   0 stopped,   0 zombie
%Cpu(s): 31.9 us, 22.8 sy,  0.0 ni, 44.8 id,  0.0 wa,  0.4 hi,  0.0 si,  0.0 st
MiB Mem :  15806.0 total,   4770.5 free,   6052.9 used,   4982.5 buff/cache
MiB Swap:      0.0 total,      0.0 free,      0.0 used.   9338.7 avail Mem

    PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND
   5222 bigdata   20   0   10.1g 311272  22484 R  99.3   1.9  10:00.91 process-logger-
   5044 bigdata   20   0   10.1g 311272  22484 R  99.0   1.9  10:03.98 process-logger-
   4730 bigdata   20   0   10.1g 311272  22484 R  98.3   1.9  10:13.24 process-logger-
   4911 bigdata   20   0   10.1g 311272  22484 R  98.3   1.9  10:06.42 process-logger-
   4523 bigdata   20   0   10.1g 311272  22484 R  98.0   1.9  10:41.55 process-logger-
   5769 bigdata   20   0   10.1g 311272  22484 R  96.7   1.9   2:46.80 process-logger-
```

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [x] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #941 from timothy65535/ky-940.

Closes #940

331b9423 [timothy65535] [KYUUBI #940] Fix process-logger-capture thread consumes high usage CPU

Authored-by: timothy65535 <timothy65535@163.com>
Signed-off-by: ulysses-you <ulyssesyou18@gmail.com>
This commit is contained in:
timothy65535 2021-08-17 16:59:43 +08:00 committed by ulysses-you
parent 5fb143e50b
commit f21e118a44
1 changed files with 16 additions and 13 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

29
kyuubi-server/src/main/scala/org/apache/kyuubi/engine/ProcBuilder.scala
View File

@ -110,24 +110,27 @@ trait ProcBuilder {
val redirect: Runnable = { () =>
try {
val maxErrorSize = conf.get(KyuubiConf.ENGINE_ERROR_MAX_SIZE)
var line: String = reader.readLine
while (true) {
if (containsIgnoreCase(line, "Exception:") &&
if (reader.ready()) {
var line: String = reader.readLine
if (containsIgnoreCase(line, "Exception:") &&
!line.contains("at ") && !line.startsWith("Caused by:")) {
val sb = new StringBuilder(line)
error = KyuubiSQLException(sb.toString() + s"\n See more: $engineLog")
line = reader.readLine()
while (sb.length < maxErrorSize && line != null &&
(line.startsWith("\tat ") || line.startsWith("Caused by: "))) {
sb.append("\n" + line)
val sb = new StringBuilder(line)
error = KyuubiSQLException(sb.toString() + s"\n See more: $engineLog")
line = reader.readLine()
}
while (sb.length < maxErrorSize && line != null &&
(line.startsWith("\tat ") || line.startsWith("Caused by: "))) {
sb.append("\n" + line)
line = reader.readLine()
}
error = KyuubiSQLException(sb.toString() + s"\n See more: $engineLog")
} else if (line != null) {
lastRowOfLog = line
error = KyuubiSQLException(sb.toString() + s"\n See more: $engineLog")
} else if (line != null) {
lastRowOfLog = line
}
} else {
Thread.sleep(300)
}
line = reader.readLine()
}
} catch {
case _: IOException =>