b8f452692a
### _Why are the changes needed?_ This pr adds a new rule `FinalStageResourceManager` to eagerly kill redundant executors We first get the final stage partition which is the actually required cores, then kill the redundant executors. The priority of kill executors follow: 1. kill executor who is younger than other (The older the JIT works better) 2. kill executor who produces less shuffle data first The reason why add this feature is that, if the previous stage contains lots executors but final stage has less, then the tasks of final stage would be scheduled randomly in all exists executors which may cause resource waste. e.g., each executor only run 1 or 2 tasks but holds 4 or 5 cores. ### _How was this patch tested?_ test manually - test for the kill executor <img width="755" alt="image" src="https://user-images.githubusercontent.com/12025282/227203809-9fe0731c-f97f-40d2-ac7f-b892a2a35289.png"> Closes #4592 from ulysses-you/eagerly-kill-executors. Closes #4592 f35208bfd [ulysses-you] nit ec627ee4f [ulysses-you] nit 28d4230f8 [ulysses-you] address comments f2492cec6 [ulysses-you] nit f44e48451 [ulysses-you] Support eagerly kill redundant executors Authored-by: ulysses-you <ulyssesyou18@gmail.com> Signed-off-by: ulyssesyou <ulyssesyou@apache.org> |
||
|---|---|---|
| .. | ||
| engines | ||
| server | ||
| index.rst | ||
