Commit Graph

53 Commits

Author SHA1 Message Date
AngersZhuuuu
e1ece6123d
[ISSUE-364][FEATURE] Worker should reject ReserveSlots request when shutting down (#365) 2022-08-17 21:12:53 +08:00
Ethan Feng
b3acb740e9
[Feature] choose usable disk while reserving slots. (#372) 2022-08-17 15:56:20 +08:00
Keyong Zhou
8207b6bff7
[ISSUE-368] Not split if file size is smaller than minimum threshold; fix concurrent issues (#369) 2022-08-16 23:15:43 +08:00
Cheng Pan
f1f4b894af
Build: Enhance build system (#349) 2022-08-15 14:59:01 +08:00
AngersZhuuuu
9477b9cf24
[ISSUE-321][Feature] Worker trigger shutdown hook when run command stop-worker.sh (#324) 2022-08-15 12:20:35 +08:00
Keyong Zhou
937ac54e7c
[ISSUE-351] Trigger split when reaching disk space limitation (#356) 2022-08-15 00:24:25 +08:00
Keyong Zhou
e372464187
[ISSUE-273][FOLLOW-UP] Only count avgTime if flush count exceeds mini… (#355) 2022-08-14 22:56:25 +08:00
Keyong Zhou
2b5e997ec3
[ISSUE-353] 1.Only remove dirOperator for IOHang bug; 2.Fix NPE when convert DestroyResponse to PB (#354) 2022-08-14 19:34:23 +08:00
Keyong Zhou
c2672c2d9d
[ISSUE-273][FOLLOW-UP] 1.Heartbeat use workerInfo's diskInfos instead… (#352) 2022-08-14 16:54:08 +08:00
Keyong Zhou
20a3ba4e56
[ISSUE-273][FOLLOW-UP] Merge MountInfo with DiskInfo (#348) 2022-08-13 22:58:13 +08:00
Keyong Zhou
89903d162e
[ISSUE-273][FOLLOW-UP] Add minimum partition size threshold for parti… (#346) 2022-08-13 18:29:43 +08:00
Keyong Zhou
766b3118d7
[ISSUE-273][FOLLOW-UP] Fix device stat file not found exception (#345) 2022-08-13 16:47:25 +08:00
Cheng Pan
08f647ea3e
Remove unexpected log4j configuration file from package jar and minor improvement (#331) 2022-08-13 11:10:48 +08:00
AngersZhuuuu
221b068773
[ISSUE-327][REFACTOR] Remove unused logAvailableFlushBuffersTask (#328) 2022-08-13 11:09:59 +08:00
Ethan Feng
f3bcb7f6a8
[ISSUE-146]update slots distribution mechanism (#273) 2022-08-12 23:38:19 +08:00
Keyong Zhou
d19b475500
[ISSUE-325] Log error msg for ChunkFetchFailure (#326) 2022-08-12 00:18:52 +08:00
Cheng Pan
efdf283918
ShuffleBlockInfo should be static inner class (#301) 2022-08-04 21:50:58 +08:00
Ethan Feng
594feab279
fix unexpected index update. (#295) 2022-08-02 18:16:26 +08:00
dxheming
8e3f48ec12
Refactor deprecated netty ConcurrentSet (#285) 2022-07-27 20:35:46 +08:00
Ethan Feng
82f0475d9b
[BUG] Fix reserve slots failed due to take buffer stuck (#283)
* [BUG] Fix reserve slots failed due to take buffer stuck
2022-07-26 18:20:39 +08:00
AngersZhuuuu
fe17914942
Refactor pom import issue (#277) 2022-07-25 17:49:55 +08:00
Keyong Zhou
e11af5d948
Support passed-in buffer supplier for FrameDecoder (#278) 2022-07-25 16:46:29 +08:00
Ethan Feng
cb42b2fa5c
[BUG] multi-thread flusher causes data inconsistent with chunk offsets (#275) 2022-07-23 11:17:38 +08:00
Keyong Zhou
6442f38a33
[ISSUE-267] Extend API to support more partition types: MapPartition,… (#268) 2022-07-17 16:28:37 +08:00
Keyong Zhou
56a0b9072b
[ISSUE-261] Refine message class hierarchy (#266) 2022-07-16 17:00:09 +08:00
Keyong Zhou
382170ad84
[ISSUE-264] Refine network code (#265) 2022-07-16 16:02:35 +08:00
Keyong Zhou
7da8f64691
[ISSUE-262] Remove unused bootstrap (#263) 2022-07-16 11:01:44 +08:00
Keyong Zhou
691beb7889
[ISSUE-247] Extract PushHandler, FetchHandler, RpcHandler from Worker… (#251) 2022-07-12 11:40:42 +08:00
AngersZhuuuu
f80c86a675
[ISSUE-222] Destroy and DestroyResponse should remove null check (#238) 2022-07-09 15:44:17 +08:00
Keyong Zhou
1b12a2e883
[ISSUE-239] Move prcessFetchChunks into ChunkFetchHandler (#243) 2022-07-09 14:21:18 +08:00
Keyong Zhou
07d7f99964
[ISSUE-233] Remove unused FlushBuffer and MinimalByteBuf (#242) 2022-07-09 12:09:10 +08:00
AngersZhuuuu
c28eeb078c
[ISSUE-222] CommitFiles and CommitFilesResponse should remove null check (#237) 2022-07-08 22:32:54 +08:00
Keyong Zhou
f0b0030898
[ISSUE-216] Support sized chunk fetch request (#218) 2022-07-08 14:10:08 +08:00
AngersZhuuuu
ad6458bea9
[ISSUE-217][FEATURE] Re-register worker after worker lost (#219) 2022-07-05 20:51:12 +08:00
nafiy
6f8fb8747f
Modify argument class and add config (#212) 2022-07-01 23:17:24 +08:00
Keyong Zhou
49f2a00943
[ISSUE-208] Refine log levels (#210) 2022-07-01 14:57:30 +08:00
dxheming
7f6e228378
[ISSUE-203][BUG] fix rss.master.address not work in rss-worker's configuration file issue (#204) 2022-06-30 22:59:47 +08:00
AngersZhuuuu
909e8b2f53
[ISSUE-190][BUG] After WorkerLost, response to worker heartbeat RPC to, then worker can clean the data. (#192) 2022-06-29 22:25:29 +08:00
Ethan Feng
9e7e38dcbd
[Feature] Multithread disk flusher (#182) 2022-06-29 15:03:02 +08:00
dxheming
566ed18088
fix worker pushserverport replicateserverport conflict (#165) 2022-06-27 15:09:39 +08:00
mingji
d4d8eb3838 update pom version. 2022-06-24 14:28:42 +08:00
Ethan Feng
6811cc22fc
[issue-146] Add storage hint to indicate storage location. (#147) 2022-06-14 15:57:11 +08:00
nafiy
491f89bbb5
[FEATURE]Add metrics source for JVM and CPU (#125)
* Add metrics source for JVM and CPU

* Fix scala style issue
2022-05-30 13:26:54 +08:00
Ethan Feng
409da82964
[Bug]fix stuck under high memory pressure. (#90) 2022-04-14 18:53:39 +08:00
mingji
8ac0167b69 optimize unit test running time. 2022-04-08 15:51:33 +08:00
Ethan Feng
baa2836216
Add metrics: (#85)
1.shuffle fetch send data time.
 2.open stream time.
 3.memory critical count.
2022-04-02 15:05:27 +08:00
Ethan Feng
9ad8254b0a
AQE support. (#67) 2022-04-01 20:19:01 +08:00
AngersZhuuuu
eacb9a1217
Refactor the configuration (#72) 2022-03-11 12:03:43 +08:00
Ethan Feng
356a1952e4
Multi Client Support (#47)
Co-authored-by: zky.zhoukeyong <zky.zhoukeyong@alibaba-inc.com>
2022-01-29 22:28:06 +08:00
Ethan Feng
1d7b59c80a
fix import order. (#49) 2022-01-27 21:23:25 +08:00