Angerszhuuuu
5edb21d210
[CELEBORN-168][FOLLOWUP] Device metrics should use long value and add size unit in metric name ( #1143 )
...
* [CELEBORN-168][FOLLOWUP] Device metrics should use long value and add size unit in metric name
2023-01-05 11:45:19 +08:00
nafiy
3e80cf2b87
[CELEBORN-168][FEATURE] Add disk usage related metrics for Worker ( #1127 )
2023-01-05 10:35:51 +08:00
Angerszhuuuu
425e31797c
[CELEBORN-182][BUG] StorageManager should not delete shuffle file when enable graceful shutdown ( #1126 )
2022-12-30 18:13:36 +08:00
Angerszhuuuu
7d7192af14
[CELEBORN-179][BUG] Repeat remove expired shuffle throw NPE ( #1124 )
2022-12-29 15:47:05 +08:00
Angerszhuuuu
6411fe71b1
[CELEBORN-178][BUG] Default registered flag should be false, not null ( #1123 )
2022-12-29 15:24:09 +08:00
nafiy
77cb7a0477
[CELEBORN-169][REFACTOR] Extract ObservedDevice out from LocalDeviceMonitor ( #1113 )
...
* [CELEBORN-169][REFACTOR] Extract ObservedDevice out from LocalDeviceMonitor
2022-12-28 14:29:00 +08:00
Ethan Feng
5aa959a335
[CELEBORN-157] Change prefix of configurations to celeborn. ( #1104 )
2022-12-21 15:17:28 +08:00
nafiy
f13dfb7421
[CELEBORN-113][FEATURE] Add metrics to monitor non-critical error number on local device ( #1100 )
2022-12-20 22:30:55 +08:00
Keyong Zhou
2f0682265e
[CELEBORN-119] Add timeout for pushdata ( #1097 )
2022-12-20 20:40:42 +08:00
zky.zhoukeyong
3a4fd04f33
Revert "[CELEBORN-120] MapPartitionFileWriter completes some functions to support mappartition ( #1062 )"
...
This reverts commit aa5a4d8c41
2022-12-17 11:10:54 +08:00
Ethan Feng
39394526a8
[CELEBORN-142]Keep committed partition locations semantic consistent when commit files on HDFS. ( #1091 )
2022-12-16 19:02:02 +08:00
zhongqiangczq
aa5a4d8c41
[CELEBORN-120] MapPartitionFileWriter completes some functions to support mappartition ( #1062 )
2022-12-16 18:40:22 +08:00
nafiy
c931663e5f
[CELEBORN-110][REFACTOR] Notify critical error after collecting a certain number of non-critical error ( #1055 )
2022-12-16 15:47:36 +08:00
Ethan Feng
65cb36c002
[CELEBORN-83][FOLLOWUP] Fix various bugs when using HDFS as storage. ( #1065 )
2022-12-15 15:20:29 +08:00
Cheng Pan
ec371c0026
[CELEBORN-132] ShuffleClient should not implement Cloneable ( #1077 )
2022-12-14 10:04:39 +08:00
Keyong Zhou
5a3d397781
[CELEBORN-130] Correct CommitFilesTime metric ( #1073 )
2022-12-13 20:02:21 +08:00
zhongqiangczq
97991a3404
[CELEBORN-126] Fileinfo adds member bufferSize ( #1068 )
2022-12-13 16:36:26 +08:00
zhongqiangczq
edf85de8f6
[CELEBORN-123] PushDataHandler handleRpcRequestCore fix bug about val isMaster ( #1063 )
2022-12-12 15:51:58 +08:00
zhongqiangczq
c7258cfc03
[CELEBORN-103] add handleMapPartitionPushData to support mappartition ( #1048 )
2022-12-08 11:22:43 +08:00
zhongqiangczq
ea1c630173
[CELEBORN-80] FileWriter supports MapPartition ( #1025 )
2022-12-08 10:46:26 +08:00
Ethan Feng
acfaf59ab3
[CELEBORN-91] Refactor memory tracker to support read buffer. ( #1038 )
...
* [CELEBORN-91] Refactor memory tracker to support read buffer.
2022-12-05 15:38:43 +08:00
zhongqiangczq
b262591da8
[CELEBORN-71] pushdatahandler supports mappartition write: handshake/regionstart/regionfinish ( #1013 )
2022-12-05 13:05:35 +08:00
Binjie Yang
d6ee3c18bc
[CELEBORN-98][IMPROVEMENT] Remove unreachable code block in master/work arguments ( #1042 )
2022-12-02 22:53:28 +08:00
Angerszhuuuu
fc5ca42c14
[CELEBORN-96][REFACTOR] PushMergedData return partition not found use same code path ( #1039 )
2022-12-02 14:09:00 +08:00
Ethan Feng
dd02070e4b
[CELEBORN-83] Fix various bug when using HDFS as storage.
...
1. fix incompatibility between Hadoop 2 and Hadoop 3.
2. fix hdfs writer will never be called when there are no healthy disks.
3. fix an NPE when HDFS file writer close.
2022-11-30 19:33:18 +08:00
Ethan Feng
02e446284d
[CELEBORN-74] Device monitor should respect storage dir configured usable space ( #1023 )
2022-11-29 17:10:18 +08:00
Angerszhuuuu
d26e73209b
[CELEBORN-76] Support batch commit hard split partition before stage end
2022-11-29 13:09:01 +08:00
Angerszhuuuu
c8e5315b9c
[CELEBORN-23][FOLLOWUP] Both master and slave data should return HARD_SPLIT during shutdown ( #1018 )
2022-11-28 22:05:07 +08:00
Keyong Zhou
61e04b77fd
[CELEBORN-70][FOLLOWUP] Add epoch for each commitFiles request. ( #1015 )
...
* [CELEBORN-70][FOLLOWUP] Add epoch for each commitFiles request. Address comments.
2022-11-28 14:08:20 +08:00
Ethan Feng
cfa9b7f700
[CELEBORN-18] Refactor stream manager to distinguish map partition and reduce partition. ( #997 )
2022-11-28 12:02:38 +08:00
Keyong Zhou
d381df71f8
[CELEBORN-70] Add epoch for each commitFiles request ( #1012 )
2022-11-27 21:05:14 +08:00
Keyong Zhou
f8bb2cd47d
[CELEBORN-12]Retry on CommitFile request ( #1011 )
2022-11-26 20:56:24 +08:00
Keyong Zhou
9214b82181
[CELEBORN-68] Client might fetch incorrect data chunk ( #1010 )
2022-11-26 18:06:06 +08:00
Ethan Feng
93dbf3f8b1
[CELEBORN-67] Revert "Fix fetch incorrect data chunk" related commits ( #1006 )
...
* Revert "[CELEBORN-50][FOLLOWUP] Channel inactive may cause new client use old stream id to fetch data (#999 )"
This reverts commit 1e8f6dc5e8#1000 )"
This reverts commit f1c4d675d6#998 )"
This reverts commit 0be4b3399c#995 )"
This reverts commit 2b05228871#926 )"
This reverts commit 6f043f8a#954 )"
This reverts commit 64e8ebf1
2022-11-25 20:57:47 +08:00
Angerszhuuuu
2b05228871
[CELEBORN-47][IMPROVEMENT] Refine logs about tracking fetch chunk ( #995 )
2022-11-23 11:56:10 +08:00
Ethan Feng
ee243f286d
[CELEBORN-4] Add metrics about top disk used apps. ( #985 )
2022-11-22 20:06:36 +08:00
Angerszhuuuu
e12000cb67
[CELEBORN-42][BUG] PushMergedData use wrong call back when partition not found ( #991 )
2022-11-22 18:29:15 +08:00
Angerszhuuuu
5ec278f99a
[ISSUE-987][FEATURE] During worker shutdown, return HARD_SPLIT for all existed partition ( #988 )
2022-11-22 14:29:55 +08:00
zhongqiangczq
7adcb5b933
[CELEBORN-6] [REFACTOR] PushDataHandler code refactor ( #966 )
2022-11-16 11:04:24 +08:00
leesf
0b8376e2c7
Cleanup some code ( #943 )
2022-11-11 13:58:39 +08:00
Ethan Feng
6f043f8ae9
[BUG] Fix fetch incorrect data chunk ( #926 )
2022-11-09 22:31:39 +08:00
leesf
aac68c3571
Rename RssException to CelebornException ( #938 )
2022-11-08 10:08:21 +08:00
leesf
496f44eda4
Shutdown worker if initialized failed. ( #931 )
2022-11-07 19:33:35 +08:00
Angerszhuuuu
99a7b85708
[ISSUE-932][REFACTOR] Device check should not directly reportError ( #933 )
...
* [ISSUE-932][REFACTOR] Device check should not directly reportError
2022-11-07 15:15:08 +08:00
nafiy
11081eac6c
[ISSUE-879][BUG] When notifyError, should destroy corresponding file writers ( #912 )
...
* [ISSUE-879][BUG] When notifyError, should destroy corresponding file writers
2022-11-07 14:01:51 +08:00
Angerszhuuuu
100e0057e8
[ISSUE-921][BUG] Flush Error should report non critical error ( #928 )
2022-11-07 11:56:11 +08:00
leesf
3699683a3b
Fix and migrate some configs ( #927 )
2022-11-07 09:41:38 +08:00
Angerszhuuuu
38e15d89e6
[ISSUE-902][IMPROVEMENT][FOLLOWUP] LifecycleManager should reserve blacklist with irrecoverable status ( #914 )
2022-11-04 15:54:45 +08:00
Angerszhuuuu
ea4ed10e5c
[ISSUE-901][BUG] During worker graceful shutdown, worker should report itself as unavailable and avoid master allocate slots on it. ( #905 )
2022-11-02 16:09:58 +08:00
Zhen Wang
643eb84541
[MINOR] Fix typo ( #898 )
2022-11-01 10:03:15 +08:00
