Commit Graph

1099 Commits

Author SHA1 Message Date
Ethan Feng
3cdc25286d
[CELEBORN-165] Fix ut RetryCommitFilesTest failure. (#1111) 2022-12-22 11:39:40 +08:00
Cheng Pan
bf97a2227b
[CELEBORN-163][BUILD] Rename Flink modules and enable Flink CI (#1110) 2022-12-21 23:54:47 +08:00
Cheng Pan
7105f98829
[CELEBORN-160][BUILD] Spilt CI workflow (#1107) 2022-12-21 23:47:01 +08:00
Cheng Pan
e71e77447f
[CELEBORN-161][BUILD] Disable scaladocs (#1108) 2022-12-21 23:35:52 +08:00
Cheng Pan
8a7204fda3
[CELEBORN-159][INFRA] Upgrade requirements.txt to keep consistent w/ celeborn-website (#1106) 2022-12-21 22:22:41 +08:00
Ethan Feng
5aa959a335
[CELEBORN-157] Change prefix of configurations to celeborn. (#1104) 2022-12-21 15:17:28 +08:00
zhongqiangczq
907364dbf2
[CELEBORN-156] add remoteShuffleResultPartition in flink-plugin (#1103) 2022-12-21 12:22:17 +08:00
nafiy
f13dfb7421
[CELEBORN-113][FEATURE] Add metrics to monitor non-critical error number on local device (#1100) 2022-12-20 22:30:55 +08:00
Keyong Zhou
2f0682265e
[CELEBORN-119] Add timeout for pushdata (#1097) 2022-12-20 20:40:42 +08:00
Cheng Pan
1822896156
[CELEBORN-151][K8S] Celeborn should run as celeborn instead of root in container (#895) 2022-12-20 15:57:14 +08:00
Keyong Zhou
a2dd72f20c
[CELEBORN-155] Wrong TimeUnit for registerShuffleRetryWait in Shuffle… (#1099) 2022-12-19 17:32:18 +08:00
Shuang
13769f0f0a
[CELEBORN-121] Refactor batchHandleCommitPartition (#1089) 2022-12-19 12:39:39 +08:00
zhongqiangczq
d31c94a903
[CELEBORN-154] replace shuffleclient's shutDown to shutdown (#1098) 2022-12-19 09:33:44 +08:00
zky.zhoukeyong
3a4fd04f33 Revert "[CELEBORN-120] MapPartitionFileWriter completes some functions to support mappartition (#1062)"
This reverts commit aa5a4d8c41.
2022-12-17 11:10:54 +08:00
Ethan Feng
39394526a8
[CELEBORN-142]Keep committed partition locations semantic consistent when commit files on HDFS. (#1091) 2022-12-16 19:02:02 +08:00
zhongqiangczq
aa5a4d8c41
[CELEBORN-120] MapPartitionFileWriter completes some functions to support mappartition (#1062) 2022-12-16 18:40:22 +08:00
nafiy
c931663e5f
[CELEBORN-110][REFACTOR] Notify critical error after collecting a certain number of non-critical error (#1055) 2022-12-16 15:47:36 +08:00
Cheng Pan
dc66369973
[CELEBORN-150][BUILD] Reduce binary tarball size by sharing jars (#1095)
* [CELEBORN-150][BUILD] Reduce binary tarball size by sharing jars

* nit

* nit

* docker

* nit

* cp -R
2022-12-16 14:30:17 +08:00
Binjie Yang
853d0df191
[CELEBORN-149] Upload failure CI unit test logs for developer debug (#1094) 2022-12-16 04:31:13 +08:00
nafiy
ddab27a1d7
[CELEBORN-145][REFACTOR] Add reason in CheckQuotaResponse (#1093)
* [CELEBORN-145][REFACTOR] Add reason in CheckQuotaResponse
2022-12-15 18:16:34 +08:00
Ethan Feng
65cb36c002
[CELEBORN-83][FOLLOWUP] Fix various bugs when using HDFS as storage. (#1065) 2022-12-15 15:20:29 +08:00
Ethan Feng
df5ed8ec09
[CELEBORN-144]Lower code coverage requirements. (#1090) 2022-12-15 13:48:47 +08:00
Shuang
e3576e4e7a
[CELEBORN-117] refactor CommitManager, implements M/R Partition Commi… (#1060) 2022-12-15 11:09:59 +08:00
Angerszhuuuu
a341ccb22c
[CELEBORN-140][FOLLOWUP] quota.yaml file not support default value -1 now (#1088) 2022-12-15 10:59:02 +08:00
Binjie Yang
f236b0bb14
[CELEBORN-143][HELM] Exposing pods details to users (#1084)
* Exposing details to users

* fix
2022-12-15 10:12:01 +08:00
nafiy
2e37830a0f
[CELEBORN-139][BUG] Fix read wrong yaml file format when loading config (#1083) 2022-12-14 20:56:04 +08:00
Angerszhuuuu
83c85fc351
[CELEBORN-141][IMPROVEMENT] Refactor error log about quota exceed (#1086) 2022-12-14 18:25:18 +08:00
Angerszhuuuu
ed26356cf7
[CELEBORN-140][IMPROVEMENT] Quota yaml should support byte string (#1085) 2022-12-14 18:24:19 +08:00
Binjie Yang
f8970a120a
[CELEBORN-138] Respect users' configurations which are set in values.yaml 2022-12-14 16:24:21 +08:00
William Song
ce86e11d50
[CELEBORN-133] Improve snapshot loading (#1078)
Co-authored-by: Cheng Pan <pan3793@gmail.com>
2022-12-14 14:26:31 +08:00
Cheng Pan
17f45453e3
[CELEBORN-135][INFRA] Remove unused GitHub slash workflow (#1080) 2022-12-14 10:13:13 +08:00
zhongqiangczq
5ae2275e3e
[CELEBORN-125] RemoteShuffleOutputGate completes functions to support shuffle write (#1067) 2022-12-14 10:10:21 +08:00
Cheng Pan
339d585469
[CELEBORN-134][INFRA] Enhance PULL_REQUEST_TEMPLATE (#1079) 2022-12-14 10:07:56 +08:00
Cheng Pan
ec371c0026
[CELEBORN-132] ShuffleClient should not implement Cloneable (#1077) 2022-12-14 10:04:39 +08:00
Ethan Feng
dcc242e650
[CELEBORN-131]Notify pull request event to jira. (#1074) 2022-12-13 20:21:59 +08:00
Cheng Pan
c9e3221075
[(ELEBORN-127] Bump Ratis 2.4.1 (#1070) 2022-12-13 20:09:52 +08:00
Keyong Zhou
5a3d397781
[CELEBORN-130] Correct CommitFilesTime metric (#1073) 2022-12-13 20:02:21 +08:00
Cheng Pan
29e0718602
[CELEBORN-81][FOLLOWUP] Rename and reorder maven properties (#1071) 2022-12-13 19:17:51 +08:00
Cheng Pan
41a2672b58
[CELEBORN-128] Bump Log4j2 to 2.17.2 (#1072) 2022-12-13 18:42:44 +08:00
zhongqiangczq
97991a3404
[CELEBORN-126] Fileinfo adds member bufferSize (#1068) 2022-12-13 16:36:26 +08:00
zhongqiangczq
edf85de8f6
[CELEBORN-123] PushDataHandler handleRpcRequestCore fix bug about val isMaster (#1063) 2022-12-12 15:51:58 +08:00
Ethan Feng
84d67d019d
[Celeborn-81][FOLLOWUP]Correct sure fire plugin args. (#1059) 2022-12-09 15:06:08 +08:00
Angerszhuuuu
dac2ba6b40
[CELEBORN-114][REFACTOR] Keep same log code in spark2/spark3 of quota exceed (#1058) 2022-12-09 12:13:01 +08:00
Ethan Feng
e68954dbae
[CELEBORN-81] Add codecov. (#1026) 2022-12-09 12:11:45 +08:00
Angerszhuuuu
c924a4ff0d
[CELEBORN-61][CELEBORN-62][FEATURE] Shuffle client support slow start, congestion avoidance and congestion control (#1052) 2022-12-08 12:41:34 +08:00
zhongqiangczq
60f6f87832
[CELEBORN-11] ShuffleClient supports MapPartition shuffle write:pushdata (#1036) 2022-12-08 12:31:47 +08:00
nafiy
426918c1cd
[CELEBORN-109][FEATURE] Add new DiskStatus to represent critical error (#1054) 2022-12-08 11:34:58 +08:00
zhongqiangczq
d3d40f730c
[CELEBORN-106] flink-plugin supports shufflewrite:OutputGate (#1051) 2022-12-08 11:24:37 +08:00
zhongqiangczq
c7258cfc03
[CELEBORN-103] add handleMapPartitionPushData to support mappartition (#1048) 2022-12-08 11:22:43 +08:00
zhongqiangczq
ea1c630173
[CELEBORN-80] FileWriter supports MapPartition (#1025) 2022-12-08 10:46:26 +08:00