kyuubi/externals
Fu Chen d0a7ca4ba8
[KYUUBI #4754] [ARROW] Use KyuubiArrowConveters#toBatchIterator instead of ArrowConveters#toBatchIterator
### _Why are the changes needed?_

to adapt Spark 3.4

the signature of function `ArrowConveters#toBatchIterator` is changed in https://github.com/apache/spark/pull/38618 (since Spark 3.4)

Before Spark 3.4:

```
private[sql] def toBatchIterator(
    rowIter: Iterator[InternalRow],
    schema: StructType,
    maxRecordsPerBatch: Int,
    timeZoneId: String,
    context: TaskContext): Iterator[Array[Byte]]
```

Spark 3.4

```
private[sql] def toBatchIterator(
    rowIter: Iterator[InternalRow],
    schema: StructType,
    maxRecordsPerBatch: Long,
    timeZoneId: String,
    context: TaskContext): ArrowBatchIterator
```

the return type is changed from `Iterator[Array[Byte]]` to `ArrowBatchIterator`

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [x] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #4754 from cfmcgrady/arrow-spark34.

Closes #4754

a3c58d0ad [Fu Chen] fix ci
32704c577 [Fu Chen] Revert "fix ci"
e32311a03 [Fu Chen] fix ci
a76af6209 [Cheng Pan] Update externals/kyuubi-spark-sql-engine/src/main/scala/org/apache/spark/sql/kyuubi/SparkDatasetHelper.scala
453b6a6b8 [Cheng Pan] Update externals/kyuubi-spark-sql-engine/src/main/scala/org/apache/spark/sql/kyuubi/SparkDatasetHelper.scala
74a9f7a9d [Cheng Pan] Update externals/kyuubi-spark-sql-engine/src/main/scala/org/apache/spark/sql/kyuubi/SparkDatasetHelper.scala
4ce5844af [Fu Chen] adapt Spark 3.4

Lead-authored-by: Fu Chen <cfmcgrady@gmail.com>
Co-authored-by: Cheng Pan <pan3793@gmail.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
2023-04-23 17:39:59 +08:00
..
kyuubi-chat-engine [KYUUBI #4739] Add operation lock instead of locking state Enumeration 2023-04-21 10:49:21 +08:00
kyuubi-download [KYUUBI #4348] [INFRA] Cache engine archives in CI jobs for maven download plugin 2023-02-18 22:34:02 +08:00
kyuubi-flink-sql-engine [KYUUBI #4739] Add operation lock instead of locking state Enumeration 2023-04-21 10:49:21 +08:00
kyuubi-hive-sql-engine [KYUUBI #4739] Add operation lock instead of locking state Enumeration 2023-04-21 10:49:21 +08:00
kyuubi-jdbc-engine [KYUUBI #4739] Add operation lock instead of locking state Enumeration 2023-04-21 10:49:21 +08:00
kyuubi-spark-sql-engine [KYUUBI #4754] [ARROW] Use KyuubiArrowConveters#toBatchIterator instead of ArrowConveters#toBatchIterator 2023-04-23 17:39:59 +08:00
kyuubi-trino-engine [KYUUBI #4739] Add operation lock instead of locking state Enumeration 2023-04-21 10:49:21 +08:00