[KYUUBI #5435][INFRA][TEST] Improve Kyuubi On Kubernetes IT

### _Why are the changes needed?_

Improve Kyuubi On Kubernetes IT

Done:
1. Copy spark submit engine log in kyuubi pod to local and upload when test failed.
2. pre install spark image into minikube to avoid image pull error

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request

### _Was this patch authored or co-authored using generative AI tooling?_

No

Closes #5437 from zwangsheng/KYUUBI#5435.

Closes #5435

0cbbafce7 [zwangsheng] add comment
1f1336c59 [zwangsheng] ready
e1c10a6ea [zwangsheng] debug
32759015c [zwangsheng] debug
8e2f1eaf1 [zwangsheng] debug
80eaae30a [zwangsheng] [KYUUBI #5435][NOT_MERGE][TEST] Improve Kyuubi On Kubernetes IT

Authored-by: zwangsheng <binjieyang@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>

.github/workflows/master.yml

@@ -403,6 +403,9 @@ jobs:
           minikube start --cpus 2 --memory 4096 --kubernetes-version=${KUBERNETES_VERSION} --force
           # https://minikube.sigs.k8s.io/docs/handbook/pushing/#7-loading-directly-to-in-cluster-container-runtime
           minikube image load apache/kyuubi:latest
+          # pre-install spark into minikube
+          docker pull apache/spark:3.4.1
+          minikube image load apache/spark:3.4.1
       - name: kubectl pre-check
         run: |
           kubectl get nodes
@@ -425,6 +428,9 @@ jobs:
       - name: Cat kyuubi server log
         if: failure()
         run: kubectl logs kyuubi-test
+      - name: Copy spark engine log from kyuubi pod
+        if: failure()
+        run: kubectl cp kyuubi-test:/opt/kyuubi/work ./target/work
       - name: Cat spark driver log
         if: failure()
         run: |
@@ -437,6 +443,7 @@ jobs:
           name: unit-tests-log-kyuubi-on-k8s-it
           path: |
             **/target/unit-tests.log
+            **/target/work/**
 
   spark-on-k8s-it:
     name: Spark Engine On Kubernetes Integration Test

integration-tests/kyuubi-kubernetes-it/src/test/scala/org/apache/kyuubi/kubernetes/test/deployment/KyuubiOnKubernetesTestsSuite.scala

@@ -54,7 +54,9 @@ class KyuubiOnKubernetesWithSparkTestsBase extends WithKyuubiServerOnKubernetes
     super.connectionConf ++
       Map(
         "spark.master" -> s"k8s://$miniKubeApiMaster",
+        // We should update spark docker image in ./github/workflows/master.yml at the same time
         "spark.kubernetes.container.image" -> "apache/spark:3.4.1",
+        "spark.kubernetes.container.image.pullPolicy" -> "IfNotPresent",
         "spark.executor.memory" -> "512M",
         "spark.driver.memory" -> "1024M",
         "spark.kubernetes.driver.request.cores" -> "250m",