[KYUUBI #3795][PYSPARK] Python 3.8+ support

### _Why are the changes needed?_

to close #3795

Python ast api is changed after python 3.8, see https://github.com/ipython/ipython/pull/11593

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #3797 from cfmcgrady/kyuubi-3795.

Closes #3795

44edcea1 [Fu Chen] Revert "debug python version"
f8171b00 [Fu Chen] Revert "print ga debug info"
16fde4ee [Fu Chen] debug python version
331602a8 [Fu Chen] print ga debug info
66eeb3fb [Fu Chen] python 3.8+ support

Authored-by: Fu Chen <cfmcgrady@gmail.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
This commit is contained in:
Fu Chen 2022-11-11 11:27:23 +08:00 committed by Cheng Pan
parent a58816609b
commit 7295049c9f
No known key found for this signature in database
GPG Key ID: 8001952629BCC75D

View File

@ -24,6 +24,15 @@ import traceback
import re
import os
# ast api is changed after python 3.8, see https://github.com/ipython/ipython/pull/11593
if sys.version_info > (3,8):
from ast import Module
else :
# mock the new API, ignore second argument
# see https://github.com/ipython/ipython/issues/11590
from ast import Module as OriginalModule
Module = lambda nodelist, type_ignores: OriginalModule(nodelist)
TOP_FRAME_REGEX = re.compile(r'\s*File "<stdin>".*in <module>')
global_dict = {}
@ -37,7 +46,7 @@ class NormalNode(object):
try:
for node in to_run_exec:
mod = ast.Module([node])
mod = Module([node], [])
code = compile(mod, '<stdin>', 'exec')
exec(code, global_dict)