============================= test session starts ============================== platform linux -- Python 3.9.0, pytest-6.2.5, py-1.11.0, pluggy-1.5.0 rootdir: /home/jenkins/mindspore/testcases/testcases/tests/st/test_trainer, configfile: ../../../../../../sault/sault/config/pytest.ini plugins: mock-3.14.0, forked-1.6.0, xdist-1.32.0 collected 0 items / 1 error ==================================== ERRORS ==================================== _______________________ ERROR collecting test_trainer.py _______________________ test_trainer.py:143: in MODEL = GPT2LMHeadModel(MODEL_CONFIG) /home/jenkins/.local/lib/python3.9/site-packages/mindformers/models/gpt2/gpt2.py:70: in __init__ super(GPT2LMHeadModel, self).__init__(config, auto_prefix=True) /home/jenkins/.local/lib/python3.9/site-packages/mindformers/models/modeling_utils.py:257: in __init__ super().__init__(**kwargs) /home/jenkins/.local/lib/python3.9/site-packages/mindspore/nn/cell.py:141: in __init__ init_pipeline() E RuntimeError: Initialize GE failed! E E ---------------------------------------------------- E - Ascend Error Message: E ---------------------------------------------------- E E49999: Inner Error! E E49999: [PID: 142624] 2024-09-21-15:29:42.131.614 Failed to parse return of function init_multi_process_env[FUNC:InitParallelCompilation][FILE:python_adapter_manager.cc][LINE:187] E TraceBack (most recent call last): E Failed to initialize TeConfigInfo. E [GraphOpt][InitializeInner][InitTbeFunc] Failed to init tbe.[FUNC:InitializeInner][FILE:tbe_op_store_adapter.cc][LINE:1721] E [SubGraphOpt][PreCompileOp][InitAdapter] InitializeAdapter adapter [tbe_op_adapter] failed! Ret [4294967295][FUNC:InitializeAdapter][FILE:op_store_adapter_manager.cc][LINE:79] E [SubGraphOpt][PreCompileOp][Init] Initialize op store adapter failed, OpsStoreName[tbe-custom].[FUNC:Initialize][FILE:op_store_adapter_manager.cc][LINE:120] E [FusionMngr][Init] Op store adapter manager init failed.[FUNC:Initialize][FILE:fusion_manager.cc][LINE:117] E PluginManager InvokeAll failed.[FUNC:Initialize][FILE:ops_kernel_manager.cc][LINE:82] E OpsManager initialize failed.[FUNC:InnerInitialize][FILE:gelib.cc][LINE:234] E GELib::InnerInitialize failed.[FUNC:Initialize][FILE:gelib.cc][LINE:162] E GEInitialize failed.[FUNC:GEInitialize][FILE:ge_api.cc][LINE:306] E E (Please search "CANN Common Error Analysis" at https://www.mindspore.cn for error code description) E E ---------------------------------------------------- E - C++ Call Stack: (For framework developers) E ---------------------------------------------------- E mindspore/ccsrc/plugin/device/ascend/hal/hardware/ge_device_context.cc:302 InitGe ------------------------------- Captured stderr -------------------------------- [INFO] DEBUG(142624,ffffab85e010,python3.9):2024-09-21-15:29:33.684.634 [mindspore/ccsrc/debug/data_dump/data_dumper.cc:43] CpuDumpRegister] Register DataDumper for cpu backend. [INFO] GE_ADPT(142624,ffffab85e010,python3.9):2024-09-21-15:29:34.688.985 [mindspore/ccsrc/transform/acl_ir/acl_data_dumper.cc:101] AclDumpRegister] Register AclDataDumper for ascend backend [INFO] CORE(142624,ffffab85e010,python3.9):2024-09-21-15:29:34.693.322 [mindspore/core/utils/ms_context.cc:167] set_backend_policy] ms set context backend policy:ge [INFO] GE_ADPT(142624,ffffab85e010,python3.9):2024-09-21-15:29:34.694.507 [mindspore/ccsrc/transform/symbol/acl_base_symbol.cc:62] LoadAclBaseApiSymbol] Load acl base api success! [INFO] GE_ADPT(142624,ffffab85e010,python3.9):2024-09-21-15:29:34.695.920 [mindspore/ccsrc/transform/symbol/acl_compiler_symbol.cc:40] LoadAclOpCompilerApiSymbol] Load acl op compiler api success! [INFO] GE_ADPT(142624,ffffab85e010,python3.9):2024-09-21-15:29:34.695.995 [mindspore/ccsrc/transform/symbol/acl_mdl_symbol.cc:116] LoadAclMdlApiSymbol] Load acl mdl api success! [INFO] GE_ADPT(142624,ffffab85e010,python3.9):2024-09-21-15:29:34.696.025 [mindspore/ccsrc/transform/symbol/acl_op_symbol.cc:56] LoadAclOpApiSymbol] Load ascend op api success! [INFO] GE_ADPT(142624,ffffab85e010,python3.9):2024-09-21-15:29:34.696.124 [mindspore/ccsrc/transform/symbol/acl_prof_symbol.cc:48] LoadProfApiSymbol] Load acl prof api success! [INFO] GE_ADPT(142624,ffffab85e010,python3.9):2024-09-21-15:29:34.696.149 [mindspore/ccsrc/transform/symbol/acl_rt_allocator_symbol.cc:49] LoadAclAllocatorApiSymbol] Load acl allocator api success! [INFO] GE_ADPT(142624,ffffab85e010,python3.9):2024-09-21-15:29:34.696.219 [mindspore/ccsrc/transform/symbol/acl_rt_symbol.cc:140] LoadAclRtApiSymbol] Load acl rt api success! [INFO] GE_ADPT(142624,ffffab85e010,python3.9):2024-09-21-15:29:34.696.237 [mindspore/ccsrc/transform/symbol/acl_symbol.cc:37] LoadAclApiSymbol] Load acl base api success! [INFO] GE_ADPT(142624,ffffab85e010,python3.9):2024-09-21-15:29:34.697.703 [mindspore/ccsrc/transform/symbol/acl_tdt_symbol.cc:83] LoadAcltdtApiSymbol] Load acl tdt api success! [INFO] GE_ADPT(142624,ffffab85e010,python3.9):2024-09-21-15:29:34.697.722 [mindspore/ccsrc/transform/symbol/symbol_utils.cc:73] LoadAscendApiSymbols] Load ascend api success! [INFO] DEVICE(142624,ffffab85e010,python3.9):2024-09-21-15:29:34.698.613 [mindspore/ccsrc/plugin/device/ascend/hal/hardware/ge_device_context.cc:634] SetContextSocVersion] The soc version :Ascend910B3 [INFO] PIPELINE(142624,ffffab85e010,python3.9):2024-09-21-15:29:34.788.009 [mindspore/ccsrc/pybind_api/ir/log_adapter_py.h:33] PyExceptionInitializer] Set exception handler [INFO] PIPELINE(142624,ffffab85e010,python3.9):2024-09-21-15:29:34.815.914 [mindspore/ccsrc/pipeline/jit/ps/init.cc:218] pybind11_init__c_expression] Start GraphExecutorPy... [INFO] PIPELINE(142624,ffffab85e010,python3.9):2024-09-21-15:29:34.818.634 [mindspore/ccsrc/pipeline/jit/ps/init.cc:321] pybind11_init__c_expression] Start ParallelContext... [INFO] PIPELINE(142624,ffffab85e010,python3.9):2024-09-21-15:29:34.819.348 [mindspore/ccsrc/pipeline/jit/ps/init.cc:444] pybind11_init__c_expression] Start CostModelContext... [INFO] PIPELINE(142624,ffffab85e010,python3.9):2024-09-21-15:29:34.819.777 [mindspore/ccsrc/pipeline/jit/ps/init.cc:546] pybind11_init__c_expression] Start OffloadContext... [INFO] ME(142624:281473559420944,MainProcess):2024-09-21-15:29:34.822.994 [mindspore/run_check/_check_version.py:565] Setting the env `PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION=python` to prevent memory overflow during save or load checkpoint file. [INFO] GE_ADPT(142624,ffffab85e010,python3.9):2024-09-21-15:29:34.823.562 [mindspore/ccsrc/transform/symbol/symbol_utils.cc:59] LoadAscendApiSymbols] Ascend api is already loaded. [INFO] DEVICE(142624,ffffab85e010,python3.9):2024-09-21-15:29:34.823.607 [mindspore/ccsrc/plugin/device/ascend/hal/hardware/ge_device_context.cc:634] SetContextSocVersion] The soc version :Ascend910B3 [INFO] CORE(142624,ffffab85e010,python3.9):2024-09-21-15:29:37.441.688 [mindspore/core/utils/ms_context.cc:308] SetDeviceTargetFromInner] ms set context device target:Ascend [INFO] PARALLEL(142624,ffffab85e010,python3.9):2024-09-21-15:29:37.441.770 [mindspore/ccsrc/frontend/parallel/costmodel_context.cc:30] GetInstance] Create costmodel_context [INFO] CORE(142624,ffffab85e010,python3.9):2024-09-21-15:29:37.441.784 [mindspore/core/utils/ms_context.cc:312] SetDeviceTargetFromInner] Set memory_optimize_level to O0 as default on other device [INFO] MD(142624,ffffab85e010,python3.9):2024-09-21-15:29:37.572.043 [mindspore/ccsrc/minddata/dataset/kernels/image/dvpp/acl_adapter.cc:63] InitPlugin] Cannot dlopen /home/jenkins/.local/lib/python3.9/site-packages/mindspore/./lib/plugin/ascend/libdvpp_utils.so, result = libacl_dvpp.so: cannot open shared object file: No such file or directory, it can be ignored if not running on ascend. [INFO] ME(142624,ffffab85e010,python3.9):2024-09-21-15:29:40.615.818 [mindspore/ccsrc/runtime/hardware/device_context_manager.cc:466] GetDeviceContext] Device context of device Ascend is not created yet. [INFO] ME(142624:281473559420944,MainProcess):2024-09-21-15:29:40.615.916 [mindspore/hal/device.py:155] Backend Ascend is not created yet. [INFO] PIPELINE(142624,ffffab85e010,python3.9):2024-09-21-15:29:40.617.082 [mindspore/ccsrc/pipeline/jit/ps/pipeline.cc:295] RecordInitStatus] Status record: system init. [INFO] CORE(142624,ffffab85e010,python3.9):2024-09-21-15:29:40.617.639 [mindspore/core/utils/ms_context.cc:457] PrintJitLevelAndExecMode] The jit_level is: O0, and enable kernelbykernel executor in the GRAPH mode. [INFO] DEVICE(142624,ffffab85e010,python3.9):2024-09-21-15:29:40.617.665 [mindspore/ccsrc/plugin/device/ascend/hal/hardware/ge_device_context.cc:231] Initialize] The current overflow detection mode is INFNAN. [INFO] DEVICE(142624,ffffab85e010,python3.9):2024-09-21-15:29:40.617.825 [mindspore/ccsrc/plugin/device/ascend/hal/device/ascend_kernel_runtime.cc:629] SetRtDevice] Enter SetRtDevice, current initialize device number:0 [INFO] DEVICE(142624,ffffab85e010,python3.9):2024-09-21-15:29:40.810.571 [mindspore/ccsrc/plugin/device/ascend/hal/device/ascend_stream_manager.cc:110] RegCallback] Register callback thread, stream : 0x30330ec0. [INFO] DEVICE(142624,ffffab85e010,python3.9):2024-09-21-15:29:40.810.602 [mindspore/ccsrc/plugin/device/ascend/hal/device/ascend_kernel_runtime.cc:651] CreateDefaultStream] Create ascend default stream, stream id: 0 [INFO] DEVICE(142624,ffffab85e010,python3.9):2024-09-21-15:29:40.812.144 [mindspore/ccsrc/plugin/device/ascend/hal/device/ascend_stream_manager.cc:110] RegCallback] Register callback thread, stream : 0x3047aee0. [INFO] DEVICE(142624,ffffab85e010,python3.9):2024-09-21-15:29:40.812.662 [mindspore/ccsrc/plugin/device/ascend/hal/device/ascend_stream_manager.cc:130] RegCallback] Register callback thread success, stream : 0x30330ec0. [INFO] DEVICE(142624,ffffab85e010,python3.9):2024-09-21-15:29:40.812.746 [mindspore/ccsrc/plugin/device/ascend/hal/device/ascend_stream_manager.cc:130] RegCallback] Register callback thread success, stream : 0x3047aee0. [INFO] DEVICE(142624,ffffab85e010,python3.9):2024-09-21-15:29:40.812.760 [mindspore/ccsrc/plugin/device/ascend/hal/device/ascend_kernel_runtime.cc:657] CreateDefaultStream] Create ascend communication stream, stream id: 1 [INFO] DEVICE(142624,ffffab85e010,python3.9):2024-09-21-15:29:40.814.527 [mindspore/ccsrc/plugin/device/ascend/hal/device/ascend_stream_manager.cc:110] RegCallback] Register callback thread, stream : 0x305a3c70. [INFO] DEVICE(142624,ffffab85e010,python3.9):2024-09-21-15:29:40.815.195 [mindspore/ccsrc/plugin/device/ascend/hal/device/ascend_stream_manager.cc:130] RegCallback] Register callback thread success, stream : 0x305a3c70. [INFO] DEVICE(142624,ffffab85e010,python3.9):2024-09-21-15:29:40.815.212 [mindspore/ccsrc/plugin/device/ascend/hal/device/ascend_kernel_runtime.cc:663] CreateDefaultStream] Create ascend copy data stream, stream id: 2 [INFO] DEVICE(142624,ffffab85e010,python3.9):2024-09-21-15:29:40.817.042 [mindspore/ccsrc/plugin/device/ascend/hal/device/ascend_stream_manager.cc:110] RegCallback] Register callback thread, stream : 0x306cc6c0. [INFO] DEVICE(142624,ffffab85e010,python3.9):2024-09-21-15:29:40.817.475 [mindspore/ccsrc/plugin/device/ascend/hal/device/ascend_stream_manager.cc:130] RegCallback] Register callback thread success, stream : 0x306cc6c0. [INFO] DEVICE(142624,ffffab85e010,python3.9):2024-09-21-15:29:40.817.493 [mindspore/ccsrc/plugin/device/ascend/hal/device/ascend_kernel_runtime.cc:670] CreateDefaultStream] Create ascend forward Send communication stream, stream id: 3 [INFO] DEVICE(142624,ffffab85e010,python3.9):2024-09-21-15:29:40.819.283 [mindspore/ccsrc/plugin/device/ascend/hal/device/ascend_stream_manager.cc:110] RegCallback] Register callback thread, stream : 0x307f5100. [INFO] DEVICE(142624,ffffab85e010,python3.9):2024-09-21-15:29:40.819.712 [mindspore/ccsrc/plugin/device/ascend/hal/device/ascend_stream_manager.cc:130] RegCallback] Register callback thread success, stream : 0x307f5100. [INFO] DEVICE(142624,ffffab85e010,python3.9):2024-09-21-15:29:40.819.729 [mindspore/ccsrc/plugin/device/ascend/hal/device/ascend_kernel_runtime.cc:676] CreateDefaultStream] Create ascend backward Send communication stream, stream id: 4 [INFO] DEVICE(142624,ffffab85e010,python3.9):2024-09-21-15:29:40.821.503 [mindspore/ccsrc/plugin/device/ascend/hal/device/ascend_stream_manager.cc:110] RegCallback] Register callback thread, stream : 0x3091dbb0. [INFO] DEVICE(142624,ffffab85e010,python3.9):2024-09-21-15:29:40.821.921 [mindspore/ccsrc/plugin/device/ascend/hal/device/ascend_stream_manager.cc:130] RegCallback] Register callback thread success, stream : 0x3091dbb0. [INFO] DEVICE(142624,ffffab85e010,python3.9):2024-09-21-15:29:40.821.958 [mindspore/ccsrc/plugin/device/ascend/hal/device/ascend_kernel_runtime.cc:682] CreateDefaultStream] Create ascend forward Receive communication stream, stream id: 5 [INFO] DEVICE(142624,ffffab85e010,python3.9):2024-09-21-15:29:40.823.733 [mindspore/ccsrc/plugin/device/ascend/hal/device/ascend_stream_manager.cc:110] RegCallback] Register callback thread, stream : 0x30a46690. [INFO] DEVICE(142624,ffffab85e010,python3.9):2024-09-21-15:29:40.824.165 [mindspore/ccsrc/plugin/device/ascend/hal/device/ascend_stream_manager.cc:130] RegCallback] Register callback thread success, stream : 0x30a46690. [INFO] DEVICE(142624,ffffab85e010,python3.9):2024-09-21-15:29:40.824.183 [mindspore/ccsrc/plugin/device/ascend/hal/device/ascend_kernel_runtime.cc:688] CreateDefaultStream] Create ascend backward Receive communication stream, stream id: 6 [INFO] DEVICE(142624,ffffab85e010,python3.9):2024-09-21-15:29:40.824.670 [mindspore/ccsrc/plugin/device/ascend/hal/device/ascend_vmm_adapter.h:57] AscendVmmAdapter] VMM align size is 2097152 [INFO] DEVICE(142624,ffffab85e010,python3.9):2024-09-21-15:29:40.824.723 [mindspore/ccsrc/plugin/device/ascend/hal/device/ascend_vmm_adapter.h:157] CheckVmmDriverVersion] Driver path is /usr/local/Ascend [INFO] DEVICE(142624,ffffab85e010,python3.9):2024-09-21-15:29:40.824.752 [mindspore/ccsrc/plugin/device/ascend/hal/device/ascend_vmm_adapter.h:186] CheckVmmDriverVersion] Driver version is 24.1.rc3.b060, major version is 24 [INFO] DEVICE(142624,ffffab85e010,python3.9):2024-09-21-15:29:40.824.772 [mindspore/ccsrc/plugin/device/ascend/hal/device/ascend_vmm_adapter.h:112] IsVmmEnabled] VMM is enabled by default. [INFO] DEVICE(142624,ffffab85e010,python3.9):2024-09-21-15:29:40.824.784 [mindspore/ccsrc/plugin/device/ascend/hal/device/ascend_memory_adapter.cc:171] Initialize] Device HBM Size:62432M, Device free HBM Size:62091M, Reserved HBM size for Other Components(HCCL/rts/etc.):3891M, Recommend Reserved HBM size for Other Components:3880M, User define MindSpore HBM Size:0G, MindSpore Used HBM Size:58200M. [INFO] DEVICE(142624,ffffab85e010,python3.9):2024-09-21-15:29:40.824.798 [mindspore/ccsrc/plugin/device/ascend/hal/device/ascend_dynamic_mem_adapter.cc:57] Initialize] Ascend Memory Adapter initialize success, Memory Statistics: Device HBM memory size: 62432M MindSpore Used memory size: 58200M Used peak memory usage (without fragments): 0M Actual peak memory usage (with fragments): 0M [INFO] DEVICE(142624,ffffab85e010,python3.9):2024-09-21-15:29:40.824.823 [mindspore/ccsrc/plugin/device/ascend/hal/device/ascend_memory_pool.cc:27] DefaultAscendMemoryPool] DefaultAscendMemoryPool constructed. [INFO] DEVICE(142624,ffffab85e010,python3.9):2024-09-21-15:29:40.826.205 [mindspore/ccsrc/plugin/device/ascend/hal/hardware/ge_device_context.cc:495] SetHcclOptions] Values for hccl options: env_table_file[], simulation_level[], env_rank_id[], env_device_id[0], enable_hccl[0], UseDynamicCluster[0]. [INFO] DEVICE(142624,ffffab85e010,python3.9):2024-09-21-15:29:40.826.223 [mindspore/ccsrc/plugin/device/ascend/hal/hardware/ge_device_context.cc:521] SetHcclOptions] No hccl mode. If use hccl, make sure [RANK_TABLE_FILE,RANK_ID,DEVICE_ID] all be set in ENV. [INFO] DEVICE(142624,ffffab85e010,python3.9):2024-09-21-15:29:40.826.235 [mindspore/ccsrc/plugin/device/ascend/hal/hardware/ge_device_context.cc:394] GetGeOptions] Set ge.exec.jobId to default value 0 [INFO] DEVICE(142624,ffffab85e010,python3.9):2024-09-21-15:29:40.826.245 [mindspore/ccsrc/plugin/device/ascend/hal/hardware/ge_device_context.cc:404] GetGeOptions] Got empty proto lib path, cannot set ge.opsProtoLibPath. [INFO] DEVICE(142624,ffffab85e010,python3.9):2024-09-21-15:29:40.826.256 [mindspore/ccsrc/plugin/device/ascend/hal/hardware/ge_device_context.cc:359] SetAscendConfig] Set GE topo mode to memory-priority. [INFO] DEVICE(142624,ffffab85e010,python3.9):2024-09-21-15:29:40.826.266 [mindspore/ccsrc/plugin/device/ascend/hal/hardware/ge_device_context.cc:362] SetAscendConfig] Set staticMemoryPolicy to default mode 2. [INFO] DEVICE(142624,ffffab85e010,python3.9):2024-09-21-15:29:40.826.276 [mindspore/ccsrc/plugin/device/ascend/hal/hardware/ge_device_context.cc:369] SetAscendConfig] The default value of jit_compile is set to 2. [INFO] DEVICE(142624,ffffab85e010,python3.9):2024-09-21-15:29:40.826.285 [mindspore/ccsrc/plugin/device/ascend/hal/hardware/ge_device_context.cc:333] SetAscendHF32Config] The default value of allow_matmul_hf32 and allow_conv_hf32 are set by CANN. [INFO] DEVICE(142624,ffffab85e010,python3.9):2024-09-21-15:29:40.826.294 [mindspore/ccsrc/plugin/device/ascend/hal/hardware/ge_device_context.cc:342] SetAscendHF32Config] allow_matmul_hf32: , allow_conv_hf32: [CRITICAL] DEVICE(142624,ffffab85e010,python3.9):2024-09-21-15:29:42.179.091 [mindspore/ccsrc/plugin/device/ascend/hal/hardware/ge_device_context.cc:302] InitGe] Initialize GE failed! E49999: Inner Error! E49999: [PID: 142624] 2024-09-21-15:29:42.131.614 Failed to parse return of function init_multi_process_env[FUNC:InitParallelCompilation][FILE:python_adapter_manager.cc][LINE:187] TraceBack (most recent call last): Failed to initialize TeConfigInfo. [GraphOpt][InitializeInner][InitTbeFunc] Failed to init tbe.[FUNC:InitializeInner][FILE:tbe_op_store_adapter.cc][LINE:1721] [SubGraphOpt][PreCompileOp][InitAdapter] InitializeAdapter adapter [tbe_op_adapter] failed! Ret [4294967295][FUNC:InitializeAdapter][FILE:op_store_adapter_manager.cc][LINE:79] [SubGraphOpt][PreCompileOp][Init] Initialize op store adapter failed, OpsStoreName[tbe-custom].[FUNC:Initialize][FILE:op_store_adapter_manager.cc][LINE:120] [FusionMngr][Init] Op store adapter manager init failed.[FUNC:Initialize][FILE:fusion_manager.cc][LINE:117] PluginManager InvokeAll failed.[FUNC:Initialize][FILE:ops_kernel_manager.cc][LINE:82] OpsManager initialize failed.[FUNC:InnerInitialize][FILE:gelib.cc][LINE:234] GELib::InnerInitialize failed.[FUNC:Initialize][FILE:gelib.cc][LINE:162] GEInitialize failed.[FUNC:GEInitialize][FILE:ge_api.cc][LINE:306] (Please search "CANN Common Error Analysis" at https://www.mindspore.cn for error code description) [INFO] DEBUG(142624,ffffab85e010,python3.9):2024-09-21-15:29:42.179.193 [mindspore/ccsrc/pipeline/jit/ps/debug/trace.cc:116] TraceGraphEval] Length of analysis graph stack is empty. [INFO] DEBUG(142624,ffffab85e010,python3.9):2024-09-21-15:29:42.179.217 [mindspore/ccsrc/pipeline/jit/ps/debug/trace.cc:670] GetEvalStackInfo] Get graph analysis information begin [INFO] DEBUG(142624,ffffab85e010,python3.9):2024-09-21-15:29:42.179.236 [mindspore/ccsrc/pipeline/jit/ps/debug/trace.cc:673] GetEvalStackInfo] Length of analysis information stack is empty. =============================== warnings summary =============================== ../../../../../../.local/lib/python3.9/site-packages/numpy/core/getlimits.py:549 /home/jenkins/.local/lib/python3.9/site-packages/numpy/core/getlimits.py:549: UserWarning: The value of the smallest subnormal for type is zero. setattr(self, word, getattr(machar, word).flat[0]) ../../../../../../.local/lib/python3.9/site-packages/numpy/core/getlimits.py:89 /home/jenkins/.local/lib/python3.9/site-packages/numpy/core/getlimits.py:89: UserWarning: The value of the smallest subnormal for type is zero. return self._float_to_str(self.smallest_subnormal) ../../../../../../.local/lib/python3.9/site-packages/numpy/core/getlimits.py:549 /home/jenkins/.local/lib/python3.9/site-packages/numpy/core/getlimits.py:549: UserWarning: The value of the smallest subnormal for type is zero. setattr(self, word, getattr(machar, word).flat[0]) ../../../../../../.local/lib/python3.9/site-packages/numpy/core/getlimits.py:89 /home/jenkins/.local/lib/python3.9/site-packages/numpy/core/getlimits.py:89: UserWarning: The value of the smallest subnormal for type is zero. return self._float_to_str(self.smallest_subnormal) ../../../../../../.local/lib/python3.9/site-packages/mindspore/ops/_op_impl/_custom_op/batchnorm_fold2.py:57 /home/jenkins/.local/lib/python3.9/site-packages/mindspore/ops/_op_impl/_custom_op/batchnorm_fold2.py:57: DeprecationWarning: te_fusion.fusion_manager.fusion_manager.register is deprecated,please replace it with tbe.common.register.register_op_compute @fusion_manager.register("batchnorm_fold2") ../../../../../../.local/lib/python3.9/site-packages/mindspore/ops/_op_impl/_custom_op/batchnorm_fold2_grad.py:56 /home/jenkins/.local/lib/python3.9/site-packages/mindspore/ops/_op_impl/_custom_op/batchnorm_fold2_grad.py:56: DeprecationWarning: te_fusion.fusion_manager.fusion_manager.register is deprecated,please replace it with tbe.common.register.register_op_compute @fusion_manager.register("batchnorm_fold2_grad") ../../../../../../.local/lib/python3.9/site-packages/mindspore/ops/_op_impl/_custom_op/batchnorm_fold2_grad_reduce.py:48 /home/jenkins/.local/lib/python3.9/site-packages/mindspore/ops/_op_impl/_custom_op/batchnorm_fold2_grad_reduce.py:48: DeprecationWarning: te_fusion.fusion_manager.fusion_manager.register is deprecated,please replace it with tbe.common.register.register_op_compute @fusion_manager.register("batchnorm_fold2_grad_reduce") ../../../../../../.local/lib/python3.9/site-packages/mindspore/ops/_op_impl/_custom_op/correction_mul.py:51 /home/jenkins/.local/lib/python3.9/site-packages/mindspore/ops/_op_impl/_custom_op/correction_mul.py:51: DeprecationWarning: te_fusion.fusion_manager.fusion_manager.register is deprecated,please replace it with tbe.common.register.register_op_compute @fusion_manager.register("correction_mul") ../../../../../../.local/lib/python3.9/site-packages/mindspore/ops/_op_impl/_custom_op/correction_mul_grad.py:51 /home/jenkins/.local/lib/python3.9/site-packages/mindspore/ops/_op_impl/_custom_op/correction_mul_grad.py:51: DeprecationWarning: te_fusion.fusion_manager.fusion_manager.register is deprecated,please replace it with tbe.common.register.register_op_compute @fusion_manager.register("correction_mul_grad") ../../../../../../.local/lib/python3.9/site-packages/mindspore/ops/_op_impl/_custom_op/correction_mul_grad.py:143 /home/jenkins/.local/lib/python3.9/site-packages/mindspore/ops/_op_impl/_custom_op/correction_mul_grad.py:143: DeprecationWarning: te_fusion.fusion_manager.fusion_manager.register is deprecated,please replace it with tbe.common.register.register_op_compute @fusion_manager.register("correction_mul_grad_reduce") ../../../../../../.local/lib/python3.9/site-packages/mindspore/ops/_op_impl/_custom_op/fake_learned_scale_quant_perlayer.py:50 /home/jenkins/.local/lib/python3.9/site-packages/mindspore/ops/_op_impl/_custom_op/fake_learned_scale_quant_perlayer.py:50: DeprecationWarning: te_fusion.fusion_manager.fusion_manager.register is deprecated,please replace it with tbe.common.register.register_op_compute @fusion_manager.register("fake_learned_scale_quant_perlayer") ../../../../../../.local/lib/python3.9/site-packages/mindspore/ops/_op_impl/_custom_op/fake_learned_scale_quant_perlayer_grad.py:92 /home/jenkins/.local/lib/python3.9/site-packages/mindspore/ops/_op_impl/_custom_op/fake_learned_scale_quant_perlayer_grad.py:92: DeprecationWarning: te_fusion.fusion_manager.fusion_manager.register is deprecated,please replace it with tbe.common.register.register_op_compute @fusion_manager.register("fake_learned_scale_quant_perlayer_grad_d") ../../../../../../.local/lib/python3.9/site-packages/mindspore/ops/_op_impl/_custom_op/fake_learned_scale_quant_perlayer_grad_reduce.py:49 /home/jenkins/.local/lib/python3.9/site-packages/mindspore/ops/_op_impl/_custom_op/fake_learned_scale_quant_perlayer_grad_reduce.py:49: DeprecationWarning: te_fusion.fusion_manager.fusion_manager.register is deprecated,please replace it with tbe.common.register.register_op_compute @fusion_manager.register("fake_learned_scale_quant_perlayer_grad_d_reduce") ../../../../../../.local/lib/python3.9/site-packages/mindspore/ops/_op_impl/_custom_op/fake_learned_scale_quant_perchannel.py:50 /home/jenkins/.local/lib/python3.9/site-packages/mindspore/ops/_op_impl/_custom_op/fake_learned_scale_quant_perchannel.py:50: DeprecationWarning: te_fusion.fusion_manager.fusion_manager.register is deprecated,please replace it with tbe.common.register.register_op_compute @fusion_manager.register("fake_learned_scale_quant_perchannel") ../../../../../../.local/lib/python3.9/site-packages/mindspore/ops/_op_impl/_custom_op/fake_learned_scale_quant_perchannel_grad.py:91 /home/jenkins/.local/lib/python3.9/site-packages/mindspore/ops/_op_impl/_custom_op/fake_learned_scale_quant_perchannel_grad.py:91: DeprecationWarning: te_fusion.fusion_manager.fusion_manager.register is deprecated,please replace it with tbe.common.register.register_op_compute @fusion_manager.register("fake_learned_scale_quant_perchannel_grad_d") ../../../../../../.local/lib/python3.9/site-packages/mindspore/ops/_op_impl/_custom_op/fake_learned_scale_quant_perchannel_grad_reduce.py:48 /home/jenkins/.local/lib/python3.9/site-packages/mindspore/ops/_op_impl/_custom_op/fake_learned_scale_quant_perchannel_grad_reduce.py:48: DeprecationWarning: te_fusion.fusion_manager.fusion_manager.register is deprecated,please replace it with tbe.common.register.register_op_compute @fusion_manager.register("fake_learned_scale_quant_perchannel_grad_d_reduce") ../../../../../../.local/lib/python3.9/site-packages/mindspore/ops/_op_impl/_custom_op/fake_quant_perchannel.py:52 /home/jenkins/.local/lib/python3.9/site-packages/mindspore/ops/_op_impl/_custom_op/fake_quant_perchannel.py:52: DeprecationWarning: te_fusion.fusion_manager.fusion_manager.register is deprecated,please replace it with tbe.common.register.register_op_compute @fusion_manager.register("fake_quant_perchannel") ../../../../../../.local/lib/python3.9/site-packages/mindspore/ops/_op_impl/_custom_op/fake_quant_perchannel_grad.py:81 /home/jenkins/.local/lib/python3.9/site-packages/mindspore/ops/_op_impl/_custom_op/fake_quant_perchannel_grad.py:81: DeprecationWarning: te_fusion.fusion_manager.fusion_manager.register is deprecated,please replace it with tbe.common.register.register_op_compute @fusion_manager.register("fake_quant_perchannel_grad") ../../../../../../.local/lib/python3.9/site-packages/mindspore/ops/_op_impl/_custom_op/fake_quant_perlayer.py:54 /home/jenkins/.local/lib/python3.9/site-packages/mindspore/ops/_op_impl/_custom_op/fake_quant_perlayer.py:54: DeprecationWarning: te_fusion.fusion_manager.fusion_manager.register is deprecated,please replace it with tbe.common.register.register_op_compute @fusion_manager.register("fake_quant_per_layer") ../../../../../../.local/lib/python3.9/site-packages/mindspore/ops/_op_impl/_custom_op/fake_quant_perlayer_grad.py:81 /home/jenkins/.local/lib/python3.9/site-packages/mindspore/ops/_op_impl/_custom_op/fake_quant_perlayer_grad.py:81: DeprecationWarning: te_fusion.fusion_manager.fusion_manager.register is deprecated,please replace it with tbe.common.register.register_op_compute @fusion_manager.register("fake_quant_per_layer_grad") ../../../../../../.local/lib/python3.9/site-packages/mindspore/ops/_op_impl/_custom_op/minmax_update_perchannel.py:50 /home/jenkins/.local/lib/python3.9/site-packages/mindspore/ops/_op_impl/_custom_op/minmax_update_perchannel.py:50: DeprecationWarning: te_fusion.fusion_manager.fusion_manager.register is deprecated,please replace it with tbe.common.register.register_op_compute @fusion_manager.register("minmax_update_perchannel") ../../../../../../.local/lib/python3.9/site-packages/mindspore/ops/_op_impl/_custom_op/minmax_update_perlayer.py:50 /home/jenkins/.local/lib/python3.9/site-packages/mindspore/ops/_op_impl/_custom_op/minmax_update_perlayer.py:50: DeprecationWarning: te_fusion.fusion_manager.fusion_manager.register is deprecated,please replace it with tbe.common.register.register_op_compute @fusion_manager.register("minmax_update_perlayer") -- Docs: https://docs.pytest.org/en/stable/warnings.html =========================== short test summary info ============================ ERROR test_trainer.py - RuntimeError: Initialize GE failed! !!!!!!!!!!!!!!!!!!!! Interrupted: 1 error during collection !!!!!!!!!!!!!!!!!!!! ===================== no tests collected, 1 error in 9.59s =====================