cann_kb_init fails: multiprocessing.Manager instantiation error (EC0009) → GEInitialize failed

一、问题现象（附报错日志上下文）：
plog 信息如下
```
[ERROR] TUNE(16732,python):2025-09-01-06:58:53.256.763 [pywrapper.cpp:95][CANNKB][Tid:16732]"Traceback (most recent call last):
[ERROR] TUNE(16732,python):2025-09-01-06:58:53.256.770 [pywrapper.cpp:95][CANNKB][Tid:16732]"  File "/usr/local/Ascend/ascend-toolkit/latest/python/site-packages/tbe/common/repository_manager/utils/multiprocess_util.py", line 48, in __init__
[ERROR] TUNE(16732,python):2025-09-01-06:58:53.256.773 [pywrapper.cpp:95][CANNKB][Tid:16732]"  File "/path/to/miniconda3/envs/torchrl/lib/python3.10/multiprocessing/context.py", line 57, in Manager
[ERROR] TUNE(16732,python):2025-09-01-06:58:53.256.776 [pywrapper.cpp:95][CANNKB][Tid:16732]"  File "/path/to/miniconda3/envs/torchrl/lib/python3.10/multiprocessing/managers.py", line 566, in start
[ERROR] TUNE(16732,python):2025-09-01-06:58:53.256.778 [pywrapper.cpp:95][CANNKB][Tid:16732]"  File "/path/to/miniconda3/envs/torchrl/lib/python3.10/multiprocessing/connection.py", line 250, in recv
[ERROR] TUNE(16732,python):2025-09-01-06:58:53.256.781 [pywrapper.cpp:95][CANNKB][Tid:16732]"  File "/path/to/miniconda3/envs/torchrl/lib/python3.10/multiprocessing/connection.py", line 414, in _recv_bytes
[ERROR] TUNE(16732,python):2025-09-01-06:58:53.256.783 [pywrapper.cpp:95][CANNKB][Tid:16732]"  File "/path/to/miniconda3/envs/torchrl/lib/python3.10/multiprocessing/connection.py", line 383, in _recv
[ERROR] TUNE(16732,python):2025-09-01-06:58:53.256.786 [pywrapper.cpp:95][CANNKB][Tid:16732]"EOFError
[ERROR] TUNE(16732,python):2025-09-01-06:58:53.256.788 [pywrapper.cpp:95][CANNKB][Tid:16732]"
[ERROR] TUNE(16732,python):2025-09-01-06:58:53.256.790 [pywrapper.cpp:95][CANNKB][Tid:16732]"Traceback (most recent call last):
[ERROR] TUNE(16732,python):2025-09-01-06:58:53.256.793 [pywrapper.cpp:95][CANNKB][Tid:16732]"  File "/usr/local/Ascend/ascend-toolkit/latest/python/site-packages/tbe/common/repository_manager/interface.py", line 33, in cann_kb_init
[ERROR] TUNE(16732,python):2025-09-01-06:58:53.256.795 [pywrapper.cpp:95][CANNKB][Tid:16732]"  File "/usr/local/Ascend/ascend-toolkit/latest/python/site-packages/tbe/common/repository_manager/route.py", line 54, in wrapper
[ERROR] TUNE(16732,python):2025-09-01-06:58:53.256.797 [pywrapper.cpp:95][CANNKB][Tid:16732]"  File "/usr/local/Ascend/ascend-toolkit/latest/python/site-packages/tbe/common/repository_manager/route.py", line 132, in initialize
[ERROR] TUNE(16732,python):2025-09-01-06:58:53.256.799 [pywrapper.cpp:95][CANNKB][Tid:16732]"  File "/usr/local/Ascend/ascend-toolkit/latest/python/site-packages/tbe/common/repository_manager/utils/multiprocess_util.py", line 51, in __init__
[ERROR] TUNE(16732,python):2025-09-01-06:58:53.256.802 [pywrapper.cpp:95][CANNKB][Tid:16732]"RuntimeError: {'errCode': 'EC0009', 'err_detail': 'Python built-in object multiprocessing.Manager instantiation failed.'}
[ERROR] TUNE(16732,python):2025-09-01-06:58:53.257.189 [py_interface.cpp:181][CANNKB][Tid:16732]"Failed to call cann_kb_init."
[ERROR] TUNE(16732,python):2025-09-01-06:58:53.257.194 [cann_kb_api.cpp:30][CANNKB][Tid:16732]"Run CannKbInit Error!"
[ERROR] TEFUSION(16732,python):2025-09-01-06:58:53.257.199 [python_adapter_manager.cc:94]16732 InitCannKB call CannKbInit failed. res = [6]. init params: [Ascend910B4, 20, ].
[ERROR] TEFUSION(16732,python):2025-09-01-06:58:53.257.238 [python_adapter_manager.cc:47]16732 Initialize AOE Failed to call InitCannKB
[ERROR] FE(16732,python):2025-09-01-06:58:53.257.284 [tbe_op_store_adapter.cc:1889]16732 InitializeTeFusion:"[GraphOpt][InitializeInner][InitTbeFunc] Failed to init tbe."
[ERROR] FE(16732,python):2025-09-01-06:58:53.257.322 [tbe_op_store_adapter.cc:1856]16732 InitializeInner:"[GraphOpt][InitializeInner][InitTeFusion]: Failed to initialize TeFusion."
[ERROR] FE(16732,python):2025-09-01-06:58:53.257.348 [op_store_adapter_manager.cc:79]16732 InitializeAdapter:"[SubGraphOpt][PreCompileOp][InitAdapter] InitializeAdapter adapter [tbe_op_adapter] failed! Ret [4294967295]"
[ERROR] FE(16732,python):2025-09-01-06:58:53.257.425 [op_store_adapter_manager.cc:120]16732 Initialize:"[SubGraphOpt][PreCompileOp][Init] Initialize op store adapter failed, OpsStoreName[tbe-custom]."
[ERROR] FE(16732,python):2025-09-01-06:58:53.257.448 [fusion_manager.cc:115]16732 Initialize:"[FusionMngr][Init] Op store adapter manager init failed."
[ERROR] GE(16732,python):2025-09-01-06:58:53.257.581 [ops_kernel_manager.cc:83]16732 Initialize: ErrorNo: 1343250441(There is no valid so about OpsKernelInfoStore or GraphOptimizer.) [INIT][OPS_KER]PluginManager InvokeAll failed.
[ERROR] GE(16732,python):2025-09-01-06:58:53.257.599 [gelib.cc:258]16732 InnerInitialize: ErrorNo: 1343250441(There is no valid so about OpsKernelInfoStore or GraphOptimizer.) [INIT][OPS_KER][Init][OpsManager]GE ops manager initial failed.
[WARNING] FE(16732,python):2025-09-01-06:58:53.273.610 [fusion_manager.cc:200]16732 Finalize:"Already Finalized, directly return."
[WARNING] FE(16732,python):2025-09-01-06:58:53.273.616 [fusion_manager.cc:200]16732 Finalize:"Already Finalized, directly return."
[ERROR] GE(16732,python):2025-09-01-06:58:53.281.984 [gelib.cc:183]16732 Initialize: ErrorNo: 1343250441(There is no valid so about OpsKernelInfoStore or GraphOptimizer.) [INIT][OPS_KER][Init][GeLib]GeLib initial failed.
[ERROR] GE(16732,python):2025-09-01-06:58:53.282.061 [ge_api.cc:288]16732 GEInitializeImpl: ErrorNo: 1343229953(GEInitialize Failed.) [INIT][OPS_KER][Init][GELib]Failed, error code = 1343250441
[ERROR] GE(16732,python):2025-09-01-06:58:53.282.072 [ge_api.cc:370]16732 GEInitialize: ErrorNo: 4294967295(failed) [INIT][OPS_KER][GEInit] GEInitialize failed, error code:4294967295.
[ERROR] ASCENDCL(16732,python):2025-09-01-06:58:53.282.101 [local_compiler.cpp:76]16732 Init: [INIT][OPS_KER][Initialize][Ge]GEInitialize failed. ge result = 4294967295
[ERROR] ASCENDCL(16732,python):2025-09-01-06:58:53.282.132 [op_compile_service.cpp:73]16732 SetCompileStrategy: [INIT][OPS_KER][Init][Compiler]Init compiler failed
[ERROR] ASCENDCL(16732,python):2025-09-01-06:58:53.282.158 [op_compile_processor.cpp:67]16732 Init: [INIT][OPS_KER][Set][Options]OpCompileProcessor init failed!
[ERROR] ASCENDCL(16732,python):2025-09-01-06:58:53.282.181 [op_compile_processor.cpp:111]16732 SetCompileOpt: [INIT][OPS_KER][Init][Env]init env failed!
```

二、软件版本:
-- CANN 版本 :  CANN 8.1.rc1
-- Pytorch 版本: 2.8.0rc1
--Python 版本 : 3.10.18

三、测试步骤：
尝试在 NPU 运行如下 torchrl 基础脚本
```
import torch

torch.manual_seed(0)

import time

from torchrl.envs import GymEnv, StepCounter, TransformedEnv

env = TransformedEnv(GymEnv("CartPole-v1"), StepCounter())
env.set_seed(0)

from tensordict.nn import TensorDictModule as Mod, TensorDictSequential as Seq

from torchrl.modules import EGreedyModule, MLP, QValueModule

value_mlp = MLP(out_features=env.action_spec.shape[-1], num_cells=[64, 64])
value_net = Mod(value_mlp, in_keys=["observation"], out_keys=["action_value"])
policy = Seq(value_net, QValueModule(spec=env.action_spec))
exploration_module = EGreedyModule(
    env.action_spec, annealing_num_steps=100_000, eps_init=0.5
)
policy_explore = Seq(policy, exploration_module)

from torchrl.collectors import SyncDataCollector
from torchrl.data import LazyTensorStorage, ReplayBuffer

init_rand_steps = 5000
frames_per_batch = 100
optim_steps = 10
collector = SyncDataCollector(
    env,
    policy_explore,
    frames_per_batch=frames_per_batch,
    total_frames=-1,
    init_random_frames=init_rand_steps,
)
rb = ReplayBuffer(storage=LazyTensorStorage(100_000))

from torch.optim import Adam

from torchrl.objectives import DQNLoss, SoftUpdate

loss = DQNLoss(value_network=policy, action_space=env.action_spec, delay_value=True)
optim = Adam(loss.parameters(), lr=0.02)
updater = SoftUpdate(loss, eps=0.99)

from torchrl._utils import logger as torchrl_logger
from torchrl.record import CSVLogger, VideoRecorder

path = "./training_loop"
logger = CSVLogger(exp_name="dqn", log_dir=path, video_format="mp4")
video_recorder = VideoRecorder(logger, tag="video")
record_env = TransformedEnv(
    GymEnv("CartPole-v1", from_pixels=True, pixels_only=False), video_recorder
)

total_count = 0
total_episodes = 0
t0 = time.time()
for i, data in enumerate(collector):
    # Write data in replay buffer
    rb.extend(data)
    max_length = rb[:]["next", "step_count"].max()
    if len(rb) > init_rand_steps:
        # Optim loop (we do several optim steps
        # per batch collected for efficiency)
        for _ in range(optim_steps):
            sample = rb.sample(128)
            loss_vals = loss(sample)
            loss_vals["loss"].backward()
            optim.step()
            optim.zero_grad()
            # Update exploration factor
            exploration_module.step(data.numel())
            # Update target params
            updater.step()
            if i % 10:
                torchrl_logger.info(f"Max num steps: {max_length}, rb length {len(rb)}")
            total_count += data.numel()
            total_episodes += data["next", "done"].sum()
    if max_length > 200:
        break

t1 = time.time()

torchrl_logger.info(
    f"solved after {total_count} steps, {total_episodes} episodes and in {t1-t0}s."
)

```
```
ulimit -n = 1048576

ulimit -u = 1048576

free -h: 1.5 TiB total RAM, only ~24 GiB used
```

Ascend/pytorch
暂停

内容风险标识

评论 (2)

Ascend/pytorch暂停 .gitee-modal { width: 500px !important; }

内容风险标识