benchmarks_dynamic example error: result has failed to un-serialize

[jingmouren]

I run rm -r mlruns
then run
python workflow.py run_all
in the benchmarks_dynamic /DDG-DA dir
Error occured as the title

[10719:MainThread] INFO - qlib.timer - [log.py:117] - Time cost: 0.000s | waiting async_log Done
train tasks: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 151/151 [02:06<00:00, 1.19it/s]
calc: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 151/151 [00:07<00:00, 19.29it/s]
[10719:MainThread](2022-06-24 15:05:49,660) ERROR - qlib.workflow - [utils.py:41] - An exception has been raised[BrokenProcessPool: A result has failed to un-serialize. Please ensure that the objects returned by the function are always picklable.].
File "workflow.py", line 259, in
fire.Fire(DDGDA)

[SunsetWolf]

I tried executing python ../baseline/rolling_benchmark.py run_all in the benchmarks_dynamic/DDG-DA folder and didn't find this issue, so I suggest creating a new environment and trying again.

[jingmouren]

创了个新虚环境，报 https://github.com/microsoft/qlib/issues/1166的错。所以暂不知道这个是否还存在

[jingmouren]

又看了下 我说得是
python rolling_benchmark.py run_all
in the benchmarks_dynamic /DDG-DA dir
报错
你回复的是

python ../baseline/rolling_benchmark.py run_all
不报错。

哦，是我复制错文件名了。
python workflow.py run_all
in the benchmarks_dynamic /DDG-DA dir
报错

[SunsetWolf]

创了个新虚环境，报 https://github.com/microsoft/qlib/issues/1166的错。所以暂不知道这个是否还存在

This bug has been fixed in PR 1141 and can be tried again.

[jingmouren]

calc: 100% 。。。151/151 这步完成了
之后执行结果
[8851:MainThread](2022-06-30 14:06:25,843) INFO - qlib.MetaDatasetDS - [dataset.py:278] - Example task for training meta model: {'dataset': {'class': 'qlib.data.dataset.DatasetH', 'kwargs': {'handler': 'file:///examples/benchmarks_dynamic/DDG-DA/handler_proxy.pkl', 'segments': {'train': (Timestamp('2008-01-02 00:00:00'), Timestamp('2010-12-03 00:00:00')), 'test': (Timestamp('2011-01-04 00:00:00'), Timestamp('2011-01-31 00:00:00'))}}}}
creating meta tasks: 100%| 。。。 | 115/115
然后报错 AssertionError: NaN loss!

[8851:MainThread](2022-06-30 14:09:14,561) INFO - qlib.workflow - [expm.py:315] - <mlflow.tracking.client.MlflowClient object at 0x7f31fbd86130>
[8851:MainThread](2022-06-30 14:09:14,564) INFO - qlib.workflow - [exp.py:257] - Experiment 3 starts running ...
[8851:MainThread](2022-06-30 14:09:14,576) INFO - qlib.workflow - [recorder.py:295] - Recorder f5bae2b9dff24b9caadd8a023ebb23c8 starts running under Experiment 3 ...
[8851:MainThread](2022-06-30 14:09:31,567) INFO - qlib.timer - [log.py:117] - Time cost: 0.000s | waiting async_log Done
[8851:MainThread](2022-06-30 14:09:31,567) ERROR - qlib.workflow - [utils.py:41] - An exception has been raised[AssertionError: NaN loss!].
File "workflow.py", line 259, in
fire.Fire(DDGDA)
line 259, in
fire.Fire(DDGDA)
。。。
File "workflow.py", line 249, in run_all
self.train_meta_model()
File "workflow.py", line 174, in train_meta_model
mm.fit(md)
File " anaconda3/envs/py38/lib/python3.8/site-packages/qlib/contrib/meta/data_selection/model.py", line 154, in fit
self.run_epoch(f"{phase}_noweight", task_list, 0, opt, {}, ignore_weight=True)
File " anaconda3/envs/py38/lib/python3.8/site-packages/qlib/contrib/meta/data_selection/model.py", line 96, in run_epoch
assert not np.isnan(loss.detach().item()), "NaN loss!"

[RRdmlearning]

hi！How to solve it 。I have the same problem

[RRdmlearning]

hi！How to solve it 。I have the same problem

[RocStone]

Same problem!

File "/data/pengqlu/miniconda3/envs/gnn/lib/python3.8/site-packages/fire/core.py", line 681, in _CallAndUpdateTrace
component = fn(*varargs, **kwargs)
File "examples/benchmarks_dynamic/DDG-DA/workflow.py", line 252, in run_all
self.train_meta_model()
File "examples/benchmarks_dynamic/DDG-DA/workflow.py", line 177, in train_meta_model
mm.fit(md)
File "/home/pengqlu/code/qlib/qlib/new_qlib/qlib/./qlib/contrib/meta/data_selection/model.py", line 154, in fit
self.run_epoch(f"{phase}_noweight", task_list, 0, opt, {}, ignore_weight=True)
File "/home/pengqlu/code/qlib/qlib/new_qlib/qlib/./qlib/contrib/meta/data_selection/model.py", line 96, in run_epoch
assert not np.isnan(loss.detach().item()), "NaN loss!"
AssertionError: NaN loss!

[github-actions]

This issue is stale because it has been open for three months with no activity. Remove the stale label or comment on the issue otherwise this will be closed in 5 days