TypeError: control character 'delimiter' cannot be a newline

[danial2008]

🐛 Bug Description

Traceback (most recent call last):
File "D:\dev\anaconda3\envs\qlib\lib\site-packages\IPython\core\interactiveshell.py", line 3398, in run_code
exec(code_obj, self.user_global_ns, self.user_ns)
File "", line 1, in <cell line: 1>
e.calendar(start_time='2010-01-01', end_time='2017-12-31', freq='day')
File "D:\dev\pyqlib\qlib\qlib\data\data.py", line 90, in calendar
_calendar, _calendar_index = self._get_calendar(freq, future)
File "D:\dev\pyqlib\qlib\qlib\data\data.py", line 171, in _get_calendar
_calendar = np.array(self.load_calendar(freq, future))
File "D:\dev\pyqlib\qlib\qlib\data\data.py", line 657, in load_calendar
backend_obj = self.backend_obj(freq=freq, future=future).data
File "D:\dev\pyqlib\qlib\qlib\data\storage\file_storage.py", line 129, in data
H["c"][key] = self._read_calendar()
File "D:\dev\pyqlib\qlib\qlib\data\storage\file_storage.py", line 111, in _read_calendar
for x in np.loadtxt(fp, str, skiprows=skip_rows, max_rows=n_rows, delimiter="\n", encoding="utf-8")
File "D:\dev\anaconda3\envs\qlib\lib\site-packages\numpy\lib\npyio.py", line 1301, in loadtxt
arr = _read(fname, dtype=dtype, comment=comment, delimiter=delimiter,
File "D:\dev\anaconda3\envs\qlib\lib\site-packages\numpy\lib\npyio.py", line 1006, in _read
next_arr = _load_from_filelike(
TypeError: control character 'delimiter' cannot be a newline (\r or \n).

To Reproduce

Steps to reproduce the behavior:
1.download the data by get_data.py firstly
2.write a simple demo
qlib.init(provider_uri="D:/dev/pyqlib/data/cn_data/")

from qlib.data import D

D.calendar(start_time='2010-01-01', end_time='2017-12-31', freq='day')[:5]

Expected Behavior

Screenshot

Environment

Note: User could run cd scripts && python collect_info.py all under project directory to get system information
and paste them here directly.

• Qlib version: 0.8.6
• Python version: 3.8
• OS (Windows, Linux, MacOS):Windows
• Commit number (optional, please provide it if you are using the dev version):

Additional Notes

[wwwadx]

Same issue encountered!

[SunsetWolf]

This issue is caused by an upgrade of the qlib dependent library and is currently being fixed.

[Viperccc]

Same issue encountered!

[SunsetWolf]

This bug has been fixed in PR 1141.

[LiquidAmmonia]

Is this PR merged into main? I pulled the latest main branch but this bug remains.

[HuihuiWu-Microsoft]

I have the same issue with latest code.

[rmallof]

https://github.com/numpy/numpy/blob/40230640b2e5c611b0d1fe913cf3285c35848026/numpy/core/src/multiarray/textreading/readtext.c#L116

[XixuHu]

Create virtual environment using Python=3.7 and install the qlib package using the codes from https://github.com/microsoft/qlib/blob/main/examples/workflow_by_code.ipynb can run properly.

[rovedream]

同样问题

[you-n-g]

This has been fixed in the latest code

qlib/qlib/data/storage/file_storage.py

Line 109 in c34051c

return [str(x) for x in np.loadtxt(fp, str, skiprows=skip_rows, max_rows=n_rows, encoding="utf-8")]

[rmallof]

Thank you for the quick resolution - it's working for me

[lichoil]

encountered the same issue.
pyqlib 0.8.6
python 3.8
os: linux

I've tried to install a new condo environment, but issue remains.

[lichoil]

[880:MainThread](2022-07-24 15:51:15,217) INFO - qlib.Initialization - [config.py:413] - default_conf: client.
[880:MainThread](2022-07-24 15:51:15,219) INFO - qlib.Initialization - [init.py:74] - qlib successfully initialized based on client settings.
[880:MainThread](2022-07-24 15:51:15,220) INFO - qlib.Initialization - [init.py:76] - data_path={'__DEFAULT_FREQ': PosixPath('/home/choil/.qlib/qlib_data/cn_data')}
[880:MainThread](2022-07-24 15:51:15,220) INFO - qlib.workflow - [expm.py:315] - <mlflow.tracking.client.MlflowClient object at 0x7f5d10e0a940>
[880:MainThread](2022-07-24 15:51:15,235) INFO - qlib.workflow - [exp.py:257] - Experiment 1 starts running ...
[880:MainThread](2022-07-24 15:51:15,325) INFO - qlib.workflow - [recorder.py:293] - Recorder c36a78bf0d8d4c3398ab4654c3db3ad6 starts running under Experiment 1 ...
Please install necessary libs for CatBoostModel.
ModuleNotFoundError. XGBModel is skipped(optional: maybe installing xgboost can fix it).
ModuleNotFoundError. PyTorch models are skipped (optional: maybe installing pytorch can fix it).
[880:MainThread](2022-07-24 15:51:15,505) INFO - qlib.timer - [log.py:117] - Time cost: 0.000s | waiting async_log Done
[880:MainThread](2022-07-24 15:51:15,506) ERROR - qlib.workflow - [utils.py:41] - An exception has been raised[TypeError: control character 'delimiter' cannot be a newline (\r or \n).].
File "/home/choil/anaconda3/envs/qlib2/bin/qrun", line 8, in
sys.exit(run())
File "/home/choil/anaconda3/envs/qlib2/lib/python3.8/site-packages/qlib/workflow/cli.py", line 72, in run
fire.Fire(workflow)
File "/home/choil/anaconda3/envs/qlib2/lib/python3.8/site-packages/fire/core.py", line 141, in Fire
component_trace = _Fire(component, args, parsed_flag_args, context, name)
File "/home/choil/anaconda3/envs/qlib2/lib/python3.8/site-packages/fire/core.py", line 466, in _Fire
component, remaining_args = _CallAndUpdateTrace(
File "/home/choil/anaconda3/envs/qlib2/lib/python3.8/site-packages/fire/core.py", line 681, in _CallAndUpdateTrace
component = fn(*varargs, **kwargs)
File "/home/choil/anaconda3/envs/qlib2/lib/python3.8/site-packages/qlib/workflow/cli.py", line 66, in workflow
recorder = task_train(config.get("task"), experiment_name=experiment_name)
File "/home/choil/anaconda3/envs/qlib2/lib/python3.8/site-packages/qlib/model/trainer.py", line 127, in task_train
_exe_task(task_config)
File "/home/choil/anaconda3/envs/qlib2/lib/python3.8/site-packages/qlib/model/trainer.py", line 46, in _exe_task
dataset: Dataset = init_instance_by_config(task_config["dataset"], accept_types=Dataset)
File "/home/choil/anaconda3/envs/qlib2/lib/python3.8/site-packages/qlib/utils/init.py", line 457, in init_instance_by_config
return klass(**cls_kwargs, **kwargs)
File "/home/choil/anaconda3/envs/qlib2/lib/python3.8/site-packages/qlib/data/dataset/init.py", line 115, in init
self.handler: DataHandler = init_instance_by_config(handler, accept_types=DataHandler)
File "/home/choil/anaconda3/envs/qlib2/lib/python3.8/site-packages/qlib/utils/init.py", line 457, in init_instance_by_config
return klass(**cls_kwargs, **kwargs)
File "/home/choil/anaconda3/envs/qlib2/lib/python3.8/site-packages/qlib/contrib/data/handler.py", line 171, in init
super().init(
File "/home/choil/anaconda3/envs/qlib2/lib/python3.8/site-packages/qlib/data/dataset/handler.py", line 434, in init
super().init(instruments, start_time, end_time, data_loader, **kwargs)
File "/home/choil/anaconda3/envs/qlib2/lib/python3.8/site-packages/qlib/data/dataset/handler.py", line 97, in init
self.setup_data()
File "/home/choil/anaconda3/envs/qlib2/lib/python3.8/site-packages/qlib/data/dataset/handler.py", line 570, in setup_data
super().setup_data(**kwargs)
File "/home/choil/anaconda3/envs/qlib2/lib/python3.8/site-packages/qlib/data/dataset/handler.py", line 141, in setup_data
self._data = lazy_sort_index(self.data_loader.load(self.instruments, self.start_time, self.end_time))
File "/home/choil/anaconda3/envs/qlib2/lib/python3.8/site-packages/qlib/data/dataset/loader.py", line 135, in load
{
File "/home/choil/anaconda3/envs/qlib2/lib/python3.8/site-packages/qlib/data/dataset/loader.py", line 136, in
grp: self.load_group_df(instruments, exprs, names, start_time, end_time, grp)
File "/home/choil/anaconda3/envs/qlib2/lib/python3.8/site-packages/qlib/data/dataset/loader.py", line 211, in load_group_df
df = D.features(
File "/home/choil/anaconda3/envs/qlib2/lib/python3.8/site-packages/qlib/data/data.py", line 1184, in features
return DatasetD.dataset(instruments, fields, start_time, end_time, freq, inst_processors=inst_processors)
File "/home/choil/anaconda3/envs/qlib2/lib/python3.8/site-packages/qlib/data/data.py", line 905, in dataset
instruments_d = self.get_instruments_d(instruments, freq)
File "/home/choil/anaconda3/envs/qlib2/lib/python3.8/site-packages/qlib/data/data.py", line 514, in get_instruments_d
instruments_d = Inst.list_instruments(instruments=instruments, freq=freq, as_list=False)
File "/home/choil/anaconda3/envs/qlib2/lib/python3.8/site-packages/qlib/data/data.py", line 695, in list_instruments
cal = Cal.calendar(freq=freq)
File "/home/choil/anaconda3/envs/qlib2/lib/python3.8/site-packages/qlib/data/data.py", line 90, in calendar
_calendar, _calendar_index = self._get_calendar(freq, future)
File "/home/choil/anaconda3/envs/qlib2/lib/python3.8/site-packages/qlib/data/data.py", line 171, in _get_calendar
_calendar = np.array(self.load_calendar(freq, future))
File "/home/choil/anaconda3/envs/qlib2/lib/python3.8/site-packages/qlib/data/data.py", line 657, in load_calendar
backend_obj = self.backend_obj(freq=freq, future=future).data
File "/home/choil/anaconda3/envs/qlib2/lib/python3.8/site-packages/qlib/data/storage/file_storage.py", line 129, in data
H["c"][key] = self._read_calendar()
File "/home/choil/anaconda3/envs/qlib2/lib/python3.8/site-packages/qlib/data/storage/file_storage.py", line 111, in _read_calendar
for x in np.loadtxt(fp, str, skiprows=skip_rows, max_rows=n_rows, delimiter="\n", encoding="utf-8")
File "/home/choil/anaconda3/envs/qlib2/lib/python3.8/site-packages/numpy/lib/npyio.py", line 1308, in loadtxt
arr = _read(fname, dtype=dtype, comment=comment, delimiter=delimiter,
File "/home/choil/anaconda3/envs/qlib2/lib/python3.8/site-packages/numpy/lib/npyio.py", line 1006, in _read
next_arr = _load_from_filelike(
TypeError: control character 'delimiter' cannot be a newline (\r or \n).