Pandas.error and shape mismatch

[partha434]

Dear Wouter,
I am getting the following error. The version of methplotlib I am using is 0.13.1. Please help.

methplotlib -m methylation_calls_CpG_sm1.tsv.gz methylation_frequency_sm1.tsv.gz -n calls frequencies -w mtDNA:1-16,626 -g /cluster/projects/nn9337k/as1_nanopolish/all_context/sequence.sorted.gtf.gz --simplify -b /cluster/projects/nn9337k/as1/reference.bed -f /cluster/projects/nn9337k/as1/reference.fasta

Traceback (most recent call last):
File "/cluster/projects/nn9337k/anaconda/envs/methplotlib/lib/python3.10/site-packages/pandas/core/indexes/base.py", line 3621, in get_loc
return self._engine.get_loc(casted_key)
File "pandas/_libs/index.pyx", line 136, in pandas._libs.index.IndexEngine.get_loc
File "pandas/_libs/index.pyx", line 142, in pandas._libs.index.IndexEngine.get_loc
TypeError: '18910 False
18973 False
19035 False
19097 False
19159 False
...
22925 False
23000 True
23074 False
23148 False
23224 False
Name: log_lik_ratio, Length: 28154, dtype: bool' is an invalid key

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/cluster/projects/nn9337k/anaconda/envs/methplotlib/lib/python3.10/site-packages/pandas/core/series.py", line 1085, in setitem
self._set_with_engine(key, value)
File "/cluster/projects/nn9337k/anaconda/envs/methplotlib/lib/python3.10/site-packages/pandas/core/series.py", line 1146, in _set_with_engine
loc = self.index.get_loc(key)
File "/cluster/projects/nn9337k/anaconda/envs/methplotlib/lib/python3.10/site-packages/pandas/core/indexes/base.py", line 3628, in get_loc
self._check_indexing_error(key)
File "/cluster/projects/nn9337k/anaconda/envs/methplotlib/lib/python3.10/site-packages/pandas/core/indexes/base.py", line 5637, in _check_indexing_error
raise InvalidIndexError(key)
pandas.errors.InvalidIndexError: 18910 False
18973 False
19035 False
19097 False
19159 False
...
22925 False
23000 True
23074 False
23148 False
23224 False
Name: log_lik_ratio, Length: 28154, dtype: bool

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/cluster/projects/nn9337k/anaconda/envs/methplotlib/bin/methplotlib", line 8, in
sys.exit(main())
File "/cluster/projects/nn9337k/anaconda/envs/methplotlib/lib/python3.10/site-packages/methplotlib/methplotlib.py", line 27, in main
meth_browser(meth_data=meth_data,
File "/cluster/projects/nn9337k/anaconda/envs/methplotlib/lib/python3.10/site-packages/methplotlib/methplotlib.py", line 51, in meth_browser
meth_traces = plots.methylation(meth_data, dotsize=dotsize)
File "/cluster/projects/nn9337k/anaconda/envs/methplotlib/lib/python3.10/site-packages/methplotlib/plots.py", line 105, in methylation
make_per_read_meth_traces_llr(table=meth.table,
File "/cluster/projects/nn9337k/anaconda/envs/methplotlib/lib/python3.10/site-packages/methplotlib/plots.py", line 167, in make_per_read_meth_traces_llr
table.loc[:, "llr_scaled"] = rescale_log_likelihood_ratio(table["log_lik_ratio"].copy())
File "/cluster/projects/nn9337k/anaconda/envs/methplotlib/lib/python3.10/site-packages/methplotlib/plots.py", line 253, in rescale_log_likelihood_ratio
llr[llr > 0] = scaler.fit_transform(llr[llr > 0].values.reshape(-1, 1)).tolist()
File "/cluster/projects/nn9337k/anaconda/envs/methplotlib/lib/python3.10/site-packages/pandas/core/series.py", line 1127, in setitem
self._set_values(indexer, value)
File "/cluster/projects/nn9337k/anaconda/envs/methplotlib/lib/python3.10/site-packages/pandas/core/series.py", line 1185, in _set_values
self._mgr = self._mgr.setitem(indexer=key, value=value)
File "/cluster/projects/nn9337k/anaconda/envs/methplotlib/lib/python3.10/site-packages/pandas/core/internals/managers.py", line 337, in setitem
return self.apply("setitem", indexer=indexer, value=value)
File "/cluster/projects/nn9337k/anaconda/envs/methplotlib/lib/python3.10/site-packages/pandas/core/internals/managers.py", line 304, in apply
applied = getattr(b, f)(**kwargs)
File "/cluster/projects/nn9337k/anaconda/envs/methplotlib/lib/python3.10/site-packages/pandas/core/internals/blocks.py", line 959, in setitem
values[indexer] = value
ValueError: shape mismatch: value array of shape (5284,1) could not be broadcast to indexing result of shape (5284,)

[wdecoster]

Can you upgrade methplotlib to the latest version? Yours is rather outdated, and it is possible that the issue you are reporting is already solved.

[partha434]

Hello,
I tried to upgrade methplotlib, but unfortunately it still shows the old version. I have installed it via conda. On the conda page the version shows 0.20.1, but still I can't install it.

[wdecoster]

I tried to upgrade methplotlib, but unfortunately it still shows the old version.

But the upgrade was successful? Have you earlier installed this with pip? You can try uninstalling with pip and conda first.
You can use which methplotlib to see where it is installed.

but still I can't install it.

You have an error?

[partha434]

I successfully installed methplotlib to 0.20.1.
But I got this error...

methplotlib -m methylation_calls_CpG_sm1.tsv methylation_frequency_sm1.tsv -n calls frequencies -w mtDNA:1-16626 -g /cluster/projects/nn9337k/as1_nanopolish/all_context/sequence.sorted.gtf.gz --simplify -b /cluster/projects/nn9337k/as1/reference.bed

Reading methylation_calls_CpG_sm1.tsv would be faster with bgzip and tabix.
Please index with 'tabix -S1 -s1 -b3 -e4'.

Reading methylation_frequency_sm1.tsv would be faster with bgzip and tabix.
Please index with 'tabix -S1 -s1 -b2 -e3'.
Traceback (most recent call last):
File "/cluster/projects/nn9337k/anaconda/envs/methplotlib/lib/python3.10/site-packages/pandas/core/indexes/base.py", line 3621, in get_loc
return self._engine.get_loc(casted_key)
File "pandas/_libs/index.pyx", line 136, in pandas._libs.index.IndexEngine.get_loc
File "pandas/_libs/index.pyx", line 142, in pandas._libs.index.IndexEngine.get_loc
TypeError: '18971 False
19033 False
19094 False
19156 False
19218 False
...
22888 False
22962 True
23035 False
23109 False
23183 False
Name: log_lik_ratio, Length: 28154, dtype: bool' is an invalid key

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/cluster/projects/nn9337k/anaconda/envs/methplotlib/lib/python3.10/site-packages/pandas/core/series.py", line 1085, in setitem
self._set_with_engine(key, value)
File "/cluster/projects/nn9337k/anaconda/envs/methplotlib/lib/python3.10/site-packages/pandas/core/series.py", line 1146, in _set_with_engine
loc = self.index.get_loc(key)
File "/cluster/projects/nn9337k/anaconda/envs/methplotlib/lib/python3.10/site-packages/pandas/core/indexes/base.py", line 3628, in get_loc
self._check_indexing_error(key)
File "/cluster/projects/nn9337k/anaconda/envs/methplotlib/lib/python3.10/site-packages/pandas/core/indexes/base.py", line 5637, in _check_indexing_error
raise InvalidIndexError(key)
pandas.errors.InvalidIndexError: 18971 False
19033 False
19094 False
19156 False
19218 False
...
22888 False
22962 True
23035 False
23109 False
23183 False
Name: log_lik_ratio, Length: 28154, dtype: bool

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/cluster/projects/nn9337k/anaconda/envs/methplotlib/bin/methplotlib", line 8, in
sys.exit(main())
File "/cluster/projects/nn9337k/anaconda/envs/methplotlib/lib/python3.10/site-packages/methplotlib/methplotlib.py", line 26, in main
meth_browser(meth_data=meth_data,
File "/cluster/projects/nn9337k/anaconda/envs/methplotlib/lib/python3.10/site-packages/methplotlib/methplotlib.py", line 53, in meth_browser
meth_traces = plots.methylation(meth_data, dotsize=dotsize, binary=binary, minqual=minqual)
File "/cluster/projects/nn9337k/anaconda/envs/methplotlib/lib/python3.10/site-packages/methplotlib/plots.py", line 106, in methylation
make_per_read_meth_traces_llr(table=meth.table,
File "/cluster/projects/nn9337k/anaconda/envs/methplotlib/lib/python3.10/site-packages/methplotlib/plots.py", line 183, in make_per_read_meth_traces_llr
table.loc[:, "llr_scaled"] = rescale_log_likelihood_ratio(table["log_lik_ratio"].copy())
File "/cluster/projects/nn9337k/anaconda/envs/methplotlib/lib/python3.10/site-packages/methplotlib/plots.py", line 269, in rescale_log_likelihood_ratio
llr[llr > 0] = scaler.fit_transform(llr[llr > 0].values.reshape(-1, 1)).tolist()
File "/cluster/projects/nn9337k/anaconda/envs/methplotlib/lib/python3.10/site-packages/pandas/core/series.py", line 1127, in setitem
self._set_values(indexer, value)
File "/cluster/projects/nn9337k/anaconda/envs/methplotlib/lib/python3.10/site-packages/pandas/core/series.py", line 1185, in _set_values
self._mgr = self._mgr.setitem(indexer=key, value=value)
File "/cluster/projects/nn9337k/anaconda/envs/methplotlib/lib/python3.10/site-packages/pandas/core/internals/managers.py", line 337, in setitem
return self.apply("setitem", indexer=indexer, value=value)
File "/cluster/projects/nn9337k/anaconda/envs/methplotlib/lib/python3.10/site-packages/pandas/core/internals/managers.py", line 304, in apply
applied = getattr(b, f)(**kwargs)
File "/cluster/projects/nn9337k/anaconda/envs/methplotlib/lib/python3.10/site-packages/pandas/core/internals/blocks.py", line 959, in setitem
values[indexer] = value
ValueError: shape mismatch: value array of shape (5284,1) could not be broadcast to indexing result of shape (5284,)

[wdecoster]

Could you please share the log file created by methplotlib?

[partha434]

Hello,
I have attached the log file generated from methplotlib.
methplotlib_20220905_0847.log

[partha434]

Dear Wouter,
Any updates related to the above error?
I have checked pip installation with pip check. It also says no broken requirements found.
But still I am getting the above error.

[wdecoster]

Would it be possible to share your input files?

[partha434]

Sure.
Please find the attached files for calls and frequency.
methylation_calls_CpG_sm1.txt
methylation_frequency_sm1.txt

[wdecoster]

It works on my system. Can you try upgrading pandas?

[partha434]

I upgraded pandas to latest version 1.4.4 (earlier version was 1.4.3).
But I got the same error.
Please help.

[wdecoster]

Can you please try to create a conda environment for just methplotlib and see if you have the error if you execute methplotlib there.

conda create -n methplotlib methplotlib
conda activate methplotlib

If you still have the error, please export the environment (conda env export > env.yaml) and share that with me so I can try to reproduce.

[partha434]

Dear Wouter,
I created a new conda environment as per your suggestions. But I am still facing the same error.
Please find the attached env.yaml file (in text here).
Waiting for your kind help.
Thank you so much in advance.
env.txt

[partha434]

Dear Wouter,
I checked my environment with pip check. There is no broken requirements.

[xiaohuli-45]

Dear Wouter, I checked my environment with pip check. There is no broken requirements.

Hello, I have encountered a similar problem. May I ask if you have resolved the issue