MXNet C++Interface reasoning leads to CPU memory leak

[chenyujing1234]

I am using the c++ interface of mxnet to carry out the reasoning of mtcnn algorithm for face detection.
Environment: Ubuntu16.04.1 + GPU cuda9.0 + MXNet1.3.0
I use the interface: MXPredCreate MXPredReshape MXPredForward MXPredFree...
When I used a lot of pictures for stress testing (running for a long time), I found that my process occupied more and more CPU memory, and eventually it occupied all the memory.
My process was forced to kill by the system.
Later, I wrote the program To Test:

1. If only do MXPredCreate, then MXPredFree, and MXPredCreate gives different width and height.
   After 100 thousand cycles, memory will increase by 1G
2. If you do MXPredCreate, and then MXPredReshape to change the width and height, so that the continuous cycle, found that memory leaks very quickly, less than half an hour to leak 4G.
   Do you want to invite MXNet's main developers to find this problem? Please track it down for me. It has been bothering me for almost a week.

[zachgk]

@mxnet-label-bot add [C++, Memory, Bug, Pending Requester Info]

Thanks for raising the issue @chenyujing1234. Can you provide a bit more info so we can better help?

Environment info (Required)

What to do:
1. Download the diagnosis script from https://raw.githubusercontent.com/apache/incubator-mxnet/master/tools/diagnose.py
2. Run the script using `python diagnose.py` and paste its output here.

Build info (If built from source)

Compiler (gcc/clang/mingw/visual studio):

MXNet commit hash:
(Paste the output of git rev-parse HEAD here.)

Build config:
(Paste the content of config.mk, or the build command.)

Minimum reproducible example

Can you share the code for the program that you used to test the memory leak?

[KellenSunderland]

@chenyujing1234 Would you be able to post some sample code, that when run in a loop will leak? If we can run this for a fixed number of iterations (say 1234 loops) we should then be able to look with an ASAN build and see what's leaking.

[leleamol]

@mxnet-label-bot update [Bug, C API, Memory, Pending Requester Info]

[chenyujing1234]

You can get my test code from this address. TKS
https://pan.baidu.com/s/19Wtd_Cf1BGF-2MS3le1mYA

[chenyujing1234]

@mxnet-label-bot update [Bug, C API, Memory, Pending Requester Info]
You can get my test code from this address. TKS
https://pan.baidu.com/s/19Wtd_Cf1BGF-2MS3le1mYA

[chenyujing1234]

@chenyujing1234 Would you be able to post some sample code, that when run in a loop will leak? If we can run this for a fixed number of iterations (say 1234 loops) we should then be able to look with an ASAN build and see what's leaking.

You can get my test code from this address. TKS
https://pan.baidu.com/s/19Wtd_Cf1BGF-2MS3le1mYA

[chenyujing1234]

@mxnet-label-bot add [C++, Memory, Bug, Pending Requester Info]

Thanks for raising the issue @chenyujing1234. Can you provide a bit more info so we can better help?

Environment info (Required)

What to do:
1. Download the diagnosis script from https://raw.githubusercontent.com/apache/incubator-mxnet/master/tools/diagnose.py
2. Run the script using `python diagnose.py` and paste its output here.

Build info (If built from source)

Compiler (gcc/clang/mingw/visual studio):

MXNet commit hash:
(Paste the output of git rev-parse HEAD here.)

Build config:
(Paste the content of config.mk, or the build command.)

Minimum reproducible example

Can you share the code for the program that you used to test the memory leak?

You can get my test code from this address. TKS
https://pan.baidu.com/s/19Wtd_Cf1BGF-2MS3le1mYA

[KellenSunderland]

Great bug report @chenyujing1234. Really appreciate the details. I'll take a look at this on a best-effort basis (might take a while to verify the leaks on my end).

If anyone else from the community has time to jump on this I think it'd be very useful. I've seen a lot of reports of leaks from users.

[chinakook]

Are you using a mxnet version that contain MKL-DNN?