Python bindings and profiles

[nbelakovski]

Current status:

BOBYQA, LINCOA, NEWUOA, and UOBYQA work on Linux and Mac
COBLYA works on Linux and Mac but only with NonlinearConstraints OR LinearConstraints at the moment (there's a minor todo to allow both), and it also has some sort of GIL issue (I think it's related to the fact that NonlinearConstraints holds a function).

TODO:

• Windows
• Let constraints be a list so that both Linear and Nonlinear constraints may be accepted for COBYLA
• Flesh out testing
  • Flesh out test_compatible_interface.py
    • Test the non-COBYLA algorithms between pdfo and prima.
    • Add support for constraint dictionaries in prima.
• Move the code about warning when providing bounds or constraints to a method which does not use them into a separate PR Warn user if they have provided information that will not be used #134
• Does prima_is_success have a complete list of successful return codes?
• Create issue to discuss static vs shared libraries (maybe both). Static vs shared libs? #125
• Create separate PR to modify the C and Fortran examples as follows: objective function shall be (x1-5)^2 + (x2-4)^2. The rationale is that this lets the initial point be (0,0), and since users will be copying the example for their own usage it makes sense to give them a sensible point to copy. This is not strictly related to python bindings, but it is something we'd like to do and it makes sense to do it ahead of the python bindings so their examples can be built correctly in the first place. This discussion is about what we call the "API example" - there will be more examples with more interesting functions to demonstrate the capabilities of the solvers. Refactor examples #131
• Include script used to run optiprofiler and create performance profiles vs pdfo
• Run latest performance profiles for all solvers
• Create performance profiles comparing Python bindings to MATLAB bindings in order to validate the Python bindings

[github-advanced-security]

check-spelling found more than 20 potential problems in the proposed changes. Check the Files changed tab for more details.

[zaikunzhang]

Ideally, the (standalone) PRIMA Python package should have the same API as pdfo.

[nbelakovski]

Ideally, the (standalone) PRIMA Python package should have the same API as pdfo.

I see from the PDFO usage notes that it aims to be API compatibility with scipy.optimize.minimize which is what I've aimed for as well, so this shouldn't be an issue. Since I plan to add a test to ensure compatibility with the scipy API, I'll add a test to ensure compatibility with PDFO as well.

[nbelakovski]

For COBYLA, it seems that many of the problems in the list don't actually have nonlinear constraints, and this was leading the Python setup to fail on those problems (because no nonlinear constraint was provided, not even a blank one), whereas it seems the MATLAB one was continuing to solve those. I modified the code to reject and problems without nonlinear constraints for COBYLA and ran it again. For default and non_default options we again get vertical lines. For different options Python edges out MATLAB.

cobyla_default_options.pdf.zip
cobyla_different_options.pdf.zip
cobyla_nondefault_options.pdf.zip

[zaikunzhang]

For COBYLA, it seems that many of the problems in the list don't actually have nonlinear constraints, and this was leading the Python setup to fail on those problems (because no nonlinear constraint was provided, not even a blank one)

This is precisely why I updated the problem lists and suggested testing all possible types of problems.

Do you mean that the current version of COBYLA cannot solve problems without nonlinear constraints? COBYLA should be able to solve all types of problems, with or without (nonlinear) constraints. If the user does not provide nonlinear constraints, COBYLA should handle it by, for example, setting the nonlinear constraints to none internally.

Similar comments apply to other solvers. For example, LINCOA should be able to solver unconstrained, bound-constrained, and linear constrained problems, not only the last type (linearly constrained problems). If the user does not specify linear constraints at all, then LINCOA handles it by, for example, setting linear constraints to none internally.

You may check how PDFO handles these situations.

Thank you.

[zaikunzhang]

I note that the discrepancy in NEWUOA becomes much more visible when the options are the default, but not so much with non-default options. Do they really receive the same default options? Thank you.

[nbelakovski]

I note that the discrepancy in NEWUOA becomes much more visible when the options are the default, but not so much with non-default options. Do they really receive the same default options? Thank you.

I modified the Fortran to print out rhobeg_loc, rhoend_loc, etc. in newuoa.f90 to examine the options are they were being received from Python, and for MATLAB I printed the options values just before the call to the Fortran code (since in MATLAB it's necessary to use mexprintf and I didn't want to deal with that, particularly since MATLAB has preprima so it doesn't rely on Fortran's default initialization code anyway). As far as I could tell the values were the same. There were some slight differences in how each program prints doubles, but when I took a look at the bit representation for various values they were identical.

Do you mean that the current version of COBYLA cannot solve problems without nonlinear constraints?

Yes, this was an oversight on my part. I've fixed it and also added tests to make sure it doesn't happen again. Below are the PDFs from re-running with COBYLA allowed to run on problems without nonlinear constraints. There appear to be a few problems that the MATLAB version takes more function evaluations to solve. I did some digging and came up with the following information:

Results for HS52 with MATLAB-COBYLA (run 1/1): f = 5.3266e+00, maxcv = 2.7756e-17 (1.13 seconds, nfev=224).
Results for HS52 with Python-COBYLA (run 1/1): f = 5.3266e+00, maxcv = 5.5511e-17 (0.06 seconds, nfev=201).

Results for DEGENLPB with MATLAB-COBYLA (run 1/1): f = -3.0735e+01, maxcv = 1.1863e-09 (0.84 seconds, nfev=114).
Results for DEGENLPB with Python-COBYLA (run 1/1): f = -3.0731e+01, maxcv = 1.6644e-13 (0.16 seconds, nfev=97).

Results for DUALC8 with MATLAB-COBYLA (run 1/1): f = 1.8309e+04, maxcv = 1.6791e-24 (1.00 seconds, nfev=208).
Results for DUALC8 with Python-COBYLA (run 1/1): f = 1.8309e+04, maxcv = 1.1102e-16 (0.11 seconds, nfev=171).

I'm confused as to why the performance ratio on the graph is so high. The ratio of the above 3 is between 1.1 and 1.2, and yet even if I just run with these 3, the x-axis still gets up to 16. Am I misunderstanding how the x-axis is supposed to work?

I'm happy to keep doing performance profiles if we think there's a serious issue with any of these algorithms worth investigating, but I would need some guidance as to what exactly to investigate or what sort of data to come up with.

cobyla_default_options.pdf.zip
cobyla_different_options.pdf.zip
cobyla_nondefault_options.pdf.zip

[zaikunzhang]

Thank you @nbelakovski for the huge efforts!

I think we are close to the merge. I will check it and come back soon. If there is nothing too serious, we will just merge and continue the investigations and refinements after that.

[zaikunzhang]

Hi @nbelakovski ,

I note that the discrepancy in NEWUOA becomes much more visible when the options are the default, but not so much with non-default options. Do they really receive the same default options? Thank you.

I modified the Fortran to print out rhobeg_loc, rhoend_loc, etc. in newuoa.f90 to examine the options are they were being received from Python, and for MATLAB I printed the options values just before the call to the Fortran code (since in MATLAB it's necessary to use mexprintf and I didn't want to deal with that, particularly since MATLAB has preprima so it doesn't rely on Fortran's default initialization code anyway). As far as I could tell the values were the same. There were some slight differences in how each program prints doubles, but when I took a look at the bit representation for various values they were identical.

Great! Let us tolerate the fact that there is a discrepency in the profiles of NEWUOA, but we also need to keep this fact in mind. It may be just an artifact of rounding errors. Otherwise, it may bother us again in the future, when we may have better insights.

Do you mean that the current version of COBYLA cannot solve problems without nonlinear constraints?

Yes, this was an oversight on my part. I've fixed it and also added tests to make sure it doesn't happen again. Below are the PDFs from re-running with COBYLA allowed to run on problems without nonlinear constraints.

Great! The profiles look good, except for the large ratios as you mention below. We will continue to investigage this issue after the merge. It may be a bug of OptiProfiler @ragonneau @OptHuang .

I'm confused as to why the performance ratio on the graph is so high. The ratio of the above 3 is between 1.1 and 1.2, and yet even if I just run with these 3, the x-axis still gets up to 16. Am I misunderstanding how the x-axis is supposed to work?

I am confused as well. Is it possible to record the numbers of function evaluations for each solver (given a tolerance) and check whether there is a difference up to 16 times? Maybe the final numbers are close but the actual numbers needed to satisfy a given tolerance are very different. If this is not the case, it will be a bug of OptiProfiler. @ragonneau @OptHuang

I am ready to press the merge button. Do you have more changes to make before that?

Many thanks!

Zaikun

[zaikunzhang]

Hi @nbelakovski ,

Results for HS52 with MATLAB-COBYLA (run 1/1): f = 5.3266e+00, maxcv = 2.7756e-17 (1.13 seconds, nfev=224).
Results for HS52 with Python-COBYLA (run 1/1): f = 5.3266e+00, maxcv = 5.5511e-17 (0.06 seconds, nfev=201).

I hope the huge difference in the computing time comes from the fact that the MATLAB version communicated with the objective function using TCP. What do you think?

Thanks!

Zaikun

[zaikunzhang]

This may be a stupid question due to my ignorance about Python.

I saw the following in python/examples/lincoa_example.py, python/examples/lincoa_example.py, and python/examples/rosenbrock.py:

from prima import minimize, LinearConstraint as LC

from prima import minimize, NonlinearConstraint as NLC

from prima import minimize, Bounds, LinearConstraint, NonlinearConstraint

Does this mean that users are expected to use Bounds, LinearConstraint, and NonlinearConstraint provided by PRIMA? Couldn't they just use the standard constraints from SciPy?

Old versions of PDFO used to provide Bounds, LinearConstraint, and NonlinearConstraint. They have been removed in the latest version. Isn't it better to use the standard facilities provided by SciPy? See https://github.com/pdfo/pdfo/blob/main/python/examples/rosenbrock.py

[nbelakovski]

I hope the huge difference in the computing time comes from the fact that the MATLAB version communicated with the objective function using TCP. What do you think?

I hope so too 😁 !

But really, I think it is. I haven't investigated but that seems likely to me. Particularly for problems that have both an objective function and a nonlinear constraint function, since that requires two separate connections.

Does this mean that users are expected to use Bounds, LinearConstraint, and NonlinearConstraint provided by PRIMA? Couldn't they just use the standard constraints from SciPy?

If you look at __init__.py you'll see that PRIMA's Bounds, etc. are just re-exports of the SciPy classes. I felt this made the interface slightly easier.

I am ready to press the merge button. Do you have more changes to make before that?

🥳 🥳 🥳

There's one outstanding item. At first I thought it would be a couple line fix, but there's a little more to it so I propose making an issue to keep track of it and fixing it in a follow-on PR. I can make that PR tomorrow or later this week, I just want to make sure the issue gets proper discussion. Does that sound good to you?

[zaikunzhang]

Hi @nbelakovski ,

If you look at __init__.py you'll see that PRIMA's Bounds, etc. are just re-exports of the SciPy classes. I felt this made the interface slightly easier.

Will it work if people do the following?

from prima import minimize
from scipy import Bounds, LinearConstraint, NonlinearConstraint

1. If yes, I believe that we should do that in the examples. This will at least eliminate the necessity of explaining what Bounds, LinearConstraint, and NonlinearConstraint are in these examples.

2. If no, then we should make it work in this way.

[zaikunzhang]

Hi @nbelakovski ,

There's one outstanding item. At first I thought it would be a couple line fix, but there's a little more to it so I propose making an issue to keep track of it and fixing it in a follow-on PR. I can make that PR tomorrow or later this week, I just want to make sure the issue gets proper discussion. Does that sound good to you?

Yes, it sounds good. Thanks.

[zaikunzhang]

I have made an announcement on Fortran Discourse:

https://fortran-lang.discourse.group/t/prima-has-got-a-python-interface/7942

Thank you very much, Nickolai @nbelakovski !