ENH: Root finding

[chrishyland]

Hi @jstac, in this PR @spvdchachan and I added in the Jitted Newton methods for root finding, In particular, the newton method and the newton_secant method. Additionally, we wrote some test cases too.

As several people mentioned in #416 , extra information about convergence, iterations, etc. is crucial.
Therefore, we provided this extra information but instead used the namedtuple collection for fields to be accessed easier through names rather than numerical indexes in normal tuples.

We followed the source code directly from Scipy but there were several technical issues with Numba.

• Optional function args: Numba only supports optional parameters of primitive types only. Also, it is not guaranteed to be 100% stable as well (explicitly mentioned in the doc). This means it is not possible at the moment to support all three variants of newton's method within one jitted function. i.e. we cannot make fprime and fprime2 optional. Currently, we've only implemented 2 of the functions.
• Warnings: Cannot issue a warning message through warnings. If this is crucial, we have to find a workaround.

@jstac

[coveralls]

Coverage increased (+0.04%) to 95.142% when pulling 876c797 on chrishyland:root-finding into d6130c1 on QuantEcon:master.

[jstac]

Great, thanks. Nice looking code.

@mmcky using namedtuple leads to an extra dependency. Are you OK with that? I am in favor of using it.

If we do use named tuples then let's also change our existing maximization function accordingly.

I can't think of a better approach to accommodating the fact that only only primitive arguments can be optional. @oyamad Do you have any suggestions on how to better structure this code?

This is a trivial point, but I thought I read somewhere that having multiple return statements within one function is frowned on these days in software circles. It that not so? In any case you might also save a few bytes by replacing your multiple return statements with break.

[oyamad]

@jstac I am not sure if I really understand the issue, but the following generates different code for different signatures (which behaves like optional arguments from the user's perspective):

from numba import jit, generated_jit, types

@generated_jit(nopython=True)
def test(x, f=None, f2=None):
    if isinstance(f, types.Callable):
        if isinstance(f2, types.Callable):
            def return_f2(x, f, f2):
                return f2(x)
            return return_f2
        # else
        def return_f(x, f, f2):
            return f(x)
        return return_f
    
    #else
    def return_0(x, f, f2):
        return 0
    return return_0

@jit(nopython=True)
def f(x):
    return x

@jit(nopython=True)
def f2(x):
    return 2*x

test(10)
0

test(10, f)
10

test(10, f, f2)
20

But, in my opinion, this would unnecessarily complicate the structure of the code and increase the maintenance costs. Having different names for different sets of arguments (newton for (func, x0, fprime), newton_secant (func, x0), and newton_??? for (func, x0, fprime, fprime2)) appears fine to me.

• I don't think issuing a warning message is crucial. Just returning the information about whether the routine is converged is fine.

[supavitd]

Thank you for the input.
@oyamad I've added another newton method called newton_halley which can take in fprime2.

@jstac I've removed multiple returns in each method as requested.

[jstac]

@oyamad @mmcky This looks good to me. I would be happy to see it merged.

I would also like to see a robust root finding method such as brent or bisection. But perhaps this can be done separately.

[mmcky]

thanks for your review @jstac and @oyamad. Thanks @chrishyland and @spvdchachan for this PR.
@jstac I have no problem with adding additional dependencies such as named tuples

[mmcky]

thanks again for everyone's input. I will merge this PR now.