bpo-39187: robotparser does not respect longest match

[andreburgaud]

• Added a sort function to sort the rules according to longest match
• Took into account equivalent allow and disallow rules to result in allow

https://bugs.python.org/issue39187

[the-knights-who-say-ni]

Hello, and thanks for your contribution!

I'm a bot set up to make sure that the project can legally accept this contribution by verifying everyone involved has signed the PSF contributor agreement (CLA).

CLA Missing

Our records indicate the following people have not signed the CLA:

@andreburgaud

For legal reasons we need all the people listed to sign the CLA before we can look at your contribution. Please follow the steps outlined in the CPython devguide to rectify this issue.

If you have recently signed the CLA, please wait at least one business day
before our records are updated.

You can check yourself to see if the CLA has been received.

Thanks again for the contribution, we look forward to reviewing it!

[serhiy-storchaka]

Thank you for your PR, @andreburgaud.

Unfortunately, this approach cannot be used when there are special characters * or $ in the patterns. But they are currently not supported in robotparser, so at least we will get more correct behavior for robots.txt files that don't contain them.

[serhiy-storchaka]

This document is now available as https://datatracker.ietf.org/doc/html/rfc9309. Please update all links.

But this test passes also if all allow rules have higher priority than any disallow rule. So please add also reversed rules (short allow, long disallow) here.

[andreburgaud]

Thank you for the suggestion @serhiy-storchaka, Working on it.

[serhiy-storchaka]

This test replaces GoogleURLOrderingTest. I think that GoogleURLOrderingTest should now be removed, because it lost its meaning.

[andreburgaud]

Agreed. I'm removing it.

[serhiy-storchaka]

Double '/folder1'.

[andreburgaud]

Stupid mistake on my end. Thx. Removing the unnecessary double 'folder1'.