Skip to content

Conversation

@steffansafey
Copy link
Contributor

@steffansafey steffansafey commented Oct 24, 2024

Description

Fixes an issue where the string :None was being appended to Playwright Proxy configurations.

proxy_info: {'server': 'http://p.webshare.io:None', 'username': 'foo', 'password': 'bar'}

Neither Firefox or Chrome were able to use a proxy URL like http://foo:bar@p.webshare.io:80/, because the :80 was being stripped off by HTTPX. httpx.URL.port cannot be set to 80 (for HTTP) or 443 (HTTPS), see example below.

This change explicitly sets the proxy port if it's unset.

In [1]: from httpx import URL

In [2]: url = URL(scheme="http", host="example.com", port=8080)

In [3]: url.port
Out[3]: 8080

In [4]: url = URL(scheme="http", host="example.com", port=80)

In [5]: url.port

In [6]: url = URL(scheme="http", host="example.com", port=443)

In [7]: url.port
Out[7]: 443

In [8]: url = URL(scheme="https", host="example.com", port=443)

In [9]: url.port

Issues

Closes #618

Testing

Tested locally, Firefox is now able to load pages using a proxy with port 80.

Checklist

  • CI passed

Fixes an issue where the string `:None` was being appended to Playwright Proxy configurations.

```
proxy_info: {'server': 'http://p.webshare.io:None', 'username': 'foo', 'password': 'bar'}
```

Neither Firefox or Chrome were able to use a URL like `http://foo:bar@p.webshare.io:80/`, because the :80 was being stripped off by HTTPX. `httpx.URL.port` cannot be set to 80 (for HTTP) or 443 (HTTPS), see example below.

This change explicitly sets the proxy port if it's unset.

```
In [1]: from httpx import URL

In [2]: url = URL(scheme="http", host="example.com", port=8080)

In [3]: url.port
Out[3]: 8080

In [4]: url = URL(scheme="http", host="example.com", port=80)

In [5]: url.port

In [6]: url = URL(scheme="http", host="example.com", port=443)

In [7]: url.port
Out[7]: 443

In [8]: url = URL(scheme="https", host="example.com", port=443)

In [9]: url.port
```
@steffansafey steffansafey changed the title Default ProxyInfo port if httpx.URL port is None fix: Default ProxyInfo port if httpx.URL port is None Oct 24, 2024
Copy link
Collaborator

@vdusek vdusek left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks good, thank you! I just did some polishing... And one more thing, could you please provide also a simple unit test covering this? It should not be hard. Place it on tests/unit/proxy_configuration/test_new_proxy_info.py.

@janbuchar
Copy link
Collaborator

...LGTM, but a test would be great, as @vdusek said.

@steffansafey
Copy link
Contributor Author

Cheers @janbuchar, @vdusek, I'll send the test through in the next hour or so 👍

@steffansafey steffansafey requested a review from vdusek October 29, 2024 13:34
@steffansafey
Copy link
Contributor Author

Ready to go again @vdusek, covered a few cases in the test to ensure we can set it explicitly & implicitly. Tests fail when the fix is rolled back so all looking good!

Happy for you to touch it up before merge if needed. Good call adding the ValueError, I'd wondered if there were any other cases this could occur... Nicer to fail loudly!

@vdusek vdusek merged commit 8107a6f into apify:master Oct 29, 2024
@vdusek
Copy link
Collaborator

vdusek commented Oct 29, 2024

Thank you @steffansafey and good job!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Unable to use proxy with port 80 or 443 with Playwright

3 participants