perf: Improve import time by lazy import "slow" modules

[hoxbro]

This PR improves the import time by postponing the import of asyncio, logging, and html. And also not updating signatures if not inside a IPython environment.

Import time + tuna

Generated with python -X importtime -c 'import param' 2> tuna.log && tuna tuna.log. Some notes only focus on the param and ignore the param.version time; also, the measurement is not exact (see hyperfine) and needs warmup time after a branch change.

Main (9ff6a9f):

After removing asyncio (5a63a9e):

+ After removing logging and HTML (3d4885b)

+ Without updating signature in __init_submodules__ (30180b8):

Pyinstrument

Also used pyinstrument with pyinstrument -r html example.py where example.py is import param

Main (9ff6a9f)

This PR (30180b8):

hyperfine

Main (9ff6a9f)

❯ hyperfine "python -c ''" "python -c 'import param'" --warmup 5
Benchmark 1: python -c ''
  Time (mean ± σ):      15.9 ms ±   0.6 ms    [User: 11.8 ms, System: 3.9 ms]
  Range (min … max):    14.4 ms …  17.6 ms    180 runs

Benchmark 2: python -c 'import param'
  Time (mean ± σ):      57.9 ms ±   1.2 ms    [User: 47.5 ms, System: 9.9 ms]
  Range (min … max):    56.0 ms …  61.2 ms    50 runs

Summary
  python -c '' ran
    3.65 ± 0.16 times faster than python -c 'import param'

This PR (30180b8):

❯ hyperfine "python -c ''" "python -c 'import param'" --warmup 5
Benchmark 1: python -c ''
  Time (mean ± σ):      15.8 ms ±   0.6 ms    [User: 11.7 ms, System: 4.0 ms]
  Range (min … max):    14.4 ms …  17.6 ms    187 runs

Benchmark 2: python -c 'import param'
  Time (mean ± σ):      37.1 ms ±   0.9 ms    [User: 29.8 ms, System: 7.0 ms]
  Range (min … max):    35.4 ms …  39.2 ms    79 runs

Summary
  python -c '' ran
    2.35 ± 0.11 times faster than python -c 'import param'

[codecov]

Codecov Report

Attention: Patch coverage is 94.11765% with 1 line in your changes missing coverage. Please review.

Project coverage is 87.46%. Comparing base (9ff6a9f) to head (eec0101).
Report is 2 commits behind head on main.

Files with missing lines	Patch %	Lines
param/_utils.py	66.66%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #1037      +/-   ##
==========================================
- Coverage   87.46%   87.46%   -0.01%     
==========================================
  Files           9        9              
  Lines        4948     4955       +7     
==========================================
+ Hits         4328     4334       +6     
- Misses        620      621       +1     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[hoxbro]

This was added to #742, and as far as I can see, it was done to improve the experience in IPython.

[hoxbro]

Maybe also the doc experience, though I'm unsure if this comes from types or the signature.

[maximlt]

Wasn't there a regression in Panel recently because of hasattr(builtins, "__IPYTHON__")?

[hoxbro]

Yes, in holoviz/panel#7734. I pushed the change before that PR.

We could likely change it to hasattr(builtins, "__IPYTHON__") or hasattr(builtins, "get_ipython"), and make it a constant so it does not need to look up all the time.

[maximlt]

For my own understanding, why was the code changed in Panel and other places from try: get_ipython()/except to hasattr(builtins, "__IPYTHON__")?

We could likely change it to hasattr(builtins, "__IPYTHON__") or hasattr(builtins, "get_ipython"), and make it a constant so it does not need to look up all the time.

Wasn't the regression in Panel that hasattr(builtins, "__IPYTHON__") was True but get_python was not available? I'd like to understand when that can happen.

Btw Param has a _in_ipython utility function.

[maximlt]

What's the Python version used for all these benchmarks? Deferring the import of modules from the stdlib is a bit sad but well.

I recommend opening 2 PRs, one for the deferred stdlib imports, one for the signature.

[hoxbro]

What's the Python version used for all these benchmarks?

Python 3.12.9 | packaged by conda-forge | (main, Feb 14 2025, 08:00:06) [GCC 13.3.0]

I recommend opening 2 PRs, one for the deferred stdlib imports, one for the signature.

Honestly, I would rather not. I don't think there is any discussion about deferring imports, only around the signature, and both are related to import time.

[maximlt]

Honestly, I would rather not. I don't think there is any discussion about deferring imports, only around the signature, and both are related to import time.

Sorry I wasn't clear enough. It was not about discussing the changes, but about separating them. The deferred imports are an implementation detail imo, while the updated signature is a breaking change.

[hoxbro]

updated signature is a breaking change.

Hopefully not? If it was always meant to only work in IPython it should not be a breaking change.

[hoxbro]

Split into two PRs see #1038.

The result in the main post still contains the combined effort of these two prs.