Python docstrings mistakenly rendered as comment

[lkct]

Does this issue occur when all extensions are disabled?: Yes/No

• VS Code Version: Code 1.79.0 (b380da4, 2023-06-07T14:26:35.552Z)
• OS Version: Windows_NT x64 10.0.22621

Steps to Reproduce:

1. Update to 1.79.0. Use any bundled theme.
2. Open any Python file with docstring (or copy-paste the example below). All docstrings are rendered in the same style as comments.

Related Issue/PR

• The issue that proposed the change: Python multi-line comment strings should be treated as comments in all themes #182163
  • Yes, as the title suggests, for multi-line comment strings, I agree they should be treated as comments because they are "multi-line comment strings".
• The PR that did the change: all color themes: treat comment docstrings as comments too #182162
  • I'm against this change as it actually swapped the concept: the title now mentions docstrings, and it does affect all docstrings instead of only "multi-line comment strings".

From Python Docs

docstring
A string literal which appears as the first expression in a class, function or module. While ignored when the suite is executed, it is recognized by the compiler and put into the __doc__ attribute of the enclosing class, function or module. Since it is available via introspection, it is the canonical place for documentation of the object.

Takeaway of the above:

1. docstring is a Python language feature associated with the dunder attribute __doc__. It is NOT a comment.
2. Not every string literal is a docstring. Only those that appear as the 1st statement of a scope are docstrings.

Therefore, I think the current behaviour is WRONG as docstrings are not comments, but I agree with issue #182163 that comment strings should be rendered as comments.

Additional context

The Python extension additionally treats some non-docstring strings similar to a docstring, like the B and e in this example:

class A:
    """This is the docstring for A. Shown on hover."""

    B = 1
    """Shown on hover for B."""

    def c(self, x:int)->int:
        """This is the docstring for c. Shown on hover."""
        d = 1
        """This is nothing but a comment."""  # pylint: pointless-string-statement / W0105

e = 1
"""Shown on hover for e."""

Hovering on A B c e will all show the respective string content in VSCode. However, for B and e they're actually not docstrings, but it's indeed useful if we can add some "doc"strings for global variables and class attributes.
Also, some linters, e.g. pylint, share the same way to interpret these string literals: pylint generates a pointless-string-statement warning only on the "comment", but not at the other 4 places.

Thus, a better solution could be to expose extra rendering tokens and let the Python extension decide whether each string literal should be treated as a docstring or a comment string.

Finally, many thanks to everyone who has participated in the fruitful discussion in #182162 (cc @ElectricRCAircraftGuy @aeschli @k98kurz @starball5 @torext @dbaeumer). Further comments are welcomed.

[sztal]

I confirm that this bug occurs also for Debian-like systems.

VS Code version:

1.79.0
b380da4ef1ee00e224a15c1d4d9793e27c2b6302
x64

OS

Distributor ID:	Pop
Description:	Pop!_OS 22.04 LTS
Release:	22.04
Codename:	jammy

[lobsterkatie]

I think the discussion over in #182162 proves that whether or not the change was a mistake is a matter of opinion. It feels kinda like overkill, but maybe it should be a setting?

[lkct]

I think the discussion over in #182162 proves that whether or not the change was a mistake is a matter of opinion. It feels kinda like overkill, but maybe it should be a setting?

Yes, it's a matter of opinion, because both are wrong, and you just pick an error that you feel less important. And Yes, it's already a setting (though not enough to get everything correct).

I don't think this issue is overkill, because what I'm proposing here is to get things correct. However, there's no immediate solution.

Maybe I was not highlighting it (and I am now). The final conclusion above is to introduce new tokens to separate "docstrings" and "comment strings" as they're different things in Python.

[PeterJCLaw]

Thanks for putting together this summary and linked PR. I broadly agree with your description of things, however I'm not a fan of treating unassigned strings as "comment strings". As far as I know "comment strings" (multi-line or otherwise) aren't a formal concept in Python¹.

Instead, as you may be aware, the fact that strings can appear anywhere as statements on their own is a consequence of any expression being a valid statement (this is of course what also allows docstrings to work too). This also applies to everything else -- the code sample below is completely valid, yet does nothing.

2 + 2

"str"
b"bytes"

"foo" + "foo"

"bar" * 2

["quox"] * 2

"a" if True else "B"

I don't think² anyone is suggesting that the syntax highlighting for bare expressions (as in the above) be changed. Following from that I would expect that strings in such places to be considered as strings, and not comments at all.
While some users do use them for commenting, that use is discouraged [^3] by PEP8 at least and I believe somewhat uncommon.

In terms of what makes a docstring I can definitely see arguments for:

• the strict set of things which end up in __doc__
• the slightly wider set used by Jedi/Sphinx/Pylance, expanding as shown in the original post here

It's not clear to me which of these is preferable. I'd probably opt for the latter for usability.
There's also the matter of the >>> highlighting for code samples/doctests, I wonder if that should only work within strings which will actually be picked up by doctests?

My preference would be for:

• strings defaulting to a common colour
• the "docstring" token being limited to true docstrings
• document the docstring token
• maybe encourage some themes to make the docstring colour stand out/back slightly? (As a variant of string, not comment)
• not treating any type of string as a comment

I think this is broadly in line with the description in this issue, with the difference that I don't think that bare string literals should ever be coloured as comments.

Footnotes

1. I'd be happy to read any references you have about them though! ↩

2. At least I really hope not! ↩

[lkct]

@PeterJCLaw I generally agree with your points -- just some comments.

As far as I know "comment strings" (multi-line or otherwise) aren't a formal concept in Python1.

Indeed non-docstring string statements are nothing special, as far as I know.

I don't think2 anyone is suggesting that the syntax highlighting for bare expressions (as in the above) be changed.

That's a point. All those you listed are essentially the same --pointless-statement as pylint would warn. It seems strings here should be strings.

While some users do use them for commenting, that use is discouraged [^3] by PEP8 at least and I believe somewhat uncommon.

I do see people using them as comments, although discouraged by PEP-8. Whether it's common really depends on what code you usually see. Since this is not the suggested way to write comments, not rendering them as comments also makes sense. (It's better if some Intellisense engine can figure out whether a string is intended to be a comment.)
(PS: I assume you missed a footnote here?)

BTW, I just found pylint also shares the same opinion as pylance etc. I'm updating it in the "Additional Context".

[PeterJCLaw]

use is discouraged [^3] by PEP8

(PS: I assume you missed a footnote here?)

Oops, yeah. I think I was going to put the link to PEP-8 plus some additional content in a footnote but then edited in a way that didn't need it and missed removing the placeholder.