JSX/regexp ambiguity in jslexer

[remram44]

After a regular expression like />/, Babel ignores all messages until the next one.

Mapping:

[javascript: **.js]
encoding = utf-8
extract_messages = gettext

Source file:

alert(gettext("First message"));
var expr = />/g;
alert(gettext("Second message"));
var expr2 = /'/g;
alert(gettext("Third message"));

Result POT:

"Generated-By: Babel 2.6.0\n"

#: test.js:1
msgid "First message"
msgstr ""

#: test.js:5
msgid "Third message"
msgstr ""

[akx]

The JavaScript lexer in Babel is far from perfect, unfortunately...
If you feel like trying to hack at the lexer to fix this, by all means please do!

It might be a good idea for a future version to use e.g. https://github.com/Kronuz/esprima-python when it's available (i.e. an optional dependency) for more robust parsing.

[remram44]

Thanks, I might take a look. I am low on time unfortunately right now.

For reference, lexer is jslexer.py

[remram44]

Seems that /> gets recognized as a JSX tag.

[remram44]

I don't see how to fix this. The tokenizer can't possibly tell if /> is the start of a regexp or the end of a JSX tag, they are the same tokens. The Javascript grammar is too ambiguous for this to be decided at the tokenizer level.

A parser would be able to tell if it's currently in a JSX tag or not and resolve the ambiguity, but there is no such parsing being done in Babel. Simply keeping track of JSX open/close tokens wouldn't be enough because regexps can appear inside JSX tags (<hr class={ /\w+/.exec(var)[0] }/>).

I think I'll have to leave that alone, sorry!

Wrote a test:

def test_regex():
    assert list(jslexer.tokenize('''
        re = />/g;
    ''')) == [
        ('name', 're', 2),
        ('operator', '=', 2),
        ('regexp', '/>/', 2),
        ('operator', ';', 2),
    ]