Skip to content

Conversation

@lukaszsamson
Copy link
Contributor

This PR adds property tests that generate random strings in various elixir contexts.
Assumptions:
in tolerant mode

  1. if original parser returns ok, spitfire has to return exactly the same AST
  2. if the original parser crashes or errors, spitfire may not crash
    in strict mode
  3. if original parser returns ok, spitfire has to return exactly the same AST
  4. if the original parser crashes or errors skip spitfire and assume the case passed

@mhanberg mhanberg changed the title Add simple character based property test test: simple character based property test Jan 31, 2026
@mhanberg mhanberg force-pushed the ls-char-property-tests branch from 18a637e to ccd9a5f Compare February 9, 2026 12:53
@mhanberg
Copy link
Contributor

@doorgan some of the property failures in this one are related to not correctly counting new lines in some places.

@doorgan
Copy link
Contributor

doorgan commented Feb 10, 2026

Yes, I'm taking a look at this one now

@doorgan doorgan force-pushed the ls-char-property-tests branch from ccd9a5f to e85bdd3 Compare February 10, 2026 17:18
@doorgan doorgan force-pushed the ls-char-property-tests branch from e7744e8 to 8128f41 Compare February 12, 2026 15:40
@doorgan
Copy link
Contributor

doorgan commented Feb 12, 2026

Besides the metadata issues, a lot of the issues flagged by these property test were around this elixir grammar rule:

% Map and structs

map_base_expr -> sub_matched_expr : '$1'.
map_base_expr -> at_op_eol map_base_expr : build_unary_op('$1', '$2').
map_base_expr -> unary_op_eol map_base_expr : build_unary_op('$1', '$2').
map_base_expr -> ellipsis_op map_base_expr : build_unary_op('$1', '$2').

We were parsing the struct name as a "lone identifier" but the elixir grammar allows a whole range of very bizarre code like %^@_{}. I can't imagine any scenario where we'd parse code like that and realistically expect to get anything remotely useful with either parser, but the tests enforce parity between them so I fixed them too.

I also added a lot of those to spitfire_test.exs because property tests don't always generate examples that trigger the same failure type and hoping for them to generate them again is unreliable. Having values that were known to regress helped ensure the property tests weren't flaky. Otherwise I was getting 10 runs with no errors and suddenly a run that found 4 new errors.

I don't know what Oracle was supposed to mean, it just referred to the elixir parser results
@doorgan doorgan changed the title test: simple character based property test fix: parity with elixir parser based on property tests Feb 12, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants