Test Templates

[mefellows]

When generating pact tests, the output does not always match the existing style, idioms, frameworks and SDK versions as the current project the engineer is working on. For example, the test might generate chai matchers and assertions, for a project that uses Jest.

Additionally, the style of the test may not be consistent with the existing tests.

One way to address this problem, and improve the output, is to allow users to provide their own templates with their existing patterns and style encoded in them, when asking PactFlow to generate tests. PactFlow will use this as a template to inject the test cases into.

1. Ability to provide a (local) template when generating tests
2. Ability to pass additional text-based instructions with the template to guide generation
3. New permission will be created to allow users to specify local templates

In a future capability, we will support a template library within PactFLow:

1. Ability to upload templates to PactFlow into a template library (via API or UI)
2. Templates may be associated with the team or no team (follow the secrets model)
3. UI to manage the templates
4. Ability to restrict all of the above via RBAC
5. Permission to allow creating/updating/deleting (managing) and reading existing templates (at the * and team scope)

[milda-a]

Following 👀 as discussed previously, it would be great if the first iteration could also include the ability to specify error code validation, as the current AI does not touch the error handling it seems. Thanks! ❤

[mefellows]

It sure does! You can provide additional instructions to request this.

You might be aware that you can already specify negative scenarios when using the OAS (see below) or request-response (it will use what's given in the example). It can occasionally be "lazy" when using the code option, but you might find that some additional comments in the code to help make any implicit scenarios explicit.

From pactflow-ai generate help:

      --endpoint <ENDPOINT>
          Endpoint(s) to generate Pacts for

          The endpoint(s) for which a Pact should be generated. This can either be a simple string to match the path:

          > --endpoint /api/v1/user/*

          Or a more complex matcher as an inline JSON object:

          > --endpoint '{"path": "/api/v1/user/*", "methods": ["GET"]}'

          For the latter, the following fields are supported:

          - path:        The path to match (supports ?, * and ** wildcards)

          - methods:     The HTTP method to match (case-insensitive)

          - status:      The HTTP status code to match. Can be an integer, a string with X denoting a wildcard (e.g., 2XX), or a list of the former two.

          - operationId: The operation ID to match (supports ?, and * wildcards)

[milda-a]

Oooh nice, I will check that out today!

[mefellows]

And we're live 🎉