[System] Add RFC 5424 support to Auth datastream

[kcreddy]

Proposed commit message

• Adds support for RFC 5424 to system.auth datastream using syslog filebeat processor.
  • For users directly utilising the assets such as ingest pipeline without agent/beat, changes are made to existing GROK patterns to support RFC 5424
• Add tags, improve error.message, and add pipeline_error to event.kind upon on_failure.

Checklist

• I have reviewed tips for building integrations and this pull request is aligned with them.
• I have verified that all data streams collect metrics or logs.
• I have added an entry to my package's changelog.yml file.
• I have verified that Kibana version constraints are current according to guidelines.

Author's Checklist

• [ ]

How to test this PR locally

elastic-package stack down && elastic-package build && elastic-package stack up --version=8.10.3 -d -v && eval "$(elastic-package stack shellinit)" && elastic-package test pipeline --generate -v

--- Test results for package: system - START ---
╭─────────┬─────────────┬───────────┬──────────────────────────┬────────┬──────────────╮
│ PACKAGE │ DATA STREAM │ TEST TYPE │ TEST NAME                │ RESULT │ TIME ELAPSED │
├─────────┼─────────────┼───────────┼──────────────────────────┼────────┼──────────────┤
│ system  │ auth        │ pipeline  │ test-5424.log            │ PASS   │  10.007542ms │
│ system  │ auth        │ pipeline  │ test-auth-rhel79.log     │ PASS   │   3.304583ms │
│ system  │ auth        │ pipeline  │ test-auth-ubuntu1204.log │ PASS   │  21.605917ms │
│ system  │ auth        │ pipeline  │ test-auth.log            │ PASS   │   4.547708ms │
│ system  │ auth        │ pipeline  │ test-multiline.log       │ PASS   │   3.199833ms │
│ system  │ auth        │ pipeline  │ test-secure-rhel7.log    │ PASS   │   3.846833ms │
│ system  │ auth        │ pipeline  │ test-timestamp.log       │ PASS   │   2.669458ms │
╰─────────┴─────────────┴───────────┴──────────────────────────┴────────┴──────────────╯
--- Test results for package: system - END   ---
Done

elastic-package stack down && elastic-package build && elastic-package stack up --version=8.10.3 -d -v && eval "$(elastic-package stack shellinit)" && elastic-package test system --generate -v

--- Test results for package: system - START ---
╭─────────┬─────────────┬───────────┬───────────┬────────┬───────────────╮
│ PACKAGE │ DATA STREAM │ TEST TYPE │ TEST NAME │ RESULT │  TIME ELAPSED │
├─────────┼─────────────┼───────────┼───────────┼────────┼───────────────┤
│ system  │ auth        │ system    │ default   │ PASS   │ 37.581683125s │
╰─────────┴─────────────┴───────────┴───────────┴────────┴───────────────╯
--- Test results for package: system - END   ---
Done

Related issues

• Relates https://github.com/elastic/sdh-beats/issues/3895

Screenshots

[elasticmachine]

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

• Start Time: 2023-10-26T10:20:46.359+0000

• Duration: 17 min 15 sec

Test stats 🧪

Test	Results
Failed	0
Passed	154
Skipped	0
Total	154

🤖 GitHub comments

Expand to view the GitHub comments

To re-run your PR in the CI, just comment with:

• /test : Re-trigger the build.

[elasticmachine]

🌐 Coverage report

Name	Metrics % (covered/total)	Diff
Packages	100.0% (3/3)	💚
Files	100.0% (4/4)	💚
Classes	100.0% (4/4)	💚
Methods	65.476% (55/84)	👎 -19.246
Lines	98.415% (2981/3029)	👍 1.801
Conditionals	100.0% (0/0)	💚

[elasticmachine]

Pinging @elastic/security-external-integrations (Team:Security-External Integrations)

[efd6]

I'm a little confused by some of this.

[muthu-mps]

I have created an issue to keep track of updating the ECS version and reference the additional sys.log fields from there.

[elasticmachine]

Package system - 1.47.0 containing this change is available at https://epr.elastic.co/search?package=system