Fix Sign-in logs location.state field to region field

[muthu-mps]

• Bug

Proposed commit message

Add a new field region_name to map the location.state data to this field. Initially this was mapped to country_name which is not appropriate.

Added script to drop the null/empty values in the document and updated with more descriptive on_failure error message.

Checklist

• I have reviewed tips for building integrations and this pull request is aligned with them.
• I have verified that all data streams collect metrics or logs.
• I have added an entry to my package's changelog.yml file.
• I have verified that Kibana version constraints are current according to guidelines.
• I have verified that any added dashboard complies with Kibana's Dashboard good practices

Author's Checklist

Install the integration and verify the signin logs captures the state in region_name field instead of country_name.

[elastic-vault-github-plugin-prod]

🚀 Benchmarks report

To see the full report comment with /test benchmark fullreport

[zmoog]

LGTM

[muthu-mps]

/test

[agithomas]

@muthu-mps , could you check the dependency of the removed field on the dashboard, especially :

kibana/dashboard/azure-91224490-f1a6-11ec-a5a8-bf965bcd5646.json ?

[agithomas]

Suggestion:

I understand that the field geo.country_name is having the value intended for the geo.region_name.

As with the case mentioned here, to minimize the impact, do you want to consider first deprecating the field geo.country_name and then later removing this field?

[elastic-sonarqube]

Quality Gate failed

Failed conditions
69.2% Coverage on New Code (required ≥ 80%)

See analysis details on SonarQube

[elasticmachine]

💚 Build Succeeded

• Buildkite Build
• Commit: 9fe2e0a

History

• 💔 Build #24641 failed 9fe2e0a
• 💚 Build #24415 succeeded 03451ed

cc @muthu-mps

[muthu-mps]

@muthu-mps , could you check the dependency of the removed field on the dashboard, especially :

kibana/dashboard/azure-91224490-f1a6-11ec-a5a8-bf965bcd5646.json ?

• The file which you have pointed above has a valid reference which is either source.geo.country_name or destination.geo.name.

• I was checking the dashboard and found the geo.country_name field reference in the Caller IP table chart of the Activity logs data stream in this dashboard kibana/dashboard/azure-87095750-f05a-11e9-90ec-112a988266d5.

This is not related to the sign-in logs data stream.

Do we have similar filed mapping issue for activity logs?

While looking into the ingest processor there is no incorrect mapping to the state field. Eventually, we need to revisit the activity logs data stream as well but not as part of this PR.

[muthu-mps]

Suggestion:

I understand that the field geo.country_name is having the value intended for the geo.region_name.

As with the case mentioned here, to minimize the impact, do you want to consider first deprecating the field geo.country_name and then later removing this field?

I am not sure that we can remove this field. This is kept for backward compatibility as mentioned here.

[efd6]

LGTM for security-service-integrations owned files.

[agithomas]

LGTM!

[elastic-vault-github-plugin-prod]

Package azure - 1.23.1 containing this change is available at https://epr.elastic.co/package/azure/1.23.1/