Skip to content

Conversation

zmoog
Copy link
Contributor

@zmoog zmoog commented Sep 11, 2024

Proposed commit message

Align client|source.geo.location fields to ECS mapping.

Users reported mapping exceptions due to Elasticsearch mapping theclient|source.geo.location fields as object instead of geo_point. See #10848 for more.

Checklist

  • I have reviewed tips for building integrations and this pull request is aligned with them.
  • I have verified that all data streams collect metrics or logs.
  • I have added an entry to my package's changelog.yml file.
  • I have verified that Kibana version constraints are current according to guidelines.

Related issues

@andrewkroh andrewkroh added the Integration:azure Azure Logs label Sep 11, 2024
@zmoog zmoog self-assigned this Sep 11, 2024
@zmoog zmoog added the bug Something isn't working, use only for issues label Sep 11, 2024
@zmoog
Copy link
Contributor Author

zmoog commented Sep 11, 2024

According to the Client Fields and Source Fields ECS reference, we should map client.geo.location and source.geo.location using the geo_point type.

However, the packages/azure/data_stream/graphactivitylogs/fields/ecs.yml has the following mapping:

- name: client.geo.location.lat
  external: ecs
- name: client.geo.location.lon
  external: ecs
- name: source.geo.location.lat
  external: ecs
- name: source.geo.location.lon
  external: ecs

That causes Elasticsearch to map the client.geo.location and source.geo.location fields as object.

We should probably change the mapping to:

- name: client.geo.location
  external: ecs
- name: source.geo.location
  external: ecs

To align these fields with ECS and produce the expected geo_point mapping.

@kcreddy, are there specific reasons to use the *.lat|.lon mapping?

@andrewkroh andrewkroh added bugfix Pull request that fixes a bug issue and removed bug Something isn't working, use only for issues labels Sep 11, 2024
@elasticmachine
Copy link

🚀 Benchmarks report

Package azure 👍(6) 💚(4) 💔(1)

Expand to view
Data stream Previous EPS New EPS Diff (%) Result
provisioning 3225.81 2398.08 -827.73 (-25.66%) 💔

To see the full report comment with /test benchmark fullreport

@zmoog zmoog marked this pull request as ready for review September 11, 2024 22:36
@zmoog zmoog requested review from a team as code owners September 11, 2024 22:36
@andrewkroh andrewkroh added the Team:Security-Service Integrations Security Service Integrations team [elastic/security-service-integrations] label Sep 11, 2024
@elasticmachine
Copy link

Pinging @elastic/security-service-integrations (Team:Security-Service Integrations)

Copy link
Contributor

@efd6 efd6 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This needs a changelog and manifest update.

Copy link
Contributor

@muthu-mps muthu-mps left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

code owner approval.

Copy link
Contributor

@kcreddy kcreddy left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@kcreddy, are there specific reasons to use the *.lat|.lon mapping?

I don't see one. Must be a miss from previous datastreams which missed this fix.
Thanks for fixing 👍🏼

- version: "1.14.1"
changes:
- description: Fix [client|source].geo.location ECS field mapping
type: enhancement
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You labeled the PR as a bug and the patch version was incremented so all signals suggest this should be type: bugfix instead of enhancement. Can you confirm the intention.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hey Andrew, thanks for the heads up. This is a bug; updating the changelog classification accordingly.

@zmoog zmoog enabled auto-merge (squash) September 17, 2024 09:24
@zmoog zmoog merged commit 6d76da8 into elastic:main Sep 17, 2024
2 checks passed
@elasticmachine
Copy link

💚 Build Succeeded

History

cc @zmoog

Copy link

harnish-crest-data pushed a commit to chavdaharnish/integrations that referenced this pull request Feb 4, 2025
)

Align `client|source.geo.location` fields to ECS mapping.

Users reported mapping exceptions due to Elasticsearch mapping the `client|source.geo.location` fields as `object` instead of `geo_point`. See elastic#10848 for more.
harnish-crest-data pushed a commit to chavdaharnish/integrations that referenced this pull request Feb 5, 2025
)

Align `client|source.geo.location` fields to ECS mapping.

Users reported mapping exceptions due to Elasticsearch mapping the `client|source.geo.location` fields as `object` instead of `geo_point`. See elastic#10848 for more.
@zmoog zmoog deleted the zmoog/fix-azire-graphactivitylogs-geo-location branch February 6, 2025 19:45
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bugfix Pull request that fixes a bug issue Integration:azure Azure Logs Team:Security-Service Integrations Security Service Integrations team [elastic/security-service-integrations]
Projects
None yet
Development

Successfully merging this pull request may close these issues.

6 participants