[beat] Create beat package #4708

klacabane · 2022-11-23T13:05:56Z

Summary

Closes elastic/kibana#144995

Creates a Beat package that supports monitoring of standalone beats and apm-server from Agent.

Testing

see [Stack Monitoring] Add support for beats datastream patterns kibana#146184 for e2e testing
the package implements static and system tests. System tests are configured with two variants, a metricbeat service and an apm-server

elasticmachine · 2022-11-23T13:11:02Z

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

Start Time: 2022-12-16T16:30:03.135+0000
Duration: 16 min 56 sec

Test stats 🧪

Test	Results
Failed	0
Passed	8
Skipped	0
Total	8

🤖 GitHub comments

Expand to view the GitHub comments

To re-run your PR in the CI, just comment with:

/test : Re-trigger the build.

elasticmachine · 2022-11-28T15:43:30Z

🌐 Coverage report

Name	Metrics % (`covered/total`)	Diff
Packages	100.0% (`0/0`)	💚
Files	100.0% (`0/0`)	💚
Classes	100.0% (`0/0`)	💚
Methods	100.0% (`6/6`)	💚
Lines	100.0% (`0/0`)	💚 7.4
Conditionals	100.0% (`0/0`)	💚

crespocarlos

Cool. I've quickly looked at it, but I haven't run any tests yet. First, I've got a question on the package name: beats or beat?

packages/beat/_dev/build/docs/README.md

klacabane · 2022-12-13T11:56:19Z

Looks like the system tests fail because the stack started by the CI is older than the minimum kibana version of the package (>= 8.7.0). I thought the constraint was ignored when testing, I'll take a look

miltonhultgren

LGTM 👍🏼 Tested and works well for both variants

packages/beat/manifest.yml

Co-authored-by: Jaime Soriano Pastor <jaime.soriano@elastic.co>

packages/beat/manifest.yml

Co-authored-by: Jaime Soriano Pastor <jaime.soriano@elastic.co>

jsoriano · 2022-12-15T12:24:36Z

Some failures now are related to the upgrade of the format version, look here for troubleshooting them: https://github.com/elastic/elastic-package/blob/main/docs/howto/update_major_package_spec.md#troubleshooting-upgrades-from-package-spec-v1-to-v2

packages/beat/manifest.yml

klacabane · 2022-12-15T14:04:49Z

System tests are complaining about the dataset format when we overwrite it in the manifest.yml, we may not check that field when doing the comparison. edit: it should probably be picked up here

[0] field "data_stream.dataset" should have value "beat.stats", it has "beats.stack_monitoring.stats"
[1] field "event.dataset" should have value "beat.stats", it has "beats.stack_monitoring.stats"

jsoriano · 2022-12-15T15:18:44Z

System tests are complaining about the dataset format when we overwrite it in the manifest.yml, we may not check that field when doing the comparison. edit: it should probably be picked up here
[0] field "data_stream.dataset" should have value "beat.stats", it has "beats.stack_monitoring.stats"
[1] field "event.dataset" should have value "beat.stats", it has "beats.stack_monitoring.stats"

Yeah, it looks like something is missing there. I will take a look.

Co-authored-by: Jaime Soriano Pastor <jaime.soriano@elastic.co>

jsoriano · 2022-12-16T15:44:49Z

@klacabane elastic-package 0.70.1 should fix the issue with dataset checks, we have just updated to this version the integrations repository, please try to update this branch.

klacabane · 2022-12-16T16:14:38Z

/test

elasticmachine · 2022-12-19T14:04:32Z

Package beat - 0.0.1 containing this change is available at https://epr.elastic.co/search?package=beat

## Summary Closes #146686 Update apm and beats queries to read from the Beat package data streams created in elastic/integrations#4708. Also updates health API to fetch from the data streams. API tests follow up in #147755 ### Testing That's a fairly heavy workflow. I'm investigating ways to make that easier - build `beat` package with `elastic-package build`. The package is not published yet so you'll need to pull elastic/integrations#4708 locally - start a stack `elastic-package stack up -v -d --version 8.7.0-SNAPSHOT`. Make sure you do this within `integrations` repo so it picks up the previously built package - start a beat service with `elastic-package service up -v --variant metricbeat_8.7.0` - install both elasticsearch and beat packages from the kibana started by the stack command (`https://localhost:5601`). Nothing shows up in SM if we don't have elasticsearch data. - both packages are prerelease versions and we need to explicitly tell Integrations plugin to show them up[1] - elasticsearch hosts is `https://elasticsearch:9200` and beat _should be_ `http://elastic-package-service_beat_1:5066` but it may differ depending on your docker version - start a Kibana with this branch connected to the elasticsearch instance from the `stack up` command. [see howto](https://github.com/elastic/observability-dev/blob/main/docs/infra-obs-ui/stack-monitoring_integration-packages.md#connecting-a-local-kibana) - navigate to Stack Monitoring and verify the metricbeat is properly monitored - start an apm server with `elastic-package service up -v --variant apm_8.7.0` - navigate to Stack Monitoring and verify apm-server is properly monitored [1] <img width="853" alt="Screenshot 2022-12-01 at 00 20 20" src="https://user-images.githubusercontent.com/5239883/204929352-c7656679-f88c-4013-b1f8-cf7c67d4c830.png"> Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>

create beat package

d7551d1

klacabane added the Team:Infra Monitoring UI - DEPRECATED Label for the Infrastructure Monitoring UI team. - DEPRECATED - Use Team:obs-ux-infra_services label Nov 23, 2022

klacabane self-assigned this Nov 23, 2022

fix aliases

e4fc615

klacabane mentioned this pull request Nov 23, 2022

[Stack Monitoring] Add support for beats datastream patterns elastic/kibana#146184

Merged

klacabane added 7 commits November 23, 2022 17:40

clean fields

441c435

lint

1c05a13

add _dev docker container

d3c56dd

add system test

059ce09

add static tests

ff45e31

add codeowner entry

5681da8

update package owner

e6e5095

klacabane added 7 commits November 28, 2022 17:38

remove user

ac97212

change user

2548a46

command arg

7fe2a94

manifest changes

c41634e

README

a209ca7

remove unused mappings

e43d727

build

4733511

klacabane added the Integration:beat Beat label Nov 30, 2022

klacabane added 6 commits November 30, 2022 16:31

add apm fields

cd1c813

add apm-server variant

5dfdd64

build

183d17d

use system module for metricbeat service

51bdced

default env variables

fb109a4

add timestamp mapping

1c040e2

klacabane marked this pull request as ready for review December 1, 2022 15:09

miltonhultgren self-requested a review December 2, 2022 10:27

klacabane added 2 commits December 2, 2022 14:07

use 8.7.0 images

d6b9a80

add release property

4ab41e3

crespocarlos reviewed Dec 13, 2022

View reviewed changes

packages/beat/_dev/build/docs/README.md Show resolved Hide resolved

packages/beat/_dev/build/docs/README.md Outdated Show resolved Hide resolved

update elastic version

a78f46b

BenB196 mentioned this pull request Dec 13, 2022

Standalone Beats Monitoring/Logging Integration #2328

Closed

8 tasks

klacabane and others added 2 commits December 13, 2022 16:01

typo

a45f2dc

Add APM Server traffic script

b14854a

miltonhultgren approved these changes Dec 15, 2022

View reviewed changes

jsoriano reviewed Dec 15, 2022

View reviewed changes

packages/beat/manifest.yml Outdated Show resolved Hide resolved

Update packages/beat/manifest.yml

e572b64

Co-authored-by: Jaime Soriano Pastor <jaime.soriano@elastic.co>

jsoriano reviewed Dec 15, 2022

View reviewed changes

packages/beat/manifest.yml Outdated Show resolved Hide resolved

Update packages/beat/manifest.yml

17e747c

Co-authored-by: Jaime Soriano Pastor <jaime.soriano@elastic.co>

jsoriano reviewed Dec 15, 2022

View reviewed changes

packages/beat/manifest.yml Outdated Show resolved Hide resolved

lint

a1f628e

jsoriano reviewed Dec 15, 2022

View reviewed changes

packages/beat/manifest.yml Show resolved Hide resolved

add missing mapping

1ca1abd

jsoriano mentioned this pull request Dec 15, 2022

Fix expected dataset check in test runners elastic/elastic-package#1072

Merged

Update packages/beat/manifest.yml

063efde

Co-authored-by: Jaime Soriano Pastor <jaime.soriano@elastic.co>

Merge branch 'master' into create-beats-package

b2442fd

klacabane merged commit 6e35a3a into elastic:main Dec 19, 2022

andrewkroh added the New Integration Issue or pull request for creating a new integration package. label Aug 13, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

[beat] Create beat package #4708

[beat] Create beat package #4708

Uh oh!

klacabane commented Nov 23, 2022 •

edited

Loading

Uh oh!

elasticmachine commented Nov 23, 2022 •

edited

Loading

Build stats

Test stats 🧪

Uh oh!

elasticmachine commented Nov 28, 2022 •

edited

Loading

Uh oh!

crespocarlos left a comment

Uh oh!

Uh oh!

Uh oh!

klacabane commented Dec 13, 2022

Uh oh!

miltonhultgren left a comment

Uh oh!

Uh oh!

Uh oh!

jsoriano commented Dec 15, 2022

Uh oh!

Uh oh!

Uh oh!

klacabane commented Dec 15, 2022 •

edited

Loading

Uh oh!

jsoriano commented Dec 15, 2022

Uh oh!

jsoriano commented Dec 16, 2022

Uh oh!

klacabane commented Dec 16, 2022

Uh oh!

elasticmachine commented Dec 19, 2022

Uh oh!

Uh oh!

[beat] Create beat package #4708

[beat] Create beat package #4708

Uh oh!

Conversation

klacabane commented Nov 23, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Testing

Uh oh!

elasticmachine commented Nov 23, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

💚 Build Succeeded

Build stats

Test stats 🧪

🤖 GitHub comments

Uh oh!

elasticmachine commented Nov 28, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🌐 Coverage report

Uh oh!

crespocarlos left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

klacabane commented Dec 13, 2022

Uh oh!

miltonhultgren left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

jsoriano commented Dec 15, 2022

Uh oh!

Uh oh!

Uh oh!

klacabane commented Dec 15, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jsoriano commented Dec 15, 2022

Uh oh!

jsoriano commented Dec 16, 2022

Uh oh!

klacabane commented Dec 16, 2022

Uh oh!

elasticmachine commented Dec 19, 2022

Uh oh!

Uh oh!

klacabane commented Nov 23, 2022 •

edited

Loading

elasticmachine commented Nov 23, 2022 •

edited

Loading

elasticmachine commented Nov 28, 2022 •

edited

Loading

klacabane commented Dec 15, 2022 •

edited

Loading