[ti_recordedfuture] Add Support for Playbook and Triggered Alerts #13494

mohitjha-elastic · 2025-04-10T12:25:11Z

Proposed Commit Message

ti_recordedfuture: add support for playbook and triggered alerts as two new data streams

This adds support for the Playbook and Triggered Alerts event type as two new data streams to
enhance the overall visibility of threat data in the Recorded Future integration.

This changes also includes updating ecs version to 8.17.0 in all existing data streams.

Sanitized test case inputs were obtained from the Recorded Future API for Playbook Alerts[1]
and from the Recorded Future Connect API for Triggered Alerts[2].

[1] https://api.recordedfuture.com/playbook-alert
[2] https://api.recordedfuture.com/v2/#!/Alerts/Alert_Notification_Search

Checklist

I have reviewed tips for building integrations and this pull request is aligned with them.
I have verified that all data streams collect metrics or logs.
I have added an entry to my package's changelog.yml file.
I have verified that Kibana version constraints are current according to guidelines.

How to test this PR locally

Clone integrations repo.
Install the elastic package locally.
Start the elastic stack using the elastic package.
Move to integrations/packages/ti_recordedfuture directory.
Run the following command to run tests.
elastic-package test -v

Related issues

Closes [recorded_future] Add 2 alerts datastreams #13146

Screenshots

elasticmachine · 2025-04-10T12:25:14Z

Pinging @elastic/security-service-integrations (Team:Security-Service Integrations)

elastic-vault-github-plugin-prod · 2025-04-10T14:42:24Z

🚀 Benchmarks report

To see the full report comment with /test benchmark fullreport

kcreddy · 2025-04-24T09:19:29Z

packages/ti_recordedfuture/manifest.yml

 categories: ["security", "threat_intel"]
 conditions:
  kibana:
-    version: "^8.13.0 || ^9.0.0"
+    version: "^8.18.0 || ^9.0.0"


Because we are coupling agentless update with the new datastreams, users before 8.18.0 wont be able to use these new datastreams. Is that fine @jamiehynds / @cpascale43 ?

@kcreddy I think that's ok, should we document it somewhere though?

Is there a reason this could not be broken into two changes, one for the new data streams and one for the agentless update?

Same being done here #13680 (comment). Same clarification also applies there. cc: @janvi-elastic

I think we should decouple the agentless update and new datastream enhancements unless there is any strong reason not to.

@efd6 @kcreddy
The new data streams and dashboards have been implemented and tested on version 8.18. While downgrading to an earlier version is possible, it would require significant effort. Is there any key reason why we cannot proceed with the current implementation without decoupling?

Going forward, we will ensure that enhancement changes and agentless implementations are decoupled to avoid similar challenges.

Please let me know your thoughts.

Is there any key reason why we cannot proceed with the current implementation without decoupling?

More users can benefit from the enhancement. Example

Going forward, we will ensure that enhancement changes and agentless implementations are decoupled to avoid similar challenges.

I'm okay with this, but please make sure to de-couple agentless update for future enhancement tasks.

cc: @piyush-elastic

packages/ti_recordedfuture/_dev/build/docs/README.md

packages/ti_recordedfuture/data_stream/legacy_alert/agent/stream/cel.yml.hbs

kcreddy · 2025-04-24T09:27:59Z

packages/ti_recordedfuture/_dev/build/docs/README.md

@@ -10,15 +10,54 @@ from multiple entities, it's necessary to define one integration for each.
 Alternatively, it's also possible to use the integration to fetch custom Fusion files
 by supplying the URL to the CSV file as the _Custom_ _URL_ configuration option.

+It also fetches [Playbook Alerts](https://api.recordedfuture.com/playbook-alert) from the Recorded Future API and [Legacy Alerts](https://api.recordedfuture.com/v2/#!/Alerts/Alert_Notification_Search) via the Recorded Future Connect API, ensuring comprehensive threat intelligence coverage.


What was the reason for calling them Legacy Alerts? I don't see them as deprecated in the API doc (/v3).

In their Recorded Future APIs doc, under the description of connect api, its being written that this is the legacy alert. The same has been mentioned in the issue requirement.

@cpascale43 can you confirm if the alerts retrieved from Connect API should be called Legacy Alerts?
For example, the Microsoft connector for recorded future calls them Triggered Alerts just like the Connect API doc.

@kcreddy good catch, let's go with Triggered Alerts then

Thanks Carrie.
@mohitjha-elastic, please update the data_stream name.

@kcreddy Updated the PR with the changes. Can you please take a look?

#13494 (review)

...cordedfuture/data_stream/legacy_alert/_dev/test/pipeline/test-legacy-alert.log-expected.json

packages/ti_recordedfuture/data_stream/playbook_alert/agent/stream/cel.yml.hbs

packages/ti_recordedfuture/img/rf-legacy.png

1. Add uri_parts processor. 2. Update navigation link style in dashboard table of contents. 3. Remove blank space in readme. 4. Update dashboard image.

packages/ti_recordedfuture/_dev/build/docs/README.md

packages/ti_recordedfuture/_dev/deploy/docker/files/config-cel.yml

efd6 · 2025-05-05T03:36:02Z

packages/ti_recordedfuture/_dev/deploy/docker/files/config-cel.yml

+                  "note_author": null,
+                  "note_date": "2025-03-31T04:03:56.425Z",
+                  "status_date": "2025-03-31T04:03:56.425Z",
+                  "triggered": "2026-03-31T04:03:56.425Z",


Suggested change

"triggered": "2026-03-31T04:03:56.425Z",

"triggered": "2500-03-31T04:03:56.425Z",

efd6 · 2025-05-05T03:36:31Z

packages/ti_recordedfuture/_dev/deploy/docker/files/config-cel.yml

+                  "note_author": null,
+                  "note_date": "2025-03-31T04:03:56.425Z",
+                  "status_date": "2025-03-31T04:03:56.425Z",
+                  "triggered": "2027-03-31T04:03:56.425Z",


Suggested change

"triggered": "2027-03-31T04:03:56.425Z",

"triggered": "3000-03-31T04:03:56.425Z",

packages/ti_recordedfuture/data_stream/playbook_alert/agent/stream/cel.yml.hbs

packages/ti_recordedfuture/data_stream/legacy_alert/agent/stream/cel.yml.hbs

packages/ti_recordedfuture/data_stream/playbook_alert/manifest.yml

efd6 · 2025-05-05T03:50:05Z

packages/ti_recordedfuture/manifest.yml

 categories: ["security", "threat_intel"]
 conditions:
  kibana:
-    version: "^8.13.0 || ^9.0.0"
+    version: "^8.18.0 || ^9.0.0"


Is there a reason this could not be broken into two changes, one for the new data streams and one for the agentless update?

packages/ti_recordedfuture/data_stream/playbook_alert/elasticsearch/ingest_pipeline/default.yml

1. Rename legacy alert to triggered alert everywhere. 2. Add more data in system test. 3. Fix indentation in CEL code.

kcreddy

@mohitjha-elastic, can you change legacy to triggered in following places as well?

PR title
commit message
pipeline test file names

Rename pipeline test files from legacy to triggered alert.

kcreddy

LGTM for my comments

packages/ti_recordedfuture/_dev/deploy/docker/files/config-cel.yml

efd6 · 2025-05-12T06:19:38Z

packages/ti_recordedfuture/data_stream/playbook_alert/agent/stream/cel.yml.hbs

+                "error": {
+                  "code": string(resp.StatusCode),
+                  "id": string(resp.Status),
+                  "message": "POST " + state.url.trim_right("/") + "/playbook-alert/search" +


Suggested change

"message": "POST " + state.url.trim_right("/") + "/playbook-alert/search" +

"message": "POST " + state.url.trim_right("/") + "/playbook-alert/search: " +

efd6 · 2025-05-12T06:21:33Z

packages/ti_recordedfuture/data_stream/playbook_alert/agent/stream/cel.yml.hbs

+                  "message": "POST " + state.url.trim_right("/") + "/playbook-alert/" +
+                  (string(state.worklist.data[state.next].category) == "cyber_vulnerability" ?
+                    "vulnerability"
+                  :
+                    string(state.worklist.data[state.next].category))
+                  + "/" + string(state.worklist.data[state.next].playbook_alert_id) +
+                  (


I'm not sure we need this much detail here; this is just to allow the user to see which part of the program the request failed in, not the exact request.

Suggested change

"message": "POST " + state.url.trim_right("/") + "/playbook-alert/" +

(string(state.worklist.data[state.next].category) == "cyber_vulnerability" ?

"vulnerability"

:

string(state.worklist.data[state.next].category))

+ "/" + string(state.worklist.data[state.next].playbook_alert_id) +

(

"message": "POST " + state.url.trim_right("/") + "/playbook-alert/:" + (

efd6 · 2025-05-12T06:22:31Z

packages/ti_recordedfuture/data_stream/triggered_alert/agent/stream/cel.yml.hbs

+            "error": {
+              "code": string(resp.StatusCode),
+              "id": string(resp.Status),
+              "message": "GET "+ state.url.trim_right("/") + "/v3/alerts:" + (


Suggested change

"message": "GET "+ state.url.trim_right("/") + "/v3/alerts:" + (

"message": "GET "+ state.url.trim_right("/") + "/v3/alerts: " + (

Minor changes in the cel files, refactor the message in case of error.

elastic-sonarqube · 2025-05-12T07:37:13Z

Quality Gate passed

Issues
0 New issues
0 Fixed issues
0 Accepted issues

Measures
0 Security Hotspots
98.1% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube

elasticmachine · 2025-05-12T07:37:24Z

💚 Build Succeeded

Buildkite Build
Commit: 26be05e

History

💚 Build #25717 succeeded 20f54a9
💚 Build #25714 succeeded 9a6e680
💚 Build #25266 succeeded 666267b
💚 Build #24659 succeeded 5d0f50c
💔 Build #24656 failed 66eef07

cc @mohitjha-elastic

elastic-vault-github-plugin-prod · 2025-05-12T08:13:38Z

Package ti_recordedfuture - 1.30.0 containing this change is available at https://epr.elastic.co/package/ti_recordedfuture/1.30.0/

Add support for legacy and playbook alerts.

085e23b

mohitjha-elastic self-assigned this Apr 10, 2025

mohitjha-elastic requested a review from a team as a code owner April 10, 2025 12:25

mohitjha-elastic added 2 commits April 10, 2025 17:55

Update changelog entry.

66eef07

Add agentless deployment.

5d0f50c

kcreddy reviewed Apr 24, 2025

View reviewed changes

Resolve review comments.

666267b

1. Add uri_parts processor. 2. Update navigation link style in dashboard table of contents. 3. Remove blank space in readme. 4. Update dashboard image.

mohitjha-elastic requested a review from kcreddy April 29, 2025 11:41

efd6 reviewed May 5, 2025

View reviewed changes

kcreddy mentioned this pull request May 5, 2025

[panw_cortex_xdr] Add event data stream and dashboards of incident and alert #13680

Merged

4 tasks

Resolve review comments by @efd6 and @kcreddy.

9a6e680

1. Rename legacy alert to triggered alert everywhere. 2. Add more data in system test. 3. Fix indentation in CEL code.

mohitjha-elastic requested a review from efd6 May 8, 2025 09:42

kcreddy reviewed May 8, 2025

View reviewed changes

Rename pipeline test file.

20f54a9

Rename pipeline test files from legacy to triggered alert.

mohitjha-elastic changed the title ~~[ti_recordedfuture] Add Support for Legacy and Playbook Alerts~~ [ti_recordedfuture] Add Support for Playbook and Triggered Alerts May 8, 2025

mohitjha-elastic requested a review from kcreddy May 8, 2025 10:24

kcreddy approved these changes May 8, 2025

View reviewed changes

efd6 requested changes May 8, 2025

View reviewed changes

packages/ti_recordedfuture/_dev/deploy/docker/files/config-cel.yml Show resolved Hide resolved

mohitjha-elastic requested a review from efd6 May 9, 2025 04:39

efd6 reviewed May 12, 2025

View reviewed changes

Resolve comment by @efd6.

26be05e

Minor changes in the cel files, refactor the message in case of error.

mohitjha-elastic requested a review from efd6 May 12, 2025 07:12

efd6 approved these changes May 12, 2025

View reviewed changes

efd6 merged commit d8c91a6 into elastic:main May 12, 2025
7 checks passed

mohitjha-elastic deleted the ti_recordedfuture-1.30.0 branch May 15, 2025 07:22

mohitjha-elastic mentioned this pull request Aug 28, 2025

[ti_recordedfuture] Add Agentless Deployment Support #15080

Closed

5 tasks

	"triggered": "2026-03-31T04:03:56.425Z",
	"triggered": "2500-03-31T04:03:56.425Z",

	"triggered": "2027-03-31T04:03:56.425Z",
	"triggered": "3000-03-31T04:03:56.425Z",

	"message": "POST " + state.url.trim_right("/") + "/playbook-alert/search" +
	"message": "POST " + state.url.trim_right("/") + "/playbook-alert/search: " +

	"message": "GET "+ state.url.trim_right("/") + "/v3/alerts:" + (
	"message": "GET "+ state.url.trim_right("/") + "/v3/alerts: " + (

[ti_recordedfuture] Add Support for Playbook and Triggered Alerts #13494

[ti_recordedfuture] Add Support for Playbook and Triggered Alerts #13494

Uh oh!

Conversation

mohitjha-elastic commented Apr 10, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Proposed Commit Message

Checklist

How to test this PR locally

Related issues

Screenshots

Uh oh!

elasticmachine commented Apr 10, 2025

Uh oh!

elastic-vault-github-plugin-prod bot commented Apr 10, 2025

🚀 Benchmarks report

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

kcreddy May 8, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

kcreddy Apr 24, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

kcreddy left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

kcreddy left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

mohitjha-elastic commented Apr 10, 2025 •

edited

Loading

kcreddy May 8, 2025 •

edited

Loading

kcreddy Apr 24, 2025 •

edited

Loading

kcreddy left a comment •

edited

Loading

kcreddy left a comment •

edited

Loading