-
Notifications
You must be signed in to change notification settings - Fork 478
[Azure] Update Azure Firewall pipeline #9428
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
lucian-ioan
merged 52 commits into
elastic:main
from
lucian-ioan:update_azure_firewall_pipeline
Jul 11, 2024
Merged
Changes from 50 commits
Commits
Show all changes
52 commits
Select commit
Hold shift + click to select a range
bc1b365
add changelog entry
lucian-ioan f39ca8c
update pull number
lucian-ioan bf36a59
add AZFWNatRule sample log
lucian-ioan c11ef32
add AZFWNetworkRule sample log
lucian-ioan 9f07b8f
add AZFWApplicationRule sample log
lucian-ioan 1595ba7
update NatRule log type
lucian-ioan 600d310
update NetworkRule log type
lucian-ioan dd80d1f
update AZFWApplicationRule log type
lucian-ioan 78ad0e1
update resource id
lucian-ioan 36f8a92
elastic-package build
lucian-ioan 37f4d20
lowercase test files
lucian-ioan ff22bb6
remove redundant fields
lucian-ioan 7c6c1f8
fix bool to bolean type
lucian-ioan 32af0f0
add structured dns query logs
lucian-ioan 4da4444
update all test cases
lucian-ioan f9d1a77
reword changelog message
lucian-ioan 4e5b4c3
update ip addresses
lucian-ioan ba3bff3
fix azfw application rule json
lucian-ioan 167ce50
add meaningful rule names
lucian-ioan f2eaf47
use ecs fields for destination ip and port
lucian-ioan 1cfcff6
test pipeline
lucian-ioan d6e3d86
Merge remote-tracking branch 'origin' into update_azure_firewall_pipe…
lucian-ioan 3d427c0
fix merge
lucian-ioan e91c8a1
add description
lucian-ioan f1c3c14
rename categories
lucian-ioan 78f7876
merge main
lucian-ioan 35773bf
update log categories description
lucian-ioan 1d34f8c
use ECS fields for destination and port
lucian-ioan 8af6e49
docs spacing
lucian-ioan 56153c7
run pipeline tests
lucian-ioan 4e81ac2
use ECS fields
lucian-ioan 94464ac
fields.yml typo and description periods
lucian-ioan e582ec2
use more ECS fields
lucian-ioan 57525b7
fix ECS fields format
lucian-ioan 60fe43d
package build
lucian-ioan e164171
update docs
lucian-ioan c216dc5
cleanup ECS fields
lucian-ioan 071bd9d
improve test
lucian-ioan a94ab77
add action, reason and question type ECS fields
lucian-ioan f4fd40b
use json properties
lucian-ioan 9bdd073
pipeline generate
lucian-ioan 2c61a5e
add dns query
lucian-ioan 8c68c23
add network transport
lucian-ioan 699a945
ecs fields cleanup
lucian-ioan 3df6029
Update packages/azure/data_stream/firewall_logs/elasticsearch/ingest_…
lucian-ioan ba077b0
Update packages/azure/data_stream/firewall_logs/elasticsearch/ingest_…
lucian-ioan f293933
Update packages/azure/data_stream/firewall_logs/elasticsearch/ingest_…
lucian-ioan 4158fee
check for json category
lucian-ioan 5099d5d
Merge remote-tracking branch 'origin' into update_azure_firewall_pipe…
lucian-ioan 53102f1
merge main
lucian-ioan c313a28
add details about resource-specific mode
lucian-ioan 88c954c
update changelog
lucian-ioan File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
1 change: 1 addition & 0 deletions
1
...ure/data_stream/firewall_logs/_dev/test/pipeline/test-applicationrules-structured-raw.log
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1 @@ | ||
| {"category":"AZFWApplicationRule","properties":{"Action":"Allow","ActionReason":"reason","DestinationIp":"1.128.0.0","DestinationPort": 123,"Fqdn":"","IsExplicitProxyRequest":false,"IsTlsInspected":false,"Policy":"policy","Protocol":"HTTP","Rule":"ApplicationRule","RuleCollection":"ApplicationRuleSet","RuleCollectionGroup":"ApplicationRuleGroup","SourceIp":"1.128.0.0","SourcePort":1234,"TargetUrl":"https://www.microsoft.com/en-us/about","WebCategory":"category"},"resourceId":"/SUBSCRIPTIONS/23103928-B2CF-472A-8CDB-0146E2849129/RESOURCEGROUPS/TEST-FW-RG/PROVIDERS/MICROSOFT.NETWORK/AZUREFIREWALLS/TEST-FW01","time":"2024-03-20T23:39:59.8494370Z"} |
92 changes: 92 additions & 0 deletions
92
...m/firewall_logs/_dev/test/pipeline/test-applicationrules-structured-raw.log-expected.json
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,92 @@ | ||
| { | ||
| "expected": [ | ||
| { | ||
| "@timestamp": "2024-03-20T23:39:59.849Z", | ||
| "azure": { | ||
| "firewall": { | ||
| "category": "AZFWApplicationRule", | ||
| "is_explicit_proxy_request": false, | ||
| "is_tls_inspected": false, | ||
| "policy": "policy", | ||
| "rule_collection_group": "ApplicationRuleGroup", | ||
| "web_category": "category" | ||
| }, | ||
| "resource": { | ||
| "group": "TEST-FW-RG", | ||
| "id": "/SUBSCRIPTIONS/23103928-B2CF-472A-8CDB-0146E2849129/RESOURCEGROUPS/TEST-FW-RG/PROVIDERS/MICROSOFT.NETWORK/AZUREFIREWALLS/TEST-FW01", | ||
| "name": "TEST-FW01", | ||
| "provider": "MICROSOFT.NETWORK/AZUREFIREWALLS" | ||
| }, | ||
| "subscription_id": "23103928-B2CF-472A-8CDB-0146E2849129" | ||
| }, | ||
| "cloud": { | ||
| "account": { | ||
| "id": "23103928-B2CF-472A-8CDB-0146E2849129" | ||
| }, | ||
| "provider": "azure" | ||
| }, | ||
| "destination": { | ||
| "as": { | ||
| "number": 1221, | ||
| "organization": { | ||
| "name": "Telstra Pty Ltd" | ||
| } | ||
| }, | ||
| "ip": "1.128.0.0", | ||
| "port": 123 | ||
| }, | ||
| "ecs": { | ||
| "version": "8.11.0" | ||
| }, | ||
| "event": { | ||
| "action": "Allow", | ||
| "category": [ | ||
| "network" | ||
| ], | ||
| "kind": "event", | ||
| "original": "{\"category\":\"AZFWApplicationRule\",\"properties\":{\"Action\":\"Allow\",\"ActionReason\":\"reason\",\"DestinationIp\":\"1.128.0.0\",\"DestinationPort\": 123,\"Fqdn\":\"\",\"IsExplicitProxyRequest\":false,\"IsTlsInspected\":false,\"Policy\":\"policy\",\"Protocol\":\"HTTP\",\"Rule\":\"ApplicationRule\",\"RuleCollection\":\"ApplicationRuleSet\",\"RuleCollectionGroup\":\"ApplicationRuleGroup\",\"SourceIp\":\"1.128.0.0\",\"SourcePort\":1234,\"TargetUrl\":\"https://www.microsoft.com/en-us/about\",\"WebCategory\":\"category\"},\"resourceId\":\"/SUBSCRIPTIONS/23103928-B2CF-472A-8CDB-0146E2849129/RESOURCEGROUPS/TEST-FW-RG/PROVIDERS/MICROSOFT.NETWORK/AZUREFIREWALLS/TEST-FW01\",\"time\":\"2024-03-20T23:39:59.8494370Z\"}", | ||
kcreddy marked this conversation as resolved.
Show resolved
Hide resolved
|
||
| "reason": "reason", | ||
| "type": [ | ||
| "connection" | ||
| ] | ||
| }, | ||
| "network": { | ||
| "protocol": "http" | ||
| }, | ||
| "observer": { | ||
| "name": "TEST-FW01", | ||
| "product": "Network Firewall", | ||
| "type": "firewall", | ||
| "vendor": "Azure" | ||
| }, | ||
| "related": { | ||
| "ip": [ | ||
| "1.128.0.0" | ||
| ] | ||
| }, | ||
| "rule": { | ||
| "name": "ApplicationRule", | ||
| "ruleset": "ApplicationRuleSet" | ||
| }, | ||
| "source": { | ||
| "as": { | ||
| "number": 1221, | ||
| "organization": { | ||
| "name": "Telstra Pty Ltd" | ||
| } | ||
| }, | ||
| "ip": "1.128.0.0", | ||
| "port": 1234 | ||
| }, | ||
| "tags": [ | ||
| "preserve_original_event" | ||
| ], | ||
| "url": { | ||
| "domain": "www.microsoft.com", | ||
| "original": "https://www.microsoft.com/en-us/about", | ||
| "path": "/en-us/about", | ||
| "scheme": "https" | ||
| } | ||
| } | ||
| ] | ||
| } | ||
1 change: 1 addition & 0 deletions
1
packages/azure/data_stream/firewall_logs/_dev/test/pipeline/test-dnsproxy-structured-raw.log
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1 @@ | ||
| {"category":"AZFWDnsQuery","properties":{"DnssecOkBit":false,"EDNS0BufferSize":512,"ErrorMessage":"","ErrorNumber":0,"Protocol":"udp","QueryClass":"IN","QueryId":35817,"QueryName":"ntp.ubuntu.com.","QueryType":"A","RequestDurationSecs":0.0000286,"RequestSize":32,"ResponseCode":"NOERROR","ResponseFlags":"qr,aa,rd,ra","ResponseSize":152,"SourceIp":"1.128.0.0","SourcePort":47785},"resourceId":"/SUBSCRIPTIONS/56D199A8-724E-436D-B7F2-5F6F7694EABD/RESOURCEGROUPS/PROD-WESTUS3/PROVIDERS/MICROSOFT.NETWORK/AZUREFIREWALLS/PROD-WESTUS3","time":"2024-01-31T23:39:50.8623870Z"} |
93 changes: 93 additions & 0 deletions
93
...ta_stream/firewall_logs/_dev/test/pipeline/test-dnsproxy-structured-raw.log-expected.json
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,93 @@ | ||
| { | ||
| "expected": [ | ||
| { | ||
| "@timestamp": "2024-01-31T23:39:50.862Z", | ||
| "azure": { | ||
| "firewall": { | ||
| "category": "AZFWDnsQuery", | ||
| "dnssec_ok_bit": false, | ||
| "edns0_buffer_size": 512, | ||
| "request_duration_secs": 2.86E-5, | ||
| "request_size": 32, | ||
| "response_size": 152 | ||
| }, | ||
| "resource": { | ||
| "group": "PROD-WESTUS3", | ||
| "id": "/SUBSCRIPTIONS/56D199A8-724E-436D-B7F2-5F6F7694EABD/RESOURCEGROUPS/PROD-WESTUS3/PROVIDERS/MICROSOFT.NETWORK/AZUREFIREWALLS/PROD-WESTUS3", | ||
| "name": "PROD-WESTUS3", | ||
| "provider": "MICROSOFT.NETWORK/AZUREFIREWALLS" | ||
| }, | ||
| "subscription_id": "56D199A8-724E-436D-B7F2-5F6F7694EABD" | ||
| }, | ||
| "cloud": { | ||
| "account": { | ||
| "id": "56D199A8-724E-436D-B7F2-5F6F7694EABD" | ||
| }, | ||
| "provider": "azure" | ||
| }, | ||
| "dns": { | ||
| "header_flags": [ | ||
| "QR", | ||
| "AA", | ||
| "RD", | ||
| "RA" | ||
| ], | ||
| "id": "35817", | ||
| "question": { | ||
| "class": "IN", | ||
| "name": "ntp.ubuntu.com.", | ||
| "type": "A" | ||
| }, | ||
| "response_code": "NOERROR", | ||
| "type": "query" | ||
| }, | ||
| "ecs": { | ||
| "version": "8.11.0" | ||
| }, | ||
| "error": { | ||
| "id": "0" | ||
| }, | ||
| "event": { | ||
| "category": [ | ||
| "network" | ||
| ], | ||
| "kind": "event", | ||
| "original": "{\"category\":\"AZFWDnsQuery\",\"properties\":{\"DnssecOkBit\":false,\"EDNS0BufferSize\":512,\"ErrorMessage\":\"\",\"ErrorNumber\":0,\"Protocol\":\"udp\",\"QueryClass\":\"IN\",\"QueryId\":35817,\"QueryName\":\"ntp.ubuntu.com.\",\"QueryType\":\"A\",\"RequestDurationSecs\":0.0000286,\"RequestSize\":32,\"ResponseCode\":\"NOERROR\",\"ResponseFlags\":\"qr,aa,rd,ra\",\"ResponseSize\":152,\"SourceIp\":\"1.128.0.0\",\"SourcePort\":47785},\"resourceId\":\"/SUBSCRIPTIONS/56D199A8-724E-436D-B7F2-5F6F7694EABD/RESOURCEGROUPS/PROD-WESTUS3/PROVIDERS/MICROSOFT.NETWORK/AZUREFIREWALLS/PROD-WESTUS3\",\"time\":\"2024-01-31T23:39:50.8623870Z\"}", | ||
kcreddy marked this conversation as resolved.
Show resolved
Hide resolved
|
||
| "type": [ | ||
| "connection" | ||
| ] | ||
| }, | ||
| "network": { | ||
| "iana_number": "17", | ||
| "transport": "udp" | ||
| }, | ||
| "observer": { | ||
| "name": "PROD-WESTUS3", | ||
| "product": "Network Firewall", | ||
| "type": "firewall", | ||
| "vendor": "Azure" | ||
| }, | ||
| "related": { | ||
| "hosts": [ | ||
| "ntp.ubuntu.com." | ||
| ], | ||
| "ip": [ | ||
| "1.128.0.0" | ||
| ] | ||
| }, | ||
| "source": { | ||
| "as": { | ||
| "number": 1221, | ||
| "organization": { | ||
| "name": "Telstra Pty Ltd" | ||
| } | ||
| }, | ||
| "ip": "1.128.0.0", | ||
| "port": 47785 | ||
| }, | ||
| "tags": [ | ||
| "preserve_original_event" | ||
| ] | ||
| } | ||
| ] | ||
| } | ||
1 change: 1 addition & 0 deletions
1
packages/azure/data_stream/firewall_logs/_dev/test/pipeline/test-natrule-structured-raw.log
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1 @@ | ||
| {"category":"AZFWNatRule","properties":{"DestinationIp":"1.128.0.0","DestinationPort":999,"Policy":"policy","Protocol":"TCP","Rule":"NATSecurityRule","RuleCollection":"NATRuleSet","RuleCollectionGroup":"NATRuleGroup","SourceIp":"1.128.0.0","SourcePort":1234,"TranslatedIp":"1.128.0.0","TranslatedPort":999},"resourceId":"/SUBSCRIPTIONS/23103928-B2CF-472A-8CDB-0146E2849129/RESOURCEGROUPS/TEST-FW-RG/PROVIDERS/MICROSOFT.NETWORK/AZUREFIREWALLS/TEST-FW01","time":"2024-01-31T23:39:49.6798940Z"} |
82 changes: 82 additions & 0 deletions
82
...ata_stream/firewall_logs/_dev/test/pipeline/test-natrule-structured-raw.log-expected.json
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,82 @@ | ||
| { | ||
| "expected": [ | ||
| { | ||
| "@timestamp": "2024-01-31T23:39:49.679Z", | ||
| "azure": { | ||
| "firewall": { | ||
| "category": "AZFWNatRule", | ||
| "policy": "policy", | ||
| "rule_collection_group": "NATRuleGroup" | ||
| }, | ||
| "resource": { | ||
| "group": "TEST-FW-RG", | ||
| "id": "/SUBSCRIPTIONS/23103928-B2CF-472A-8CDB-0146E2849129/RESOURCEGROUPS/TEST-FW-RG/PROVIDERS/MICROSOFT.NETWORK/AZUREFIREWALLS/TEST-FW01", | ||
| "name": "TEST-FW01", | ||
| "provider": "MICROSOFT.NETWORK/AZUREFIREWALLS" | ||
| }, | ||
| "subscription_id": "23103928-B2CF-472A-8CDB-0146E2849129" | ||
| }, | ||
| "cloud": { | ||
| "account": { | ||
| "id": "23103928-B2CF-472A-8CDB-0146E2849129" | ||
| }, | ||
| "provider": "azure" | ||
| }, | ||
| "destination": { | ||
| "as": { | ||
| "number": 1221, | ||
| "organization": { | ||
| "name": "Telstra Pty Ltd" | ||
| } | ||
| }, | ||
| "ip": "1.128.0.0", | ||
| "port": 999 | ||
| }, | ||
| "ecs": { | ||
| "version": "8.11.0" | ||
| }, | ||
| "event": { | ||
| "category": [ | ||
| "network" | ||
| ], | ||
| "kind": "event", | ||
| "original": "{\"category\":\"AZFWNatRule\",\"properties\":{\"DestinationIp\":\"1.128.0.0\",\"DestinationPort\":999,\"Policy\":\"policy\",\"Protocol\":\"TCP\",\"Rule\":\"NATSecurityRule\",\"RuleCollection\":\"NATRuleSet\",\"RuleCollectionGroup\":\"NATRuleGroup\",\"SourceIp\":\"1.128.0.0\",\"SourcePort\":1234,\"TranslatedIp\":\"1.128.0.0\",\"TranslatedPort\":999},\"resourceId\":\"/SUBSCRIPTIONS/23103928-B2CF-472A-8CDB-0146E2849129/RESOURCEGROUPS/TEST-FW-RG/PROVIDERS/MICROSOFT.NETWORK/AZUREFIREWALLS/TEST-FW01\",\"time\":\"2024-01-31T23:39:49.6798940Z\"}", | ||
| "type": [ | ||
| "connection" | ||
| ] | ||
| }, | ||
| "network": { | ||
| "iana_number": "6", | ||
| "transport": "tcp" | ||
| }, | ||
| "observer": { | ||
| "name": "TEST-FW01", | ||
| "product": "Network Firewall", | ||
| "type": "firewall", | ||
| "vendor": "Azure" | ||
| }, | ||
| "related": { | ||
| "ip": [ | ||
| "1.128.0.0" | ||
| ] | ||
| }, | ||
| "rule": { | ||
| "name": "NATSecurityRule", | ||
| "ruleset": "NATRuleSet" | ||
| }, | ||
| "source": { | ||
| "as": { | ||
| "number": 1221, | ||
| "organization": { | ||
| "name": "Telstra Pty Ltd" | ||
| } | ||
| }, | ||
| "ip": "1.128.0.0", | ||
| "port": 1234 | ||
| }, | ||
| "tags": [ | ||
| "preserve_original_event" | ||
| ] | ||
| } | ||
| ] | ||
| } |
1 change: 1 addition & 0 deletions
1
...es/azure/data_stream/firewall_logs/_dev/test/pipeline/test-networkrule-structured-raw.log
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1 @@ | ||
| {"category":"AZFWNetworkRule","properties":{"Action":"Allow","ActionReason":"action reason","DestinationIp":"1.128.0.0","DestinationPort":1234,"Policy":"policy","Protocol":"TCP","Rule":"NetworkSecurityRule","RuleCollection":"NetworkRuleSet","RuleCollectionGroup":"NetworkRuleGroup","SourceIp":"1.128.0.0","SourcePort":1234},"resourceId":"/SUBSCRIPTIONS/23103928-B2CF-472A-8CDB-0146E2849129/RESOURCEGROUPS/TEST-FW-RG/PROVIDERS/MICROSOFT.NETWORK/AZUREFIREWALLS/TEST-FW01","time":"2024-03-20T23:39:59.8494370Z"} |
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It's good we're adding more info about the collection mode (azure diagnostics vs. resource specific)!
I would expand this section a little to convey more context, using the info mentioned in the doc from Azure you linked (and leaving all the other fine details to the Azure doc).
From https://learn.microsoft.com/en-gb/azure/azure-monitor/essentials/resource-logs#resource-specific:
So it's also clear that users can have the log categories from Azure Diagnostics or resource-specific, but not both.