Body Required

• _meta object

  Optional metadata about the ingest pipeline. May have any contents. This map is not automatically generated by Elasticsearch.

  Hide _meta attribute Show _meta attribute object

  • * object Additional properties
• description string

  Description of the ingest pipeline.

• on_failure array[object]

  Processors to run immediately after a processor failure. Each processor supports a processor-level on_failure value. If a processor without an on_failure value fails, Elasticsearch uses this pipeline-level parameter as a fallback. The processors in this parameter run sequentially in the order specified. Elasticsearch will not attempt to run the pipeline's remaining processors.

  Hide on_failure attributes Show on_failure attributes object

  • append object

    Appends one or more values to an existing array if the field already exists and it is an array. Converts a scalar to an array and appends one or more values to it if the field exists and it is a scalar. Creates an array containing the provided values if the field doesn’t exist. Accepts a single value or an array of values.

    Hide append attributes Show append attributes object

    • description string

      Description of the processor. Useful for describing the purpose of the processor or its configuration.

    • if object

      Conditionally execute the processor.

    • ignore_failure boolean

      Ignore failures for the processor.

    • on_failure array[object]

      Handle failures for the processor.
    • tag string

      Identifier for the processor. Useful for debugging and metrics.

    • field string Required

      The field to be appended to. Supports template snippets.

    • value object | array[object] Required

      The value to be appended. Supports template snippets.

      One of:

      object-1 object array-2 array[object]
    • allow_duplicates boolean

      If false, the processor does not append values already present in the field.

      Default value is true.

  • attachment object

    The attachment processor lets Elasticsearch extract file attachments in common formats (such as PPT, XLS, and PDF) by using the Apache text extraction library Tika.

    Hide attachment attributes Show attachment attributes object

    • description string

      Description of the processor. Useful for describing the purpose of the processor or its configuration.

    • if object

      Conditionally execute the processor.

    • ignore_failure boolean

      Ignore failures for the processor.

    • on_failure array[object]

      Handle failures for the processor.
    • tag string

      Identifier for the processor. Useful for debugging and metrics.

    • field string Required

      The field to get the base64 encoded field from.

    • ignore_missing boolean

      If true and field does not exist, the processor quietly exits without modifying the document.

      Default value is false.

    • indexed_chars number

      The number of chars being used for extraction to prevent huge fields. Use -1 for no limit.

      Default value is 100000.

    • indexed_chars_field string

      Field name from which you can overwrite the number of chars being used for extraction.

    • properties array[string]

      Array of properties to select to be stored. Can be content, title, name, author, keywords, date, content_type, content_length, language.

    • target_field string

      The field that will hold the attachment information.

    • remove_binary boolean

      If true, the binary field will be removed from the document

      Default value is false.

    • resource_name string

      Field containing the name of the resource to decode. If specified, the processor passes this resource name to the underlying Tika library to enable Resource Name Based Detection.

  • bytes object

    Converts a human readable byte value (for example 1kb) to its value in bytes (for example 1024). If the field is an array of strings, all members of the array will be converted. Supported human readable units are "b", "kb", "mb", "gb", "tb", "pb" case insensitive. An error will occur if the field is not a supported format or resultant value exceeds 2^63.

    Hide bytes attributes Show bytes attributes object

    • description string

      Description of the processor. Useful for describing the purpose of the processor or its configuration.

    • if object

      Conditionally execute the processor.

    • ignore_failure boolean

      Ignore failures for the processor.

    • on_failure array[object]

      Handle failures for the processor.
    • tag string

      Identifier for the processor. Useful for debugging and metrics.

    • field string Required

      The field to convert.

    • ignore_missing boolean

      If true and field does not exist or is null, the processor quietly exits without modifying the document.

      Default value is false.

    • target_field string

      The field to assign the converted value to. By default, the field is updated in-place.

  • circle object

    Converts circle definitions of shapes to regular polygons which approximate them.

    Hide circle attributes Show circle attributes object

    • description string

      Description of the processor. Useful for describing the purpose of the processor or its configuration.

    • if object

      Conditionally execute the processor.

    • ignore_failure boolean

      Ignore failures for the processor.

    • on_failure array[object]

      Handle failures for the processor.
    • tag string

      Identifier for the processor. Useful for debugging and metrics.

    • error_distance number Required

      The difference between the resulting inscribed distance from center to side and the circle’s radius (measured in meters for geo_shape, unit-less for shape).

    • field string Required

      The field to interpret as a circle. Either a string in WKT format or a map for GeoJSON.

    • ignore_missing boolean

      If true and field does not exist, the processor quietly exits without modifying the document.

      Default value is false.

    • shape_type string Required

      Which field mapping type is to be used when processing the circle: geo_shape or shape.

      Values are geo_shape or shape.

    • target_field string

      The field to assign the polygon shape to By default, the field is updated in-place.

  • community_id object

    Computes the Community ID for network flow data as defined in the Community ID Specification. You can use a community ID to correlate network events related to a single flow.

    Hide community_id attributes Show community_id attributes object

    • description string

      Description of the processor. Useful for describing the purpose of the processor or its configuration.

    • if object

      Conditionally execute the processor.

    • ignore_failure boolean

      Ignore failures for the processor.

    • on_failure array[object]

      Handle failures for the processor.
    • tag string

      Identifier for the processor. Useful for debugging and metrics.

    • source_ip string

      Field containing the source IP address.

    • source_port string

      Field containing the source port.

    • destination_ip string

      Field containing the destination IP address.

    • destination_port string

      Field containing the destination port.

    • iana_number string

      Field containing the IANA number.

    • icmp_type string

      Field containing the ICMP type.

    • icmp_code string

      Field containing the ICMP code.

    • transport string

      Field containing the transport protocol name or number. Used only when the iana_number field is not present. The following protocol names are currently supported: eigrp, gre, icmp, icmpv6, igmp, ipv6-icmp, ospf, pim, sctp, tcp, udp

    • target_field string

      Output field for the community ID.

    • seed number

      Seed for the community ID hash. Must be between 0 and 65535 (inclusive). The seed can prevent hash collisions between network domains, such as a staging and production network that use the same addressing scheme.

      Default value is 0.

    • ignore_missing boolean

      If true and any required fields are missing, the processor quietly exits without modifying the document.

      Default value is true.

  • convert object

    Converts a field in the currently ingested document to a different type, such as converting a string to an integer. If the field value is an array, all members will be converted.

    Hide convert attributes Show convert attributes object

    • description string

      Description of the processor. Useful for describing the purpose of the processor or its configuration.

    • if object

      Conditionally execute the processor.

    • ignore_failure boolean

      Ignore failures for the processor.

    • on_failure array[object]

      Handle failures for the processor.
    • tag string

      Identifier for the processor. Useful for debugging and metrics.

    • field string Required

      The field whose value is to be converted.

    • ignore_missing boolean

      If true and field does not exist or is null, the processor quietly exits without modifying the document.

      Default value is false.

    • target_field string

      The field to assign the converted value to. By default, the field is updated in-place.

    • type string Required

      The type to convert the existing value to.

      Values are integer, long, double, float, boolean, ip, string, or auto.

  • csv object

    Extracts fields from CSV line out of a single text field within a document. Any empty field in CSV will be skipped.

    Hide csv attributes Show csv attributes object

    • description string

      Description of the processor. Useful for describing the purpose of the processor or its configuration.

    • if object

      Conditionally execute the processor.

    • ignore_failure boolean

      Ignore failures for the processor.

    • on_failure array[object]

      Handle failures for the processor.
    • tag string

      Identifier for the processor. Useful for debugging and metrics.

    • empty_value object

      Value used to fill empty fields. Empty fields are skipped if this is not provided. An empty field is one with no value (2 consecutive separators) or empty quotes ("").

    • field string Required

      The field to extract data from.

    • ignore_missing boolean

      If true and field does not exist, the processor quietly exits without modifying the document.

    • quote string

      Quote used in CSV, has to be single character string.

      Default value is ".

    • separator string

      Separator used in CSV, has to be single character string.

      Default value is ,.

    • target_fields string | array[string] Required

      The array of fields to assign extracted values to.

    • trim boolean

      Trim whitespaces in unquoted fields.

  • date object

    Parses dates from fields, and then uses the date or timestamp as the timestamp for the document.

    Hide date attributes Show date attributes object

    • description string

      Description of the processor. Useful for describing the purpose of the processor or its configuration.

    • if object

      Conditionally execute the processor.

    • ignore_failure boolean

      Ignore failures for the processor.

    • on_failure array[object]

      Handle failures for the processor.
    • tag string

      Identifier for the processor. Useful for debugging and metrics.

    • field string Required

      The field to get the date from.

    • formats array[string] Required

      An array of the expected date formats. Can be a java time pattern or one of the following formats: ISO8601, UNIX, UNIX_MS, or TAI64N.

    • locale string

      The locale to use when parsing the date, relevant when parsing month names or week days. Supports template snippets.

      Default value is ENGLISH.

    • target_field string

      The field that will hold the parsed date.

    • timezone string

      The timezone to use when parsing the date. Supports template snippets.

      Default value is UTC.

    • output_format string

      The format to use when writing the date to target_field. Must be a valid java time pattern.

      Default value is yyyy-MM-dd'T'HH:mm:ss.SSSXXX.

  • date_index_name object

    The purpose of this processor is to point documents to the right time based index based on a date or timestamp field in a document by using the date math index name support.

    Hide date_index_name attributes Show date_index_name attributes object

    • description string

      Description of the processor. Useful for describing the purpose of the processor or its configuration.

    • if object

      Conditionally execute the processor.

    • ignore_failure boolean

      Ignore failures for the processor.

    • on_failure array[object]

      Handle failures for the processor.
    • tag string

      Identifier for the processor. Useful for debugging and metrics.

    • date_formats array[string]

      An array of the expected date formats for parsing dates / timestamps in the document being preprocessed. Can be a java time pattern or one of the following formats: ISO8601, UNIX, UNIX_MS, or TAI64N.

    • date_rounding string Required

      How to round the date when formatting the date into the index name. Valid values are: y (year), M (month), w (week), d (day), h (hour), m (minute) and s (second). Supports template snippets.

    • field string Required

      The field to get the date or timestamp from.

    • index_name_format string

      The format to be used when printing the parsed date into the index name. A valid java time pattern is expected here. Supports template snippets.

      Default value is yyyy-MM-dd.

    • index_name_prefix string

      A prefix of the index name to be prepended before the printed date. Supports template snippets.

    • locale string

      The locale to use when parsing the date from the document being preprocessed, relevant when parsing month names or week days.

      Default value is ENGLISH.

    • timezone string

      The timezone to use when parsing the date and when date math index supports resolves expressions into concrete index names.

      Default value is UTC.

  • dissect object

    Extracts structured fields out of a single text field by matching the text field against a delimiter-based pattern.

    Hide dissect attributes Show dissect attributes object

    • description string

      Description of the processor. Useful for describing the purpose of the processor or its configuration.

    • if object

      Conditionally execute the processor.

    • ignore_failure boolean

      Ignore failures for the processor.

    • on_failure array[object]

      Handle failures for the processor.
    • tag string

      Identifier for the processor. Useful for debugging and metrics.

    • append_separator string

      The character(s) that separate the appended fields.

      Default value is "".

    • field string Required

      The field to dissect.

    • ignore_missing boolean

      If true and field does not exist or is null, the processor quietly exits without modifying the document.

      Default value is false.

    • pattern string Required

      The pattern to apply to the field.

  • dot_expander object

    Expands a field with dots into an object field. This processor allows fields with dots in the name to be accessible by other processors in the pipeline. Otherwise these fields can’t be accessed by any processor.

    Hide dot_expander attributes Show dot_expander attributes object

    • description string

      Description of the processor. Useful for describing the purpose of the processor or its configuration.

    • if object

      Conditionally execute the processor.

    • ignore_failure boolean

      Ignore failures for the processor.

    • on_failure array[object]

      Handle failures for the processor.
    • tag string

      Identifier for the processor. Useful for debugging and metrics.

    • field string Required

      The field to expand into an object field. If set to *, all top-level fields will be expanded.

    • override boolean

      Controls the behavior when there is already an existing nested object that conflicts with the expanded field. When false, the processor will merge conflicts by combining the old and the new values into an array. When true, the value from the expanded field will overwrite the existing value.

      Default value is false.

    • path string

      The field that contains the field to expand. Only required if the field to expand is part another object field, because the field option can only understand leaf fields.

  • drop object

    Drops the document without raising any errors. This is useful to prevent the document from getting indexed based on some condition.

    Hide drop attributes Show drop attributes object

    • description string

      Description of the processor. Useful for describing the purpose of the processor or its configuration.

    • if object

      Conditionally execute the processor.

    • ignore_failure boolean

      Ignore failures for the processor.

    • on_failure array[object]

      Handle failures for the processor.
    • tag string

      Identifier for the processor. Useful for debugging and metrics.

  • enrich object

    The enrich processor can enrich documents with data from another index.

    Hide enrich attributes Show enrich attributes object

    • description string

      Description of the processor. Useful for describing the purpose of the processor or its configuration.

    • if object

      Conditionally execute the processor.

    • ignore_failure boolean

      Ignore failures for the processor.

    • on_failure array[object]

      Handle failures for the processor.
    • tag string

      Identifier for the processor. Useful for debugging and metrics.

    • field string Required

      The field in the input document that matches the policies match_field used to retrieve the enrichment data. Supports template snippets.

    • ignore_missing boolean

      If true and field does not exist, the processor quietly exits without modifying the document.

      Default value is false.

    • max_matches number

      The maximum number of matched documents to include under the configured target field. The target_field will be turned into a json array if max_matches is higher than 1, otherwise target_field will become a json object. In order to avoid documents getting too large, the maximum allowed value is 128.

      Default value is 1.

    • override boolean

      If processor will update fields with pre-existing non-null-valued field. When set to false, such fields will not be touched.

      Default value is true.

    • policy_name string Required

      The name of the enrich policy to use.

    • shape_relation string

      A spatial relation operator used to match the geoshape of incoming documents to documents in the enrich index. This option is only used for geo_match enrich policy types.

      Supported values include:

      • intersects: Return all documents whose geo_shape or geo_point field intersects the query geometry.
      • disjoint: Return all documents whose geo_shape or geo_point field has nothing in common with the query geometry.
      • within: Return all documents whose geo_shape or geo_point field is within the query geometry. Line geometries are not supported.
      • contains: Return all documents whose geo_shape or geo_point field contains the query geometry.

      Values are intersects, disjoint, within, or contains.

    • target_field string Required

      Field added to incoming documents to contain enrich data. This field contains both the match_field and enrich_fields specified in the enrich policy. Supports template snippets.

  • fail object

    Raises an exception. This is useful for when you expect a pipeline to fail and want to relay a specific message to the requester.

    Hide fail attributes Show fail attributes object

    • description string

      Description of the processor. Useful for describing the purpose of the processor or its configuration.

    • if object

      Conditionally execute the processor.

    • ignore_failure boolean

      Ignore failures for the processor.

    • on_failure array[object]

      Handle failures for the processor.
    • tag string

      Identifier for the processor. Useful for debugging and metrics.

    • message string Required

      The error message thrown by the processor. Supports template snippets.

  • fingerprint object

    Computes a hash of the document’s content. You can use this hash for content fingerprinting.

    Hide fingerprint attributes Show fingerprint attributes object

    • description string

      Description of the processor. Useful for describing the purpose of the processor or its configuration.

    • if object

      Conditionally execute the processor.

    • ignore_failure boolean

      Ignore failures for the processor.

    • on_failure array[object]

      Handle failures for the processor.
    • tag string

      Identifier for the processor. Useful for debugging and metrics.

    • fields string | array[string] Required

      Array of fields to include in the fingerprint. For objects, the processor hashes both the field key and value. For other fields, the processor hashes only the field value.

    • target_field string

      Output field for the fingerprint.

    • salt string

      Salt value for the hash function.

    • method string

      The hash method used to compute the fingerprint. Must be one of MD5, SHA-1, SHA-256, SHA-512, or MurmurHash3.

      Values are MD5, SHA-1, SHA-256, SHA-512, or MurmurHash3.

    • ignore_missing boolean

      If true, the processor ignores any missing fields. If all fields are missing, the processor silently exits without modifying the document.

      Default value is false.

  • foreach object

    Runs an ingest processor on each element of an array or object.

    Hide foreach attributes Show foreach attributes object

    • description string

      Description of the processor. Useful for describing the purpose of the processor or its configuration.

    • if object

      Conditionally execute the processor.

    • ignore_failure boolean

      Ignore failures for the processor.

    • on_failure array[object]

      Handle failures for the processor.
    • tag string

      Identifier for the processor. Useful for debugging and metrics.

    • field string Required

      Field containing array or object values.

    • ignore_missing boolean

      If true, the processor silently exits without changing the document if the field is null or missing.

      Default value is false.

    • processor object Required

      Ingest processor to run on each element.

  • ip_location object

    Currently an undocumented alias for GeoIP Processor.

    Hide ip_location attributes Show ip_location attributes object

    • description string

      Description of the processor. Useful for describing the purpose of the processor or its configuration.

    • if object

      Conditionally execute the processor.

    • ignore_failure boolean

      Ignore failures for the processor.

    • on_failure array[object]

      Handle failures for the processor.
    • tag string

      Identifier for the processor. Useful for debugging and metrics.

    • database_file string

      The database filename referring to a database the module ships with (GeoLite2-City.mmdb, GeoLite2-Country.mmdb, or GeoLite2-ASN.mmdb) or a custom database in the ingest-geoip config directory.

      Default value is GeoLite2-City.mmdb.

    • field string Required

      The field to get the ip address from for the geographical lookup.

    • first_only boolean

      If true, only the first found IP location data will be returned, even if the field contains an array.

      Default value is true.

    • ignore_missing boolean

      If true and field does not exist, the processor quietly exits without modifying the document.

      Default value is false.

    • properties array[string]

      Controls what properties are added to the target_field based on the IP location lookup.

    • target_field string

      The field that will hold the geographical information looked up from the MaxMind database.

    • download_database_on_pipeline_creation boolean

      If true (and if ingest.geoip.downloader.eager.download is false), the missing database is downloaded when the pipeline is created. Else, the download is triggered by when the pipeline is used as the default_pipeline or final_pipeline in an index.

  • geo_grid object

    Converts geo-grid definitions of grid tiles or cells to regular bounding boxes or polygons which describe their shape. This is useful if there is a need to interact with the tile shapes as spatially indexable fields.

    Hide geo_grid attributes Show geo_grid attributes object

    • description string

      Description of the processor. Useful for describing the purpose of the processor or its configuration.

    • if object

      Conditionally execute the processor.

    • ignore_failure boolean

      Ignore failures for the processor.

    • on_failure array[object]

      Handle failures for the processor.
    • tag string

      Identifier for the processor. Useful for debugging and metrics.

    • field string Required

      The field to interpret as a geo-tile.= The field format is determined by the tile_type.

    • tile_type string Required

      Three tile formats are understood: geohash, geotile and geohex.

      Values are geotile, geohex, or geohash.

    • target_field string

      The field to assign the polygon shape to, by default, the field is updated in-place.

    • parent_field string

      If specified and a parent tile exists, save that tile address to this field.

    • children_field string

      If specified and children tiles exist, save those tile addresses to this field as an array of strings.

    • non_children_field string

      If specified and intersecting non-child tiles exist, save their addresses to this field as an array of strings.

    • precision_field string

      If specified, save the tile precision (zoom) as an integer to this field.

    • ignore_missing boolean

      If true and field does not exist, the processor quietly exits without modifying the document.

      Default value is false.

    • target_format string

      Which format to save the generated polygon in.

      Values are geojson or wkt.

  • geoip object

    The geoip processor adds information about the geographical location of an IPv4 or IPv6 address.

    Hide geoip attributes Show geoip attributes object

    • description string

      Description of the processor. Useful for describing the purpose of the processor or its configuration.

    • if object

      Conditionally execute the processor.

    • ignore_failure boolean

      Ignore failures for the processor.

    • on_failure array[object]

      Handle failures for the processor.
    • tag string

      Identifier for the processor. Useful for debugging and metrics.

    • database_file string

      The database filename referring to a database the module ships with (GeoLite2-City.mmdb, GeoLite2-Country.mmdb, or GeoLite2-ASN.mmdb) or a custom database in the ingest-geoip config directory.

      Default value is GeoLite2-City.mmdb.

    • field string Required

      The field to get the ip address from for the geographical lookup.

    • first_only boolean

      If true, only the first found geoip data will be returned, even if the field contains an array.

      Default value is true.

    • ignore_missing boolean

      If true and field does not exist, the processor quietly exits without modifying the document.

      Default value is false.

    • properties array[string]

      Controls what properties are added to the target_field based on the geoip lookup.

    • target_field string

      The field that will hold the geographical information looked up from the MaxMind database.

    • download_database_on_pipeline_creation boolean

      If true (and if ingest.geoip.downloader.eager.download is false), the missing database is downloaded when the pipeline is created. Else, the download is triggered by when the pipeline is used as the default_pipeline or final_pipeline in an index.

  • grok object

    Extracts structured fields out of a single text field within a document. You choose which field to extract matched fields from, as well as the grok pattern you expect will match. A grok pattern is like a regular expression that supports aliased expressions that can be reused.

    Hide grok attributes Show grok attributes object

    • description string

      Description of the processor. Useful for describing the purpose of the processor or its configuration.

    • if object

      Conditionally execute the processor.

    • ignore_failure boolean

      Ignore failures for the processor.

    • on_failure array[object]

      Handle failures for the processor.
    • tag string

      Identifier for the processor. Useful for debugging and metrics.

    • ecs_compatibility string

      Must be disabled or v1. If v1, the processor uses patterns with Elastic Common Schema (ECS) field names.

      Default value is disabled.

    • field string Required

      The field to use for grok expression parsing.

    • ignore_missing boolean

      If true and field does not exist or is null, the processor quietly exits without modifying the document.

      Default value is false.

    • pattern_definitions object

      A map of pattern-name and pattern tuples defining custom patterns to be used by the current processor. Patterns matching existing names will override the pre-existing definition.

      Hide pattern_definitions attribute Show pattern_definitions attribute object

      • * string Additional properties
    • patterns array[string] Required

      An ordered list of grok expression to match and extract named captures with. Returns on the first expression in the list that matches.

    • trace_match boolean

      When true, _ingest._grok_match_index will be inserted into your matched document’s metadata with the index into the pattern found in patterns that matched.

      Default value is false.

  • gsub object

    Converts a string field by applying a regular expression and a replacement. If the field is an array of string, all members of the array will be converted. If any non-string values are encountered, the processor will throw an exception.

    Hide gsub attributes Show gsub attributes object

    • description string

      Description of the processor. Useful for describing the purpose of the processor or its configuration.

    • if object

      Conditionally execute the processor.

    • ignore_failure boolean

      Ignore failures for the processor.

    • on_failure array[object]

      Handle failures for the processor.
    • tag string

      Identifier for the processor. Useful for debugging and metrics.

    • field string Required

      The field to apply the replacement to.

    • ignore_missing boolean

      If true and field does not exist or is null, the processor quietly exits without modifying the document.

      Default value is false.

    • pattern string Required

      The pattern to be replaced.

    • replacement string Required

      The string to replace the matching patterns with.

    • target_field string

      The field to assign the converted value to By default, the field is updated in-place.

  • html_strip object

    Removes HTML tags from the field. If the field is an array of strings, HTML tags will be removed from all members of the array.

    Hide html_strip attributes Show html_strip attributes object

    • description string

      Description of the processor. Useful for describing the purpose of the processor or its configuration.

    • if object

      Conditionally execute the processor.

    • ignore_failure boolean

      Ignore failures for the processor.

    • on_failure array[object]

      Handle failures for the processor.
    • tag string

      Identifier for the processor. Useful for debugging and metrics.

    • field string Required

      The string-valued field to remove HTML tags from.

    • ignore_missing boolean

      If true and field does not exist or is null, the processor quietly exits without modifying the document,

      Default value is false.

    • target_field string

      The field to assign the converted value to By default, the field is updated in-place.

  • inference object

    Uses a pre-trained data frame analytics model or a model deployed for natural language processing tasks to infer against the data that is being ingested in the pipeline.

    Hide inference attributes Show inference attributes object

    • description string

      Description of the processor. Useful for describing the purpose of the processor or its configuration.

    • if object

      Conditionally execute the processor.

    • ignore_failure boolean

      Ignore failures for the processor.

    • on_failure array[object]

      Handle failures for the processor.
    • tag string

      Identifier for the processor. Useful for debugging and metrics.

    • model_id string Required

      The ID or alias for the trained model, or the ID of the deployment.

    • target_field string

      Field added to incoming documents to contain results objects.

    • field_map object

      Maps the document field names to the known field names of the model. This mapping takes precedence over any default mappings provided in the model configuration.

      Hide field_map attribute Show field_map attribute object

      • * object Additional properties
    • inference_config object

      Contains the inference type and its options.

    • input_output object | array[object]

      Input fields for inference and output (destination) fields for the inference results. This option is incompatible with the target_field and field_map options.

      One of:

      InputConfig object array-2 array[object]
    • ignore_missing boolean

      If true and any of the input fields defined in input_ouput are missing then those missing fields are quietly ignored, otherwise a missing field causes a failure. Only applies when using input_output configurations to explicitly list the input fields.

  • join object

    Joins each element of an array into a single string using a separator character between each element. Throws an error when the field is not an array.

    Hide join attributes Show join attributes object

    • description string

      Description of the processor. Useful for describing the purpose of the processor or its configuration.

    • if object

      Conditionally execute the processor.

    • ignore_failure boolean

      Ignore failures for the processor.

    • on_failure array[object]

      Handle failures for the processor.
    • tag string

      Identifier for the processor. Useful for debugging and metrics.

    • field string Required

      Field containing array values to join.

    • separator string Required

      The separator character.

    • target_field string

      The field to assign the joined value to. By default, the field is updated in-place.

  • json object

    Converts a JSON string into a structured JSON object.

    Hide json attributes Show json attributes object

    • description string

      Description of the processor. Useful for describing the purpose of the processor or its configuration.

    • if object

      Conditionally execute the processor.

    • ignore_failure boolean

      Ignore failures for the processor.

    • on_failure array[object]

      Handle failures for the processor.
    • tag string

      Identifier for the processor. Useful for debugging and metrics.

    • add_to_root boolean

      Flag that forces the parsed JSON to be added at the top level of the document. target_field must not be set when this option is chosen.

      Default value is false.

    • add_to_root_conflict_strategy string

      When set to replace, root fields that conflict with fields from the parsed JSON will be overridden. When set to merge, conflicting fields will be merged. Only applicable if add_to_root is set to true.

      Supported values include:

      • replace: Root fields that conflict with fields from the parsed JSON will be overridden.
      • merge: Conflicting fields will be merged.

      Values are replace or merge.

    • allow_duplicate_keys boolean

      When set to true, the JSON parser will not fail if the JSON contains duplicate keys. Instead, the last encountered value for any duplicate key wins.

      Default value is false.

    • field string Required

      The field to be parsed.

    • target_field string

      The field that the converted structured object will be written into. Any existing content in this field will be overwritten.

  • kv object

    This processor helps automatically parse messages (or specific event fields) which are of the foo=bar variety.

    Hide kv attributes Show kv attributes object

    • description string

      Description of the processor. Useful for describing the purpose of the processor or its configuration.

    • if object

      Conditionally execute the processor.

    • ignore_failure boolean

      Ignore failures for the processor.

    • on_failure array[object]

      Handle failures for the processor.
    • tag string

      Identifier for the processor. Useful for debugging and metrics.

    • exclude_keys array[string]

      List of keys to exclude from document.

    • field string Required

      The field to be parsed. Supports template snippets.

    • field_split string Required

      Regex pattern to use for splitting key-value pairs.

    • ignore_missing boolean

      If true and field does not exist or is null, the processor quietly exits without modifying the document.

      Default value is false.

    • include_keys array[string]

      List of keys to filter and insert into document. Defaults to including all keys.

    • prefix string

      Prefix to be added to extracted keys.

      Default value is null.

    • strip_brackets boolean

      If true. strip brackets (), <>, [] as well as quotes ' and " from extracted values.

      Default value is false.

    • target_field string

      The field to insert the extracted keys into. Defaults to the root of the document. Supports template snippets.

    • trim_key string

      String of characters to trim from extracted keys.

    • trim_value string

      String of characters to trim from extracted values.

    • value_split string Required

      Regex pattern to use for splitting the key from the value within a key-value pair.

  • lowercase object

    Converts a string to its lowercase equivalent. If the field is an array of strings, all members of the array will be converted.

    Hide lowercase attributes Show lowercase attributes object

    • description string

      Description of the processor. Useful for describing the purpose of the processor or its configuration.

    • if object

      Conditionally execute the processor.

    • ignore_failure boolean

      Ignore failures for the processor.

    • on_failure array[object]

      Handle failures for the processor.
    • tag string

      Identifier for the processor. Useful for debugging and metrics.

    • field string Required

      The field to make lowercase.

    • ignore_missing boolean

      If true and field does not exist or is null, the processor quietly exits without modifying the document.

      Default value is false.

    • target_field string

      The field to assign the converted value to. By default, the field is updated in-place.

  • network_direction object

    Calculates the network direction given a source IP address, destination IP address, and a list of internal networks.

    Hide network_direction attributes Show network_direction attributes object

    • description string

      Description of the processor. Useful for describing the purpose of the processor or its configuration.

    • if object

      Conditionally execute the processor.

    • ignore_failure boolean

      Ignore failures for the processor.

    • on_failure array[object]

      Handle failures for the processor.
    • tag string

      Identifier for the processor. Useful for debugging and metrics.

    • source_ip string

      Field containing the source IP address.

    • destination_ip string

      Field containing the destination IP address.

    • target_field string

      Output field for the network direction.

    • internal_networks array[string]

      List of internal networks. Supports IPv4 and IPv6 addresses and ranges in CIDR notation. Also supports the named ranges listed below. These may be constructed with template snippets. Must specify only one of internal_networks or internal_networks_field.

    • internal_networks_field string

      A field on the given document to read the internal_networks configuration from.

    • ignore_missing boolean

      If true and any required fields are missing, the processor quietly exits without modifying the document.

      Default value is true.

  • pipeline object

    Executes another pipeline.

    Hide pipeline attributes Show pipeline attributes object

    • description string

      Description of the processor. Useful for describing the purpose of the processor or its configuration.

    • if object

      Conditionally execute the processor.

    • ignore_failure boolean

      Ignore failures for the processor.

    • on_failure array[object]

      Handle failures for the processor.
    • tag string

      Identifier for the processor. Useful for debugging and metrics.

    • name string Required

      The name of the pipeline to execute. Supports template snippets.

    • ignore_missing_pipeline boolean

      Whether to ignore missing pipelines instead of failing.

      Default value is false.

  • redact object

    The Redact processor uses the Grok rules engine to obscure text in the input document matching the given Grok patterns. The processor can be used to obscure Personal Identifying Information (PII) by configuring it to detect known patterns such as email or IP addresses. Text that matches a Grok pattern is replaced with a configurable string such as <EMAIL> where an email address is matched or simply replace all matches with the text <REDACTED> if preferred.

    Hide redact attributes Show redact attributes object

    • description string

      Description of the processor. Useful for describing the purpose of the processor or its configuration.

    • if object

      Conditionally execute the processor.

    • ignore_failure boolean

      Ignore failures for the processor.

    • on_failure array[object]

      Handle failures for the processor.
    • tag string

      Identifier for the processor. Useful for debugging and metrics.

    • field string Required

      The field to be redacted

    • patterns array[string] Required

      A list of grok expressions to match and redact named captures with

    • pattern_definitions object
      Hide pattern_definitions attribute Show pattern_definitions attribute object

      • * string Additional properties
    • prefix string

      Start a redacted section with this token

      Default value is <.

    • suffix string

      End a redacted section with this token

      Default value is >.

    • ignore_missing boolean

      If true and field does not exist or is null, the processor quietly exits without modifying the document.

      Default value is false.

    • skip_if_unlicensed boolean

      If true and the current license does not support running redact processors, then the processor quietly exits without modifying the document

      Default value is false.

    • trace_redact boolean Generally available; Added in 8.16.0

      If true then ingest metadata _ingest._redact._is_redacted is set to true if the document has been redacted

      Default value is false.

  • registered_domain object

    Extracts the registered domain (also known as the effective top-level domain or eTLD), sub-domain, and top-level domain from a fully qualified domain name (FQDN). Uses the registered domains defined in the Mozilla Public Suffix List.

    Hide registered_domain attributes Show registered_domain attributes object

    • description string

      Description of the processor. Useful for describing the purpose of the processor or its configuration.

    • if object

      Conditionally execute the processor.

    • ignore_failure boolean

      Ignore failures for the processor.

    • on_failure array[object]

      Handle failures for the processor.
    • tag string

      Identifier for the processor. Useful for debugging and metrics.

    • field string Required

      Field containing the source FQDN.

    • target_field string

      Object field containing extracted domain components. If an empty string, the processor adds components to the document’s root.

    • ignore_missing boolean

      If true and any required fields are missing, the processor quietly exits without modifying the document.

      Default value is true.

  • remove object

    Removes existing fields. If one field doesn’t exist, an exception will be thrown.

    Hide remove attributes Show remove attributes object

    • description string

      Description of the processor. Useful for describing the purpose of the processor or its configuration.

    • if object

      Conditionally execute the processor.

    • ignore_failure boolean

      Ignore failures for the processor.

    • on_failure array[object]

      Handle failures for the processor.
    • tag string

      Identifier for the processor. Useful for debugging and metrics.

    • field string | array[string] Required

      Fields to be removed. Supports template snippets.

    • keep string | array[string]

      Fields to be kept. When set, all fields other than those specified are removed.

    • ignore_missing boolean

      If true and field does not exist or is null, the processor quietly exits without modifying the document.

      Default value is false.

  • rename object

    Renames an existing field. If the field doesn’t exist or the new name is already used, an exception will be thrown.

    Hide rename attributes Show rename attributes object

    • description string

      Description of the processor. Useful for describing the purpose of the processor or its configuration.

    • if object

      Conditionally execute the processor.

    • ignore_failure boolean

      Ignore failures for the processor.

    • on_failure array[object]

      Handle failures for the processor.
    • tag string

      Identifier for the processor. Useful for debugging and metrics.

    • field string Required

      The field to be renamed. Supports template snippets.

    • ignore_missing boolean

      If true and field does not exist, the processor quietly exits without modifying the document.

      Default value is false.

    • target_field string Required

      The new name of the field. Supports template snippets.

  • reroute object

    Routes a document to another target index or data stream. When setting the destination option, the target is explicitly specified and the dataset and namespace options can’t be set. When the destination option is not set, this processor is in a data stream mode. Note that in this mode, the reroute processor can only be used on data streams that follow the data stream naming scheme.

    Hide reroute attributes Show reroute attributes object

    • description string

      Description of the processor. Useful for describing the purpose of the processor or its configuration.

    • if object

      Conditionally execute the processor.

    • ignore_failure boolean

      Ignore failures for the processor.

    • on_failure array[object]

      Handle failures for the processor.
    • tag string

      Identifier for the processor. Useful for debugging and metrics.

    • destination string

      A static value for the target. Can’t be set when the dataset or namespace option is set.

    • dataset string | array[string]

      Field references or a static value for the dataset part of the data stream name. In addition to the criteria for index names, cannot contain - and must be no longer than 100 characters. Example values are nginx.access and nginx.error.

      Supports field references with a mustache-like syntax (denoted as {{double}} or {{{triple}}} curly braces). When resolving field references, the processor replaces invalid characters with _. Uses the part of the index name as a fallback if all field references resolve to a null, missing, or non-string value.

      default {{data_stream.dataset}}

      One of:

      string-1 string array-2 array[string]

    • namespace string | array[string]

      Field references or a static value for the namespace part of the data stream name. See the criteria for index names for allowed characters. Must be no longer than 100 characters.

      Supports field references with a mustache-like syntax (denoted as {{double}} or {{{triple}}} curly braces). When resolving field references, the processor replaces invalid characters with _. Uses the part of the index name as a fallback if all field references resolve to a null, missing, or non-string value.

      default {{data_stream.namespace}}

      One of:

      string-1 string array-2 array[string]
  • script object

    Runs an inline or stored script on incoming documents. The script runs in the ingest context.

    Hide script attributes Show script attributes object

    • description string

      Description of the processor. Useful for describing the purpose of the processor or its configuration.

    • if object

      Conditionally execute the processor.

    • ignore_failure boolean

      Ignore failures for the processor.

    • on_failure array[object]

      Handle failures for the processor.
    • tag string

      Identifier for the processor. Useful for debugging and metrics.

    • id string

      ID of a stored script. If no source is specified, this parameter is required.

    • lang
    • params object

      Object containing parameters for the script.

      Hide params attribute Show params attribute object

      • * object Additional properties
    • source
  • set object

    Adds a field with the specified value. If the field already exists, its value will be replaced with the provided one.

    Hide set attributes Show set attributes object

    • description string

      Description of the processor. Useful for describing the purpose of the processor or its configuration.

    • if object

      Conditionally execute the processor.

    • ignore_failure boolean

      Ignore failures for the processor.

    • on_failure array[object]

      Handle failures for the processor.
    • tag string

      Identifier for the processor. Useful for debugging and metrics.

    • copy_from string

      The origin field which will be copied to field, cannot set value simultaneously. Supported data types are boolean, number, array, object, string, date, etc.

    • field string Required

      The field to insert, upsert, or update. Supports template snippets.

    • ignore_empty_value boolean

      If true and value is a template snippet that evaluates to null or the empty string, the processor quietly exits without modifying the document.

      Default value is false.

    • media_type string

      The media type for encoding value. Applies only when value is a template snippet. Must be one of application/json, text/plain, or application/x-www-form-urlencoded.

    • override boolean

      If true processor will update fields with pre-existing non-null-valued field. When set to false, such fields will not be touched.

      Default value is true.

    • value object

      The value to be set for the field. Supports template snippets. May specify only one of value or copy_from.

  • set_security_user object

    Sets user-related details (such as username, roles, email, full_name, metadata, api_key, realm and authentication_type) from the current authenticated user to the current document by pre-processing the ingest.

    Hide set_security_user attributes Show set_security_user attributes object

    • description string

      Description of the processor. Useful for describing the purpose of the processor or its configuration.

    • if object

      Conditionally execute the processor.

    • ignore_failure boolean

      Ignore failures for the processor.

    • on_failure array[object]

      Handle failures for the processor.
    • tag string

      Identifier for the processor. Useful for debugging and metrics.

    • field string Required

      The field to store the user information into.

    • properties array[string]

      Controls what user related properties are added to the field.

  • sort object

    Sorts the elements of an array ascending or descending. Homogeneous arrays of numbers will be sorted numerically, while arrays of strings or heterogeneous arrays of strings + numbers will be sorted lexicographically. Throws an error when the field is not an array.

    Hide sort attributes Show sort attributes object

    • description string

      Description of the processor. Useful for describing the purpose of the processor or its configuration.

    • if object

      Conditionally execute the processor.

    • ignore_failure boolean

      Ignore failures for the processor.

    • on_failure array[object]

      Handle failures for the processor.
    • tag string

      Identifier for the processor. Useful for debugging and metrics.

    • field string Required

      The field to be sorted.

    • order string

      The sort order to use. Accepts "asc" or "desc".

      Supported values include:

      • asc: Ascending (smallest to largest)
      • desc: Descending (largest to smallest)

      Values are asc or desc.

    • target_field string

      The field to assign the sorted value to. By default, the field is updated in-place.

  • split object

    Splits a field into an array using a separator character. Only works on string fields.

    Hide split attributes Show split attributes object

    • description string

      Description of the processor. Useful for describing the purpose of the processor or its configuration.

    • if object

      Conditionally execute the processor.

    • ignore_failure boolean

      Ignore failures for the processor.

    • on_failure array[object]

      Handle failures for the processor.
    • tag string

      Identifier for the processor. Useful for debugging and metrics.

    • field string Required

      The field to split.

    • ignore_missing boolean

      If true and field does not exist, the processor quietly exits without modifying the document.

      Default value is false.

    • preserve_trailing boolean

      Preserves empty trailing fields, if any.

      Default value is false.

    • separator string Required

      A regex which matches the separator, for example, , or \s+.

    • target_field string

      The field to assign the split value to. By default, the field is updated in-place.

  • terminate object

    Terminates the current ingest pipeline, causing no further processors to be run. This will normally be executed conditionally, using the if option.

    Hide terminate attributes Show terminate attributes object

    • description string

      Description of the processor. Useful for describing the purpose of the processor or its configuration.

    • if object

      Conditionally execute the processor.

    • ignore_failure boolean

      Ignore failures for the processor.

    • on_failure array[object]

      Handle failures for the processor.
    • tag string

      Identifier for the processor. Useful for debugging and metrics.

  • trim object

    Trims whitespace from a field. If the field is an array of strings, all members of the array will be trimmed. This only works on leading and trailing whitespace.

    Hide trim attributes Show trim attributes object

    • description string

      Description of the processor. Useful for describing the purpose of the processor or its configuration.

    • if object

      Conditionally execute the processor.

    • ignore_failure boolean

      Ignore failures for the processor.

    • on_failure array[object]

      Handle failures for the processor.
    • tag string

      Identifier for the processor. Useful for debugging and metrics.

    • field string Required

      The string-valued field to trim whitespace from.

    • ignore_missing boolean

      If true and field does not exist, the processor quietly exits without modifying the document.

      Default value is false.

    • target_field string

      The field to assign the trimmed value to. By default, the field is updated in-place.

  • uppercase object

    Converts a string to its uppercase equivalent. If the field is an array of strings, all members of the array will be converted.

    Hide uppercase attributes Show uppercase attributes object

    • description string

      Description of the processor. Useful for describing the purpose of the processor or its configuration.

    • if object

      Conditionally execute the processor.

    • ignore_failure boolean

      Ignore failures for the processor.

    • on_failure array[object]

      Handle failures for the processor.
    • tag string

      Identifier for the processor. Useful for debugging and metrics.

    • field string Required

      The field to make uppercase.

    • ignore_missing boolean

      If true and field does not exist or is null, the processor quietly exits without modifying the document.

      Default value is false.

    • target_field string

      The field to assign the converted value to. By default, the field is updated in-place.

  • urldecode object

    URL-decodes a string. If the field is an array of strings, all members of the array will be decoded.

    Hide urldecode attributes Show urldecode attributes object

    • description string

      Description of the processor. Useful for describing the purpose of the processor or its configuration.

    • if object

      Conditionally execute the processor.

    • ignore_failure boolean

      Ignore failures for the processor.

    • on_failure array[object]

      Handle failures for the processor.
    • tag string

      Identifier for the processor. Useful for debugging and metrics.

    • field string Required

      The field to decode.

    • ignore_missing boolean

      If true and field does not exist or is null, the processor quietly exits without modifying the document.

      Default value is false.

    • target_field string

      The field to assign the converted value to. By default, the field is updated in-place.

  • uri_parts object

    Parses a Uniform Resource Identifier (URI) string and extracts its components as an object. This URI object includes properties for the URI’s domain, path, fragment, port, query, scheme, user info, username, and password.

    Hide uri_parts attributes Show uri_parts attributes object

    • description string

      Description of the processor. Useful for describing the purpose of the processor or its configuration.

    • if object

      Conditionally execute the processor.

    • ignore_failure boolean

      Ignore failures for the processor.

    • on_failure array[object]

      Handle failures for the processor.
    • tag string

      Identifier for the processor. Useful for debugging and metrics.

    • field string Required

      Field containing the URI string.

    • ignore_missing boolean

      If true and field does not exist, the processor quietly exits without modifying the document.

      Default value is false.

    • keep_original boolean

      If true, the processor copies the unparsed URI to <target_field>.original.

      Default value is true.

    • remove_if_successful boolean

      If true, the processor removes the field after parsing the URI string. If parsing fails, the processor does not remove the field.

      Default value is false.

    • target_field string

      Output field for the URI object.

  • user_agent object

    The user_agent processor extracts details from the user agent string a browser sends with its web requests. This processor adds this information by default under the user_agent field.

    Hide user_agent attributes Show user_agent attributes object

    • description string

      Description of the processor. Useful for describing the purpose of the processor or its configuration.

    • if object

      Conditionally execute the processor.

    • ignore_failure boolean

      Ignore failures for the processor.

    • on_failure array[object]

      Handle failures for the processor.
    • tag string

      Identifier for the processor. Useful for debugging and metrics.

    • field string Required

      The field containing the user agent string.

    • ignore_missing boolean

      If true and field does not exist, the processor quietly exits without modifying the document.

      Default value is false.

    • regex_file string

      The name of the file in the config/ingest-user-agent directory containing the regular expressions for parsing the user agent string. Both the directory and the file have to be created before starting Elasticsearch. If not specified, ingest-user-agent will use the regexes.yaml from uap-core it ships with.

    • target_field string

      The field that will be filled with the user agent details.

    • properties array[string]

      Controls what properties are added to target_field.

      Values are name, os, device, original, or version. Default value is ["name", "major", "minor", "patch", "build", "os", "os_name", "os_major", "os_minor", "device"].

    • extract_device_type boolean Beta; Added in 8.9.0

      Extracts device type from the user agent string on a best-effort basis.

      Default value is false.

• processors array[object]

  Processors used to perform transformations on documents before indexing. Processors run sequentially in the order specified.

  Hide processors attributes Show processors attributes object

  • append object

    Appends one or more values to an existing array if the field already exists and it is an array. Converts a scalar to an array and appends one or more values to it if the field exists and it is a scalar. Creates an array containing the provided values if the field doesn’t exist. Accepts a single value or an array of values.

    Hide append attributes Show append attributes object

    • description string

      Description of the processor. Useful for describing the purpose of the processor or its configuration.

    • if object

      Conditionally execute the processor.

    • ignore_failure boolean

      Ignore failures for the processor.

    • on_failure array[object]

      Handle failures for the processor.
    • tag string

      Identifier for the processor. Useful for debugging and metrics.

    • field string Required

      The field to be appended to. Supports template snippets.

    • value object | array[object] Required

      The value to be appended. Supports template snippets.

      One of:

      object-1 object array-2 array[object]
    • allow_duplicates boolean

      If false, the processor does not append values already present in the field.

      Default value is true.

  • attachment object

    The attachment processor lets Elasticsearch extract file attachments in common formats (such as PPT, XLS, and PDF) by using the Apache text extraction library Tika.

    Hide attachment attributes Show attachment attributes object

    • description string

      Description of the processor. Useful for describing the purpose of the processor or its configuration.

    • if object

      Conditionally execute the processor.

    • ignore_failure boolean

      Ignore failures for the processor.

    • on_failure array[object]

      Handle failures for the processor.
    • tag string

      Identifier for the processor. Useful for debugging and metrics.

    • field string Required

      The field to get the base64 encoded field from.

    • ignore_missing boolean

      If true and field does not exist, the processor quietly exits without modifying the document.

      Default value is false.

    • indexed_chars number

      The number of chars being used for extraction to prevent huge fields. Use -1 for no limit.

      Default value is 100000.

    • indexed_chars_field string

      Field name from which you can overwrite the number of chars being used for extraction.

    • properties array[string]

      Array of properties to select to be stored. Can be content, title, name, author, keywords, date, content_type, content_length, language.

    • target_field string

      The field that will hold the attachment information.

    • remove_binary boolean

      If true, the binary field will be removed from the document

      Default value is false.

    • resource_name string

      Field containing the name of the resource to decode. If specified, the processor passes this resource name to the underlying Tika library to enable Resource Name Based Detection.

  • bytes object

    Converts a human readable byte value (for example 1kb) to its value in bytes (for example 1024). If the field is an array of strings, all members of the array will be converted. Supported human readable units are "b", "kb", "mb", "gb", "tb", "pb" case insensitive. An error will occur if the field is not a supported format or resultant value exceeds 2^63.

    Hide bytes attributes Show bytes attributes object

    • description string

      Description of the processor. Useful for describing the purpose of the processor or its configuration.

    • if object

      Conditionally execute the processor.

    • ignore_failure boolean

      Ignore failures for the processor.

    • on_failure array[object]

      Handle failures for the processor.
    • tag string

      Identifier for the processor. Useful for debugging and metrics.

    • field string Required

      The field to convert.

    • ignore_missing boolean

      If true and field does not exist or is null, the processor quietly exits without modifying the document.

      Default value is false.

    • target_field string

      The field to assign the converted value to. By default, the field is updated in-place.

  • circle object

    Converts circle definitions of shapes to regular polygons which approximate them.

    Hide circle attributes Show circle attributes object

    • description string

      Description of the processor. Useful for describing the purpose of the processor or its configuration.

    • if object

      Conditionally execute the processor.

    • ignore_failure boolean

      Ignore failures for the processor.

    • on_failure array[object]

      Handle failures for the processor.
    • tag string

      Identifier for the processor. Useful for debugging and metrics.

    • error_distance number Required

      The difference between the resulting inscribed distance from center to side and the circle’s radius (measured in meters for geo_shape, unit-less for shape).

    • field string Required

      The field to interpret as a circle. Either a string in WKT format or a map for GeoJSON.

    • ignore_missing boolean

      If true and field does not exist, the processor quietly exits without modifying the document.

      Default value is false.

    • shape_type string Required

      Which field mapping type is to be used when processing the circle: geo_shape or shape.

      Values are geo_shape or shape.

    • target_field string

      The field to assign the polygon shape to By default, the field is updated in-place.

  • community_id object

    Computes the Community ID for network flow data as defined in the Community ID Specification. You can use a community ID to correlate network events related to a single flow.

    Hide community_id attributes Show community_id attributes object

    • description string

      Description of the processor. Useful for describing the purpose of the processor or its configuration.

    • if object

      Conditionally execute the processor.

    • ignore_failure boolean

      Ignore failures for the processor.

    • on_failure array[object]

      Handle failures for the processor.
    • tag string

      Identifier for the processor. Useful for debugging and metrics.

    • source_ip string

      Field containing the source IP address.

    • source_port string

      Field containing the source port.

    • destination_ip string

      Field containing the destination IP address.

    • destination_port string

      Field containing the destination port.

    • iana_number string

      Field containing the IANA number.

    • icmp_type string

      Field containing the ICMP type.

    • icmp_code string

      Field containing the ICMP code.

    • transport string

      Field containing the transport protocol name or number. Used only when the iana_number field is not present. The following protocol names are currently supported: eigrp, gre, icmp, icmpv6, igmp, ipv6-icmp, ospf, pim, sctp, tcp, udp

    • target_field string

      Output field for the community ID.

    • seed number

      Seed for the community ID hash. Must be between 0 and 65535 (inclusive). The seed can prevent hash collisions between network domains, such as a staging and production network that use the same addressing scheme.

      Default value is 0.

    • ignore_missing boolean

      If true and any required fields are missing, the processor quietly exits without modifying the document.

      Default value is true.

  • convert object

    Converts a field in the currently ingested document to a different type, such as converting a string to an integer. If the field value is an array, all members will be converted.

    Hide convert attributes Show convert attributes object

    • description string

      Description of the processor. Useful for describing the purpose of the processor or its configuration.

    • if object

      Conditionally execute the processor.

    • ignore_failure boolean

      Ignore failures for the processor.

    • on_failure array[object]

      Handle failures for the processor.
    • tag string

      Identifier for the processor. Useful for debugging and metrics.

    • field string Required

      The field whose value is to be converted.

    • ignore_missing boolean

      If true and field does not exist or is null, the processor quietly exits without modifying the document.

      Default value is false.

    • target_field string

      The field to assign the converted value to. By default, the field is updated in-place.

    • type string Required

      The type to convert the existing value to.

      Values are integer, long, double, float, boolean, ip, string, or auto.

  • csv object

    Extracts fields from CSV line out of a single text field within a document. Any empty field in CSV will be skipped.

    Hide csv attributes Show csv attributes object

    • description string

      Description of the processor. Useful for describing the purpose of the processor or its configuration.

    • if object

      Conditionally execute the processor.

    • ignore_failure boolean

      Ignore failures for the processor.

    • on_failure array[object]

      Handle failures for the processor.
    • tag string

      Identifier for the processor. Useful for debugging and metrics.

    • empty_value object

      Value used to fill empty fields. Empty fields are skipped if this is not provided. An empty field is one with no value (2 consecutive separators) or empty quotes ("").

    • field string Required

      The field to extract data from.

    • ignore_missing boolean

      If true and field does not exist, the processor quietly exits without modifying the document.

    • quote string

      Quote used in CSV, has to be single character string.

      Default value is ".

    • separator string

      Separator used in CSV, has to be single character string.

      Default value is ,.

    • target_fields string | array[string] Required

      The array of fields to assign extracted values to.

    • trim boolean

      Trim whitespaces in unquoted fields.

  • date object

    Parses dates from fields, and then uses the date or timestamp as the timestamp for the document.

    Hide date attributes Show date attributes object

    • description string

      Description of the processor. Useful for describing the purpose of the processor or its configuration.

    • if object

      Conditionally execute the processor.

    • ignore_failure boolean

      Ignore failures for the processor.

    • on_failure array[object]

      Handle failures for the processor.
    • tag string

      Identifier for the processor. Useful for debugging and metrics.

    • field string Required

      The field to get the date from.

    • formats array[string] Required

      An array of the expected date formats. Can be a java time pattern or one of the following formats: ISO8601, UNIX, UNIX_MS, or TAI64N.

    • locale string

      The locale to use when parsing the date, relevant when parsing month names or week days. Supports template snippets.

      Default value is ENGLISH.

    • target_field string

      The field that will hold the parsed date.

    • timezone string

      The timezone to use when parsing the date. Supports template snippets.

      Default value is UTC.

    • output_format string

      The format to use when writing the date to target_field. Must be a valid java time pattern.

      Default value is yyyy-MM-dd'T'HH:mm:ss.SSSXXX.

  • date_index_name object

    The purpose of this processor is to point documents to the right time based index based on a date or timestamp field in a document by using the date math index name support.

    Hide date_index_name attributes Show date_index_name attributes object

    • description string

      Description of the processor. Useful for describing the purpose of the processor or its configuration.

    • if object

      Conditionally execute the processor.

    • ignore_failure boolean

      Ignore failures for the processor.

    • on_failure array[object]

      Handle failures for the processor.
    • tag string

      Identifier for the processor. Useful for debugging and metrics.

    • date_formats array[string]

      An array of the expected date formats for parsing dates / timestamps in the document being preprocessed. Can be a java time pattern or one of the following formats: ISO8601, UNIX, UNIX_MS, or TAI64N.

    • date_rounding string Required

      How to round the date when formatting the date into the index name. Valid values are: y (year), M (month), w (week), d (day), h (hour), m (minute) and s (second). Supports template snippets.

    • field string Required

      The field to get the date or timestamp from.

    • index_name_format string

      The format to be used when printing the parsed date into the index name. A valid java time pattern is expected here. Supports template snippets.

      Default value is yyyy-MM-dd.

    • index_name_prefix string

      A prefix of the index name to be prepended before the printed date. Supports template snippets.

    • locale string

      The locale to use when parsing the date from the document being preprocessed, relevant when parsing month names or week days.

      Default value is ENGLISH.

    • timezone string

      The timezone to use when parsing the date and when date math index supports resolves expressions into concrete index names.

      Default value is UTC.

  • dissect object

    Extracts structured fields out of a single text field by matching the text field against a delimiter-based pattern.

    Hide dissect attributes Show dissect attributes object

    • description string

      Description of the processor. Useful for describing the purpose of the processor or its configuration.

    • if object

      Conditionally execute the processor.

    • ignore_failure boolean

      Ignore failures for the processor.

    • on_failure array[object]

      Handle failures for the processor.
    • tag string

      Identifier for the processor. Useful for debugging and metrics.

    • append_separator string

      The character(s) that separate the appended fields.

      Default value is "".

    • field string Required

      The field to dissect.

    • ignore_missing boolean

      If true and field does not exist or is null, the processor quietly exits without modifying the document.

      Default value is false.

    • pattern string Required

      The pattern to apply to the field.

  • dot_expander object

    Expands a field with dots into an object field. This processor allows fields with dots in the name to be accessible by other processors in the pipeline. Otherwise these fields can’t be accessed by any processor.

    Hide dot_expander attributes Show dot_expander attributes object

    • description string

      Description of the processor. Useful for describing the purpose of the processor or its configuration.

    • if object

      Conditionally execute the processor.

    • ignore_failure boolean

      Ignore failures for the processor.

    • on_failure array[object]

      Handle failures for the processor.
    • tag string

      Identifier for the processor. Useful for debugging and metrics.

    • field string Required

      The field to expand into an object field. If set to *, all top-level fields will be expanded.

    • override boolean

      Controls the behavior when there is already an existing nested object that conflicts with the expanded field. When false, the processor will merge conflicts by combining the old and the new values into an array. When true, the value from the expanded field will overwrite the existing value.

      Default value is false.

    • path string

      The field that contains the field to expand. Only required if the field to expand is part another object field, because the field option can only understand leaf fields.

  • drop object

    Drops the document without raising any errors. This is useful to prevent the document from getting indexed based on some condition.

    Hide drop attributes Show drop attributes object

    • description string

      Description of the processor. Useful for describing the purpose of the processor or its configuration.

    • if object

      Conditionally execute the processor.

    • ignore_failure boolean

      Ignore failures for the processor.

    • on_failure array[object]

      Handle failures for the processor.
    • tag string

      Identifier for the processor. Useful for debugging and metrics.

  • enrich object

    The enrich processor can enrich documents with data from another index.

    Hide enrich attributes Show enrich attributes object

    • description string

      Description of the processor. Useful for describing the purpose of the processor or its configuration.

    • if object

      Conditionally execute the processor.

    • ignore_failure boolean

      Ignore failures for the processor.

    • on_failure array[object]

      Handle failures for the processor.
    • tag string

      Identifier for the processor. Useful for debugging and metrics.

    • field string Required

      The field in the input document that matches the policies match_field used to retrieve the enrichment data. Supports template snippets.

    • ignore_missing boolean

      If true and field does not exist, the processor quietly exits without modifying the document.

      Default value is false.

    • max_matches number

      The maximum number of matched documents to include under the configured target field. The target_field will be turned into a json array if max_matches is higher than 1, otherwise target_field will become a json object. In order to avoid documents getting too large, the maximum allowed value is 128.

      Default value is 1.

    • override boolean

      If processor will update fields with pre-existing non-null-valued field. When set to false, such fields will not be touched.

      Default value is true.

    • policy_name string Required

      The name of the enrich policy to use.

    • shape_relation string

      A spatial relation operator used to match the geoshape of incoming documents to documents in the enrich index. This option is only used for geo_match enrich policy types.

      Supported values include:

      • intersects: Return all documents whose geo_shape or geo_point field intersects the query geometry.
      • disjoint: Return all documents whose geo_shape or geo_point field has nothing in common with the query geometry.
      • within: Return all documents whose geo_shape or geo_point field is within the query geometry. Line geometries are not supported.
      • contains: Return all documents whose geo_shape or geo_point field contains the query geometry.

      Values are intersects, disjoint, within, or contains.

    • target_field string Required

      Field added to incoming documents to contain enrich data. This field contains both the match_field and enrich_fields specified in the enrich policy. Supports template snippets.

  • fail object

    Raises an exception. This is useful for when you expect a pipeline to fail and want to relay a specific message to the requester.

    Hide fail attributes Show fail attributes object

    • description string

      Description of the processor. Useful for describing the purpose of the processor or its configuration.

    • if object

      Conditionally execute the processor.

    • ignore_failure boolean

      Ignore failures for the processor.

    • on_failure array[object]

      Handle failures for the processor.
    • tag string

      Identifier for the processor. Useful for debugging and metrics.

    • message string Required

      The error message thrown by the processor. Supports template snippets.

  • fingerprint object

    Computes a hash of the document’s content. You can use this hash for content fingerprinting.

    Hide fingerprint attributes Show fingerprint attributes object

    • description string

      Description of the processor. Useful for describing the purpose of the processor or its configuration.

    • if object

      Conditionally execute the processor.

    • ignore_failure boolean

      Ignore failures for the processor.

    • on_failure array[object]

      Handle failures for the processor.
    • tag string

      Identifier for the processor. Useful for debugging and metrics.

    • fields string | array[string] Required

      Array of fields to include in the fingerprint. For objects, the processor hashes both the field key and value. For other fields, the processor hashes only the field value.

    • target_field string

      Output field for the fingerprint.

    • salt string

      Salt value for the hash function.

    • method string

      The hash method used to compute the fingerprint. Must be one of MD5, SHA-1, SHA-256, SHA-512, or MurmurHash3.

      Values are MD5, SHA-1, SHA-256, SHA-512, or MurmurHash3.

    • ignore_missing boolean

      If true, the processor ignores any missing fields. If all fields are missing, the processor silently exits without modifying the document.

      Default value is false.

  • foreach object

    Runs an ingest processor on each element of an array or object.

    Hide foreach attributes Show foreach attributes object

    • description string

      Description of the processor. Useful for describing the purpose of the processor or its configuration.

    • if object

      Conditionally execute the processor.

    • ignore_failure boolean

      Ignore failures for the processor.

    • on_failure array[object]

      Handle failures for the processor.
    • tag string

      Identifier for the processor. Useful for debugging and metrics.

    • field string Required

      Field containing array or object values.

    • ignore_missing boolean

      If true, the processor silently exits without changing the document if the field is null or missing.

      Default value is false.

    • processor object Required

      Ingest processor to run on each element.

  • ip_location object

    Currently an undocumented alias for GeoIP Processor.

    Hide ip_location attributes Show ip_location attributes object

    • description string

      Description of the processor. Useful for describing the purpose of the processor or its configuration.

    • if object

      Conditionally execute the processor.

    • ignore_failure boolean

      Ignore failures for the processor.

    • on_failure array[object]

      Handle failures for the processor.
    • tag string

      Identifier for the processor. Useful for debugging and metrics.

    • database_file string

      The database filename referring to a database the module ships with (GeoLite2-City.mmdb, GeoLite2-Country.mmdb, or GeoLite2-ASN.mmdb) or a custom database in the ingest-geoip config directory.

      Default value is GeoLite2-City.mmdb.

    • field string Required

      The field to get the ip address from for the geographical lookup.

    • first_only boolean

      If true, only the first found IP location data will be returned, even if the field contains an array.

      Default value is true.

    • ignore_missing boolean

      If true and field does not exist, the processor quietly exits without modifying the document.

      Default value is false.

    • properties array[string]

      Controls what properties are added to the target_field based on the IP location lookup.

    • target_field string

      The field that will hold the geographical information looked up from the MaxMind database.

    • download_database_on_pipeline_creation boolean

      If true (and if ingest.geoip.downloader.eager.download is false), the missing database is downloaded when the pipeline is created. Else, the download is triggered by when the pipeline is used as the default_pipeline or final_pipeline in an index.

  • geo_grid object

    Converts geo-grid definitions of grid tiles or cells to regular bounding boxes or polygons which describe their shape. This is useful if there is a need to interact with the tile shapes as spatially indexable fields.

    Hide geo_grid attributes Show geo_grid attributes object

    • description string

      Description of the processor. Useful for describing the purpose of the processor or its configuration.

    • if object

      Conditionally execute the processor.

    • ignore_failure boolean

      Ignore failures for the processor.

    • on_failure array[object]

      Handle failures for the processor.
    • tag string

      Identifier for the processor. Useful for debugging and metrics.

    • field string Required

      The field to interpret as a geo-tile.= The field format is determined by the tile_type.

    • tile_type string Required

      Three tile formats are understood: geohash, geotile and geohex.

      Values are geotile, geohex, or geohash.

    • target_field string

      The field to assign the polygon shape to, by default, the field is updated in-place.

    • parent_field string

      If specified and a parent tile exists, save that tile address to this field.

    • children_field string

      If specified and children tiles exist, save those tile addresses to this field as an array of strings.

    • non_children_field string

      If specified and intersecting non-child tiles exist, save their addresses to this field as an array of strings.

    • precision_field string

      If specified, save the tile precision (zoom) as an integer to this field.

    • ignore_missing boolean

      If true and field does not exist, the processor quietly exits without modifying the document.

      Default value is false.

    • target_format string

      Which format to save the generated polygon in.

      Values are geojson or wkt.

  • geoip object

    The geoip processor adds information about the geographical location of an IPv4 or IPv6 address.

    Hide geoip attributes Show geoip attributes object

    • description string

      Description of the processor. Useful for describing the purpose of the processor or its configuration.

    • if object

      Conditionally execute the processor.

    • ignore_failure boolean

      Ignore failures for the processor.

    • on_failure array[object]

      Handle failures for the processor.
    • tag string

      Identifier for the processor. Useful for debugging and metrics.

    • database_file string

      The database filename referring to a database the module ships with (GeoLite2-City.mmdb, GeoLite2-Country.mmdb, or GeoLite2-ASN.mmdb) or a custom database in the ingest-geoip config directory.

      Default value is GeoLite2-City.mmdb.

    • field string Required

      The field to get the ip address from for the geographical lookup.

    • first_only boolean

      If true, only the first found geoip data will be returned, even if the field contains an array.

      Default value is true.

    • ignore_missing boolean

      If true and field does not exist, the processor quietly exits without modifying the document.

      Default value is false.

    • properties array[string]

      Controls what properties are added to the target_field based on the geoip lookup.

    • target_field string

      The field that will hold the geographical information looked up from the MaxMind database.

    • download_database_on_pipeline_creation boolean

      If true (and if ingest.geoip.downloader.eager.download is false), the missing database is downloaded when the pipeline is created. Else, the download is triggered by when the pipeline is used as the default_pipeline or final_pipeline in an index.

  • grok object

    Extracts structured fields out of a single text field within a document. You choose which field to extract matched fields from, as well as the grok pattern you expect will match. A grok pattern is like a regular expression that supports aliased expressions that can be reused.

    Hide grok attributes Show grok attributes object

    • description string

      Description of the processor. Useful for describing the purpose of the processor or its configuration.

    • if object

      Conditionally execute the processor.

    • ignore_failure boolean

      Ignore failures for the processor.

    • on_failure array[object]

      Handle failures for the processor.
    • tag string

      Identifier for the processor. Useful for debugging and metrics.

    • ecs_compatibility string

      Must be disabled or v1. If v1, the processor uses patterns with Elastic Common Schema (ECS) field names.

      Default value is disabled.

    • field string Required

      The field to use for grok expression parsing.

    • ignore_missing boolean

      If true and field does not exist or is null, the processor quietly exits without modifying the document.

      Default value is false.

    • pattern_definitions object

      A map of pattern-name and pattern tuples defining custom patterns to be used by the current processor. Patterns matching existing names will override the pre-existing definition.

      Hide pattern_definitions attribute Show pattern_definitions attribute object

      • * string Additional properties
    • patterns array[string] Required

      An ordered list of grok expression to match and extract named captures with. Returns on the first expression in the list that matches.

    • trace_match boolean

      When true, _ingest._grok_match_index will be inserted into your matched document’s metadata with the index into the pattern found in patterns that matched.

      Default value is false.

  • gsub object

    Converts a string field by applying a regular expression and a replacement. If the field is an array of string, all members of the array will be converted. If any non-string values are encountered, the processor will throw an exception.

    Hide gsub attributes Show gsub attributes object

    • description string

      Description of the processor. Useful for describing the purpose of the processor or its configuration.

    • if object

      Conditionally execute the processor.

    • ignore_failure boolean

      Ignore failures for the processor.

    • on_failure array[object]

      Handle failures for the processor.
    • tag string

      Identifier for the processor. Useful for debugging and metrics.

    • field string Required

      The field to apply the replacement to.

    • ignore_missing boolean

      If true and field does not exist or is null, the processor quietly exits without modifying the document.

      Default value is false.

    • pattern string Required

      The pattern to be replaced.

    • replacement string Required

      The string to replace the matching patterns with.

    • target_field string

      The field to assign the converted value to By default, the field is updated in-place.

  • html_strip object

    Removes HTML tags from the field. If the field is an array of strings, HTML tags will be removed from all members of the array.

    Hide html_strip attributes Show html_strip attributes object

    • description string

      Description of the processor. Useful for describing the purpose of the processor or its configuration.

    • if object

      Conditionally execute the processor.

    • ignore_failure boolean

      Ignore failures for the processor.

    • on_failure array[object]

      Handle failures for the processor.
    • tag string

      Identifier for the processor. Useful for debugging and metrics.

    • field string Required

      The string-valued field to remove HTML tags from.

    • ignore_missing boolean

      If true and field does not exist or is null, the processor quietly exits without modifying the document,

      Default value is false.

    • target_field string

      The field to assign the converted value to By default, the field is updated in-place.

  • inference object

    Uses a pre-trained data frame analytics model or a model deployed for natural language processing tasks to infer against the data that is being ingested in the pipeline.

    Hide inference attributes Show inference attributes object

    • description string

      Description of the processor. Useful for describing the purpose of the processor or its configuration.

    • if object

      Conditionally execute the processor.

    • ignore_failure boolean

      Ignore failures for the processor.

    • on_failure array[object]

      Handle failures for the processor.
    • tag string

      Identifier for the processor. Useful for debugging and metrics.

    • model_id string Required

      The ID or alias for the trained model, or the ID of the deployment.

    • target_field string

      Field added to incoming documents to contain results objects.

    • field_map object

      Maps the document field names to the known field names of the model. This mapping takes precedence over any default mappings provided in the model configuration.

      Hide field_map attribute Show field_map attribute object

      • * object Additional properties
    • inference_config object

      Contains the inference type and its options.

    • input_output object | array[object]

      Input fields for inference and output (destination) fields for the inference results. This option is incompatible with the target_field and field_map options.

      One of:

      InputConfig object array-2 array[object]
    • ignore_missing boolean

      If true and any of the input fields defined in input_ouput are missing then those missing fields are quietly ignored, otherwise a missing field causes a failure. Only applies when using input_output configurations to explicitly list the input fields.

  • join object

    Joins each element of an array into a single string using a separator character between each element. Throws an error when the field is not an array.

    Hide join attributes Show join attributes object

    • description string

      Description of the processor. Useful for describing the purpose of the processor or its configuration.

    • if object

      Conditionally execute the processor.

    • ignore_failure boolean

      Ignore failures for the processor.

    • on_failure array[object]

      Handle failures for the processor.
    • tag string

      Identifier for the processor. Useful for debugging and metrics.

    • field string Required

      Field containing array values to join.

    • separator string Required

      The separator character.

    • target_field string

      The field to assign the joined value to. By default, the field is updated in-place.

  • json object

    Converts a JSON string into a structured JSON object.

    Hide json attributes Show json attributes object

    • description string

      Description of the processor. Useful for describing the purpose of the processor or its configuration.

    • if object

      Conditionally execute the processor.

    • ignore_failure boolean

      Ignore failures for the processor.

    • on_failure array[object]

      Handle failures for the processor.
    • tag string

      Identifier for the processor. Useful for debugging and metrics.

    • add_to_root boolean

      Flag that forces the parsed JSON to be added at the top level of the document. target_field must not be set when this option is chosen.

      Default value is false.

    • add_to_root_conflict_strategy string

      When set to replace, root fields that conflict with fields from the parsed JSON will be overridden. When set to merge, conflicting fields will be merged. Only applicable if add_to_root is set to true.

      Supported values include:

      • replace: Root fields that conflict with fields from the parsed JSON will be overridden.
      • merge: Conflicting fields will be merged.

      Values are replace or merge.

    • allow_duplicate_keys boolean

      When set to true, the JSON parser will not fail if the JSON contains duplicate keys. Instead, the last encountered value for any duplicate key wins.

      Default value is false.

    • field string Required

      The field to be parsed.

    • target_field string

      The field that the converted structured object will be written into. Any existing content in this field will be overwritten.

  • kv object

    This processor helps automatically parse messages (or specific event fields) which are of the foo=bar variety.

    Hide kv attributes Show kv attributes object

    • description string

      Description of the processor. Useful for describing the purpose of the processor or its configuration.

    • if object

      Conditionally execute the processor.

    • ignore_failure boolean

      Ignore failures for the processor.

    • on_failure array[object]

      Handle failures for the processor.
    • tag string

      Identifier for the processor. Useful for debugging and metrics.

    • exclude_keys array[string]

      List of keys to exclude from document.

    • field string Required

      The field to be parsed. Supports template snippets.

    • field_split string Required

      Regex pattern to use for splitting key-value pairs.

    • ignore_missing boolean

      If true and field does not exist or is null, the processor quietly exits without modifying the document.

      Default value is false.

    • include_keys array[string]

      List of keys to filter and insert into document. Defaults to including all keys.

    • prefix string

      Prefix to be added to extracted keys.

      Default value is null.

    • strip_brackets boolean

      If true. strip brackets (), <>, [] as well as quotes ' and " from extracted values.

      Default value is false.

    • target_field string

      The field to insert the extracted keys into. Defaults to the root of the document. Supports template snippets.

    • trim_key string

      String of characters to trim from extracted keys.

    • trim_value string

      String of characters to trim from extracted values.

    • value_split string Required

      Regex pattern to use for splitting the key from the value within a key-value pair.

  • lowercase object

    Converts a string to its lowercase equivalent. If the field is an array of strings, all members of the array will be converted.

    Hide lowercase attributes Show lowercase attributes object

    • description string

      Description of the processor. Useful for describing the purpose of the processor or its configuration.

    • if object

      Conditionally execute the processor.

    • ignore_failure boolean

      Ignore failures for the processor.

    • on_failure array[object]

      Handle failures for the processor.
    • tag string

      Identifier for the processor. Useful for debugging and metrics.

    • field string Required

      The field to make lowercase.

    • ignore_missing boolean

      If true and field does not exist or is null, the processor quietly exits without modifying the document.

      Default value is false.

    • target_field string

      The field to assign the converted value to. By default, the field is updated in-place.

  • network_direction object

    Calculates the network direction given a source IP address, destination IP address, and a list of internal networks.

    Hide network_direction attributes Show network_direction attributes object

    • description string

      Description of the processor. Useful for describing the purpose of the processor or its configuration.

    • if object

      Conditionally execute the processor.

    • ignore_failure boolean

      Ignore failures for the processor.

    • on_failure array[object]

      Handle failures for the processor.
    • tag string

      Identifier for the processor. Useful for debugging and metrics.

    • source_ip string

      Field containing the source IP address.

    • destination_ip string

      Field containing the destination IP address.

    • target_field string

      Output field for the network direction.

    • internal_networks array[string]

      List of internal networks. Supports IPv4 and IPv6 addresses and ranges in CIDR notation. Also supports the named ranges listed below. These may be constructed with template snippets. Must specify only one of internal_networks or internal_networks_field.

    • internal_networks_field string

      A field on the given document to read the internal_networks configuration from.

    • ignore_missing boolean

      If true and any required fields are missing, the processor quietly exits without modifying the document.

      Default value is true.

  • pipeline object

    Executes another pipeline.

    Hide pipeline attributes Show pipeline attributes object

    • description string

      Description of the processor. Useful for describing the purpose of the processor or its configuration.

    • if object

      Conditionally execute the processor.

    • ignore_failure boolean

      Ignore failures for the processor.

    • on_failure array[object]

      Handle failures for the processor.
    • tag string

      Identifier for the processor. Useful for debugging and metrics.

    • name string Required

      The name of the pipeline to execute. Supports template snippets.

    • ignore_missing_pipeline boolean

      Whether to ignore missing pipelines instead of failing.

      Default value is false.

  • redact object

    The Redact processor uses the Grok rules engine to obscure text in the input document matching the given Grok patterns. The processor can be used to obscure Personal Identifying Information (PII) by configuring it to detect known patterns such as email or IP addresses. Text that matches a Grok pattern is replaced with a configurable string such as <EMAIL> where an email address is matched or simply replace all matches with the text <REDACTED> if preferred.

    Hide redact attributes Show redact attributes object

    • description string

      Description of the processor. Useful for describing the purpose of the processor or its configuration.

    • if object

      Conditionally execute the processor.

    • ignore_failure boolean

      Ignore failures for the processor.

    • on_failure array[object]

      Handle failures for the processor.
    • tag string

      Identifier for the processor. Useful for debugging and metrics.

    • field string Required

      The field to be redacted

    • patterns array[string] Required

      A list of grok expressions to match and redact named captures with

    • pattern_definitions object
      Hide pattern_definitions attribute Show pattern_definitions attribute object

      • * string Additional properties
    • prefix string

      Start a redacted section with this token

      Default value is <.

    • suffix string

      End a redacted section with this token

      Default value is >.

    • ignore_missing boolean

      If true and field does not exist or is null, the processor quietly exits without modifying the document.

      Default value is false.

    • skip_if_unlicensed boolean

      If true and the current license does not support running redact processors, then the processor quietly exits without modifying the document

      Default value is false.

    • trace_redact boolean Generally available; Added in 8.16.0

      If true then ingest metadata _ingest._redact._is_redacted is set to true if the document has been redacted

      Default value is false.

  • registered_domain object

    Extracts the registered domain (also known as the effective top-level domain or eTLD), sub-domain, and top-level domain from a fully qualified domain name (FQDN). Uses the registered domains defined in the Mozilla Public Suffix List.

    Hide registered_domain attributes Show registered_domain attributes object

    • description string

      Description of the processor. Useful for describing the purpose of the processor or its configuration.

    • if object

      Conditionally execute the processor.

    • ignore_failure boolean

      Ignore failures for the processor.

    • on_failure array[object]

      Handle failures for the processor.
    • tag string

      Identifier for the processor. Useful for debugging and metrics.

    • field string Required

      Field containing the source FQDN.

    • target_field string

      Object field containing extracted domain components. If an empty string, the processor adds components to the document’s root.

    • ignore_missing boolean

      If true and any required fields are missing, the processor quietly exits without modifying the document.

      Default value is true.

  • remove object

    Removes existing fields. If one field doesn’t exist, an exception will be thrown.

    Hide remove attributes Show remove attributes object

    • description string

      Description of the processor. Useful for describing the purpose of the processor or its configuration.

    • if object

      Conditionally execute the processor.

    • ignore_failure boolean

      Ignore failures for the processor.

    • on_failure array[object]

      Handle failures for the processor.
    • tag string

      Identifier for the processor. Useful for debugging and metrics.

    • field string | array[string] Required

      Fields to be removed. Supports template snippets.

    • keep string | array[string]

      Fields to be kept. When set, all fields other than those specified are removed.

    • ignore_missing boolean

      If true and field does not exist or is null, the processor quietly exits without modifying the document.

      Default value is false.

  • rename object

    Renames an existing field. If the field doesn’t exist or the new name is already used, an exception will be thrown.

    Hide rename attributes Show rename attributes object

    • description string

      Description of the processor. Useful for describing the purpose of the processor or its configuration.

    • if object

      Conditionally execute the processor.

    • ignore_failure boolean

      Ignore failures for the processor.

    • on_failure array[object]

      Handle failures for the processor.
    • tag string

      Identifier for the processor. Useful for debugging and metrics.

    • field string Required

      The field to be renamed. Supports template snippets.

    • ignore_missing boolean

      If true and field does not exist, the processor quietly exits without modifying the document.

      Default value is false.

    • target_field string Required

      The new name of the field. Supports template snippets.

  • reroute object

    Routes a document to another target index or data stream. When setting the destination option, the target is explicitly specified and the dataset and namespace options can’t be set. When the destination option is not set, this processor is in a data stream mode. Note that in this mode, the reroute processor can only be used on data streams that follow the data stream naming scheme.

    Hide reroute attributes Show reroute attributes object

    • description string

      Description of the processor. Useful for describing the purpose of the processor or its configuration.

    • if object

      Conditionally execute the processor.

    • ignore_failure boolean

      Ignore failures for the processor.

    • on_failure array[object]

      Handle failures for the processor.
    • tag string

      Identifier for the processor. Useful for debugging and metrics.

    • destination string

      A static value for the target. Can’t be set when the dataset or namespace option is set.

    • dataset string | array[string]

      Field references or a static value for the dataset part of the data stream name. In addition to the criteria for index names, cannot contain - and must be no longer than 100 characters. Example values are nginx.access and nginx.error.

      Supports field references with a mustache-like syntax (denoted as {{double}} or {{{triple}}} curly braces). When resolving field references, the processor replaces invalid characters with _. Uses the part of the index name as a fallback if all field references resolve to a null, missing, or non-string value.

      default {{data_stream.dataset}}

      One of:

      string-1 string array-2 array[string]

    • namespace string | array[string]

      Field references or a static value for the namespace part of the data stream name. See the criteria for index names for allowed characters. Must be no longer than 100 characters.

      Supports field references with a mustache-like syntax (denoted as {{double}} or {{{triple}}} curly braces). When resolving field references, the processor replaces invalid characters with _. Uses the part of the index name as a fallback if all field references resolve to a null, missing, or non-string value.

      default {{data_stream.namespace}}

      One of:

      string-1 string array-2 array[string]
  • script object

    Runs an inline or stored script on incoming documents. The script runs in the ingest context.

    Hide script attributes Show script attributes object

    • description string

      Description of the processor. Useful for describing the purpose of the processor or its configuration.

    • if object

      Conditionally execute the processor.

    • ignore_failure boolean

      Ignore failures for the processor.

    • on_failure array[object]

      Handle failures for the processor.
    • tag string

      Identifier for the processor. Useful for debugging and metrics.

    • id string

      ID of a stored script. If no source is specified, this parameter is required.

    • lang
    • params object

      Object containing parameters for the script.

      Hide params attribute Show params attribute object

      • * object Additional properties
    • source
  • set object

    Adds a field with the specified value. If the field already exists, its value will be replaced with the provided one.

    Hide set attributes Show set attributes object

    • description string

      Description of the processor. Useful for describing the purpose of the processor or its configuration.

    • if object

      Conditionally execute the processor.

    • ignore_failure boolean

      Ignore failures for the processor.

    • on_failure array[object]

      Handle failures for the processor.
    • tag string

      Identifier for the processor. Useful for debugging and metrics.

    • copy_from string

      The origin field which will be copied to field, cannot set value simultaneously. Supported data types are boolean, number, array, object, string, date, etc.

    • field string Required

      The field to insert, upsert, or update. Supports template snippets.

    • ignore_empty_value boolean

      If true and value is a template snippet that evaluates to null or the empty string, the processor quietly exits without modifying the document.

      Default value is false.

    • media_type string

      The media type for encoding value. Applies only when value is a template snippet. Must be one of application/json, text/plain, or application/x-www-form-urlencoded.

    • override boolean

      If true processor will update fields with pre-existing non-null-valued field. When set to false, such fields will not be touched.

      Default value is true.

    • value object

      The value to be set for the field. Supports template snippets. May specify only one of value or copy_from.

  • set_security_user object

    Sets user-related details (such as username, roles, email, full_name, metadata, api_key, realm and authentication_type) from the current authenticated user to the current document by pre-processing the ingest.

    Hide set_security_user attributes Show set_security_user attributes object

    • description string

      Description of the processor. Useful for describing the purpose of the processor or its configuration.

    • if object

      Conditionally execute the processor.

    • ignore_failure boolean

      Ignore failures for the processor.

    • on_failure array[object]

      Handle failures for the processor.
    • tag string

      Identifier for the processor. Useful for debugging and metrics.

    • field string Required

      The field to store the user information into.

    • properties array[string]

      Controls what user related properties are added to the field.

  • sort object

    Sorts the elements of an array ascending or descending. Homogeneous arrays of numbers will be sorted numerically, while arrays of strings or heterogeneous arrays of strings + numbers will be sorted lexicographically. Throws an error when the field is not an array.

    Hide sort attributes Show sort attributes object

    • description string

      Description of the processor. Useful for describing the purpose of the processor or its configuration.

    • if object

      Conditionally execute the processor.

    • ignore_failure boolean

      Ignore failures for the processor.

    • on_failure array[object]

      Handle failures for the processor.
    • tag string

      Identifier for the processor. Useful for debugging and metrics.

    • field string Required

      The field to be sorted.

    • order string

      The sort order to use. Accepts "asc" or "desc".

      Supported values include:

      • asc: Ascending (smallest to largest)
      • desc: Descending (largest to smallest)

      Values are asc or desc.

    • target_field string

      The field to assign the sorted value to. By default, the field is updated in-place.

  • split object

    Splits a field into an array using a separator character. Only works on string fields.

    Hide split attributes Show split attributes object

    • description string

      Description of the processor. Useful for describing the purpose of the processor or its configuration.

    • if object

      Conditionally execute the processor.

    • ignore_failure boolean

      Ignore failures for the processor.

    • on_failure array[object]

      Handle failures for the processor.
    • tag string

      Identifier for the processor. Useful for debugging and metrics.

    • field string Required

      The field to split.

    • ignore_missing boolean

      If true and field does not exist, the processor quietly exits without modifying the document.

      Default value is false.

    • preserve_trailing boolean

      Preserves empty trailing fields, if any.

      Default value is false.

    • separator string Required

      A regex which matches the separator, for example, , or \s+.

    • target_field string

      The field to assign the split value to. By default, the field is updated in-place.

  • terminate object

    Terminates the current ingest pipeline, causing no further processors to be run. This will normally be executed conditionally, using the if option.

    Hide terminate attributes Show terminate attributes object

    • description string

      Description of the processor. Useful for describing the purpose of the processor or its configuration.

    • if object

      Conditionally execute the processor.

    • ignore_failure boolean

      Ignore failures for the processor.

    • on_failure array[object]

      Handle failures for the processor.
    • tag string

      Identifier for the processor. Useful for debugging and metrics.

  • trim object

    Trims whitespace from a field. If the field is an array of strings, all members of the array will be trimmed. This only works on leading and trailing whitespace.

    Hide trim attributes Show trim attributes object

    • description string

      Description of the processor. Useful for describing the purpose of the processor or its configuration.

    • if object

      Conditionally execute the processor.

    • ignore_failure boolean

      Ignore failures for the processor.

    • on_failure array[object]

      Handle failures for the processor.
    • tag string

      Identifier for the processor. Useful for debugging and metrics.

    • field string Required

      The string-valued field to trim whitespace from.

    • ignore_missing boolean

      If true and field does not exist, the processor quietly exits without modifying the document.

      Default value is false.

    • target_field string

      The field to assign the trimmed value to. By default, the field is updated in-place.

  • uppercase object

    Converts a string to its uppercase equivalent. If the field is an array of strings, all members of the array will be converted.

    Hide uppercase attributes Show uppercase attributes object

    • description string

      Description of the processor. Useful for describing the purpose of the processor or its configuration.

    • if object

      Conditionally execute the processor.

    • ignore_failure boolean

      Ignore failures for the processor.

    • on_failure array[object]

      Handle failures for the processor.
    • tag string

      Identifier for the processor. Useful for debugging and metrics.

    • field string Required

      The field to make uppercase.

    • ignore_missing boolean

      If true and field does not exist or is null, the processor quietly exits without modifying the document.

      Default value is false.

    • target_field string

      The field to assign the converted value to. By default, the field is updated in-place.

  • urldecode object

    URL-decodes a string. If the field is an array of strings, all members of the array will be decoded.

    Hide urldecode attributes Show urldecode attributes object

    • description string

      Description of the processor. Useful for describing the purpose of the processor or its configuration.

    • if object

      Conditionally execute the processor.

    • ignore_failure boolean

      Ignore failures for the processor.

    • on_failure array[object]

      Handle failures for the processor.
    • tag string

      Identifier for the processor. Useful for debugging and metrics.

    • field string Required

      The field to decode.

    • ignore_missing boolean

      If true and field does not exist or is null, the processor quietly exits without modifying the document.

      Default value is false.

    • target_field string

      The field to assign the converted value to. By default, the field is updated in-place.

  • uri_parts object

    Parses a Uniform Resource Identifier (URI) string and extracts its components as an object. This URI object includes properties for the URI’s domain, path, fragment, port, query, scheme, user info, username, and password.

    Hide uri_parts attributes Show uri_parts attributes object

    • description string

      Description of the processor. Useful for describing the purpose of the processor or its configuration.

    • if object

      Conditionally execute the processor.

    • ignore_failure boolean

      Ignore failures for the processor.

    • on_failure array[object]

      Handle failures for the processor.
    • tag string

      Identifier for the processor. Useful for debugging and metrics.

    • field string Required

      Field containing the URI string.

    • ignore_missing boolean

      If true and field does not exist, the processor quietly exits without modifying the document.

      Default value is false.

    • keep_original boolean

      If true, the processor copies the unparsed URI to <target_field>.original.

      Default value is true.

    • remove_if_successful boolean

      If true, the processor removes the field after parsing the URI string. If parsing fails, the processor does not remove the field.

      Default value is false.

    • target_field string

      Output field for the URI object.

  • user_agent object

    The user_agent processor extracts details from the user agent string a browser sends with its web requests. This processor adds this information by default under the user_agent field.

    Hide user_agent attributes Show user_agent attributes object

    • description string

      Description of the processor. Useful for describing the purpose of the processor or its configuration.

    • if object

      Conditionally execute the processor.

    • ignore_failure boolean

      Ignore failures for the processor.

    • on_failure array[object]

      Handle failures for the processor.
    • tag string

      Identifier for the processor. Useful for debugging and metrics.

    • field string Required

      The field containing the user agent string.

    • ignore_missing boolean

      If true and field does not exist, the processor quietly exits without modifying the document.

      Default value is false.

    • regex_file string

      The name of the file in the config/ingest-user-agent directory containing the regular expressions for parsing the user agent string. Both the directory and the file have to be created before starting Elasticsearch. If not specified, ingest-user-agent will use the regexes.yaml from uap-core it ships with.

    • target_field string

      The field that will be filled with the user agent details.

    • properties array[string]

      Controls what properties are added to target_field.

      Values are name, os, device, original, or version. Default value is ["name", "major", "minor", "patch", "build", "os", "os_name", "os_major", "os_minor", "device"].

    • extract_device_type boolean Beta; Added in 8.9.0

      Extracts device type from the user agent string on a best-effort basis.

      Default value is false.

• version number

  Version number used by external systems to track ingest pipelines. This parameter is intended for external systems only. Elasticsearch does not use or validate pipeline version numbers.

• deprecated boolean

  Marks this ingest pipeline as deprecated. When a deprecated ingest pipeline is referenced as the default or final pipeline when creating or updating a non-deprecated index template, Elasticsearch will emit a deprecation warning.

  Default value is false.