.. Licensed to the Apache Software Foundation (ASF) under one or more contributor license agreements. See the NOTICE file distributed with this work for additional information regarding copyright ownership. The ASF licenses this file to you under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. Copyright 2020, Verizon Media .. include:: ../../../../common.defs .. highlight:: yaml .. default-domain:: txb .. _modifier_reference: ******************* Modifier Reference ******************* Modifiers are a way to modify or filter features. Modifiers ********* .. modifier:: else The :mod:`else` modifier leaves the feature unchanged unless it is the :code:`NULL` value or is an empty string. In that case, the feature is changed to be the value of the modifier key. .. modifier:: hash Compute the hash of a feature and take the result modulo the value. This modifier requires a value that is a positive integer. .. modifier:: filter Filter is intended to operate on lists, although it will work on a single value as if it were a list of length 1. The purpose of filter is to create a new list based on the input list. The common form of this is to remove elements. The modifier takes a list of comparisons, each of which can have an attached action as a key. The valid action keys are pass Pass the item through to the new list unchanged. drop Do not pass the item to the new list, in effect removing it. replace Add an item to the new list in place of the original element. This key takes a feature expression which is the new value to put in the new list. It is an error to have more than one action for a comparison. If no action is specified with a comparison, a match will ``pass`` the item. If an action has no comparison, it always matches. The comparisons are checked in order for each element and the first match determines the action for that element. If there are no matches, the item is dropped. This makes it easy to remove elements. To remove any "set-cookie" field with the loopback address in it - :: [ proxy-rsp-field , { filter: [ { contains: "127.0.0.1", drop: }, { pass: } ] } ] or long hand :: - proxy-rsp-field - filter: - contains: "127.0.0.1" drop: - pass: It is not permitted to have a :code:`do` key with any of the comparisons. See :ref:`filter-guide` for examples of using this modifier. .. modifier:: join :arg: Separator Join features in to a string. The value is used as the separator between elements. If used on a scalar feature it will simply convert that feature to a string. If used on tuple it will convert each tuple element to a string and concentate the result, placing the separator between the strings. Nested tuples are placed in brackets and the modifier recursively applied. NULL elements are discared, but empty strings are retained. .. modifier:: concat :arg: List of separator, string Concatenate a string. This takes a list of two values, the separator and the string. If the active feature is a non-empty string, and the string value for the modifier is not empty, the latter is appended to the active feature. The separator is appended first if the active feature does not already end with the separator. For example :: [ pre-remap-path , { concat: [ "/" , "albums" ] } ] will add "albums" to the pre-remap path. If that path is "delain" the result is "delain/albums". If the path is "delain/" the result is still "delain/albums". A common use is to attach a query string to a URL while avoiding adding "?" if there is no query stirng. E.g. :: [ "http://delain.nl/albums/{pre-remap-path}", { concat: [ "?" , pre-remap-query ] } ] which propagate the query string without creating a URL ending in "?". If there was no query string :ex:`pre-remap-query` will be the empty string and the modifier will not change the string. This can be used to append separated strings even on empty fields. For instance, to make sure the list of rock bands contains "Delain", it would be .. literalinclude:: ../../../../../tests/gold_tests/pluginTest/txn_box/basic/mod.replay.yaml :start-after: doc-concat-empty-< :end-before: doc-concat-empty-> An empty field is changed to "Delain", while if there is already a value, a comma is added before adding "Delain". .. modifier:: as-bool Coerce the feature to a Boolean if possible. The following types can be coerced Boolean Identity. Integer 0 is false, any other value is true. Float 0.0 is false, anything else is true. String False unless the string is one of "true", "1", "on", "enable", "Y", "yes". IP Address A valid IP address is true, otherwise false. Duration A non-zero duration is true, otherwise false. Tuple A non-zero length tuple is true, otherwise false. Note the value(s) in the tuple are not examined. Null False. .. modifier:: as-integer Coerce the feature to an Integer type if possible. If the feature is already an Integer, it is left unchanged. If it is a string, it is parsed as an integer string and if that is successfull, the feature is changed to be the resulting integer value. If not, the modifier value is used. This is the :code:`NULL` value if unspecified. .. modifier:: as-duration Convert to a duration. If the feature is a string the conversion is done based on following table. ============ ======================== Duration Names ============ ======================== nanoseconds ns, nanoseconds microseconds us, microseconds milliseconds ms, milliseconds second s, sec, second, seconds minute m, min, minute, minutes hour h, hour, hours day d, day, days week w, week, weeks ============ ======================== The string must consist of pairs, each pair an integer followed by a name. Spaces are ignored, but can be added for clarity. The duration for the string is the sum of all the pairs, irrespective of order. For instance, a duration of ninety minutes could be "90 minutes", "90m", "1h 30m", "30 m 1 hour", "5400 sec", or even "900 s1 hours15 minute". Note the singular vs. plural forms are purely for convenience. "1 day" and "1 days" are indistinguishable, as are "10 minute" and "10 minutes". If used on a tuple, each element will be coerced to a duration and all of the durations summed. .. modifier:: as-ip-addr Convert to an IP address or range. If the conversion can not be done, the result is :code:`NULL`. .. modifier:: ip-space :arg: IP Space name This modifier must be used in conjunction with :drtv:`ip-space-define`. The latter defines and loads an IP space. This modifier changes an IP address feature into a row from the named IP Space. The value for the modifier is a feature expression in which :ex:`ip-col` can be used to extract data from the row. The value of that expression replaces the IP address. .. modifier:: as-text-block :arg: Block name as a string. Convert a string in to the contents of a text block. The active value is treated as the name. The modifier value is used if there is no block for the name. If there is no block and no name the result is an empty string. .. modifier:: url-encode :value: string The :mod:`url-encode` perform percent-encoding of a feature, this provides a mechanism for encoding information in a Uniform Resource Identifier (URI). This modifier uses the |TS| api :code:`TSStringPercentEncode` to perform the encoding. In addition we are using a custom map optional in the mentioned API) to deal with reserved characters(RFC2396:Sec 2.2). .. code:: reserved = ";" | "/" | "?" | ":" | "@" | "&" | "=" | "+" | "$" | "," The following example shows how a particular field can be encoded. .. literalinclude:: ../../../../../tests/gold_tests/pluginTest/txn_box/basic/mod.replay.yaml :start-after: doc-url-encode-< :end-before: doc-url-encode-> Perform a redirect and apply the :mod:`url-encode` to :ex:`pre-remap-query` in combination with :mod:`concat`. .. literalinclude:: ../../../../../tests/gold_tests/pluginTest/txn_box/basic/redirect.replay.yaml :start-after: doc-redirect-url-encode-form-< :end-before: doc-redirect-url-encode-form-> .. modifier:: url-decode :value: string The :mod:`url-decode` perform percent-decoding of a feature, this provides a mechanism for encoding information in a Uniform Resource Identifier (URI).This internally uses the |TS| api :code:`TSStringPercentDecode` to perform the decoding. The following example shows how a particular field can be encoded. .. literalinclude:: ../../../../../tests/gold_tests/pluginTest/txn_box/basic/mod.replay.yaml :start-after: doc-url-decode-< :end-before: doc-url-decode-> Perform a redirect and apply the :mod:`url-decode` to :ex:`pre-remap-query` in combination with :mod:`concat`. .. literalinclude:: ../../../../../tests/gold_tests/pluginTest/txn_box/basic/redirect.replay.yaml :start-after: doc-redirect-url-decode-form-< :end-before: doc-redirect-url-decode-form-> .. modifier:: query-sort :arg: ``nc``, ``rev`` Sort a query string by name. ``nc`` Caseless comparison for sorting. ``rev`` Sort in reverse (descending) order. .. modifier:: query-filter This is designed to work with URL query strings in a manner similar to :mod:`filter`. The filter is applied to each element in the query string. Comparisons match on the name. The basic actions are the same as :mod:`filter`. pass Pass element to the output unchanged. drop Do not pass the element to the output (the element is removed). replace Replace the element with a new element. This can have two nested keys, ``name`` and ``value``. These are optional and should contain feature expressions which will replace the existing text. It is an error to have more than one action for a comparison. It is an error to have a :code:`do` key with any of the comparisons. If no action is specified with a comparison, a match will ``pass`` the item. If an action has no comparison, it always matches. The comparisons are checked in order for each element and the first match determines the action for that element. If there are no matches, the item is dropped. In addition to actions there are a set of options which are attached to the ``option`` key with the comparison. These nested keys are value This must contain a comparison which is applied to the value for the element. append Append a new element to the output. This requires a list of two elements, the name and value. This will be added after the current element (if passed or replaced). It is valid to ``drop`` the current element and then ``append``, although that is more easily done by using the list form of ``replace``. append-unique **NOT IMPLEMENTED** As ``append`` but only once. If this is used, the name for the appended element will appear exactly once in the modified query string. Using this will effect all elements with the name, regardless of how such elements are added to the output. If some previous element was passed with this name, this append will be dropped to satisfy the uniqueness. pass-rest Stop filtering - all remaining elements are passed. drop-rest Stop filtering - all remaining elements are dropped. For feature expressions in the scope of this modifier the extractors ``name`` and ``value`` are locally bound to the name and value of the current element. Example - drop all elements with the name "email" if the address is for "base.ex". :: query-filter: - match: "email" option: value: suffix: "@base.ex" drop: - pass: # everything else pass on through. Change the name "x-email" to "email" for all elements. :: query-filter: - match: "x-email" replace: name: "email" - pass: Make every value more exciting. :: query-filter: - replace: value: "{value}!" .. modifier:: rxp-replace :arg: nc, g Perform a regular expression based search and replace on a string. This takes a list of two elements, the *pattern* and the *replacement*. As is common this modifier searches the value a match for the regular expression *pattern*. If found the text matching *pattern* is removed and replaced by replacement*. Capture group extractors are available to extract text from the matched text. PCRE syntax is used for the regular expression. The arguments are nc Case insensitive matching. g Global - after a match and replacement, the remaining string is searched for further matches.