Skip to main content

YAML Reference

This page lists all components, interpolation variables and interpolation macros that can be used when defining a low code YAML file.

For the technical JSON schema definition that low code manifests are validated against, see here.

Components​

DeclarativeSource object​

An API source that extracts data according to its declarative components.

Properties:

AddedFieldDefinition object​

Defines the field to add on a record.

Properties:

  • path array

    List of strings defining the path where to add the value on the record.

    Examples:
    [
  "segment_id"
]
    [
  "metadata",
  "segment_id"
]

  • value string

    Value of the new field. Use {{ record['existing_field'] }} syntax to refer to other fields in the record.

    Available variables:
    Examples:
    "{{ record['updates'] }}"
    "{{ record['MetaData']['LastUpdatedTime'] }}"
    "{{ stream_partition['segment_id'] }}"

  • value_type #/definitions/ValueType

    Type of the value. If not specified, the type will be inferred from the value.

  • $parameters object

    Set parameters that are inherited to all children. See the section in the advanced topics for more details.

AddFields object​

Transformation which adds field to an output record. The path of the added field can be nested.

Properties:

  • fields array

    List of transformations (path and corresponding value) that will be added to the record.

  • $parameters object

    Set parameters that are inherited to all children. See the section in the advanced topics for more details.

ApiKeyAuthenticator object​

Authenticator for requests authenticated with an API token injected as an HTTP request header.

Properties:

  • api_token string

    The API key to inject in the request. Fill it in the user inputs.

    Available variables:
    Examples:
    "{{ config['api_key'] }}"
    "Token token={{ config['api_key'] }}"

  • header string

    The name of the HTTP header that will be set to the API key. This setting is deprecated, use inject_into instead. Header and inject_into can not be defined at the same time.

    Available variables:
    Examples:
    "Authorization"
    "Api-Token"
    "X-Auth-Token"

  • inject_into #/definitions/RequestOption

    Configure how the API Key will be sent in requests to the source API. Either inject_into or header has to be defined.

    Examples:
    {
  "inject_into": "header",
  "field_name": "Authorization"
}
    {
  "inject_into": "request_parameter",
  "field_name": "authKey"
}

  • $parameters object

    Set parameters that are inherited to all children. See the section in the advanced topics for more details.

AuthFlow object​

Additional and optional specification object to describe what an 'advanced' Auth flow would need to function.

  • A connector should be able to fully function with the configuration as described by the ConnectorSpecification in a 'basic' mode.
  • The 'advanced' mode provides easier UX for the user with UI improvements and automations. However, this requires further setup on the server side by instance or workspace admins beforehand. The trade-off is that the user does not have to provide as many technical inputs anymore and the auth process is faster and easier to complete.
Properties:

  • auth_flow_type string

    The type of auth to use

  • predicate_key array

    JSON path to a field in the connectorSpecification that should exist for the advanced auth to be applicable.

    Example:
    [
  "credentials",
  "auth_type"
]

  • predicate_value string

    Value of the predicate_key fields for the advanced auth to be applicable.

    Example:
    "Oauth"

  • oauth_config_specification #/definitions/OAuthConfigSpecification

BasicHttpAuthenticator object​

Authenticator for requests authenticated with the Basic HTTP authentication scheme, which encodes a username and an optional password in the Authorization request header.

Properties:

  • username string

    The username that will be combined with the password, base64 encoded and used to make requests. Fill it in the user inputs.

    Available variables:
    Examples:
    "{{ config['username'] }}"
    "{{ config['api_key'] }}"

  • password string

    The password that will be combined with the username, base64 encoded and used to make requests. Fill it in the user inputs.

    Available variables:
    Examples:
    "{{ config['password'] }}"
    ""

  • $parameters object

    Set parameters that are inherited to all children. See the section in the advanced topics for more details.

BearerAuthenticator object​

Authenticator for requests authenticated with a bearer token injected as a request header of the form Authorization: Bearer <token>.

Properties:

  • api_token string

    Token to inject as request header for authenticating with the API.

    Available variables:
    Examples:
    "{{ config['api_key'] }}"
    "{{ config['token'] }}"

  • $parameters object

    Set parameters that are inherited to all children. See the section in the advanced topics for more details.

SelectiveAuthenticator object​

Authenticator that selects concrete authenticator based on config property.

Properties:

  • authenticator_selection_path array

    Path of the field in config with selected authenticator name

    Examples:
    [
  "auth"
]
    [
  "auth",
  "type"
]

  • authenticators object

    Authenticators to select from.

    Example:
    {
  "authenticators": {
    "token": "#/definitions/ApiKeyAuthenticator",
    "oauth": "#/definitions/OAuthAuthenticator"
  }
}

  • $parameters object

    Set parameters that are inherited to all children. See the section in the advanced topics for more details.

CheckStream object​

Defines the streams to try reading when running a check operation.

Properties:

  • stream_names array

    Names of the streams to try reading from when running a check operation.

    Examples:
    [
  "users"
]
    [
  "users",
  "contacts"
]

CompositeErrorHandler object​

Error handler that sequentially iterates over a list of error handlers.

Properties:

  • error_handlers array

    List of error handlers to iterate on to determine how to handle a failed response.

  • $parameters object

    Set parameters that are inherited to all children. See the section in the advanced topics for more details.

ConstantBackoffStrategy object​

Backoff strategy with a constant backoff interval.

Properties:

  • backoff_time_in_seconds

    Backoff time in seconds.

    Type:
    • number
    • string

    Available variables:
    Examples:
    30
    30.5
    "{{ config['backoff_time'] }}"

  • $parameters object

    Set parameters that are inherited to all children. See the section in the advanced topics for more details.

CursorPagination object​

Pagination strategy that evaluates an interpolated string to define the next page to fetch.

Properties:

  • cursor_value string

    Value of the cursor defining the next page to fetch.

    Available variables:
    Examples:
    "{{ headers.link.next.cursor }}"
    "{{ last_records[-1]['key'] }}"
    "{{ response['nextPage'] }}"

  • page_size integer

    The number of records to include in each pages.

    Example:
    100

  • stop_condition string

    Template string evaluating when to stop paginating.

    Available variables:
    Examples:
    "{{ response.data.has_more is false }}"
    "{{ 'next' not in headers['link'] }}"

  • decoder #/definitions/JsonDecoder

    Component decoding the response so records can be extracted.

  • $parameters object

    Set parameters that are inherited to all children. See the section in the advanced topics for more details.

CustomAuthenticator object​

Authenticator component whose behavior is derived from a custom code implementation of the connector.

Properties:

  • class_name string

    Fully-qualified name of the class that will be implementing the custom authentication strategy. Has to be a sub class of DeclarativeAuthenticator. The format is source_<name>.<package>.<class_name>.

    Example:
    "source_railz.components.ShortLivedTokenAuthenticator"

  • $parameters object

    Set parameters that are inherited to all children. See the section in the advanced topics for more details.

CustomBackoffStrategy object​

Backoff strategy component whose behavior is derived from a custom code implementation of the connector.

Properties:

  • class_name string

    Fully-qualified name of the class that will be implementing the custom backoff strategy. The format is source_<name>.<package>.<class_name>.

    Example:
    "source_railz.components.MyCustomBackoffStrategy"

  • $parameters object

    Set parameters that are inherited to all children. See the section in the advanced topics for more details.

CustomErrorHandler object​

Error handler component whose behavior is derived from a custom code implementation of the connector.

Properties:

  • class_name string

    Fully-qualified name of the class that will be implementing the custom error handler. The format is source_<name>.<package>.<class_name>.

    Example:
    "source_railz.components.MyCustomErrorHandler"

  • $parameters object

    Set parameters that are inherited to all children. See the section in the advanced topics for more details.

CustomIncrementalSync object​

Incremental component whose behavior is derived from a custom code implementation of the connector.

Properties:

  • class_name string

    Fully-qualified name of the class that will be implementing the custom incremental sync. The format is source_<name>.<package>.<class_name>.

    Example:
    "source_railz.components.MyCustomIncrementalSync"

  • cursor_field string

    The location of the value on a record that will be used as a bookmark during sync.

  • $parameters object

    Set parameters that are inherited to all children. See the section in the advanced topics for more details.

CustomPaginationStrategy object​

Pagination strategy component whose behavior is derived from a custom code implementation of the connector.

Properties:

  • class_name string

    Fully-qualified name of the class that will be implementing the custom pagination strategy. The format is source_<name>.<package>.<class_name>.

    Example:
    "source_railz.components.MyCustomPaginationStrategy"

  • $parameters object

    Set parameters that are inherited to all children. See the section in the advanced topics for more details.

CustomRecordExtractor object​

Record extractor component whose behavior is derived from a custom code implementation of the connector.

Properties:

  • class_name string

    Fully-qualified name of the class that will be implementing the custom record extraction strategy. The format is source_<name>.<package>.<class_name>.

    Example:
    "source_railz.components.MyCustomRecordExtractor"

  • $parameters object

    Set parameters that are inherited to all children. See the section in the advanced topics for more details.

CustomRecordFilter object​

Record filter component whose behavior is derived from a custom code implementation of the connector.

Properties:

  • class_name string

    Fully-qualified name of the class that will be implementing the custom record filter strategy. The format is source_<name>.<package>.<class_name>.

    Example:
    "source_railz.components.MyCustomCustomRecordFilter"

  • $parameters object

    Set parameters that are inherited to all children. See the section in the advanced topics for more details.

CustomRequester object​

Requester component whose behavior is derived from a custom code implementation of the connector.

Properties:

  • class_name string

    Fully-qualified name of the class that will be implementing the custom requester strategy. The format is source_<name>.<package>.<class_name>.

    Example:
    "source_railz.components.MyCustomRecordExtractor"

  • $parameters object

    Set parameters that are inherited to all children. See the section in the advanced topics for more details.

CustomRetriever object​

Retriever component whose behavior is derived from a custom code implementation of the connector.

Properties:

  • class_name string

    Fully-qualified name of the class that will be implementing the custom retriever strategy. The format is source_<name>.<package>.<class_name>.

    Example:
    "source_railz.components.MyCustomRetriever"

  • $parameters object

    Set parameters that are inherited to all children. See the section in the advanced topics for more details.

CustomPartitionRouter object​

Partition router component whose behavior is derived from a custom code implementation of the connector.

Properties:

  • class_name string

    Fully-qualified name of the class that will be implementing the custom partition router. The format is source_<name>.<package>.<class_name>.

    Example:
    "source_railz.components.MyCustomPartitionRouter"

  • $parameters object

    Set parameters that are inherited to all children. See the section in the advanced topics for more details.

CustomTransformation object​

Transformation component whose behavior is derived from a custom code implementation of the connector.

Properties:

  • class_name string

    Fully-qualified name of the class that will be implementing the custom transformation. The format is source_<name>.<package>.<class_name>.

    Example:
    "source_railz.components.MyCustomTransformation"

  • $parameters object

    Set parameters that are inherited to all children. See the section in the advanced topics for more details.

DatetimeBasedCursor object​

Cursor to provide incremental capabilities over datetime.

Properties:

  • cursor_field string

    The location of the value on a record that will be used as a bookmark during sync. To ensure no data loss, the API must return records in ascending order based on the cursor field. Nested fields are not supported, so the field must be at the top level of the record. You can use a combination of Add Field and Remove Field transformations to move the nested field to the top.

    Available variables:
    Examples:
    "created_at"
    "{{ config['record_cursor'] }}"

  • datetime_format string

    The datetime format used to format the datetime values that are sent in outgoing requests to the API. Use placeholders starting with "%" to describe the format the API is using. The following placeholders are available:

    • %s: Epoch unix timestamp - 1686218963
    • %ms: Epoch unix timestamp (milliseconds) - 1686218963123
    • %a: Weekday (abbreviated) - Sun
    • %A: Weekday (full) - Sunday
    • %w: Weekday (decimal) - 0 (Sunday), 6 (Saturday)
    • %d: Day of the month (zero-padded) - 01, 02, ..., 31
    • %b: Month (abbreviated) - Jan
    • %B: Month (full) - January
    • %m: Month (zero-padded) - 01, 02, ..., 12
    • %y: Year (without century, zero-padded) - 00, 01, ..., 99
    • %Y: Year (with century) - 0001, 0002, ..., 9999
    • %H: Hour (24-hour, zero-padded) - 00, 01, ..., 23
    • %I: Hour (12-hour, zero-padded) - 01, 02, ..., 12
    • %p: AM/PM indicator
    • %M: Minute (zero-padded) - 00, 01, ..., 59
    • %S: Second (zero-padded) - 00, 01, ..., 59
    • %f: Microsecond (zero-padded to 6 digits) - 000000
    • %z: UTC offset - (empty), +0000, -04:00
    • %Z: Time zone name - (empty), UTC, GMT
    • %j: Day of the year (zero-padded) - 001, 002, ..., 366
    • %U: Week number of the year (starting Sunday) - 00, ..., 53
    • %W: Week number of the year (starting Monday) - 00, ..., 53
    • %c: Date and time - Tue Aug 16 21:30:00 1988
    • %x: Date standard format - 08/16/1988
    • %X: Time standard format - 21:30:00
    • %%: Literal '%' character

    Some placeholders depend on the locale of the underlying system - in most cases this locale is configured as en/US. For more information see the Python documentation.

    Examples:
    "%Y-%m-%dT%H:%M:%S.%f%z"
    "%Y-%m-%d"
    "%s"
    "%ms"

  • start_datetime

    The datetime that determines the earliest record that should be synced.

    Type:
    Available variables:
    Examples:
    "2020-01-1T00:00:00Z"
    "{{ config['start_time'] }}"

  • cursor_datetime_formats array

    The possible formats for the cursor field, in order of preference. The first format that matches the cursor field value will be used to parse it. If not provided, the datetime_format will be used.

  • cursor_granularity string

    Smallest increment the datetime_format has (ISO 8601 duration) that is used to ensure the start of a slice does not overlap with the end of the previous one, e.g. for %Y-%m-%d the granularity should be P1D, for %Y-%m-%dT%H:%M:%SZ the granularity should be PT1S. Given this field is provided, step needs to be provided as well.

    Example:
    "PT1S"

  • end_datetime

    The datetime that determines the last record that should be synced. If not provided, {{ now_utc() }} will be used.

    Type:
    Available variables:
    Examples:
    "2021-01-1T00:00:00Z"
    "{{ now_utc() }}"
    "{{ day_delta(-1) }}"

  • end_time_option #/definitions/RequestOption

    Optionally configures how the end datetime will be sent in requests to the source API.

  • is_data_feed boolean

    A data feed API is an API that does not allow filtering and paginates the content from the most recent to the least recent. Given this, the CDK needs to know when to stop paginating and this field will generate a stop condition for pagination.

  • lookback_window string

    Time interval before the start_datetime to read data for, e.g. P1M for looking back one month.

    Available variables:
    Examples:
    "P1D"
    "P{{ config['lookback_days'] }}D"

  • partition_field_end string

    Name of the partition start time field.

    Example:
    "ending_time"

  • partition_field_start string

    Name of the partition end time field.

    Example:
    "starting_time"

  • start_time_option #/definitions/RequestOption

    Optionally configures how the start datetime will be sent in requests to the source API.

  • step string

    The size of the time window (ISO8601 duration). Given this field is provided, cursor_granularity needs to be provided as well.

    Examples:
    "P1W"
    "{{ config['step_increment'] }}"

  • $parameters object

    Set parameters that are inherited to all children. See the section in the advanced topics for more details.

OAuthAuthenticator object​

Authenticator for requests using OAuth 2.0 authorization flow.

Properties:

  • client_id string

    The OAuth client ID. Fill it in the user inputs.

    Examples:
    "{{ config['client_id }}"
    "{{ config['credentials']['client_id }}"

  • client_secret string

    The OAuth client secret. Fill it in the user inputs.

    Examples:
    "{{ config['client_secret }}"
    "{{ config['credentials']['client_secret }}"

  • refresh_token string

    Credential artifact used to get a new access token.

    Examples:
    "{{ config['refresh_token'] }}"
    "{{ config['credentials]['refresh_token'] }}"

  • token_refresh_endpoint string

    The full URL to call to obtain a new access token.

    Example:
    "https://connect.squareup.com/oauth2/token"

  • access_token_name string

    The name of the property which contains the access token in the response from the token refresh endpoint.

    Example:
    "access_token"

  • expires_in_name string

    The name of the property which contains the expiry date in the response from the token refresh endpoint.

    Example:
    "expires_in"

  • grant_type string

    Specifies the OAuth2 grant type. If set to refresh_token, the refresh_token needs to be provided as well. For client_credentials, only client id and secret are required. Other grant types are not officially supported.

    Examples:
    "refresh_token"
    "client_credentials"

  • refresh_request_body object

    Body of the request sent to get a new access token.

    Example:
    {
  "applicationId": "{{ config['application_id'] }}",
  "applicationSecret": "{{ config['application_secret'] }}",
  "token": "{{ config['token'] }}"
}

  • scopes array

    List of scopes that should be granted to the access token.

    Example:
    [
  "crm.list.read",
  "crm.objects.contacts.read",
  "crm.schema.contacts.read"
]

  • token_expiry_date string

    The access token expiry date.

    Examples:
    "2023-04-06T07:12:10.421833+00:00"
    1680842386

  • token_expiry_date_format string

    The format of the time to expiration datetime. Provide it if the time is returned as a date-time string instead of seconds.

    Example:
    "%Y-%m-%d %H:%M:%S.%f+00:00"

  • refresh_token_updater

    When the token updater is defined, new refresh tokens, access tokens and the access token expiry date are written back from the authentication response to the config object. This is important if the refresh token can only used once.

  • $parameters object

    Set parameters that are inherited to all children. See the section in the advanced topics for more details.

DeclarativeStream object​

A stream whose behavior is described by a set of declarative low code components.

Properties:

DefaultErrorHandler object​

Component defining how to handle errors. Default behavior includes only retrying server errors (HTTP 5XX) and too many requests (HTTP 429) with an exponential backoff.

Properties:

  • backoff_strategies array

    List of backoff strategies to use to determine how long to wait before retrying a retryable request.

  • max_retries integer

    The maximum number of time to retry a retryable request before giving up and failing.

    Examples:
    5
    0
    10

  • response_filters array #/definitions/HttpResponseFilter

    List of response filters to iterate on when deciding how to handle an error. When using an array of multiple filters, the filters will be applied sequentially and the response will be selected if it matches any of the filter's predicate.

  • $parameters object

    Set parameters that are inherited to all children. See the section in the advanced topics for more details.

DefaultPaginator object​

Default pagination implementation to request pages of results with a fixed size until the pagination strategy no longer returns a next_page_token.

Properties:

DpathExtractor object​

Record extractor that searches a decoded response over a path defined as an array of fields.

Properties:

  • field_path array

    List of potentially nested fields describing the full path of the field to extract. Use "*" to extract all values from an array. See more info in the docs.

    Examples:
    [
  "data"
]
    [
  "data",
  "records"
]
    [
  "data",
  "{{ parameters.name }}"
]
    [
  "data",
  "*",
  "record"
]

  • decoder #/definitions/JsonDecoder

    Component decoding the response so records can be extracted.

  • $parameters object

    Set parameters that are inherited to all children. See the section in the advanced topics for more details.

ExponentialBackoffStrategy object​

Backoff strategy with an exponential backoff interval. The interval is defined as factor * 2^attempt_count.

Properties:

  • factor

    Multiplicative constant applied on each retry.

    Type:
    • number
    • string

    Available variables:
    Examples:
    5
    5.5
    "10"

  • $parameters object

    Set parameters that are inherited to all children. See the section in the advanced topics for more details.

SessionTokenAuthenticator object​

Properties:

  • login_requester #/definitions/HttpRequester

    Description of the request to perform to obtain a session token to perform data requests. The response body is expected to be a JSON object with a session token property.

    Example:
    {
  "type": "HttpRequester",
  "url_base": "https://my_api.com",
  "path": "/login",
  "authenticator": {
    "type": "BasicHttpAuthenticator",
    "username": "{{ config.username }}",
    "password": "{{ config.password }}"
  }
}

  • session_token_path array

    The path in the response body returned from the login requester to the session token.

    Examples:
    [
  "access_token"
]
    [
  "result",
  "token"
]

  • expiration_duration string

    The duration in ISO 8601 duration notation after which the session token expires, starting from the time it was obtained. Omitting it will result in the session token being refreshed for every request.

    Examples:
    "PT1H"
    "P1D"

  • request_authentication

    Authentication method to use for requests sent to the API, specifying how to inject the session token.

    Type:

  • $parameters object

    Set parameters that are inherited to all children. See the section in the advanced topics for more details.

SessionTokenRequestApiKeyAuthenticator object​

Authenticator for requests using the session token as an API key that's injected into the request.

Properties:

  • inject_into #/definitions/RequestOption

    Configure how the API Key will be sent in requests to the source API.

    Examples:
    {
  "inject_into": "header",
  "field_name": "Authorization"
}
    {
  "inject_into": "request_parameter",
  "field_name": "authKey"
}

SessionTokenRequestBearerAuthenticator​

Authenticator for requests using the session token as a standard bearer token.

Properties:

    HttpRequester object​

    Requester submitting HTTP requests and extracting records from the response.

    Properties:

    HttpResponseFilter object​

    A filter that is used to select on properties of the HTTP response received. When used with additional filters, a response will be selected if it matches any of the filter's criteria.

    Properties:

    • action string

      Action to execute if a response matches the filter.

      Examples:
      "SUCCESS"
      "FAIL"
      "RETRY"
      "IGNORE"

    • error_message string

      Error Message to display if the response matches the filter.

      Available variables:

    • error_message_contains string

      Match the response if its error message contains the substring.

    • http_codes array

      Match the response if its HTTP code is included in this list.

      Examples:
      [
  420,
  429
]
      [
  500
]

    • predicate string

      Match the response if the predicate evaluates to true.

      Available variables:
      Examples:
      "{{ 'Too much requests' in response }}"
      "{{ 'error_code' in response and response['error_code'] == 'ComplexityException' }}"

    • $parameters object

      Set parameters that are inherited to all children. See the section in the advanced topics for more details.

    InlineSchemaLoader object​

    Loads a schema that is defined directly in the manifest file.

    Properties:

    • schema object

      Describes a streams' schema. Refer to the <a href="https://docs.airbyte.com/understanding-airbyte/supported-data-types/">Data Types documentation</a> for more details on which types are valid.

    JsonFileSchemaLoader object​

    Loads the schema from a json file.

    Properties:

    • file_path string

      Path to the JSON file defining the schema. The path is relative to the connector module's root.

      Available variables:

    • $parameters object

      Set parameters that are inherited to all children. See the section in the advanced topics for more details.

    JsonDecoder object​

    Properties:

      ListPartitionRouter object​

      A Partition router that specifies a list of attributes where each attribute describes a portion of the complete data set for a stream. During a sync, each value is iterated over and can be used as input to outbound API requests.

      Properties:

      • cursor_field string

        While iterating over list values, the name of field used to reference a list value. The partition value can be accessed with string interpolation. e.g. "{{ stream_partition['my_key'] }}" where "my_key" is the value of the cursor_field.

        Available variables:
        Examples:
        "section"
        "{{ config['section_key'] }}"

      • values

        The list of attributes being iterated over and used as input for the requests made to the source API.

        Type:
        • string
        • array

        Available variables:
        Examples:
        [
  "section_a",
  "section_b",
  "section_c"
]
        "{{ config['sections'] }}"

      • request_option #/definitions/RequestOption

        A request option describing where the list value should be injected into and under what field name if applicable.

      • $parameters object

        Set parameters that are inherited to all children. See the section in the advanced topics for more details.

      MinMaxDatetime object​

      Compares the provided date against optional minimum or maximum times. The max_datetime serves as the ceiling and will be returned when datetime exceeds it. The min_datetime serves as the floor.

      Properties:

      • datetime string

        Datetime value.

        Available variables:
        Examples:
        "2021-01-01"
        "2021-01-01T00:00:00Z"
        "{{ config['start_time'] }}"

      • datetime_format string

        Format of the datetime value. Defaults to "%Y-%m-%dT%H:%M:%S.%f%z" if left empty. Use placeholders starting with "%" to describe the format the API is using. The following placeholders are available:

        • %s: Epoch unix timestamp - 1686218963
        • %ms: Epoch unix timestamp - 1686218963123
        • %a: Weekday (abbreviated) - Sun
        • %A: Weekday (full) - Sunday
        • %w: Weekday (decimal) - 0 (Sunday), 6 (Saturday)
        • %d: Day of the month (zero-padded) - 01, 02, ..., 31
        • %b: Month (abbreviated) - Jan
        • %B: Month (full) - January
        • %m: Month (zero-padded) - 01, 02, ..., 12
        • %y: Year (without century, zero-padded) - 00, 01, ..., 99
        • %Y: Year (with century) - 0001, 0002, ..., 9999
        • %H: Hour (24-hour, zero-padded) - 00, 01, ..., 23
        • %I: Hour (12-hour, zero-padded) - 01, 02, ..., 12
        • %p: AM/PM indicator
        • %M: Minute (zero-padded) - 00, 01, ..., 59
        • %S: Second (zero-padded) - 00, 01, ..., 59
        • %f: Microsecond (zero-padded to 6 digits) - 000000, 000001, ..., 999999
        • %z: UTC offset - (empty), +0000, -04:00
        • %Z: Time zone name - (empty), UTC, GMT
        • %j: Day of the year (zero-padded) - 001, 002, ..., 366
        • %U: Week number of the year (Sunday as first day) - 00, 01, ..., 53
        • %W: Week number of the year (Monday as first day) - 00, 01, ..., 53
        • %c: Date and time representation - Tue Aug 16 21:30:00 1988
        • %x: Date representation - 08/16/1988
        • %X: Time representation - 21:30:00
        • %%: Literal '%' character

        Some placeholders depend on the locale of the underlying system - in most cases this locale is configured as en/US. For more information see the Python documentation.

        Examples:
        "%Y-%m-%dT%H:%M:%S.%f%z"
        "%Y-%m-%d"
        "%s"

      • max_datetime string

        Ceiling applied on the datetime value. Must be formatted with the datetime_format field.

        Available variables:
        Examples:
        "2021-01-01T00:00:00Z"
        "2021-01-01"

      • min_datetime string

        Floor applied on the datetime value. Must be formatted with the datetime_format field.

        Available variables:
        Examples:
        "2010-01-01T00:00:00Z"
        "2010-01-01"

      • $parameters object

        Set parameters that are inherited to all children. See the section in the advanced topics for more details.

      NoAuth object​

      Authenticator for requests requiring no authentication.

      Properties:

      NoPagination object​

      Pagination implementation that never returns a next page.

      Properties:

        OAuthConfigSpecification object​

        Specification describing how an 'advanced' Auth flow would need to function.

        Properties:

        • oauth_user_input_from_connector_config_specification object

          OAuth specific blob. This is a Json Schema used to validate Json configurations used as input to OAuth. Must be a valid non-nested JSON that refers to properties from ConnectorSpecification.connectionSpecification using special annotation 'path_in_connector_config'. These are input values the user is entering through the UI to authenticate to the connector, that might also shared as inputs for syncing data via the connector. Examples: if no connector values is shared during oauth flow, oauth_user_input_from_connector_config_specification=[] if connector values such as 'app_id' inside the top level are used to generate the API url for the oauth flow, oauth_user_input_from_connector_config_specification={ app_id: { type: string path_in_connector_config: ['app_id'] } } if connector values such as 'info.app_id' nested inside another object are used to generate the API url for the oauth flow, oauth_user_input_from_connector_config_specification={ app_id: { type: string path_in_connector_config: ['info', 'app_id'] } }

          Examples:
          {
  "app_id": {
    "type": "string",
    "path_in_connector_config": [
      "app_id"
    ]
  }
}
          {
  "app_id": {
    "type": "string",
    "path_in_connector_config": [
      "info",
      "app_id"
    ]
  }
}

        • complete_oauth_output_specification object

          OAuth specific blob. This is a Json Schema used to validate Json configurations produced by the OAuth flows as they are returned by the distant OAuth APIs. Must be a valid JSON describing the fields to merge back to ConnectorSpecification.connectionSpecification. For each field, a special annotation path_in_connector_config can be specified to determine where to merge it, Examples: complete_oauth_output_specification={ refresh_token: { type: string, path_in_connector_config: ['credentials', 'refresh_token'] } }

          Example:
          {
  "refresh_token": {
    "type": "string,",
    "path_in_connector_config": [
      "credentials",
      "refresh_token"
    ]
  }
}

        • complete_oauth_server_input_specification object

          OAuth specific blob. This is a Json Schema used to validate Json configurations persisted as Airbyte Server configurations. Must be a valid non-nested JSON describing additional fields configured by the Airbyte Instance or Workspace Admins to be used by the server when completing an OAuth flow (typically exchanging an auth code for refresh token). Examples: complete_oauth_server_input_specification={ client_id: { type: string }, client_secret: { type: string } }

          Example:
          {
  "client_id": {
    "type": "string"
  },
  "client_secret": {
    "type": "string"
  }
}

        • complete_oauth_server_output_specification object

          OAuth specific blob. This is a Json Schema used to validate Json configurations persisted as Airbyte Server configurations that also need to be merged back into the connector configuration at runtime. This is a subset configuration of complete_oauth_server_input_specification that filters fields out to retain only the ones that are necessary for the connector to function with OAuth. (some fields could be used during oauth flows but not needed afterwards, therefore they would be listed in the complete_oauth_server_input_specification but not complete_oauth_server_output_specification) Must be a valid non-nested JSON describing additional fields configured by the Airbyte Instance or Workspace Admins to be used by the connector when using OAuth flow APIs. These fields are to be merged back to ConnectorSpecification.connectionSpecification. For each field, a special annotation path_in_connector_config can be specified to determine where to merge it, Examples: complete_oauth_server_output_specification={ client_id: { type: string, path_in_connector_config: ['credentials', 'client_id'] }, client_secret: { type: string, path_in_connector_config: ['credentials', 'client_secret'] } }

          Example:
          {
  "client_id": {
    "type": "string,",
    "path_in_connector_config": [
      "credentials",
      "client_id"
    ]
  },
  "client_secret": {
    "type": "string,",
    "path_in_connector_config": [
      "credentials",
      "client_secret"
    ]
  }
}

        OffsetIncrement object​

        Pagination strategy that returns the number of records reads so far and returns it as the next page token.

        Properties:

        • page_size

          The number of records to include in each pages.

          Type:
          • integer
          • string

          Available variables:
          Examples:
          100
          "{{ config['page_size'] }}"

        • inject_on_first_request boolean

          Using the offset with value 0 during the first request

        • $parameters object

          Set parameters that are inherited to all children. See the section in the advanced topics for more details.

        PageIncrement object​

        Pagination strategy that returns the number of pages reads so far and returns it as the next page token.

        Properties:

        • page_size

          The number of records to include in each pages.

          Type:
          • integer
          • string

          Available variables:
          Examples:
          100
          "100"
          "{{ config['page_size'] }}"

        • start_from_page integer

          Index of the first page to request.

          Examples:
          0
          1

        • inject_on_first_request boolean

          Using the page number with value defined by start_from_page during the first request

        • $parameters object

          Set parameters that are inherited to all children. See the section in the advanced topics for more details.

        ParentStreamConfig object​

        Describes how to construct partitions from the records retrieved from the parent stream..

        Properties:

        • parent_key string

          The primary key of records from the parent stream that will be used during the retrieval of records for the current substream. This parent identifier field is typically a characteristic of the child records being extracted from the source API.

          Examples:
          "id"
          "{{ config['parent_record_id'] }}"

        • stream #/definitions/DeclarativeStream

          Reference to the parent stream.

        • partition_field string

          While iterating over parent records during a sync, the parent_key value can be referenced by using this field.

          Examples:
          "parent_id"
          "{{ config['parent_partition_field'] }}"

        • request_option #/definitions/RequestOption

          A request option describing where the parent key value should be injected into and under what field name if applicable.

        • $parameters object

          Set parameters that are inherited to all children. See the section in the advanced topics for more details.

        PrimaryKey​

        The stream field to be used to distinguish unique records. Can either be a single field, an array of fields representing a composite key, or an array of arrays representing a composite key where the fields are nested fields.

        Examples:
        "id"
        [
  "code",
  "type"
]

        RecordFilter object​

        Filter applied on a list of records.

        Properties:

        RecordSelector object​

        Responsible for translating an HTTP response into a list of records by extracting records from the response and optionally filtering records based on a heuristic.

        Properties:

        SchemaNormalization string​

        Responsible for normalization according to the schema.

        Examples:
        "None"
        "Default"

        RemoveFields object​

        A transformation which removes fields from a record. The fields removed are designated using FieldPointers. During transformation, if a field or any of its parents does not exist in the record, no error is thrown.

        Properties:

        • condition string

          The predicate to filter a property by a property value. Property will be removed if it is empty OR expression is evaluated to True.,

          Available variables:
          Examples:
          "{{ property|string == '' }}"
          "{{ property is integer }}"
          "{{ property|length > 5 }}"
          "{{ property == 'some_string_to_match' }}"

        • field_pointers array

          Array of paths defining the field to remove. Each item is an array whose field describe the path of a field to remove.

          Examples:
          [
  "tags"
]
          [
  [
    "content",
    "html"
  ],
  [
    "content",
    "plain_text"
  ]
]

        RequestPath object​

        Specifies where in the request path a component's value should be inserted.

        Properties:

          RequestOption object​

          Specifies the key field and where in the request a component's value should be injected.

          Properties:

          • field_name string

            Configures which key should be used in the location that the descriptor is being injected into

            Available variables:
            Example:
            "segment_id"

          • inject_into

            Configures where the descriptor should be set on the HTTP requests. Note that request parameters that are already encoded in the URL path will not be duplicated.

            Examples:
            "request_parameter"
            "header"
            "body_data"
            "body_json"

          Schemas object​

          The stream schemas representing the shape of the data emitted by the stream.

          LegacySessionTokenAuthenticator object​

          Deprecated - use SessionTokenAuthenticator instead. Authenticator for requests authenticated using session tokens. A session token is a random value generated by a server to identify a specific user for the duration of one interaction session.

          Properties:

          • header string

            The name of the session token header that will be injected in the request

            Example:
            "X-Session"

          • login_url string

            Path of the login URL (do not include the base URL)

            Example:
            "session"

          • session_token string

            Session token to use if using a pre-defined token. Not needed if authenticating with username + password pair

          • session_token_response_key string

            Name of the key of the session token to be extracted from the response

            Example:
            "id"

          • username string

            Username used to authenticate and obtain a session token

            Example:
            " {{ config['username'] }}"

          • password string

            Password used to authenticate and obtain a session token

            Examples:
            "{{ config['password'] }}"
            ""

          • validate_session_url string

            Path of the URL to use to validate that the session token is valid (do not include the base URL)

            Example:
            "user/current"

          • $parameters object

            Set parameters that are inherited to all children. See the section in the advanced topics for more details.

          SimpleRetriever object​

          Retrieves records by synchronously sending requests to fetch records. The retriever acts as an orchestrator between the requester, the record selector, the paginator, and the partition router.

          Properties:

          Spec object​

          A source specification made up of connector metadata and how it can be configured.

          Properties:

          • connection_specification object

            A connection specification describing how a the connector can be configured.

          • documentation_url string

            URL of the connector's documentation page.

            Example:
            "https://docs.airbyte.com/integrations/sources/dremio"

          • advanced_auth #/definitions/AuthFlow

            Advanced specification for configuring the authentication flow.

          SubstreamPartitionRouter object​

          Partition router that is used to retrieve records that have been partitioned according to records from the specified parent streams. An example of a parent stream is automobile brands and the substream would be the various car models associated with each branch.

          Properties:

          • parent_stream_configs array #/definitions/ParentStreamConfig

            Specifies which parent streams are being iterated over and how parent records should be used to partition the child stream data set.

          • $parameters object

            Set parameters that are inherited to all children. See the section in the advanced topics for more details.

          ValueType string​

          A schema type.

          WaitTimeFromHeader object​

          Extract wait time from a HTTP header in the response.

          Properties:

          • header string

            The name of the response header defining how long to wait before retrying.

            Available variables:
            Example:
            "Retry-After"

          • regex string

            Optional regex to apply on the header to extract its value. The regex should define a capture group defining the wait time.

            Example:
            "([-+]?\\d+)"

          • $parameters object

            Set parameters that are inherited to all children. See the section in the advanced topics for more details.

          WaitUntilTimeFromHeader object​

          Extract time at which we can retry the request from response header and wait for the difference between now and that time.

          Properties:

          • header string

            The name of the response header defining how long to wait before retrying.

            Available variables:
            Example:
            "wait_time"

          • min_wait

            Minimum time to wait before retrying.

            Type:
            • number
            • string

            Available variables:
            Examples:
            10
            "60"

          • regex string

            Optional regex to apply on the header to extract its value. The regex should define a capture group defining the wait time.

            Available variables:
            Example:
            "([-+]?\\d+)"

          • $parameters object

            Set parameters that are inherited to all children. See the section in the advanced topics for more details.

          Interpolation variables​

          All string properties that list out available variables allow jinja expressions. They can be used by placing them in double curly braces: {{ config.property }}. The following variables are available

          config object​

          The connector configuration. The object's keys are the same as the the keys defined in the connection specification.

          Example:
          {
  "start_date": "2010-01-01",
  "api_key": "*****"
}

          parameters object​

          Additional runtime parameters, to be used for string interpolation. Parameters can be passed down from a parent component to its subcomponents using the $parameters key. This can be used to avoid repetitions.

          Example:
          {
  "path": "automations",
  "data_export_path": "automations",
  "cursor_field": "updated_at"
}

          headers object​

          The HTTP headers from the last response received from the API. The object's keys are the header names from the response.

          Example:
          {
  "Server": "nginx",
  "Date": "Mon, 24 Apr 2023 20:17:21 GMT",
  "Content-Type": "application/json",
  "Content-Length": "420",
  "Connection": "keep-alive",
  "referrer-policy": "strict-origin-when-cross-origin",
  "x-content-type-options": "nosniff",
  "x-ratelimit-limit": "600",
  "x-ratelimit-remaining": "598",
  "x-ratelimit-reset": "39"
}

          last_records list​

          List of records extracted from the last response received from the API.

          Example:
          [
  {
    "name": "Test List: 19",
    "id": "0236d6d2",
    "contact_count": 20,
    "_metadata": {
      "self": "https://api.sendgrid.com/v3/marketing/lists/0236d6d2"
    }
  },
  {
    "name": "List for CI tests, number 30",
    "id": "041ee031",
    "contact_count": 0,
    "_metadata": {
      "self": "https://api.sendgrid.com/v3/marketing/lists/041ee031"
    }
  }
]

          next_page_token object​

          Object describing the token to fetch the next page of records. The object has a single key "next_page_token".

          Examples:
          {
  "next_page_token": 3
}
          {
  "next_page_token": "https://api.sendgrid.com/v3/marketing/lists/0236d6d2-75d2-42c5-962d-603e0deaf8d1"
}

          record object​

          The record being processed. The object's keys are the same keys as the records produced by the RecordSelector.

          response object​

          The body of the last response received from the API. The object's keys are the same keys as the response body's.

          Example:
          {
  "result": [
    {
      "name": "Test List: 19",
      "id": "0236d6d2-75d2-42c5-962d-603e0deaf8d1",
      "contact_count": 20,
      "_metadata": {
        "self": "https://api.sendgrid.com/v3/marketing/lists/0236d6d2"
      }
    }
  ],
  "_metadata": {
    "self": "https://api.sendgrid.com/v3/marketing/lists?page_size=1&page_token=",
    "next": "https://api.sendgrid.com/v3/marketing/lists?page_size=1&page_token=0236d6d2",
    "count": 82
  }
}

          stream_interval object​

          The current stream interval being processed. The keys are defined by the incremental sync component. Default keys are start_time and end_time.

          Example:
          {
  "start_time": "2020-01-01 00:00:00.000+00:00",
  "end_time": "2020-01-02 00:00:00.000+00:00"
}

          stream_partition object​

          The current stream partition being processed. The keys are defined by the partition router component.

          Examples:
          {
  "survey_id": 1234
}
          {
  "strategy": "DESKTOP"
}
          {
  "survey_id": 1234,
  "strategy": "MOBILE"
}

          stream_slice object​

          This variable is deprecated. Use stream_interval or stream_partition instead.

          stream_state object​

          The current state of the stream. The object's keys are defined by the incremental sync's cursor_field the and partition router's values.

          Examples:
          {
  "created_at": "2020-01-01 00:00:00.000+00:00"
}
          {
  "updated_at": "2020-01-02 00:00:00.000+00:00"
}

          Interpolation macros​

          Besides referencing variables, the following macros can be called as part of jinja expressions, for example like this: {{ now_utc() }}.

          Now (UTC)​

          Returns the current date and time in the UTC timezone.

          Examples:
          "'{{ now_utc() }}' -> '2021-09-01 00:00:00+00:00'"
          "'{{ now_utc().strftime('%Y-%m-%d') }}' -> '2021-09-01'"

          Today (UTC)​

          Returns the current date in UTC timezone. The output is a date object.

          Examples:
          "'{{ today_utc() }}' -> '2021-09-01'"
          "'{{ today_utc().strftime('%Y/%m/%d')}}' -> '2021/09/01'"

          Timestamp​

          Converts a number or a string representing a datetime (formatted as ISO8601) to a timestamp. If the input is a number, it is converted to an int. If no timezone is specified, the string is interpreted as UTC.

          Arguments:
          • datetime: A string formatted as ISO8601 or an integer representing a unix timestamp
          Examples:
          "'{{ timestamp(1646006400) }}' -> 1646006400"
          "'{{ timestamp('2022-02-28') }}' -> 1646006400"
          "'{{ timestamp('2022-02-28T00:00:00Z') }}' -> 1646006400"
          "'{{ timestamp('2022-02-28 00:00:00Z') }}' -> 1646006400"
          "'{{ timestamp('2022-02-28T00:00:00-08:00') }}' -> 1646035200"

          Max​

          Returns the largest object of a iterable, or or two or more arguments.

          Arguments:
          • args: iterable or a sequence of two or more arguments
          Examples:
          "'{{ max(2, 3) }}' -> 3"
          "'{{ max([2, 3]) }}' -> 3"

          Day Delta​

          Returns the datetime of now() + num_days.

          Arguments:
          • num_days: The number of days to add to now
          • format: How to format the output string
          Examples:
          "'{{ day_delta(1) }}' -> '2021-09-02T00:00:00.000000+0000'"
          "'{{ day_delta(-1) }}' -> '2021-08-31:00:00.000000+0000'"
          "'{{ day_delta(25, format='%Y-%m-%d') }}' -> '2021-09-02'"

          Duration​

          Converts an ISO8601 duratioin to datetime.timedelta.

          Arguments:
          • duration_string: A string representing an ISO8601 duration. See https://www.digi.com/resources/documentation/digidocs//90001488-13/reference/r_iso_8601_duration_format.htm for more details.
          Examples:
          "'{{ duration('P1D') }}' -> '1 day, 0:00:00'"
          "'{{ duration('P6DT23H') }}' -> '6 days, 23:00:00'"
          "'{{ (now_utc() - duration('P1D')).strftime('%Y-%m-%dT%H:%M:%SZ') }}' -> '2021-08-31T00:00:00Z'"

          Format Datetime​

          Converts a datetime or a datetime-string to the specified format.

          Arguments:
          • datetime: The datetime object or a string to convert. If datetime is a string, it must be formatted as ISO8601.
          • format: The datetime format
          Examples:
          "{{ format_datetime(config['start_time'], '%Y-%m-%d') }}"
          "{{ format_datetime(config['start_date'], '%Y-%m-%dT%H:%M:%S.%fZ') }}"
          Was this page helpful?