Target Data Models (TDMs)

Target data models define the structure of the output data in all nuvo pipelines. They ensure that the mapped and transformed data is organized according to the desired format and schema required by the destination system.

You can create, read (all/by id), update and delete target data models using:

the target data model API
the target data model embeddables (coming soon)
the nuvo user platform (coming soon)

Within a target data model, you can define the name of each column at both a UI-based level (label) and a technical level (key). Additionally, you can apply required, unique, and regex validations, as well as specify the column type (e.g., float, int, date, currency, etc.)

This ensures that the pipeline’s output not only has the correct structure but also that the values are in the correct format.

For example, if you want the output to contain objects with the keys customer_name, domain_name, region, deal_size, address, and done, you would define the target data model as follows:

Example

[
  {
    label: "Customer Name",
    key: "customer_name",
    columnType: "string"
  },
  {
    label: "Domain Name",
    key: "domain_name",
    columnType: "url"
  },
  {
    label: "Region",
    key: "region",
    columnType: "string"
  },
  {
    label: "Deal Size",
    key: "deal_size",
    columnType: "float"
  },
  {
    label: "Address",
    key: "address",
    columnType: "string"
  },
  {
    label: "Done",
    key: "done",
    columnType: "boolean"
  }
]

How to create a TDM with the nuvo pipeline API?

The nuvo pipeline API provides a convenient way to create a target data model (TDM) for importing data via a pipeline.

Learn how to authenticate yourself with our authentication API and then use the target data model API to create your TDMs.

After setting up the general structure of your target data model, we recommend you to add more validation rules so that you can ensure that the user-submitted data is not only in the correct schema but also the values are in the preferred format.

The column class includes several properties. In the following, all properties of the column class are displayed, including its default value, data type and description.

label (required)


Type	`string`
Description	`label` is one of two required properties of a column object. The value is displayed within the UI to the user, who goes through the importing workflow. This value and the `key` value is used for the column matching.

key (required)


Type	`string`
Description	The value of `key` defines how your application calls a column. This value is not displayed to your users, but like the label, it is used for column matching.

alternativeMatches

Type [ string, ... ]

Description nuvo utilizes machine learning with a logic-based component to provide an intelligent and advanced mapping functionality. To be able to force a more precise matching, we offer an additional matching layer with the alternativeMatches property. This should help match the target and imported columns more accurately because every item of the alternativeMatches array is considered for calculating the similarity between imported and target model columns. We recommend adding abbreviations and synonyms here.


Type	`[ string, ... ]`
Description	nuvo utilizes machine learning with a logic-based component to provide an intelligent and advanced mapping functionality. To be able to force a more precise matching, we offer an additional matching layer with the `alternativeMatches` property. This should help match the target and imported columns more accurately because every item of the `alternativeMatches` array is considered for calculating the similarity between imported and target model columns. We recommend adding abbreviations and synonyms here.

mappingValidation


Type	`[ MappingValidation, ... ]`
Description	With the mapping validation option, you can define the conditions for when a column is required to be mapped. You can also add custom error messages to guide users on which columns need to be mapped and when. The `mappingValidation` object can contain two keys:

logic
errorMessage

logic


Type	`string`
Description	This option enables you create complex conditional validation rules using logical operators and the `mapped()` helper function. See the validation rules below for details.

errorMessage


Type	`string`
Description	This option lets you add a fully customized error message to a validation rule, which appears at the column level. It replaces the default message.

dataValidations


Type	`[ DataValidation, ... ]`
Description	With the data validations option, you can define the preferred format for the values in one column as well as whether the values are required or have to be unqiue. You can also add customized error messages to guide your users on how to correct their input. Each object in the `dataValidations` array can contain two keys:

logic
errorMessage

logic


Type	`string`
Description	With this option, you can write complex, conditional validation rules with logical operators and our helper functions (`unique`, `regex`, `includes`). This property offers you the possibility to ensure the highest-possible data quality (see Validation rules below for more details)

errorMessage


Type	`string`
Description	With this option, you can add an fully customized error message to each validation rule. This notification text overwrittes the pre-built error message.

advancedValidations


Type	`[ AdvancedValidation, ... ]`
Description	With the advanced validations option, you can specify the endpoints for validating imported data. You have full control over which columns are sent for validation and which ones trigger the validation process. You can also add customized error messages to guide your users on how to correct their input. Each object in the `advancedValidtions` array can contain two keys:

url
method
headers
authentication
payloadConfiguration
triggerConfiguration

url


Type	`string`
Description	With this option, you can define the endpoint that the data is sent to for validation.

method


Type	`string`
Description	Define the REST API method that should be used when sending the data to the defined endpoint. You can set it to "POST", "PUT" or "PATCH".

headers


Type	`object`
Description	With this option, you can define the REST API headers of the request sent to the defined endpoint.

authentication


Type	`object`
Description	With this option, you can add the authentication endpoint the is used to retrieve the required authentication information that is sent with the request to the defined endpoint. The object contains the following fields: `refresh_url`, `method`, and `headers`.

payloadConfiguration

Type object

Description With this option, you can control which columns are sent to your endpoint and how the data is batched. Use the columns field to specify which columns to include in the request payload. If you leave this field empty, all columns will be sent to the endpoint. Additionally, you can define the batchSize (default 25.000 rows) that determines how many rows are sent in a single request. If your total number of rows exceeds the defined batch size, the system will automatically split the data and send multiple requests to your endpoint.

triggerConfiguration

Type object

Description With this option, you can define which columns trigger sending the data to the defined endpoint. With the columns field, you can specify which columns should trigger sending data to your endpoint. Add column names to the columns list to define these trigger columns. When a change occurs in any of these specified columns, the data will be sent to your endpoint. If you leave this list empty, the system will send data for every change in any row or column.

advancedCleanings


Type	`[ AdvancedCleaning, ... ]`
Description	With the advanced cleanings option, you can specify the endpoints for cleaning imported data. You have full control over which columns are sent for cleaning and which ones trigger the cleaning process. Your users are then informed which values were changed through those cleanings. Each object in the `advancedCleanings` array can contain two keys:

url
method
headers
authentication
payloadConfiguration
triggerConfiguration

url


Type	`string`
Description	With this option, you can define the endpoint that the data is sent to for cleaning.

method


Type	`string`
Description	Define the REST API method that should be used when sending the data to the defined endpoint. You can set it to "POST", "PUT" or "PATCH".

headers


Type	`object`
Description	With this option, you can define the REST API headers of the request sent to the defined endpoint.

authentication


Type	`object`
Description	With this option, you can add the authentication endpoint the is used to retrieve the required authentication information that is sent with the request to the defined endpoint. The object contains the following fields: `refresh_url`, `method`, and `headers`.

payloadConfiguration

Type object

triggerConfiguration

Type object

Validation rules

Mapping Validation

Mapping validations help to ensure data integrity by letting you define when a column must be mapped.

You can define conditions using logical operators like AND (&&) and OR (||) to create complex validation rules with nested conditions. To create a validation rule, write an expression representing the valid case. If the logic returns false, the defined error message will be displayed.

Here are some examples of mapping validations with different levels of complexity and how to add them into your column definitions:

Column A needs to be mapped

[
  {
    key: "column_a",
    label: "Column A",
    columnType: "string",
    mappingValidation: {
        logic: "mapped('column_a')",
        errorMessage: "Column A needs to be mapped."
      },
    ...
  }
]

Column A needs to be mapped when column B is not mapped

[
  {
    key: "column_a",
    label: "Column A",
    columnType: "string",
    mappingValidation: {
        logic: "mapped('column_b')",
        errorMessage: "Column A needs to be mapped when column B is not mapped."
      },
    ...
  }
]

Column A needs to be mapped when column B is mapped and column C is not mapped

[
  {
    key: "column_a",
    label: "Column A",
    columnType: "string",
    mappingValidation: {
        logic: "mapped('column_a') || !mapped('column_b') || mapped('column_c')",
        errorMessage: "Column A needs to be mapped when column B is mapped and column C is not mapped"
      },
    ...
  }
]

Either first name & last name or full name needs to be mapped

[
  {
    key: "first_name",
    label: "First name",
    columnType: "string",
    mappingValidation: {
        logic: "mapped('first_name') || mapped('full_name')",
        errorMessage: "Either first name & last name or full name needs to be mapped"
      },
    ...
  },
  {
    key: "last_name",
    label: "Last name",
    columnType: "string",
    mappingValidation: {
        logic: "mapped('last_name') || mapped('full_name')",
        errorMessage: "Either first name & last name or full name needs to be mapped"
      },
    ...
  },
  {
    key: "full_name",
    label: "Full name",
    columnType: "string",
    mappingValidation: {
        logic: "mapped('first_name') || mapped('last_name') || mapped('full_name')",
        errorMessage: "Either first name & last name or full name needs to be mapped"
      },
    ...
  }
]

Either customerID or companyID or employeeID has to be mapped

[
  {
    key: "company_id",
    label: "Company ID",
    columnType: "string",
    mappingValidation: {
        logic: "mapped('company_id') || mapped('customer_id') || mapped('employee_id')",
        errorMessage: "Either customerID or companyID or employeeID has to be mapped"
      },
    ...
  },
  {
    key: "customer_id",
    label: "Customer ID",
    columnType: "string",
    mappingValidation: {
        logic: "mapped('company_id') || mapped('customer_id') || mapped('employee_id')",
        errorMessage: "Either customerID or companyID or employeeID has to be mapped"
      },
    ...
  },
  {
    key: "employee_id",
    label: "Employee ID",
    columnType: "string",
    mappingValidation: {
        logic: "mapped('company_id') || mapped('customer_id') || mapped('employee_id')",
        errorMessage: "Either customerID or companyID or employeeID has to be mapped"
      },
    ...
  }
]

Data Validations

Data validations are a powerful tool for ensuring data integrity within your columns. They allow you to specify criteria that a value must meet to be considered valid, and you can add multiple validations to a each column. You can define conditions using logical operators like AND (&&) and OR (||) to create complex validation rules with nested conditions. To create a validation rule, write an expression representing the valid case. If the logic returns false, the defined error message will be displayed.

We also provide helper functions that you can use in conjunction with your validation expressions to create more sophisticated rules:

unique(['<column_key>', ...])
regex(row.<column_key>, { expression: '' })
includes(['<value>', ...], <column_key>)
isEmpty('<column_key>')

Here are some examples of data validations with different levels of complexity and how to add them into your column definitions:

Each entry in column_a requires a value

[
  {
    key: "column_a",
    label: "Column A",
    columnType: "string",
    dataValidations: [
      {
        logic: "row.column_a !== ''",
        errorMessage: "Column A is required."
      }
    ],
    ...
  }
]

Each entry in column_a requires a value if the entry in column_b is empty

[
  {
    key: "column_a",
    label: "Column A",
    columnType: "string",
    dataValidations: [
      {
        logic: "row.column_a !== '' || row.column_b !== ''",
        errorMessage: "Column A is required when Column B is empty."
      }
    ],
    ...
  }
]

Either customer_id, company_id, or employee_id have to have a value

[
  {
    key: "customer_id",
    label: "Customer ID",
    columnType: "string",
    dataValidations: [
      {
        logic: "row.customerID !== '' || row.companyID !== '' || row.employeeID !== ''",
        errorMessage: "At least one of Customer ID, Company ID, or Employee ID must be provided."
      }
    ],
    ...
  }
]

Each entry in column_a requires a value if the entry in column_b is empty AND the entry in column_c has a value

[
  {
    key: "column_a",
    label: "Column A",
    columnType: "string",
    dataValidations: [
      {
        logic: "row.column_b !== '' || row.column_c === '' || row.column_a !== ''",
        errorMessage: "Column A is required when Column B is empty and Column C has a value."
      }
    ],
    ...
  }
]

Each entry in column_a requires a number between 100 and 99999

[
  {
    key: "column_a",
    label: "Column A",
    columnType: "string",
    dataValidations: [
      {
        logic: "regex(row.column_a, { expression: '^\\d{3,5}$' })",
        errorMessage: "Column A must be a number with 3 to 5 digits."
      }
    ],
    ...
  }
]

All entries in column_a have to be unique

[
  {
    key: "column_a",
    label: "Column A",
    columnType: "string",
    dataValidations: [
      {
        logic: "unique(['column_a'])",
        errorMessage: "Column A must be unique across all entries."
      }
    ],
    ...
  }
]

All entries across column_a, column_b, and column_c have to be unique

[
  {
    key: "column_a",
    label: "Column A",
    columnType: "string",
    dataValidations: [
      {
        logic: "unique(['column_a', 'column_b', 'column_c'])",
        errorMessage: "The combination of Column A, Column B, and Column C must be unique."
      }
    ],
    ...
  }
]

column_a requires a value if column_b is "Germany" and column_c is "Hamburg"

[
  {
    key: "column_a",
    label: "Column A",
    columnType: "string",
    dataValidations: [
      {
        logic: "!(includes(['Germany', 'Canada'], row.column_b) || row.column_a !== '')",
        errorMessage: "Column A is required when Column B is Germany and Column C is Hamburg."
      }
    ],
    ...
  }
]

Advanced Validations

Advanced validations are a powerful way to ensure data integrity with your system. They allow you to define specific endpoints for validating imported data, giving you control over which columns are sent for validation and which ones trigger the process. To set up an advanced validation rule, simply specify the endpoint, method, and headers. If the validation logic returns false, the designated error message will be displayed.

Example

"advancedValidations": [
  {
    "url": "https://your-endpoint.com",
    "method": "POST",
    "headers": {},
    "authentication":{
      "refresh_url": "https://your-authentication-endpoint.com",
      "method": "POST",
      "headers": {}
    },
    "payloadConfiguration":{
      "columns": ['company_name', 'company_id'],
      "batchSize": 10000
    }
    "triggerConfiguration":{
      "columns": ['company_id', 'company_name']
    }
  }
]

Payload

[
  {
    "data": {
      "company_id": "13132",
      "company_name": "company A"
    },
    "index": 2
  }
  {
    "data": {
      "company_id": "879632",
      "company_name": "company B"
    },
    "index": 3
  }
]

Response

[
  {
    "errors": {
      "company_id": "Key should be unique",
      "company_name": null
    },
    "index": 2
  }
  {
    "errors": {
      "company_id": "Key should be unique",
      "ncompany_name": null
    },
    "index": 3
  }
]

Advanced Cleanings

Advanced cleanings are a powerful way to ensure data integrity with your system. They allow you to define specific endpoints for cleaning imported data, giving you control over which columns are sent for cleaning and which ones trigger the process. To set up an advanced cleaning, simply specify the endpoint, method, and headers.

Example

"advancedCleanings": [
  {
    "url": "https://your-endpoint.com",
    "method": "POST",
    "headers": {},
    "authentication":{
      "refresh_url": "https://your-authentication-endpoint.com",
      "method": "POST",
      "headers": {}
    },
    "payloadConfiguration":{
      "columns": ['company_name', 'company_id'],
      "batchSize": 10000
    }
    "triggerConfiguration":{
      "columns": ['company_id', 'company_name']
    }
  }
]

Payload

[
  {
    "data": {
      "company_id": "13132",
      "company_name": "company A"
    },
    "index": 2
  }
  {
    "data": {
      "company_id": "879632",
      "company_name": "company B"
    },
    "index": 3
  }
]

Response

[
  {
    data: {
      company_id: "12345",
      company_name: null,
    },
    index: 2,
  },
  {
    data: {
      company_id: "098",
    },
    index: 5,
  },
];

columnType


Type	`"int"`, `"float"`, `"string"`, and more
Optional
Description	This option allows you to define the type of the column. You can either choose if the column should contain values which are an int, a float, a string or many more.

info

You can find a full list of column types with pre-build data validation rules in our column types documentation.

dateTimeFormat


Type	`string`
Optional & usable if	`columnType: "date"`
Description	With this key, you can support all your preferred date and timestamp formats such as `MM/DD/YYYY`, `DD.MM.YYYY`, `YYYY-MM-DD`, etc.

Use the following date variables to create your desired format:

Type	Syntax	Output
Month	M	1, 2, 3, ..., 12
Month	Mo	1st, 2nd, 3rd, ..., 12th
Month	MM	01, 02, 03, ..., 12
Month	MMM	Jan, Feb, Mar, ..., Dec
Month	MMMM	January, February, March, ..., December
Day	D	1, 2, 3, ..., 31
Day	Do	1st, 2nd, 3rd, ..., 31st
Day	DD	01, 02, 03, ..., 31
Day	DDD	1, 2, 3, ..., 365
Day	DDDD	001, 002, ..., 365
Year	Y	1970, 1971, 1972, ..., +10000
Year	YY	70, 71, 72, ..., 27
Year	YYYY	1970, 1971, 1972, ...., 2027
Year	YYYYYY	-001970, -001971, -001972, ..., +001907
Hour	H	0, 1, 2, ..., 23
Hour	HH	00, 01, 02, ..., 23
Hour	h	1, 2, 3, ..., 12
Hour	hh	01, 02, 03, ..., 12
Hour	k	1, 2, 3, ..., 24
Hour	kk	01, 02, 03, ..., 24
Minute	m	0, 1, 2, ..., 59
Minute	mm	00, 01, 02, ..., 59
Second	s	0, 1, 2, ..., 59
Second	ss	00, 01, 02, ..., 59
Time zone	Z	-07:00, -06:00, -05:00, ..., +07:00
Time zone	ZZ	-0700, -0600, -0500, ..., +0700
Unix timestamp	X	855599530642
AM/PM	A	AM, PM
AM/PM	a	am, pm
Quarter	Q	1, 2, 3, 4
Quarter	Qo	1st, 2nd, 3rd, 4th

info

This table is based on the original table from the open source library Moment.js. You can find the original table and its documentation here. Please note that the table has been adjusted. You can use all variables given in the original table apart from the Day of Week ones.

In the following, you can find an example of how to implement a date column with the format MM/DD/YYYY and a timestamp column with the format YYYY-MM-DDTHH:mm:ss:

Example

[
  {
    label: "Date",
    key: "date",
    columnType: "date",
    dateTimeFormat: "MM/DD/YYYY",
  },
  {
    label: "Timestamp",
    key: "timestamp",
    columnType: "date",
    dateTimeFormat: "YYYY-MM-DDTHH:mm:ss",
  },
]

numberFormat


Type	`"eu"` \| `"us"`
Default	`"eu"`
Description	This property is applicable for `int`, `float`, `currency_eur`, `currency_usd`, and `percentage` column types. It affects how the numbers will be displayed at the review step. If the value is `"eu"`, then a comma will used as a decimal delimiter, and dots will be used as thousands delimiters. If the value is `"us"`, then a dot will used as a decimal delimiter, and commas will be used as thousands delimiters.

How to create a TDM with the nuvo pipeline API?​

label (required)​

key (required)​

alternativeMatches​

mappingValidation​

logic​

errorMessage​

dataValidations​

logic​

errorMessage​

advancedValidations​

url​

method​

headers​

authentication​

payloadConfiguration​

triggerConfiguration​

advancedCleanings​

url​

method​

headers​

authentication​

payloadConfiguration​

triggerConfiguration​

Validation rules​

Mapping Validation​

Data Validations​

Advanced Validations​

Advanced Cleanings​

columnType​

dateTimeFormat​

numberFormat​

How to create a TDM with the nuvo pipeline API?

label (required)

key (required)

alternativeMatches

mappingValidation

logic

errorMessage

dataValidations

logic

errorMessage

advancedValidations

url

method

headers

authentication

payloadConfiguration

triggerConfiguration

advancedCleanings

url

method

headers

authentication

payloadConfiguration

triggerConfiguration

Validation rules

Mapping Validation

Data Validations

Advanced Validations

Advanced Cleanings

columnType

dateTimeFormat

numberFormat