Skip to main content

Cleaning Functions

Advanced

Concept​

Our cleaning functions serve as callbacks triggered after specific import events, allowing you to provide feedback to the user and transform the imported data. This improves the user efficiency by enabling faster data cleaning at the desired quality without interacting directly with the user.

These functions complement the pre-built validation rules defined in the validations property. With these callbacks, you can implement advanced cleaning and validation logic to address all your use cases. It ranges from data transformation, displaying info, warnings, or customized error messages to users to performing external server requests to retrieve data, comparing this data with what's imported, and flagging duplicates with error messages. Additionally, you can enhance entries, establish dependencies between different columns, and utilize third-party APIs for verification, among other possibilities.

In the following sections, we highlight the contrast between an import process conducted without cleaning functions and an import empowered with a cleaning function. This callback not only reshapes the data but also has the capability to enrich entries using pre-existing data from your data source.

Without cleaning functionsWith cleaning functions
info

To help you get started with the cleaning functions, we provide a comprehensive library within our knowledge base. This library contains pre-written callbacks tailored to frequent data transformation requirements. These functions can be readily adapted to suit your specific use case.

Implementation​

In our cleaning functions, you can utilize JavaScript/TypeScript to its fullest. This includes variables, safe data storage, async server calls, and more.

Each cleaning function can return an array with info and value for each cell. value transforms data for "Review Entries." Using info with level and message, you can display user notifications during import. Notifications can be informational (blue), warning (yellow), or error (red). Errors affect the importing behavior. The entries with at least one error are not included in the result array object of the onResults function by default (You can change that behavior by adjusting completeImportAction). Moreover, you can customize the displayed text of the notification by setting a value for the message key.

For the implementation, it's essential to grasp the available cleaning functions and comprehend their utilization. Below, you'll discover a list providing an overview of our cleaning functions:

columnHooks​

DescriptionWithin this function, you have the ability to retrieve all the data from a specific column. You can implement data modifications and present customized UI notifications, including info messages, warnings, and/or errors
ParametercolumnName: Callback function that must have the identical name as the column to which it will be employed

values: The callback function's parameter consists of a 2d array. Each inner array contains a value at position 1 and an index at position 2
Run EventThe columnHooks function is executed before the user enters the "Review Entries"-step

In the given example, the columnHooks cleaning function is used to restructure phone numbers by eliminating leading zeros and replacing them with a plus sign:

<NuvoImporter
  licenseKey="Your License Key"
  settings={{
    developerMode: true,
    identifier: "customer_data",
    columns: [
      {
        label: "Customer Code",
        key: "customer_code",
        columnType: "string",
      },
      {
        label: "Phone Number",
        key: "phone_number",
        columnType: "string",
      },
    ],
  }}
  columnHooks={{
    phoneNumber: (values) => {
      return values.map(([item, index]) => {
        let phoneNumber = item;
        if (/^[0]{2}/.test(phoneNumber)) {
          phoneNumber = `+${item.slice(2, item.length)}`;
        }
        return [
          {
            value: phoneNumber,
            info: [
              {
                message: 'We have automatically transformed your input into the correct format by converting "00" to "+".',
                level: "info",
              },
            ],
          },
          index,
        ];
      });
    },
  }}
/>

If you’re interested in exploring additional use cases, kindly navigate to our knowledge base available on the user platform.

onEntryInit​

DescriptionThis function iterates through every imported entry. In each iteration, you have access to each row value. With this cleaning function, you can perfectly define column-cross dependencies between two or more columns
Parameterrow: This refers to an object that includes column keys paired with their corresponding values for a given row
  rowIndex: Index of the current iteration
Run EventThis function is executed immediately once after the columnHooks callback

In the following example, we check the Country field of each row after the mapping step. If the value is Germany, we add DE to its Country Code field. Similarly, if the value is Italy, we add IT to the Country Code field of that row:

<NuvoImporter
  licenseKey="Your License Key"
  settings={{
    developerMode: true,
    identifier: "data",
    columns: [
      {
        label: "Country",
        key: "country",
        columnType: "string",
      },
      {
        label: "Country Code",
        key: "country_code",
        columnType: "string",
      },
    ],
  }}
  onEntryInit={(row, rowIndex) => {
    if (row.country === "Germany") {
      return {
        country_code: {
          value: "DE",
        },
      };
    } else if (row.country === "Italy") {
      return {
        country_code: {
          value: "IT",
        },
      };
    }
  }}
/>

For additional use cases, check out our knowledge base available on the user platform.

onEntryChange​

DescriptionThis cleaning function is applied after every action involving changes, creations, or deletions of at least one row. The data for each affected row is available within this function, allowing it to handle information from multiple rows after a single interaction, such as "Replace all", dragging a value via the drag functionality, or copy-pasting.
Parameterrows: An array of row objects containing all rows that either have been changed, deleted or created
Each row object contains:
  • rowIndex: Index of the row
  • data: Object of all column keys with their current values
  • changeLog: Object of key-value pairs including the column key and their old values before the change or deletion. If a row was created, changeLog is null.
  • info: Object of key-value pairs including the column key and an array of objects with all custom infos/warnings/errors of this particular column
  • actionType: The type of change that was performed:
    • "edit": or Manually changing a cell value, copy-pasting or using the drag functionality on an existing cell
    • "delete": Deleting a row
    • "create": Creating a new row
    • "replace": Replacing a cell value via "Find & Replace"
    • "cleaningAssistant": Applying a cleaning suggestion of our Cleaning Assistant on an existing cell
logs: The logs contain details about custom-added options and columns during the mapping and the review step:
  • columns: an object with addedColumns and addedOptions keys. If allowCustomColumns is set to true and new columns are added to the schema, the addedColumns field will be present in the logs and will display an array of added columns including the label, key and column type. If allowCustomOptions is set to true and new options are added to the columns, the addedOptions field will be present in the logs and will display an array of added options with the corresponding column.
Run EventThis function is executed each time a user creates or deletes one or multiple rows as well as each time a user changes one or multiple entries within the "Review Entries"-step. Changing an entry can be done through a manual change, the drag functionality, copy & paste, find & replace and through our cleaning assistant. If the user changes the values of multiple entries simultaneously, the callback function runs only once

If the user changes the age value within the review step in the example given, it is validated again, and an error is displayed accordingly:

<NuvoImporter
  licenseKey="Your License Key"
  settings={{
    developerMode: true,
    identifier: "data",
    columns: [
      {
        label: "ID",
        key: "user_id",
        columnType: "string",
      },
      {
        label: "Name",
        key: "user_name",
        columnType: "string",
      },
      {
        label: "Age",
        key: "age",
        columnType: "int",
      },
    ],
  }}
  onEntryChange={(rows, logs) => {
    return rows
      .filter((row) => Number(row.data["age"]) > 50)
      .map((row) => {
        return {
          rowIndex: row.rowIndex,
          data: {
            age: {
              value: row.data.age,
              info: [
                {
                  message: "Age of the user should not be greater than 50",
                  level: "error",
                },
              ],
            },
          },
        };
      });
  }}
/>

Here is an example of rows if you use copy and paste, where one row is changed, and one row is updated, and here is an example of logs if you add one column and add an additional dropdown option to the column deal_stage during the import process:

[
  {
    "actionType": "edit",
    "changeLog": {
      "user_id": null,
      "full_name": null,
      "age": null
    },
    "info": {
      "user_id": [
        {
          "level": "error",
          "message": "This user already exists in our database."
        }
      ]
    },
    "data": {
      "user_id": "0118",
      "full_name": "Jane Doe",
      "age": "55"
    },
    "rowIndex": 119
  },
  {
    "actionType": "create",
    "changeLog": null,
    "info": {
      "age": [
        {
          "level": "warning",
          "message": "This age is suspiciously high. Please recheck this value."
        }
      ]
    },
    "data": {
      "user_id": "0119",
      "full_name": "Jack Smith",
      "age": "45"
    },
    "rowIndex": 120
  }
]

You can also manipulate the values of other rows that are not part of the onEntryChange function. This can be a quite powerful tool to serve your use case. To see an example, check out our sandbox.

If you want to discover other use cases, please navigate to our knowledge base, which is available on the user platform.