Full Dataset Matching API

Overview

This explains how to leverage our high-performance, parallel-processing cloud architecture to run and/or automate the running of matching jobs for datasets, files, or database tables, all with a single API call. Not only can you identify and matching inconsistent and duplicate data quickly and easily, you can also schedule processing, add to business processes, match & merge multiple datasets, and integrate into ETL/ELT processes.

How It Works

The matching process is initiated via API using an HTTP request "query string", which can be embedded into any process, batch file, scheduler, or scripted series of commands, including from a browser address bar, a command line, using cURL, and any other method that enables an API call.

API Parameters

Use these parameters in your HTTP query string/API call.

Supported Data Sources: Connection Strings

Values to use for the API source and connection parameters

Note: to process and analyze local files, use our browser-based wizard.

Running with cURL Example

You can run the command from a Linux, Windows, or macOS command line using cURL:

Redirecting Output

Output from these curl commands can be redirected to output files for further processing using the greater-than symbol in both Linux & Windows.

Other File Source Examples

Additional examples where a file is the data source that will be used for data matching.

Connecting to Cloud SQL Data Tables

JSON:

There are two primary ways to provide JSON as input to the Full Dataset Matching API. These are a source input file, or alternatively a multi-value supporting batch parameter. in the API call (up to 100 values at a time, no file necessary).

Example JSON input file data for matching.

This example illustrates how input data might look when performing similarity matching using the API with source=json parameter, using JSON structured values from an input file. Use one data column for the data to be matched from and from which similarity keys will be generated. Using the target parameter, output can still be JSON, CSV, TSV, etc.

[
    {
        "Data": "IBM"
    },
    {
        "Data": "International Business Machines"
    },
    {
        "Data": "ibm corp"
    }
]

Example JSON input data for matching with reference data, such as a primary key, that will also appear in output for reference purposes only and not used as part of the matching.

This example illustrates how input data along with a reference parameter might look when performing similarity matching using the API with source=jsonref parameter. Use one data column for the data to be matched from and from which similarity keys will be generated. The reference value will also appear in the output. Using the target parameter, output can still be JSON, CSV, TSV, etc.

[
    {
        "Data": "IBM",
        "Reference": "376152"
    },
    {
        "Data": "International Business Machines",
        "Reference": "419044"
    },
    {
        "Data": "ibm corp",
        "Reference": "277383"
    }
]

Batch: Encoded JSON as input parameter - no source file necessary

This example illustrates how input data can look when performing similarity matching using the API with the source=jsonbatch parameter, using encoded JSON via the data parameter as input.

[
    {
        "Data": "IBM"
    },
    {
        "Data": "International Business Machines"
    },
    {
        "Data": "ibm corp"
    }
]

Batch: Encoded JSON as input parameter with a reference value - no source file necessary

This example illustrates how input data can look when performing similarity matching using the API with source=jsonrefbatch parameter, using encoded JSON via the data parameter as input. The reference parameter is simply a value, such as a primary key, you want to appear along with the generated similarity key. Using the target parameter, output can still be JSON, CSV, TSV, etc.

[
    {
        "Data": "IBM",
        "Reference": "376152"
    },
    {
        "Data": "International Business Machines",
        "Reference": "419044"
    },
    {
        "Data": "ibm corp",
        "Reference": "277383"
    }
]

Need help with your query/API call? Contact us at [email protected]