Get data into Occtoo

This guide displays how to get data into Occtoo in three easy steps.

tip

In order to get the maximum amount of value from this guide it is recommended to read up on the following concepts:

Step One - Preparations

Before pushing data into the system a data source and a data provider is needed. These steps are managed in the Occtoo Studio.

Create a Data Source

One prerequisite for onboarding data is having a location to place the data. In the Occtoo domain, this is known as a data source. To onboard data, you need to know the identifier of the data source you will be using. For detailed instructions on how to create a data source, see the full article on how to create a data source.

Create a Data Provider

The other prerequisite for onboarding data is having a data provider that has permissions to onboard data into the selected data source. Using the data provider's ID and secret, you can request a token, which is subsequently used to authenticate when calling the ingest API. See the article on how to create a data provider.

Step Two - Generate ingest token

The next step is to generate a token for interacting with the ingest API. To do this, you need the ID and secret of the data provider you are using.

By making a POST request to the URL https://ingest.occtoo.com/dataProviders/tokens and providing the data provider ID and secret in the request body as JSON, a new token is generated. For more information, see the token endpoint documentation.

{
  "id": "{@DATA-PROVIDER-ID}",
  "secret": "{@DATA-PROVIDER-SECRET}"
}

Below, you can find rudimentary code examples of how to request a token from the ingest API using the token endpoint.

Code examples

cURL
C#
JavaScript

curl --location 'https://ingest.occtoo.com/dataProviders/tokens' \
--header 'Content-Type: application/json' \
--data '

{
    "id" : "{@DATA-PROVIDER-ID}",
    "secret": "{@DATA-PROVIDER-SECRET}"
}'

using System.Net.Http.Headers;
using System.Text;
using System.Text.Json;

var baseAddress = "https://ingest.occtoo.com";
var dataProviderId = "{@DATA-PROVIDER-ID}";
var dataProviderSecret = "{@DATA-PROVIDER-SECRET}";

var httpClient = new HttpClient();
httpClient.BaseAddress = new Uri(baseAddress);

var tokenRequest = new HttpRequestMessage(HttpMethod.Post, "dataProviders/tokens");
tokenRequest.Content = new StringContent(JsonSerializer.Serialize(new
{
    id = dataProviderId,
    secret = dataProviderSecret
}), Encoding.UTF8, "application/json");
var tokenResponse = await httpClient.SendAsync(tokenRequest);
var tokenResponseContent = await tokenResponse.Content.ReadAsStreamAsync();
var tokenDocument = JsonSerializer.Deserialize<JsonDocument>(tokenResponseContent);
var token = tokenDocument.RootElement.GetProperty("result").GetProperty("accessToken").GetString();

// Example assumes you've patched fetch with node-fetch:
// npm install node-fetch
// globalThis.fetch = await import('node-fetch').then(m => m.default)

(async () => {
  const baseAddress = "https://ingest.occtoo.com";
  const dataProviderId = "{@DATA-PROVIDER-ID}";
  const dataProviderSecret = "{@DATA-PROVIDER-SECRET}";

  function getToken() {
    return fetch(baseAddress + "/dataProviders/tokens", {
      method: 'POST',
      headers: {
        ContentType: 'application/json',
      },
      body: JSON.stringify({
        id: dataProviderId,
        secret: dataProviderSecret
      })
    });
  }

})();

Step Three - Onboard Data

The third and final step is to import data into a data source using the generated token and the import endpoint. When onboarding data, each entry requires a unique key within the data source. If an existing key is provided, the entry is updated by merging the new data with the existing data: new or replaced properties will be updated, while existing properties not included in the import remain unchanged. If a new key is provided, a new entry is created.

Regardless of which data source is being updated the format of payload is always the same. A data entry has a key, a delete flag, and a list of properties. Each property has a property id, a value, and an optional language parameter.

Payload format example

 
 {
    "entities" : [
    {
        "key" : "Entry-key",
        "delete" : false,
        "properties" : [
        {
            "id" : "Property-ID",
            "value" : "Property Value",
            "language" : "Code"
        }
        ]
    }
    ]
}

Importing Data

Imports are initiated by sending a POST request to the URL ending with the identifier of the data source to which the data is intended for import. The generated ingest token needs to be included in the authorization header as a bearer token. The body should consist of the entries to import, update, or delete.

https://ingest.occtoo.com/import/{@DATA-SOURCE-IDENTIFIER}

Below are code examples of how data ingestion can look. There is also an onboarding SDK created to ease the interaction with the ingest API, available on Occtoo's GitHub page.

Code examples

cURL
C#
JavaScript

curl --location 'https://test-ingest.occtoo.com/import/mynewsource' \
--header 'Content-Type: application/json' \
--header 'Authorization: Bearer {@INGEST-ACCESS-TOKEN}' \
--data '{
        "entities" : [
        {
            "key" : "some-id",
            "properties" : [
            {
                "id": "propertyId",
                "value":"some value",
                "language": "lang-code"
            }
            ]
        }
        ]
    }'

using System.Net.Http.Headers;
using System.Text;
using System.Text.Json;

var baseAddress = "https://ingest.occtoo.com";
var dataSource = "{@DATA-SOURCE-IDENTIFIER}";

var httpClient = new HttpClient();
httpClient.BaseAddress = new Uri(baseAddress);

var token = GetToken();// see Generate token section for code example

var ingestRequest = new HttpRequestMessage(HttpMethod.Post, $"import/{dataSource}");
ingestRequest.Headers.Authorization = new AuthenticationHeaderValue("Bearer", token);
ingestRequest.Content = new StringContent(JsonSerializer.Serialize(new
{
    entities = new []
    {
        new
        {
            key = "some-id",
            delete = false,
            properties = new []
            {
                new
                {
                    id = "propertyId",
                    value = "some value",
                    language = "lang-code"
                }
            }
        }
    }
}), Encoding.UTF8, "application/json");
var ingestResponse = await httpClient.SendAsync(ingestRequest);
ingestResponse.EnsureSuccessStatusCode();

// Example assumes you've patched fetch with node-fetch:
// npm install node-fetch
// globalThis.fetch = await import('node-fetch').then(m => m.default)

(async () => {

  const baseAddress = "https://ingest.occtoo.com";
  const dataSource = "{@DATA-SOURCE-IDENTIFIER}";

  const response = await getToken();// see Generate token section for code example

  if (!response.ok) {
    throw new Error("Response not ok: " + response.status, response);
  }

  const { accessTokenResult } = await response.json();

  const examplePayload = {
    entities: [
      {
        key: "some-id",
        delete: false,
        properties: [
          {
            id: "propertyId",
            value: "some value",
            language: "lang-code"
          }
        ]
      }
    ]
  };

  fetch(baseAddress + "/import/campaign", {
    method: 'POST',
    headers: {
      ContentType: 'application/json',
      Authorization: "Bearer " + accessTokenResult.accessToken,
    },
    body: JSON.stringify(examplePayload),
  });
})();

For more details on the import see the import endpoint documentation.

Summary

Getting data into Occtoo is an easy process involving three steps: First, create a data source and a data provider in Occtoo Studio. Next, generate a token using the data provider ID and secret. Finally, onboard data by posting it to the designated endpoint with the token.

For detailed steps and code examples, visit the ingest documentation.

Step One - Preparations​

Create a Data Source​

Create a Data Provider​

Step Two - Generate ingest token​

Step Three - Onboard Data​

Importing Data​

Summary​