Skip to main content

Get data into Occtoo

This guide displays how to get data into Occtoo in three easy steps.

tip

In order to get the maximum amount of value from this guide it is recommended to read up on the following concepts:

1 Preparations

Before pushing data into the system a data source and a data provider is needed. These steps are managed in the Occtoo Studio.

1.1. Create data source

To create a data source go to Sources in Studio and click on 'Add data source'. A name and identifier is mandatory, the name is a user friendly way of identifying the source and the identifier is used to identify the source when pushing in data. The identifier has to be unique.

An image from the static

1.2 Create data provider

To create a data provider go to Sources in Studio, select the tab 'Data providers 'and click on 'Add data provider'. Select a display name for your provider and which sources that the provider should be allowed to update. After creation the secret will be shown. Make sure to copy it as it will not be possible to retrieve later, However, it is possible to reset the secret. In the listing the data provider id is shown and this is needed together with the secret to authenticate.

An image from the static

2 Generate token

To generate a token the following is needed:

  • Data provider id: found in the data provider listing
  • Data provider secret: from data provider registration

By doing a POST request to the url https://ingest.occtoo.com/dataProviders/tokens and providing a body of the provider id and provider secret a new token is retrieved.

/*Token request body*/

{
id : "@dataProviderId",
secret: "@dataProviderSecret"
}
using System.Net.Http.Headers;
using System.Text;
using System.Text.Json;

var baseAddress = "https://ingest.occtoo.com";
var dataProviderId = "@application_id@";
var dataProviderSecret = "@application_secret@";
var dataSource = "@datasource_identifier@";

var httpClient = new HttpClient();
httpClient.BaseAddress = new Uri(baseAddress);

var tokenRequest = new HttpRequestMessage(HttpMethod.Post, "dataProviders/tokens");
tokenRequest.Content = new StringContent(JsonSerializer.Serialize(new
{
id = dataProviderId,
secret = dataProviderSecret
}), Encoding.UTF8, "application/json");
var tokenResponse = await httpClient.SendAsync(tokenRequest);
var tokenResponseContent = await tokenResponse.Content.ReadAsStreamAsync();
var tokenDocument = JsonSerializer.Deserialize<JsonDocument>(tokenResponseContent);
var token = tokenDocument.RootElement.GetProperty("result").GetProperty("accessToken").GetString();

caution

Do not include secrets in frontend JavaScript code or add to source control. Remove secrets that are no longer needed.

3 Onboarding data

When onboarding data, each entry needs a key. The key has to be unique within the data source. If an existing key is provided the data entry will be updated, and if a new key is provided the data entry will be created.

For updates the existing data will be merged with the new data. Properties in the import will be created or replaced, and already existing properties not in the import will be unmodified.

3.1 Payload format

Regardless of which data source is being updated the format of payload is always the same. One entry has a key, a delete flag, and a list of properties. Each property has a property id, a value, and an optional language parameter.

  //data source entry
{
key = "some-id",
delete = false,
properties = new []
{
new
{
id = "propertyId",
value = "some value",
language = "optional language code"
}
}
}

3.2 Importing data

All data import is done by posting to the same endpoint (https://ingest.occtoo.com/import/[@sourceIdentifier]). The request needs to contain a authorization header containing the authentication token, and the URL should also contain the identifier for the source that is being updated.

The POST body is an array of data entries and a successful request will result in a 202 response code.

note

The data provider needs to be configured for the source for the import to succeed.

caution

Supported characters

  • Entry Key - Maximum length 256 characters - Allowed characters a-z,A-Z,0-9,_-
  • PropertyId - Maximum length 256 characters - Allowed characters a-z,A-Z,0-9,_-
  • Language- Maximum length 10 characters- Allowed characters a-z,A-Z,0-9,_
using System.Net.Http.Headers;
using System.Text;
using System.Text.Json;

var baseAddress = "https://ingest.occtoo.com";
var dataSource = "@datasource_identifier@";

var httpClient = new HttpClient();
httpClient.BaseAddress = new Uri(baseAddress);

var token = GetToken();// see Generate token section for code example

var ingestRequest = new HttpRequestMessage(HttpMethod.Post, $"import/{dataSource}");
ingestRequest.Headers.Authorization = new AuthenticationHeaderValue("Bearer", token);
ingestRequest.Content = new StringContent(JsonSerializer.Serialize(new
{
entities = new []
{
new
{
key = "some-id",
delete = false,
properties = new []
{
new
{
id = "propertyId",
value = "some value",
language = "optional language code"
}
}
}
}
}), Encoding.UTF8, "application/json");
var ingestResponse = await httpClient.SendAsync(ingestRequest);
ingestResponse.EnsureSuccessStatusCode();
caution

A data entry key can only exist once in an import batch. If it exists more than once the batch will fail in validation.

3.3 Deleting data

For deleting entries the same request as importing data is used, but the delete flag should be set to true for the entries to delete. No properties are needed as they will be ignored

  //data source entry for deleting
{
key = "some-id",
delete = true,
properties = new []
}