> ## Documentation Index
> Fetch the complete documentation index at: https://docs.occtoo.com/llms.txt
> Use this file to discover all available pages before exploring further.
# Ingest
Ingesting data into Occtoo is facilitated by the Ingest API. This powerful API serves as the backbone of the data ingestion process, allowing both data and media to be seamlessly pushed into Occtoo. All [providers](/integrations/providers/overview) utilize this API via the [onboarding SDK](/api-reference/overview#onboarding-sdk), acting as bridges between external systems and Occtoo's ecosystem.
## Data ingest
The Ingest API supports typed JSON and the legacy string-based entity format. Both formats validate the complete request before accepting it for asynchronous processing.
### Typed source entries
Use `POST https://api.occtoo.com/sources/{sourceId}` to send native JSON strings, numbers, booleans, `null` values, and arrays of strings. Occtoo validates values against configured source property types and infers types for new properties. New list properties use `,` as their delimiter unless the source already defines one.
Typed ingest requires a tenant-level Application token with the `write:sources` scope and access to the target source. A successful `202 Accepted` response includes the batch correlation ID, accepted entry count, and any new properties inferred from the request.
View authentication requirements, type mapping and inference rules, request and response schemas, and the interactive API playground.
### Legacy import
The legacy import format serializes property values as strings. Exchange registered Data Provider credentials through the Occtoo authorization service, then use that bearer token to call the import endpoint.
```mermaid theme={null}
sequenceDiagram
autonumber
participant Integration
participant Auth as Occtoo authorization
participant API as Ingest API
Integration->>Auth: POST /oauth2/token (scope: write:sources)
Auth-->>Integration: 200 access token
Integration->>API: POST /sources/{sourceId} with typed JSON
API-->>Integration: 202 correlation ID and accepted count
```
```mermaid theme={null}
sequenceDiagram
autonumber
participant Integration
participant Auth as Occtoo authorization
participant API as Ingest API
Integration->>Auth: POST /oauth2/token (scope: import-datasource)
Auth-->>Integration: 200 access token
Integration->>API: POST /datasources/{dataSource}/import
API-->>Integration: 202 correlation ID
```
### Rules
There are a set of rules related to the ingest data that need to be respected for the payload to be accepted by the ingest API:
* Entry key can have a length of 1-256 characters.
* Property ID can have a length of 1-256 characters.
* Property language can have a length of 2-10 characters.
* The following characters are allowed in entry key, property ID, or property language:
* a-z (lowercase a to z)
* A-Z (uppercase A to Z)
* 0-9 (numbers 0 to 9)
* \_ (underscore)
If one of the rules is broken for *any* of the entries in a payload, the **entire import** will be rejected. The API response will indicate what the error is.
```javascript theme={null}
// Response HTTP Code 400 Bad Request
{
"result": null,
"errors": [
{
"message": "Invalid request",
"details": {
"propertyName": "entities",
"message": "Entity keys contain restricted characters. The following ids containing restricted characters: @2"
}
}
],
"requestId": "254db272f8af8cfe0e01051951cb1550"
}
```
### Recommendations
We recommend that payloads containing entities adhere to the following guidelines for optimal performance and efficiency:
* **Total payload size:** Ensure that the total payload size does not exceed 20 MB. This helps maintain efficient data transfer and processing.
* **Single entity size:** Each individual entity within the payload should not exceed 1 MB in size. Keeping entity sizes manageable improves processing speed and reduces the risk of errors.
* **Payload entity limit:** It is recommended that a single payload does not contain more than 1000 entities. This limit helps maintain optimal performance during data ingestion and processing.
Adhering to these guidelines will help ensure smooth and efficient handling of payloads containing entities, leading to improved system performance and reliability.
### API endpoints
#### Data Provider access token
Request a token through the standard OAuth 2.0 client credentials flow at `https://auth.occtoo.com/oauth2/token`. Use your Data Provider OAuth client ID and client secret, your tenant ID as `audience`, and `import-datasource` as `scope`.
Cache and reuse the returned `access_token` until it expires instead of requesting a new token for every import.
See [Data Provider access token](/api-reference/authentication/application-token#data-provider-access-token) for the request fields, interactive playground, and response schema.
#### Import
Imports are initiated by sending a POST request to the URL ending with the identifier of the [data source](/guides/studio/sources/add-a-data-source) the data is intended for.
```text theme={null}
https://api.occtoo.com/datasources/{@DATA-SOURCE-IDENTIFIER}/import
```
The import endpoint accepts a JSON payload consisting of one or several entities. All import requests need to be authenticated by providing a bearer token as the authentication method. Upon successful import, the API will respond with an HTTP Code 202 Accepted, provided that the import content follows the [rules](#rules).
##### Request payload
The import format is structured as follows:
* **entities**: Array of entities to be included in the import
* **key** (Mandatory): Unique identifier of the entity
* **delete** (Optional): Flag indicating if the entity is to be deleted (true) or added/updated (false)
* **properties** (Mandatory): Array of properties of the entity
* **id** (Mandatory): Unique identifier of the property
* **value** (Mandatory): Value of the property
* **language** (Optional): The language code for the property
If an entry is to be added or updated, the **delete property** *can optionally* be omitted.
```javascript theme={null}
{
"entities" : [
{
"key" : "Entry-key",
"delete" : false,
"properties" : [
{
"id" : "Property-ID",
"value" : "Property Value",
"language" : "Code"
}
]
}
]
}
```
It is possible to send multiple entries, each with multiple properties, in a single payload.
```javascript theme={null}
{
"entities" : [
{
"key" : "item123",
"delete" : false,
"properties" : [
{
"id" : "my-property",
"value" : "My property value for item123",
"language" : "en-us"
}
]
},
{
"key" : "item456",
"delete" : false,
"properties" : [
{
"id" : "my-property",
"value" : "My property value for item456. This text is slightly longer",
"language" : "en-us"
}
]
}
]
}
```
To provide a property in multiple languages, repeat the property ID with its corresponding language code and value. If a property is not to be localized, the **language property** can optionally be omitted, left blank, or set to null.
```javascript theme={null}
{
"entities" : [
{
"key" : "item123",
"properties" : [
{
"id": "My-first-prop",
"value":"This is the value of my-first-prop in English",
"language": "en"
},
{
"id": "My-first-prop",
"value":"これがmy-first-propの値です",
"language": "ja"
},
{
"id": "My-second-prop",
"value":"This is the value of My-second-prop"
},
{
"id": "My-third-prop",
"value":"This is the value of My-third-prop",
"language": null
},
{
"id": "My-fourth-prop",
"value":"This is the value of My-fourth-prop",
"language": ""
}
]
}
]
}
```
When removing entities, there is *no need* to send any properties in the payload as the item is to be deleted. Omitting properties also reduces the import payload size and thus saves the planet, one byte at a time 😉.
```javascript theme={null}
{
"entities" : [
{
"key" : "item123",
"delete" : true,
"properties" : []
}
]
}
```
One has the option to mix actions in the payload, adding entities at the same time as updating or removing others.
```javascript theme={null}
{
"entities" : [
{
"key" : "item123",
"delete" : true,
"properties" : []
},
{
"key" : "item456",
"properties" : [
{
"id": "My-prop",
"value":"This is a multi action payload example",
"language": "en"
}
]
}
]
}
```
##### Response payload
The status of the import is given by the HTTP response code of the request.
```javascript theme={null}
// Response HTTP Code 202 Accepted
{
"result": {
"id": "9d4a30c2-949d-4c4e-afb2-dc9279dbbd95"
},
"errors": [],
"requestId": "e9162cfaa9a41cccc919db13356ef5b7"
}
```
## Media ingest
Media ingestion can be performed using the ingest API, but the media file to be uploaded must be available at a public URL. The Occtoo [onboarding SDK](/api-reference/overview#onboarding-sdk) supports uploading via streaming content directly. Before calling a media endpoint, request a Data Provider access token from `https://auth.occtoo.com/oauth2/token`.
All uploaded media files are provided with an Occtoo media file identifier. The media file ID is used for all API interactions related to the file. When uploading a file, users have the option to provide their own unique identifiers as an attribute to the file. The unique ID attribute is respected, meaning that once it is used, any additional upload attempts with that same unique ID will fail. If a media file needs to be replaced, users must first delete the original and then upload the updated file.
All uploads are made asynchronously, and the ingest API allows for multiple uploads to be initiated in a single request. To track the status of uploads, users can use the [Upload Status API](#upload-status).
Media information can be retrieved either using user-provided unique IDs via the [Media File Information by Unique Identifier API](#media-file-information-by-unique-identifier), or using the Occtoo media file ID via the [Media File Information by File ID API](#media-file-information-by-file-id).
Each flow sends a [Data Provider access token](/api-reference/authentication/application-token#data-provider-access-token) in the `Authorization` header.
Upload media files from public links with the [Upload via Links API](#upload-via-links).
```mermaid theme={null}
sequenceDiagram
autonumber
participant Integration
participant API as Ingest API
Integration->>API: PUT /media/uploads/links with bearer token
API-->>Integration: 202 accepted media IDs
```
Check an asynchronous upload with the [Upload Status API](#upload-status).
```mermaid theme={null}
sequenceDiagram
autonumber
participant Integration
participant API as Ingest API
Integration->>API: GET /media/uploads/{mediaId} with bearer token
API-->>Integration: 200 upload progress and state
```
Get metadata for one file with the [Media File Information by File ID API](#media-file-information-by-file-id).
```mermaid theme={null}
sequenceDiagram
autonumber
participant Integration
participant API as Ingest API
Integration->>API: GET /media/files/{mediaId} with bearer token
API-->>Integration: 200 metadata and public URL
```
Get metadata for several files with the [Media File Information by Unique Identifier API](#media-file-information-by-unique-identifier).
```mermaid theme={null}
sequenceDiagram
autonumber
participant Integration
participant API as Ingest API
Integration->>API: POST /media/files/batch with bearer token
API-->>Integration: 200 results grouped by unique ID
```
Delete a file by its Occtoo media ID with the [Delete Media endpoint](#delete-media).
```mermaid theme={null}
sequenceDiagram
autonumber
participant Integration
participant API as Ingest API
Integration->>API: DELETE /media/files/{mediaId} with bearer token
API-->>Integration: 204 no content
```
### API endpoints
#### Data Provider access token
Request a token through the standard OAuth 2.0 client credentials flow at `https://auth.occtoo.com/oauth2/token`. Use your Data Provider OAuth client ID and client secret, your tenant ID as `audience`, and `import-datasource` as `scope`.
Cache and reuse the returned `access_token` until it expires instead of requesting a new token for every media API call.
See [Data Provider access token](/api-reference/authentication/application-token#data-provider-access-token) for the request fields, interactive playground, and response schema.
#### Upload via links
Media uploads are initiated by sending a PUT request containing a list of one or more media files to be uploaded. Upon successful acceptance of the payload, the API will respond with an HTTP Code 202 Accepted.
```text theme={null}
https://api.occtoo.com/media/uploads/links
```
##### Request payload
The upload via links payload format is structured as follows:
* **links**: Array of links to the media to be included in the import
* **filename** (Mandatory): The filename of the media file when uploaded
* **link** (Mandatory): The URL to the media to be uploaded
* **uniqueidentifier** (Optional): Unique identifier of the media to be uploaded
If the media file that is to be uploaded contains illegal characters one can use the filename attribute to rename the file as this is the name that will be used for the file when uploaded to the Occtoo media service.
You can mix and match different types of media and locations when uploading.
```javascript theme={null}
{
"links": [
{
"filename": "my-media-file-name.png",
"link": "https://www.the.url/to/the/media/file/my-media-file-name.png",
"uniqueidentifier": "1709807830"
},
{
"filename": "another-media-file.tif",
"link": "https://www.another.url/to/a/media/file/location/another-media-file.tif",
"uniqueidentifier": "1709808053"
}
]
}
```
It is possible to either set **uniqueidentifier** to *null* or omit it entirely from the payload. However, it is our **strong recommendation** that a unique identifier is used to avoid reuploading the same binary multiple times.
```javascript theme={null}
{
"links": [
{
"filename": "my-media-file-name.png",
"link": "https://www.the.url/to/the/media/file/my-media-file-name.png"
},
{
"filename": "another-media-file.tif",
"link": "https://www.the.url/to/the/media/file/another-media-file.tif",
"uniqueidentifier": null
}
]
}
```
##### Response payload
The response is divided into two parts: `succeeded` and `failures`. Since file uploads are handled asynchronously, each linked media's information in the sent payload indicates whether the file has finished uploading to Occtoo. Accepted links receive an Occtoo media ID, which can be used to query the upload status of individual links using the [upload status](#upload-status) endpoint. This same ID can also be used to retrieve media file details using the [media information by file ID](#media-file-information-by-file-id) endpoint. If an upload link was rejected for any reason (such as conflicting unique identifiers or unreachable URLs), it will be categorized under the `failures` section with an error message explaining the cause of the failed upload.
```javascript theme={null}
{
"result": {
"succeeded": {
"https://www.the.url/to/the/media/file/my-media-file-name.png": {
"id": "6601c9e3-b4df-4610-ba0c-cfeb29432a63",
"progress": {
"totalSize": 107508,
"uploadedSize": 107508,
"completedPercentage": 100.0,
"isCompleted": true
},
"state": "Completed",
"sourceUrl": "https://www.the.url/to/the/media/file/my-media-file-name.png",
"createdAt": "2024-03-07T12:40:42.9386452+00:00",
"updatedAt": "2024-03-07T12:40:42.9386453+00:00",
"metadata": {
"filename": "my-media-file-name.png",
"mimeType": "application/octet-stream",
"uniqueIdentifier": "1709807830",
"size": 107508
}
}
},
"failures": {
"https://www.the.url/to/the/media/file/another-media-file.tif": {
"error": "Media File with uniqueIdentifier [1709808053] already exists"
}
}
},
"errors": [],
"requestId": "cc29d09e6625df925891351523566d1a"
}
```
#### Upload status
The upload status of an individual file can be checked by sending a GET request with the provided Occtoo media ID. Upon successful request, the API will respond with an HTTP Code 200 OK.
```text theme={null}
https://api.occtoo.com/media/uploads/{@OCCTOO-MEDIA-ID}
```
##### Response payload
The response payload contains detailed information on the current status of the upload.
```javascript theme={null}
{
"result": {
"id": "6601c9e3-b4df-4610-ba0c-cfeb29432a63",
"progress": {
"totalSize": 107508,
"uploadedSize": 107508,
"completedPercentage": 100.0,
"isCompleted": true
},
"state": "Completed",
"sourceUrl": "https://www.the.url/to/the/media/file/my-media-file-name.png",
"createdAt": "2024-03-07T12:40:42.9386452+00:00",
"updatedAt": "2024-03-07T12:40:42.9386453+00:00",
"metadata": {
"filename": "my-media-file-name.png",
"mimeType": "application/octet-stream",
"uniqueIdentifier": "1709807830",
"size": 107508
}
},
"errors": [],
"requestId": "c9e82ec7e9d49018e7cc076ea1336bb2"
}
```
#### Media file information by File ID
Detailed media information can be fetched by doing a GET request with the Occtoo media ID. Upon successful request, the API will respond with an HTTP Code 200 OK.
```text theme={null}
https://api.occtoo.com/media/files/{@OCCTOO-MEDIA-ID}
```
##### Response payload
The response contains the **public URL** that is to be used when accessing the media. Additionally, the response contains the `metadata` object, which provides more details about the media object, such as its size and MIME type. If the media object is an image file (e.g., jpg, png) or a video file (e.g., mp4), there is detailed information about the file encapsulated in the `mediaInfo` object.
```javascript theme={null}
{
"result": {
"id": "6601c9e3-b4df-4610-ba0c-cfeb29432a63",
"publicUrl": "https://cdn.occtoo-media.com/24885f4e-9f70-44ad-870e-d18362a20b2b/6601c9e3-b4df-4610-ba0c-cfeb29432a63/my-media-file-name.png",
"sourceUrl": "https://www.the.url/to/the/media/file/my-media-file-name.png",
"metadata": {
"filename": "my-media-file-name.png",
"mimeType": "image/png",
"size": 107508,
"extension": ".png",
"mediaInfo": {
"image": {
"width": 400,
"height": 668,
"resolution": {
"vertical": 3779.0,
"horizontal": 3779.0
}
}
}
}
},
"errors": [],
"requestId": "0efc737c456fddfca0cd0d0b85ab19b6"
}
```
#### Media file information by unique identifier
To fetch detailed media information for multiple objects simultaneously, you can make a POST request to the batch endpoint, providing a list of unique identifiers. Upon a successful request, the API will respond with an HTTP Code 200 OK.
```text theme={null}
https://api.occtoo.com/media/files/batch
```
##### Request payload
The endpoint accepts a JSON payload in the body, which consists of a list of unique identifiers used during the media upload process.
```javascript theme={null}
{
"UniqueIdentifiers": [
"1709807830",
"1709808053",
"my-fake-unique-id"
]
}
```
##### Response payload
The response is divided into two parts: `succeeded` and `failures`. Media information for each media object that could be identified via its unique identifier is listed under `succeeded`. The response contains the **public URL** to access the media. Additionally, it includes the `metadata` object, which provides details such as size and MIME type. If the media object is an image file (e.g., jpg, png) or a video file (e.g., mp4), detailed information is encapsulated in the `mediaInfo` object.
Unique identifiers that were not found are sorted under `failures`.
```javascript theme={null}
{
"result": {
"succeeded": {
"1709807830": {
"id": "6601c9e3-b4df-4610-ba0c-cfeb29432a63",
"publicUrl": "https://cdn.occtoo-media.com/24885f4e-9f70-44ad-870e-d18362a20b2b/6601c9e3-b4df-4610-ba0c-cfeb29432a63/my-media-file-name.png",
"sourceUrl": "https://www.the.url/to/the/media/file/my-media-file-name.png",
"metadata": {
"filename": "my-media-file-name.png",
"mimeType": "image/png",
"size": 107508,
"extension": ".png",
"mediaInfo": {
"image": {
"width": 400,
"height": 668,
"resolution": {
"vertical": 3779.0,
"horizontal": 3779.0
}
}
}
}
},
"1709808053": {
"id": "1321b879-99a7-421c-b74f-0e756450e9f5",
"publicUrl": "https://cdn.occtoo-media.com/24885f4e-9f70-44ad-870e-d18362a20b2b/1321b879-99a7-421c-b74f-0e756450e9f5/another-media-file.tif",
"sourceUrl": "https://www.another.url/to/a/media/file/location/another-media-file.tif",
"metadata": {
"filename": "another-media-file.tif",
"mimeType": "image/tiff",
"size": 13498048,
"extension": ".tif",
"mediaInfo": {
"image": {
"width": 1000,
"height": 1000,
"resolution": {
"vertical": 1,
"horizontal": 1
}
}
}
}
}
},
"failures": {
"my-fake-unique-id": {
"message": "MediaFile not found in tenant 24885f4e-9f70-44ad-870e-d18362a20b2b"
}
}
},
"errors": [],
"requestId": "327015f59f3148dafd5fcb8ced83ee5c"
}
```
#### Delete media
To delete a media object one sends a DELETE request containing the Occtoo media ID. Upon successful request, the API will respond with an HTTP Code 204 No Content.
```text theme={null}
https://api.occtoo.com/media/files/{@OCCTOO-MEDIA-ID}
```