Chunked Uploader

Chunked Uploader is a special Uploader implementation to support Multiple-Step Uploads. Chunked Uploader can be used if the artifact (pdf) must be uploaded in multiple chunks to a destination system. E.g., the system allows only a certain size of a single file upload.

The typical use case is a multistep upload where the first step retrieves a URL for a chunked upload, the second step uploads the chunks in a loop, and in a third step the upload is registered in the system.

To invoke the Chunked Uploader, users must set the uploadType: "chunked" in the configuration. The details of the upload process are configured via values in the uploadParams section.

The documentation is given by an extended example of a typical multiple-step upload.

Configuration

Here comes a complete configuration that also involves all three steps a typical multiple-step upload.

First we need to define the upload URLs in restapi configuration - or alternatively in the POST /renderpdf request parameter.

/restapi/config.ion

{
  upload: "https://my.dam/uploads https://my.dam/uploads/{uploadId} https://my.dam/register-upload"
}

Here, the three URLs point to the same server. Note that this is no requirement. Often the second step points to a different system, like, for instance, to a Google Storage bucket URL.

/upload/config.ion

The upload configuration is split into three connection parts:

{
  connections: [
    {  // STEP 1 - retrieve url for chunked upload 
      url: "https://my.dam/uploads", 
      method: "POST",
      authentication: { /* */ },
      exportParams: {
        "uploadLocation": {
          "source": "jsonBody",
          "path": "upload_url"
        },
        "objectLocation": {
          "source": "jsonBody",
          "path": "object_url"
        }
      }
    },
    {  // STEP 2 - upload chunks in a loop
      url: "https://my.dam/uploads/",
      method: "PUT",
      uploadType: "chunked",
      uploadTimeout: 60000,
      uploadParams: {
        chunkSize: 20000000,
        repeatMax: 20,
        repeatUntil: "statusCode=200 or statusCode=201",
        repeatWhile: "statusCode=308",
        repeatUrl: "{uploadLocation}"
      },
      header : {
        "Content-Length": "{rangeSize}",
        "Content-Range": "bytes {rangeStart}-{rangeEnd}/{totalSize}"
      },
      exportParams: [
        {
          "uploadLocation": {
            "source": "headers",
            "path": "Location"
          }
        }
      ]
    },
    {  // STEP 3 - register media-asset
      url: "https://my.dam/register-upload",
      method: "POST",
      authentication: { /* */ },
      header: {
        "content-type": "application/json"
      },
      payload: [
        {
          value: "{\"object_url\":\"{objectUrl}\",referenceId:\"{foreignKey}\"}"
        }
      ]
    }
  ]
}

Comments

• The specific exportParams are fictitious.
• uploadTimeout is the timeout of the whole process (complete series of PUT requests), requestTimeout is the timeout of an individual PUT request.
• All three steps must have their own connection configuration.
  • URLs must be unique for all three steps we would not support different HTTP methods on the same URL (as listed in the connection array).
  • Information between the steps is transported using exportParams.
• Polling for any result is not supported.
• In case of an error for one chunk, the whole upload fails and is repeated according to the redelivery configuration of the upload worker.
• Systems might want to use different URLs for each chunk ("Dynamic URLs") in a way that the response for request A contains the URL for request B (e.g., in the "Location" header).
  • When creating the URL used for the next chunk (repeatUrl) we must check for security reasons that the new URL still matches the connection URL pattern.
• Header names must be used in a case-insensitive way (e.g., "Location").

Special Configuration via uploadParams

Upload Parameter	Description
chunkSize	Number of bytes per chunk. Default is 50000000 (50 MB). Maximum is also set to 50 MB.
repeatMax	Maximum number for chunked requests. Default is 10. Maximum is 50.
repeatUntil	Optional JXPath expression to check whether we the current response is the last one.
repeatWhile	Optional JXPath expression to check whether the server expects more upload requests.
repeatUrl	Optional pattern for the continuation URL for dynamic URLs.

Iteration Control

Normally, the process iterates as long as there is still data to be sent. When there is no more data, the step is completed and the upload worker will execute the next upload url (which is the next step). For most cases this is enough. Users can, though, define more explicit rules for continuing of ending the iteration through the repeatUntil and/or repeatWhile options. These options contain a JXPath expression to check whether the current response is the last one or if the destination expects for data. Configurations may for instance check for specific status codes sent by the destination server like repeatUntil: "responseCode=200 or responseCode=201" or repeatWhile: "responseCode=308"

Process Parameters

Process Parameters are available in expression evaluation.

Process Parameter	Description
chunkSize	size of a chunk as defined by configuration
rangeEnd	0-based index of the last byte in the current chunk
rangeSize	number of bytes in the current chunk - can be smaller than chunkSize
rangeStart	0-based index of the first byte in the current chunk
responseCode	last HTTP response code
responseMessage	last HTTP response message - if any
responseTime	roundtrip time in millis for the last HTTP exchange
requestIndex	0-based index of exchanges
requestMillis	aggregated time in millis for all completed exchanges
totalSize	size of the uploaded document

Range Requests

Although HTTP Range Requests are mainly specified for GET requests and not for PUT, POST, or PATCH, some servers use "Range" headers to specify the current chunk. A configurator may use the process parameters to construct the appropriate range header for a connection like in the example below:

{ url: "https://my.dam/uploads/",
	header : {
      "Content-Length": "{rangeSize}", 
      "Content-Range": "bytes {rangeStart}-{rangeEnd}/{totalSize}"
	}
}