Adding and Removing Files

With Caddie, you can bring your existing data and easily make it accessible to your Agents, allowing them to respond to user's prompts with answers enriched by your custom or proprietary data. Behind the scenes, your documents will be processed and stored in a format that makes them more accessible for retrieval by the Agent.

You can add files in one of two ways: through uploading them individually through our backend APIs, and through automated data synchronization (the next page in this documentation).

Generating an upload url

To upload a file, retrieve a presigned upload url from /api/file. You must include the id of the folder the file will be placed in, and the name of the file. To upload a file to the root folder of an organization, pass the folderId as null.

fetch('http://localhost:3000/api/file', {
  method: 'POST',
  headers: {
    'X-API-KEY': '',
    'X-ORGANIZATION-ID': "",
  },
  body: {
	"folderId": "", //null for root folder
	"name": "" //filename
  }
})

If successful, you will recieve a response with a presigned URL, which will expire in 5 minutes.

{
	"success":true,
	"data": {
		"presignedUrl":"presigned-url-string"
	}
}

Uploading via presigned url

You can upload the file to this url using an octet-stream in a PUT request. Here is a sample curl request:

curl -X PUT -T "path/to/local/file" -H "Content-Type: application/octet-stream" "presigned-url"

For more information on uploading via presigned url, check out the AWS documentation

Adding custom metadata

Files can also be added with custom metadata fields, which can later be used for custom metadata filtering in chat sessions. To include custom fields, use the "metadata" object in the request body:

fetch('http://localhost:3000/api/file', {
  method: 'POST',
  headers: {
    'X-API-KEY': '',
    'X-ORGANIZATION-ID': "",
  },
  body: {
	"folderId": "",
	"name": "", //filename
	"metadata": {
		"metadata-field-1": "firstValue",
		"metadata-field-2": "secondValue"
	}
  }
})

WARNING: The keys of the metadata field you provide are NOT case-sensitive. They will be converted to lower-case in the knowledgebase. It is recommended that you use kebab case for metadata keys. Metadata values, however, are case-sensitive and don't have to be changed. For example, if you're metadata key-value pair is "userTeam"

WARNING: You should avoid naming any of your keys any of the following names, as it will be overriden by default values. The default metadata fields are:

  • fileId: id of uploaded file,
  • organizationId: x-organization-id,
  • organizationSlug: slug of x-organization-id,
  • fileName: name of uploaded file,
  • fileMetadataS3Key: s3 key, created from filename + filepath,
  • createdAt: date created,
  • updatedAt: date last updated,

Requirements for /upload request fields:

  • folderId: UUID string, optional. If provided, it must be a folder within the X-ORGANIZATION-ID. If not provided, the file will be uploaded at the root level of organization file structure.
  • name: required string, minimum 1 character.

File upload requirements

The following filetypes are supported:

Supported filetypes

  • Plain text (ASCII only) .txt
  • Markdown .md
  • HyperText Markup Language .html
  • Microsoft Word document .doc/.docx
  • Portable Document Format .pdf

File requirements

The maximum supported filesize is 50MB. If uploading multiple files, the total filesize of all the files in your upload request cannot exceed 100GB.

If you have PDFs, paginate them according to a normal 8.5 x 11 paper size. A vertically long PDF that isn't paginated (with lots of information on the same page) may fail to index properly into the knowledgebase.

Removing files

Files can be deleted using a DELETE request to /api/file/file-id-to-delete:

fetch('http://localhost:3000/api/file/file-id-to-delete', {
  method: 'DELETE',
  headers: {
    'X-API-KEY': '',
    'X-ORGANIZATION-ID': '',
  }
})

Adding and removing folders

Folders can be used to organize your files and are also a key component in the default permissions system. By default, you grant an agent access to files by placing them in a folder then granting the agent access to that folder.

To create a folder, issue a POST request to the /api/folder endpoint. You must provide a name for the folder. You can also optionally provide the id of the new folder's parent, if you wish to create a nested directory structure. If no parentFolderId is provided, the folder will be created in the organization's root directory.

fetch('http://localhost:3000/api/folder', {
  method: 'POST',
  headers: {
    'X-API-KEY': '',
    'X-ORGANIZATION-ID': '',
  },
  body: JSON.stringify({
    name: '',
    parentFolderId: '' //optional
  })
})

If successful, the response to your folder creation will include the following metadata for the new folder:

{
  "id": "",
  "name": "",
  "createdAt": "",
  "userId": "", // id of your user, who created the folder
  "organizationId": "",
  "parentId": "" // id of parent folder, or null if new root-level folder

}

Deleting a folder is nearly identical to deleting a file: provide the id of the folder you wish to delete as a URL parameter:

fetch('http://localhost:3000/api/folder/folder-id-to-delete', {
  method: 'DELETE',
  headers: {
    'X-API-KEY': '',
    'X-ORGANIZATION-ID': '',
  }
})

Listing all contents of a folder

To show the contents of a folder (including both files and subfolders), issue a GET request with the folder id as a URL parameter:

fetch('http://localhost:3000/api/folder/folder-id-to-get', {
  method: 'GET',
  headers: {
    'X-API-KEY': '',
    'X-ORGANIZATION-ID': '',
  }
})

To view the contents of the root level directory, don't pass any "folder-id-to-get" URL param, instead making your GET request to "/api/folder". Caddie will show you the root level directory contents for X-ORGANIZATION-ID.