> ## Documentation Index
> Fetch the complete documentation index at: https://docs.cloudglue.dev/llms.txt
> Use this file to discover all available pages before exploring further.
# Bulk Imports
> Import an entire connected source into a collection with saved, re-runnable import definitions
A bulk import lists a connected data source and brings everything matching your filters into a
collection — instead of calling the API once per file. It is a **saved definition** you can re-run,
so "keep this collection in sync with that Zoom account" is a durable thing rather than a script you
have to babysit.
## Two kinds of import
What an import ingests follows the **collection you create it on**. You never choose this directly —
it is reported back as `import_type`.
| Collection type | `import_type` | What a run does | Cost |
| --------------------------------------------------------------------- | ------------- | ----------------------------------------------------------------------------- | --------------- |
| `metadata` | `metadata` | Indexes whatever metadata the source exposes. No media is downloaded. | Free |
| `rich-transcripts`, `media-descriptions`, `entities`, `face-analysis` | `media` | Ingests and processes the actual files, exactly like adding each one by hand. | Billed per file |
Which fields you get depends entirely on the connector — a metadata import stores what that source
happens to publish about each file, and sources differ a lot. A Zoom recording carries things like a
topic, host, start time and duration; a Gong call adds parties and its own call analysis; Grain
contributes AI summaries and highlights; a Dropbox or Drive file may be little more than a name,
size and timestamps; iconik brings whatever custom metadata your workspace has configured. Anything
present is indexed and becomes searchable and filterable.
The two compose into the workflow most teams want: bulk-index a whole source
into a **metadata** collection for free, search it to find the files actually
worth processing, then run a **media** import with narrower filters against a
processing collection to ingest only those.
## Create an import
```bash theme={null}
curl -X POST "https://api.cloudglue.dev/v1/collections/$COLLECTION_ID/imports" \
-H "Authorization: Bearer $CLOUDGLUE_API_KEY" \
-H 'Content-Type: application/json' \
-d '{
"name": "All sales calls",
"connector_id": "'"$CONNECTOR_ID"'",
"max_files": 500
}'
```
The first run starts immediately unless you pass `start: false`. The response echoes the definition
plus `latest_run`.
### Filtering what gets imported
`filters` is an array of listing passes, mirroring the parameters you would pass when browsing a
connector. Each set runs as its own pass and results are deduplicated, so overlapping sets are safe.
```json theme={null}
{
"filters": [
{ "from": "2026-07-01", "to": "2026-09-30", "title_search": "customer" }
]
}
```
Which keys a connector honors depends on the connector — `folder_id` for Google Drive, `path` and
`recursive` for Dropbox, `team` and `meeting_type` for Grain, `title_search` and date ranges for most.
Unsupported keys are **rejected when you create the import** rather than silently ignored, because a
saved definition re-runs forever and a quietly dropped filter would import the whole source.
Zoom and Gong default to a six-month lookback when no `from` is given. That
date is frozen into the definition at creation so re-runs keep listing the
same window — otherwise a `refresh` + `delete_missing` run would remove files
that had merely aged out of the window.
## Watch a run
```bash theme={null}
curl "https://api.cloudglue.dev/v1/collections/$COLLECTION_ID/imports/$IMPORT_ID" \
-H "Authorization: Bearer $CLOUDGLUE_API_KEY"
```
`files_queued` is the denominator; `files_imported` (media) or `files_indexed` (metadata) plus
`files_failed` settle against it.
```json theme={null}
{
"import_type": "media",
"runs": [
{
"status": "processing",
"progress": {
"files_listed": 180,
"files_queued": 154,
"files_imported": 96,
"files_skipped": 26,
"files_failed": 2
}
}
]
}
```
Large runs take as long as the work takes — hundreds of long videos is normal for hours. A run keeps
going in the background; you can close the connection and poll later.
## Re-running: append, refresh, and mirroring
Every run picks a mode, defaulting to the definition's `default_mode`.
Imports files the collection does not have yet and retries previously failed
ones. Files already imported are skipped: not re-downloaded, not re-billed.
This is the mode to re-run on a schedule.
Additionally re-fetches source metadata for files this import already
brought in. For media imports the bytes are never re-downloaded — only the
metadata is refreshed.
After a **complete** listing, removes files this import previously added
that the source no longer returns. Files added by a *different* import,
files you added yourself, and files this import has never listed are all
left alone. Only the file's place in this collection is removed — the file
itself, and its use in any other collection, is untouched. A run that
stopped at `max_files` skips this step entirely, because a truncated
listing cannot prove anything is missing.
To keep a file regardless of what the source does, add it to a collection
this import does not target.
```bash theme={null}
curl -X POST "https://api.cloudglue.dev/v1/collections/$COLLECTION_ID/imports/$IMPORT_ID/runs" \
-H "Authorization: Bearer $CLOUDGLUE_API_KEY" \
-H 'Content-Type: application/json' \
-d '{ "mode": "refresh", "delete_missing": true }'
```
## When a run stops early
Media imports are billed per file, so a run can stop on something account-wide — running out of
credits, or hitting a usage limit. When that happens the run ends with a `failed` status and a
plain-language `error`, and **everything already imported is kept**.
Recovery is always the same move: fix the cause, then re-run in `append` mode. Already-imported files
are skipped and only the remainder is attempted.
```bash theme={null}
curl -X POST "https://api.cloudglue.dev/v1/collections/$COLLECTION_ID/imports/$IMPORT_ID/runs" \
-H "Authorization: Bearer $CLOUDGLUE_API_KEY" \
-H 'Content-Type: application/json' \
-d '{ "mode": "append" }'
```
Problems with a single file — too long, unsupported type, no longer in the source — fail only that
file. The run continues and reports it in `files_failed`.
## Cancelling
```bash theme={null}
curl -X POST "https://api.cloudglue.dev/v1/collections/$COLLECTION_ID/imports/$IMPORT_ID/runs/$RUN_ID/cancel" \
-H "Authorization: Bearer $CLOUDGLUE_API_KEY"
```
Files already imported stay in the collection. Deleting the import definition likewise removes only
the definition and its run history — imported content is never removed.
## Things to know
* **One active run per collection.** Starting a second while one is in flight returns a `409`;
creating a definition still succeeds, it just comes back with `latest_run: null` so you can run it
once the current one finishes.
* **Definitions are immutable.** To change filters or connector, delete and recreate.
* **`max_files`** caps the files a run takes from the listing, and a capped run never removes anything.
* **Runs use the workspace's default API key** to read the connector, so the workspace needs one.
## Reference