Streamed datasets and agents

A streamed dataset is not uploaded as a file. An agent sends data to a certified node over time. When you run an experiment, the script reads a snapshot of that data on the node, in the same folder layout as a regular dataset.

Streamed type appears on Upload Dataset when your portal environment has streamed datasets enabled.

Create the dataset

1. Follow Make your data accessible through Info and Contributors.
2. On File, choose Streamed instead of Regular. Do not upload a file.
3. Finish the draft. Status becomes Schema setup required.

Schema and maDMP

On the dataset details page choose Schema & DMP.

1. Define at least one column (names, types, no duplicates). Saving moves the dataset to maDMP setup required.
2. Create the machine-actionable DMP (maDMP). After that, status becomes Agent Initialization Pending.

Schema and maDMP cannot be changed after streaming starts. You cannot delete a maDMP while the dataset is linked to an agent.

Create an agent

1. Open Agents and choose Create Agent.
2. Copy the API key immediately. You will not see it again. Use it to configure your streaming agent.
3. Open the agent and Add Dataset. Only streamed datasets in Agent Initialization Pending can be linked.
4. Leave Allowed to stream on when the agent should deliver data. Status becomes Streaming in progress.

Caution

Regenerate Key breaks existing agent connections. Update the agent configuration with the new key.

Streamed datasets cannot be downloaded with Download Dataset. Samples, marketplace listing, and experiments still follow the same access and credits rules as regular datasets.