description: Repster's execution model: flow discovery, local vs. cloud runs, immutable deployments, secrets injection, and observability.¶
How Repster works¶
Repster adds scheduling, secrets management, cloud execution, and observability on top of dlt, without changing how you write pipelines.
The core idea¶
A Repster project is a Python package with flows:
@rp.flow(schedule="0 * * * *")
def my_pipeline():
pipeline = dlt.pipeline(destination="duckdb")
pipeline.run(my_source())
Repster discovers flows by scanning your project files for @rp.flow() decorators. Each flow is a callable that runs a dlt pipeline. The decorator adds a name, an optional schedule, and hooks into the Repster execution model.
Local vs. cloud execution¶
Repster uses the same @rp.flow() code for both local and cloud execution; the runner is swapped, not the flow definition.
Local: rp run <flow> or rp dev (scheduler). Runs in your local Python environment, writes state to .repster/ (SQLite).
Cloud: rp run --cloud <flow>. Dispatches to an isolated worker managed by Repster. The worker runs the same @rp.flow() code in an environment built from your uv.lock.
Deployments¶
rp deploy packages your project into a source archive with your uv.lock and uploads it to the Repster control plane. Each deployment is:
- Immutable: the archive is content-addressed by
sha256. Deploying the same code twice is a no-op. - Complete: the lockfile is included so workers build a reproducible environment.
Cloud runs always execute against a specific deployment. rp run --cloud uses the latest.
Secrets¶
Secrets (API keys, database credentials) live in .env locally and in the Repster secrets vault in the cloud.
rp secrets push reads .env and stores values encrypted in the vault. At run time, the worker receives the secrets as environment variables, never in logs or run records.
Repster uses dlt's env-var naming convention, so the same secrets work locally and in the cloud without any code changes.
Observability¶
Every run produces a log stream captured from dlt's output. Logs include:
- dlt pipeline progress (rows loaded, tables updated)
- Any
print()/loggingoutput from your code - Full stack traces on failure
Runs are stored in the control plane with status, timing, and row counts. Local runs are stored in .repster/ and merged with cloud runs in rp status.
What belongs in a flow¶
Keep the flow body a thin wrapper: construct a dlt.pipeline(...) and call .run(...). Extraction logic (HTTP calls, pagination, parsing) belongs inside your @dlt.resource/@dlt.source functions, where dlt already expects arbitrary Python. One flow run should map to one observable dlt pipeline run.
Repster is not a general-purpose script or task runner. If you need to run arbitrary non-dlt workloads on a schedule, use a dedicated orchestrator (Dagster, Airflow, plain cron) instead of stretching a flow to cover it. See Why Repster? for how the two fit together.
What Repster owns and what you own¶
| Repster owns | You own |
|---|---|
| Execution (cloud workers) | Pipeline code |
| Scheduling (cron dispatch) | Connectors and sources |
| Secrets injection | Destination schema |
| Log capture and retention | .env secrets |
| Run state and history | pyproject.toml config |
| Infrastructure |
Your pipeline code runs unchanged. Repster adds the operational layer around it.