Run a closed test from the terminal: create a cohort, invite people, see where they stand, chase whoever needs chasing, and deploy the daily pass that does it on a schedule.
The dashboard is the paid tier; this is not. Every operation the control-plane routes expose has a command here, each non-interactive and each with --json.
This is the command surface of the testers capability; provisioning Testers stands the inbox up first.
Synopsis
pithy testers provision [--env <env>] [--worker <name>] [--json]
pithy testers deprovision [--env <env>] [--json]
pithy testers create <name> [--env <env>] [--target-size <n>] [--window-days <n>] [--max-roster <n>] [--platform <android|ios>] [--store-url <url>] [--json]
pithy testers list [--env <env>] [--json]
pithy testers invite <cohort> --email <address> [--email <address>…] [--name <name>] [--env <env>] [--json]
pithy testers pending <cohort> [--env <env>] [--json]
pithy testers roster <cohort> [--env <env>] [--json]
pithy testers status <cohort> [--env <env>] [--trend-days <n>] [--json]
pithy testers remove <cohort> --email <address> [--reason <text>] [--env <env>] [--json]
pithy testers close <cohort> [--env <env>] [--json]
pithy testers run [--cohort <cohort>] [--env <env>] [--skip-nudges] [--json]Which operations need a Cloudflare account. provision and deprovision always do — they deploy and delete a Worker. The nine roster subcommands read and write the database directly, so at the default --env dev they run entirely locally through Miniflare.
The roster commands talk to the database, not to the Worker over HTTP. The control-plane routes exist for a management client holding a credential; a developer at a terminal in their own repo already has the database, and minting a token to read their own roster would be ceremony with no security benefit.
Flags
--env is two different sets, deliberately. The roster subcommands default to dev and accept any environment, because they talk to a local D1 and dev is where you use them. provision and deprovision deploy a Worker to a Cloudflare account, where there is no dev to deploy to, so they take a managed environment only — and omitting the flag means every one.
provision and deprovision
| Flag | Default | Purpose |
|---|---|---|
--env <env> | every managed environment | Act on one environment only. --env dev is refused with the reason |
--worker <name> | the project’s only Worker | provision only. The app Worker whose wrangler.jsonc carries the per-environment DB binding and receives the TESTERS_DAILY binding. Required when a project has several |
--json | false | Machine-readable output |
create <name>
| Flag | Default | Purpose |
|---|---|---|
<name> | — | Required positional. A label for the cohort, e.g. closed-test |
--target-size <n> | 12, Play’s floor | Testers required simultaneously |
--window-days <n> | 14, Play’s window | Continuous days required |
--max-roster <n> | 100 | Roster cap |
--platform <android|ios> | android | Which store’s program this cohort serves |
--store-url <url> | the capability default | The store’s own opt-in link |
--env <env>, --json | dev, false | As everywhere |
The three numeric flags must each be a whole number above zero, and --platform must be one of the two. Both are checked at the flag rather than at the schema, so the answer names what you typed.
invite <cohort>
| Flag | Default | Purpose |
|---|---|---|
<cohort> | — | Required positional. The cohort name or id |
--email <address> | — | Required. Address to invite. Repeat the flag for several |
--name <name> | — | A display name for the roster, applied to every address in this run |
invite adds people to the roster and sends nothing. pithy testers run is what asks them whether they will test.
status <cohort>, remove <cohort>, run
| Flag | Applies to | Default | Purpose |
|---|---|---|---|
--trend-days <n> | status | 30 | How many daily snapshots to include in the trend series |
--email <address> | remove | — | Required. The tester’s address |
--reason <text> | remove | — | A short note recorded on the event |
--cohort <cohort> | run | every open cohort | Run one cohort only |
--skip-nudges | run | false | Advance state and record the day, but send nothing |
What it does
provision deploys the daily-pass Workflow worker — <project>-<env>-testers-daily — for each requested environment and then writes its TESTERS_DAILY Workflow binding into the app worker’s wrangler.jsonc. pithy add testers cannot write that binding — wrangler requires both a name and a class_name, and the deployed name is per environment. Preflight runs once before the first deploy, so failing means failing before one environment is half provisioned.
Its Worker deploy is gated. The Worker is deployed carrying a stamp naming the package version and a hash of its resolved configuration, and a run whose stamp matches both ships nothing. Anything the gate cannot establish — no Worker, no stamp, an unreachable account — deploys. pithy deploy --env <env> ships the same Worker without provisioning anything else, and pithy deploy --env <env> --kit --force re-uploads it regardless.
Each environment’s deploy resolves two ids first: the app DB id from that environment’s stanza in the app worker’s wrangler.jsonc, and this project’s shared email-suppression database, looked up by name — the daily pass reads it to reconcile bounced addresses, so pithy email provision has to have run.
An environment is ready when its stanza carries a DB binding with a database_id — that is, when pithy provision --env <name> has run for it. One that is not is skipped and reported, never fatal: standing staging up and proving it before production resources exist is the ordinary bring-up. --env and readiness are two different narrowings and both are reported — the flag says which environments you meant, readiness says which of those can be acted on. If every environment in the selection is skipped the run exits 3 (core/precondition_failed, with a fix) rather than reporting a success for a run that did nothing. The suppression database stays a refusal: it is project-global, and its absence means pithy email provision has not run at all, which no per-environment skip could express.
The sending identity is read off the composed email capability rather than asked for. When no email capability is composed, the host is still deployed and the pass records the day but sends nothing — said at provision time rather than discovered on the first silent morning. The pass runs daily at 05:00 UTC.
A Worker that owns what the host binds gets its own. When --worker names a Worker that lists a binding the testers host binds in its ownBindings, provision deploys that Worker’s own host, <project>-<env>-<worker>---testers, bound to the Worker’s own resources, and writes its Workflow bindings to reach it, <project>-<env>-<worker>---testers-<job>. The shared <project>-<env>-testers is left alone. The TESTERS_DAILY binding it writes names <project>-<env>-<worker>---testers-daily. See a Worker’s own kit hosts.
deprovision deletes the host’s daily Workflow (<project>-<env>-testers-daily), then the host, and every Worker’s own daily-pass host and its Workflow, found by recomputing their names, and nothing else. The Workflow goes first: deleting a Worker leaves the Workflows registered against it. The cohorts, the roster and the whole snapshot series are rows in your own D1 and are not this command’s to remove.
create creates a cohort, freezing the target, window, roster cap, platform and reset policy onto the row. They are stored rather than read from config at query time: raising targetSize from twelve to fifteen later would otherwise retroactively rewrite whether last Tuesday counted.
list, roster and status read. All three degrade rather than fail when the activity read cannot run — auth tables absent, or a transient D1 error — and say so on stderr at warn, which shows without --debug and leaves the --json contract on stdout untouched. Without that line, a whole roster reading never signed in is indistinguishable from a cohort nobody ever used.
invite adds addresses to the roster, refusing one already on it in a live state rather than quietly resetting the date the streak is measured from. A previously removed member is revived, keeping their id so their history stays attached to one person. A tester who withdrew is not revived at all.
pending prints the addresses that have agreed to test and are waiting to be added to the store’s own tester list. This is the one step Pithy cannot do for you: the Play Developer API has no way to add an address to an email list, so you paste these into the console. Until that is done the store link answers App not available, which is why the second email waits.
remove takes a tester off the roster. close closes a cohort: its roster and trend stay readable, and nothing further is sent.
run runs the daily pass now — advance state, chase whoever is due, record the day. Sends are real: the pass writes a row into the email job table and the email worker delivers it, so the CLI needs no sending domain of its own. The row stays pending until that worker’s every-minute scheduler picks it up, which costs up to a minute and loses nothing.
--json
One line, one object, one shape per subcommand. Every line also carries schemaVersion, an integer bumped only when a key is removed or retyped; pithy schema publishes the current one. A failure is one {"error": …} line on stderr, with exit, retryable and, where the cause is known, fix, and the process exits by its class.
testers provision and testers deprovision
| Key | Type | Meaning |
|---|---|---|
command | string | The subcommand that produced this line |
schemaVersion | integer | The version of this payload’s shape. Bumped only when a key is removed or retyped; pithy schema publishes the current one |
results | array | provision: one entry per environment acted on. deprovision: one entry per daily-pass worker deleted, in each environment torn down; an environment with none deployed has one entry, naming the shared worker |
results[].env | string | The environment this entry describes |
results[].worker | string | The daily-pass worker name, <project>-<env>-testers, or <project>-<env>-<worker>---testers for a Worker with a host of its own. On teardown, absent is success — it is idempotent |
skippedEnvironments | object[] | One entry per selected environment nothing was provisioned for |
skippedEnvironments[].env | string | The environment that was skipped |
skippedEnvironments[].reason | string | Why — env.<name> has no DB database_id. |
skippedEnvironments[].action | string | What resolves it. For any environment but production, a command that runs as written: Run pithy provision --env staging --yes. Production takes a --confirm phrase only a person gives, so its line says the run is a person’s decision rather than naming a command an agent can run alone |
sends | boolean | provision only. Whether an email capability is composed. false means the pass will record the day and send nothing |
exitCode | integer | provision only. The status the run exits with: 0. A run that provisions nothing writes no report |
One outcome, never two. A run that went ahead writes this report alone and exits 0, whether or not some environments were skipped — exitCode says so, and it matches $?. A run in which every selected environment was skipped provisioned nothing, so it writes nothing on stdout: the {"error":…} line on stderr is the whole answer, exit 3, with the provisioning command as its fix.
$ pithy testers provision --worker api --json
{"command":"testers provision","schemaVersion":1,"results":[{"env":"staging","worker":"acme-staging-testers"}],"skippedEnvironments":[{"env":"prod","reason":"env.prod has no DB database_id.","action":"`pithy provision --env prod --yes` with its --confirm phrase, which only a person gives, runs against prod, a production environment. That is a person's decision, not a fix."}],"sends":true,"exitCode":0}
$ echo $?
0$ pithy testers provision --worker api --json
{"error":{"code":"core/precondition_failed","status":412,"message":"No environment is ready — staging and prod have no DB database_id.","action":"Run pithy provision --env staging --yes to create its app database. `pithy provision --env prod --yes` with its --confirm phrase, which only a person gives, runs against prod, a production environment. That is a person's decision, not a fix. Once their app databases exist, this run is next: `pithy testers provision --json --worker api` runs against every environment, production included. That is a person's decision, not a fix.","exit":3,"retryable":false,"fix":"pithy provision --env staging --yes"}}
$ echo $?
3The second line is on stderr, and stdout is empty. Run without --env, the command acts on every environment, production included, so running it again is said as a person’s decision rather than named to run. Run with --env staging, the command to repeat carries it, and runs as written: pithy testers provision --json --env staging --worker api.
testers create and testers close
Both emit the whole cohort row, under the same key and the same shape.
| Key | Type | Meaning |
|---|---|---|
command | string | The subcommand that produced this line |
schemaVersion | integer | The version of this payload’s shape. Bumped only when a key is removed or retyped; pithy schema publishes the current one |
cohort.id | string | The cohort’s UUID. Text rather than a sequential id, which would leak how many test programs a project has run |
cohort.name | string | The human label |
cohort.targetPlatform | "android" or "ios" | Which store’s program this cohort serves |
cohort.targetSize | integer | How many testers must be opted in simultaneously. Frozen on the row |
cohort.windowDays | integer | How many continuous days the target must hold. Frozen on the row |
cohort.maxRosterSize | integer | The most members this roster may hold |
cohort.storeOptInUrl | string or null | The store’s own opt-in page. null until set — and until it is, nobody can actually join the test |
cohort.resetPolicy | "reset" or "pause" | Pithy’s assumption about what a dip below target does to the streak. Google documents neither behavior |
cohort.closedAt | ISO-8601 or null | When the cohort was closed |
cohort.createdAt | ISO-8601 | The zero point of its day axis |
cohort.updatedAt | ISO-8601 | When the row was last written |
testers list
| Key | Type | Meaning |
|---|---|---|
cohorts[].estimatedOptedIn | integer | Pithy’s estimate of the currently opted-in count |
cohorts[].estimatedHeldDays | integer | The estimated unbroken at-target run ending today |
cohorts[].targetSize, .windowDays | integer | The target and window in force for this cohort |
cohorts[].closed | boolean | Whether the cohort has been closed |
testers invite, pending, remove
| Key | Type | Meaning |
|---|---|---|
cohort | string | The resolved cohort id, whether you passed a name or an id |
invited[].id | string | invite. The member id. Stable across a revival, so history stays attached to one person |
invited[].email | string | invite. The invited address, normalized |
invited[].created | boolean | invite. false when the row already existed and was revived. Re-inviting is not an error, so this is how the two are told apart |
emails | string[] | pending. Every tester who has agreed to test and is waiting for the store’s tester list |
member | object | remove. The removed tester’s id and email |
testers run
| Key | Type | Meaning |
|---|---|---|
results[].cohortId | string | The cohort this entry describes |
results[].snapshotOn | string | The UTC day the pass recorded |
results[].nudged | object | Nudges enqueued by kind — confirm, store, inactive, closing. Enqueued, not delivered: that is the email worker’s job |
results[].estimatedOptedInCount | integer | Pithy’s estimate after the pass |
results[].estimatedHeldDays | integer | The estimated unbroken at-target run ending on this day |
results[].trendDirection | "improving", "steady", "declining" or "unknown" | The direction as of this pass |
results[].pruned | integer | Snapshots pruned by the retention policy in this pass |
results[].nudgesSkipped | "no_base_url" | Why nothing was sent, when sending was on and no link could be built. Absent otherwise, including under --skip-nudges — a pass that mailed nobody because it could not build a link looks identical to a pass where nobody was due, and those are very different states to be in on day 9 |
testers roster and testers status
Both emit one cohort view under cohort, and the view is the same shape the control plane serves a dashboard. The two differ in exactly two ways: roster includes cohort.members and loads no snapshots; status omits cohort.members entirely and loads the last --trend-days of them.
That difference reaches cohort.trend. Direction and every delta come off the most recent stored snapshot, so under roster — which loads none — the direction is "unknown", the deltas are null and the series is empty. Read the trend from status. fragile is the one field that still answers under roster: with no snapshot to read it is computed live from the clock.
The view carries five blocks beside the cohort’s own fields:
| Block | What it holds |
|---|---|
cohort.roster | Current composition: size, headroom to the cap, and a count per state — invited, accepted, optedIn, lapsed, unreachable, neverLinked. Headroom counts against live states only, so a lapsed member does not occupy a slot |
cohort.estimatedClock | Pithy’s estimate, named estimated* field by field so it cannot be mistaken for Google’s. Carries source: "pithy_estimate" as a literal discriminator, and dayBoundary: "UTC" — Google’s boundary is undocumented and may differ |
cohort.activity | The observed half, read from the auth tables. source: "observed" — this block is fact rather than estimate. Buckets testers by how long they have been quiet, with 8–13 days the strongest silent-uninstall signal |
cohort.projection | Pithy’s forecast. basis says why the forecast reads as it does and why any null in it is null; calibration: "default" says out loud that the survival priors are numbers Pithy chose rather than values fitted to your data; method is named and versioned, because a chart must not splice two methods into one line |
cohort.trend | Direction, four deltas, and the daily series |
Testers reference has the full field list.
Errors
The capability is not configured, or will not load — classified rather than assumed, exactly as the other provisioning commands classify it. No Worker composing testers is core/precondition_failed, exit 3. With one Worker its fix is pithy add testers --worker <worker> --with-prerequisites — testers declares email, and the Worker is named by its apps/ directory — and the action names the same command. With several there is no fix, because nothing says which Worker should compose it, and the action names one command per Worker, each runnable as written.
--env dev on provision or deprovision. Refused with the reason: there is no dev to deploy to.
A numeric flag that is not a whole number above zero, or a --platform outside the two. Checked at the flag, so the answer names what you typed.
A cohort that does not resolve. By name or by id, and the refusal says which were tried.
An address already on the roster in a live state. Refused rather than quietly resetting the date the streak is measured from.
Cloudflare credentials are missing, on the two commands that deploy, and on any roster command pointed at a deployed environment.
The project has no name. Every deployed name leads with it.
Examples
pithy testers create closed-test --store-url https://play.google.com/apps/testing/com.example.app
pithy testers invite closed-test --email ada@example.com --email grace@example.com
pithy testers run --cohort closed-test
pithy testers pending closed-test
pithy testers status closed-test --trend-days 14Then, once the daily pass should run without you:
pithy testers provision --env prod