pithy vector

Stand up an environment’s search indexes, rebuild them from the corpus, and re-embed what they hold.

This is the provisioning half of the vector capability — provisioning Vector has the account steps, and semantic search is what it is for.

Synopsis

pithy vector provision [--env <environment>] [--worker <name>] [--json]
pithy vector reset [--env <environment>] [--worker <name>] [--confirm-reset <phrase>] [--json]
pithy vector reprocess [--env <environment>] [--worker <name>] [--index <name>] [--all] [--filter <json>] [--json]

All three subcommands need a Cloudflare account, --env dev included. Cloudflare ships no local emulation for Vectorize, so a dev search reaches a real remote index — which is also why the capability’s bindings are declared remote.

Flags

FlagApplies toDefaultPurpose
--env <environment>all threedevThe environment to act on. dev is a real remote index here, not a local store
--worker <name>all threethe project’s only WorkerThe app Worker whose vector(...) registration declares the indexes, and whose wrangler.jsonc carries the per-environment DB binding, receives the vectorize and workflows bindings, and holds the VECTOR_PROVISIONED record. Required when a project has several
--confirm-reset <phrase>reset—Unlock a non-dev reset non-interactively. The exact, environment-named phrase: yes, i really want to reset <env>
--index <name>reprocessevery indexThe index to re-embed: a vector() index as named in pithy.config.ts, or one another capability owns, such as support-messages
--allreprocessfalseRe-embed every document, not only the ones whose model differs from config
--filter <json>reprocess—Narrow the run to matching documents, as a JSON metadata object. Parsed here, so a malformed filter fails in this terminal rather than inside a running Workflow
--jsonall threefalseOne line of machine-readable output

There is no --yes on reset, deliberately. --yes elsewhere means yes, this is not dev, and it was designed to authorize additive writes. A reset deletes every vector in an index, so it carries the stricter gate — the same one pithy seed --redo uses. dev is free; every other environment needs the exact phrase. A run that may not ask — --json, --no-input, PITHY_NONINTERACTIVE=1, or no terminal — without --confirm-reset is refused core/person_required, exit 5, and the action names --confirm-reset.

What it does

provision creates each index the config declares, or reuses one that already exists, and creates every metadata index the index’s schema marks filterable.

It returns only once each declared metadata index is actually live — listed under its own name and type on three reads in a row, which Vectorize gives no time for and provision waits up to ten minutes on — because the very next step deploys the Worker that writes vectors, and a write landing before an index exists is permanently unfilterable.

Then it deploys the prebuilt reprocess Worker for that environment and writes two things into the app’s wrangler.jsonc: the VECTOR_PROVISIONED var recording what it observed, and this environment’s vectorize and workflows bindings. pithy add vector cannot write those bindings — wrangler requires an index_name on one and a name plus class_name on the other, and all three are provisioning outputs.

Its Worker deploy is gated. The Worker is deployed carrying a stamp naming the package version and a hash of its resolved configuration, and a run whose stamp matches both ships nothing. Anything the gate cannot establish — no Worker, no stamp, an unreachable account — deploys. pithy deploy --env <env> ships the same Worker without provisioning anything else, and pithy deploy --env <env> --kit --force re-uploads it regardless.

The VECTOR_PROVISIONED var is what the Worker compares its declarations against at boot, refusing to serve on drift. That check is the only way somebody who edits a metadata schema and deploys without re-provisioning hears about it, because Vectorize answers such a filter with partial results and no error. The var is written last, and only on success, so it never claims more than provisioning got done.

A Worker that owns what the host binds gets its own. When --worker names a Worker that lists a binding the vector host binds in its ownBindings, provision deploys that Worker’s own host, <project>-<env>-<worker>---vector, bound to the Worker’s own resources, and writes its Workflow bindings to reach it, <project>-<env>-<worker>---vector-<job>. The shared <project>-<env>-vector is left alone. Its indexes are the Worker’s own too, <project>-<env>-<worker>---vector-<index>, since their vectors are keyed to the Worker’s own rows; reset and reprocess with that --worker act on them and on the Worker’s own reprocess Workflow, never the shared ones. See a Worker’s own kit hosts.

The refusal covers every route but the admin surface. It is a readiness guard, so every composed capability’s admin routes, the control plane’s ping and manifest included, keep answering, and a drifted Worker stays reachable by its management client. GET /health answers too. Everything else answers a 500 vector/metadata_index_drift. That surface is also where the drift is reported: GET /vector/indexes names each drifted field, distinctly from an index provision never recorded.

reset is the repair for the one failure Vectorize cannot repair in place: a metadata index added after vectors were written covers none of them, and there is no backfill. It deletes every configured index, rebuilds each through the ordinary provision path, and re-embeds the corpus from D1 — since the rebuilt index holds nothing.

Destructive by definition: only the D1 corpus survives, and only what it holds comes back. A non-dev reset is audited at critical severity, truthfully — recorded as failure when it dies partway, with the command still failing.

reprocess runs the re-embed Workflow for one index or for every configured one, and waits for each. With no --index it covers them all, which is what re-embed after a model change usually means. Every named index is checked against the config before the first run starts, so a typo costs nothing.

It covers the indexes other capabilities own, too. A capability that builds an index of its own declares it on its Capability (vectorIndexes), and reprocess reads those declarations off the Worker’s composition beside vector()’s config — @pithy-sh/support’s semantic message index, support-messages, is one. Such an index is re-embedded by the owning capability’s own reprocess Workflow, <project>-<env>-<capability>-reprocess, or <project>-<env>-<worker>---<capability>-reprocess when --worker names a Worker with that capability’s host to itself, with the same { index, all } every reprocess takes: the default pass re-embeds what its model stamp says drifted, and --all re-embeds everything. It works in a project that never composes vector(). Its vectors carry no metadata, so --filter with one is refused. An index in vector()’s config may not take a name a capability owns; the run is refused until one is renamed.

--json

vector provision

$ pithy vector provision --env staging --json
{"command":"vector provision","schemaVersion":1,"env":"staging","indexes":[{"index":"docs","indexName":"acme-staging-vector-docs","created":[{"propertyName":"ownerId","indexType":"string"}],"extra":[],"observed":[{"propertyName":"ownerId","indexType":"string"}]}]}
KeyTypeMeaning
command"vector provision"The subcommand that produced the line
schemaVersionintegerThe version of this payload’s shape. Bumped only when a key is removed or retyped; pithy schema publishes the current one
envstringThe environment provisioned
indexesobject[]One entry per configured index
indexes[].indexstringThe config’s name for the index
indexes[].indexNamestringThe Vectorize index name it was provisioned as
indexes[].createdobject[]The metadata indexes created on this run. Empty on a re-run, which is what idempotent looks like
indexes[].created[].propertyNamestringThe metadata property the index covers
indexes[].created[].indexType"string", "number" or "boolean"How Vectorize stores and compares the property’s values
indexes[].extraobject[]Metadata indexes that exist but the config does not declare. Not fatal; they still spend a slot
indexes[].observedobject[]Every metadata index live on the index when provisioning finished — declared and undeclared alike. This is what the VECTOR_PROVISIONED var carries, and what the Worker’s boot check compares against
indexes[].observed[].indexTypestringThe type Vectorize reports the index was created with, recorded verbatim rather than narrowed to the three types this package declares

vector reset

reset emits everything provision does — it rebuilds through the same path — plus two fields.

KeyTypeMeaning
command"vector reset"The subcommand that produced the line
schemaVersionintegerThe version of this payload’s shape. Bumped only when a key is removed or retyped; pithy schema publishes the current one
envstringThe environment reset
indexesobject[]Each rebuilt index, in the shape provision reports
deletedstring[]The indexes that were deleted and rebuilt, by their Vectorize names
reprocessedstring[]The reprocess runs started, one per configured index, by their config names

vector reprocess

$ pithy vector reprocess --env staging --index docs --json
{"command":"vector reprocess","schemaVersion":1,"env":"staging","runs":[{"index":"docs","report":{"indexName":"acme-staging-vector-docs","pages":2,"scanned":140,"reembedded":140,"skipped":0}}]}
KeyTypeMeaning
command"vector reprocess"The subcommand that produced the line
schemaVersionintegerThe version of this payload’s shape. Bumped only when a key is removed or retyped; pithy schema publishes the current one
envstringThe environment the runs happened in
runsobject[]One entry per index reprocessed, in run order
runs[].indexstringThe config’s name for the index
runs[].reportobjectThe Workflow’s own return value, passed through verbatim

runs[].report is the deployed Workflow’s output, not the CLI’s. It crosses no schema on the way out, and it is absent from its entry when a completed Workflow returned nothing. What the capability returns today:

report keyTypeMeaning
indexNamestringThe Vectorize index that was reprocessed
pagesnumberPages the run took — one Workflow step each
scannednumberRows read
reembeddednumberRows re-embedded and written back
skippednumberRows selected but skipped because the corpus holds no text for them

Errors

Each is a problem line then an action line. Under --json it is one {"error": …} line on stderr with exit, retryable and, where the cause is known, fix, and the process exits by its class.

The Worker does not compose vector. The target — --worker, or the project’s only Worker — has no vector(...) registration. The config is read off the Worker the run writes into and never off a sibling: the indexes created are the ones the Worker holding the VECTOR_PROVISIONED record declares. Named with --worker, it is a usage error, exit 2. The project’s only Worker, unnamed, is core/precondition_failed, exit 3, with pithy add vector --worker <worker> as its fix.

<worker> does not compose the vector capability.
Add `vector({ ... })` to <worker>'s pithy.config.ts (run `pithy add vector --worker <worker>`), or name the Worker that composes it with --worker.

Cloudflare credentials are missing.

No DB id for this environment. The document corpus lives in the app database, so provisioning refuses rather than standing up an index with nothing to fill it.

A metadata index is accepted but not live yet. Ten minutes passed without three agreeing reads. Vectorize applies a metadata index asynchronously, and this one has not landed yet. Write no vectors, and re-run provision. It creates nothing twice, and it waits again.

The metadata indexes on `acme-staging-vector-docs` were accepted but are not live yet.
Re-run `pithy vector provision --worker api --env staging` in a minute. Do not write vectors until it reports them ready.

The command is this run’s own, with its Worker and environment. In a production environment it is not offered as a fix: the line names it and says the run is a person’s decision.

An illegal --env. Checked at the flag, before any config is loaded or any Cloudflare call is made.

A reset that is not confirmed. The gate runs before a single Cloudflare call, so a refused reset costs nothing.

Resetting staging destroys all of its data.
Pass --confirm-reset "yes, i really want to reset staging" to drop and recreate the staging schema.

A supplied-but-wrong phrase is refused separately, and the phrase names its environment — one authorizing a staging reset cannot be pasted into a command targeting another. Either refusal is core/person_required, exit 5: the phrase is a person’s to give.

An unknown --index. Raised for every named index before the first run starts, listing the ones that are configured.

A malformed --filter. An array and a bare scalar are both refused.

The project has no name. An index is found by name and reused, so the name must be identical on every run, and it is never guessed: a wrong one adopts another project’s corpus.

The Workflow did not finish. reset and reprocess wait on dispatched instances and raise when one ends errored or terminated, or is still running at the poll cap.

Examples

$ pithy vector provision --env staging
docs: acme-staging-vector-docs ready, 1 metadata index(es) created.
Done.

An index carrying a metadata index the config no longer declares says so, and does not remove it:

$ pithy vector provision --env staging
docs: acme-staging-vector-docs ready.
  legacyTag is indexed but not declared. It still costs a slot.
Done.
pithy vector reprocess --env staging --all
pithy vector reprocess --env staging --index docs --filter '{"ownerId":"ada"}'

Re-embed support’s message index after changing search.semantic’s model, in a project with no vector():

$ pithy vector reprocess --env staging --index support-messages
support-messages: 312 re-embedded of 312 scanned.
Done.
$ pithy vector reset --env staging --confirm-reset "yes, i really want to reset staging"
DESTRUCTIVE. Every vector in staging was deleted and rebuilt from the corpus.
docs: acme-staging-vector-docs rebuilt and re-embedded.
Done.
ESC