You rotate a production API key. The provider confirms it: new key issued, old key dead. Then the write to your secret store fails.
The new value exists in one place — the process that is about to exit. The old value no longer works anywhere. Your Worker is holding a credential that stopped being valid a second ago, and no amount of retrying will fix it, because retrying the roll produces a third key and loses the second.
pithy add secrets gives you encrypted secret storage. The interesting part is not the encryption. It is that the whole rotation path is shaped around that one paragraph.
The ordinary half
Every secret is sealed with AES-256-GCM. The master key is read from a Cloudflare Secrets Store binding and never leaves the Worker — every encrypt and decrypt runs in process, so there is no round trip that could carry plaintext anywhere.
There are two version axes, and keeping them apart matters:
- The encryption-key version — which master key sealed a given row. Each row records its own, so a rotation job can re-encrypt under a new key with an overlap window instead of a flag day.
- The value version — which generation of the secret itself you are reading.
Both use the same envelope shape: { currentVersion, versions }. That explicit pointer is not decoration. Version keys are stringified integers, and if you sorted them to find the newest you would eventually discover that "10" < "2". The pointer means nothing ever sorts them.
The half that is about one failure
Rotating a value means two systems have to agree: the provider that issues it, and the store that holds it. They can disagree in exactly one direction that cannot be walked back.
So the ordering is the design, not a caveat on it.
1. Refuse everything refusable before anything is called. An unknown keyspace, an undeclared rotation, a provider secret with no rotator configured, a missing master key — each is answered with nothing rolled and nothing written. A refusal after a roll would be the worst of both worlds: the old credential dead, the new one rejected on a technicality you could have caught first.
2. Produce the value exactly once. local secrets mint it, provider secrets call the rotator. One call. It is never repeated, for any reason.
3. Store with retries — against that same value. Every attempt writes the identical string. If the store cannot be made to accept it, the run ends. It never reaches back for a fresh value, because a fresh value would orphan the one the provider already committed to.
Read those together and the shape is clear: all the retrying happens on the side that is safe to retry.
Two failures that look alike and are not
When something goes wrong, the run reports which environments the new value reached and which it did not — and it distinguishes two outcomes that a less careful tool would merge.
| outcome | what is true | what you do |
|---|---|---|
failed | nothing rolled; the old value is still live | run the command again |
unrecorded | rolled at the issuer, not written to the store | go to a console now |
Those demand completely different responses, which is why they are separate members rather than one error with a message. failed is an inconvenience. unrecorded is an outage with a clock on it.
A tool that prints “3 of 4 rotated” over that distinction has told you the least useful true thing available.
The decision with a real cost
Here is the part I would push back on if I had not read the reasoning, and the reasoning is why I did not.
When the store refuses the new value, the value is discarded. It is not printed to your terminal. Not written to a file. Not attached to an audit event. Not returned to the caller — the outcome type has no field that could carry one.
You just rolled a production credential and the tool throws it away.
The alternative is worse, and it is worth being explicit about why. A live production credential printed to a terminal is in your shell scrollback, in your CI logs, and in every terminal-recording buffer on the machine. That is a permanent leak, and you cannot un-leak it. A rolled-but-unrecorded credential is an outage with a known remedy the tool can name for you: roll again at the issuer by hand, then run pithy secrets update.
What you get day to day
Most of the time none of the above is visible, which is the point.
Secrets are declared in a registry and read through two accessors in your Worker. pithy add secrets provisions the store, wires the bindings, and mints the manager’s own runtime credential with the narrowest scope that works — Secrets Store read and write, nothing else.
Other capabilities read their credentials through the same seam. This is why payments will not compose without secrets: four payment rails need four sets of provider credentials, and there is exactly one place they come from rather than four opinions about environment variables.
At-rest key rotation runs as a Cloudflare Workflow, which is the right shape for it: re-encrypting a whole store is long-running work that must survive a restart and resume where it stopped. It re-encrypts under a new master key while both keys remain valid, so nothing has to stop while it runs.
The rest of the time, the thing worth knowing is that somebody already thought carefully about the ten seconds in which your credential exists in exactly one place.