Railway Told Every Customer to Rotate Their Keys. Here Is What That Means.

The Platform Paused Builds Because of Secrets

On October 20, Railway’s monitoring paged an engineer for abnormal latency. By that evening the company had decided an outside party was trying to escalate access inside its build systems, and it took builds offline. The public status page followed. For the next two days, customers could not ship a new deploy, and for long stretches the dashboard, API, and CLI were down with them.

Running apps stayed up. The thing that stopped was the path used to change them.

Railway’s preliminary report, published October 22 by Jacob Cooper and Angelo Saraceno, says the company believes the intruder’s access ran from October 20 at 13:58 UTC until October 22 at 03:32 UTC, when Railway confirmed the intruder was out of internal systems. The status incident was marked resolved on October 23 at 22:06 UTC. Builds, deployments, the API, and the dashboard were operating normally. New signups were still paused, and the support site station.railway.com was still down under a separate notice.

The outage was the visible part. The part that lasts longer is the keys.

What Railway Says the Intruder Got

The report is still a preliminary one. Railway says a forensic investigation is ongoing, that outside incident-response specialists and authorities are involved, and that it is withholding exploit detail while that work continues.

What it does say:

• The campaign was noticed on October 20 at 19:53 UTC, after an earlier page at 13:58 UTC. Railway’s early read is that the intruder was after AI provider API keys, to run distillation attacks. TechCrunch reported the same motive from the company’s account to its customers: a broader pattern of attacks aimed at AI inference providers.
• On October 21, between 04:24 and 05:23 UTC, the intruder pulled secrets in encrypted form and, at that moment, could not decrypt them.
• Between 05:27 and 07:25 UTC, the intruder reached Railway’s CI environment and decrypted artifacts that held further API access.
• Between 07:25 and 09:18 UTC, Railway found an unauthorized query of encrypted environment variables. A subset were decrypted. Because the values were envelope-encrypted, that query does not tell Railway which individual key inside an affected project was read.
• Around 09:18 to 09:25 UTC, staff read logs from the intruder’s payloads and saw the API key names being searched for. At 09:25 UTC Railway notified vendors and asked them to revoke keys, before the public disclosure later that day.
• Railway user accounts were not logged into. Source code access is not ruled out; the company says it will confirm that in the final report.
• All Railway API tokens and project tokens created before the incident were invalidated. Storage bucket credentials were rolled. An audit-log entry from Railway employee Faraz Patankar for that bucket reset is expected.

The Hacker News also reported that security researcher Taylor Monahan saw cryptocurrency wallets drained where private keys had been stored on Railway, with the first known theft at least two hours before Railway’s public disclosure. Railway’s own preliminary report does not describe those thefts. The practical reading is the same either way: a private key, a database password, or an AI token sitting in an environment variable is a live credential, not a config detail.

“1.85%” Is a Ceiling, Not a Clean List

Railway says its design is meant to keep one compromised key set from opening the whole platform. On that basis it says at most 1.85% of workspaces had some environment variables potentially compromised. That is about 100,000 workspaces.

Two groups sit higher in its forensics, because the intruder appeared to be hunting high-value targets:

• Workspaces with a very large number of environment variables
• Environments holding a lot of AI vendor keys

Railway has been emailing customers it believes were directly affected, and a wider set it considers high risk. On October 23 the status page said notifications for workspaces that met the increased-risk criteria were complete as of 18:57:54 UTC that day. It also said workspaces outside those groups are not confirmed safe. If later findings change a workspace’s status, Railway said it will contact that customer. If you already rotated after the incident was contained, the October 23 note does not ask you to do it again.

A revocation email from GitHub, GitLab, Anthropic, OpenAI, OpenRouter, Stripe, or another provider is a different signal. Railway says it worked with those companies, and with SpaceXSI and others, and that for some vendors it handed over every key stored on the platform, not only the ones it could tie to the decrypted subset. More people were caught in upstream resets than the 100,000 figure. A vendor email means the key was on the list Railway sent. It does not, by itself, mean that workspace was one of the decrypted ones.

What to Do If You Deploy There

Railway published provider-specific guides on its engineering blog for GitHub, AWS, GCP, Stripe, OpenAI, Anthropic, OpenRouter, Grok, and for Postgres, MySQL, MongoDB, and Redis. The short version, for a team that is not going to read twelve guides before lunch:

If a provider already killed the key, create a new one and deploy that. Do not paste the old value back in because production is down. Production is down because the old value is no longer trustworthy.

The Lesson Is Where the Secrets Live

A hosting platform is a reasonable place to inject configuration. It is a bad place to be the only copy of a secret you cannot rotate in an afternoon.

The teams that got through this cleanly already knew three things: which keys existed, which provider could revoke each one, and who was allowed to put the new value into production. The teams that got stuck had a database password in an environment variable, a public database port, and nobody sure which GitHub app was allowed to deploy.

That inventory is ordinary operational work. It is also the difference between a status-page afternoon and a week of guessing which integration is still using the old token.

We see the same shape in custom systems we take over. The app is fine. The accounts around it are shared, unnamed, and one incident away from a full stop. If that is the state of a tool your staff depend on, a systems review is where we list the credentials, the host, and the person who can rotate both.

Book a Free Systems Review Keeping Software Running After Launch