Skip to content
mdlake

Troubleshooting

Messages and symptoms you can meet while running the CMS, building a site, deploying it and using forms and backends, with the cause and the fix for each. For developers and admins.

On this page

This page lists problems by where they show up. Each row gives the message or symptom as it appears, why it happens and what to do. Messages are quoted from the tools, so you can search this page for the text you see. If your problem is not here, run npm run check in the site and read the first error: the checks are written to say what is wrong and what to run.

How to read a failing check

npm run check runs several checks in a row and stops at the first one that fails. Run the failing part alone to see more:

PartCommand
Ruleset versionnpm run check:ruleset
Ownershipnpm run check:ownership
Hostingnpm run check:hosting
Backend contract (split)npm run check:api
Typesnpm run check:types
Buildnpm run build
Bindings and contentnpm run check:bindings
Accessibilitynpm run check:a11y (needs a build)

Lines start with error, warning or note. Only errors fail the check.

The CMS

Docker Compose

SymptomCause and fix
set MSSQL_SA_PASSWORD in .envCompose requires the variable. Create .env next to docker-compose.yml and set it.
set SEED_ADMIN_EMAIL in .envThe full profile requires the first administrator’s address. Set it in .env.
The sql container is unhealthy for a minute after startIt has a start period of 20 seconds and checks every 10. On Apple Silicon it runs under emulation (linux/amd64) and is slower. Wait, then check docker compose ps.
The API stops at start with a setting name in the messageA required setting is missing or out of range. See CMS configuration. Read docker compose logs cms-api.
Connection string 'Cms' is missing.ConnectionStrings__Cms is not set. Compose sets it from MSSQL_SA_PASSWORD; outside Compose set it yourself.
Everyone is signed out after a rebuildThe Data Protection keys were lost. Set DataProtection__KeysPath and keep the folder on a volume (Compose does this). On a server, keep the keys in Blob Storage with DataProtection__BlobUri.
The database has the old schemaThe API applies migrations at startup only in Development. In other environments run the migration step before the new version takes traffic: the migration job of the pipelines, or the bundle or script from cms/scripts/build-migrations.sh. See Install the CMS.

On a server

The setup is in Install the CMS. For deploys, rollback and restores see infra/cms/README.md.

SymptomCause and fix
GET /health/ready answers 503, or a new revision never takes trafficOne of the checks database, blobStorage, lakeFolder, converter or dataProtection failed. Sign in as an admin and open /health/ready to read the description. “The database is missing N migration(s) of this build” means the migration step has not run for this version. Run it, then the revision turns ready. Meanwhile its background workers (publish, import, mirror push) claim no work, and the same check reports workersWaitingForMigration. “The lake repository folder … does not exist (is the share mounted?)” means the lakes volume is not mounted.
The API stops at startup with “Set exactly one of Storage:ConnectionString … and Storage:ServiceUri”Blob Storage is reached one way only. Remove the leftover Storage__ConnectionString (also a Storage--ConnectionString secret in Key Vault) when you use Storage__ServiceUri, or the other way round.
Uploads or /health/ready (blobStorage) fail with 403 AuthorizationPermissionMismatch or AuthenticationFailed while Storage__ServiceUri is setThe identity lacks Storage Blob Data Contributor on the storage account (role assignments take a few minutes to apply), or Storage__ManagedIdentityClientId / AZURE_CLIENT_ID names another identity than the one that has the role.
The API stops at startup with “Data Protection has nowhere to keep its keys”Outside Development the API needs a key store. Set DataProtection__BlobUri with DataProtection__KeyIdentifier, or DataProtection__KeysPath, not both. See CMS configuration.
Publishing is refused with “Connect the lake repository first”Remotes:RequireMirror is on and the workspace has no connected mirror. An admin connects it under Workspace settings, Lake repository. See Connect GitHub. At startup, “Remotes:RequireMirror is on, but no Git host is configured” means Git:Remote:Host is not set.

Sign-in

SymptomCause and fix
No sign-in mail arrivesLocally every mail lands in Mailpit at http://localhost:8025. In production check Email:Provider, Email:From and Email:Resend:ApiKey, and that the sender’s domain is verified in Resend.
Asking for a sign-in link fails with HTTP 429The magic-link limits apply per email address (3 per 15 minutes) and per client address (10 per 15 minutes). Wait, or raise RateLimiting:MagicLink:*. Behind a proxy, set App:TrustForwardedHeaders so the limit sees visitors and not the proxy.
Passkey ceremony refused: the credential's origin ... is not an allowed editor originThe browser’s address is not in App:EditorBaseUrl or Cors:Origins. Add it.
A passkey stops working after a change of addressPasskeys:ServerDomain is part of the passkey. After changing it, sign in with a magic link and register a new passkey.
The first administrator cannot sign inThe seeder creates the user only when no administrator exists yet, and logs Seed:AdminEmail is not set; skipping admin seed. when the setting is empty. Check Seed:AdminEmail and the log. The administrator has no password: sign in with a magic link sent to that address.

Publishing, mirror and media

SymptomCause and fix
Publishing is slow or stuckA publish is queued and written by a worker that polls every Publishing:PollIntervalMs. A run whose heartbeat is older than Publishing:StaleAfterMinutes is treated as abandoned and failed, so the queue moves on. Check that Publishing:RunWorker is not false.
GitHub:AppId is required. or GitHub:PrivateKeyPath does not point to a file. at startupGit:Remote:Host is GitHub, so all GitHub settings are required. Set them or set the host to None. See Connect GitHub.
The mirror repository is behindPushes are retried with a growing wait, from Git:Remote:RetryInitialSeconds up to RetryMaxMinutes. Use Check access in the workspace settings to see the cause.
Markdown or Word import refusedThe upload hit a limit. See MarkdownImport:* and Assets:MaxUploadBytes in CMS configuration.
AI suggestions are missingAnthropic:ApiKey (or ANTHROPIC_API_KEY) is not set, Anthropic:Enabled is false, or the workspace’s AI switch is off.

Fetching the lake

These come from mdlake prepare, which runs before dev, build and the checks. More detail in The lake in CI, and caches.

MessageCause and fix
git was not found. The site fetches its lake with the system gitInstall Git and open a new terminal.
Could not read the lake at ... git was not allowed inYour Git account cannot read the lake repository, or in CI the token is missing, expired or too narrow. Locally: git ls-remote <remote>. In CI: MDLAKE_LAKE_TOKEN.
The commit locked in lake.lock is not in <remote>The CMS has not mirrored that commit yet, or the history was rewritten. Check the mirror in the CMS, or lock a commit the remote has.
Expected the full 40-character git commit hash (also lake.lock must name a full commit hash (40 characters))lake.lock holds an abbreviated hash. Run the refresh flow, or node "<plugin folder>/scripts/write-lake-lock.mjs" <site> --commit <hash>, which writes the full hash from the managed clone.
lake.lock must not contain credentials in its remoteRemove the user or token from the address. Use MDLAKE_LAKE_TOKEN in CI.
The lake at ... is at ..., but lake.lock locks ...The lake is not at the locked commit. Run npx mdlake prepare. To take newer content on purpose, run the refresh flow.
The lake folder .mdlake/lake does not existRun npx mdlake prepare.
LAKE_PATH (...) has origin ..., but lake.lock records ...A warning. LAKE_PATH points at a clone of another repository. Check it, or unset it to use the managed copy.
WARNING: LAKE_LOCK_IGNORE=1, building without checking lake.lockSomeone set the emergency variable. Do not deploy that build.

Building and checking a site

Bindings and content

Message or symptomCause and fix
error lines from check-bindings that name a file and a reasonThe loader found a lake file that is invalid, has a repeated ID or cannot be parsed. Fix it in the CMS, publish, and refresh.
A page, section or link disappeared after a refreshThe refresh report lists it. See Safe content refresh.
Generated content: lines marked stale or hand-editedThe sources of generated copy changed, or the file was edited. Regenerate it with the generate-copy skill, or adopt it.
An asset fails the build with its ID and the pages that use itThe runner cannot download it, or the ID has no sidecar. Check the sidecar’s url and the storage firewall.
Images are blurry or oversizedASSET_OPTIMIZE=false copies the originals. Remove it for production.

Ownership

MessageCause and fix
error for a hand-edited kit fileA kit file was changed. Revert it and override the component in site.config.ts. See Custom components.
warning for a hand-edited generated fileKeep the edit with npx mdlake adopt <file>, or regenerate. See Ownership zones.
site-snapshot.json is missing or is older than lake.lockBuild the site and run npx mdlake snapshot. Commit it.
.mdlake/types/lake.d.ts is missingRun npx mdlake prepare. npm run dev, build and check do.
provenance.json has no ownership zones yetRun npx mdlake migrate.
A skill reports that a file was not written and exits with code 3The file is adopted or was edited. The skill prints what it would have changed. Apply that by hand, or choose --overwrite if you agree to lose your edit.

Ruleset and upgrades

MessageCause and fix
This site uses ruleset X; the plugin is Y (N versions behind). Run /mdlake-builder:upgrade-ruleset.The plugin is newer than the site. Run the skill, or npx mdlake migrate. See Upgrade and migrate.
This site uses ruleset X, which is newer than the pluginUpdate the plugin.
Could not find the mdlake-builder plugin, so the ruleset version wasn't checkedSet MDLAKE_PLUGIN_ROOT or install the plugin. With --max-behind this exits 2.
Button colours changed after an upgradeRun npx mdlake migrate --buttons --check, then the steps in the upgrade-ruleset skill, and compare-buttons.

Accessibility

MessageCause and fix
No pages in dist/. Run npm run build first.Build first.
Cannot launch a browserRun npx playwright install chromium (--with-deps on Linux), or set PLAYWRIGHT_CHROMIUM_EXECUTABLE_PATH.
A serious or critical violationThe message names the page, the rule and the element. Images need alt text from the lake: an empty alt is content an editor must add.
An alt text that starts with “Photo of” or “Image of”The check and the CMS warn about words that name the kind of file. An editor should reword the alt text.

Hosting and deployment

The hosting check

MessageCause and fix
... is an on-demand route outside src/pages/api/Server code lives only in src/pages/api/** and src/server/**. Move the route, or remove export const prerender = false.
... is in src/pages/api/ but is built as a static fileAdd export const prerender = false, or move the file out.
The site has on-demand routes (...) but .mdlake/hosting.json does not exist (or says static)Run npx mdlake prepare and commit .mdlake/hosting.json.
.mdlake/hosting.json says the site has on-demand routes, but none is leftRun npx mdlake prepare, then npx mdlake hosting init --ci ... so the pipelines and Bicep are static again.
astro.config.mjs sets an adapter, but the site has no on-demand routeRemove adapter. The kit adds @astrojs/node when a route needs it.
... sets an adapter other than @astrojs/nodeThe pipelines deploy the Node standalone server. Remove adapter, or adopt your own hosting.
<file> (GitHub Actions workflow) is for a static site, but this is a server siteRun npx mdlake hosting init --ci github. A hand-edited file is refused; adopt it or use --overwrite.
No pipeline or Bicep yet for this server siteA note, not an error. Run the setup-hosting skill.
.mdlake/hosting.json says the site is split but names no backendRun backend init again, or restore the file from Git.

See Choose a hosting model for the other split-site messages.

Bicep and az

SymptomCause and fix
The subscription is not registered to use namespace 'Microsoft.Web' (or Microsoft.KeyVault)Register the provider once: az provider register --namespace Microsoft.Web.
A site or Web App name is takenNames are global. Change name in the parameter file. The Key Vault name is <name>-kv cut to 24 characters, and soft delete keeps a deleted vault’s name for 30 days.
The Static Web App location is refusedStatic Web Apps exist in a few regions only. The generated default is westeurope.
Linking the backend fails in the deploymentLinked backends need the Standard plan of Static Web Apps. The generated Bicep sets it for the split model; check that nobody changed it.
az keyvault secret set is refusedGive yourself Key Vault Secrets Officer on the vault.
The app shows @Microsoft.KeyVault(...) as a value or cannot read a secretThe secret does not exist yet, or the identity cannot read the vault. Set the secret, then restart the Web App.
A re-deployment asks for secrets againSecrets are passed on the command line and never stored. Pass them again, or set them in the vault and leave secrets out.

Pipelines

SymptomCause and fix
The pipeline does not startThe file is not on main, or the push was to another branch.
Login failed or No matching federated identity record found (GitHub)The federated credential’s subject does not match the repository and branch.
AuthorizationFailed on the Web App stepThe identity needs Website Contributor on that Web App.
The Static Web Apps step says the token is invalidRead the token again with az staticwebapp secrets list and update the secret.
The build is slow every timeThe caches do not hit. See The lake in CI, and caches.
The live site has old contentThe site shows the commit in lake.lock. Publishing in the CMS does not change it.
The API answers 403 when called directlyAfter linking, the API accepts traffic only from the Static Web App. Change its Authentication settings to test directly.

Forms, mail and Turnstile

SymptomCause and fix
The form answers 404On the server model the route src/pages/api/forms/[name].ts is missing. On a split site the backend does not serve that form.
The form answers 422A field failed validation. The answer lists the errors per field.
The form answers 429The rate limit: 5 submissions per 10 minutes per address by default, counted per server instance.
The form answers 403 captcha-failedTurnstile rejected the answer. With test keys, the site key 2x00000000000000000000AB always fails.
The form answers 500 captcha-not-configuredTURNSTILE_SECRET_KEY is not set in production.
The form answers 502 captcha-unavailableThe server could not reach Cloudflare. Turnstile fails closed.
No mail arrivesCheck MAIL_PROVIDER, RESEND_API_KEY, MAIL_FROM (on a verified domain) and the recipient variable. With MAIL_PROVIDER=log the mail is printed in the server output.
Every visitor shares one rate limitTRUSTED_PROXY_HOPS is wrong. Behind Azure’s proxy the default is 1; the split API needs 2.

More in Forms, Mail and Turnstile.

The separate backend

See the table in A separate backend. The three that come up most:

SymptomFix
src/api/generated/api.ts is stalenpm run api:generate, then commit.
api/openapi.json is out of datenpm run api:openapi, then npm run api:generate, and commit both.
calls /api with a hand-written fetchUse the generated function from src/api/generated/api.

Preview

MessageCause and fix
mdlake preview needs MDLAKE_API_URL and MDLAKE_TOKENSet them in .env. The token needs the preview scope. See Workspace tokens.
The port is in useThe CMS’s preview address names one port. Stop the other process, or change the address in the workspace settings.
astro build refuses MDLAKE_PREVIEWA preview build contains drafts and must never be deployed. Unset it. For a static preview on a private host, use --preview-deploy.
A preview link to an image stops workingSigned links to unpublished assets expire after Tokens:SignedUrlMinutes (15 by default). Reload the preview.

What’s next