The lake in CI, and caches

On your machine the build reads the lake from a clone that the kit manages for you. In CI there is no such clone, so the pipeline has to fetch the lake first, with a token that can read the lake repository and nothing else, and it should keep the slow parts between runs. The pipelines from npx mdlake hosting init already do all of this. This page explains what they do, so you can set up the token, change a pipeline you adopted, or fix a failure.

For the concepts behind it, see How sites work.

How the lake reaches the build

A site has a file lake.lock, committed in the repository:

{
  "lake": {
    "workspace": "acme",
    "remote": "https://github.com/acme-org/lake-acme.git",
    "commit": "9b1c0e2d7f4a6c3b8e5d1a0f2c4b6d8e9a7c5b3d",
    "lockedAt": "2026-10-20T09:14:00Z"
  },
  "mode": "snapshot"
}

npm run check starts with mdlake prepare, and so does npm run build (through its prebuild script). prepare does this:

1. If LAKE_PATH is not set and lake.lock has a remote, it makes the folder .mdlake/lake a git repository whose origin is that remote.
2. If the folder is already at the locked commit, it does nothing and does not touch the network.
3. Otherwise it fetches exactly that commit (git fetch --depth 1 origin <commit>) and checks it out. The fetch is shallow, so only that commit’s files are downloaded, not the history.

The commit must be the full 40-character hash in lower-case hexadecimal. A short hash cannot be fetched, so lake.lock with one is invalid.

In snapshot mode (the default) the build then verifies that the lake is at the locked commit, and fails if it is not. In live mode the check is skipped. See How sites work.

Two rules follow for CI:

• Do not set LAKE_PATH in CI. It overrides the managed clone, and the build would read whatever folder it names. Leave it unset so the commit in lake.lock is what you build.
• The remote in lake.lock must not contain credentials. prepare refuses a remote such as https://user:token@github.com/.... Use the plain URL and pass the token as MDLAKE_LAKE_TOKEN.

The lake token

The lake repository is private (lake-<slug> under the owner you set in the CMS; see Connect GitHub). The runner needs read access. Give it a token that can read this one repository and nothing more.

Requirements on the runner

• git installed. GitHub-hosted runners and Microsoft-hosted agents have it. A self-hosted agent might not. The error then reads “git was not found”.
• Network access to the lake’s host (github.com).
• The variable MDLAKE_LAKE_TOKEN, set as a secret.

Create a fine-grained GitHub token

1. In GitHub, open Settings, Developer settings, Personal access tokens, Fine-grained tokens, and choose Generate new token.
2. Set the Resource owner to the owner of the lake repository (the user or organization).
3. Under Repository access choose Only select repositories and select the lake-<slug> repository.
4. Under Permissions, Repository permissions, set Contents to Read-only. Leave everything else at no access.
5. Set an expiry. Put the date in your calendar: when the token expires, every build fails until you replace it.
6. Generate the token and store it as MDLAKE_LAKE_TOKEN: as a repository secret in GitHub Actions or a secret variable in Azure DevOps. See GitHub Actions and Azure DevOps.

A token that belongs to a person stops working when that person leaves the organization. A GitHub App installation token avoids this. The CMS’s own GitHub App writes to the lake and is not meant for this. Create a second app with Contents: read-only permission, install it on the lake repository, and generate an installation token in the pipeline before the check step. The kit sends MDLAKE_LAKE_TOKEN as HTTP Basic authentication with the user name x-access-token, which is also the form GitHub expects for an installation token.

How the token is used

The kit hands the token to the one git fetch process that talks to the remote, as an HTTP header for the remote’s host only. It does not appear in the command line, in .git/config or in the log. Error messages that quote the remote or git’s output have the token removed.

What to cache

A cold build downloads the lake and every asset, and converts every image. A warm build skips all three. Three folders are worth keeping:

All three are keyed on lake.lock. The key changes with every new lake commit, so a refresh starts with an empty cache, and the next builds of that commit are warm. A stale cache would only cost time: image entries are keyed on the source image and download entries on the asset id and the sidecar’s content, so a cache can never serve the wrong bytes.

The generated steps look like this in GitHub Actions:

- name: Cache the lake clone and the image caches
  uses: actions/cache@v4
  with:
    path: |
      .mdlake/lake
      node_modules/.astro/assets
      node_modules/.cache/mdlake-assets
    key: mdlake-${{ runner.os }}-${{ hashFiles('lake.lock') }}

And in Azure DevOps, three Cache@2 tasks with keys of the form mdlake0 | "$(Agent.OS)" | lake.lock. Both come after npm ci, because npm ci deletes and recreates node_modules.

Keep the cache step after npm ci and before npm run check if you rewrite the pipeline.

Cache limits

• GitHub removes a cache that has not been used for seven days. Each repository also has a total size limit (10 GB by default), and GitHub removes the oldest caches when it is exceeded. These are GitHub’s limits, not mdlake’s; they can change, so see GitHub’s documentation for the current values.
• Azure DevOps caches are scoped to the pipeline and the branch, with a fallback to the default branch.
• A pull request in GitHub cannot write a cache that the main branch reads. The generated workflow does not run on pull requests, so this does not matter for it.

The asset host must be reachable

Pages never link to the asset host. The build copies the files they use into dist/files and the pages link to /files/<id>/<name>.<ext> on your own domain. To do that it downloads each asset from the url in its sidecar (the public address the CMS wrote, in Blob Storage), or from PUBLIC_ASSET_BASE_URL when you set it.

So the runner must be able to reach that address. If the storage account sits behind a firewall or a private network, add the runner’s address to the allow list, or put a CDN in front of the storage and set PUBLIC_ASSET_BASE_URL to it (as a pipeline variable). An asset that cannot be downloaded, or an asset:// link whose sidecar is missing, fails the build with the asset id and the pages that use it.

Other variables in the build

See Site configuration for all site variables.

Check a pipeline locally

You can reproduce what CI does from a clean state:

rm -rf .mdlake/lake node_modules/.astro node_modules/.cache/mdlake-assets
MDLAKE_LAKE_TOKEN=<token> npm run check

Remove-Item -Recurse -Force .mdlake/lake, node_modules/.astro, node_modules/.cache/mdlake-assets -ErrorAction SilentlyContinue
$env:MDLAKE_LAKE_TOKEN = "<token>"
npm run check

You see the same messages as the pipeline: “Fetching lake commit … from …” and, at the end, “lake … fetched …”. Unset LAKE_PATH in your shell and .env first.

Problems

What’s next

• GitHub Actions and Azure DevOps for the pipelines.
• Refresh content to move lake.lock to newer content.
• Troubleshooting.