Connect GitHub

Every lake lives in a Git repository that the CMS keeps on its own disk. You can use mdlake that way for ever, unless the CMS is set to require a mirror (see On a server). Connecting GitHub adds a mirror: after every publish the CMS pushes the lake to a private repository named lake-<short name>, so developers, the builder and backups get a normal Git remote. This page is for admins and developers. You set up a GitHub App once, give its details to the CMS, and then connect each workspace.

The CMS only pushes. It never forces a push and never overwrites what is on GitHub. Publishing never waits for GitHub: if GitHub is down, publishes still work and the push is retried.

Before you start

• The CMS runs and you can sign in as an admin (Install the CMS).
• You have a GitHub account or organisation that will own the lake repositories, and the right to register an App on it.
• You know whether that owner is a personal account or an organisation. This changes one step, see Personal account or organisation.

Create the GitHub App

The CMS acts as a GitHub App, not as a person. It needs no personal token. It uses short-lived tokens that GitHub issues to the App. One App serves every workspace.

1. On GitHub, open Settings, then Developer settings, then GitHub Apps, then New GitHub App.
2. Give it a name, for example mdlake-cms. The homepage URL can be anything.
3. Untick Webhook, Active. The CMS never receives events.
4. Leave the callback URL and the user authorization options alone. No user sign-in is used.
5. Under Repository permissions, set Contents to Read and write. Metadata: Read is added for you. If the owner is an organisation, also set Administration to Read and write, so the CMS can create the lake-<short name> repository itself. Grant nothing else.
6. Under Where can this GitHub App be installed, choose Only on this account.
7. Choose Create GitHub App. Note the App ID at the top of the App’s page.
8. Under Private keys, choose Generate a private key. A .pem file downloads once. Keep it out of Git. The repository’s .gitignore already covers *.pem.
9. Choose Install App and install it on the owner. Choose All repositories, so that new lake-* repositories are reachable at once, or select repositories one by one.
10. Note the installation ID. It is the number at the end of the installation’s settings address: https://github.com/settings/installations/<id>, or for an organisation https://github.com/organizations/<org>/settings/installations/<id>.

You now have an App ID, an installation ID, a .pem file and the owner’s name.

Give the settings to the CMS

Optional settings:

The owner must be a valid GitHub name: letters, digits and single dashes, at most 39 characters.

With Docker Compose

Put these lines in the .env file next to docker-compose.yml:

GIT_REMOTE_HOST=GitHub
GIT_REMOTE_OWNER=acme-corp
GITHUB_APP_ID=123456
GITHUB_INSTALLATION_ID=7654321
GITHUB_PRIVATE_KEY_FILE=/home/you/keys/mdlake-cms.pem
GITHUB_PRIVATE_KEY_PATH=/run/secrets/github-app.pem

GITHUB_PRIVATE_KEY_FILE is the path of the .pem on your machine. On Windows use the path of the file on your machine, for example C:/Users/you/keys/mdlake-cms.pem. If the container cannot read the key, run docker compose config and check the path in the volumes entry of the API service. Compose mounts that file read-only inside the container at /run/secrets/github-app.pem. GITHUB_PRIVATE_KEY_PATH is that path inside the container. .env.example already sets it, so leave it as it is. Then restart the stack:

docker compose --profile full up -d --build

docker compose --profile full up -d --build

On a server

Mount the key as a secret file and set GitHub__PrivateKeyPath, or store the key text in a secret called GitHub__PrivateKey. With Key Vault configured (KeyVault__Uri), name the secret GitHub--PrivateKey.

In production, also set Remotes__RequireMirror=true. The lake then lives on one disk, and the mirror is its second copy, so a workspace cannot publish until it is connected. Until an admin connects it, publishing answers “Connect the lake repository first” and the workspace home shows a warning. The API does not start when Remotes__RequireMirror is true and Git__Remote__Host is not set.

Personal account or organisation

GitHub does not let an App create repositories in a personal account. It can create them in an organisation. So the first connection differs:

Moving the lakes to an organisation later removes the manual steps.

Connect a workspace

When the CMS has a Git host, every new workspace starts connecting at once. For existing workspaces, you connect from the settings page.

1. Open the workspace, then choose Workspace settings in the header.

2. Find Lake repository. If the section is missing, the CMS has no Git host. Check the settings above.

3. Read the state:

   You see	It means
   The choice Repository on GitHub with Create a new private repository or Link an existing repository, and a Repository name field	No repository is chosen yet. Go to step 4.
   A box with Check access	A repository is chosen but not reachable yet. Go to step 5.
   Clone URL with Push now, Fetch and check and Disconnect	The workspace is connected. Go to Check that it works.

4. Choose how to connect, then choose Connect to GitHub:

   • Create a new private repository (organisations). The name is filled in as lake-<short name>. The CMS creates it.
   • Use a new private repository I create (personal accounts). You create it in step 5.
   • Link an existing repository. Type its name in Repository name. It must be empty, or already hold this lake’s history.

5. For a personal account, the box with Check access lists the steps:

   1. Choose Create it on GitHub. Create a private repository with exactly the name shown. Leave it empty: no README, no licence, no .gitignore.
   2. If you installed the App on selected repositories only, choose Open the installation and add the new repository. With All repositories this step is not needed.
   3. Come back and choose Check access.

   The CMS finds the repository, checks that it is private and that the App can write to it, and pushes the whole history. A message says “Connected. The full history was pushed.” If the lake has no commits yet it says “Connected. The first publish will be pushed.”

The repository name may use letters, digits, dots, dashes and underscores, up to 100 characters.

Check that it works

In Lake repository:

• Clone URL shows the address developers use, https://github.com/<owner>/lake-<short name>.git. Choose Copy clone URL to copy it. Developers need their own GitHub access to clone it. See Install the builder and CI, the lake and caches.
• A status line shows the push state: “Up to date. Last push 2 minutes ago.” or “Waiting to push the latest publish.”
• The section refreshes every five seconds, so a publish made by someone else shows up without a reload.

Test it: publish a document (Publishing) and watch the line. A new commit should appear on GitHub within a few seconds.

The buttons

Fetch and check answers with one of these sentences:

When something is wrong

What’s next

• Give developers and tools read access with a token: Workspace tokens.
• Build a site from the lake: Install the builder.