Importing content
How to bring existing material into the lake - a Word, PowerPoint or PDF file as one draft, or a whole folder of Markdown files with images as many drafts. Covers the dry run, re-imports, the frontmatter fields including order and _folder.yaml, limits and the report.
On this page
You do not have to retype what you already have. mdlake has two ways to bring content in. Both create drafts, so nothing is public until you publish it.
| You have | Use | Who can do it |
|---|---|---|
| One Word, PowerPoint or PDF file | Import a Word, PowerPoint or PDF file | Editors and admins |
| A folder of Markdown files, with images | Markdown folder import | Admins, and tools with a token that has the import permission |
Before you start
- You have a workspace and you are a member of it (Users and invites).
- For Word, PowerPoint and PDF files, the document converter service runs. It is part of the standard install (Install the CMS). Without it, the import says “The document converter is not available right now. Try again in a moment.”
Import a Word, PowerPoint or PDF file
This turns one file into one draft document. The file itself is not kept. If you want to keep the original too, upload it to the media library as well (The media library).
- Open the workspace home page. It always shows three cards at the top.
- Choose Import Word, PowerPoint or PDF (“Turn a file into a draft document”). Pick a
.docx,.pptxor.pdffile. - Wait while the card says Importing….
Expected result: a new draft opens in the editor. If the file had pictures, or something needs your attention, a notice appears first: “Fleet services is a draft now. 3 pictures were added to the media library.” Choose Open the document, or Open the media library.
What the import does
- Title. The title in the file is used. If there is none, the file name is.
- Text. The converter turns the file into Markdown: headings, paragraphs, lists and tables. Anything the editor cannot show may be flattened.
- Folder. The draft goes into the folder you are working in.
- Pictures in Word and PowerPoint files become media files in a folder named after the document. The alt text comes from the picture description in the file, when the author wrote one. Pictures with identical content are stored once and reused.
- Pictures in PDF files are not extracted. Only the text is imported. The notice says “Images in PDF files are not extracted yet; only the text was imported. Upload the pictures to the media library and add them to the document.”
- Pictures without a description get no alt text. They are listed under Needs attention on the home page, and cannot be published until you add it (The media library).
Limits and failures
| What you see | What to do |
|---|---|
| “The file is larger than 25 MB.” | Use a smaller file, or split it. The limit is the media upload limit (Assets:MaxUploadBytes). |
| “Choose a Word (.docx), PowerPoint (.pptx) or PDF file.” | Other types are refused. The check looks at the contents, not the name. Old .doc and .ppt files must be saved as .docx and .pptx first. |
| “The text of this file could not be read. It may be damaged or password protected.” | Remove the password or save the file again. |
| “The document converter took too long. Try again or use a smaller file.” | Try again, or a smaller file. |
| “The file is empty.” | Check that you chose the right file. |
After the import, read the draft with care. Check headings, tables and lists, then add a summary and tags (Writing documents).
Markdown folder import
This is for a larger move: a documentation folder, an export from another system, or a site’s content. You give the CMS a folder of .md files with the images and files they use. Folders become folders, images become media, and links between the files become lake:// links that survive renaming.
It is a feature for admins. The menu item Import Markdown… under New is shown only to them. Tools can do the same through the API with a workspace token that has the import permission (Workspace tokens).
Four rules protect your workspace:
- Everything arrives as a draft. The menu never publishes.
- Nothing is ever deleted. A file that disappears from your folder does not remove the document it made earlier.
- You see a dry run first, and nothing is written until you confirm.
- The import can be repeated. A second run updates the drafts it made before.
Prepare the folder
- Use
.mdor.markdownfiles. Put images and files (.jpg,.jpeg,.png,.webp,.pdf,.docx,.pptx,.xlsx) next to the Markdown that uses them, and refer to them with relative paths, as in. - Link between files with relative paths too:
[Pricing](../pricing.md). A link to a heading works when the heading is lower-case words joined by hyphens, as inpricing.md#per-seat. - Give each file frontmatter if you want control. Every field is optional:
---
id: 01M5YPMJRG2SN2C75GJWV9DAWF
title: Fleet services
summary: What the fleet team offers and how to book it.
tags: [services, fleet]
slug: fleet-services
template: service
order: 20
---| Field | What the import does |
|---|---|
title | The document title. Without it, the first level-1 heading is used and removed from the text. Without that, the file name. The title is cut at 200 characters. |
summary | The summary. description is read as a summary when summary is missing. Over 300 characters it is shortened and a warning is given. |
tags | A list, or one comma-separated string. The workspace’s own spelling is used when a tag exists. Invalid tags are dropped with a warning. At most 20 are kept. |
slug | Kept. A value that is not lower-case hyphenated words is converted if possible (Writing documents). |
template | Kept. A hint for the site about which layout to use. Up to 100 characters. |
id | The document’s ID, a valid ULID. If a draft with that ID exists in the workspace, the file updates it. If the ID is new, it is used for the new document. An ID that belongs to another workspace, or to a document in the trash, is ignored and the file gets a new ID. |
order | The document’s place among the items of its folder: a number, lower first. The CMS keeps it as the document’s order and not as an extra field, so you can drag the document to another place afterwards (Reorder documents and folders). A number with decimals is rounded to a whole number. Text that reads as a number, such as "20", is accepted. Without order, the document keeps the place it has. |
| any other field | Kept as an extra field of the document and published with it. A site can read it. |
status, created, updated | Managed by the CMS. Not imported. |
The first-level heading (# Title) is removed from the body because the title is a field. Other headings are moved into the levels the editor has (2 to 4). Raw HTML, MDX components and other things the editor cannot show stay as written, and are reported. Fenced code blocks stay as they are, and indented code gets a fence so that it opens as a code block.
A folder can carry its own place in the same way. Put a file called _folder.yaml inside the folder, with one line:
order: 20The import reads it, gives that folder the number, and does not import the file as anything else. Other keys in the file are ignored. A _folder.yaml at the top of the zip does nothing when you import into the top level of the workspace, because the top level has no parent to be ordered in. When you import into a folder (Put it in), the top-level _folder.yaml of the zip orders that folder.
Folder names in the zip become document folders. Names that do not fit the folder rules are fixed, and the report says so. Hidden files and folders (names that start with a dot), node_modules and __MACOSX are left out.
Run the import
- Open the workspace. Choose New, then Import Markdown…. The dialog Import Markdown opens.
- Choose Choose a zip, or Choose a folder. A folder is zipped in your browser. The line under the buttons says what you picked, for example “Folder docs: 142 files (8.1 MB), 12 other files left out”.
- Under Put it in, pick the folder in the CMS where everything should go. The default is the folder you have open. The line below says “The top level of the workspace.” or “Inside guide.”.
- Decide on Replace drafts that were edited here since the last import. Leave it off the first time. Off means that documents someone changed in the CMS are left alone and listed in the report.
- Choose Check. The dialog says Checking… and then Check the import.
- Read the plan, described below. Nothing has been written yet.
- Choose Import 24 documents as drafts. The button says Nothing to import when there is nothing to do, and is disabled.
- The dialog Importing… shows “12 of 40 done”. You can leave it open while it works.
- When it says Import finished, read the report. Choose Done.
Expected result: the folders and drafts appear in the tree and the pictures in the media library. Each document shows a hollow dot, because it is a draft.
If you close the dialog after checking and do not confirm within 60 minutes, the staged upload is thrown away and you must start again.
Read the plan and the report
The plan before you confirm and the report after the import have the same shape. They show eight numbers: documents to create, update, skip or fail; media to upload or already there; links rewritten; links kept as text.
Below the numbers, a sentence says “Nothing needs your attention.” or “3 warnings and 0 errors to look at.” Tick Only show items with warnings or errors to focus on those.
The sections are:
- Documents. Each row shows the title, the source path and a badge: Will be created, Will be updated, Will be updated and moved, Unchanged, Edited in the CMS, left alone, Being edited, left alone, or Cannot be imported. Open a row to see where the file Becomes, its Id and where the ID came from (from the file, from an earlier import or new), the tags, the messages, and Kept as written, the links that could not be resolved, with their line numbers.
- Media. Each row shows a file and Will be uploaded, Already in the library or Cannot be imported. Open a row for the folder, the size, the alt text and the documents that use it.
- In the workspace, not in the upload. Documents that an earlier import made, which are not in this upload. Nothing is deleted. Remove them yourself if they are gone for good.
- Ignored files. Files left out, each with a reason: not used by any document, file type not imported, no file extension, path too long, hidden file.
After the import, Download report (JSON) and Download report (Markdown) save the report. The Markdown version is readable and good to keep.
Messages come in three levels: information, warnings and errors. Messages about order are:
| Message | Level | Meaning |
|---|---|---|
“The order “soon” is not a number and was dropped.” (order-invalid) | Warning | The order of a document is not a number. The document is imported without it. |
“The order 2.5 is not a whole number and became 3.” (order-rounded) | Information | The number was rounded. |
”‘_folder.yaml’ is at the top level, which has no parent to be ordered in; it was left out.” (folder-file-root) | Information | A _folder.yaml at the top level of the workspace is ignored. |
“The order in ‘Guides/_folder.yaml’ is not a number; it was left out.” or “‘Guides/_folder.yaml’ could not be read (…); it was left out.” (folder-file-invalid) | Warning | The file is not valid YAML, has no number, or sits in a folder whose name cannot be used. That folder gets no number. |
| “2 folders got their order from _folder.yaml.” | Information | How many folder numbers the import will set. |
“The order of the documents and folders could not be set. Import again to retry.” (order-failed) | Warning | The documents and folders were imported but the numbers were not. Run the import again. |
Typical warnings are “images/map.png is not in the upload; kept as written”, “No alt text in the Markdown. Add it in the media library: an image without alt text cannot be published.” and “TOML frontmatter (+++) is not supported; it was kept as text.”
Import again
Run the import again with a changed folder to update what you imported before. The CMS recognises a document by, in this order, the id in its frontmatter and the path it had in the earlier import.
The path includes the folder under Put it in, so choose the same folder as the first time. In another folder, files without an id become new documents. When most of the new documents match what an earlier import put in another folder, the plan warns you and names that folder (other-folder).
For each file, the plan decides:
| Situation | Result |
|---|---|
| The file is new | A new draft is created |
| The file changed | The draft is updated |
| The file did not change since the last import | Unchanged, skipped |
| Someone changed the draft in the CMS after the last import | Edited in the CMS, left alone. Tick the box Replace drafts that were edited here since the last import to overwrite it |
| Someone is editing the document right now | Being edited, left alone |
| The file moved to another folder in the source | The document is updated and moved, keeping its ID and its links |
| The document an earlier import made is in the trash | A new document is created |
| A file is gone from the source | Nothing happens. It is listed under In the workspace, not in the upload |
Media is matched by content, wherever the file was. Identical files are stored once. A changed image is uploaded as a new file, and the old one is kept.
Published documents keep their published version until you publish again. An import never changes the text of what is public. The one exception is order: when the import gives a published document or folder a different number, the CMS writes the new number to the lake in a small commit, Reorder: <folder>, as it does when you drag (Reorder documents and folders). Documents that were left alone because they were edited in the CMS or are being edited keep their number.
Limits
These limits are checked while the zip is read. Admins can change them under MarkdownImport (CMS configuration).
| Limit | Default | Setting |
|---|---|---|
| Size of the zip | 100 MB | MarkdownImport:MaxZipBytes |
| Files and folders in the zip | 2,000 | MarkdownImport:MaxEntries |
| Everything together, unpacked | 300 MB | MarkdownImport:MaxTotalBytes |
| Packing ratio of a file of 1 MB or more | 200 times | MarkdownImport:MaxCompressionRatio |
| Wait between Check and Import | 60 minutes | MarkdownImport:StagingMinutes |
| One image or file | 25 MB | Assets:MaxUploadBytes |
| One Markdown document | 2,000,000 characters | not configurable |
| Path of a file | 260 characters | not configurable |
The zip is also refused when it contains an absolute path, a .. path, control characters in a name, a symbolic link or two entries with the same path. These checks protect the server.
If something goes wrong
| What you see | What to do |
|---|---|
| “That folder has no Markdown files (.md) in it.” | Choose the folder that holds the .md files. |
| “The file is not a readable zip archive.” | Make the zip again. |
| “The zip has 2400 entries; at most 2000 are accepted.” | Split the folder, or ask an admin to raise MarkdownImport:MaxEntries. |
| “The files add up to 340 MB unpacked; at most 300 MB are accepted.” | Remove large files, or raise MarkdownImport:MaxTotalBytes. |
| “The checked upload has expired” | Choose the zip or folder again. |
| “The import was already confirmed” | Each check can be confirmed once. Start again to run it a second time. |
| Cannot be imported on a document | Open the row. The reason is one of: the frontmatter cannot be read, the document is too large, the path is too long, or the file is not text. |
| A link is Kept as written | The target is not in the upload or earlier imports. Add it and import again, or fix the link in the editor. |
| Images appear but cannot be published | They have no alt text. Add it in the media library (The media library). |
What’s next
- Review and publish the new drafts: Publishing.
- Fix missing alt text and summaries from the home page: The media library and Writing documents.
- Let tools call the import through the API: Workspace tokens.
