RFD 124: Source-Mapped Markdown Rendering
ImplementedDesign2026-11-16Jean Mertz
Summary
This RFD adds md-html, an internal crate that renders Markdown to HTML with source byte ranges attached to every rendered text span. A page displaying that HTML can map a text selection directly to an exact source byte range and map a stored source range back to a highlight without searching for text.
Motivation
Any tool that lets a reader annotate rendered Markdown (to comment on a phrase, suggest a replacement, or link to a passage) must identify the exact source text the selection covers. Searching the Markdown source for the selected text is unreliable:
- Duplicate phrases are ambiguous.
- Rendered text differs from source text:
**Abandoned**renders as "Abandoned", and\|in a table cell renders as|. - A paragraph that appears on one line in the browser may span several lines in the source, so a selection crosses line endings that are invisible on the page.
JP renders Markdown to HTML in two places, and neither preserves source positions:
- The documentation site renders with markdown-it via VitePress. markdown-it tokens carry line ranges only for block tokens. Its maintainer closed a request for column positions as "impossible with current architecture" (markdown-it issue 821), and its paragraph rule passes a rebuilt, trimmed string to the inline parser, so inline offsets do not correspond to source offsets.
jp-serve-webrenders conversation messages with comrak'smarkdown_to_html.
comrak records a source position on every AST node, and six workspace crates already use it to parse Markdown. What is missing is an HTML renderer that emits those positions in a form a web page can consume, along with verification that the positions are exact.
Design
What a Caller Gets
let rendered = md_html::render(source);
rendered.html // StringThis RFD is **Abandoned**.<p><span data-s="327" data-e="339">This RFD is </span><strong><span data-s="341" data-e="350">Abandoned</span></strong><span data-s="352" data-e="353">.</span></p>Every contiguous run of rendered text is wrapped in a <span> carrying data-s and data-e, the half-open byte range ([start, end)) in source that produced it. Markdown syntax characters (**, backticks, link destinations, list markers) produce no rendered text and appear in no span. Whitespace inserted by the renderer between spans (line endings, or the space that replaces a line break inside a wrapped code span) is also outside any span; a cursor position there lies in a gap.
The renderer emits three kinds of spans:
| Span | Meaning |
|---|---|
data-s, data-e | The rendered text matches the source bytes character for character. An offset into the text maps 1:1 to the source. |
data-s, data-e, data-atom | The rendered text represents the source range as an indivisible unit. Selections snap to its boundaries. |
class="unmapped" | The source range could not be verified. The text cannot be selected as an annotation endpoint. |
Atoms are used in two places: the | rendered from an escaped pipe (\|) in a table cell, and an alert's title, which comrak synthesizes from the alert kind to represent the [!NOTE] marker.
The renderer never guesses: if a node's source range does not resolve back to the exact rendered text, the renderer marks the span unmapped.
Positions are UTF-8 byte offsets because browser code can compute them directly from a DOM text node with TextEncoder. The crate also converts between byte offsets and 1-based line:column positions (with columns counted in Unicode code points) for callers that persist line-and-column anchors.
Client-Side Mapping
md-html ships no JavaScript; the client-side mapping contract has two operations:
- Selection to source range. For each endpoint of the DOM selection, locate its enclosing span. The source byte offset is
data-splus the UTF-8 byte length of the span text preceding the selection offset. Inside adata-atomspan, a selection start snaps todata-sand a selection end snaps todata-e. If an endpoint falls in a gap outside any span (such as between two paragraphs), a selection start snaps forward to the next span'sdata-sand a selection end snaps backward to the previous span'sdata-e. If either endpoint falls inside anunmappedspan, the selection has no source range and cannot be annotated. A selection whose endpoints are both mapped may span acrossunmappedtext, because the byte range between the two endpoints still covers the underlying source. - Source range to highlight. Every span whose
[data-s, data-e)interval overlaps the target byte range is highlighted over the intersecting character subrange;data-atomspans are highlighted in full.
This mapping was prototyped in a browser across three RFDs using the CSS Custom Highlight API (which highlights DOM ranges without mutating the HTML), covering single-line selections, selections across emphasis, code spans, links, and paragraphs, selections in table cells and code blocks, and persistence across page reloads.
Deriving Exact Ranges
md-html parses Markdown with comrak (enabling tables, strikethrough, autolinks, task lists, and alerts) and walks the AST to emit HTML directly. A text node's byte range comes from its comrak sourcepos, with three adjustments:
parse.escaped_char_spansis enabled. Without this option, a text node containing a backslash escape reports asourceposthat includes the backslash, shifting character offsets after the escape.- Code spans are split by source line. A code span's
sourceposincludes its delimiting backticks, and a multi-line code span collapses each line ending (along with the next line's indentation or blockquote prefix) into a single rendered space. The renderer strips the backticks and emits a separate mapped<span>for each source line, separated by an unmapped space. - Escaped pipes in table cells are compensated. Inside a table cell, comrak counts an escaped pipe (
\|) as one column instead of two bytes, shifting subsequent column positions on the same line left by one byte per\|. The renderer scans from the start of the table cell and adds one byte for each preceding\|.
Fenced code blocks are mapped line by line: each content line corresponds to the trailing slice of its source line, after any indentation or blockquote prefix.
Corpus Verification
Measured across 403 Markdown files in this repository (docs/rfd, docs/rfd/drafts, docs/wiki, and docs/ticket) with comrak 0.52:
| Node | Count | Resolves back to exact source |
|---|---|---|
| Text | 115,404 | 100%. 8 fail without escaped_char_spans. |
| Code span | 41,942 | 100%. 632 wrap across lines; 5 contain an escaped pipe in a table. |
| Fenced code block | 1,634 | 100%, line by line. |
By container, the text nodes include 44,402 in list items, 4,647 in table cells, 6,442 in headings, and 526 in alerts.
The same pass found two inaccuracies in block-level sourcepos ranges, which md-html does not rely on because it maps leaf text and code nodes directly:
- 1,163 of 11,078 list items report their end position as column 0 of the following line.
- One table nested inside a list item reports its end position two columns short.
Scope Boundaries
- Raw HTML is escaped and rendered as literal text. Callers render untrusted or semi-trusted Markdown inside pages with access to local APIs.
- Image
alttext is written into an HTML attribute and receives no span. - JP-specific syntax. Mention autolinks (such as
T-02wt0kx) and embeds belong todocandjp-wiki, notmd-html.md-htmlreports byte offsets into the exact string passed torender. A caller that transforms the Markdown source before rendering (for example, by expanding embeds) is responsible for mapping offsets back to the original file; a caller that post-processes the HTML may wrap spans in outer elements as long as it preserves their text and attributes. - Syntax highlighting in code blocks is omitted.
Drawbacks
- A second HTML renderer alongside VitePress. Documents rendered by
md-htmldo not share VitePress's exact HTML structure or plugins, so any Markdown extension added to the documentation site must either be ported tomd-htmlor differ between the two views. - Coupling to comrak's column accounting. A comrak upgrade that changes how column offsets are calculated could invalidate one of the range adjustments. The corpus test over JP's documentation catches regressions at test time.
- Larger HTML output. Wrapping every text node and code line in a
<span>increases HTML size compared to plain comrak output.
Alternatives
comrak's render.sourcepos
comrak has a built-in render.sourcepos option that adds data-sourcepos attributes to HTML elements. However, it annotates block and inline elements rather than text nodes: plain text between child elements has no wrapper element of its own, element ranges include surrounding Markdown delimiters, and the caller would still need to apply the backslash, multi-line code span, and table-pipe adjustments on the client.
Patch markdown-it
Adding byte-accurate inline source spans to markdown-it would let VitePress and review views share one renderer. Doing so requires wrapping every inline rule and mapping each block rule's rebuilt, trimmed input string back to the original file offsets: a change the markdown-it maintainer rejected as incompatible with its architecture, inside a VitePress release line that is still in alpha. Because stored anchors are source positions rather than DOM positions, nothing in this RFD precludes switching renderers later.
pulldown-cmark
pulldown-cmark's offset iterator yields a byte range for every event, including text events. We did not adopt it because six workspace crates already parse Markdown with comrak; introducing a second Markdown parser would risk subtle parsing divergences where md-html and doc or jp-wiki disagree on document structure.
Text search in the source
Locating selections by searching the raw Markdown source requires no renderer changes, but fails on duplicate phrases, inline formatting syntax, and wrapped lines as described in Motivation.
Non-Goals
- Replacing the VitePress documentation renderer.
- Migrating
jp-serve-webtomd-html. - Shipping client-side selection or highlighting JavaScript in
md-html.
Risks and Open Questions
- Markdown constructs absent from JP's corpus. JP's documents contain no inline HTML and no HTML entity or character references outside code spans, and neither footnotes nor description lists are enabled (comrak notes that description list source positions are unreliable). When a construct's source range does not match its rendered text (for example, an HTML entity like
&whose 5-byte source renders as&), the node falls back toclass="unmapped", failing closed rather than producing a wrong offset. - Tab characters. JP's Markdown files indent with spaces; how comrak's column counter advances across literal
\tcharacters has not been verified.
Implementation Plan
Phase 1: md-html crate
Create crates/internal/md-html with render, byte-to-line:column conversion helpers, and unit tests covering each span type and each range adjustment.
Include a corpus test that renders every Markdown file under docs/rfd, docs/wiki, and docs/ticket and asserts that no text node or code span is emitted as unmapped, catching comrak regressions or newly introduced edge cases in CI.
This phase is self-contained and mergeable before any consumer is wired up.
References
- markdown-it issue 821, on column positions
- comrak's
render.sourcepos