Open source · MIT License

Fast text processing for Python, TypeScript, and Rust

Textlane parses Markdown, HTML and natural language into syntax trees, transforms them with plugins and writes them back out. One native Rust core does the work, with the same API in every language.

Get started View source

md.parse
md.gfm
md.gemoji
md.toHtml
html.slug
html.stringify

input.md

# Hello, *lane*

- [x] parse :zap:
- [ ] ~~rewrite~~

output.html

<h1 id="hello-lane">Hello, <em>lane</em></h1>
<ul class="contains-task-list">
<li class="task-list-item"><input type="checkbox" checked disabled> parse ⚡</li>
<li class="task-list-item"><input type="checkbox" disabled> <del>rewrite</del></li>
</ul>

md.parse
md.gemoji
md.smartypants
md.toc
md.stringify

input.md

# Release notes

## Contents

## What's new

"Lanes" -- now *3x faster*... :rocket:

## Upgrading

Run `npm install textlane` -- that's it.

output.md

# Release notes

## Contents

* [What’s new](#whats-new)
* [Upgrading](#upgrading)

## What’s new

“Lanes” — now *3x faster*… 🚀

## Upgrading

Run `npm install textlane` — that’s it.

nl.parse
nl.passive
nl.repeatedWords
nl.stringify

input.txt

It was hit by the the ball.

report

input.txt
  1:8-1:11   warning  Unexpected use of the passive voice  hit  retext-passive
  1:15-1:22  warning  Unexpected repeated `the`, remove one occurrence  the  retext-repeated-words

⚠ 2 warnings

md.parse
md.toHtml
html.slug
JSrehypeAutolinkHeadings
html.stringify

main.ts

import rehypeAutolinkHeadings from "rehype-autolink-headings"

const file = await textlane()
  .use(md.parse())
  .use(md.toHtml())
  .use(html.slug())
  .use(rehypeAutolinkHeadings)
  .use(html.stringify())
  .process("# Lanes\n\n## Native stages")

output.html

<h1 id="lanes"><a aria-hidden="true" tabindex="-1" href="#lanes"><span class="icon icon-link"></span></a>Lanes</h1>
<h2 id="native-stages"><a aria-hidden="true" tabindex="-1" href="#native-stages"><span class="icon icon-link"></span></a>Native stages</h2>

unified-compatible

Built with AI, in Rust

100×faster natively

All credit goes to the unified community for creating the project: its plugins, syntax trees and pipeline design are theirs.

remark-parseremark-gfmremark-gemojiremark-rehyperehype-slugrehype-stringify

remark-parseremark-gemojiremark-smartypantsremark-tocremark-stringify

retext-englishretext-passiveretext-repeated-wordsretext-stringify

remark-parseremark-rehyperehype-slugrehype-stringify

Languages

One pipeline in Python, TypeScript and Rust; each stage ports the unified plugin of the same name.

Python · main.py

from textlane import textlane, md, html

processor = (
    textlane()
    .use(md.parse())
    .use(md.gfm())
    .use(md.to_html())
    .use(html.slug())
    .use(html.stringify())
)

file = processor.process(markdown)

TypeScript · main.ts

import { textlane } from "textlane"
import * as md from "textlane/md"
import * as html from "textlane/html"

const file = await textlane()
  .use(md.parse())
  .use(md.gfm())
  .use(md.toHtml())
  .use(html.slug())
  .use(html.stringify())
  .process(markdown)

Rust · main.rs

use textlane::{Processor, html, md};

let processor = Processor::new()
    .with(md::Parse::default())
    .with(md::Gfm::default())
    .with(md::ToHtml::default())
    .with(html::Slug::default())
    .with(html::Stringify::default())
    .freeze()?;

let output = processor.process(markdown)?;

Plugins

Markdown, HTML and natural language, each a crate of its own, bridged by the same converters unified uses.

textlane-md/mdx

Markdown

A port of micromark and the mdast utilities.

md.parseremark-parse
md.stringifyremark-stringify
md.toHtmlremark-rehype
↳ breaksremark-breaks
md.toNaturalLanguageremark-retext
md.gfmremark-gfm
↳ alertsremark-github-blockquote-alert
md.cjkFriendlyremark-cjk-friendly
md.frontmatterremark-frontmatter
md.mathremark-math
md.directiveremark-directive
md.gemojiremark-gemoji
↳ emoticonremark-emoji
md.smartypantsremark-smartypants
md.definitionListremark-definition-list
md.headingIdremark-heading-id
md.stripstrip-markdown
md.tocremark-toc
md.packageCommandnpm-to-yarn
md.pathRewritetextlane
md.componenttextlane
md.mdxremark-mdx
mdx.compile@mdx-js/mdx

textlane-html

HTML

A port of parse5 and the hast utilities.

html.parserehype-parse
html.stringifyrehype-stringify
html.toMarkdownrehype-remark
html.toNaturalLanguagerehype-retext
html.rawrehype-raw
html.sanitizerehype-sanitize
html.slugrehype-slug
↳ autolinkrehype-autolink-headings
html.extractTocrehype-extract-toc
html.externalLinksrehype-external-links
html.unwrapImagesrehype-unwrap-images
html.pathRewritetextlane
html.mathtextlane
html.linkifytextlane
html.gemojitextlane
html.formatrehype-format
↳ minifyrehype-minify-whitespace
html.highlight@shikijs/rehype
html.katexrehype-katex

textlane-nl

Natural Language

A port of parse-english and the retext plugins.

nl.parseretext-english
nl.stringifyretext-stringify
nl.smartypantsretext-smartypants
nl.quotesretext-quotes
nl.contractionsretext-contractions
nl.repeatedWordsretext-repeated-words
nl.indefiniteArticleretext-indefinite-article
nl.passiveretext-passive
nl.readabilityretext-readability
nl.equalityretext-equality
nl.sentenceSpacingretext-sentence-spacing

Compatibility

Each port is checked case by case against its upstream test suite.

Native execution

Parsing, transforms and serialization run in Rust over trees held in a per-document arena, not a heap of objects.

One crate, three languages

crates.io gets the crate, npm a prebuilt addon per platform, PyPI a PyO3 wheel: the same code at the same version.

unified's semantics

use, data and freeze; parse, run and stringify; presets, sync and async; and JavaScript or Python plugins joining a pipeline as transformers.

Compatible trees

mdast, hast and nlcst nodes with unified's types, fields and positions, read and written as unified's JSON.

10k+ conformance tests

Ported from CommonMark, html5lib and WPT, remark, rehype and retext. Every implementation runs the same JSON cases.

Agent skill

The npm and PyPI packages ship a skill that teaches coding agents to install and call textlane; this site publishes it too.

Conformance suite

Shared across languages

Python, TypeScript and Rust run the same language-neutral cases, ported from CommonMark, html5lib, remark, rehype and retext: the same pipeline, the same input, the same expected output.

tests: 10k+
plugins: 40+
languages: 3

Browse the suite →

conformance/tests/md/gfm-strikethrough.json

{
  "description": "micromark-extension-gfm-strikethrough",
  "pipeline": [
    ["md.parse"],
    ["md.gfm"],
    ["md.toHtml"],
    ["html.stringify"]
  ],
  "tests": [{
    "description": "should support nested strikethrough",
    "input": "a ~~b ~~c~~ d~~ e",
    "expected": {
      "html": "<p>a <del>b <del>c</del> d</del> e</p>"
    }
  }]
}

PythonTypeScriptRust

Performance

Markdown-to-HTML throughput from Rust, Node.js and Python, against other libraries. textlane exposes one pluggable tree (AST), or two once an HTML plugin needs them; comrak and cmark expose one, pulldown-cmark none.

benchmark · throughput on a 100 KB CommonMark README

pulldown-cmark0 ASTs340.5 MB/s
textlane (Rust)1 AST198.2 MB/s
textlane (Python)1 AST192.5 MB/s
textlane (TypeScript)1 AST171.7 MB/s
comrak1 AST155.4 MB/s
textlane (Python)2 ASTs143.2 MB/s
textlane (Rust)2 ASTs140.7 MB/s
textlane (TypeScript)2 ASTs129.3 MB/s
cmark (C, via cmarkgfm)1 AST120.9 MB/s
marked1 AST25.1 MB/s
markdown-it0 ASTs24.6 MB/s
markdown-rs0 ASTs9.5 MB/s
mistune1 AST4.8 MB/s
markdown-it-py0 ASTs3.8 MB/s
Python-Markdown1 AST3.2 MB/s
micromark0 ASTs2.2 MB/s
unified (remark + rehype)2 ASTs1.9 MB/s

Input megabytes per second at the median, on an AMD Ryzen 5 PRO 7545U. Reproduce with pnpm -C benchmark measure.

Get started

Textlane is in technical preview: the crate and its plugins are here, the Node.js and Python pipeline API is on its way.

$pip install textlane$npm install textlane$cargo add textlane

Read the docs Browse the conformance suite

Built by

Datist — Software development, end to end.