plugin system
This commit is contained in:
parent
1c69a01aa6
commit
03cab97dac
7 changed files with 599 additions and 251 deletions
37
README.md
37
README.md
|
|
@ -1,11 +1,11 @@
|
|||
# md2amb utilities
|
||||
|
||||
This repository contains command line helpers for transforming Markdown into formats that work well on retro hardware or constrained text viewers.
|
||||
This repository contains command line helpers for transforming Markdown into formats that work well on retro hardware or constrained text viewers. A small plugin API lets you mix-and-match parsers and renderers so additional formats can plug into the same preprocessing pipeline.
|
||||
|
||||
## Tools
|
||||
|
||||
- `md2amb.py` – converts Markdown into Amber-screen formatted text (see script for details).
|
||||
- `md2txt.py` – converts Markdown into 80-column, DOS-compatible plain text with extensive formatting support:
|
||||
- `md2txt.py` – converts Markdown into 80-column, DOS-compatible plain text with extensive formatting support. It ships with the default `markdown` parser and `text` renderer plugins and exposes the core pipeline so you can add your own parser or renderer modules:
|
||||
- FIGlet-rendered headings (H1–H3) driven by optional YAML frontmatter (`h1_font`, `h2_font`, `h3_font`).
|
||||
- H4+ headings rendered in uppercase with dashed underlines.
|
||||
- Emphasis styles converted to spaced or delimited characters, e.g. `**bold**` → `B O L D`, `__strong__` → `_s_t_r_o_n_g_`, `~~strike~~` → `~s~t~r~i~k~e~`.
|
||||
|
|
@ -29,9 +29,11 @@ This repository contains command line helpers for transforming Markdown into for
|
|||
python md2txt.py input.md -o output.txt # convert to DOS-friendly text
|
||||
python md2txt.py input.md # write result to stdout
|
||||
python md2txt.py input.md --width 72 # override column width
|
||||
python md2txt.py input.md --parser markdown --renderer text # explicitly select defaults
|
||||
python md2txt.py input.md --renderer-option width=68 # pass KEY=VALUE to a renderer
|
||||
```
|
||||
|
||||
Both scripts accept `--help` for the full option list.
|
||||
`--parser` and `--renderer` select a plugin by name (defaults are `markdown` and `text`). Repeatable `--parser-option KEY=VALUE` and `--renderer-option KEY=VALUE` pairs are forwarded to the plugin factories as keyword arguments in addition to the defaults supplied by the CLI. Both scripts accept `--help` for the full option list.
|
||||
|
||||
## FIGlet Fonts via Frontmatter
|
||||
|
||||
|
|
@ -114,3 +116,32 @@ All values are optional—defaults maintain the legacy behaviour.
|
|||
## Output Line Endings
|
||||
|
||||
Generated text uses CRLF line endings to maintain DOS compatibility.
|
||||
|
||||
## Plugin Architecture
|
||||
|
||||
The conversion pipeline lives in `conversion_core.py` and is exposed via `run_conversion`. It is designed around lightweight factories:
|
||||
|
||||
- **Parser factories** receive `base_style: BlockStyle` plus any extra keyword arguments and must return an object with a `parse(lines: Iterable[str]) -> Iterator[BlockEvent | StyleUpdateEvent]` method. The bundled `MarkdownParser` implements this interface.
|
||||
- **Renderer factories** receive `frontmatter: FrontMatter` and arbitrary keyword arguments and must return an object that provides `handle_event(event)` and `finalize() -> Any`. The `TextRenderer` returns a list of DOS-friendly output lines; other renderers may return any data appropriate for their target format.
|
||||
|
||||
Plugins register themselves through the helpers in `plugins.py`:
|
||||
|
||||
```python
|
||||
from plugins import register_parser, register_renderer
|
||||
|
||||
def my_parser_factory(*, base_style, **options):
|
||||
return MyParser(base_style, **options)
|
||||
|
||||
register_parser("my-markdown", my_parser_factory)
|
||||
```
|
||||
|
||||
```python
|
||||
def my_renderer_factory(*, frontmatter, **options):
|
||||
return MyRenderer(frontmatter, target=options.get("target"))
|
||||
|
||||
register_renderer("ansi", my_renderer_factory)
|
||||
```
|
||||
|
||||
Once registered (for example in a small module that imports `md2txt.py`), the new plugins are available via `--parser my-markdown` or `--renderer ansi`. The CLI lists registered plugin names in `--help`, and the helper functions `available_parsers()` / `available_renderers()` return the sorted names if you need to build higher-level tooling.
|
||||
|
||||
The shared preprocessing helpers—YAML frontmatter parsing, recursive include expansion, and ASCII art sentinels—also live in `conversion_core.py`, allowing alternate front-ends to reuse exactly the same behaviour without duplicating code.
|
||||
|
|
|
|||
306
conversion_core.py
Normal file
306
conversion_core.py
Normal file
|
|
@ -0,0 +1,306 @@
|
|||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import re
|
||||
from pathlib import Path
|
||||
from typing import Any, Dict, Iterable, Iterator, List, Optional, Protocol, Set, Tuple, TypeVar
|
||||
|
||||
from md_types import BlockEvent, BlockStyle, FrontMatter, StyleUpdateEvent
|
||||
|
||||
|
||||
FRONTMATTER_PATTERN = re.compile(r"^---\s*$")
|
||||
INCLUDE_WIKILINK_PATTERN = re.compile(r"^\s*!\[\[(.+?)\]\]\s*$")
|
||||
INCLUDE_DIRECTIVE_PATTERN = re.compile(r"^\s*\{\s*\.include\s+(.+?)\s*\}\s*$")
|
||||
ASCII_BLOCK_PATTERN = re.compile(
|
||||
r"^\s*#\[(?P<label>[^\]]+)\]\((?P<target>[^)]+)\)\s*(?P<attr>\{\s*:[^}]+\s*\})?\s*$"
|
||||
)
|
||||
ASCII_INLINE_PATTERN = re.compile(r"#\[(?P<label>[^\]]+)\]\((?P<target>[^)]+)\)")
|
||||
MMD_ATTR_TAIL_RE = re.compile(r"(.*?)\s*\{\s*:(.+?)\}\s*$")
|
||||
ASCII_SENTINEL_PREFIX = "\u0000ASCII:"
|
||||
|
||||
Event = BlockEvent | StyleUpdateEvent
|
||||
RendererOutput = TypeVar("RendererOutput")
|
||||
|
||||
|
||||
class Parser(Protocol):
|
||||
def parse(self, lines: Iterable[str]) -> Iterator[Event]:
|
||||
...
|
||||
|
||||
|
||||
class Renderer(Protocol[RendererOutput]):
|
||||
def handle_event(self, event: Event) -> None:
|
||||
...
|
||||
|
||||
def finalize(self) -> RendererOutput:
|
||||
...
|
||||
|
||||
|
||||
class ParserFactory(Protocol):
|
||||
def __call__(self, *, base_style: BlockStyle, **kwargs: Any) -> Parser:
|
||||
...
|
||||
|
||||
|
||||
class RendererFactory(Protocol[RendererOutput]):
|
||||
def __call__(self, *, frontmatter: FrontMatter, **kwargs: Any) -> Renderer[RendererOutput]:
|
||||
...
|
||||
|
||||
|
||||
def _parse_int(value: Optional[str], default: int = 0) -> int:
|
||||
if value is None:
|
||||
return default
|
||||
match = re.search(r"-?\d+", value)
|
||||
if not match:
|
||||
return default
|
||||
try:
|
||||
return int(match.group())
|
||||
except ValueError:
|
||||
return default
|
||||
|
||||
|
||||
def _parse_bool(value: Optional[str], default: bool = False) -> bool:
|
||||
if value is None:
|
||||
return default
|
||||
lowered = value.strip().lower()
|
||||
if lowered in {"true", "yes", "1", "on"}:
|
||||
return True
|
||||
if lowered in {"false", "no", "0", "off"}:
|
||||
return False
|
||||
return default
|
||||
|
||||
|
||||
def parse_frontmatter(lines: List[str]) -> Tuple[FrontMatter, List[str]]:
|
||||
if not lines or not FRONTMATTER_PATTERN.match(lines[0]):
|
||||
return FrontMatter(), lines
|
||||
frontmatter: Dict[str, str] = {}
|
||||
idx = 1
|
||||
while idx < len(lines):
|
||||
if FRONTMATTER_PATTERN.match(lines[idx]):
|
||||
break
|
||||
if ":" in lines[idx]:
|
||||
key, value = lines[idx].split(":", 1)
|
||||
frontmatter[key.strip()] = value.strip()
|
||||
idx += 1
|
||||
if idx >= len(lines):
|
||||
return FrontMatter(), lines
|
||||
remaining = lines[idx + 1 :] if idx + 1 < len(lines) else []
|
||||
paragraph_spacing_value = frontmatter.get("paragraph_spacing")
|
||||
if paragraph_spacing_value is None:
|
||||
paragraph_spacing_value = frontmatter.get("lines_between_paragraphs")
|
||||
if paragraph_spacing_value is None:
|
||||
paragraph_spacing_value = frontmatter.get("paragraph_lines")
|
||||
default_wrap_indent = 2
|
||||
wrap_code_blocks = _parse_bool(frontmatter.get("wrap_code_blocks"), False)
|
||||
code_block_wrap_indent = default_wrap_indent if wrap_code_blocks else 0
|
||||
code_block_wrap_value = frontmatter.get("code_block_wrap")
|
||||
if code_block_wrap_value is not None:
|
||||
normalized_wrap = code_block_wrap_value.strip()
|
||||
if normalized_wrap:
|
||||
if re.fullmatch(r"-?\d+", normalized_wrap):
|
||||
wrap_code_blocks = True
|
||||
code_block_wrap_indent = max(0, _parse_int(normalized_wrap, default_wrap_indent))
|
||||
else:
|
||||
wrap_flag = _parse_bool(normalized_wrap, wrap_code_blocks)
|
||||
wrap_code_blocks = wrap_flag
|
||||
code_block_wrap_indent = default_wrap_indent if wrap_flag else 0
|
||||
code_block_line_numbers = _parse_bool(frontmatter.get("code_block_line_numbers"), True)
|
||||
blockquote_bars = _parse_bool(frontmatter.get("blockquote_bars"), True)
|
||||
list_marker_indent = max(0, _parse_int(frontmatter.get("list_marker_indent"), 0))
|
||||
list_text_spacing = max(0, _parse_int(frontmatter.get("list_text_spacing"), 1))
|
||||
fm = FrontMatter(
|
||||
h1_font=frontmatter.get("h1_font", "standard").strip() or "standard",
|
||||
h2_font=frontmatter.get("h2_font", "standard").strip() or "standard",
|
||||
h3_font=frontmatter.get("h3_font", "standard").strip() or "standard",
|
||||
margin_left=_parse_int(frontmatter.get("margin_left"), 0),
|
||||
margin_right=_parse_int(frontmatter.get("margin_right"), 0),
|
||||
paragraph_spacing=max(0, _parse_int(paragraph_spacing_value, 0)),
|
||||
hyphenate=_parse_bool(frontmatter.get("hyphenate"), False),
|
||||
hyphen_lang=(frontmatter.get("hyphen_lang") or "en_US").strip() or "en_US",
|
||||
figlet_fallback=_parse_bool(frontmatter.get("figlet_fallback"), False),
|
||||
header_spacing=max(0, _parse_int(frontmatter.get("header_spacing"), 2)),
|
||||
wrap_code_blocks=wrap_code_blocks,
|
||||
code_block_wrap_indent=code_block_wrap_indent,
|
||||
code_block_line_numbers=code_block_line_numbers,
|
||||
blockquote_bars=blockquote_bars,
|
||||
list_marker_indent=list_marker_indent,
|
||||
list_text_spacing=list_text_spacing,
|
||||
)
|
||||
return fm, remaining
|
||||
|
||||
|
||||
def read_lines(path: Path) -> List[str]:
|
||||
with path.open("r", encoding="utf-8") as handle:
|
||||
return handle.readlines()
|
||||
|
||||
|
||||
def expand_includes(
|
||||
lines: List[str],
|
||||
base_dir: Path,
|
||||
include_stack: Set[Path],
|
||||
) -> List[str]:
|
||||
expanded: List[str] = []
|
||||
for line in lines:
|
||||
ascii_segments = _extract_ascii_segments(line, base_dir)
|
||||
if ascii_segments is not None:
|
||||
for sentinel_line, attr_line in ascii_segments:
|
||||
expanded.append(sentinel_line)
|
||||
if attr_line is not None:
|
||||
expanded.append(attr_line)
|
||||
continue
|
||||
target = _extract_include_target(line)
|
||||
if target is None:
|
||||
expanded.append(line)
|
||||
continue
|
||||
target_path = (base_dir / target).resolve()
|
||||
if target_path in include_stack:
|
||||
raise RuntimeError(f"Circular include detected for '{target_path}'.")
|
||||
if not target_path.exists():
|
||||
raise FileNotFoundError(f"Included file '{target_path}' was not found.")
|
||||
include_stack.add(target_path)
|
||||
included_lines = read_lines(target_path)
|
||||
_, include_body = parse_frontmatter(included_lines)
|
||||
included_content = expand_includes(include_body, target_path.parent, include_stack)
|
||||
expanded.extend(included_content)
|
||||
include_stack.remove(target_path)
|
||||
return expanded
|
||||
|
||||
|
||||
def _extract_include_target(line: str) -> Optional[str]:
|
||||
stripped = line.rstrip("\n")
|
||||
match = INCLUDE_WIKILINK_PATTERN.match(stripped)
|
||||
if match:
|
||||
return _normalize_include_target(match.group(1))
|
||||
match = INCLUDE_DIRECTIVE_PATTERN.match(stripped)
|
||||
if match:
|
||||
return _normalize_include_target(match.group(1))
|
||||
return None
|
||||
|
||||
|
||||
def _normalize_include_target(value: str) -> str:
|
||||
trimmed = value.strip()
|
||||
if len(trimmed) >= 2 and ((trimmed[0] == trimmed[-1]) and trimmed[0] in {"'", '"'}):
|
||||
trimmed = trimmed[1:-1].strip()
|
||||
return trimmed
|
||||
|
||||
|
||||
def _extract_ascii_segments(line: str, base_dir: Path) -> Optional[List[Tuple[str, Optional[str]]]]:
|
||||
stripped_line = line.rstrip("\n")
|
||||
block_match = ASCII_BLOCK_PATTERN.match(stripped_line)
|
||||
if block_match:
|
||||
label = block_match.group("label")
|
||||
target = block_match.group("target")
|
||||
attr_text = block_match.group("attr")
|
||||
sentinel = _make_ascii_sentinel(label, target, base_dir)
|
||||
attr_line = f"{attr_text}\n" if attr_text else None
|
||||
return [(sentinel, attr_line)]
|
||||
|
||||
matches = list(ASCII_INLINE_PATTERN.finditer(stripped_line))
|
||||
if not matches:
|
||||
return None
|
||||
|
||||
pieces: List[Dict[str, Optional[str]]] = []
|
||||
last_end = 0
|
||||
for match in matches:
|
||||
prefix = stripped_line[last_end : match.start()]
|
||||
if prefix.strip():
|
||||
return None
|
||||
label = match.group("label")
|
||||
target = match.group("target")
|
||||
block_type, block_name, align = _parse_ascii_label(label)
|
||||
normalized_target = _normalize_include_target(target)
|
||||
target_path = (base_dir / normalized_target).resolve()
|
||||
if not target_path.exists():
|
||||
raise FileNotFoundError(f"ASCII art file '{target_path}' was not found.")
|
||||
pieces.append(
|
||||
{
|
||||
"type": block_type,
|
||||
"name": block_name,
|
||||
"path": str(target_path),
|
||||
"align": align,
|
||||
}
|
||||
)
|
||||
last_end = match.end()
|
||||
|
||||
suffix = stripped_line[last_end:]
|
||||
if suffix.strip():
|
||||
return None
|
||||
|
||||
if not pieces:
|
||||
return None
|
||||
|
||||
sentinel = f"{ASCII_SENTINEL_PREFIX}{json.dumps({'pieces': pieces})}\n"
|
||||
return [(sentinel, None)]
|
||||
|
||||
|
||||
def _make_ascii_sentinel(label: str, target: str, base_dir: Path) -> str:
|
||||
block_type, block_name, align = _parse_ascii_label(label)
|
||||
normalized_target = _normalize_include_target(target)
|
||||
target_path = (base_dir / normalized_target).resolve()
|
||||
if not target_path.exists():
|
||||
raise FileNotFoundError(f"ASCII art file '{target_path}' was not found.")
|
||||
payload = {
|
||||
"pieces": [
|
||||
{
|
||||
"type": block_type,
|
||||
"name": block_name,
|
||||
"path": str(target_path),
|
||||
"align": align,
|
||||
}
|
||||
]
|
||||
}
|
||||
return f"{ASCII_SENTINEL_PREFIX}{json.dumps(payload)}\n"
|
||||
|
||||
|
||||
def _parse_ascii_label(label: str) -> Tuple[str, str, Optional[str]]:
|
||||
tokens = label.strip().split()
|
||||
non_colon: List[str] = []
|
||||
align: Optional[str] = None
|
||||
for token in tokens:
|
||||
if token.startswith(":"):
|
||||
tag = token[1:].strip().lower()
|
||||
if tag in {"left", "right", "center", "centre"}:
|
||||
align = "center" if tag in {"center", "centre"} else tag
|
||||
else:
|
||||
non_colon.append(token)
|
||||
|
||||
block_type = non_colon[0] if non_colon else "custom"
|
||||
block_name = " ".join(non_colon[1:]) if len(non_colon) > 1 else ""
|
||||
return block_type, block_name, align
|
||||
|
||||
|
||||
def run_pipeline(
|
||||
lines: Iterable[str],
|
||||
*,
|
||||
parser: Parser,
|
||||
renderer: Renderer[RendererOutput],
|
||||
base_path: Optional[Path] = None,
|
||||
) -> RendererOutput:
|
||||
base_dir = (base_path or Path.cwd()).resolve()
|
||||
expanded_lines = expand_includes(list(lines), base_dir, set())
|
||||
for event in parser.parse(expanded_lines):
|
||||
renderer.handle_event(event)
|
||||
return renderer.finalize()
|
||||
|
||||
|
||||
def run_conversion(
|
||||
lines: Iterable[str],
|
||||
*,
|
||||
frontmatter: FrontMatter,
|
||||
parser_factory: ParserFactory,
|
||||
renderer_factory: RendererFactory[RendererOutput],
|
||||
parser_options: Optional[Dict[str, Any]] = None,
|
||||
renderer_options: Optional[Dict[str, Any]] = None,
|
||||
base_path: Optional[Path] = None,
|
||||
) -> RendererOutput:
|
||||
parser = parser_factory(
|
||||
base_style=BlockStyle(
|
||||
align="left",
|
||||
margin_left=max(0, frontmatter.margin_left),
|
||||
margin_right=max(0, frontmatter.margin_right),
|
||||
),
|
||||
**(parser_options or {}),
|
||||
)
|
||||
renderer = renderer_factory(
|
||||
frontmatter=frontmatter,
|
||||
**(renderer_options or {}),
|
||||
)
|
||||
return run_pipeline(lines, parser=parser, renderer=renderer, base_path=base_path)
|
||||
|
|
@ -5,6 +5,7 @@ import re
|
|||
from pathlib import Path
|
||||
from typing import Dict, Iterable, Iterator, List, Optional, Union
|
||||
|
||||
from conversion_core import ASCII_SENTINEL_PREFIX
|
||||
from md_types import (
|
||||
AsciiArtPayload,
|
||||
AsciiArtPiece,
|
||||
|
|
@ -31,7 +32,6 @@ PARA_OPEN_RE = re.compile(r"^\s*<p\b([^>]*)>\s*$", re.IGNORECASE)
|
|||
PARA_CLOSE_RE = re.compile(r"^\s*</p>\s*$", re.IGNORECASE)
|
||||
MMD_ATTR_LINE_RE = re.compile(r"^\{\s*:(.+)\}\s*$")
|
||||
MMD_ATTR_TAIL_RE = re.compile(r"(.*?)\s*\{\s*:(.+?)\}\s*$")
|
||||
ASCII_SENTINEL_PREFIX = "\u0000ASCII:"
|
||||
|
||||
|
||||
class MarkdownParser:
|
||||
|
|
|
|||
99
md2mu.py
Normal file
99
md2mu.py
Normal file
|
|
@ -0,0 +1,99 @@
|
|||
from mistune.core import BlockState
|
||||
from mistune.util import strip_end
|
||||
from mistune.renderers._list import render_list
|
||||
from mistune.renderers.markdown import MarkdownRenderer
|
||||
from typing import Dict, Any
|
||||
from textwrap import indent
|
||||
|
||||
class MicronRenderer(MarkdownRenderer):
|
||||
"""A renderer to format Micron text."""
|
||||
NAME = 'micron'
|
||||
|
||||
def __call__(self, tokens, state: BlockState):
|
||||
out = self.render_tokens(tokens, state)
|
||||
# special handle for line breaks
|
||||
out += '\n\n'.join(self.render_referrences(state)) + '\n'
|
||||
return strip_end(out)
|
||||
|
||||
def render_children(self, token, state: BlockState):
|
||||
children = token['children']
|
||||
return self.render_tokens(children, state)
|
||||
|
||||
def text(self, token: Dict[str, Any], state: BlockState) -> str:
|
||||
return token['raw']
|
||||
|
||||
def emphasis(self, token: Dict[str, Any], state: BlockState) -> str:
|
||||
return '`*' + self.render_children(token, state) + '`*'
|
||||
|
||||
def strong(self, token: Dict[str, Any], state: BlockState) -> str:
|
||||
return '`!' + self.render_children(token, state) + '`!'
|
||||
|
||||
def link(self, token: Dict[str, Any], state: BlockState) -> str:
|
||||
label = token.get('label')
|
||||
text = self.render_children(token, state)
|
||||
out = '`[' + text + '`'
|
||||
if label:
|
||||
return out + '`[' + label + '`'
|
||||
attrs = token['attrs']
|
||||
url = attrs['url']
|
||||
if text == url:
|
||||
return '`[' + text + '`'
|
||||
elif 'mailto:' + text == url:
|
||||
return '`[' + text + '`'
|
||||
out += url
|
||||
return out + ']'
|
||||
|
||||
def image(self, token: Dict[str, Any], state: BlockState) -> str:
|
||||
return self.link(token, state)
|
||||
|
||||
def codespan(self, token: Dict[str, Any], state: BlockState) -> str:
|
||||
return '`=' + token['raw'] + '`='
|
||||
|
||||
def linebreak(self, token: Dict[str, Any], state: BlockState) -> str:
|
||||
return ' \n'
|
||||
|
||||
def softbreak(self, token: Dict[str, Any], state: BlockState) -> str:
|
||||
return '\n'
|
||||
|
||||
def blank_line(self, token: Dict[str, Any], state: BlockState) -> str:
|
||||
return ''
|
||||
|
||||
def inline_html(self, token: Dict[str, Any], state: BlockState) -> str:
|
||||
return ''
|
||||
|
||||
def paragraph(self, token: Dict[str, Any], state: BlockState) -> str:
|
||||
text = self.render_children(token, state)
|
||||
return text + '\n\n'
|
||||
|
||||
def heading(self, token: Dict[str, Any], state: BlockState) -> str:
|
||||
level = token['attrs']['level']
|
||||
if level > 3:
|
||||
level = 3
|
||||
marker = '>' * level
|
||||
text = self.render_children(token, state)
|
||||
return marker + ' ' + text + '\n\n'
|
||||
def thematic_break(self, token: Dict[str, Any], state: BlockState) -> str:
|
||||
return '-\n\n'
|
||||
|
||||
def block_text(self, token: Dict[str, Any], state: BlockState) -> str:
|
||||
return self.render_children(token, state) + '\n'
|
||||
|
||||
def block_code(self, token: Dict[str, Any], state: BlockState) -> str:
|
||||
code = token['raw']
|
||||
if code and code[-1] != '\n':
|
||||
code += '\n'
|
||||
marker = '`='
|
||||
return marker + '\n' + code + marker + '\n\n'
|
||||
|
||||
def block_quote(self, token: Dict[str, Any], state: BlockState) -> str:
|
||||
text = indent(self.render_children(token, state), '>>>>')
|
||||
return text + '\n\n'
|
||||
|
||||
def block_html(self, token: Dict[str, Any], state: BlockState) -> str:
|
||||
return ''
|
||||
|
||||
def block_error(self, token: Dict[str, Any], state: BlockState) -> str:
|
||||
return ''
|
||||
|
||||
def list(self, token: Dict[str, Any], state: BlockState) -> str:
|
||||
return render_list(self, token, state)
|
||||
327
md2txt.py
327
md2txt.py
|
|
@ -5,49 +5,51 @@ Convert Markdown into 80-column DOS-compatible plain text.
|
|||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import json
|
||||
import re
|
||||
import sys
|
||||
from pathlib import Path
|
||||
from typing import Iterable, List, Optional, Set, Tuple
|
||||
from typing import Any, Dict, Iterable, List, Optional, Tuple
|
||||
|
||||
from conversion_core import parse_frontmatter, read_lines, run_conversion
|
||||
from md_types import BlockStyle, FrontMatter
|
||||
from markdown_parser import MarkdownParser
|
||||
from plugins import (
|
||||
available_parsers,
|
||||
available_renderers,
|
||||
get_parser_factory,
|
||||
get_renderer_factory,
|
||||
register_parser,
|
||||
register_renderer,
|
||||
)
|
||||
from text_renderer import TextRenderer
|
||||
|
||||
|
||||
FRONTMATTER_PATTERN = re.compile(r"^---\s*$")
|
||||
INCLUDE_WIKILINK_PATTERN = re.compile(r"^\s*!\[\[(.+?)\]\]\s*$")
|
||||
INCLUDE_DIRECTIVE_PATTERN = re.compile(r"^\s*\{\s*\.include\s+(.+?)\s*\}\s*$")
|
||||
ASCII_BLOCK_PATTERN = re.compile(
|
||||
r"^\s*#\[(?P<label>[^\]]+)\]\((?P<target>[^)]+)\)\s*(?P<attr>\{\s*:[^}]+\s*\})?\s*$"
|
||||
)
|
||||
ASCII_INLINE_PATTERN = re.compile(r"#\[(?P<label>[^\]]+)\]\((?P<target>[^)]+)\)")
|
||||
MMD_ATTR_TAIL_RE = re.compile(r"(.*?)\s*\{\s*:(.+?)\}\s*$")
|
||||
ASCII_SENTINEL_PREFIX = "\u0000ASCII:"
|
||||
def _split_option(token: str) -> Tuple[str, str]:
|
||||
if "=" not in token:
|
||||
raise argparse.ArgumentTypeError("Expected KEY=VALUE format.")
|
||||
key, value = token.split("=", 1)
|
||||
key = key.strip()
|
||||
if not key:
|
||||
raise argparse.ArgumentTypeError("Option key cannot be empty.")
|
||||
return key, value
|
||||
|
||||
|
||||
def _markdown_parser_factory(*, base_style: BlockStyle, **_: Any) -> MarkdownParser:
|
||||
return MarkdownParser(base_style)
|
||||
|
||||
|
||||
def _text_renderer_factory(*, frontmatter: FrontMatter, width: int = 80, **_: Any) -> TextRenderer:
|
||||
return TextRenderer(width=width, frontmatter=frontmatter)
|
||||
|
||||
|
||||
def _parse_int(value: Optional[str], default: int = 0) -> int:
|
||||
if value is None:
|
||||
return default
|
||||
match = re.search(r"-?\d+", value)
|
||||
if not match:
|
||||
return default
|
||||
try:
|
||||
return int(match.group())
|
||||
register_parser("markdown", _markdown_parser_factory)
|
||||
except ValueError:
|
||||
return default
|
||||
pass
|
||||
|
||||
|
||||
def _parse_bool(value: Optional[str], default: bool = False) -> bool:
|
||||
if value is None:
|
||||
return default
|
||||
lowered = value.strip().lower()
|
||||
if lowered in {"true", "yes", "1", "on"}:
|
||||
return True
|
||||
if lowered in {"false", "no", "0", "off"}:
|
||||
return False
|
||||
return default
|
||||
try:
|
||||
register_renderer("text", _text_renderer_factory)
|
||||
except ValueError:
|
||||
pass
|
||||
|
||||
|
||||
def convert_markdown(
|
||||
|
|
@ -56,215 +58,28 @@ def convert_markdown(
|
|||
width: int,
|
||||
frontmatter: FrontMatter,
|
||||
base_path: Optional[Path] = None,
|
||||
parser_name: str = "markdown",
|
||||
renderer_name: str = "text",
|
||||
parser_options: Optional[Dict[str, Any]] = None,
|
||||
renderer_options: Optional[Dict[str, Any]] = None,
|
||||
) -> List[str]:
|
||||
base_dir = (base_path or Path.cwd()).resolve()
|
||||
expanded_lines = _expand_includes(list(lines), base_dir, set())
|
||||
base_style = BlockStyle(
|
||||
align="left",
|
||||
margin_left=max(0, frontmatter.margin_left),
|
||||
margin_right=max(0, frontmatter.margin_right),
|
||||
parser_factory = get_parser_factory(parser_name)
|
||||
renderer_factory = get_renderer_factory(renderer_name)
|
||||
effective_renderer_options: Dict[str, Any] = {"width": width}
|
||||
if renderer_options:
|
||||
effective_renderer_options.update(renderer_options)
|
||||
rendered = run_conversion(
|
||||
lines,
|
||||
frontmatter=frontmatter,
|
||||
parser_factory=parser_factory,
|
||||
parser_options=parser_options,
|
||||
renderer_factory=renderer_factory,
|
||||
renderer_options=effective_renderer_options,
|
||||
base_path=base_path,
|
||||
)
|
||||
parser = MarkdownParser(base_style)
|
||||
renderer = TextRenderer(width=width, frontmatter=frontmatter)
|
||||
for event in parser.parse(expanded_lines):
|
||||
renderer.handle_event(event)
|
||||
return renderer.finalize()
|
||||
|
||||
|
||||
def _expand_includes(lines: List[str], base_dir: Path, include_stack: Set[Path]) -> List[str]:
|
||||
expanded: List[str] = []
|
||||
for line in lines:
|
||||
ascii_segments = _extract_ascii_segments(line, base_dir)
|
||||
if ascii_segments is not None:
|
||||
for sentinel_line, attr_line in ascii_segments:
|
||||
expanded.append(sentinel_line)
|
||||
if attr_line is not None:
|
||||
expanded.append(attr_line)
|
||||
continue
|
||||
target = _extract_include_target(line)
|
||||
if target is None:
|
||||
expanded.append(line)
|
||||
continue
|
||||
target_path = (base_dir / target).resolve()
|
||||
if target_path in include_stack:
|
||||
raise RuntimeError(f"Circular include detected for '{target_path}'.")
|
||||
if not target_path.exists():
|
||||
raise FileNotFoundError(f"Included file '{target_path}' was not found.")
|
||||
include_stack.add(target_path)
|
||||
included_lines = read_lines(target_path)
|
||||
_, include_body = parse_frontmatter(included_lines)
|
||||
included_content = _expand_includes(include_body, target_path.parent, include_stack)
|
||||
expanded.extend(included_content)
|
||||
include_stack.remove(target_path)
|
||||
return expanded
|
||||
|
||||
|
||||
def _extract_include_target(line: str) -> Optional[str]:
|
||||
stripped = line.rstrip("\n")
|
||||
match = INCLUDE_WIKILINK_PATTERN.match(stripped)
|
||||
if match:
|
||||
return _normalize_include_target(match.group(1))
|
||||
match = INCLUDE_DIRECTIVE_PATTERN.match(stripped)
|
||||
if match:
|
||||
return _normalize_include_target(match.group(1))
|
||||
return None
|
||||
|
||||
|
||||
def _normalize_include_target(value: str) -> str:
|
||||
trimmed = value.strip()
|
||||
if len(trimmed) >= 2 and ((trimmed[0] == trimmed[-1]) and trimmed[0] in {"'", '"'}):
|
||||
trimmed = trimmed[1:-1].strip()
|
||||
return trimmed
|
||||
|
||||
|
||||
def _extract_ascii_segments(line: str, base_dir: Path) -> Optional[List[Tuple[str, Optional[str]]]]:
|
||||
stripped_line = line.rstrip("\n")
|
||||
block_match = ASCII_BLOCK_PATTERN.match(stripped_line)
|
||||
if block_match:
|
||||
label = block_match.group("label")
|
||||
target = block_match.group("target")
|
||||
attr_text = block_match.group("attr")
|
||||
sentinel = _make_ascii_sentinel(label, target, base_dir)
|
||||
attr_line = f"{attr_text}\n" if attr_text else None
|
||||
return [(sentinel, attr_line)]
|
||||
|
||||
matches = list(ASCII_INLINE_PATTERN.finditer(stripped_line))
|
||||
if not matches:
|
||||
return None
|
||||
|
||||
pieces: List[dict] = []
|
||||
last_end = 0
|
||||
for match in matches:
|
||||
prefix = stripped_line[last_end : match.start()]
|
||||
if prefix.strip():
|
||||
return None
|
||||
label = match.group("label")
|
||||
target = match.group("target")
|
||||
block_type, block_name, align = _parse_ascii_label(label)
|
||||
normalized_target = _normalize_include_target(target)
|
||||
target_path = (base_dir / normalized_target).resolve()
|
||||
if not target_path.exists():
|
||||
raise FileNotFoundError(f"ASCII art file '{target_path}' was not found.")
|
||||
pieces.append(
|
||||
{
|
||||
"type": block_type,
|
||||
"name": block_name,
|
||||
"path": str(target_path),
|
||||
"align": align,
|
||||
}
|
||||
)
|
||||
last_end = match.end()
|
||||
|
||||
suffix = stripped_line[last_end:]
|
||||
if suffix.strip():
|
||||
return None
|
||||
|
||||
if not pieces:
|
||||
return None
|
||||
|
||||
sentinel = f"{ASCII_SENTINEL_PREFIX}{json.dumps({'pieces': pieces})}\n"
|
||||
return [(sentinel, None)]
|
||||
|
||||
|
||||
def _make_ascii_sentinel(label: str, target: str, base_dir: Path) -> str:
|
||||
block_type, block_name, align = _parse_ascii_label(label)
|
||||
normalized_target = _normalize_include_target(target)
|
||||
target_path = (base_dir / normalized_target).resolve()
|
||||
if not target_path.exists():
|
||||
raise FileNotFoundError(f"ASCII art file '{target_path}' was not found.")
|
||||
payload = {
|
||||
"pieces": [
|
||||
{
|
||||
"type": block_type,
|
||||
"name": block_name,
|
||||
"path": str(target_path),
|
||||
"align": align,
|
||||
}
|
||||
]
|
||||
}
|
||||
return f"{ASCII_SENTINEL_PREFIX}{json.dumps(payload)}\n"
|
||||
|
||||
|
||||
def _parse_ascii_label(label: str) -> Tuple[str, str, Optional[str]]:
|
||||
tokens = label.strip().split()
|
||||
non_colon: List[str] = []
|
||||
align: Optional[str] = None
|
||||
for token in tokens:
|
||||
if token.startswith(":"):
|
||||
tag = token[1:].strip().lower()
|
||||
if tag in {"left", "right", "center", "centre"}:
|
||||
align = "center" if tag in {"center", "centre"} else tag
|
||||
# ignore other colon tags for now
|
||||
else:
|
||||
non_colon.append(token)
|
||||
|
||||
block_type = non_colon[0] if non_colon else "custom"
|
||||
block_name = " ".join(non_colon[1:]) if len(non_colon) > 1 else ""
|
||||
return block_type, block_name, align
|
||||
|
||||
|
||||
def parse_frontmatter(lines: List[str]) -> Tuple[FrontMatter, List[str]]:
|
||||
if not lines or not FRONTMATTER_PATTERN.match(lines[0]):
|
||||
return FrontMatter(), lines
|
||||
frontmatter: dict[str, str] = {}
|
||||
idx = 1
|
||||
while idx < len(lines):
|
||||
if FRONTMATTER_PATTERN.match(lines[idx]):
|
||||
break
|
||||
if ":" in lines[idx]:
|
||||
key, value = lines[idx].split(":", 1)
|
||||
frontmatter[key.strip()] = value.strip()
|
||||
idx += 1
|
||||
if idx >= len(lines):
|
||||
return FrontMatter(), lines
|
||||
remaining = lines[idx + 1 :] if idx + 1 < len(lines) else []
|
||||
paragraph_spacing_value = frontmatter.get("paragraph_spacing")
|
||||
if paragraph_spacing_value is None:
|
||||
paragraph_spacing_value = frontmatter.get("lines_between_paragraphs")
|
||||
if paragraph_spacing_value is None:
|
||||
paragraph_spacing_value = frontmatter.get("paragraph_lines")
|
||||
default_wrap_indent = 2
|
||||
wrap_code_blocks = _parse_bool(frontmatter.get("wrap_code_blocks"), False)
|
||||
code_block_wrap_indent = default_wrap_indent if wrap_code_blocks else 0
|
||||
code_block_wrap_value = frontmatter.get("code_block_wrap")
|
||||
if code_block_wrap_value is not None:
|
||||
normalized_wrap = code_block_wrap_value.strip()
|
||||
if normalized_wrap:
|
||||
if re.fullmatch(r"-?\d+", normalized_wrap):
|
||||
wrap_code_blocks = True
|
||||
code_block_wrap_indent = max(0, _parse_int(normalized_wrap, default_wrap_indent))
|
||||
else:
|
||||
wrap_flag = _parse_bool(normalized_wrap, wrap_code_blocks)
|
||||
wrap_code_blocks = wrap_flag
|
||||
code_block_wrap_indent = default_wrap_indent if wrap_flag else 0
|
||||
code_block_line_numbers = _parse_bool(frontmatter.get("code_block_line_numbers"), True)
|
||||
blockquote_bars = _parse_bool(frontmatter.get("blockquote_bars"), True)
|
||||
list_marker_indent = max(0, _parse_int(frontmatter.get("list_marker_indent"), 0))
|
||||
list_text_spacing = max(0, _parse_int(frontmatter.get("list_text_spacing"), 1))
|
||||
fm = FrontMatter(
|
||||
h1_font=frontmatter.get("h1_font", "standard").strip() or "standard",
|
||||
h2_font=frontmatter.get("h2_font", "standard").strip() or "standard",
|
||||
h3_font=frontmatter.get("h3_font", "standard").strip() or "standard",
|
||||
margin_left=_parse_int(frontmatter.get("margin_left"), 0),
|
||||
margin_right=_parse_int(frontmatter.get("margin_right"), 0),
|
||||
paragraph_spacing=max(0, _parse_int(paragraph_spacing_value, 0)),
|
||||
hyphenate=_parse_bool(frontmatter.get("hyphenate"), False),
|
||||
hyphen_lang=(frontmatter.get("hyphen_lang") or "en_US").strip() or "en_US",
|
||||
figlet_fallback=_parse_bool(frontmatter.get("figlet_fallback"), False),
|
||||
header_spacing=max(0, _parse_int(frontmatter.get("header_spacing"), 2)),
|
||||
wrap_code_blocks=wrap_code_blocks,
|
||||
code_block_wrap_indent=code_block_wrap_indent,
|
||||
code_block_line_numbers=code_block_line_numbers,
|
||||
blockquote_bars=blockquote_bars,
|
||||
list_marker_indent=list_marker_indent,
|
||||
list_text_spacing=list_text_spacing,
|
||||
)
|
||||
return fm, remaining
|
||||
|
||||
|
||||
def read_lines(path: Path) -> List[str]:
|
||||
with path.open("r", encoding="utf-8") as handle:
|
||||
return handle.readlines()
|
||||
if not isinstance(rendered, list):
|
||||
raise TypeError("Renderer plugin returned unsupported output for md2txt CLI.")
|
||||
return rendered
|
||||
|
||||
|
||||
def write_output(path: Optional[Path], lines: List[str]) -> None:
|
||||
|
|
@ -281,6 +96,34 @@ def build_parser() -> argparse.ArgumentParser:
|
|||
parser.add_argument("input_path", type=Path, help="Path to the Markdown input file.")
|
||||
parser.add_argument("-o", "--output", type=Path, help="Optional path to write the resulting text file.")
|
||||
parser.add_argument("--width", type=int, default=80, help="Maximum column width (default: 80).")
|
||||
parser.add_argument(
|
||||
"--parser",
|
||||
default="markdown",
|
||||
choices=available_parsers() or ["markdown"],
|
||||
help="Name of the parser plugin to use.",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--renderer",
|
||||
default="text",
|
||||
choices=available_renderers() or ["text"],
|
||||
help="Name of the renderer plugin to use.",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--parser-option",
|
||||
action="append",
|
||||
default=[],
|
||||
type=_split_option,
|
||||
metavar="KEY=VALUE",
|
||||
help="Additional parser option in KEY=VALUE form (may repeat).",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--renderer-option",
|
||||
action="append",
|
||||
default=[],
|
||||
type=_split_option,
|
||||
metavar="KEY=VALUE",
|
||||
help="Additional renderer option in KEY=VALUE form (may repeat).",
|
||||
)
|
||||
return parser
|
||||
|
||||
|
||||
|
|
@ -289,12 +132,22 @@ def main(argv: Optional[List[str]] = None) -> int:
|
|||
args = parser.parse_args(argv)
|
||||
lines = read_lines(args.input_path)
|
||||
frontmatter, content = parse_frontmatter(lines)
|
||||
parser_options = dict(args.parser_option or [])
|
||||
renderer_options = dict(args.renderer_option or [])
|
||||
try:
|
||||
converted_lines = convert_markdown(
|
||||
content,
|
||||
width=args.width,
|
||||
frontmatter=frontmatter,
|
||||
base_path=args.input_path.parent,
|
||||
parser_name=args.parser,
|
||||
renderer_name=args.renderer,
|
||||
parser_options=parser_options or None,
|
||||
renderer_options=renderer_options or None,
|
||||
)
|
||||
except (KeyError, TypeError) as exc:
|
||||
sys.stderr.write(f"{exc}\n")
|
||||
return 2
|
||||
write_output(args.output, converted_lines)
|
||||
return 0
|
||||
|
||||
|
|
|
|||
25
plugin_registry.py
Normal file
25
plugin_registry.py
Normal file
|
|
@ -0,0 +1,25 @@
|
|||
from __future__ import annotations
|
||||
|
||||
from typing import Dict, Generic, List, TypeVar
|
||||
|
||||
|
||||
T = TypeVar("T")
|
||||
|
||||
|
||||
class PluginRegistry(Generic[T]):
|
||||
def __init__(self) -> None:
|
||||
self._factories: Dict[str, T] = {}
|
||||
|
||||
def register(self, name: str, factory: T) -> None:
|
||||
if name in self._factories:
|
||||
raise ValueError(f"Plugin '{name}' is already registered.")
|
||||
self._factories[name] = factory
|
||||
|
||||
def get(self, name: str) -> T:
|
||||
try:
|
||||
return self._factories[name]
|
||||
except KeyError as exc:
|
||||
raise KeyError(f"Plugin '{name}' is not registered.") from exc
|
||||
|
||||
def names(self) -> List[str]:
|
||||
return sorted(self._factories.keys())
|
||||
34
plugins.py
Normal file
34
plugins.py
Normal file
|
|
@ -0,0 +1,34 @@
|
|||
from __future__ import annotations
|
||||
|
||||
from typing import Any
|
||||
|
||||
from conversion_core import ParserFactory, RendererFactory
|
||||
from plugin_registry import PluginRegistry
|
||||
|
||||
|
||||
parser_plugins = PluginRegistry[ParserFactory]()
|
||||
renderer_plugins = PluginRegistry[RendererFactory[Any]]()
|
||||
|
||||
|
||||
def register_parser(name: str, factory: ParserFactory) -> None:
|
||||
parser_plugins.register(name, factory)
|
||||
|
||||
|
||||
def register_renderer(name: str, factory: RendererFactory[Any]) -> None:
|
||||
renderer_plugins.register(name, factory)
|
||||
|
||||
|
||||
def get_parser_factory(name: str) -> ParserFactory:
|
||||
return parser_plugins.get(name)
|
||||
|
||||
|
||||
def get_renderer_factory(name: str) -> RendererFactory[Any]:
|
||||
return renderer_plugins.get(name)
|
||||
|
||||
|
||||
def available_parsers() -> list[str]:
|
||||
return parser_plugins.names()
|
||||
|
||||
|
||||
def available_renderers() -> list[str]:
|
||||
return renderer_plugins.names()
|
||||
Loading…
Add table
Add a link
Reference in a new issue