LLM Synthesis Logic

Moltext does more than just scrape web pages; it transforms human-centric prose into high-density, agent-readable context. This process is governed by a multi-stage synthesis pipeline that prioritizes technical accuracy and structural density over narrative flow.

The Transformation Pipeline

When you run Moltext without the --raw flag, documentation passes through four distinct phases:

Clutter Extraction: The system uses cheerio to strip non-content elements including navigation bars, footers, scripts, styles, and sidebars. This isolates the core documentation body.
Structural Normalization: Cleaned HTML is converted into GitHub Flavored Markdown (GFM) using Turndown.
Agentic Synthesis: The Markdown is fed into an LLM (e.g., gpt-4o-mini or a local model) with a specialized system prompt designed to "compile" the text.
Deterministic Output: The final context.md is appended with the source URL and the synthesized content, separated by deterministic delimiters.

The Synthesis Prompt

The core of Moltext’s "intelligence" lies in its internal system instructions. When synthesizing, the LLM is directed to act as a Strict Documentation Compiler. The logic follows these five pillars:

Density over Narrative: Strips all conversational filler (e.g., "Welcome to this guide," "In the next section, we will explore...").
Vector Optimization: Highlights keywords and logical hierarchies to ensure the resulting file is easily indexed by RAG (Retrieval-Augmented Generation) systems.
Signature Preservation: Enforces a "Zero-Loss" policy for code blocks, API signatures, and technical constraints.
Structural Compression: Collapses redundant explanations into concise bullet points or logical definitions.
Markdown Repair: Fixes broken syntax or table structures inherited from the raw HTML-to-Markdown conversion.

Configuration and Model Selection

You can influence the synthesis logic by selecting different models or pointing to local inference servers. This is useful for balancing cost, speed, and privacy.

Using OpenAI (Default)

The default model is gpt-4o-mini, which provides a high balance of speed and structural reasoning.

moltext https://docs.example.com -k your_openai_key

Using Local Models (Ollama/LM Studio)

For privacy-sensitive documentation or offline use, you can redirect the synthesis logic to a local LLM. We recommend models like llama3 or mistral for high-density compression.

moltext https://docs.example.com \
  --base-url http://localhost:11434/v1 \
  --model llama3

Raw Mode vs. Synthesized Mode

Fallback Mechanism

If the LLM synthesis fails (due to rate limits, API errors, or connectivity issues), Moltext gracefully falls back to Raw Mode for that specific page. This ensures that your context.md remains complete, even if the "agentic cleanup" for a specific section is skipped.