Introduction

Overview

Moltext is an agent-native documentation compiler designed to transform human-centric web documentation into high-density, deterministic context for Large Language Models (LLMs) and autonomous agents.

Traditional documentation is built for humans; it is fractured across multiple pages and cluttered with navigation menus, CSS, and JavaScript. For AI agents, this "noise" leads to context window bloat and hallucinations. Moltext solves this by crawling, cleaning, and compiling entire documentation sites into a single, structured context.md file—the "gold standard" for agentic memory.

Key Capabilities

• Recursive Crawling: Automatically traverses documentation domains to gather all relevant sub-pages while staying within the specified scope.
• Noise Reduction: Strips HTML boilerplate (navbars, footers, scripts) to extract only the core technical content.
• LLM-Powered Normalization: Optionally uses models (OpenAI or local via Ollama) to compress content into a high-density, agent-optimized format.
• Raw Mode: Provides a zero-dependency, pure structural conversion from HTML to Markdown without requiring an LLM.
• Agentic Integration: Serves as a native skill for the OpenClaw and Moltbot ecosystems, acting as the primary ingestion layer for new tools and libraries.

Core Concepts

The context.md File

The primary output of Moltext is a context.md file. This file is designed to be dropped directly into an agent's context window or a RAG (Retrieval-Augmented Generation) vector store. It prioritizes API signatures, code blocks, and technical constraints over conversational filler.

Processing Modes

Moltext offers three primary processing flows depending on your infrastructure:

1. Raw Mode (--raw): The fastest method. It uses turndown and cheerio to create clean Markdown. No API keys or inference required.
2. Local Inference: Connects to local providers like Ollama or LM Studio via the OpenAI-compatible /v1 endpoint.
3. Cloud AI: Uses OpenAI models (e.g., gpt-4o-mini) to perform sophisticated structural compression and error correction on broken Markdown.

Quick Start

Moltext is a CLI tool. Once installed via NPM, you can begin compiling documentation immediately.

Installation

npm install -g moltext

Basic Usage

To compile a documentation site using the recommended Raw Mode (no LLM required):

moltext https://docs.example.com --raw -o project_context.md

To use an LLM for high-density compression (requires OPENAI_API_KEY or a local base URL):

moltext https://docs.example.com --model gpt-4o-mini --output context.md

Use Cases

• Agent Onboarding: Give an autonomous agent the ability to use a new library by compiling the library's docs into its memory.
• Offline Docs: Create a portable, searchable Markdown version of online documentation.
• RAG Pre-processing: Prepare clean, high-signal data for vector databases without manual cleaning.