Testing Strategy

To maintain the integrity of the agentic context produced by Moltext, any new features—specifically custom crawlers or processor enhancements—must undergo a rigorous validation process. Since Moltext is designed to produce deterministic output for autonomous agents, testing focuses on structural accuracy and noise reduction.

Core Testing Objectives

Crawler Determinism: Ensure the crawler stays within the specified domain and correctly identifies all relevant documentation nodes without entering infinite loops.
Sanitization Integrity: Verify that Processor.cleanHtml() removes "human-first" noise (navbars, footers, scripts) without stripping critical technical data (code signatures, API tables).
Markdown Consistency: Ensure the TurndownService configuration produces valid, high-density Markdown that an LLM or agent can parse without hallucination.

Component-Level Validation

1. Crawler Logic

When extending the Crawler class or modifying URL normalization, tests should validate:

Domain Scoping: Links outside the base URL hostname must be ignored.
Protocol Handling: Only http: and https: protocols should be queued.
Recursion Limits: The --limit flag must strictly terminate crawling once the page count is reached.

Example Test Case (Mocking Axios):

// Ensure the crawler correctly extracts local links and ignores external ones
const crawler = new Crawler('https://docs.example.com');
const links = await crawler.crawl(5); 
// Validate that no link in 'links' belongs to 'google.com'

2. Processor & HTML Cleaning

The Processor is the most sensitive component. Testing involves passing raw HTML snippets through the cleanHtml method to ensure the output is "Agent-Ready."

Requirements for New Processors:

Element Stripping: Verify <nav>, <footer>, and <script> tags are removed.
Content Preservation: Ensure <main>, <article>, or specific technical containers are preserved.
Raw vs. LLM Mode: Compare output from --raw (pure Markdown) against LLM-processed output to ensure no technical data was lost during summarization.

const processor = new Processor('mock-key');
const html = '<html><body><nav>Menu</nav><main><h1>API</h1><code>GET /v1</code></main></body></html>';
const cleaned = processor.cleanHtml(html);
// Assert: cleaned should NOT contain "Menu"
// Assert: cleaned SHOULD contain "GET /v1"

Integration Testing via CLI

Before a release, run the tool against a controlled local documentation server to verify the end-to-end flow.

The Raw Pipeline: Run moltext <local_url> --raw and check the context.md for broken Markdown links or malformed code blocks.
The LLM Pipeline: Run with a local inference engine (e.g., Ollama) to ensure the system prompt correctly compresses the content without stripping API signatures.
```
moltext http://localhost:8080 --base-url http://localhost:11434/v1 --model llama3
```

Mocking External Services

Network: Use libraries like nock or msw to intercept axios requests to prevent external network calls during unit tests.
OpenAI API: When testing the Processor, mock the openai.chat.completions.create method. Tests should verify that the processor handles API timeouts or "dummy-key" scenarios gracefully by falling back to raw Markdown.

Quality Benchmarks

An ingestion is considered "Successful" if the resulting context.md:

Contains 0% HTML tags.
Retains 100% of the code blocks found in the source.
Follows a flat, header-driven structure (## Source: [Title](URL)).