Docfind

A high-performance document search engine built in Rust with WebAssembly support.

Website (Demo) | Source

A high-performance document search engine built in Rust with WebAssembly support. Combines full-text search using FST (Finite State Transducers) with FSST compression for efficient storage and fast fuzzy matching capabilities.

Installation

macOS/Linux:

curl -fsSL https://microsoft.github.io/docfind/install.sh | sh

Windows (PowerShell):

irm https://microsoft.github.io/docfind/install.ps1 | iex

The installer will:

Manual Installation

Download the binary for your platform from the latest release:

Rename it to docfind (or docfind.exe on Windows), make it executable, and place it in your PATH.

Building from Source

Before building from source, ensure you have the following installed:

  1. Rust - rustup.rs
  2. wasm-pack - drager.github.io/wasm-pack
  3. Node.js - nodejs.org (required for esbuild)

Usage

Creating a Search Index

Prepare a JSON file with your documents:

[
  {
    "title": "Getting Started",
    "category": "docs",
    "href": "/docs/getting-started",
    "body": "This guide will help you get started."
  },
  {
    "title": "API Reference",
    "category": "reference",
    "href": "/docs/api",
    "body": "Complete API documentation for all search functions and configuration options."
  }
]

Build the index and generate a WASM module:

docfind documents.json output

This creates:

Using in the Browser

<script type="module">
import search from 'docfind.js';

const documents = await search('needle');
console.log(documents);
</script>

How It Works

Loading

```mermaid
flowchart LR
    A(\[documents.json\]) --> B\[docfind\]
    B --> C\[Keyword Extraction<br/>RAKE\]
    B --> E\[FSST Compression<br/>document strings\]
    C --> D\[FST Map<br/>keywords → docs\]
    D --> F\[\[Index\]\]
    E --> F
    F --> G(\[docfind\_bg.wasm<br/>+ docfind.js\])
    
    style A fill:#e1f5ff
    style G fill:#e1f5ff
    style F fill:#ffffcc
```

  1. Indexing Phase (CLI):

    • Extracts keywords from document titles, categories, and bodies
    • Uses RAKE algorithm to identify important multi-word phrases
    • Assigns relevance scores based on keyword source (metadata > title > body)
    • Builds an FST mapping keywords to document indices
    • Compresses all document strings using FSST
    • Serializes the index using Postcard (binary format)

  2. Embedding Phase (CLI):

    • Parses the pre-compiled WASM module
    • Expands WASM memory to accommodate the index
    • Patches global variables (INDEX_BASE, INDEX_LEN) with actual values
    • Adds the index as a new data segment in the WASM binary

  3. Search Phase (WASM):

    • Deserializes the embedded index on first use
    • Performs fuzzy matching using Levenshtein automaton
    • Combines results from multiple keywords with score accumulation
    • Decompresses matching document strings on demand
    • Returns ranked results as JavaScript objects

Dependencies

Performance

The combination of FST and FSST provides:

References

Prior Art

This project builds on the rich ecosystem of search technologies:

Technical Foundations

Key technologies and concepts that inspired and power docfind:

Tags: presentation   browser   search   web assembly  

Last modified 11 September 2026