Module: Lich::Util::TextStripper

Defined in:
documented/util/textstripper.rb

Overview

Utility module for stripping markup, HTML, and XML from text

This module provides methods to remove various types of formatting from text strings, including HTML tags, XML tags, and Markdown markup. It uses the Kramdown library for HTML and Markdown parsing, and Ox for proper XML parsing.

Examples:

Basic usage

TextStripper.strip("<p>Hello</p>", TextStripper::Mode::HTML)
# => "Hello"

Stripping XML

TextStripper.strip("<root>data</root>", TextStripper::Mode::XML)
# => "data"

Stripping Markdown

TextStripper.strip("**bold** text", TextStripper::Mode::MARKUP)
# => "bold text"

TextStripper.strip("**bold** text", TextStripper::Mode::MARKDOWN)
# => "bold text"

Using symbol shortcuts (backward compatible)

TextStripper.strip("<p>Hello</p>", :html)
# => "Hello"

TextStripper.strip("**bold** text", :markdown)
# => "bold text"

Defined Under Namespace

Modules: Mode

Constant Summary collapse

MODE_TO_INPUT_FORMAT =

This constant is part of a private API. You should avoid using this constant if possible, as it may be removed or be changed in the future.

Note:

XML mode does not use Kramdown; it uses Ox instead

Note:

MARKDOWN is an alias for MARKUP and uses the same input format

Map of modes to their corresponding Kramdown input formats

Returns:

  • (Hash<Symbol, String>) —

    Mapping of modes to Kramdown input types

{
  Mode::HTML     => 'html',
  Mode::MARKUP   => 'GFM',
  Mode::MARKDOWN => 'GFM'
}.freeze

Class Method Summary collapse

Class Method Details

.entity_to_char(entity) ⇒ String

This method is part of a private API. You should avoid using this method if possible, as it may be removed or be changed in the future.

Convert HTML entity codes to characters

Parameters:

  • entity (Kramdown::Utils::Entities::Entity, Symbol) —

    The entity to convert

Returns:

  • (String) —

    The character representation



435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
 
# File 'documented/util/textstripper.rb', line 435

def self.entity_to_char(entity)
  if entity.respond_to?(:char)
    entity.char
  else
    # Fallback for symbol entities
    case entity
    when :nbsp then ' '
    when :lt then '<'
    when :gt then '>'
    when :amp then '&'
    when :quot then '"'
    else entity.to_s
    end
  end
end

.extract_text(element) ⇒ String

This method is part of a private API. You should avoid using this method if possible, as it may be removed or be changed in the future.

Note:

This method handles different element types:

  • :text - Returns the text value directly
  • :entity - Converts HTML entities to characters
  • :smart_quote - Converts smart quotes to regular quotes
  • :codeblock, :codespan - Returns code content as plain text
  • :br - Converts line breaks to newlines
  • :blank - Converts blank lines to newlines
  • All other elements - Recursively processes children

Extract plain text from a Kramdown element tree

Recursively traverses the Kramdown element tree and extracts all text content, ignoring markup and formatting.

Parameters:

  • element (Kramdown::Element) —

    The root element to extract text from

Returns:

  • (String) —

    The extracted plain text



398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
 
# File 'documented/util/textstripper.rb', line 398

def self.extract_text(element)
  return '' if element.nil?

  case element.type
  when :text
    element.value
  when :entity
    # Convert HTML entities (e.g., &nbsp; -> space)
    entity_to_char(element.value)
  when :smart_quote
    # Convert smart quotes to regular quotes
    smart_quote_to_char(element.value)
  when :codeblock, :codespan
    # Return code content as plain text
    element.value
  when :br
    # Convert line breaks to newlines
    "\n"
  when :blank
    # Blank lines become newlines
    "\n"
  else
    # For all other elements (p, div, span, etc.), recursively process children
    if element.children
      element.children.map { |child| extract_text(child) }.join
    else
      ''
    end
  end
end

.extract_xml_text(element) ⇒ String

This method is part of a private API. You should avoid using this method if possible, as it may be removed or be changed in the future.

Note:

This method processes all child nodes including:

  • Text nodes (plain Strings in Ox's generic model)
  • CDATA sections (Ox::CData)
  • Nested elements (recursively)

Extract plain text from an Ox element tree

Recursively traverses the Ox element tree and extracts all text content, including CDATA sections.

Parameters:

  • element (Ox::Element) —

    The root element to extract text from

Returns:

  • (String) —

    The extracted plain text



360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
 
# File 'documented/util/textstripper.rb', line 360

def self.extract_xml_text(element)
  return '' if element.nil?

  # In Ox's generic model text nodes are plain Strings and CDATA sections
  # are Ox::CData; comments/PIs are other node types and are ignored.
  element.nodes.map do |node|
    case node
    when Ox::CData
      node.value
    when Ox::Element
      extract_xml_text(node)
    when String
      node
    else
      ''
    end
  end.join
end

.log_error(message, exception) ⇒ void

This method is part of a private API. You should avoid using this method if possible, as it may be removed or be changed in the future.

This method returns an undefined value.

Log an error message to both the response output and Lich log

Parameters:

  • message (String) —

    The base error message

  • exception (Exception) —

    The exception that occurred



261
262
263
264
265
 
# File 'documented/util/textstripper.rb', line 261

def self.log_error(message, exception)
  full_message = "TextStripper: #{message} (#{exception.class}: #{exception.message}). Returning original."
  respond(full_message)
  Lich.log(full_message)
end

.requires_kramdown?(mode) ⇒ Boolean

This method is part of a private API. You should avoid using this method if possible, as it may be removed or be changed in the future.

Check if a mode requires kramdown

Parameters:

  • mode (Symbol) —

    The mode to check

Returns:

  • (Boolean) —

    true if the mode requires kramdown, false otherwise



134
135
136
 
# File 'documented/util/textstripper.rb', line 134

def self.requires_kramdown?(mode)
  MODE_TO_INPUT_FORMAT.key?(mode)
end

.smart_quote_to_char(quote_type) ⇒ String

This method is part of a private API. You should avoid using this method if possible, as it may be removed or be changed in the future.

Convert smart quote symbols to regular characters

Parameters:

  • quote_type (Symbol) —

    The smart quote type (:lsquo, :rsquo, :ldquo, :rdquo)

Returns:

  • (String) —

    The quote character



457
458
459
460
461
462
463
 
# File 'documented/util/textstripper.rb', line 457

def self.smart_quote_to_char(quote_type)
  case quote_type
  when :lsquo, :rsquo then "'"
  when :ldquo, :rdquo then '"'
  else quote_type.to_s
  end
end

.strip(text, mode) ⇒ String

Note:

If Kramdown or Ox parsing fails, a warning is issued and the original text is returned unchanged

Strip markup/code from text based on the specified mode

This method provides a unified interface for removing different types of markup from text. It handles HTML tags, XML tags, and Markdown formatting based on the mode parameter.

Examples:

Stripping HTML with constant

TextStripper.strip("<p>Hello <strong>World</strong></p>", Mode::HTML)
# => "Hello World"

Stripping HTML with symbol (backward compatible)

TextStripper.strip("<p>Hello <strong>World</strong></p>", :html)
# => "Hello World"

Stripping XML

TextStripper.strip("<root><item>data</item></root>", Mode::XML)
# => "data"

Stripping XML with namespaces

TextStripper.strip("<root xmlns='http://example.com'><item>data</item></root>", Mode::XML)
# => "data"

Stripping Markdown with MARKUP constant

TextStripper.strip("**bold** and *italic*", Mode::MARKUP)
# => "bold and italic"

Stripping Markdown with MARKDOWN constant

TextStripper.strip("**bold** and *italic*", Mode::MARKDOWN)
# => "bold and italic"

Stripping Markdown with symbol

TextStripper.strip("**bold** and *italic*", :markdown)
# => "bold and italic"

Invalid mode

TextStripper.strip("text", :invalid)
# raises ArgumentError: Invalid mode: invalid. Use one of: html, xml, markup, markdown

Parameters:

  • text (String) —

    The text to process

  • mode (Symbol, String, Mode constant) —

    The stripping mode to use. Valid options are:

    • Mode::HTML or :html - Strip HTML tags using Kramdown
    • Mode::XML or :xml - Strip XML tags using Ox
    • Mode::MARKUP or :markup - Strip Markdown formatting (GitHub Flavored Markdown) using Kramdown
    • Mode::MARKDOWN or :markdown - Strip Markdown formatting (alias for MARKUP)

Returns:

  • (String) —

    The stripped text with formatting removed

  • (String) —

    Empty string if input text is nil or empty

  • (String) —

    Original text if parsing fails

Raises:

  • (ArgumentError) —

    if mode is not one of the valid modes or is not a Symbol/String



191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
 
# File 'documented/util/textstripper.rb', line 191

def self.strip(text, mode)
  return "" if text.nil? || text.empty?

  # Validate mode BEFORE entering the rescue block
  # This allows ArgumentError to propagate to the caller as documented
  validated_mode = validate_mode(mode)

  # Check if kramdown is required and available
  if requires_kramdown?(validated_mode) && !KRAMDOWN_LOADED
    respond("Need to restart Lich5 in order to use this method.")
    return text
  end

  # Route to appropriate parsing method based on mode
  case validated_mode
  when Mode::XML
    strip_xml_with_ox(text)
  else
    strip_with_kramdown(text, validated_mode)
  end
rescue Kramdown::Error => e
  # Handle Kramdown parsing errors (HTML/MARKUP/MARKDOWN modes)
  log_error("Failed to parse #{validated_mode}", e)
  text
rescue Ox::ParseError => e
  # Handle Ox parsing errors (XML mode)
  log_error("Failed to parse #{validated_mode}", e)
  text
rescue StandardError => e
  # Catch any other unexpected errors during parsing
  log_error("Unexpected error during #{validated_mode} parsing", e)
  text
end

.strip_html(text) ⇒ String

This method is part of a private API. You should avoid using this method if possible, as it may be removed or be changed in the future.

Note:

This method is called internally by #strip when mode is Mode::HTML

Note:

Uses Kramdown for HTML parsing

Strip HTML tags and return plain text

Parses the input as HTML and removes all HTML tags, returning only the text content. This is a convenience wrapper around #strip_with_kramdown.

Examples:

Basic HTML stripping

TextStripper.strip_html("<p>Hello</p>")
# => "Hello"

Nested tags

TextStripper.strip_html("<div><p>Hello <strong>World</strong></p></div>")
# => "Hello World"

Parameters:

  • text (String) —

    The HTML text to process

Returns:

  • (String) —

    Plain text with HTML tags removed and whitespace trimmed

See Also:

  • #strip


487
488
489
490
491
492
493
494
 
# File 'documented/util/textstripper.rb', line 487

def self.strip_html(text)
  unless KRAMDOWN_LOADED
    respond("Need to restart Lich5 in order to use this method.")
    return text
  end

  strip_with_kramdown(text, Mode::HTML)
end

.strip_markdown(text) ⇒ String

This method is part of a private API. You should avoid using this method if possible, as it may be removed or be changed in the future.

Note:

This is functionally identical to strip_markup

Note:

Uses GitHub Flavored Markdown (GFM) as the input format

Strip Markdown formatting and return plain text (alias for strip_markup)

This is an alias for #strip_markup that provides a more explicit method name for working with Markdown content. Both :markup and :markdown modes are functionally identical.

Examples:

Bold and italic

TextStripper.strip_markdown("**bold** and *italic*")
# => "bold and italic"

Links

TextStripper.strip_markdown("[link text](http://example.com)")
# => "link text"

Parameters:

  • text (String) —

    The Markdown text to process

Returns:

  • (String) —

    Plain text with Markdown formatting removed and whitespace trimmed

See Also:

  • #strip_markup


591
592
593
594
595
596
597
598
 
# File 'documented/util/textstripper.rb', line 591

def self.strip_markdown(text)
  unless KRAMDOWN_LOADED
    respond("Need to restart Lich5 in order to use this method.")
    return text
  end

  strip_with_kramdown(text, Mode::MARKDOWN)
end

.strip_markup(text) ⇒ String

This method is part of a private API. You should avoid using this method if possible, as it may be removed or be changed in the future.

Note:

Uses GitHub Flavored Markdown (GFM) as the input format

Note:

Uses Kramdown for Markdown parsing

Note:

This method is called internally by #strip when mode is Mode::MARKUP

Strip Markdown formatting and return plain text

Parses Markdown (GitHub Flavored Markdown) and removes all formatting, returning only the plain text content. This is a convenience wrapper around #strip_with_kramdown.

Examples:

Bold and italic

TextStripper.strip_markup("**bold** and *italic*")
# => "bold and italic"

Links

TextStripper.strip_markup("[link text](http://example.com)")
# => "link text"

Headers

TextStripper.strip_markup("# Heading")
# => "Heading"

Parameters:

  • text (String) —

    The Markdown text to process

Returns:

  • (String) —

    Plain text with Markdown formatting removed and whitespace trimmed

See Also:

  • #strip
  • #strip_markdown


560
561
562
563
564
565
566
567
 
# File 'documented/util/textstripper.rb', line 560

def self.strip_markup(text)
  unless KRAMDOWN_LOADED
    respond("Need to restart Lich5 in order to use this method.")
    return text
  end

  strip_with_kramdown(text, Mode::MARKUP)
end

.strip_with_kramdown(text, mode) ⇒ String

This method is part of a private API. You should avoid using this method if possible, as it may be removed or be changed in the future.

Note:

This method uses Kramdown's conversion chain to ensure proper text extraction. The process is:

  1. Parse input according to format (HTML/GFM)
  2. Convert internal representation to plain text
  3. Strip leading/trailing whitespace
Note:

For HTML mode, Kramdown's HTML parser extracts text content while preserving text nodes and ignoring markup. For markup/markdown modes, the GFM parser processes Markdown syntax and then extracts plain text.

Strip tags using Kramdown based on the input format

This is a shared helper method that handles the actual parsing and tag removal for HTML, MARKUP, and MARKDOWN modes. It converts the parsed document to plain text using Kramdown's standard conversion methods.

Parameters:

  • text (String) —

    The text to process

  • mode (Symbol) —

    The stripping mode, which determines the input format

Returns:

  • (String) —

    Plain text with tags/formatting removed and whitespace trimmed



289
290
291
292
293
294
295
296
297
298
299
300
 
# File 'documented/util/textstripper.rb', line 289

def self.strip_with_kramdown(text, mode)
  unless KRAMDOWN_LOADED
    respond("Need to restart Lich5 in order to use this method.")
    return text
  end

  input_format = MODE_TO_INPUT_FORMAT[mode]
  doc = Kramdown::Document.new(text, input: input_format)

  # Extract plain text from the parsed document by traversing the element tree
  extract_text(doc.root).strip
end

.strip_xml(text) ⇒ String

This method is part of a private API. You should avoid using this method if possible, as it may be removed or be changed in the future.

Note:

This method uses Ox for proper XML parsing, which correctly handles XML-specific features (CDATA, namespaces, etc.)

Note:

This method is called internally by #strip when mode is Mode::XML

Strip XML tags and return plain text

Removes XML tags from the input text using Ox for proper XML parsing. This method correctly handles XML-specific features like namespaces, CDATA sections, and processing instructions. This is a convenience wrapper around #strip_xml_with_ox.

Examples:

Basic XML stripping

TextStripper.strip_xml("<root>content</root>")
# => "content"

Nested XML elements

TextStripper.strip_xml("<root><item>data</item></root>")
# => "data"

XML with CDATA

TextStripper.strip_xml("<root><![CDATA[Special <characters>]]></root>")
# => "Special <characters>"

XML with namespaces

TextStripper.strip_xml("<root xmlns='http://example.com'><item>data</item></root>")
# => "data"

Parameters:

  • text (String) —

    The XML text to process

Returns:

  • (String) —

    Plain text with XML tags removed and whitespace trimmed

See Also:

  • #strip


528
529
530
 
# File 'documented/util/textstripper.rb', line 528

def self.strip_xml(text)
  strip_xml_with_ox(text)
end

.strip_xml_with_ox(text) ⇒ String

This method is part of a private API. You should avoid using this method if possible, as it may be removed or be changed in the future.

Note:

skip: :skip_off keeps whitespace verbatim, including whitespace-only text nodes between sibling elements. (:skip_none still drops those inter-element nodes, so "x y" would strip to "xy" not "x y".)

Note:

This method handles:

  • XML namespaces
  • CDATA sections (content is preserved as text)
  • Nested elements
  • Mixed content (text and elements)
  • HTML named entities (e.g. &nbsp;) and numeric character references
  • Unescaped special characters in plain text (wraps in CDATA if needed)

Strip XML tags using Ox and return plain text

This method uses Ox (generic mode) to properly parse XML content and extract all text nodes. Unlike the HTML parser, Ox correctly handles XML-specific features like namespaces, CDATA sections, and processing instructions, and it decodes HTML named entities such as &nbsp; that are not part of XML's predefined set.

Parameters:

  • text (String) —

    The XML text to process

Returns:

  • (String) —

    Plain text with XML tags removed and whitespace trimmed



328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
 
# File 'documented/util/textstripper.rb', line 328

def self.strip_xml_with_ox(text)
  # Try to parse as-is first (in case it's already well-formed XML)
  begin
    parsed = Ox.load("<root>#{text}</root>", mode: :generic, skip: :skip_off)
  rescue Ox::ParseError
    # If parsing fails due to unescaped characters, wrap in CDATA
    parsed = Ox.load("<root><![CDATA[#{text}]]></root>", mode: :generic, skip: :skip_off)
  end

  # Ox.load returns an Ox::Document when the input has an XML prolog and
  # the bare root Ox::Element otherwise; the root is <root> either way.
  root = parsed.is_a?(Ox::Document) ? parsed.root : parsed

  # Extract all text content from the document
  extract_xml_text(root).strip
end

.validate_mode(mode) ⇒ Symbol

This method is part of a private API. You should avoid using this method if possible, as it may be removed or be changed in the future.

Validate and normalize a mode value

Parameters:

  • mode (Symbol, String, Object) —

    The mode to validate

Returns:

  • (Symbol) —

    The validated and normalized mode as a symbol

Raises:

  • (ArgumentError) —

    if mode is not a Symbol or String, or is not a valid mode



234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
 
# File 'documented/util/textstripper.rb', line 234

def self.validate_mode(mode)
  # Ensure mode is a Symbol or String
  unless mode.is_a?(Symbol) || mode.is_a?(String)
    raise ArgumentError,
          "Mode must be a Symbol or String, got #{mode.class}"
  end

  # Normalize to symbol
  normalized_mode = mode.to_sym

  # Validate against allowed modes
  unless Mode.valid?(normalized_mode)
    raise ArgumentError,
          "Invalid mode: #{mode}. Use one of: #{Mode.list}"
  end

  normalized_mode
end