Module: Lich::Common::ShutdownWatchdog

Defined in:
documented/common/shutdown_watchdog.rb

Overview

Guarantees that a shutting-down Lich process actually terminates.

The teardown sequence runs several steps that can block indefinitely -- notably inline +before_dying+/+at_exit+ script hooks, Vars.save, Game.close (socket linger), database close, and lifecycle unregister IO. None of them are individually time-bounded. If one hangs, the process never reaches exit and continues to hold its OS resources (open sockets, advisory locks) until it is killed by hand.

This watchdog is armed at the start of teardown and disarmed once teardown completes. If the deadline elapses first, it dumps every thread's backtrace (so the debug log records what was stuck) and then forces the process to exit, letting the OS reclaim all resources.

It is intentionally dependency-light and cross-platform: plain Ruby threads, a condition variable for prompt disarm, and Process.exit!.

Constant Summary collapse

DEFAULT_TIMEOUT_SECONDS =

Default deadline, in seconds, before a stuck shutdown is forced.

Chosen to comfortably exceed a healthy teardown (bounded script drain plus state/socket/database closeout) while still bounding a hang.

Returns:

  • (Integer) 
60
SETTING_NAME =

Name of the lich_settings row that overrides DEFAULT_TIMEOUT_SECONDS.

Operator reference:

  • The value is a whole number of seconds.
  • A positive value sets the force-exit deadline.
  • An explicit 0 or negative value disables the watchdog.
  • A missing or non-numeric value falls back to DEFAULT_TIMEOUT_SECONDS; a malformed value is logged and never silently disables the watchdog.

Examples:

Set the deadline to 90 seconds from an in-game console

;e Lich.db.execute("INSERT OR REPLACE INTO lich_settings(name,value) VALUES('shutdown_watchdog_timeout','90')")

Disable the watchdog

;e Lich.db.execute("INSERT OR REPLACE INTO lich_settings(name,value) VALUES('shutdown_watchdog_timeout','0')")

Returns:

'shutdown_watchdog_timeout'

Class Method Summary collapse

Class Method Details

.arm(timeout: configured_timeout, on_expire: -> { Process.exit!(1) }) ⇒ Boolean

Arms the watchdog.

Spawns a single background thread that waits up to timeout seconds. If the watchdog is still armed when the deadline elapses, it dumps diagnostics and invokes on_expire. A subsequent disarm wakes the thread immediately and prevents on_expire from running.

Parameters:

  • timeout (Numeric) (defaults to: configured_timeout) —

    deadline in seconds; <= 0 disables (no-op)

  • on_expire (#call) (defaults to: -> { Process.exit!(1) }) —

    action taken when the deadline elapses; defaults to an immediate, un-trappable process exit. Must be a real, resolvable call: Process.exit! is used (not a bare exit!, which resolves against this module and raises NoMethodError, silently defeating the force-exit guarantee).

Returns:

  • (Boolean) —

    true when a watchdog thread was started



71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
 
# File 'documented/common/shutdown_watchdog.rb', line 71

def arm(timeout: configured_timeout, on_expire: -> { Process.exit!(1) })
  return false if timeout.to_f <= 0

  @mutex.synchronize do
    return false if @armed

    @armed = true

    # Create the thread and record @thread while still holding the lock
    # so the thread can identify itself. An arm/disarm/arm sequence can
    # leave an earlier thread parked in the wait; gating on
    # @thread == Thread.current ensures only the currently armed
    # watchdog can expire, so a superseded thread never forces an exit.
    @thread = Thread.new do
      # Anchor the deadline to the monotonic clock and re-wait on every
      # wakeup. ConditionVariable#wait can return early on a spurious
      # wakeup, and disarm/broadcast also wakes it; only a wakeup at or
      # after the genuine deadline counts as expiry. Without this loop a
      # spurious wakeup while still armed would force an exit before the
      # configured timeout.
      deadline = Process.clock_gettime(Process::CLOCK_MONOTONIC) + timeout.to_f
      expired = @mutex.synchronize do
        loop do
          break false unless @armed && @thread == Thread.current

          remaining = deadline - Process.clock_gettime(Process::CLOCK_MONOTONIC)
          break true if remaining <= 0

          @condition.wait(@mutex, remaining)
        end
      end
      if expired
        dump_diagnostics(timeout)
        on_expire.call
      end
    end
  end
  true
end

.armed? ⇒ Boolean

Returns whether the watchdog is currently armed.

Returns:

  • (Boolean) 


126
127
128
 
# File 'documented/common/shutdown_watchdog.rb', line 126

def armed?
  @mutex.synchronize { @armed }
end

.configured_timeout ⇒ Integer

Resolves the configured deadline from lich_settings.

Falls back to DEFAULT_TIMEOUT_SECONDS when the setting is unset, unreadable, or non-numeric. A malformed value is logged and treated as the default rather than being coerced to 0, so a typo cannot silently disable the watchdog; only an explicit numeric value of 0 or less disables it.

Returns:

  • (Integer) 


139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
 
# File 'documented/common/shutdown_watchdog.rb', line 139

def configured_timeout
  return DEFAULT_TIMEOUT_SECONDS unless defined?(Lich) && Lich.respond_to?(:db) && Lich.db

  raw = Lich.db.get_first_value("SELECT value FROM lich_settings WHERE name='#{SETTING_NAME}';")
  return DEFAULT_TIMEOUT_SECONDS if raw.nil?

  parsed = Integer(raw.to_s.strip, exception: false)
  if parsed.nil?
    log_line("invalid #{SETTING_NAME}=#{raw.inspect}; falling back to #{DEFAULT_TIMEOUT_SECONDS}s")
    return DEFAULT_TIMEOUT_SECONDS
  end
  parsed
rescue StandardError
  DEFAULT_TIMEOUT_SECONDS
end

.disarm ⇒ void

This method returns an undefined value.

Disarms the watchdog, waking the waiting thread so it exits without forcing termination. Idempotent and safe to call when not armed.



115
116
117
118
119
120
121
 
# File 'documented/common/shutdown_watchdog.rb', line 115

def disarm
  @mutex.synchronize do
    @armed = false
    @condition.broadcast
  end
  nil
end