lazydns

Plugin-Based File Downloader

Overview

implements a lightweight, plugin-based approach for automatically downloading and updating DNS rule files. This approach combines the Downloader Plugin with the Cron Plugin to provide scheduled file updates without requiring an HTTP server or Admin API.

Architecture

┌─────────────────────────────────────────────────────────────────┐
│ Schedule: Daily at 02:05 UTC                                    │
│ (Cron Expression: "0 5 2 * * *")                                │
└─────────────────────────────┬───────────────────────────────────┘
                              │
                              ▼
                    ┌──────────────────┐
                    │ CronPlugin       │
                    │ (Scheduler)      │
                    └────────┬─────────┘
                             │
                    invoke_plugin action
                             │
                    ┌────────▼─────────┐
                    │ DownloaderPlugin │
                    │ (Download Files) │
                    └────────┬─────────┘
                             │
        ┌────────────────────┼────────────────────┐
        │                    │                    │
        ▼                    ▼                    ▼
    gfw.txt          reject-list.txt         hosts.txt
        │                    │                    │
        └────────────────────┼────────────────────┘
                             │
                    ┌────────▼────────┐
                    │ File Watcher    │
                    │ (auto_reload)   │
                    └────────┬────────┘
                             │
        ┌────────────────────┼────────────────────┐
        │                    │                    │
        ▼                    ▼                    ▼
   DomainSet           IPSet                  Hosts
   (auto reload)   (auto reload)           (auto reload)

Key Components

1. DownloaderPlugin

Handles the actual file downloads with robust error handling and atomic updates.

Configuration Options:

- tag: file_downloader
  type: downloader
  args:
    files:
      - url: "https://example.com/gfw.txt"
        path: "gfw.txt"
      - url: "https://example.com/reject.txt"
        path: "reject.txt"

    # Download timeout per file (seconds)
    timeout_secs: 30

    # Concurrent download strategy
    concurrent: false

Features:

Downloads multiple files with configurable timeout
Supports both sequential and concurrent download modes
Atomic file operations (temp → rename) prevents partial updates
Built-in retry mechanism for network errors
Detailed logging of download progress and results

2. CronPlugin

Triggers the downloader on a specified schedule.

Configuration Options:

- tag: auto_update_scheduler
  type: cron
  args:
    # Cron expression: second minute hour day month day-of-week
    cron: "0 5 2 * * *" # Daily at 02:05 UTC

    # Action type
    action: invoke_plugin

    # Target plugin
    args:
      plugin: "file_downloader"

Cron Expression Examples:

0 5 2 * * * - Every day at 02:05:00 UTC
0 0 */6 * * * - Every 6 hours at 00:00 minutes
0 0 0 1 * * - First day of every month at 00:00 UTC
0 0 0 * * 0 - Every Sunday at 00:00 UTC (weekly)

3. Auto-Reload Mechanism

DomainSet, IPSet, and Hosts plugins automatically reload files when they change:

- tag: domain_gfw
  type: domain_set
  args:
    files:
      - "gfw.txt"
    auto_reload: true # Key: enables automatic reload on file change

Complete Workflow

1. Initialization (Server Start)

Plugin Registration: DownloaderPlugin and CronPlugin register themselves
Data Loading: DomainSet, IPSet, Hosts load initial rule files
Scheduler Setup: CronPlugin starts background scheduler thread
File Watching: auto_reload mechanism starts watching file changes

2. Scheduled Event (02:05 UTC)

Cron Trigger: CronPlugin detects scheduled time
Plugin Invocation: CronPlugin calls DownloaderPlugin via invoke_plugin action
Download Phase: DownloaderPlugin downloads all configured files
Atomic Update: Files are written to temp locations, then atomically renamed
File Change Detection: File watcher detects modification
Automatic Reload: DomainSet/IPSet/Hosts reload updated files
DNS Updates: Subsequent DNS queries use new rules

3. DNS Query Processing

Rules from reloaded files are immediately available for DNS queries:

DNS Query (e.g., facebook.com)
    │
    ▼
Check domain_gfw matcher (uses updated gfw.txt)
    │
    ├─ Found → Forward to upstream_proxy
    │
    └─ Not found → Forward to upstream_direct

Configuration Examples

Simple Daily Update (02:05 UTC)

plugins:
  # Download files
  - tag: file_downloader
    type: downloader
    args:
      files:
        - url: "https://example.com/gfw.txt"
          path: "gfw.txt"
      timeout_secs: 30
      concurrent: false

  # Schedule the download
  - tag: scheduler
    type: cron
    args:
      cron: "0 5 2 * * *"
      action: invoke_plugin
      args:
        plugin: "file_downloader"

  # Load with auto-reload
  - tag: domain_list
    type: domain_set
    args:
      files: ["gfw.txt"]
      auto_reload: true

Concurrent Downloads (Off-Peak Hours)

- tag: file_downloader
  type: downloader
  args:
    files:
      - url: "https://example.com/gfw.txt"
        path: "gfw.txt"
      - url: "https://example.com/reject.txt"
        path: "reject.txt"
      - url: "https://example.com/hosts.txt"
        path: "hosts.txt"
    timeout_secs: 60
    concurrent: true # Download 3 files simultaneously

Weekly Update (Sunday 03:00 UTC)

- tag: scheduler
  type: cron
  args:
    cron: "0 0 3 * * 0" # 0 = Sunday
    action: invoke_plugin
    args:
      plugin: "file_downloader"

Multiple Update Schedules

# Fast daily update (lightweight lists)
- tag: daily_downloader
  type: downloader
  args:
    files:
      - url: "https://example.com/gfw.txt"
        path: "gfw.txt"
    timeout_secs: 20
    concurrent: false

- tag: daily_scheduler
  type: cron
  args:
    cron: "0 2 2 * * *"
    action: invoke_plugin
    args:
      plugin: "daily_downloader"

# Weekly comprehensive update (all lists)
- tag: weekly_downloader
  type: downloader
  args:
    files:
      - url: "https://example.com/gfw.txt"
        path: "gfw.txt"
      - url: "https://example.com/reject.txt"
        path: "reject.txt"
      - url: "https://example.com/hosts.txt"
        path: "hosts.txt"
    timeout_secs: 60
    concurrent: true

- tag: weekly_scheduler
  type: cron
  args:
    cron: "0 0 3 * * 0" # Weekly
    action: invoke_plugin
    args:
      plugin: "weekly_downloader"

Performance Characteristics

Download Performance

Sequential Mode (concurrent: false):

6 files, average 500KB each: ~10-15 seconds
Bandwidth usage: 1 file at a time (~500KB/s network)
CPU usage: Minimal
Memory usage: Minimal (streaming mode)

Concurrent Mode (concurrent: true):

6 files, average 500KB each: ~2-3 seconds
Bandwidth usage: All files simultaneously (~3MB/s total)
CPU usage: Slightly higher (tokio tasks)
Memory usage: Buffered content for all files

Reload Performance

File Replace + Reload:

Temp file write: ~100-500ms (depends on file size)
Atomic rename: <1ms (filesystem operation)
File watch detection: <100ms (inotify/FSEvents)
Reload execution: ~50-200ms (parsing + memory update)
Total: <1 second typical

No Impact on DNS Queries

Downloads happen in background
File reload is atomic (no partial updates visible)
DNS queries continue seamlessly
No server restart required

Logging and Monitoring

Key Log Lines

# Plugin initialization
INFO downloader: Downloader plugin initialized

# Cron trigger
INFO cron: Cron scheduler started

# Download start
INFO downloader: Starting file downloads count=3 concurrent=false

# File downloaded
INFO downloader: File downloaded url="https://..." path="gfw.txt" size_bytes=524288 duration_ms=2500

# All downloads complete
INFO downloader: All files downloaded successfully count=3 duration_ms=5200

# File reload
INFO domain_set: Reloading file "gfw.txt" from file system

# Reload complete
INFO domain_set: Domain set loaded: 10234 entries

Monitoring Downloads

# Watch for failed downloads
tail -f logs/lazydns.log | grep "download failed"

# Monitor file sizes after update
ls -lh *.txt

# Check last update time
stat gfw.txt

# Verify log rotation
ls -la logs/lazydns.log*

Troubleshooting

No Downloads Happening

Check 1: Cron scheduler running

grep "Cron scheduler started" logs/lazydns.log

Check 2: Cron expression correct

# Test cron expression with cronexpr tool or online calculator
# "0 5 2 * * *" should trigger at 02:05:00

Check 3: Plugin registered

grep "Downloader plugin initialized" logs/lazydns.log

Downloads Timing Out

Issue: Files too large or network too slow

Solution: Increase timeout_secs

- tag: file_downloader
  type: downloader
  args:
    files: [...]
    timeout_secs: 60 # Increased from 30

Files Not Reloading

Issue: auto_reload not enabled

Solution: Enable auto_reload on data plugins

- tag: domain_list
  type: domain_set
  args:
    files: ["gfw.txt"]
    auto_reload: true # Must be explicitly true

Concurrent Downloads Slow

Issue: Network bottleneck or overloaded disk

Solution: Use sequential mode

- tag: file_downloader
  type: downloader
  args:
    files: [...]
    concurrent: false # Download one at a time

Implementation Details

Atomic File Operations

// 1. Download to temporary file
let temp_path = format!("{}.tmp", spec.path);
write_to_file(&temp_path, &content)?;

// 2. Atomic rename (OS-level guarantee)
fs::rename(&temp_path, &spec.path)?;

// 3. File watcher detects change
// 4. auto_reload triggers reload

Benefits:

No partial files visible to the system
Process crash during download doesn’t corrupt files
File watchers get clean “file modified” event

Cron to Plugin Invocation

invoke_plugin Action Flow:

CronPlugin detects schedule match
Looks up target plugin by tag: plugin: "file_downloader"
Calls plugin’s execute() method
Plugin processes download task asynchronously
Cron continues without waiting for completion

// Pseudo-code
if let Some(plugin_tag) = args.get("plugin") {
    if let Some(plugin) = plugin_registry.get(plugin_tag) {
        // Spawn async task
        tokio::spawn(async move {
            let _ = plugin.execute(&mut ctx).await;
        });
    }
}

Advanced Configuration

Custom File Processing

For formats requiring transformation (e.g., dnsmasq → POSIX domain list):

# Download raw dnsmasq format
- tag: downloader
  type: downloader
  args:
    files:
      - url: "https://example.com/dnsmasq.conf"
        path: "dnsmasq-raw.conf"
    timeout_secs: 30
# Post-process with external command (requires Plan A elements)
# TODO: Add exec_plugin to run conversion script

Fallback Domains

Ensure service availability if downloads fail:

- tag: domain_gfw
  type: domain_set
  args:
    files:
      - "gfw.txt" # Updated by downloader
      - "gfw-fallback.txt" # Static fallback
    auto_reload: true

This site is open source. Improve this page.