Mail::SpamAssassin::FuzzyHash::ZOrder(3pm)

NAME
FuzzyHash::ZOrder - Fuzzy text similarity via DNS, using exactly 4 queries

SYNOPSIS
  use FuzzyHash::ZOrder qw(zorder_digest zorder_compare zorder_dns_keys);

  my $query_digest = zorder_digest($incoming_text);
  for my $key (zorder_dns_keys($incoming_text, 'fbl.example.com')) {
    for my $stored_digest (dns_txt_lookup($key)) {
      my $sim = zorder_compare($query_digest, $stored_digest);
      if ($sim >= 85) { # rule fires }
    }
  }

DESCRIPTION
FuzzyHash::ZOrder detects texts with ~90% token overlap using exactly 4 DNS TXT lookups followed by a local similarity check via zorder_compare().

  Algorithm

Normalisation
    Lowercase, replace non-alphanumeric with spaces, collapse whitespace, deduplicate tokens.

32-bit MinHash signature
    32 independent hash functions are applied to the deduplicated token set. Each function finds the token with the minimum hash value; the LSB parity of that minimum becomes one output bit. The result is a 32-bit signature. Each bit agrees between two texts with probability exactly equal to their Jaccard similarity.

4 bands x 8 bits
    The 32-bit signature is split into 4 consecutive 8-bit bands. Two texts share a band key when all 8 bits in that band agree: probability S^8 at Jaccard similarity S.

Local similarity verification
    Each TXT response carries the full 32-bit digest of the indexed string. zorder_compare() verifies similarity locally using all 32 bits. Unrelated texts will have a score of ~50%.

FUNCTIONS

  zorder_digest($text)
Computes the 32-bit MinHash signature of $text. Returns an 8-character lowercase hex string.

This is the value stored in each DNS TXT record at index time and used for local similarity verification at query time.

  zorder_dns_keys($text, $domain)
Returns a list of exactly 4 DNS FQDNs for $text under $domain.

Use for both indexing (store digest as TXT at each key) and querying (look up each key, verify TXT responses locally with zorder_compare).

  zorder_body_band_labels($text)
Returns a list of exactly 4 bare band labels without a trailing domain, each of the form "fz2BVV".

  zorder_compare($digest_a, $digest_b)
Compares two 8-char hex digests. Returns integer similarity 0-100, computed as "100 * (1 - hamming_distance / 32)".

  100  = identical token sets (0 bits differ)
   91  = ~3 bits differ  -> very high overlap
   85  = ~5 bits differ  -> high overlap
   75  = ~8 bits differ  -> moderate overlap
   50  = ~16 bits differ -> noise floor for unrelated texts

The noise floor at ~50% provides a clean gap below the suggested 90% threshold, so coincidental DNS band hits from unrelated indexed strings are reliably rejected without any additional DNS queries.

  zorder_explain($digest_a, $digest_b)
Returns a human-readable breakdown: Hamming distance, similarity score, shared band count, firing decision, and a per-band bit-agreement table. Used only for debugging RBL entries and tuning the similarity threshold.