BLACKED

// URL Blacklist Aggregator · v0.3.0 · 12 providers · 1,070,538 entries

Models Used deepseek-v4-pro · minimax-m2.7 · glm-5 · deepseek-v4-flash
Total Tokens ~840k in / ~215k out
PRs #5 · #16 · #50 · #53 · #54 · #55 · #57–66
Read the editorial story
HUMAN → AI Handover // 16 May 2026 → 28 May 2026 // 9 sessions // 4 models // 1,070,538 entries // v0.1.0-alpha → v0.3.0 DECLASSIFIED
SYS.INDEX // SEC_00 // BLACKED // SIGNAL FROM NOISE

BLACKED
Signal From Noise

DIR_ORIGINAL Design a high-performance URL blacklist aggregator in Go that uses parallel bloom filters for sub-millisecond lookups.
Status: AI Owned GO 1.22 · 12 PROVIDERS · v0.3.0
Session Split // 9 total
// Human-anchored (16–25 May)
#01 16 May MVP validation
#02 18 May Research pipeline
#03 22 May PR #16 review
#04 25 May Mad-man analysis
#05 26 May E2E bootstrap
↓ Handover ↓
// AI-led (26–28 May)
#06 26 May ProcessID cascade
#07 26 May Provider hardening
#08 27 May Pipeline audit
#09 28 May CodeGraph MCP
Token In
~840k
Token Out
~215k
Models
4
PRs
12+
COORD: [X:5.2, Y:18.7]
// 01 // Bloom Filter Anatomy

How 6 parallel bloom filters say "no" in 0.4 ms

OR-merge · first hit wins
INPUT // URL → 6 parallel hash channels
GET /evil.com/malware/exploit.php?ref=bad
→ domain channel → host channel → path channel → file channel → fullurl channel → ip channel
Layer 01
Domain
"evil.com"
h1h2h3
hits
[3, 7, 12]
w: 0.3
Layer 02
Host
"cdn.evil.com"
h1h2h3
hits
[2, 8, 14]
w: 0.5
Layer 03
Path
"/malware/"
h1h2h3
hits
[5, 9, 11]
w: 1.0
Layer 04
File
"exploit.php"
h1h2h3
hits
[1, 6, 13]
w: 0.7
Layer 05
FullURL
"?ref=bad"
h1h2h3
hits
[4, 10, 15]
w: 1.5
Layer 06
IP
"93.127.x.x"
h1h2h3
hits
[0, 5, 9]
w: 0.8
// First Hit Wins
Layer 01 (Domain) lights bit 3 first → ctx cancel → layers 02-06 are aborted. Latency stops growing.
// OR-Gate
≥1 HIT
→ CONFIDENCE LOOKUP
// FP Rate Math
7 hash functions × 1.07M entries / 7M bit capacity.
FP ≈ 7.8% — mitigated by mandatory DB hit verification on every bloom "yes".
COORD: [X:18.4, Y:42.0]
// 02 // Cron Scheduler

12 providers, 1 timeline, 1 minute per sync

domain url ip mixed
provider / TTL
00:0003:0006:0009:0012:0015:0018:0021:0024:00
last sync
P-01 DOMAIN
OISD (big)
0 6 * * *
NOW
06:00 (1× / day)
P-02 DOMAIN
OISD (nsfw)
0 6 * * *
06:00 (1× / day)
P-03 MIXED
ThreatFox
*/30 * * * *
23:30 (48× / day)
P-04 URL
URLHaus
0 * * * *
23:00 (24× / day)
P-05 IP
RTBH Turkey
0 */2 * * *
22:00 (12× / day)
P-06 IP
Blocklist.de
0 */4 * * *
20:00 (6× / day)
P-07 IP
CINS Army
0 */6 * * *
18:00 (4× / day)
P-08 IP
AbuseIPDB
0 0 * * *
— AUTH —
P-09 IP
GreenSnow
0 */6 * * *
18:00 (4× / day)
P-10 MIXED
AlienVault OTX
*/30 * * * *
23:30 (48× / day)
P-11 IP
Tor Exit Nodes
0 */12 * * *
12:00 (2× / day)
P-12 IP
Emerging Threats
0 0 * * *
00:00 (1× / day)
// Single Sync Cycle — State Machine
S1 · IDLE S2 · TRIGGER S3 · FETCH S4 · PARSE S5 · POND S6 · COMMIT S1 · IDLE
CRON fires at the scheduled tick → goroutine pulls the provider feed → parser streams to the pond collector → async batch commits to SQLite. The next cycle can start before this one finishes. No backpressure. No starvation. No missed ticks.
DECLASSIFIED_A03 // BUG CASCADE

Declassified Incident Report

// 8 critical bugs. 12 PR. 2 weeks. Every cascade listed below cost real time, real data, and one near-miss with the production DB.
CRITICAL 16 May
PR #5

scoring.toml not loading

// Root cause

NewScorer(nil) call — the configuration file is never injected. Every trust score falls back to 0.5. The scorer runs in isolation, but the scoring system cannot tell providers apart by reliability.

// Patch applied

cmd/serve.go: NewScorer(configPath) — scoring.toml hot-reload added. Provider weights now read from the configuration file.

CRITICAL 16 May
PR #5

DROP TABLE runs without CASCADE

// Root cause

During migrations, DROP TABLE blacklisted_urls was being called without CASCADE. A destructive operation could wipe FK references by accident. Caught in the test environment.

// Patch applied

migrations/0001_init.sql: DROP TABLE → DROP TABLE IF EXISTS … CASCADE. Refactored: all migrations now run in a single tx.

CRITICAL 16 May
PR #5

SELECT on a column that does not exist

// Root cause

The hit query was writing SELECT reason FROM … but the table had no "reason" column (only source, url, process_id). Returned 500. Caught before reaching production.

// Patch applied

repository/hits.go: SELECT source, url, process_id, inserted_at FROM blacklisted_urls. Index added on (process_id, inserted_at).

HIGH 16 May
PR #5

Hit path skips the DB (bloom only)

// Root cause

The /check endpoint was looking only at the bloom filter; it never verified the DB hit. On a false positive the caller received a wrong "blocked" verdict. There was no reverse path.

// Patch applied

check.go: when bloom.TestString(hash) is true, a DB lookup is now mandatory. Result: {InBloom: true, InDB: bool, Reason: string}.

CRITICAL 26 May
PR #50 + #53

ProcessID cache poisoning (cascade)

// Root cause

process.go:269 — strProcessID = meta.ProcessID was overwriting the fresh UUID. Every entry was bound to the wrong processID. RemoveOlderInsertions then wiped them all. The first fix (PR #50) failed to find the rogue uuid.New() generator inside parallel_parser.go → a second patch (PR #53) forced the parser to take the processID from the provider.

// Patch applied

process.go:269-274: ignore the cached processID, use a fresh UUID. parallel_logger.go:107-134: added a processID parameter to ParseLinesParallel. 6 callers updated.

★ Multi-PR Cascade
HIGH 26 May
PR #54

AlienVault 403 Forbidden

// Root cause

AlienVault OTX Fetch() defaulted to ctx.Background(). The User-Agent header was missing. 403 came back. The pagination strategy (page-based, max 56) was unusable because not a single fetch was getting through.

// Patch applied

alienvault/provider.go: added a FetchWithContext() override (delegating to Fetch()). Correct UA + rate-limit aware retry. The multi-page fetch chain was restored.

MEDIUM 26 May
PR #54

NULL scan error (openphish)

// Root cause

openphish-feed and other providers were returning NULL confidence. The sql.NullString migration was incomplete. SQLite was raising a scan error.

// Patch applied

sqlite_blacklist_repository.go: confidence, source → sql.NullString. NullInt64 added too. Insert and select paths updated.

CRITICAL 28 May
PR #57

RemoveStoredResponse on every restart (AlienVault 300+ page)

// Root cause

process.go:328-330 was calling RemoveStoredResponse in the development environment → after every run response.dat was being deleted → on restart AlienVault had to refetch 300+ pages. The 2.5s bloom bootstrap was paying the price.

// Patch applied

process.go:328-330: added an env == "dev" check. In production, response.dat is preserved. Folded into the async startup plan (PR #59).

Critical Bugs 8
PRs Merged 12+
Lines Patched ~+5,400 / −1,200
Near-Miss 1 (DROPTABLE)
DECLASSIFIED_A04 // PROVIDER MATRIX

Provider Matrix

// 12 live providers. 1,070,538 entries. The signal that Blacked is filtering from.
P-01 DOMAIN
OISD (big)
438,790
entries · 40.8% of total
0 6 * * * OK
P-02 DOMAIN
OISD (nsfw)
332,539
entries · 30.9% of total
0 6 * * * OK
P-03 MIXED
ThreatFox (online)
103,628
entries · 9.6% of total
*/30 * * * * OK
P-04 URL
URLHaus (online)
76,148
entries · 7.1% of total
0 * * * * OK
P-05 IP
RTBH Turkey
62,804
entries · 5.8% of total
0 */2 * * * OK
P-06 IP
Blocklist.de
23,583
entries · 2.2% of total
0 */4 * * * OK
P-07 IP
CINS Army
15,000
entries · 1.4% of total
0 */6 * * * OK
P-08 IP
AbuseIPDB
10,000
entries · 0.9% of total
0 0 * * * AUTH
P-09 IP
GreenSnow
5,995
entries · 0.6% of total
0 */6 * * * OK
P-10 MIXED
AlienVault OTX
6,000
entries · 0.6% of total
*/30 * * * * OK
P-11 IP
Tor Exit Nodes
1,280
entries · 0.1% of total
0 */12 * * * OK
P-12 IP
Emerging Threats
516
entries · 0.0% of total
0 0 * * * OK
Total Entries (all 12)
1,076,283
As of 26 May 2026, 19:37
Largest Provider
OISD (big)
438,790 · 0 6 * * *
Smallest Provider
Emerging Threats
516 · 0 0 * * *
COORD: [X:22.0, Y:5.0]
Total Sessions 9 Main · 25+ sub
Active Providers 12 (1.07M entries)
Token In ~850k
Token Out ~220k
Model / Latency GPT-4o · 0.4ms
E2E Tests 14/14 (0.59s)
COORD: [X:12.4, Y:67.8]
// 06 // Heuristic Algorithm

Confidence Score Matrix

confidence =
Σ(trust_score × depth_weight) Σ(trust_score)

Depth weights: Domain 0.3 · Host 0.5 · HostPath 1.0 · File 0.7 · FullURL 1.5 · IP 0.8. Provider trust scores resolved from scoring.toml.

MATH_SIG_VERIFIED

Blacked Chat Timeline

// 9 sessions. 4 models. 1 human-anchored handoff to AI. Every row is a real conversation from May 2026.
#Date / TimeChannelPhaseModelOperational TasksTokens in/outBadge
01
2026-05-16
22:00
Telegram01 // Human Anchordeepseek-v4-pro:cloud
MVP validation & v0.1.0-alpha prep
7-layer research methodology extracted. go build PASS + 7/7 unit + 14/14 E2E. 5 MVP phases verified. 4 critical gaps in scoring.toml / drop-table / select / hit-DB found via Copilot PR review. PR #5 (33 commit, 75 file) merged. // Human-led.
~140k in / 35k outFEAT
HIGH
02
2026-05-18
multi
Telegram02 // Research Driftdeepseek-v4-flash:cloud
Research: AlienVault + provider pipeline
AlienVault OTX indicator mapping (domain/url → bloom; hash → skip; pulse → category). 6-provider capability analysis. Architecture decisions for new provider onboarding.
~70k in / 18k outDOCS
MEDIUM
03
2026-05-22
multi
Telegram03 // Provider Takeoverminimax-m2.7:cloud
PR #16 review + provider enrichment
PR #16 review: 37 files (+5,220/-18). AbuseIPDB REST API, AlienVault pagination, 4 new providers. SourceType 12→14. Pagination page-based max 56. 10+ providers auto-enabled. // AI takes the initiative at provider level.
~110k in / 28k outFEAT
HIGH
04
2026-05-25
15:51
Telegram04 // Mad-Man Passminimax-m2.7:cloud
Mad-man analysis — 20 critical issues
37KB analysis report. 20 critical issues (bloom pool corruption, DB write saturation, SQLite WAL contention, bloom race condition, 7.8% FP rate, missing provider reliability scoring). 14 quality items. 4-week backlog. 14 metrics proposed.
~85k in / 22k outALERT
EXTREME
05
2026-05-26
02:04
Telegram05 // Bootstrap Verifyglm-5:cloud
E2E bootstrap + port collision
Port 8082 collision → kill old process → fresh start. Bootstrap observation: 888K → 1M+ bloom. Provider sync confirmed. ProcessID cache poisoning discovered at process.go:269.
~75k in / 20k outFIX
HIGH
06
2026-05-26
17:56
Telegram06 // Cascade Patchglm-5:cloud
ProcessID fix + bug cascade
process.go:269-274 patch (ignore cached processID, use fresh UUID). parallel_logger.go signature update. 6 callers updated. PR #50 merged. Second bug in parallel_parser.go → PR #53 fix.
~95k in / 24k outFIX
EXTREME
07
2026-05-26
19:37
Telegram07 // Provider Hardeningglm-5:cloud
ThreatFox / AlienVault / AbuseIPDB fix
AlienVault FetchWithContext() override (403 fix). NULL scan migration. abuseipdb parseFunc fix. ThreatFox API key logging. 12 providers totalling 1,070,538 entries. PR #54 + PR #55 cherry-pick.
~85k in / 22k outFIX
HIGH
08
2026-05-27
11:57
Telegram08 // Pipeline Auditglm-5:cloud
Provider pipeline analysis
Pond collector + bloom bootstrap analysis. 2.5s duration / 734K entry. AlienVault scheduled TTL bug (every 10min full refetch). /api/v1/bloom/stats endpoint. 4 PRs for provider enable/disable subset.
~70k in / 18k outCHORE
MEDIUM
09
2026-05-28
multi
Telegram09 // AI Takeoverdeepseek-v4-flash:cloud
CodeGraph MCP + async startup
CodeGraph MCP setup (140 files, 2,312 nodes). 10 kanban sub-tasks. PR #57–60. Server startup blocking bug fix (async). Mad-man dispatch → PR #64 (AlienVault nil), PR #66 (IPv6). // AI now makes architecture decisions.
~110k in / 28k outFEAT
EXTREME
SOURCE // THE RECEIPT
root@blacked:/core/engine.go · COLLECTOR_POND_MONITOR
// Engine Source — finally, at the dead end of the page
// INITIALIZING CORE ROUTINES...
package blacklist

import (
	"sync"
	"context"
	"log"
)

// 6-layer parallel check chain: Domain -> Host -> HostPath -> File -> FullURL -> IP. First hit wins.
func (b *Blacklist) Check(url string) bool {
	var wg sync.WaitGroup
	result := make(chan bool, 6)
	ctx, cancel := context.WithCancel(context.Background())
	defer cancel()

	layers := []func() bool{
		func() bool { return b.checkDomain(url) },
		func() bool { return b.checkHost(url) },
		func() bool { return b.checkHostPath(url) },
		func() bool { return b.checkFile(url) },
		func() bool { return b.checkFullURL(url) },
		func() bool { return b.checkIP(url) },
	}

	for _, layer := range layers {
		wg.Add(1)
		go func(check func() bool) {
			defer wg.Done()
			if check() {
				select {
				case result <- true:
					cancel()
				case <-ctx.Done():
				}
			}
		}(layer)
	}

	go func() {
		wg.Wait()
		close(result)
	}()

	for hit := range result {
		if hit {
			return true
		}
	}
	return false
}
// NODE SYNC STATUS: SECURE