UTIX

From raw IOCs to actionable confidence — automated threat intel scoring.

Comment

Inspiration

Every security team I've seen operates the same way: ingest threat intelligence feeds, dump them into a database, and then block everything—forever. That IP that hosted a botnet command-and-control server two years ago? Still blocked. That domain that briefly resolved to a malicious server? Still flagged. No one ever cleans up the list, because cleaning it up means potentially letting something bad through. Stale threat intelligence is a liability masquerading as safety.

The other thing that frustrated me was the black-box score problem. Commercial Threat Intelligence Platforms (TIPs) will hand you a confidence number between 0 and 100 and tell you nothing about where it came from. Is that 84 from VirusTotal's consensus? From a single community report three years ago? You can't tell, and that matters enormously when you're deciding whether to block a customer's IP at the firewall.

I wanted to build something that answers two questions a SOC analyst actually cares about:

"Why does this IOC have this score?" — full provenance, no black boxes. "Is this IOC still relevant today?" — because threats have an expiry date.

What it does- UTIX (Unified Threat Intelligence Connector) aggregates Indicators of Compromise (IOCs) from multiple open threat intelligence feeds — ThreatFox, abuse.ch (URLhaus, MalwareBazaar, Feodo Tracker, SSLBL), and VirusTotal — into a single PostgreSQL database with a live web dashboard.

But v2 goes far beyond aggregation. It adds three things no free TIP does out of the box:

Explainable scoring — every IOC carries a full provenance card showing exactly which sources flagged it, their reliability weights, recency factors, and the step-by-step math behind its confidence score. No black-box numbers. Self-cleaning exports — blocklists and detection rules (Palo Alto EDL, Suricata, Snort, Sigma, STIX 2.1) that automatically drop stale IOCs as their confidence decays over time. Threat intelligence has an expiry date; UTIX enforces it automatically. Log matching — paste or upload raw firewall/DNS/proxy logs and UTIX tells you which of your hosts communicated with known malicious infrastructure. The dashboard headline changes from "500,000 IOCs ingested" to "3 of your hosts talked to a Feodo C2 this week."

How we built it- The backend is Python (Flask) with a PostgreSQL database. The project evolved in two stages:

v1 — Data warehouse: Modular connector architecture with one Python file per feed source under connectors/. Each connector fetches, normalizes, and upserts IOCs with deduplication. A scheduler runs ingestion on a configurable cron-like loop. A single-page HTML dashboard visualizes the data.

v2 — Defense tool: We added a formal scoring engine (scoring.py) fully aligned to a structured Intelligence Processing Architecture document, implementing Sections 2.2–2.6 (multi-source confidence aggregation) and Section 3.2–3.6 (MISP polynomial decay). The corroboration engine (corroborate.py) groups IOCs by identity across sources, computes weighted-average confidence with pairwise vendor agreement bonuses, applies time decay, and auto-revokes IOCs that fall below the threshold — all triggered automatically after every ingestion run.

The provenance API (provenance.py) reconstructs the full scoring audit trail on demand. The export engine (exports.py) renders live-filtered feeds in six formats. The log-matching engine (logmatch.py) extracts observables from raw text (including defanged notation like hxxp:// and 1.2.3[.]4), skips private/reserved ranges, and cross-references against the live IOC database, persisting hits for SOC audit. All secrets were moved out of source code into environment variables with .env support.

Challenges we ran into- Getting the multi-vendor math right. The edge case that nearly broke the scoring model: a low-reliability source (e.g. a Twitter OSINT bot) giving an extreme score of 95/100. A naive average would inflate confidence dangerously. The solution — reliability-weighted averaging combined with a pairwise disagreement penalty — required working through concrete examples by hand before trusting the code. The corroboration matrix adds points for close inter-vendor agreement and subtracts them for conflict, so a lone outlier from an untrusted source barely moves the needle.

Decay curve calibration. Choosing the decay shape parameter λ λ per observable type required understanding the real-world threat lifecycle. IPs and URLs cycle in weeks; file hashes remain valid for months. Getting the curves to match practitioner intuition meant plotting decay trajectories and iterating.

Defanged IOC parsing. Threat analysts deliberately mangle IOCs to prevent accidental resolution — hxxp://, [dot], [.], and more. Building an extractor that correctly refangs all variants without generating false positives from email addresses and other look-alike patterns was fiddly, edge-case-heavy work.

Idempotent database migrations. Adding new columns, views, and tables to an existing v1 schema without breaking live installs required careful SQL that could be safely re-run multiple times — not as simple as it sounds when dealing with constraints and materialized views.

Accomplishments that we're proud of- The blocklist that cleans itself. The decay + auto-revocation pipeline is something we hadn't seen in any free tool. Point a Palo Alto firewall or pfSense instance at the EDL endpoint, and the list stays fresh indefinitely — stale IOCs quietly expire without anyone having to manually curate it.

Full score transparency. The provenance card (/api/provenance) shows every source, its reliability, its recency weight, the weighted average, the corroboration adjustment, the decay rule applied, points already lost to decay, and the projected revocation date. Analysts can audit exactly why an IOC has the score it has — something most commercial platforms charge significantly for and still don't fully expose.

The shift from vanity metrics to actionable alerts. The log-matching feature transforms UTIX from a data warehouse into an operational tool. "3 of your hosts talked to a known C2" is a fundamentally different product than "500,000 IOCs ingested."

Mathematically grounded scoring from first principles. Implementing the full Section 2.4 multi-vendor aggregation model — recency-weighted averages, pairwise corroboration matrix, MISP polynomial decay — in clean, unit-testable Python that maps directly to the underlying architecture document.

What we learned- The biggest lesson was that trust is not binary. Every design decision in the scoring model is really a question about how much to trust a source, how much to trust a sighting from six months ago, and how much to trust that two sources agreeing independently is stronger signal than either alone. The math formalizes intuitions threat analysts already have but rarely make explicit.

We also learned that threat intelligence has a half-life. An IP used for botnet C2 infrastructure in January is likely irrelevant by March — the adversary has moved on. File hashes last longer. Domain names are somewhere in between. Encoding these decay rates directly into the data model, rather than leaving cleanup to human operators, is a design choice that makes the system more trustworthy over time, not less.

Finally: secrets in source code always get committed. The v1 codebase had API keys hardcoded in main.py. Moving everything to .env / environment variables with a proper .env.example template and .gitignore coverage is not optional hygiene — it's the minimum bar before any tool touches production data.

What's next for UTIX- Reaction scoring. When an IOC is confirmed by a SIEM hit or endpoint detection, that's the strongest possible signal — it should spike confidence back up and reset the decay clock. The data model is ready; the scoring logic is the next piece.

Independence-corrected aggregation. The current model treats all sources as independent, but many threat feeds cross-pollinate — ThreatFox data appears in OTX, and both draw from the same honeypot networks. Naively aggregating dependent sources inflates confidence. Correcting for feed overlap with a provenance graph would produce significantly more calibrated scores.

MISP / OpenCTI push integration. Right now UTIX pulls from feeds and exports static STIX bundles. The next step is bidirectional sync — pushing high-confidence IOCs and match hits back into a MISP instance or OpenCTI graph in real time.

Analyst feedback loop. A simple thumbs-up/thumbs-down on IOC cards would let analysts push reliability signal back into the scoring model, personalizing source weights to their environment over time.

Built With

Share this project:

Updates