Rare Not Random Using Token Efficiency for Secrets Scanning
Researcher proposes token efficiency (string length divided by BPE token count) as a better post-regex filter than entropy for secrets scanning, validated on CredData.
The post explores whether Byte-Pair Encoding tokenization can replace Shannon entropy as the primary filter for candidate secrets captured by regex in tools like Gitleaks. It defines 'token efficiency' as string length divided by token count under the cl100k_base tokenizer; secret-like strings such as GitHub tokens tokenize into many small tokens and score low, while natural text scores high. Evaluating labeled secrets from the CredData dataset shows a usable separation, with roughly 2.5 suggested as a minimum cutoff versus Gitleaks' 3.5 entropy threshold. The technique is positioned as a post-regex filtering step rather than a standalone detector.
- BPE tokenization reflects training-data frequency: rare strings split into many short tokens
- Token efficiency = string length / token count; secrets score low, natural text high
- CredData evaluation suggests a minimum token efficiency cutoff around 2.5
- Proposed as a post-regex filter to reduce false positives, not a standalone detector
- Idea originated from a GitHub issue on the Gitleaks repository
- Vendors
- Gitleaks
- Products
- cl100k_base tokenizerCredData
From the original article
10 min read at lookingatcomputer.substack.com
In Regex is (almost) All You Need we learned that using a combination of regular expression patterns, entropy, and rule-based filters are an effective way to detect candidate secrets. Regex is used for casting a wide net to identify candidates. Entropy is used as a primary filter on the captured candidates and additional filters like presence of commonly used english words, or filtering on known “safe” files like go.
By lookingatcomputer.substack.com via boyter · lookingatcomputer.substack.com
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.