From a Silent Math Error to Certificate Bypass: Uncovering an Integer Overflow in a TLS Parser

From a Silent Math Error to Certificate Bypass: Uncovering an Integer Overflow in a TLS Parser | by Hacker MD - Freedium Milestone: 20GB Reached We’ve reached 20GB of stored data — thank you for helping us grow! Patreon Ko-fi Liberapay Close < Go to the original From a Silent Math Error to Certificate Bypass: Uncovering an Integer Overflow in a TLS Parser Bug hunting isn't always about popping XSS alerts or finding chained SSRFs. Sometimes, the most critical vulnerabilities are hidden deep… Hacker MD Follow ~4 min read · March 22, 2026 (Updated: March 22, 2026) · Free: Yes Bug hunting isn't always about popping XSS alerts or finding chained SSRFs. Sometimes, the most critical vulnerabilities are hidden deep within the foundational cryptographic parsing logic of an application. Recently, while auditing the TLS and certificate validation module of a highly popular open-source Java/Kotlin library, I discovered a CWE-190 (Integer Overflow or Wraparound) vulnerability. This wasn't a bug that caused a loud, noisy crash. Instead, it was a silent failure in the ASN.1 DER parser that could theoretically allow an attacker to spoof Object Identifiers (OIDs) and bypass certificate validation. Here is a deep dive into what the bug was, how I exploited it, and what other researchers can learn from this journey. 1. The Background: Understanding VLQs and ASN.1 DER To understand the vulnerability, we first need to look at how X.509 certificates store large numbers, like Tags and OIDs. They use an encoding format called Variable-Length Quantity (VLQ) in ASN.1 DER. In this Base-128 format: The data is processed byte by byte. The Most Significant Bit (MSB) of each byte is a flag: 1 means "more bytes are coming." 0 means "this is the final byte." The remaining 7 bits of each byte hold the actual data, which are shifted and added together to form the final large integer. 2. The Vulnerability: The Silent Overflow While reviewing the library's DerReader component, I found the function responsible for decoding these VLQ integers. The code looked exactly like this: private fun readVariableLengthLong(): Long { // A developer comment here literally said: "TODO: detect overflow." 🚩 var result = 0L while (true) { val byteN = source.readByte().toLong() and 0xff if ((byteN and 0b1000_0000L) == 0b1000_0000L) { // If continuation bit is 1 result = (result + (byteN and 0b0111_1111)) shl 7 // Shift left by 7 } else { return result + byteN } } } The Logical Flaw: The result variable is a 64-bit signed Long . The while loop continuously shifts bits to the left ( shl 7 ) as long as the continuation bit is set. There is absolutely no boundary or size validation. If an attacker feeds the parser a malicious sequence of 11 bytes (10 continuation bytes + 1 final byte), the parser performs 70 bits of left shifts ($10 \times 7 = 70$). Because the container is only 64 bits wide, the most significant bits simply "fall off" the edge of the register and disappear. The program doesn't crash; it silently truncates the massive number into a tiny, incorrect value. 3. The Exploitation: Crafting the Payload To prove this, I built a local test harness linked directly to the target library's source files. I crafted a payload designed to perfectly overflow the register and leave behind a specific value: 1 . The Malicious Byte Array (Hex): 0x1F, 0x81, 0x80, 0x80, 0x80, 0x80, 0x80, 0x80, 0x80, 0x80, 0x80, 0x01, 0x00 0x1F : Extended Tag marker to trigger the readVariableLengthLong function. 0x81...0x01 : The 11-byte VLQ representing a massive 77-bit logical integer. 0x00 : Dummy length byte. The Result: When I executed the parser against this payload, it effortlessly processed the 77-bit number and outputted: Resulting Tag Value: 1 I had successfully forced a massive identifier to masquerade as the number 1 . 4. The Security Impact: OID Spoofing Why does a mathematical truncation matter? Because this parser is the gatekeeper for TLS trust. During a TLS handshake, the library parses X.509 certificates and relies on OIDs to identify critical elements, such as the Signature Algorithm or Root CA identifiers . By utilizing this overflow, an attacker can engineer a rogue certificate with a "Massive OID." When parsed, the library silently truncates this massive OID into a smaller value. If the attacker crafts the payload so that the truncated value matches the internal ID of a Trusted Standard OID , the library will process the malicious certificate as "Valid." This creates a logic bypass, opening the door to certificate spoofing and Man-in-the-Middle (MitM) attacks. 5. The Remediation The fix is straightforward. I proposed adding a boundary check before the bitwise shift occurs to ensure the 64-bit limit isn't breached: if (result > (Long.MAX_VALUE ushr 7)) { throw ProtocolException("Variable length long overflow") } result = (result + bits) shl 7 With this patch, the parser immediately detects the malicious input and throws a fatal exception, closing the connection instead of processing corrupted trust data. Pro Tips for Bug Hunters If you want to dive into deep-code auditing, here are a few tips from my experience with this bug: Follow the // TODOs : Developers often leave breadcrumbs. When you see comments like // TODO: detect overflow or // FIXME: add validation , stop everything and test that exact function. It is literally a map to a vulnerability. Build Local Harnesses: Don't just rely on black-box testing. Extract the specific vulnerable classes/files, link them in your own local environment (like compiling with kotlinc or javac ), and write isolated unit tests to trigger the bug. It makes debugging and creating PoCs 10x easier. A Crash is a Bug, A Bypass is a Bounty: Triage teams often reject simple math errors or crashes as "Not Applicable" if they don't see a business impact. Always ask yourself: "How does this math error break a security decision?" Translating a silent overflow into an OID Spoofing scenario is what elevates a finding from "informative" to a "security vulnerability." Test the Boundaries: Whenever you see data being shifted ( << , >> , shl ) or concatenated in a loop, check the size of the variable holding that data. If an attacker controls the loop iterations, you have an overflow. Happy Hunting! 🎯 #BugBounty #CyberSecurity #InfoSec #AppSec #EthicalHacking #IntegerOverflow #Cryptography #CodeReview #VulnerabilityResearch #TLSSecurity #BugBountyTips #OffensiveSecurity #TechBlog #cybersecurity #bug-bounty #infosec #cryptography #application-security Reporting a Problem Sometimes we have problems displaying some Medium posts. If you have a problem that some images aren't loading - try using VPN. Probably you have problem with access to Medium CDN (or fucking Cloudflare's bot detection algorithms are blocking you).