IDN and Punycode

DNS was designed for ASCII. The original label rules allowed only letters, digits, and hyphens — all ASCII characters. For decades, this meant non-English scripts had no native domain names.

Internationalized Domain Names (IDN) solves this by encoding Unicode labels into ASCII-compatible form using punycode.

How Punycode Works

Punycode (RFC 3492) encodes Unicode labels using only ASCII characters. The process:

1. Separate ASCII characters from non-ASCII characters in the label
2. Output the ASCII characters first
3. Append a hyphen if there were any ASCII characters
4. Encode the non-ASCII characters and their positions using a compact numeric encoding

The result is always a valid ASCII string. To signal that a label is punycode-encoded, it is prefixed with xn--.

Examples:

The encoding is reversible: any valid punycode label can be decoded back to its Unicode form.

IDNA: The Standard for Applications

IDNA (Internationalized Domain Names in Applications, RFC 5891) defines how applications should handle IDN:

Input processing (domain registration and display):

1. Accept Unicode input from users
2. Apply Unicode normalization (NFC)
3. Apply IDNA mapping (case fold, remove disallowed characters)
4. Verify the label is valid per IDNA rules
5. Convert to ACE form using punycode for DNS queries and storage

Output processing (displaying stored domains):

1. Detect xn-- prefixed labels
2. Decode from punycode to Unicode
3. Display the Unicode form to the user

Security consideration:

• Some Unicode code points are disallowed in IDN (they look like other characters, creating homograph attack risk)
• The IDNA standard includes a list of permitted characters based on script mixing rules

The xn-- Prefix Convention

Labels are punycode-encoded only when they contain non-ASCII characters. A purely ASCII label (even one containing hyphens) is not punycode-encoded and does not get the xn-- prefix.

The xn-- prefix is reserved for IDNA-encoded labels. This means:

• xn--example.com claims to be a punycode label — the part after xn-- must decode to valid Unicode
• xn--invalid.com claims to be punycode, but invalid is not valid punycode — this should be rejected by IDNA-aware validators
• Labels with -- in positions 3-4 (like ab--example) that don't start with xn are reserved for future extensions

Testing IDN Handling

For any application that accepts email addresses or URLs with international domains:

Test 1: Unicode input acceptance

• Submit user@münchen.de
• Verify: accepted without error

Test 2: Punycode form acceptance

• Submit user@xn--mnchen-3ya.de
• Verify: accepted without error (same domain in ACE form)

Test 3: Consistency check

• Submit both forms in steps 1 and 2
• Verify: both are stored as the same value (either both normalized to Unicode or both to punycode)
• Verify: no duplicate account is created

Test 4: Display round-trip

• Submit user@münchen.de
• Retrieve the stored address from the profile or API
• Verify: the address displays as user@münchen.de (not garbled)

Test 5: Double-encoding check

• Submit user@münchen.de
• If stored as punycode, verify the stored form is user@xn--mnchen-3ya.de
• Verify it is NOT double-encoded as user@xn--xn--mnchen-3ya-... or similar

Homograph Attacks as a Testing Consideration

If your application displays domain names from user input (e.g., "you are logging in to domain X"), IDN creates a homograph attack surface:

• Cyrillic а (U+0430) looks identical to Latin a (U+0061)
• A domain pаypal.com using Cyrillic а is a different domain from paypal.com
• Both are valid IDN domains and both would be accepted by an IDNA-compliant validator

As a security tester: submit domain names that contain lookalike Unicode characters from other scripts. Verify that the application either:

• Rejects mixed-script domains (the safest approach)
• Displays the punycode form when mixed scripts are detected (what modern browsers do)
• Has some other mechanism to prevent homograph-based phishing

This is only relevant for applications that display domain names to users in a security context (OAuth authorization screens, login confirmation dialogs, etc.).