The Complete Guide to Text Encoding on the Web

Every character you have ever typed is a fiction. Storage and networks move bytes — numbers from 0 to 255 — and a character like é, →, or 🙂 only exists because some agreed-upon rule says which bytes stand for it. Text encoding is that rule. When two systems agree on it, text moves invisibly and nobody thinks about it. When they disagree, you get é, ’, %2520, and support tickets titled "weird symbols in the export".

This guide covers the encodings you actually encounter on the web: the Unicode/UTF-8 foundation that decides how characters become bytes, and the transport encodings — Base64, percent-encoding, HTML entities, hex — that wrap those bytes so they survive a channel with rules of its own. It also covers the failure mode that ties them all together, and how to reason about which one you need.

Two different jobs, both called "encoding"

The word gets used for two distinct things, and conflating them is the source of most confusion.

Character encoding answers: how do I store this character as bytes? UTF-8, UTF-16, ASCII, Windows-1252, and Shift-JIS are character encodings. This is the base layer — every piece of text has one, whether or not anyone declared it.

Transport encoding answers: how do I move these bytes through a channel that forbids some of them? Base64, percent-encoding, HTML entities, and hex are transport encodings. They take bytes or characters that would break a container and rewrite them in a restricted alphabet the container tolerates.

Both are reversible, both are public, and neither one is a security control. That last point deserves its own emphasis: encoding provides no confidentiality whatsoever, which is why Base64 is not encryption is a distinction worth internalizing before it costs you a credential leak.

Unicode is the character set; UTF-8 is the encoding

These two get used interchangeably in conversation and they are not the same thing.

Unicode is a catalogue. It assigns every character a unique number called a code point, written as U+ plus hex: A is U+0041, é is U+00E9, → is U+2192, 🙂 is U+1F642. Unicode says nothing about bytes. It is a list of characters and their numbers, currently covering more than 150,000 of them across every writing system in active use.

UTF-8, UTF-16, and UTF-32 are encodings of Unicode — three different ways to serialize those code point numbers into bytes.

UTF-8 won the web, and its design explains why:

• It is variable-width, using 1 to 4 bytes per code point. ASCII characters take 1 byte, most Latin and Cyrillic and Greek letters take 2, most CJK characters take 3, and emoji and rarer scripts take 4.
• It is backward-compatible with 7-bit ASCII. Code points U+0000 through U+007F encode to a single byte with the same value ASCII used. A pure-ASCII file is byte-for-byte identical whether you call it ASCII or UTF-8, which meant existing tooling kept working during the transition.
• It is self-synchronizing. Lead bytes and continuation bytes are distinguishable, so a decoder that starts mid-stream can find the next character boundary instead of producing garbage forever.

ASCII, by contrast, is a 1963 standard with exactly 128 characters and no room for anything else. The 8-bit "extended ASCII" charsets — Latin-1, Windows-1252 — bolted 128 more characters onto the top half of the byte range, and every region picked a different set. That fragmentation is precisely the problem Unicode was invented to end. The UTF-8 vs ASCII vs Unicode breakdown goes deeper into the byte-level mechanics.

Practical takeaway: use UTF-8 everywhere — files, database columns, HTTP headers, <meta charset="utf-8"> — and declare it explicitly rather than hoping something guesses right.

Base64: binary through a text-only door

Plenty of channels only accept text: email bodies, JSON string values, XML documents, HTTP headers, data: URIs. Raw binary sent through them gets mangled — a byte that happens to equal a newline, or a byte above 127 that some intermediary reinterprets, and the payload is corrupt.

Base64 solves this by re-expressing arbitrary bytes in a 64-character alphabet that survives anything: A-Z, a-z, 0-9, +, and /, with = used for padding.

The mechanism is straightforward arithmetic. Take 3 input bytes — 24 bits. Split those 24 bits into four 6-bit groups. Each 6-bit group is a number from 0 to 63, which indexes into the alphabet. So 3 bytes in, 4 characters out, always. When the input length is not a multiple of 3, the final group is padded and the output gets one or two = characters to signal how many bytes were real.

That 4:3 ratio is where the well-known ~33% size inflation comes from. Base64 is not compression — it is the opposite of compression, and applying it to a 5 MB image to embed as a data URI gives you about 6.7 MB of text.

Common places you will meet it: data: URIs for inlined images and fonts, email attachments via MIME, binary blobs stuffed into JSON, HTTP Basic authentication headers, and the three segments of a JWT. That last one uses Base64URL, a variant that swaps + and / for - and _ so the result is safe inside a URL without further escaping.

You can watch the transformation happen with our Base64 Encoder / Decoder — encode a short string, then change one input character and see how it shifts the output. If you want the fuller tour, what Base64 is and why developers use it covers the alphabet, padding, and typical use cases in detail.

URL encoding: escaping inside an address

URLs have a grammar. Characters like ?, #, &, =, /, and : are reserved — they delimit the parts of the address. Others, including spaces and most non-ASCII characters, are simply unsafe to transmit literally.

Percent-encoding handles both cases with one rule: take the character's byte value and write it as % followed by two hex digits. A space is byte 0x20, so it becomes %20. An ampersand is 0x26, so it becomes %26. Non-ASCII characters are first encoded to UTF-8 bytes, then each byte gets its own %XX — which is why é becomes %C3%A9, two escapes for one character.

The nuance that catches people: a space is not always %20. In a URL path it is. But in a query string submitted as application/x-www-form-urlencoded — the default for HTML form GETs — a space is traditionally written as +. Both conventions are alive in the wild, so a decoder that assumes the wrong one will silently turn a genuine plus sign into a space. Anyone handling query strings should read URL encoding explained for the full reserved-character table and the encoder-selection rules.

The critical discipline is encode the parts, not the whole. Encode each parameter name and value individually, then assemble them with & and =. Encoding an already-assembled URL destroys its structure. Our URL Encoder / Decoder lets you test a single value in isolation, which is usually the fastest way to confirm what your backend is actually receiving.

HTML entities: escaping inside markup

HTML has its own reserved characters, and for the same structural reason. A browser reading < starts looking for a tag name. Reading & it starts looking for an entity. If your content contains those characters literally, the parser interprets them as markup instead of text — which is a rendering bug at best and a cross-site scripting hole at worst.

Entities give each problem character a safe spelling. There are two forms, and they mean exactly the same thing:

• Named: & < > " ' — memorable, but only defined for a specific list of characters.
• Numeric: decimal & or hexadecimal & — works for any Unicode code point, since it just names the code point directly.

Both &, &, and & produce a literal &. The five that genuinely matter are the ampersand, the angle brackets, and the quote characters — the ampersand first, because it starts every other entity, so escaping it out of order double-encodes everything else.

Since UTF-8 handles accents and symbols natively, entities like é or © are optional today; the escaping ones are not. See HTML entities explained for when each form is required, and use the HTML Entity Encoder / Decoder to check markup you did not write.

Hex and binary: seeing the bytes themselves

Hex is not really a transport encoding — it is a human-readable view of bytes, two hex digits per byte, no interpretation applied. It shows up in colour codes (#FF5733), MAC addresses, hash digests, and every hex dump you have ever squinted at.

Its value during encoding debugging is that it is unambiguous. When text is rendering wrong and you cannot tell why, converting it to hex with the Hex to Text Converter shows you the actual byte sequence. Seeing C3 A9 where you expected E9 immediately tells you the data is UTF-8 and something downstream is reading it as Latin-1 — a diagnosis you cannot make by staring at rendered glyphs.

When encodings stack

Real systems layer these, and layering is where the bugs live.

A binary file gets Base64-encoded to fit in JSON, and the JSON gets URL-encoded to fit in a query parameter. That is fine — as long as each layer is unwrapped exactly once, in reverse order.

The classic failure is double encoding. A value is percent-encoded by application code, then a framework percent-encodes it again. The % in %20 is itself byte 0x25, so it becomes %25, and %20 becomes %2520. One decode pass yields %20 — a literal string, not a space. The same happens in HTML when & gets escaped a second time into & and renders visibly as & on the page.

Two rules prevent nearly all of it: know which layer owns the encoding and let only that layer do it, and never encode a value you did not personally produce in raw form.

Mojibake: the signature of a charset mismatch

café displaying as café is not corruption. The bytes are perfectly intact. They were written as UTF-8, where é is the two bytes C3 A9, and then read by something that assumed a single-byte charset like Windows-1252 — which maps C3 to Ã and A9 to ©. Two bytes, two characters, both wrong.

The tell is diagnostic. Patterns like é, ’, and “ are near-certain evidence of UTF-8 read as Windows-1252. Black diamonds with question marks (�) mean the decoder hit a byte sequence that is not valid in the encoding it was told to use. Plain ? characters mean something converted to a charset that had no room for the character at all — and unlike the others, that one is lossy and unrecoverable.

Fixes are almost always about declaration rather than transformation: set <meta charset="utf-8">, send Content-Type: text/html; charset=utf-8, use utf8mb4 in MySQL, and open CSVs with the encoding stated explicitly instead of letting a spreadsheet guess. Diagnosing and fixing mojibake walks through each pattern and the repair path.

Picking the right one

You need to	Use	Why
Put binary in JSON, email, or a data URI	Base64	Text-safe alphabet; costs ~33% size
Put a value in a URL path or query string	Percent-encoding	Protects reserved delimiters
Put binary in a URL or JWT	Base64URL	No + or / to re-escape
Show user text inside HTML	HTML entities	Prevents markup injection
Store or transmit any text at all	UTF-8	Universal coverage, ASCII-compatible
Keep something secret	None of these	Encoding is not encryption — use AES

Encode without shipping your data anywhere

Encoding tools handle exactly the material you should be careful with: API responses, tokens, customer records, config fragments. Most online encoders POST your input to a server to do the work, which means a debugging session quietly becomes a data transfer.

Every tool on this site runs entirely in your browser. The encoding and decoding happen in local JavaScript, nothing is uploaded, and nothing is logged — you can confirm it by opening your network tab and watching it stay empty while you type, or by disconnecting from the internet and using the page anyway.

Conclusion

Text encoding is one layer with two floors. Underneath, Unicode names every character and UTF-8 turns those names into bytes; use it everywhere and declare it explicitly. On top, transport encodings adapt those bytes to a channel's rules — Base64 for binary through text, percent-encoding for URLs, entities for HTML, hex for inspection.

Almost every encoding bug reduces to one of three things: an undeclared or wrongly-declared charset, a layer applied twice, or a layer not removed. Once you can name which encoding is involved and which direction it needs to go, the fix is usually one line. And when you need to check what a string really contains, the encoder, decoder, and hex tools linked above will tell you in seconds — without your data leaving the tab.