Full guide for character set conversion: UTF-8, Unicode point and random code

Update 2026-08-09 About 13 minutes development tools How-to

The interface returns the "cracker" and the Chinese in the log becomes the question mark, and the emoji is broken after the JSON paste, and most of it is not the back-end business logic that is broken, but the characters and bytes are misread between different codes. This Guide sets out the code points, UTF-8 multibytes, UTF-16 agents around a KiloCell-character set, gives replicable steps, error contrasts and inter-column scenes, and avoids re-checking coding as business Bug.

Use tools immediately Open character set converter

Open tool →

The core sentence.

The hyphen is almost "the correct byte with the wrong character set". First, we'll check the code points and bytes, then we'll talk business.

Thousands of. Character set converter Enter text in the browser local analysis: number of statistical characters, number of UTF-8 bytes, number of UTF-16 units, and expand the detail by code points. It's not a "one-size-fits-all" machine to pretend to support all historical coding tables, but to help you see what this text is in the Unicode world."

Characters, code points, code: 3-story boundary

Many people mix character sets with code. In the screen-up, it is recommended to split into three layers:

Level	Meaning	Common expression	Performance in error
Character	The symbols that people see in their eyes.	Hello, emoji.	Semantic Layer, not directly equal to byte
Code Point	Unicode Abstract Numbering	U+4F60、U+1F600	Unique number of the same character worldwide
Coding	How the code points become bytes	UTF-8、UTF-16、GBK	There are different bytes at the same level.

UTF-8: Variable length, compatible ASCII

UTF-8 is a code point in 1-4 bytes:

ASCII (U+000-U+007F) is still single-bytes, and the old system is friendly.
Most common Chinese is in 3 byte area.
Most emojis need 4 bytes.

So " One Chinese = 3 bytes" is just a rough experience, not an absolute formula. The character set converter gives the real thing. utf8BytesMore steady than heart.

UTF-16: Unit versus Agent

JavaScript is closer to UTF-16 code elements inside the string. BMP characters are usually 1 unit; Supplementary plane (e.g., most emoji) is paired with a high- and low-altitude agent, representing 2 units. So:

str.length The unit UTF-16 is counted and does not necessarily equal "number of characters seen in the eyes".
Page utf16Units and charCount It may not be consistent -- it's not the tools that are broken, but the Unicode reality.

Relationship with GBK / Latin-1

The point of the KiloCell of pages is Analysis in the Unicode Perspective(Code, UTF-8/UTF-16 spread). If you have a "spoiled file with a GBK, " you need to decode it in an environment that correctly identifies the source code before it is converted to UTF-8. Don't expect to fix the code by "Base64" alone -- it'll only wrap the wrong bytes in one layer.

What do you do on the 1,000?

tools page:Character set converter。 Processing is done locally in browsers and is suitable for pasting de-sensitive samples and for keeping production keys on public computers for long periods.

Basic analytical processes

Open /tools/dev/charset-converter。
Yes. Input Box（inputText) Paste to analyze text, which may include Chinese, English, points, emoji.
Observation of statistical regions:

Characters charCount
UTF-8 bytes utf8Bytes
UTF-16 Unit utf16Units
Maximum size points maxCodepoint

View the bottom breakdown detailsTable: The number points, byte expressions and description of each character.
The "service-end statement" is consistent with "real bytes" by comparing the interface document with the result of the catch.

Suggested comparison of sample

Input	Expecting observation point
`A`	UTF-8 = 1, UTF-16 = 1, code point U+0041
`You.`	UTF-8 Common 3 bytes, code points U+4F60
`😀`	UTF-8 4 bytes; UTF-16 is often Unit 2 (Active pair)
`Hello.😀`	Byte is raised with the unit, and the rows are visible

If a character is abnormal in the table number points, or if the whole of the UTF-8 bytes is much smaller than "Chinese q. 3", the priority is to suspect that the input itself is a hyphen, not "not converted".

Common error contrast

phenomena	Possible cause	Treatment of recommendations
Page Display `锟- What?拷`	Read UTF-8 as a GBK (or reverse) with the wrong encoding	(a) Return source to get the original bytes and decode them correctly by charset; Do not continue to coding over already broken text
Emoji becomes two "freak characters"	Disassembly UTF-16 proxy into two separate characters	Using a code point/word frame perspective; Check if cut is pressed `length` Half-Agent.
JSON Normal Chinese and Writing Library Question Number	Connect string/ library chart charset not utf8mb4	(a) The Charcoux and Connecting Character Set; This tool only proves that "Unicode in memory is right"
The length of the same string is not consistent	Bytes for one party, by bits/chars for one party	View UTF-8 bytes and UTF-16 units with this page, with uniform measures
Only in the log. `?`	Transfer links replace non-magnifying characters with question marks	(a) The question mark is irreversible; We must return to the original response before replacement.
Copy "unseen character" after pasting	BOM, zero width characters, combination of sound signs	A detailed table of the anomaly points can be exposed; We'll clean it up when we need it.

Intentional unlawful / easily confused examples

Half-tone UTF-8: Only two bytes in front of multi-byte series are retained and displayed in the text, which is necessarily confused. The correct approach is to clearly report errors when the entire paragraph decoded by UTF-8 fails, rather than to replace them silently.
Make Base64 as a character set: Base64 is byte text transport mode, see Base64 Guide； It doesn't change the "what character code is in it."
URL %1 encoding：%E4%BD%A0 It's the UTF-8 byte number form, decoded and decoded before "you". Complementarity URL encoding。

Real work scenarios

1. Back-to-back connection: Content-Type says one set, body does one set

Reply Header charset=utf-8, but the gateway or old middle is recoded by default system. Paste this tool to the body-response "looks normal/unnormal" clip to check if the code point is a target word. If the code points are right and the pages are still in disarray, the problem is on the presentation level; If the code is wrong, the problem is explained earlier by by bytes.

Logs and surveillance alerts

The alarm robots put the unusual news into the crowd, and the Chinese occasionally gave the question mark. Take the same original log file (confirming that the editor opened with UTF-8) and compare the text with the text to be sent: check the robot or webbook encoding if the internal code point of the file is correct, the transfer error is wrong; Check the data in case the document itself is wrong.

Data Migration and CSV Import

Operations export CSV, Excel from Windows to save by ANSI system, import Linux conduits read by UTF-8. This tool quickly validates the "a sample of a column" after it is opened correctly. The batch code should still be used for a special ETL, but the sample level is sufficiently rapid.

4. Interception and storage length

The product requires "up to 12 characters by nickname". If backend by bytes limit 12, frontend press length The limit is 12, and the Chinese and the emoji will step on the pits. Read UTF-8 bytes and UTF-16 units simultaneously with this page, and the product confirms the definition of " character " , and then write a verification.

5. Security and privacy

Please be allergic to pasting the user ' s real name, ID number and coded experiment. The tool is not locally run, but it may still be entered into the browser extension or the synchronized clipboard, which is a public computer that has run out of empty input boxes.

And what about the tools?

Requirements	Priority tool	Reason
Seep point / UTF-8 / UTF-16 spread	Character set converter	Home Path
Text ↔ Binary Bitstring	Binary Translator	Focus 0/1 indicates, not a table of points
Plug in any byte in text channel	Base64	Transport mode, not solving charset dispute
Chinese in URL query parameters	URL encoding	Percentage Encoding Layer
Need Hash's fingerprints, not readable text.	Hash Generators	It's irreversible. It's not about code conversion.

Decision-making phrase: Ask first, "I want to read characters or just move bytes." (a) Understand the character set analysis; Removes the Base64/URL encoding.

Checklist

[ ] The source bytes are confirmed to be consistent with the declared charset
[ ] Check the sample code points with this tool, not just "is it good?"
[ ] Distinguishing between three measures of character number, UTF-8 bytes, UTF-16 units
[ ] emoji / Visible word is verified by proxy with 4 byte UTF-8
[ ] The original file copy was retained before the scuffled sample was repaired
[ ] Production data are dissensitized

FAQ

Why is it also "Hello" with a system showing six bytes and four characters?

Under UTF-8, "you" is usually 3 bytes, 6 bytes total; 2 characters by word/code points; It may also be 2 in some environments measured in UTF-16 units. The questions answered by the three were different and the limits were not to be used in combination.

Could the tool directly convert the GBK file to UTF-8?

The core of this page is the Unicode text analysis. If the input box is already a correctly decoded JS string, you see the Unicode truth. If the file is still a GBK original byte, it needs to be decoded in the environment where the code is supported, then paste the analysis or save it as UTF-8.

Can you save it if you show up?

In most cases, the torture represents an error in decoded and recoded information. If you can also get the original byte stream before the error is converted, you can reset it correctly by charset; Only hard copies are often irreversible.

Is it a front-end problem that emoji becomes two boxes or a obscurity in the database?

First confirm in this tool whether the code points are on the supplementary plane, and whether the UTF-8 is 4 bytes. If the front end is correct and the library is utf8(max. 3 bytes) utf8mb4It's the storage. Don't just change the CSS.

Why is there a possible inconsistency between the `String.length ' and the number of characters on the page?

Depending on whether it is achieved by UTF-16, Unicode or user sensory phrase. This page gives a variety of statistics that will facilitate your behaviour when you run against your language.

What's the difference between coding problems and encryption, Hash?

(a) The code is reversible (under the correct rule) and the objective is to indicate; (a) Encryption to protect confidentiality; Hash made prints. The password "Unique UTF-8" or "Beace 64" is not protected, as detailed in the relevant security tool document.

A big log card?

Browser-end resolvers large text eat memory and main threads. First, you cut the smallest piece of the problem that can be repeated, then you analyze it. The complete log is processed with local scripts.

How do you prove that "I'm already UTF-8 on my side"?

Don't make any promises. Export hexadecimals or view UTF-8 bytes on this page against the original body in the grab package; Check HTTP header Content-Type The charset parameter is consistent with the facts.

Summary

The nature of the character set problem is The coded and byte contracts were broken.。 The KiloCell char set converter allows you to see whether the contract is valid locally: number of characters, UTF-8 bytes, UTF-16 units and word-code points. It is much cheaper to characterize it, then to change the gateway, to change the database or to change the front-end cut-off logic than to repeat the error in the business code.