Skip to content
Back to top
View this page

# Charset Normalizer

## Overview

A Library that helps you read text from unknown charset encoding. This project is motivated by chardet, I’m trying to resolve the issue by taking another approach. All IANA character set names for which the Python core library provides codecs are supported.

It aims to be as generic as possible.

It is released under MIT license, see LICENSE for more details. Be aware that no warranty of any kind is provided with this package.

Copyright (C) 2025 Ahmed TAHRI <tahri(dot)ahmed(at)proton.me>

## Introduction

This library aim to assist you in finding what encoding suit the best to content. It DOES NOT try to uncover the originating encoding, in fact this program does not care about it.

By originating we means the one that was precisely used to encode a text file.

Precisely

    my_byte_str = 'Bonjour, je suis à la recherche d\'une aide sur les étoiles'.encode('cp1252')

We ARE NOT looking for cp1252 BUT FOR `Bonjour, je suis à la recherche d'une aide sur les étoiles`. Because of this

    my_byte_str.decode('cp1252') == my_byte_str.decode('cp1256') == my_byte_str.decode('cp1258') == my_byte_str.decode('iso8859_14')
    # Print True !

There is no wrong answer to decode `my_byte_str` to get the exact same result. This is where this library differ from others. There’s not specific probe per encoding table.

## Features

- Encoding detection on a fp (file pointer), bytes or PathLike.

- Transpose any encoded content to Unicode the best we can.

- Detect spoken language in text.

- Ship with a great CLI.

- Also, detect binaries.

### Start Guide

- Support
  - Supported Encodings
  - Supported Languages
  - Incomplete Sequence / Stream
- Installation
  - Using PIP
  - From git via master
- Basic Usage
  - The new way
  - Backward compatibility
- Advanced Search
  - Using CharsetMatches
  - List behaviour
  - Using best()
  - Calling first()
  - Class aliases
  - Verbose output
- Handling Result
  - Using CharsetMatch
- Miscellaneous
  - Convert to str
  - Logging
  - Detect binaries
- Command Line Interface
  - Arguments
  - Main JSON Output
  - Multiple File Input
  - Unicode Conversion

### Community Guide

- Optional speedup extension
  - Why?
  - How?
  - How not to?
- Frequently asked questions
  - Is UTF-8 everywhere already?
  - Should I bother using detection?
  - Is it backward-compatible with Chardet?
  - Isn’t it the same as Chardet?
  - I can’t build standalone executable
- Why should I migrate to Charset-Normalizer?
- Featured projects
  - Niquests
  - httpie-next
  - Wassima

### Developer Guide

- Developer Interfaces
  - Main Interfaces
    - `from_bytes()`
    - `from_fp()`
    - `from_path()`
    - `is_binary()`
    - `CharsetMatches`
    - `CharsetMatch`
    - `detect()`
    - `set_logging_handler()`
  - Mess Detector
    - `mess_ratio()`
    - `MessDetectorPlugin`
    - `is_suspiciously_successive_range()`
  - Coherence Detector
    - `coherence_ratio()`
  - Utilities
    - `is_accentuated()`
    - `remove_accent()`
    - `unicode_range()`
    - `is_latin()`
    - `is_punctuation()`
    - `is_symbol()`
    - `is_emoticon()`
    - `is_separator()`
    - `is_case_variable()`
    - `is_cjk()`
    - `is_hiragana()`
    - `is_katakana()`
    - `is_hangul()`
    - `is_thai()`
    - `is_unicode_range_secondary()`
    - `any_specified_encoding()`
    - `is_multi_byte_encoding()`
    - `identify_sig_or_bom()`
    - `should_strip_sig_or_bom()`
    - `iana_name()`
    - `is_cp_similar()`
    - `os.PathLike`
    - `typing.BinaryIO`

## Indices and tables

- Index

- Module Index

- Search Page