Detect strings with non English characters in Python

Posted on

Question :

Detect strings with non English characters in Python

I have some strings that have a mix of English and none English letters. For example:


How can I recognize these types of string using Regex or any other fast method in Python?

I prefer not to compare letters of the string one by one with a list of letters, but to do this in one shot and quickly.

Asked By: TJ1


Answer #1:

You can just check whether the string can be encoded only with ASCII characters (which are Latin alphabet + some other characters). If it can not be encoded, then it has the characters from some other alphabet.

Note the comment # -*- coding: ..... It should be there at the top of the python file (otherwise you would receive some error about encoding)

# -*- coding: utf-8 -*-
def isEnglish(s):
    except UnicodeDecodeError:
        return False
        return True

assert not isEnglish('slabiky, ale liší se podle významu')
assert isEnglish('English')
assert not isEnglish('?? ???????? ?? ?????? ??')
assert not isEnglish('how about this one : ? asf?')
assert isEnglish('?fd4))45s&')
Answered By: Salvador Dali

Answer #2:

If you work with strings (not unicode objects), you can clean it with translation and check with isalnum(), which is better than to throw Exceptions:

import string

def isEnglish(s):
    return s.translate(None, string.punctuation).isalnum()

print isEnglish('slabiky, ale liší se podle významu')
print isEnglish('English')
print isEnglish('?? ???????? ?? ?????? ??')
print isEnglish('how about this one : ? asf?')
print isEnglish('?fd4))45s&')
print isEnglish('????? ?? ???????')

> False
> True
> False
> False
> True
> False

Also you can filter non-ascii characters from string with this function:

ascii = set(string.printable)   

def remove_non_ascii(s):
    return filter(lambda x: x in ascii, s)

remove_non_ascii('slabiky, ale liší se podle významu')
> slabiky, ale li se podle vznamu
Answered By: Katerina

Answer #3:

IMHO it is the simpliest solution:

def isEnglish(s):
  return s.isascii()


Answered By: Torello

Answer #4:

import re

english_check = re.compile(r'[a-z]')

if english_check.match(w):
    print "english",w
    print "other:",w
Answered By: PemaGrg

Answer #5:


You can easily see the method in docs:

Return true if the string is a valid identifier according to the language definition, section Identifiers and keywords.

Answered By: Furkan

Answer #6:

I believe this one would have a minimal runtime since it stops once it finds a character which is not a latin letter. It also uses a generator for better memory usage.

import string

def has_only_latin_letters(name):
    char_set = string.ascii_letters
    return all((True if x in char_set else False for x in name))

>>> has_only_latin_letters('_1991_??_??2')
>>> has_only_latin_letters('bla bla')
>>> has_only_latin_letters('blä blä')
>>> has_only_latin_letters('????????')
>>> has_only_latin_letters('also a string with numbers and punctuation 1, 2, 4')

You can also use a different set of characters:

>>> string.ascii_letters

>>> string.ascii_lowercase

>>> string.ascii_uppercase

>>> string.punctuation

>>> string.digits

>>> string.digits + string.lowercase

>>> string.printable
'()*+,-./:;<=>?@[\]^_`{|}~ tnrx0bx0c'

To add latin accented letters, you can refer to this post.

Answered By: roi3363

Leave a Reply

Your email address will not be published. Required fields are marked *