Python Unicode Tutorial

Why is Python getting Unicode support?

Data Storage

If your customer database is all English, or even all Japanese, you can store it any way you like. But if you have to keep English, Japanese, Russian and Thai in the same file or database column, you can;t use a native encoding - you really need something like Unicode.

Encoding Conversion

If a new encoding needs to be added to a library, it is only necessary to establish a mapping to and from Unicode, and not to every other encoding in the world

Operations on wide characters

Asian languages have to use more tha one byte per character. Most native encodings use a mix of single bytes for ASCII, and two bytes per chinese character. Software that needs to slice strings can potentially cut a character in half. It is much, much easier to write string- processing operations in Unicode, where every character is the same width.

Operating System Compatibility

For the above reasons, operating systems and low-level APIs have been moving to support Unicode, and there are more and more functions around which expect Unicode strings as arguments, or which return them.

Installation and Setup

If you are used to using a compiled binary distribution of Python, such as the one on Windows, you need to make some potentially destructive changes to your existing environment. Here are some guidelines: (TODO)

TODO - extract the diffs for the library - exceptions

overwrite python.exe and pythonw.exe in C:\program files\python with the ones in the zip file

if using Pythonwin, it won't start, as there is an illegal use of list.append() in the scintilla code, which Guido is going to ban for 1.6. Look in ..\python\pythonwin\pywin\scintilla\view.py lines 71 and 73, and add the extra brackets so they look like this:

                event_commands.append((event, val))
        for name, id in _extra_event_commands:
                event_commands.append((name, id))

Viewing data in different encodings

Let's imagine we have an HTML file containing the name of the author of the unicode extensions, encoded in ISO-latin-1. This contains an acute letter e, which is not available in ASCII. If you have this encoding selected in the browser, and have any Asian fonts installed, you should see this:

If you now go and select (say) UTF8, the bytes will be interpreted differently and you will see this instead:

That's because the three bytes from the acute-e to the 'L' in Lemburg are UTF8 for a Chinese character. (If you don't have the fonts installed, you'll see a round blob, which is the generic 'I don't know how to display this' symbol in IE5.)

Seen from a long distance away, the Unicode extensions are largely about how to prevent this kind of thing: letting you explicitly control the encodings of the files you work with, and converting between them as needed. Unicode itself is an internal technology to make this easier.

Basics about Unicode strings

Creating Unicode Strings

>>> u"Hello World!"  #create a Unicode string
u'Hello World!'

Python 1.6 also gets a "unicode" built-in function, to which you can specify the encoding:

>>> unicode('hello')
u'hello'
>>> unicode('hello', 'ascii')
u'hello'
>>> unicode('hello', 'iso-8859-1')
u'hello'
>>>

Now let's encode something with a European accent, which is outside of ASCII. What you see at a console may depend on your operating system locale; Windows lets me type in ISO-Latin-1.

>>> a = unicode('André','latin-1')
>>> a
u'Andr\202'

Unicode supports all the common operations such as iteration and splitting. We won't run over them here.

Encoding conversions

>>> a.encode('latin-1')
'Andr\202'
>>> print a.encode('latin-1')
André
>>> a.encode('utf8')
'Andr\302\202'
>>>

Not all conversions are possible because not every encoding includes every Unicode character. If we try to convert to ASCII, we should expect an error:

>>> a.encode('ascii')
Traceback (innermost last):
  File "", line 1, in ?
UnicodeError: ASCII encoding error: ordinal not in range(128)

>>> a.encode('ascii', 'strict')  # the default, raise exception
Traceback (innermost last):
  File "", line 1, in ?
UnicodeError: ASCII encoding error: ordinal not in range(128)
>>> a.encode('ascii', 'ignore')  # turn to zero and continue 
'Andr\000'
>>> a.encode('ascii', 'replace') # replace with a readable error character
'Andr?'
>>>

Looking at the raw data

>>> a.encode('utf-16')
'\377\376A\000n\000d\000r\000\202\000'
>>>

>>> for char in a:
...     print ord(char)
...
65
110
100
114
130
>>>

Marc-Andre's original snippets

>>> u"Hello World!"
u'Hello World!'

>>> # Strings and Unicode do auto-coercion, e.g.
>>> u"Hello World!".split(u' ')
[u'Hello', u'World!']
>>> # This also works with normal strings:
>>> u"Hello World!".split(' ')
[u'Hello', u'World!']

>>> # Strings and Unicode can interoperate:
>>> 'Hello' + u' ' + 'World' + u'!'
u'Hello World!'

>>> # The string module handles both worlds using a single API
>>> import string
>>> string.split(u'Hello World!')
[u'Hello', u'World!']
>>> string.split('Hello World!')
['Hello', 'World!']

>>> # Using Codecs is easy:
>>> unicode('Hello World!','latin-1')
u'Hello World!'
>>> # ... in both directions:
>>> unicode('Hello World!','latin-1').encode('ascii')
'Hello World!'
>>> # The Unicode-Escape encoding simplifies entering Unicode
>>> # directly:
>>> u'Hello\u1234World!'
u'Hello\u1234World!'

>>> # Codecs raise a UnicodeError in case conversion is not
>>> # possible (note the center dot between the words):
>>> unicode('Hello·World!','latin-1').encode('ascii')
Traceback (innermost last):
  File "", line 1, in ?
UnicodeError: ASCII encoding error: ordinal not in range(128)
>>> unicode('Hello·World!','latin-1').encode('utf-8')
'Hello\302\267World!'

Here is an example of stackable streams:
import codecs,sys

# Convert Unicode -> UTF-8
(e,d,sr,sw) = codecs.lookup('utf-8')
unicode_to_utf8 = sw(sys.stdout)

# Convert Latin-1 -> Unicode during .write
(e,d,sr,sw) = codecs.lookup('latin-1')
class StreamRewriter(codecs.StreamWriter):

    encode = e
    decode = d

    def write(self,object):

        """ Writes the object's contents encoded to self.stream
            and returns the number of bytes written.
        """
        data,consumed = self.decode(object,self.errors)
        self.stream.write(data)
        return len(data)
    
latin1_to_utf8 = StreamRewriter(unicode_to_utf8)

# Now install
sys.stdout = latin1_to_utf8

# All subsequent prints will output Latin-1 strings using UTF-8
# characters...

>>> print 'Hello World!'
Hello World!
>>> print 'Hello·World!'
Hello·World!