Python write Unicode string to file


You can use the “codecs” module in Python to read or write text in various encodings. To read or write Unicode (also known as “UTF-8”) text, you need to use the “codecs” module.

The “codecs” module provides functions for encoding and decoding text in various encodings. The most common encoding is probably “UTF-8”. You can also use other encodings, such as “Latin-1”, “Windows-1252”, etc.

To read or write Unicode text, you first need to open a file in the proper encoding. For example, to read or write Unicode (UTF-8) text, you would do something like this:

f =“filename.txt”, encoding=”utf-8″)
text =
To write Unicode text to a file, you would do something like this:

f =“filename.txt”, mode=”w”, encoding=”utf-8″)

Basic steps

Unicode strings are stored internally as sequences of codepoints (to be precise as Py_UNICODE arrays). When a unicode string is written to a file, it is first encoded using a particular encoding scheme. The most common encodings are utf-8 and utf-16.

There are now two types of strings in Python 3.x, str for binary data and unicode for text data. When you write a unicode string to a file, you first need to encode it into a bytes object. The most common encoding is utf-8. For example:

f = open(‘test.txt’, ‘w’)
s = ‘\u00e9’ #unicode string
f.write(s) #error! can’t write unicode strings to files in python 2.x
UnicodeEncodeError: ‘ascii’ codec can’t encode character u’\xe9′ in position 0: ordinal not in range(128)

To fix this, you need to encode the unicode string into a byte string first:

f = open(‘test.txt’, ‘w’)
s = ‘\u00e9’ #unicode string
b = s.encode(‘utf-8’) #byte string encoded using utf-8 encoding scheme

Code example

def write_unicode_string(filename, ustring):
with, ‘w’, ‘utf-8’) as f:


Unicode is an industry standard for representing text in most of the world’s writing systems. Python 3 uses the codec system to deal with Unicode, and in this post we’ll look at some of the common problems you may encounter when trying to write Unicode strings to a file.

One common issue is the encoding of special characters such as é or ß. In Python 2, you can write these strings to a file using the correct codec:

import codecs

with‘output.txt’, ‘w’, ‘utf-8’) as f:

In Python 3, however, this will raise an error:

UnicodeEncodeError: ‘ascii’ codec can’t encode character ‘\xe9’ in position 0: ordinal not in range(128)
f = open(‘’, encoding=”utf8″) line 1

The reason for this error is that the default encoding for Python 3 is ASCII, which cannot represent all Unicode characters. To fix this, you need to explicitly set the encoding when you open the file:

import io

with‘output.txt’, ‘w’, encoding=’utf-8′) as f: # note instead of!

Alternatively, you can use sys.setdefaultencoding() to set the default encoding globally: import sys

sys.setdefaultencoding(‘utf8’) # this line only needs to be run once! Then you can remove it and your scripts will still work correctly 🙂

with open(‘output2.txt’, ‘w’) as f: # no need for “encoding” kwarg now! This will work with any kind of text file on any system! f

Leave a Reply

Your email address will not be published.