Python encoding problems on startup script in Raspberry Pi debian

Question

So I have a Raspberry Pi running Debian and the WebIOPi system. The daemon runs just fine when I start it using a standard start ´$ sudo /etc/init.d/webiopi start´ command but the auto start doesn't work ´$ sudo update-rc.d webiopi defaults´.

The log suggests it is a encoding problem in Python, my guess python is using ascii but the script file is using utf8.

Why does it work normally but not in auto start and what is a good way to fix this?

Answer 1

To confirm the log and your suspicion, this is an encoding issue (bytes vs string most likely). Even if you set the encoding of your file to UTF-8 with # -*- coding: utf-8 -*-, you can still have issues working with text which has been changed from one form to another.

Strings are not really strings, but representations of bytes in a particular order. UTF-8 provides for the encoding of many more characters than ASCII can handle, so if you try to convert a character which exists in a UTF-8 encoded string to an ASCII encoded string, then you will get an error because no such encoding exists.

I cannot give a much better answer without more information, such as code and / or data source.

Reading https://docs.python.org/2/howto/unicode.html#the-unicode-type we learn by studying the following example(s):

>>> unicode('abcdef')
u'abcdef'
>>> s = unicode('abcdef')
>>> type(s)
<type 'unicode'>
>>> unicode('abcdef' + chr(255))    
Traceback (most recent call last):
...
UnicodeDecodeError: 'ascii' codec can't decode byte 0xff in position 6:
ordinal not in range(128)

The documentation also mentions that you can choose to handle these exceptions by either replacing or ignoring them, like so:

>>> unicode('\x80abc', errors='strict')     
Traceback (most recent call last):
    ...
UnicodeDecodeError: 'ascii' codec can't decode byte 0x80 in position 0:
ordinal not in range(128)
>>> unicode('\x80abc', errors='replace')
u'\ufffdabc'
>>> unicode('\x80abc', errors='ignore')
u'abc'

NOTE 1: In Python 3 things have changed. for writing code which is compatible with Python 3, I recommend the following read:

https://docs.python.org/3.0/whatsnew/3.0.html#text-vs-data-instead-of-unicode-vs-8-bit

NOTE 2: It is also worth noting that if your encoding issue is encountered while trying to display the string on the console then, python has a -u switch which can be used in certain situations, such as when you are serving a binary file through a CGI script, which will turn off buffering of strings, but that opens up another can of worms. But, none the less, mimicking this behavior without invoking -u:

>>> print 'test'
'test'
>>> import os
>>> import sys
>>> sys.stdout = os.fdopen(sys.stdout.fileno(), 'w', 0)
>>> print 'test'
test

Answer 2

This truly depends on that program.

However if there's a difference between starting it manually and as a part of system startup, then the core difference ought to be in environment variables. This typical for e.g. inittab.

Two options here -- either your locale is not set up, or script ends up using different version of python.

For the earlier, consider adding LANG (precise) or if needed LC_ALL (a hammer) to this script or global environment for all startup tasks.

You can try manually by comparing these two invocations:

sudo LC_ALL=en_US.UTF-8 /etc/init.d/webiopi start

vs.

sudo LC_ALL=C /etc/init.d/webiopi start

For the latter try changing PATH to only include some of /bin:/usr/bin:/usr/local/bin