My Tech Notes

This post continues a topic "how to deal with UTF-16-LE encoding", see this post.

If writing UTF-16-LE is no picnic, reading is even more challenging, unless you want to read the whole file as one line. If you prefer or are forced to use per-line approach, then you'd better be aware that properly encoded UTF-16-LE file includes zero byte '\0' after the end-of-line symbol, that is, every line ends up with four bytes 0x0d000a00, or if you will '\r\x00\n\x00'. However, when reading line-by-line, processing stops at '\n' and following '\x00' is interpreted as belonging to the next line!

To rectify this problem, you can use this Python-based "generator" to read UTF-16-LE-encoded file lile-by-line as unicode:

def readiterator(file) :
    fh = open ( file, "rb" )
    for line in fh :
        if line == '\x00' : continue
        if line[:2] == '\xff\xfe' :
            line = line[2:] + "\x00"
        else :
            line = line[1:] + "\x00"
        res = unicode ( line, "utf_16_le" )
        yield res.encode ( "utf-8" )
    fh.close ()

One typical application of that would be parsing CSV file (e.g., reasult of Excel CSV export) using Python built-in "csv" module, which has no knowledge of encoding, though luckily does support unicode input:

cvsreader = csv.reader(readiterator(input_csv_file))

If you decide to (for example) make changes to the table and save it again as CSV file, you'll quickly discover that you can't similarly use csv.write() directly, since it does not work with unicode strings, at all. You will have to play a trick taken directly from official Python documentation, to first convert to UTF-8 and dump CSV to a temporary string , and then read this string and convert back to unicode. Here one way to do that:

class MyCSVWriter:
    def __init__ (self,file_writer) :
        self.stream = file_writer
        self.queue = StringIO.StringIO ()
        self.writer = csv.writer(self.queue)

    def writerow (self,row) :
        self.writer.writerow([s.encode("utf-8") for s in row])
        self.stream.write(unicode(self.queue.getvalue(),"utf-8"))
        self.queue.truncate(0)

    def close(self) :
        self.stream.close ()

Of course, you still need a backend to dump unicode data as UTF-16-LE-encoded file:

class MyFileWriter:
    def __init__ ( self, file ) :
        self.fh = open ( file, "wb" )
        self.lineno = 0
            
    def write (self, line) :
        if self.lineno == 0 :
            self.fh.write ( '\xff\xfe' )
        self.lineno += 1
        self.fh.write ( line.encode ( "utf_16_le" ) )

    def close (self) :
        self.fh.close()

These two classes finally make it possible to create a CSV "writer" which can be used to write data just retrieved by aforementioned "reader"

csvwriter = MyCSVWriter(MyFileWriter(output_csv_file))

All of these code snippets are taken from an utility parsegab.py which I write to make some very specific changes to Google Address Book, using workflow "export – fix – erase all – import back".

Labels: CSV, python, unicode, windows

Procedures to mount Windows shares from remote machines changed so many times during lifespan of Linux, that an attempt to do a search in Google gives a mix of solutions from using "smbmount" to "mount -t smb".

Here is the latest and greates solution. Make sure you have "smbfs" package installed and do this :

% sudo mount -t cifs //192.168.1.102/share_name /media/my_share -o \
  username=theuser,password=thepass,iocharset=utf8,file_mode=0777,dir_mode=0777

With option "iocharset=utf8" it ought to handle international characters in all Windows NT-based systems. If you need to mount drives from Windows 95/98/Me, different translation options will be necessary.

see this page for more detailed explanations.

Labels: linux, Samba, Ubuntu, unicode, web-services, windows

Microsoft Windows and other Microsoft utilities, like Microsoft Office, use encoding "UTF-16LE" by default; if they offer you multiple choices of encoding, they call it simply "Unicode". If the goal is to generate Unicode files which could be opened by all Microsoft applications, these better be in UTF-16LE.

Multiple language and libraries offer built-in conversion to UTF-16LE; however, one must be aware of two potential problems with that: (1) standard 4-byte header that Windows expects (and writes on output), and (2) potential problem with built-in DOS line ending mode ("text mode"); files must be written in "binary" mode.

Proper way to create UTF-16LE file in Python would be this:

fh = open ( "Test.txt", "wb" )
fh.write ( "\xff\xfe")
fh.write ( u"Проверка\r\n".encode("UTF-16LE" ) )
fh.close()

Labels: python, unicode, windows

I am trying to get comfortable using Ruby (having already written a fair amount of code in Perl, Python, and PHP). It is apparent that one of the serious problems with Ruby is lack of support for Unicode (in the current 1.8.x Ruby version). Class String in Ruby is just a byte string and nothing else.

A few things of interest:

1. Standard Ruby distribution does support Iconv library. Fox example, to convert a string of text from UTF-16LE (used by Windows) to UTF8, one can use:

   require 'iconv'
   utf8_str = Iconv.iconv( "UTF-8", "UTF-16LE", win_str ).join

   (obviously, this is merely a wrapper around "standard" GNU iconv library. To see full list of currently installed encodings, use `iconv -l')
2. Also, there is a special option 'U' to standard String 'unpack' function, which effectively treats string as having UTF-8 encoding and then splits it into array of Unicode integers;
3. Using the above, there is an attempt made on rubyforge to create well-behaved Unicode String class purely in Ruby (the only way to install above seems to be to download the only source file directly). However, this project was obviously not finished and is barely used, given that last modification occurred 18 months ago and there seems to be no activity on project public forums. While many features of "regular" strings are supported and others could be added, it is not clear whether for example full regular expression support is feasible;
4. Interestingly, Ruby, being originally developed in Japan, was designed to deal with non-ascii encodings from its day one. In particular, Ruby supports the notion of "default encoding", which corresponds to "global variable" $KCODE . It can be set with command line option -K, e.g. `-Ku' for UTF-8. This variable can be assigned to at any moment to overwrite default value. The only trouble is, in my (Cygwin) Ruby and iconv installation, actual value of $KCODE which corresponds to utf8 is "UTF8", whereas iconv only has "UTF-8" (note the dash) and refuses to understand "UTF8". Assigning explicitly `$KCODE = "UTF-8"' does not help, as Ruby still resets $KCODE to "UTF8". It means that to use Ruby unicode library mentioned above I had to make changes to have it pass to Iconv "UTF-8" whenever previously it wanted to pass "UTF8";
5. In this e-mail thread, someone explains why he thinks Ruby is fine without any (more) Unicode support;
6. Here some Unicode-related changes planned for Ruby 2.0 are described;
7. However, here is the detailed description of what has already been done in Ruby 1.9, with an understanding that 2.0 will be a subset of 1.9; I cannot see Unicode ever mentioned there;

In a nutshell, all of the above means that at least before Ruby 2.0 is released, the best option to deal with Unicode strings is to make all String's (=byte arrays) used share same Unicode-compatible encoding, e.g. UTF-16LE . Then, we will need to always keep that in mind while dealing with such strings. Fox example, to split TAB-separated string, we would use:

words = line.split( "\t\0" )

Appendix 1. Some reference material on Ruby:

1. Two similarly brief introductions to Ruby: Programming Ruby: The Pragmatic Programmer's Guide and Ruby User's Guide (by language creator);
2. As part of Pragmatic Guide mentioned above there is a library reference and built-in reference; also, seemingly same documents are also available from here; I am not why if there is any difference or not;
3. The Ruby Language FAQ;
4. Ruby Cheatsheet;
5. Ruby-doc.org server features, among other things, standard Ruby documentation and customized Google search;
6. Ruby QuickRef;
7. Book Programming Ruby (2-nd edition) is not available for free (you can buy it as a regular book or download PDF file for $25), but number of useful excerpts are available, and judging from these excerpts book appears to be well-done and useful;
8. Server rubylearning.com offers some on-line tutorial, as well as (huge) PDF file download (link is only provided via e-mail, but see this for book and this for accompanying Ruby programs)
9. One more PDF book from the Scribd collection.

Appendix 2. On main Ruby web site, there is a special section Ruby From Other Languages. Especially interesting to us are possible confusion and incompatibilities compared to Python and Perl. A few things, however, are missing from the list of incompatibilities.

1. Ranges in Ruby are inclusive while in Python they are not; that is, in Python `range(2,10)' consists of all numbers from 2 to 9 while in Ruby similar n spirit notation `2..10' is used for all numbers from 2 to 10 inclusively. As a result, to, for example, drop the last character from a string in Python, we use `str[:-1]' while in Ruby this translates to `str[0..-2]';
2. Including function from a file in Ruby does not imply any consequences in terms on namespace, whatsoever. To take advantage of namespaces, you must use `module' explicitly;
3. Apparently Ruby does not use popular Perl regular expression library, so its regular expression syntax, while very similar to that from Perl, might not be fully compatible. E.g., Ruby does not support look-behind zero-width assertions;
4. On a positive side, Ruby does implement Python-style '%' operator for parameters substitution, though it is not used anywhere in the standard documentation; instead, Ruby-own parameter substitution ("x = '#{my_var + 57}'") is used. I prefer Python style, besides, it allows for formatted output;

(Happy Ruby programming! :-)

UPD [18-Jun-07]. Slashdot reviews new book "Practical Ruby Gems"

Labels: ruby, unicode