Monday, April 28, 2008
Dealing with "native" Windows encoding
Microsoft Windows and other Microsoft utilities, like Microsoft Office, use encoding "UTF-16LE" by default; if they offer you multiple choices of encoding, they call it simply "Unicode". If the goal is to generate Unicode files which could be opened by all Microsoft applications, these better be in UTF-16LE.
Multiple language and libraries offer built-in conversion to UTF-16LE; however, one must be aware of two potential problems with that: (1) standard 4-byte header that Windows expects (and writes on output), and (2) potential problem with built-in DOS line ending mode ("text mode"); files must be written in "binary" mode.
Proper way to create UTF-16LE file in Python would be this:
fh = open ( "Test.txt", "wb" )
fh.write ( "\xff\xfe")
fh.write ( u"Проверка\r\n".encode("UTF-16LE" ) )
fh.close()
Labels: python, unicode, windows
Monday, April 21, 2008
My first bugzilla bug report
Just submitted my first ever Firefox bug report: 
bug 430140
"HTML source as simple as that:
<html>
<body style="direction: rtl;">
<img src="http://www.freeiconsweb.com/Icons/16x16_File_icons/Computer_File_001.gif"> First<br>
<img src="http://www.freeiconsweb.com/Icons/16x16_File_icons/Computer_File_001.gif"> Second<br>
</body>
</html>
(use any image you want) causes two lines to be inconsistent: in the 1-st line
image is on the right side of text (correct RTL behavior), but on the 2-nd line
it is on the left size (seems to be inconsistent with RTL requested) while line
itself is right-aligned.
Attempt to separate lines with techniques other than "<BR>" solves the problem
and restores consistent RTL behavior.
Even if I am wrong about correct behavior, there is no reason these two lines
should display inconsistently.
Note: while MSIE and Safari display this correctly, Opera mysteriously
(mis-)behaves consistently with Firefox.
Labels: firefox
Saturday, April 12, 2008
ImageMagick and RMagick
There is now a "standard" (free, BSD-style) image manipulation library ImageMagick. As it is pretty customary these days, it provides bindings for many popular higher-level languages, including Java, .NET, PHP, Perl, Python and Ruby. Ruby binding is known as RMagick.
One problem with ImageMagick is that it is notoriously difficult to install. The root of this problem may be that library insists on integration with X-server libraries (and indeed provides some X-client capabilities, though interestingly enough not supported on Windows).
(As a result, people are coming up with various ways to make a smaller library with restricted functionality, e.g. mini-magick. Unfortunately, there seems to be no documentation available, other than this tutorial, so I didn't want to spend time researching this)
Cygwin does include ImageMagic (you'll need at least 2 cygwin packages for ImageMagic: image-magick and libMagick-devel and various dependents for some reason not picked up by cygwin setup utility, e.g. libXft-devel and libbz2-devel), but all my attempts to first install ImageMagic and then RMagic failed (perhaps bundled version was too old, though judging from version numbers it was ok). ImageMagick web site includes pre-built version of all libraries for cygwin, but it appears to be very poorly built, includes many hardcoded paths from original build machine, etc.
So, the only option was to install ImageMagick from the sources, and it succeeded, after I explicitly excluded Perl bindings
./configure --without-perl
(be aware that built could take several hours to complete). After that, RMagick can be installed with command as simple as:
gem install rmagick
To try it out, you can use this program; try to uncomment line "canvas.display" near the end and look at the result. This didn't work well under Cygwin at all (though displaying same image just read from file worked fine).
Library itself is very powerful indeed. Documentation is good and detailed. One interesting thing that I discovered is that library has no less than 3 different methods to resize an image: "to sample", "to scale", and "to resize". You can compare all three by picking some random picture (sufficiently large) and running on it program like that:
#! /usr/local/bin/ruby -w
require 'rubygems'
require 'RMagick'
include Magick
imgfile = ARGV[0]
img = (Image.read imgfile)[0]
factor = 0.25
osize = File.size imgfile
puts "Original file #{imgfile} : #{osize} bytes, factor=#{factor}"
[:sample, :resize, :scale].each { |resize_alg|
t = Time.now
img_resized = img.method(resize_alg).call factor
dt = Time.now - t
fname = "test_#{resize_alg}.jpg"
img_resized.write( fname );
size = File.size fname
puts "#{sprintf("%-6s",resize_alg)}: #{dt.to_f} secs, #{size} bytes =\
#{sprintf("%.2f",100*size.to_f/osize)}%"
}
On my machine, it generated this output:
sample: 0.012 secs, 258985 bytes= 11.08%
resize: 1.287 secs, 227424 bytes= 9.73%
scale : 0.469 secs, 222147 bytes= 9.51%
Which is mathematically interesting: apparently, "resize" and "scale" provide some smoothing, therefore reducing size of the compressed image; "sample" is by far the fastest way to resize, but by just picking some more or less random pixels from the original image it generated result which is not only visually worse, but is harder to compress.
Friday, April 04, 2008
Global $PATH with Debian
Let's say I want to add new directory to "global" system $PATH. That is, I want all applications, no matter how started, from the command line or from GUI, no matter by what user, is to be able to find executables in certain directory, among others like "/usr" or "/usr/local/bin".
Interestingly, there is no simple or at least standard way of accomplishing this simple task.
On a very basic level, there are two main "sources" of the environment
1. Shell initialization file, system-wide or per-user. For "bash", these are:
/etc/bash.bashrc
/etc/profile
~/.bashrc
~/.profile
~/.bash_profile
It is important that all files from this list which have "profile" in their names are only invoked by "login" shells; that is, only for shells which are invoked when user logins remotely to the system or for shells started with "--login" (or just "-l") command line options.
2. Various "display managers", like GDM = "Gnome Display Manager".
Now, here is a tricky part: I could not figure it out how GDM sets it environments. Environment itself which comes out from GDM is actually identical to that set up in file /etc/login.defs, but changing said file appears to have no effect.
However, there is another file which GDM appears to honor: /etc/environment (note that this is NOT a bash script, but a simple configuration file). And yes, you can add PATH definition there and GDM will use it with no interference.
The only problem is that GNOME (and all other modern desktops) have a feature "run as root", and that requires different environment (again, I believe default one is identical to one defined in /etc/login.defs).
Given this situation, there are two basic strategies to define your system $PATH. You can of course customize both to fit your needs.
Approach 1 is described here.
The basic idea is that if there is a local user-defined executable file ~/.xsession, GDM will invoke it when firing up default user session. Therefore, this ~/.xsession file
#! /bin/bash --login
exec x-session-manager
makes sure windows environment is consistent with login environment.
Of course, you can change header to simply "#! /bin/sh" and set any environment you want in the same file before invoking x-session-manager.
Approach 2
Set your global $PATH in /etc/environment, but then make sure it is good both for regular login and applications requiring super-user privileges. Here is mine:
PATH=/usr/local/bin:/opt/jdk/bin:/usr/bin:/bin:/usr/bin/X11:/usr/games:/sbin:/usr/sbin
You can and probably should customize your bash configuration files to set local $PATH for user (by default, file ~/.bash_profile prepends "~/bin" to $PATH)
![http://www.freeiconsweb.com/Icons/16x16_File_icons/Computer_File_001.gif"](http://www.freeiconsweb.com/Icons/16x16_File_icons/Computer_File_001.gif"){kind=link}