My Tech Notes

Why use href="#"?

2011-11-24T17:12:00.001-05:00

Yes, I finally figured why people so often use construct

<A href="#" onclick=do something...

whereas I have always been using

<A href="javascript:foo()" ...

Let's look at this test file:

<html>
<script>
function foo(elm) {
  if (elm == window)
    alert('received Window!');
  else
    alert("received <" + elm.tagName + ">");
}
</script>
<A href="javascript:foo(this)">Click 1</A><br />
<A href="#" onclick="foo(this); return false">Click 2</A>
</html>

Second click works as "expected," passing relevant "<A>" node into Javascript function, but first one does not; instead; 'this' is resolved to it's "default" value as current document "window". Also, this behavior seems to be consistent across all major browsers.

MPM

2011-02-17T20:52:00.007-05:00

In my mod-python/apache based server implementation at some point I generate certain amount of data (from MySQL database) used in processing user input. This generation takes a while (~10 secs on my development server, ~30 secs on production VPS-based server) and is virtually unchanged from one invocation to the next. Why can't I save it in a disk cache and use it instead? OK, I tried just that.

You can imagine my surprise when I realized that reading from cache took exactly same time as original data generation! How is it even possible?

It turned out that the reason for this phenomenon was that my data consisted mostly of complied Python RE's (regular expressions). However, Python's underlying serialization engine ("pickle") cannot save compiled RE's; when you try to, on the pickle.load() call these RE's will be automatically (re-)compiled. This is what I experienced: most of the time which I initially attributed to DB access and processing results was in fact spent compiling RE's. When I switched to file cache, I gained nothing: same RE's were recompiled again on cache load.

Why such limitation? No one is really sure, however, this is exactly how things are at the moment:

Q: Is it possible to store these regular expressions in a cache on disk in a pre-compiled manner to avoid having to execute the regex compilations on each import?
A: Not easily. You'd have to write a custom serializer that hooks into the C sre implementation of the Python regex engine. Any performance benefits would be vastly outweighed by the time and effort required.

So what should or could I do to optimize my server? It is really annoying to have to wait half a minute every time for a server to generate the same data over and over again.

Looking for an answer to this question, I turned to mod-pyton documentation, section on session management. It appears that mod-python supports three kinds of section management engines: MemorySession, DbmSession, FileSession; naturally, MemorySession implements persistent storage in memory, whereas DbmSession and FileSession are essentially different ways to provide disk-based caching.

Now, there is little doubt that said caching will use internally Python's standard pickle engine, which will take me back exactly to square one.

Can I use MemorySession? mod-python "Session" implementation can make a determination which session engine to use. Documentation has this to say regarding when MemorySession is chosen:

If session type option is not found, the function queries the MPM and based on that returns either a new instance of DbmSession or MemorySession. MemorySession will be used if the MPM is threaded and not forked (such is the case on Windows), or if it threaded, forked, but only one process is allowed (the worker MPM can be configured to run this way). In all other cases DbmSession is used.

What is it talking about, and what is "MPM"? It took me a while to figure that out. MPM in fact stands for Multi-Processing Module and it has to do with how apache distributes incoming requests. Borrowing an excellent explanation from here,

Apache can operate in a number of different modes dependent on the platform being used and the way in which it is configured. This ranges from multiple processes being used with only one request being handled at a time within each process, to one or more processes being used with concurrent requests being handled in distinct threads executing within the same process or distinct processes.
The UNIX "prefork" Mode
This mode is the most commonly used. It was the only mode of operation available in Apache 1.3 and is still the default mode on UNIX systems in Apache 2.0 and 2.2. In this setup, the main Apache process will at startup create multiple child processes. When a request is received by the parent process, it will be handed off to one of the child processes to be handled.
The UNIX "worker" Mode
The "worker" mode is similar to "prefork" mode except that within each child process there will exist a number of worker threads. Instead of a request being handed off to the next available child process with the handling of the request being the only thing the child process is doing, the request will be handed off to a specific worker thread within a child process with other worker threads in the same child process potentially handling other requests at the same time.

You can find out which mode is used by these function calls (from mod-python request handler)

threaded = apache.mpm_query(apache.AP_MPMQ_IS_THREADED)
forked = apache.mpm_query(apache.AP_MPMQ_IS_FORKED)

The last question is, how to switch from one mode to another? In debian, simply install one of apache2-mpm-XXX packages:

debian-linux% apt-cache search ^apache2-mpm
apache2-mpm-itk - multiuser MPM for Apache 2.2
apache2-mpm-event - Apache HTTP Server - event driven model
apache2-mpm-prefork - Apache HTTP Server - traditional non-threaded model
apache2-mpm-worker - Apache HTTP Server - high speed threaded model

I am not sure what apache2-mpm-event is, but apache2-mpm-prefork and apache2-mpm-worker, when installed, will automatically uninstall (!) the other one and automatically make changes to /etc/apache2/apache2.conf to turn on respective mode, like that:

# worker MPM
# StartServers: initial number of server processes to start
# MaxClients: maximum number of simultaneous client connections
# MinSpareThreads: minimum number of worker threads which are kept spare
# MaxSpareThreads: maximum number of worker threads which are kept spare
# ThreadsPerChild: constant number of worker threads in each server process
# MaxRequestsPerChild: maximum number of requests a server process serves
<IfModule mpm_worker_module>
    StartServers          2
    MaxClients          150
    MinSpareThreads      25
    MaxSpareThreads      75
    ThreadsPerChild      25
    MaxRequestsPerChild   0
</IfModule>

All that remains is to add somewhere to apache configuration

PythonOption mod_python.session.session_type MemorySession

... and verify it in Python code

if req.get_options().get('mod_python.session.session_type') == MemorySession and \
   apache.mpm_query(apache.AP_MPMQ_IS_THREADED) != 0

   # using persistent storage
   session = Session.Session(req)
   ......................

And that's it - the processing which used to take half a minute now done in a fraction of a second!

Time delta in Python

2011-02-17T07:04:00.003-05:00

Obviously, Python is a very nice and intuitive language (some subtle problems like described here notwithstanding), but one thing which always confused me immensely is time management in Python, for example relations between datetime module, time module and date object.

By way of example, if you want to measure time delta between two events, there are two ways to do it. Run an interactive Python session and try it yourself:

Unix% python
Python 2.5.2 (r252:60911, Jan 24 2010, 14:53:14) 
[GCC 4.3.2] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import time, datetime
>>> x = time.time(); y = datetime.datetime.now()
>>> time.time() - x;  (datetime.datetime.now() - y).seconds + (datetime.datetime.now() - y).microseconds/1000000.0
11.006932973861694
11.007103000000001

(In Python 2.7 they at least added method timedelta.total_seconds(), making it slightly more consistent).

XONIX, now available in Google Web application store

2011-02-14T20:51:00.006-05:00

Browser support: All major browsers except IE7 and earlier (interestingly, IE7 seems to have "canvas" element support, but it does not work at all as expected - not sure how this is even possible). IE8 and Firefox 3.6 work fine, but somewhat slower than Chrome, Safari, and Opera. Latest Chrome and Opera support the game perfectly.

Slow performance: There should be no problem playing this game on a modern computer in supported browser even in a full-screen mode; however, your mileage may vary, especially with older browsers or on poor hardware. In this case, you can modify default setting to decrease number of "ticks" and "redraw time", or make dimensions smaller (especially if you have a large screen). By playing with these parameters, I successfully made it work (even if not perfectly) on every system I tried so far.

Parsing Windows-encoded CSV file, again

2009-12-21T15:56:00.001-05:00

This post continues a topic "how to deal with UTF-16-LE encoding", see this post.

If writing UTF-16-LE is no picnic, reading is even more challenging, unless you want to read the whole file as one line. If you prefer or are forced to use per-line approach, then you'd better be aware that properly encoded UTF-16-LE file includes zero byte '\0' after the end-of-line symbol, that is, every line ends up with four bytes 0x0d000a00, or if you will '\r\x00\n\x00'. However, when reading line-by-line, processing stops at '\n' and following '\x00' is interpreted as belonging to the next line!

To rectify this problem, you can use this Python-based "generator" to read UTF-16-LE-encoded file lile-by-line as unicode:

def readiterator(file) :
    fh = open ( file, "rb" )
    for line in fh :
        if line == '\x00' : continue
        if line[:2] == '\xff\xfe' :
            line = line[2:] + "\x00"
        else :
            line = line[1:] + "\x00"
        res = unicode ( line, "utf_16_le" )
        yield res.encode ( "utf-8" )
    fh.close ()

One typical application of that would be parsing CSV file (e.g., reasult of Excel CSV export) using Python built-in "csv" module, which has no knowledge of encoding, though luckily does support unicode input:

cvsreader = csv.reader(readiterator(input_csv_file))

If you decide to (for example) make changes to the table and save it again as CSV file, you'll quickly discover that you can't similarly use csv.write() directly, since it does not work with unicode strings, at all. You will have to play a trick taken directly from official Python documentation, to first convert to UTF-8 and dump CSV to a temporary string , and then read this string and convert back to unicode. Here one way to do that:

class MyCSVWriter:
    def __init__ (self,file_writer) :
        self.stream = file_writer
        self.queue = StringIO.StringIO ()
        self.writer = csv.writer(self.queue)

    def writerow (self,row) :
        self.writer.writerow([s.encode("utf-8") for s in row])
        self.stream.write(unicode(self.queue.getvalue(),"utf-8"))
        self.queue.truncate(0)

    def close(self) :
        self.stream.close ()

Of course, you still need a backend to dump unicode data as UTF-16-LE-encoded file:

class MyFileWriter:
    def __init__ ( self, file ) :
        self.fh = open ( file, "wb" )
        self.lineno = 0
            
    def write (self, line) :
        if self.lineno == 0 :
            self.fh.write ( '\xff\xfe' )
        self.lineno += 1
        self.fh.write ( line.encode ( "utf_16_le" ) )

    def close (self) :
        self.fh.close()

These two classes finally make it possible to create a CSV "writer" which can be used to write data just retrieved by aforementioned "reader"

csvwriter = MyCSVWriter(MyFileWriter(output_csv_file))

All of these code snippets are taken from an utility parsegab.py which I write to make some very specific changes to Google Address Book, using workflow "export – fix – erase all – import back".

GSM call forwarding

2009-11-04T09:14:00.001-05:00

Forward All Calls - Activate - Cancel & De-register - Cancel & Retain - Status - Re-establish	21[Phone Number]# ##21# #21# #21# 21#
Forward if Busy - Activate - Cancel & De-register - Cancel & Retain - Status - Re-establish	67[Phone Number]# ##67# #67# #67# 67#
Forward if Not Answered - Activate - Cancel & De-register - Cancel & Retain - Status - Re-establish	61[Phone Number]# ##61# #61# #61# 61#
Forward if Out of Reach - Activate - Cancel & De-register - Cancel & Retain - Status - Re-establish	62[Phone Number]# ##62# #62# #62# 62#
Simultaneously perform ALL FOUR forwards: - Activate - Cancel & De-register - Cancel & Retain - Re-establish	002[Phone Number]# ##002# #002# *002#
Simultaneously perform ALL Conditional Forwards: - Activate - Cancel & De-register - Cancel & Retain - Re-establish	004[Phone Number]# ##004# #004# *004#

Source

Simple shell implementation

2009-10-22T18:07:00.001-04:00

Just done a simple exercise in C programming and wrote a trivial implementation of Unix-style "shell" utility in C.

The source is here: http://code.google.com/p/inet-lab/source/browse/trunk/utils/Misc/myshell.c

All it does is basically accepts a user's command, parses it into "utility" and "arguments", does a basic sanity check, finds utility in $PATH, forks a child process and executes an utility with arguments via execv().

It also optionally supports GNU readline library and could be compiled e.g. with gcc like that :

gcc  -g --pedantic –Wall [-DRL] -o myshell myshell.c [-lreadline]

Double quotes can be used as part of command line to pass a single argument with spaces. Other than that, no other shell substitution or expansion is done; in particular, there is no way to escape a double quote to pass it as part of an argument.

Why would anyone need this primitive shell?

Well, other that a simple training in a few basic C-programming concepts (also serving as a basic GNU readline example), it allows one to create a "back door" bypassing system security. Indeed, if such a "shell" is owned by root and is granted setuid privilege, it allows a regular user to execute any administrative/privileged command, a goal which cannot be accomplished with either regular system shells (which all have built-in protection against setuid flag) or any tools written in a scripting language; it must be a native system executable.

Auto-generating JavaScript code

2009-09-21T16:19:00.001-04:00

Languages which are trying to be smarter make it harder to auto-generate code.

Indeed, I wrote a small piece of code to output JavaScript Array, something along these lines (Python):

write ("var myarr=Array(" + ",".join([str(o.id) for  o in objects]) + "); \n")

Of course, it was working just fine till I encountered a case where length of incoming Python list objects is one, where it immediately broke, because meaning of JavaScript initialization

var myarr=Array(10);

is not at all what the above piece of code was silently expecting.

Potential problems with mutable default arguments in Python

2009-08-25T22:51:00.001-04:00

Just spent about half an hour tracking down a mysterious problem in one of my web servers implemented in Python, where same page occasionally was showing obviously duplicate info in some table cells while loading correctly on other invocations.

The bug was, of course, a rather stupid one. I was using my own implementation of a container and wanted to initialize it with a regular array, like that:

class Container :
    def __init__ (self, initial_arr) :
        self.content = initial_arr
.........................................
a = Container( ["one","two","three"] )

Of course, this was already problematic enough, since I was initializing my container with (original) reference and not a copy, but provided that a consumer of this class is wary of this behavior, this is not incorrect per se.

The real problem emerges when, naturally, I wanted to provide a default initialization to an empty container, and did it like that:

class Container :
    def __init__ (self, initial_arr=[]) :
        self.content = initial_arr

So that I could write

a = Container()

With this "improvement", this is no longer just a poor style, but a clear bug: new instance of class is initialized with a reference to default argument, and if class is modified, so is default argument, in effect carrying over changed to the next, unrelated, instance.

No wonder I was seeing duplicated cells returned by server…

Here is the complete script which illustrates the problem:

#! /usr/bin/env python

class Container :
    def __init__ (self, initial_arr=[]) :
        self.content = initial_arr
    def __repr__(self) :
        return repr(self.content)
    def append(self,elm) :
        self.content.append(elm)

def foo (use_default) :
    if use_default :
        a = Container ()
    else :
        a = Container ( ["one","two","three"] )
    a.append ( "four" )
    print "a = %r" % a

print "\nUsing explicit argument"
foo (use_default=False)
foo (use_default=False)

print "\nUsing default argument"
foo (use_default=True)
foo (use_default=True)

This script generates the following output:

Using explicit argument
a = ['one', 'two', 'three', 'four']
a = ['one', 'two', 'three', 'four']

Using default argument
a = ['four']
a = ['four', 'four']

Ideal solution would have been to force default argument (if mutable) to be read-only. Unfortunately, Python does not support read-only values.

Lacking that, the only option available is to never use mutable values as default arguments. For example, above script could have been written like that:

class Container :
    def __init__ (self, initial_arr=None) :
        if initial_arr is None : initial_arr = []
        self.content = initial_arr
.........................................

With this simple update, class behaves "as intended" (whether this is a good behavior to begin with is another question entirely).

Update (14-Sep-09). One more manifestation of the same problem is initializing class members to mutable values outside of method __init__(). For example, this code

class Foo :
    m_x = []              # DANGEROUS!!!
    def __init__ (self) :
        self.m_y = []     # This is much better!
    def append(self,obj) :
        self.m_x.append (obj)
        self.m_y.append (obj)
    def __repr__ (self) :
        return "Foo(m_x=%r,m_y=%r)" % (self.m_x,self.m_y)

v = Foo ()
v.append ("A")
w = Foo ()
w.append ("B")
print "v = %r, w = %r" %(v,w)

Generates output

v = Foo(m_x=['A', 'B'],m_y=['A']), w = Foo(m_x=['A', 'B'],m_y=['B'])

Flash/SSD drive comparison

2009-07-23T13:12:00.002-04:00

This is a totally unscientific comparison of two drives, one is a Solid State Drive which works both as internal or external USB/SATA, and a traditional "keychain" flash drive, both have the same capacity of 32GB and formatted as FAT32.

Name	bought	for	Read	Write
Cavalry Pelican 32GB ssd	Dec'08	$80	28.42	21.81
Kingston 32GB DataTraveler 120 USB Flash Drive	July'09	$62	16.26	11.93

Read/Write speed is given in MB/s. 1 MB = 1048576 bytes.

Conclusion:

SSD is roughly 75% faster;
In both instances, reading speed is about 33% faster than writing speed.

Inline JavaScript

2009-07-22T16:27:00.002-04:00

I write a lot of Web Interfaces in Python. Occasionally generated Web pages are using moderately complicated JavaScript code. In this situation, you can do one of two things:

Insert JavaScript code in Python source as a (multi-line) string;
Use a separate JavaScript file.

First approach is a good one when there are relatively few lines of JavaScript code, while second approach makes sense if this stand-alone file could be re-used.

However, neither approach works if JavaScript logic is complicated but current page-specific.

A possible alternative is to keep JavaScript in a separate source file in the depository, but embed a copy of it into python files immediately before check-in.

This solution could be used directly, or combined with some JavaScript compression and/or obfuscation technology.

First, about compression and obfuscation.

I found it most productive to use combination of two separate tools: a popular compression tool JSMin (available in 10 different languages, I am using Python version) and ShrinkSafe, an obfuscation/compression tool written in Java.

Provided that you have SUN java available, js.jar (Rhino “JavaScript in pure Java” engine from Mozilla) is somewhere in you CLASSPATH, and jsmin.py is in PATH, the complete process looks like that:

java -jar <path>/shrinksafe.jar <original>.js | jsmin.py > <new>.js

Next step is to use a Python utility jsprocess.py which I wrote. It is reading a list of Python input files and is looking for all calls to the utility inline_js() (which must not exists other than version created by a previous run of jsprocess.py). For very call which looks like inline_js ("file_name.js"), e.g.

fh.write ( inline_js("docform.js") )

it will append to the end of the file definition of inline_js() which embeds compressed/obfuscated copy of "docform.js". If function inline_js() is already defined (anywhere), it will be replaced.

UbuntuOne

2009-07-14T17:18:00.001-04:00

My number finally came up, I was able to register for new revolutionary file sharing service by Ubuntu Team called UbuntuOne. The service is still in closed-beta; see some reviews here, here and here.

(There isn’t much information (currently) on the official site, other than installation instructions, mailing list and bugs database)

This is a potentially brilliant idea, because it gives people a convenient way to support financially their favorite distribution, and with solid financial support this could become a reliable backup solution. Correct, there may be cheaper options available on the market, but you never know how long they’ll last (Streamload being a good example of that)

On the other hand, as of right now the service is hardly usable.

It is clearly inferior compared to the popular (among Linux crowd) DropBox service, which provided 5 times more of paid storage, supports versions, has be better UI (web-based and client-side) and is compatible with every modern Linux distributions, Windows and Mac;
It is still very buggy and incomplete. For example, when I added a file through Web UI, it never showed up in my Ubuntu folder, though synchronization the other way around worked fine;
There are also browser-compatibility issues, e.g. with Google Chrome;
Is this service supposed to be open source? I could not find any reference to the server source code anywhere, nor to any plans for future client-side API.

Mailing list appears to be relatively low-volume, and I am not sure many people are actually using this – which, given the modest, at best, progress of development and plenty of competitors, does not promise a bright future for this project any time soon, in my opinion.

But, whatever this is going to turn out, the idea was really a good one.

Skype Web Services

2009-07-13T09:44:00.001-04:00

Apparently, if you installed latest Skype 4.1, it is advisable to go to Control Panel => Add or Remove Programs and remove something which is called "Skype Web Services".

This will save you from some unnecessary authorization requests, communication failure messages, and slow down of Internet Explorer, and should also clean up a mysterious "skypenames.exe" executable.

For some background info, see this thread.

Mounting windows shares

2009-05-31T15:32:00.002-04:00

Procedures to mount Windows shares from remote machines changed so many times during lifespan of Linux, that an attempt to do a search in Google gives a mix of solutions from using "smbmount" to "mount -t smb".

Here is the latest and greates solution. Make sure you have "smbfs" package installed and do this :

% sudo mount -t cifs //192.168.1.102/share_name /media/my_share -o \
  username=theuser,password=thepass,iocharset=utf8,file_mode=0777,dir_mode=0777

With option "iocharset=utf8" it ought to handle international characters in all Windows NT-based systems. If you need to mount drives from Windows 95/98/Me, different translation options will be necessary.

see this page for more detailed explanations.

Magic of C preprocessor

2009-05-09T16:03:00.004-04:00

Curtis Krauskopf discusses a neat trick which forces C preprocessor to merge __FILE__ and __LINE__ as one string, which isn't straightforward since __FILE__ is a string but __LINE__ is an integer.

It turns out you can solve the problem with two additional macros:

#include <stdio.h>
#define STRINGIFY(x) #x
#define TOSTRING(x) STRINGIFY(x)
#define AT __FILE__ ":" TOSTRING(__LINE__)
void error(const char *location, const char *msg)
{
  printf("Error at %s: %s\n", location, msg);
}
int main(int , char**)
{
  error(AT, "fake error");
  return 0;
}

Now, if only someone could find a way to merge __func__ and __LINE__ ....

WebDAV, problems all around

2009-04-26T22:42:00.005-04:00

It seems almost incomprehensible how many problems emerge with simple WebDAV file hosting. While apache DAV server, and clients built-in into both Windows Shell and GNOME have been available fr many years, I still can't make it to work properly.

Here is a brief list of problems:

Apache 2 Server. Its problem is that it is bit difficult to make server work with both "resular" and SSL modes. This typically emerges as failure to "copy", "move" or "rename" file in non-SSL mode, while everything else seems to work.

Good explanation is available here:

The main issue was that the production system runs on https and there was a reverse proxy setup for this system. So, all the "https" requests were converted to "http" at this proxy level and forwarded to the main system. This was the main culprit.

Here is an example of HTTP request for the MOVE resource request for a WebDAV resource. For brevity I removed all unnecessary details.

MOVE contentLocation //request line, some https location, URI of webDAV resource
Destination:destinationLocation // this is the HTTP request header, should be absolute URI according to specifications.

Overwrite: "F" // this is also a HTTP request header

So, when the reverse proxy sees the request line, it knows that it has to convert this to HTTP request but the header Destination also contains an HTTPS request which would be ignored by the proxy. So, when the request reached the server, we are moving the resource from an URI which begins with a http to a URI which begins with an https. Server treats this request as a request to move a WebDAV resource from one server location to another server location.(Refer RFC: http://greenbytes.de/tech/webdav/rfc2518.html#METHOD_MOVE). This was the source of the main problem.

Linux/GNOME GVFS driver: Seems to work most of the time,but

Fails to handle paths with user name "davs://username@server.com/path". "Top listing" is shown correctly, but an attempt to change directory fails. Reason unknown;
When using drag-and-drop, or "cut/paste" interface, it attempts to "copy" file and then "delete" the original. I could not find any way to invoke "MOVE" command.

Windows (old driver). This is "old" Windows-2000 implementation, which identifies itself as "Microsoft Data Access Internet Publishing Provider DAV". This seems almost the most reliable implementation, and it worked for me for a while, but now it shows directory "Temp" as "temp" (all other directories similarly named are OK), and obviously fails to chdir to it. Reason unknown.

Windows (new driver, XP and newer). Identifies itself as "Microsoft-WebDAV-MiniRedir/5.2.3790". Apparently only intended to work with Microsoft IIS, not compatible with Apache. This pages gives a consice overview:

For implementation of WebDAV on Windows XP and later , MSFT made it's own interpretation of the standard to work best with the Windows IIS servers. The problem due to this is three fold:

Windows XP authenticates users using the format "domain\username" by the mechanism of "Microsoft-WebDAV-MiniRedir/5.1.2600". Whereas Windows 98SE/2000 authenticates users as "username" using the mechanism of "Microsoft Data Access Internet Publishing Provider DAV 1.1".
The problem lies with the implementation of "Microsoft-WebDAV-MiniRedir/5.1.2600". If authentication is sent as "domain\username" then it would be received as "usernamedomain" or "usernamehostname" by the Web server and not as "username".
Also as per "Microsoft Knowledge Base, Article ID: 841215" Windows XP disables "Basic Auth" in its "Microsoft-WebDAV-MiniRedir/5.1.2600" mechanism by default for security reasons. But WebDAV expects "Basic Auth".

There are hundreds pages how to trick it to invoke "old" implementation, including some on the same page mentioned above; there is also a separate discussion regarding Windows Vista, where "old" implementation has to be separately installed. Windows 7 status is unknown.

Platform-independent clients.

SkunkDAV Java-based client is simple and reliable, but unfortunately I could not make it support SSL-based access;
Cadaver is CLI-based tool. It seems to work fine.

Update (12-May-09). "Official" subversion book has a list of DAV clients. Among some known clients, some of them mentioned above, there is another Jaba-based client "DAV Explorer", it has been last updated in 2005 and looks a bit better than SkunkDAV, though I can't say there is a big difference.

Also, its help file says that in order to enable SSL in Java one has to run Java with -Dssl=true, and only from version 1.4 on; for earlier versions, one has to download special Java Secure Socket Extensions. It is likely with if run with these options, SkunkDAV will work ok with SSL DAV.

iPhone: jailbreaking and unlocking

2009-04-15T17:50:00.003-04:00

OK, since it seems everyone will have to do that sooner or later, I am going to describe here brifly my understanding of the process and what specifically I did to "jailbreak" and "unlock" my phone; whether or not it is applicable to anyone else's situation I cannot say.

First, let's clarify some terminology.

Jailbreaking is the process of "freeing" iPhone from (artificial) restrictions imposed by Apple, in terms of which software it is allowed to run. "Freed", or "Jailbroken" iPhone becomes similar to any other WiFi device, completely open to any changes, run under special version of MacOS. Jailbreaking can be applied to any device working under "iPhone OS", such as (currently) iPhone 2G (older), iPhone 3G (newer), or iPod Touch (any of them).

Unlocking is, generally speaking, a process of making a cellular phone work with any SIM card, as opposed to being "locked" to a particular carrier. Speaking of Apple devices, "unlocking" can apply to (any) iPhone, but not iPod Touch which is not a phone.

These two notions are totally independent.

There could be phones unlocked but not jailbroken. For examples, Apple cells such phones (officially) either for a premium, or because in certain foreign countries "locking" is illegal, or perhaps for some other practical reasons;
Apple never officially sells "jailbroken" phones; process of "jailbreaking", while probably legal, is 100% unofficial, can void any warranty you might have, could damage the phone (though this never happened to anyone) and must always be carried at your own risk;
There could be phones jailbroken but still locked. Someone might do that to be able to install some "unauthorized" software, while still using "official" authorized carrier under "official" plan, so there would not be any need for him to "unlock" the phone;
To unlock the phone by your own, you must jailbreak it first. At least, I havn't heard of any other way;
Reportedly, Apple's iTunes allows you to "restore" iPhone/iPod Touch to the "factory" condition no matter what you did to it, so in principle playing with jailbreaking and unlocking should be safe.

Another common source of confusion is when one needs to unlock the phone. Can I buy a phone from an online auction site or from a friend and still used it completely officially, without unlocking, if, let's say, I am already a Cingular customer?

The answer is, "it depends". For "newer" iPhone 3G, you probably can, though you might be forced to subscribe to additional services from AT&T like unlimited 3G internet, even if you don't want it.

For "older" iPhone's, this is a lot more involved. The reason is, Apple originally used a strange marketing maneuver when you were first buying a phone for $399 from any retailer or Internet, and then during activation were forced to subscribe to 2-year contract + additional services - or "unlock" it and use it any way you want. Later, with iPhone 3G Apple switched to more "standard" strategy when in order to buy iPhone you already had to subscribe (and commit), and thus any (new) iPhone no mаtter where from is considered "paid for".

Not so for iPhone 2, where AT&T has no obvious way of knowing if this specific phone has already been "paid for" by completing 2-year subscription or not, and obviously assumes the latter, forcing you to "commit", again. Reportedly, if you can bring a person from whom you got the phone and he could demonstrate that the phone has been fully paid for, AT&T representative might remove this commitment. But initially, you will need it anyway just to get behind iTunes "new service" page - there is no way around it. You can read some rather interesting discussion of the subject here.

OK, now going back to jailbreaking and unlocking. This is basically a 3-step process: Upgrading; Jailbreaking; Unlocking.

Upgrading. Make sure that the latest firmware version (2.2.1 at the time of this writing) has already been jailbroken. If not, you might find some tricks how to "upgrade" you phone to the version which is not the latest. The following however assumes that you are OK with upgrading the iPhone/iPod Touch to the latest firmware version from Apple.

Install the latest iTunes software from Apple site. Make sure to reboot the computer as instructed;
Attach iPhone to USB port, fire up iTines and see what it will tell you. It might want to upgrade right away, which is the best; but make sure first download upgrade, and only then install. After download, make sure there is a corresponding file *.ipsw in the diectory
```
%USERPROFILE%\Application Data\Apple Computer\iTunes\iPhone Software Updates
```
If your phone already has latest version, still try to force it somehow to "upgrade", there must be a way. Also, you can (optionally) "reset" the iPhone to remove any settings and files from a previous user. Whatever you do, make sure that you end up with *.ipsw file in aforementioned directory.
Exit from iTines and kill all relevant processes. This is important! (or else QuickPWN will be crashing during the installation)

Jailbreaking:

Download the latest version of QuickPWN and also files bl39.bin, bl46.bin , unzip QuickPWN archive and start the executable (no installation required)
QuickPWN will first detect your iPhone and will offer you to select *.ipsw file (see above). Make sure it is the right one and confirm.
On the next screen, select what you want to do: install Cydia (recommended and required for unlock), Installer (recommended), Replace logos (not recommended), Unlock (required)
You will have to identify locations of *.bin files mentioned above;
Next screen is tricky. You'll need to play some games with your phone in order to switch it to special "upgrade" mode: hold Home button, then power, then both, etc. Program screen will tell you exactly what to do - just follow very carefully!
At this moment program commences completely automated process of "Jailbreaking". You can relax and enjoy the show, just be patient and do not touch the phone (or the program) till it is all finished, and phone reboots into new "Jailbroken" mode.

Unlocking. For this to work, you must have an available WiFi network not separated by any NAT mapping or firewall from another computer (does not have to be Windows or Mac) you can use. The simplest approach is of course to use your home WiFi network.

Remove any SIM card from iPhone (note: this could be tricky). Do NOT insert a new one.
Make sure you can access Internet (e.g., via Safari browser) from the phone. Configure you WiFi using regular phone settings UI as required;
Start Cydia, configure it as appropriate, and use it to install OpenSSH;
Make sure you can access your iPhone from another computer through a SSH (with your favorite client), using it network IP address, user name "root", (default) password "alpine". Enjoy the beauty of accessing your phone with SSH command prompt! (Thanks to Unix-like kernel of MacOS, of course)

So far, we just made sure your iPhone performs well as SSH server, which could be useful to you in many ways; also, keep in mind, that it seems you could only use your phone this way till "power saving" feature kicks in, so either do everything really fast, or adjust power saving mode, or constantly touch the screen to keep it alive.

From this point on, we follow the instructions from here or here.
Download tiny Lockdown.zip file from here; unzip it to get a few files in a directory named "Lockdown";
Using your faivorite SSH-based file transfer program, replace iPhone directory /private/var/root/Library/Lockdown with new Lockdown directory you created in a previous step. It could be a good idea to backup "original" content of this directory someplace safe.
Shut down the phone, insert your SIM card, and boot it up. Everything should work.

Good luck!

Formatting new internal hard drive

2009-04-11T19:26:00.002-04:00

It is painfully apparent that latest software still lags behind latest hardware.

I had to add another internal drive to my Linux system, and though this is by far not the first time I had to do something like that, I keep forgetting the right sequence of commands, especially since these command do differ somewhat between Linix and BSD-based systems.

That's why following suggestion from somewhere I installed and tried to use new Ubuntu front-end utility for disk management, "GParted".

Indeed, this utility does conveniently show list of available disks, mounted or not, and allows you to create "disk label" (which is a first step before new file system cam be created). The only thing is, there are different "types" of "disk label", and default type "msdos" seems a bit strange, and there is no help or any explanation what other types are for, but still, this was indeed a correct default, in my case (but see below).

Unfortunately, the program only goes this far. An attempt to create actual file system hit me with error message "A partition cannot have a length of -1 sectors". Quick Internet search revealed the following:

This error only happens with disk sizes > 1TB (mine is 1.5TB);
No one has a clue as to why;
"msdos" disk label only supports disks < 2TB. There should be no problem with 1.5TB size though;
Nevertheless, some suggest to use disk label type GPT, which is supposed to handle any disks.

OK, I tried using GPT as disk label type, but it didn't make any difference.

Therefore, I followed advice from this blog post and use these commands to create filesystem:

sudo cfdisk /dev/sdc
sudo mke2fs -j /dev/sdc1
sudo vol_id /dev/sdc1

Some comments:

"cfdisk" is ncurses-based utility which is more or less equivalent to "GParted", but it does seem to work. However, it refuses to do anything if you already created GPT disk label, seems like it only works with "msdos". You can always use "GParted" to create another disk label.
Option "-j" ("journaling") essentially means "ext3" file system;
vol_id prints volume "ID" which can then be used in fstab (it is considered to be more stable than device names like "/dev/sdc1"). Entry in /etc/fstab could then look like that:
```
UUID=c49f9c07-1025-4e6b-988a-276ac67a1b6e /ext ext3 defaults 0 0
```
However, in order for "mount /ext" to succeed, there must be a sym link (automatically) created in /dev/disk/by-uuid/ , and apparently opinions differ as to what should trigger creation of new link after a new drive has been installed and formatted. For me, this command
```
sudo /etc/init.d/udev restart
```
worked.

So, at the end of the day, everything worked, but it appears that if/when next time I'll want to add a disk of 2TB or more, the real problems will begin.

Deep Zoom

2009-03-03T19:20:00.002-05:00

Following a question asked by a friend, I did a small research on the subject of using modern browsers capabilities to implement "photo zoom" feature, that is, make a "photo gallery" which allows visitors to "zoom in" into a picture to see specific areas of interest while not loading the whole image (potentially huge) in advance.

On a basic level, there are three relatively straightforward ways to do that.

(Let me say here right away that the fourth way, which might be better than all three, is to use Adobe Flash, but as far as I could tell there is no readily and freely available "player" which can do the job, thus one has to resort to either evaluating some commercial products or writing relevant code himself. At any rate, I didn't have neither time or inclination to research this any further)

Google Map API allows to use arbitrary images instead of actual maps, and it could be used to achieve desired effect. One commercial site using this approach is iFloor (just click on any image which has "click to zoom" markup and enjoy Google Map UI), and here author of this implementation posted complete instructions and all relevant source code (under open license), along with another "demo" of this technology in action.

I shall speak in more details on this approach later, let's just say for now that while it is not without its shortcomings, it is easy to use, portable and flexible.
Unlike Adobe, Microsoft decided to include in the latest release of its Silverlight browser plugin a special feature which they call "Deep Zoom", intended very specifically for this very purpose: web-based presentation of large images with "on demand" loading of detailed view on "zoom in" requests. An extremely nice showcase of this technology can be seen here.

There are two ways you cal play with it:

(a) Download special application "Deep Zoom Composer" (proprietary, but free to download and use) and follow tutorial.
Unfortunately, no matter how hard I tried I could not make it work: following any attempt to load an image, after processing it for a while application always reported that this was not a valid image; therefore, I had to give up on this approach;

(b) Alternatively, one can use free experimental site by Microsoft Labs to upload your images (you can upload many images in one operation) for similar on-line processing. Then, you can include presentation into your own web site using IFRAME.
Again, this all sounds good in theory, but in reality I could hardly make it work. About half of my images refused to be processed for unknown reason, and remaining few that I did upload demonstrated strange behaviors, such as view suddenly "darkening" (sometimes going totally black) on some intermediate zoom levels, etc.

Conclusion: like many other things from Microsoft, this looks promising and demo looks fantastic but implementation sucks big time, and since this is based entirely on proprietary technology I am powerless to do anything about that :(
Finally, the third option is really a variation of option number two. It would appear that the whole "Deep Zoom" technology came from an acquisition of software called "SeaDragon", which was not using any browser plugins but rather relied only on JavaScript, therefore providing a portable alternative to "Deep Zoom"-based solutions.

Since this technology is now owned by Microsoft, there is an easy integration between SeaDragon and "Deep Zoom", so whether you published your work in photozoom.mslivelabs.com or used Deep Zoom Composer, you can re-use your results with SeaDragon following these simple instructions.

The result might not look as nice and smooth as Silverlight-based solution (price to be paid for better portability), but it is basically the same thing underneath and so it shares all of its shortcomings as explained above.

So, that's it. As it is equally evident in many other situations, solution from Microsoft is nice when it works, but completely proprietary and basically useless (in my case, anyway), while Google approach, not perfect as it is, and perhaps not looking as "professional", is much more open (though not technically speaking an "open source" one) and much easier to tweak to do something close to what you want.

In the next post I'll provide more details on my experience with implementing "PhotoZoom" with Google map API.

ls color

2009-02-27T12:01:00.004-05:00

Modern Linux distributions are smart enough to support colored "ls" output and even enable that by default, but stupid enough not to adjust it in any way to current terminal color scheme. Default built-in colors "kind of work" for a black-on-white terminal, but should you try to make even small adjustment, some file types quickly become completely unreadable.

At the same time, procedure to customize these colors, while not complicated, is rather poorly documented.

Here are a few simple steps to follow;

Execute command "dircolors > ~/.dircolors";
Generated file "~/.dircolors" is self-documented, customize it to your liking;

Add these lines to your ~/.bashrc file or equivalent:

if [ -f ~/.dircolors ]; then
   eval $(dircolors ~/.dircolors)
fi

That's it!

P.S. You can also disable color output altogether by doing "unalias ls".

Debian 5 Lemmy

2009-02-26T09:18:00.003-05:00

Tried installing new Debian version, just released.

Looks fine, except that it does seem slow (but my impression might be subjective), and there are number of points to be taken into account when using as VMware guest under VMware 6.5 (the latest versions as of now):

You need to install Linux headers and "make" utility, like that:

# apt-get install make linux-headers-$(uname -r)

Some also recommend installing

# apt-get install build-essential psmisc automake

You need to use 4.1 compiler, and not "default" 4.3, so prior to running ./vmware-install.pl do this
```
# export CC=/usr/bin/gcc-4.1
```

Module vsync fails to compile with these messages:

Building modules, stage 2.
MODPOST 1 modules
WARNING: "VMCIDatagram_CreateHnd" [/tmp/vmware-config0/vsock-only/vsock.ko] undefined!
WARNING: "VMCIDatagram_DestroyHnd" [/tmp/vmware-config0/vsock-only/vsock.ko] undefined!
Building modules, stage 2.
MODPOST 1 modules
WARNING: "VMCIDatagram_CreateHnd" [/tmp/vmware-config0/vsock-only/vsock.ko] undefined!
WARNING: "VMCIDatagram_DestroyHnd" [/tmp/vmware-config0/vsock-only/vsock.ko] undefined!
WARNING: "VMCIEvent_Subscribe" [/tmp/vmware-config0/vsock-only/vsock.ko] undefined!
WARNING: "VMCI_DeviceGet" [/tmp/vmware-config0/vsock-only/vsock.ko] undefined!
WARNING: "VMCIMemcpyFromQueueV" [/tmp/vmware-config0/vsock-only/vsock.ko] undefined!
WARNING: "VMCIQueuePair_Detach" [/tmp/vmware-config0/vsock-only/vsock.ko] undefined!
WARNING: "VMCI_GetContextID" [/tmp/vmware-config0/vsock-only/vsock.ko] undefined!
WARNING: "VMCIDatagram_Send" [/tmp/vmware-config0/vsock-only/vsock.ko] undefined!
WARNING: "VMCIQueuePair_Alloc" [/tmp/vmware-config0/vsock-only/vsock.ko] undefined!
WARNING: "VMCIEvent_Unsubscribe" [/tmp/vmware-config0/vsock-only/vsock.ko] undefined!
WARNING: "VMCIMemcpyToQueueV" [/tmp/vmware-config0/vsock-only/vsock.ko] undefined!
CC /tmp/vmware-config0/vsock-only/vsock.mod.o
LD [M] /tmp/vmware-config0/vsock-only/vsock.ko
make[1]: Leaving directory `/usr/src/linux-headers-2.6.27-7-generic'
cp -f vsock.ko ./../vsock.o
make: Leaving directory `/tmp/vmware-config0/vsock-only'
Unable to make a vsock module that can be loaded in the running kernel:
insmod: error inserting '/tmp/vmware-config0/vsock.o': -1 Unknown symbol in module

Apparently, this is known problem which should not by itself cause any noticable problems. Relevant internet threads: [1], [2].

After performing standard installation, mouse pointer is not going to move between guest and host as it should. This can be fixed by adding one line "Option "CorePointer" to relevant portion of your /etc/X11/xorg.conf file, so it would look like that:

Section "InputDevice"
        Identifier      "VMware Mouse"
        Driver          "vmmouse"
        Option          "CorePointer"
        Option          "Device"                "/dev/input/mice"
        Option          "Emulate3Buttons"       "true"
        Option          "ZAxisMapping"          "4 5"
EndSection

(from here)

See also earlier post regarding installing VMware tools under Debian 4.

It seems like that few problems mentioned above notwithstanding, overall process has become simler, e.g. there is no longer any need to manually specify location of Linux headers (though you still must install them manually), mouse wheel works "out of the box", etc.

Mounting USB devices under Linux

2008-10-01T06:10:00.003-04:00

For some reason, Ubuntu/Gnome often fails to mount properly USB drives when plugged-in. It is not hard to mount them manually, as long as user has super-user access and remembers necessary commands and options.

To mount anything, one has to know name of corresponding "device" file in "/dev"; under modern Linux, these are typically "/dev/sdc1", "/dev/sdd1", etc.

For some peculiar reason, there is no single utility which would print simple map "Name/manufacturer of USB disk => device file". Rather, there are four utilities which together allow user to accomplish this task:

"lshw" is the only utility I know of which can print device names. To do that, run it like that:
```
sudo lshw -businfo | fgrep volume
```
"lsusb" can print information about all USB devices (a lot of information with "-v"), including enough hints to identify specific device. It could be used both in "super user" mode and "regular user" mode;
"fdisk -l" (super-user mode only) can print information about a device (if you have guessed its name).
Finally, when you have figured out device name and file system type (FAT or NTFS), you run mount in super-user mode like that:
```
mount -t [vfat|ntfs] <device name, e.g. /dev/sdc1> <empty directory, e.g. /media/mydisk> -o \
     rw,nosuid,nodev,uhelper=hal,shortname=mixed,uid=1000,utf8,umask=077,flush
```
Options can be adjusted, I believe above is what Gnome uses by default when mount is successful

[sudo] lsusb [-v] sudo fdisk -l /dev/sdc1 sudo lshw -businfo sudo mount -t [vfat|ntfs] /dev/sdc1 /lacie -o rw,nosuid,nodev,uhelper=hal,shortname=mixed,uid=1000,utf8,umask=077,flush

Samba installation under Ubuntu

2008-09-29T22:46:00.003-04:00

In one of the recent posts, I listed some basic steps to configure Samba server under Debian 4; when I tried recently to similarly activate Samba under Ubuntu 8.04, it appeared that due to consistent improvement of Linux distributions, some steps needed to be corrected in some minor ways. Here I list the most important changes for future reference:

Ubuntu is distributed by default with xinit as opposed to init (xinit has a "compatibility mode" which makes it to read and interpret "init" configuration file, but by default it is not created or updated since "init" is not installed). To enable Samba server under xinit, create file "samba" in directory "/etc/xinetd.d" with this (or similar) content (taking a hint from here):

service netbios-ssn
{
# "port" by default is taken from /etc/services
# port  = 139
 socket_type = stream
 wait  = no
 only_from = 192.168.0.0
 user  = root
 server  = /usr/sbin/smbd
 log_on_failure += HOST
 disable  = no
 mdns  = yes
 instances = 2
}

There is now option "passdb backend = tdbsam", which is very similar to using "smb passwd file", except that there is no need to use specific file; other than that, you can (and should) use smbpasswd to setup encrypted passwords;
"Home directory" special share "[homes]" is commented out by default. Uncomment it and make changes as described.

System.Environment.Exit and Mono.GetOptions

2008-08-27T22:03:00.001-04:00

C# is apparently such a well-structured language (or its authors wished it to be one) so that it doesn't have normal "exit" function. Indeed, why does one need a special "exit" utility? Program should end simply when "Main" function reaches its end. Simple, ah?

Well, let's take a step back. Microsoft .NET platform apparently does not have its own "standard" argument-parsing utility similar to GNU "getopt". Obviously, it isn't a huge problem to do a straightforward port of "getopt" to C#; see one such port (not free, unfortunately) here. However, Mono project provides a better alternative: Mono.GetOptions assembly.

Blog post from PumaCode.org blog gives a very neat overview of how it is done; basically, for testing you can create one C# file like that:

using System;
using System.Reflection;
using Mono.GetOptions;

// Attributes visible in " --help"
[assembly: AssemblyTitle ("go.exe")]
[assembly: AssemblyVersion ("1.0.*")]
[assembly: AssemblyDescription ("Mono.GetOptions Sample Program")]
[assembly: AssemblyCopyright ("Public Domain")]

// This is text that goes after " [options]" in help output.
[assembly: Mono.UsageComplement ("")]

// Attributes visible in " -V"
[assembly: Mono.About("Insert About Text Here.")]
[assembly: Mono.Author ("Your name here")]

class SampleOptions : Options
{
    // Long option is the variable name ("--file"), short option is -f
    [Option ("Write report to FILE", 'f')]
    public string file;

    // Long option is the variable name ("--quiet"), short option is -q
    [Option ("don't print status messages to stdout", 'q')]
    public bool quiet;

    // Long option is as specified ("--use-int"), no short option
    [Option ("Sample int option", "use-int")]
    public int use_int;

    public SampleOptions ()
    {
        base.ParsingMode = OptionsParsingMode.Both;
    }
}

class TestApp
{
    public static void Main (string[] args)
    {
        SampleOptions options = new SampleOptions ();
        options.ProcessArgs (args);

        Console.WriteLine ("Specified Program Options:");
        Console.WriteLine ("\t           file: {0}", options.file);
        Console.WriteLine ("\t          quiet: {0}", options.quiet);
        Console.WriteLine ("\t        use_int: {0}", options.use_int);

        Console.WriteLine ("Remaining Program Options:");
        foreach (string s in options.RemainingArguments) {
            Console.WriteLine ("\t{0}", s);
        }
    }
}

and try to compile and run it. With mono, it is as simple as:

gmcs -r:Mono.GetOptions getopt.cs

Under .NET, we have to copy Mono.GetOptions.dll from somewhere first. If you have Mono installed on some Linux machine, you can find this DLL under "/usr/lib/mono/gac".

Of course, you need to know where to copy it it to and how to use it. This question has two parts: (a) how to compile code which uses Mono.GetOptions; (b) How to run/distribute code which uses Mono.GetOptions .

How to compile.

You can use compile-option "-r:<full path to Mono.GetOptions.dll>". This will work.

How to make it work by simply using -r:Mono.GetOptions.dll with no path? That's a good question which I cannot answer at this point. A "standard" location for "global" assemblies is "c:\WINNT\assembly" (or similar); note that you can only copy something there with Windows Explorer UI, since it is using special Shell Extension to properly distribute DLLs to some internal directories, not visible from the shell. If you've copied your DLL there, you can reference it as:

/r:c:\WINNT\assembly\GAC_MSIL\Mono.GetOptions\2.0.0.0__0738eb9f132ed756\Mono.GetOptions.dll

Or, alternatively, you can append the above line to your CSC.RSP file (e.g. "c:\WINNT\Microsoft.NET\Framework\v3.5\csc.rsp")

How to run.

This is simple enough - if you have copied your assembly to "c:\WINNT\assembly", it will run fine.

End of story?

No, not yet.

Your test executable will indeed run... till you'll try to copy it to network share and run from there. If you do that and execute "getopt.exe -help", you'll observe "security violation" failure.

One might ask, what kind of security could be violated by parsing a command line?

It is very simple, indeed. As one can see from the source code, function OptionList.ProcessArgs() calls function System.Environment.Exit(), and as it follows from both mono documentation and Microsoft documentation, this function for some reason needs permission to "execute unmanaged code" to run; by default, such permission is not afforded to programs run from "local intranet", including mounted shares.

Is there any way to override this default and grant all required permissions? Perhaps. There is (or was?) utility called caspol.exe which could help you; whether this utility is still supported is not clear. Rather, Microsoft suggest using UI-based configuration tool (part of Windows "Administrative tools", available from Control Panel). When I tried changing that saying that policies from "My computer" shall apply to everything coming from "Local intranet", it indeed fixed the problem... except that as a result Windows Live Writer refused to start! Fortunately, this "configuration tool" has an option to restore default behavior....

.... And all this trouble from simple desire to use some kind of argument-parsing in C#.....

The mystery of a plus

2008-08-12T19:31:00.006-04:00

Well, today I noticed that "long" listing in one of my directories appends "+" sign to each permissions string, like that:

drwxrwxr-x+ 13 user      500 4096 2008-05-10 02:07 DivX
drwxrwxr-x+ 34 user      500 4096 2008-05-10 08:44 Raw

I was used to seeing this extra plus a lot under Cygwin, and never really gave it much thought always attributing this to some Cygwin peculiarities. But this time around this wasn't Cygwin at all - this was pristine Ubuntu 8.04. Given the specific path where this pluses were present, it was apparent that this is somehow related to Samba - these directories were originally uploaded with Samba daemon. So, I did understand from the get-go that in all likelihood Samba server is assigning some special flag to these files and "ls" is trying to notify me of the existence of this flag. I only need to figure out which specific flag it is (and how to remove it). This should be easy. Or so I thought.

The first page I stumbled upon in Google search had this to say:

The character after permissions is an ACL or extended attributes indicator. This character is an @ if extended attributes are associated with the file and the -@ option is in effect. Otherwise, this character is a plus sign (+) character if a nontrivial ACL is associated with the file or a space character if not.

ACL stands for "Access Control Lists" and allows for some fine-tuning of users' access permission on the level of individual user.

Ok, so what now? Same page listed some ACL-related command line options to ls, like '-v'; none of them was recognized by my ls. Well, that was to be expected since these were options for Solaris ls. How about Linux? No problem, said Google, you can use commands getfacl/setfacl to get/set ACL for a given file.

getfacl does indeed generate some output, but it turned out it does so even in absence of any ACLs, simply interpreting default Unix behavior based on file ownership and permissions. Attempts to set and/or clean ACLs for a file failed, and for a good reason: apparently, in order for ACL to be in effect for a mounted file system, mount option 'acl' must be included, as explained in some details e.g. here.

Fine, so if this is not ACL to blame, how about extended attributes? (Actually, extended attributes is not something parallel to ACLs; ACL implementation is based on extended attributes, which merely allows to attach more or less arbitrary "extended" info to each file in a supported file system; ACL is about how its attributes are to be interpreted by file system and kernel; attributes themselves bear no such interpretation). For one thing, this seemed as unlikely, since to use extended attributes one needs another mount option, "user_xattr". In fact, I didn't even have a package "attr" installed, which provided utilities getfattr/setfattr very similar to getfacl/setfacl.

Nevertheless, I installed the package and tried running getfattr on my files. None, as expected.

At this point in time I decided that the fastest way to get to the bottom of this puzzle was to look at the source of "ls" to see what kind of test makes it decide to append this extra "plus".

This was surprisingly simple enough to do; though this is somewhat off-topic, here is a brief overview what I did to compile my own debuggable "ls", for future references:

Install packages dbs and debhelper (for build);
Install packages xxgdb or ddd (for debugging);
Run command "dpkg -S `which ls`" to figure out which package "ls" belongs to. It is "coreutils";
Retrieve the source from http://packages.ubuntu.com/hardy/coreutils;
Untar, change dir to coreutils-6.10 and following instructions from file debian/README.build run command "debian/rules setup";
Now change dir to build-tree/coreutils-6.10 and run " CFLAGS='-g' ./configure";
Finally, do "make";
Now your fresh debuggable "ls" is available as "src/ls".

Anyway, it quickly turned out "ls" is calling function getfilecon() and upon receiving some non-trivial output appends "+" markup.

So, what is getfilecon()? It is part of SELinux toolkit; SELinux is, of course, a special security-enhancement system for Linux, which is totally separate from traditional Linux access restriction tools, including ACL; it works by assigning special "roles" to files, processes and users and then keeping a (potentially) huge policy database of who-can-do-what; it was originally developed by NSA and open-sourced in 2000.

On a practical level, SELinux-enabled system (Ubuntu Hardy being one of them) have special CLI switch "-Z" for many file and process-management utilities to report/take into account SELinux "context". Indeed, this finally allowed me to solve first piece of the mystery - force "ls" to tell me what it means my "+", by using "-Z":

% ls -1Z
user_u:object_r:file_t DivX
user_u:object_r:file_t Raw

The only trouble is, of course, that SELinux was never enabled on my system! This could be seen with e.g. "id -Z"

% id -Z
id: --context (-Z) works only on an SELinux-enabled kernel

Anyway, enough suspence, here is what (I think) is going on. Not unlike ACL, SELinux keeps its "context" in extended attributes. Whether kernel has SELinux enabled or not, utilities which are told to assign certain SELinux "types" to files, are free to do so using e.g. setfattr. These attributes, of course, will have no impact whatsoever on security of the system, but will be reported by utilities like "ls" which, again, simply looks at the specific attribute.

In fact, the name of this attribute is "security.selinux"; indeed, I can verify this with "getfattr" :

% getfattr -n security.selinux -d Raw
# file: Raw
security.selinux="user_u:object_r:file_t\000"

So why woudn't "getfattr -d" report this? Isn't it in the absence of "-n" supposed to return all attributes?

Well, yes and no. By default, it reports all user attributes, that is, attributes with names which begin with "user.". To truly report all attributes, add "-m -" :

% getfattr -m - -d Raw
# file: Raw
security.selinux="user_u:object_r:file_t\000"

You can, of course, assign this attribute with command "setfattr -n security.selinux -v <value> <file name>" or remove it altogether with "setfattr -x security.selinux <file name>".

This, finally, answers the question we asked in the beginning - how to remove this "plus" from "ls -l" output - and thus concluded our investigation.

Bonus track. Some SELinux-related references:

Zimbra

2008-07-29T12:50:00.001-04:00

Well-respected and popular blog TechCrunch recently published a review of new version of Zimbra desktop e-mail client; as blog authors put it, Zimbra Releases Version 3 Of Open Source Email Client, And It’s Awesome.

OK, I tried playing a little bit with Zimbra and I think that the concept is OK, application is responsive and useful. However, there are a few things that must be fixed or improved before it could indeed outplace Thunderbird on my desktop. In no particular order, these are:

Account setup is strange. Software insists that SMTP server must be specified and available at the time of the setup, or else it refuses to allow you to proceed. Why is that? What's wrong with using Zimbra to simply read e-mail from one of my IMAP servers, or e.g. setup SMTP server later?
It refuses to show you any information at all about a folder unless it is "synchronized", that is, completely copied to the local machine and indexed. Without doing that (which can take hours and gigabytes in traffic) you can't even (in Zimbra) figure out if there is anything in this folder which is worth your time (you can of course start synchronization and then cancel it);
Spellchecker options are nowhere no be found, but this does not actually mean that spellchecking is completely unavailable: since Zimbra UI is really nothing more than a "headless" Mozilla browser (they call this technology "Mozilla Prism") and Mozilla now has built-in spellchecker, you will see incorrectly spelled words underlined in red, except that you can neither see list of corrections (since Zimbra overrides RMB) nor change the language (for the same reason);
Zimbra is trying to use (by default) "conversation" mode, which is to say, combine together in the list of messages deemed (based on subject) to belong to the same thread. This is actually done very nicely from the UI standpoint (although GMail is still much better), but background engine which generates "conversations" is faulty; it constantly fails to realize that messages with subjects "foo" and "Re: foo" belong to the same conversation;
Zimbra is allowing custom-made tags, but for some reason only one tag per message. Why?
Update. This is incorrect, you can assigned as many tags as you want to a message.
When retrieving message headers from IMAP server, in the message list Zimbra shows column "Received" which is actually the time when IMAP server registered the message, and not date of the message itself. Most people probably won't notice the difference, but since I am using my IMAP server as a backup for various other sources (e.g., GMail) and copy email there only occasionally, it is highly inconvenient for me;
I really like the concept of searches and conversations working across the IMAP folders. However, why not across separate "accounts"?
Unlike Thunderbird, "search" folders and (probably using same internal implementation) "tag" folders do not display number of unread messages inside them in the folder tree; I have to manually click on each of them to see if there are any new messages;
Also regarding search folders, and also unlike Thunderbird, there is no way to edit existing search folder, you can only remove it and re-create;
Working with contacts and address books is painful. Zimbra does not support any kind of LDAP queries and uploading complete corporate address book is buggy; even after I wrote a special tool to re-order and re-name properly fields in CSV file, ZImbra import missed about half of e-mails (why only e-mails?) from the import. Worse, I could not manually edit these contacts to add missing emails; the only possibility was to delete these contacts and then manually add (identical) new ones. This kind of worked....
For the life of me, I can't figure out where sent messages disappear to; I can't see anything in "Sent" folder (and I am not even mentioning a capability to push sent messages to server). I guess the only available workaround is to always sent a copy to myself...
And last but not the least, after a while Zimbra simply stopped updating some of my folder as new messages continued to show up on server. Till now, I couldn't say what specifically triggered this behavior.

Honestly, I can probably live with all these bugs and shortcomings - on a technical side, that it. But there is a bigger issue here of a psychological conformability. When I select a IMAP folder without any filters or adjustment - I expect my mail client to show me just that, folder, precisely as IMAP server reports it to be. If, instead, my e-mail client inserts some intermediate logical layers between me and my folder, even if these layers are supposed to be completely transparent, I know that in practice I can never be really sure that what I see is indeed my folder "as is" and not some internally generated skewed view of it. And that's the problem - for me, anyway.

P.S. Some clarification on the last problem of Zimbra not auto-updating some or all folders.

First, problem was likely triggered by an attempt to synchronize with some "weird" Microsoft Exchange folders which did not really have "messages" as such;

Second, you can get dome useful debugging info (mostly Java tracebacks) from <...>/zimbra/zdesktop/log/mailbox.log, (especially) if you checked "enable debug trace" in account setup.

Accessing SMB shares under firewall

2008-07-27T21:08:00.001-04:00

There are, generally speaking, five "standard" ways to make files on (Linux) server available to clients:

Using FTP server;
If sshd is running, files could be accessed with SFTP;
If Web server is running, WebDAV could be used;
Using NFS (see earlier port);
Using SMB-shares.

In principle, last choice - using Samba shares - is supposed to be most "native" with respect to Windows clients; let's consider how difficult it is to use it practically...

For the following, we assume that SMB server is running on "SERVER" and we'll be using it to access file of local regular user "user".

1. Install and configure samba

1.1. Install

# apt-get install samba smbclient

1.2. Modify config file.

Here we zero in on "minimalistic" approach, which only requires minimal changes to default config file (as distributed with Debian, anyway). It has one built-in share "homes" which provides access to each user's home directory (it is enabled by default, but in read-only mode)

1.2.1. Backup default config file:

cp /etc/samba/smb.conf /etc/samba/smb.conf.original

1.2.2. Modify global settings:

workgroup = <Enter some name>
interfaces = <enter some interfaces from /sbin/ifconfig, e.g. lo venet0:0>
printcap name = /dev/null (shut down all complaints in logs about printers)
encrypt passwords = yes (or else you won't connect from Windows NT and up)
security = user (this should be the default anyway)
smb passwd file = /etc/samba/smbpasswd (see below why/when this is necessary)

1.2.3. Modify setting for share.

If you are satisfied with read-only access to user's directory, there is nothing more to change. If you want read-write access, there are some settings to adjust:

writable = yes
create mask = 0644
directory mask = 0755

Note on "encrypt password": if encrypt password = false, you don't need "smb passwd file", system password file will be used. For some reason it did NOT work for me if "encrypt password = true". As suggested in [8], I did this:

cat /etc/passwd | mksmbpasswd > /etc/samba/smbpasswd
smbpasswd user (for each use who needs his home dir access via SMB)

1.2.4. Restart server:

/etc/init.d/samba restart

1.2.5. Test installation (on the server as user)

smbclient //localhost/homes

If you can, test from another location which does not block outgoing ports 139, 445

smbclient //SERVER/homes -U user

2. Setup Windows computer

In principle, the following command

net use [<drive letter>:] \\SERVER\homes /user:user /persistent:no

should be able to mount corresponding share. However, for this to work it is necessary that client computer had direct access to server using ports 139 and/or 445. If server is to be used in local subnet, this is undoubtedly so and no more setup is required. However, if you are accessing server from the Internet, and your ISP is blocking these ports (like RCN), read on.

The idea is to try to mount SMB shares on SERVER as if they were available on localhost; intercept requests made on (local) ports 139 and 445 and somehow forward them to SERVER.

This however appears to be more difficult than it sounds. The problem is, Windows by default binds all adapters on port 445 :

> netstat -ano | grep 445
TCP 0.0.0.0:445 0.0.0.0:0 LISTENING 4

and the only way to make this port available for binding is to disable NetBios completely, which is rather pointless, since then you won't be able to amount anything at all (if, however, you want to play with this, refer to [5] and [6]).

Fortunately, however, usually Windows, after failing to mount using port 445, falls back to port 139, which apparently you can bind, albeit not on "standard" loopback adapter 127.0.0.1; (well, you can bind to it, but it won't work for whatever reason), thus necessity to create new loopback connection. This "fallback" logic is in no way guaranteed and moreover has reportedly been broken by a recent Vista patch; but at least on XP this seems to work, as long as you implement 2.1 and 2.2 below.

2.1. If you are using Windows XP, install this Windows patch [3].

2.2. For your existing Internet connection, enable option "EnableBIOS over TCP/IP" (in fact, it may be sufficient to do so for any Internet connection. Quoting from [1]: "It also appears that if there are no valid interfaces with NetBIOS over tcp enabled, then windows will not attempt to use port 139")

2.3. Add new Microsoft loopback interface and bind it to port 10.0.0.1, see [4] and [1] ([4] has some screenshots from Vista; XP installation is similar).

2.4. It may or may not be necessary, but I also (a) disabled all "items" for new 10.0.0.1 loopback connection (except TCP/IP, see first screenshot above) ; (b) enabled LMHOSTS lookup; (c) disabled NetBIOS over TCP/IP (see second screenshot) ; (d) disabled "File and Printer Sharing for Microsoft Networks" for ALL connections;

2.5. Reboot and you are all set!

3. Setup port forwarding

This can be done in one of the two ways.

You can use of the many existing utilities for port forwarding. (It is more correct in fact to speak of reversed port forwarding, but people usually call it port forwarding all the same). In this case you need a (non-blocked) port to forward to, and your Samba server should be told to listen on this port.

Alternatively, you can use ssh port tunneling capability and tunnel port 139 through ssh. This requires more complicated setup and is more difficult to automate, but does not require any additional port to be used and has an additional benefit of securing your Samba traffic via ssh.

3.1. Using ssh tunneling.

3.1.1. With Cygwin (or similar CLI) ssh:

ssh -L 10.0.0.1:139:localhost:139 SERVER

3.1.2. Using PuTTY

(see screenshot in [4]). There are various ways to automate this using pageant, but I won't get into this here.

3.2. Using port forwarding.

3.2.1. Add another option to smb.conf :

smb ports = 445 139 8445 8139

3.2.2. Configure AUTAPF (shareware) or PassPort (Open Source) to forward port 139 on local adapter 10.0.0.1 to port 8139 on SERVER.

AUTAPF is more convenient as it immediately tells you if it can't bind a port thus it is better for testing; once you are comfortable with the setup, you can switch to free PassPort.

4. Using SMB shares

4.1. Under Windows

net use [<drive letter>:] \\10.0.0.1\homes /user:user /persistent:no

(Unless user if Windows user name and password matches, you'll be asked to enter server password at this point)

4.2. Under Linux

Make sure you have "smbfs" installed ("apt-get install smbfs") and issue this command as super-user:

mount -t smbfs -o username=user%<user password>,uid=<local username>,port=8139 //SERVER/homes <local mount point>

(You can use option -credential in place of plain text user name/password, see "man smbmount" and this page)

4.3. Using smbclient, any platform

For any platform which has utility "smbclient" or equivalent, and provided that you've configured SMB server to listen to port 8138, you can use this

smbclient //SERVER/homes -U user -p 8139

to access your files.

You can get MinGW-based smbclient.exe for Windows from here (it works fine except that you have to specify password in the command line); alternatively, you can build your own Cygwin-based version using one of the patched published here (you'll need to disable first test in Samba source file source/tests/summary.c)

References:

[1] Sharing (tunneling) Samba/CIFS/SMB file systems over SSH

[2] How to tunnel Samba via ssh from Windows XP without having to disable local NetBIOS

[3] Programs that connect to IP addresses that are in the loopback address range may not work as you expect in Windows XP Service Pack 2

[4] Vista or XP Accessing Samba shares securely over SSH

[5] Disabling Port 445 (SMB) Entirely

[6] After you disable the "Client for Microsoft Networks" option for a dial-up connection, the dial-up connection is still active on a Windows XP-based computer

[7] "Network Location Cannot be Reached" when accessing shares

[8] SMBCLIENT CONNECTION ERROR

Installing TTF fonts under Debian

2008-07-20T10:07:00.001-04:00

I am still not sure I am fully understand the inner workings of fonts under Linux; using hints from this page, here is how I recently installed RedHat Liberations fonts under Debian:

cd /tmp
# using link from https://fedorahosted.org/liberation-fonts
wget https://fedorahosted.org/releases/l/i/liberation-fonts/liberation-fonts-1.04.tar.gz
tar -xvzf liberation-fonts-1.04.tar.gz
cd liberation-fonts-1.04
# remove everything except font *.ttf files
rm `ls -1 | grep -iv \.ttf$`
# I prefer to install under /usr/local, but it's irrelevant
mkdir -p /usr/local/share/fonts/truetype
cp -pr /tmp/liberation-fonts-1.04  /usr/local/share/fonts/truetype/liberation-fonts
# defoma is Perl-based; for some reason, some required Perl dependencies weren't
# installed by default
apt-get install libft-perl
defoma-hints -c --no-question truetype \
            /usr/local/share/fonts/truetype/liberation-fonts/* > 
            /etc/defoma/hints/liberation-fonts.hints
defoma-font register-all /etc/defoma/hints/liberation-fonts.hints
defoma-reconfigure 
xset fp rehash

I just hope one day I'll understand what all of that really means....

Backing store

2008-06-27T16:39:00.003-04:00

"Backing store" is an old, albeit not well-known, feature of X servers, going back to the 80's or may be even before that. More often when not, servers do not enable it by default, and then users are left to search through server customization option to discover appropriate magic to enable it.

This page, which, I believe, is devoted to some specific Java drawing problems, has some good and concise explanation of "backing store" mechanism (emphasis added):

Usage of backing store is another way of improving our painting procedure. Backing store is a feature of most of X servers: provided that a window has the backing store attribute set, such an X server automatically maintains the contents of the window while it is obscured. When (a part of) a window is exposed, its contents is automatically copied to the screen, and expose events are not generated as the contents is restored. Though the attribute is only a hint to an X server.
Backing store does conflict with DGA, so backing store attribute is not set for a window if Java2D uses DGA on at least one screen. (DGA is used only on some Solaris/SPARC machines.)
To allow setting of the backing store attribute for a window (if DGA is used), one may set the env var NO_J2D_DGA; refer to http://java.sun.com/j2se/1.5.0/docs/guide/2d/flags.html.
To leverage backing store feature, in addition to setting of the backing store attribute for a window, the X server must have backing store support enabled. One should consult his/her X server documentation in order to enable X server's support for backing store. Let's consider typical setting of 3 most popular X servers.

Xsun has backing store support enabled by default.
XFree86: put
Option "backingstore"
in each Screen section of /etc/X11/XF86Config-4 or /etc/X11/XF86Config if the former isn't present. (See also http://www.xfree86.org/current/ati5.html)
Xorg: put
Option "backingstore"
in each Screen section of /etc/X11/xorg.conf. (See also http://www.x.org/X11R6.8.2/doc/ati5.html)

It turns out that there is also option "SaveUnders" somehow related to this. In any case, this is an example of "Screen" section of /etc/X11/xorg.conf with backing store enabled:

Section "Screen"
   Identifier     "Screen0"
   Device         "Device0"
   Monitor        "Monitor0"
   DefaultDepth    24
   Option "BackingStore" "on"
   Option "SaveUnders" "on"
   SubSection     "Display"
       Depth       24
   EndSubSection
EndSection

Enjoy!

Difficult turn to Firefox 3

2008-06-17T14:34:00.001-04:00

Someone is trying to edit main page of Mozilla.com in real time (apparently by doing some merge and resolving conflicts as we speak)

Visual Studio tricks

2008-06-10T20:51:00.002-04:00

Well, this is not really a trick but rather a trivial remark, but Visual Studio entertains some peculiar notion of "default libraries", which means, basically, that object file "knows" which basic system libraries like LIBCMT.LIB (basic C multithreaded library) or LIBCMTD.LIB (debugging version). That means that if you have certain 3-rd party or external libraries which you only have available to you in "non-debugging" version, and when try to link them with your project in "Debug" configuration, linker may want to include both LIBCMT.LIB and LIBCMTD.LIB, creating multiply-defined symbols and failing the build.

For some reason, problem (for me) only appears when I try to use STL library, which then causes diagnostics like this:

LIBCMTD.lib(dbgheap.obj) : error LNK2005: __heap_alloc already defined in LIBCMT.lib(malloc.obj)
LIBCMTD.lib(dbgheap.obj) : error LNK2005: __recalloc already defined in LIBCMT.lib(recalloc.obj)
LIBCMTD.lib(dbgheap.obj) : error LNK2005: __msize already defined in LIBCMT.lib(msize.obj)
LIBCMTD.lib(malloc.obj) : error LNK2005: _V6_HeapAlloc already defined in LIBCMT.lib(malloc.obj)
LIBCMTD.lib(dbghook.obj) : error LNK2005: __crt_debugger_hook already defined in LIBCMT.lib(dbghook.obj)
LIBCMTD.lib(sbheap.obj) : error LNK2005: __get_sbh_threshold already defined in LIBCMT.lib(sbheap.obj)
LIBCMTD.lib(sbheap.obj) : error LNK2005: __set_sbh_threshold already defined in LIBCMT.lib(sbheap.obj)
LIBCMTD.lib(sbheap.obj) : error LNK2005: __set_amblksiz already defined in LIBCMT.lib(sbheap.obj)
LIBCMTD.lib(sbheap.obj) : error LNK2005: __get_amblksiz already defined in LIBCMT.lib(sbheap.obj)
LIBCMTD.lib(sbheap.obj) : error LNK2005: ___sbh_heap_init already defined in LIBCMT.lib(sbheap.obj)
LIBCMTD.lib(sbheap.obj) : error LNK2005: ___sbh_find_block already defined in LIBCMT.lib(sbheap.obj)
LIBCMTD.lib(sbheap.obj) : error LNK2005: ___sbh_free_block already defined in LIBCMT.lib(sbheap.obj)
LIBCMTD.lib(sbheap.obj) : error LNK2005: ___sbh_alloc_block already defined in LIBCMT.lib(sbheap.obj)
LIBCMTD.lib(sbheap.obj) : error LNK2005: ___sbh_alloc_new_region already defined in LIBCMT.lib(sbheap.obj)
LIBCMTD.lib(sbheap.obj) : error LNK2005: ___sbh_alloc_new_group already defined in LIBCMT.lib(sbheap.obj)
LIBCMTD.lib(sbheap.obj) : error LNK2005: ___sbh_resize_block already defined in LIBCMT.lib(sbheap.obj)
LIBCMTD.lib(sbheap.obj) : error LNK2005: ___sbh_heapmin already defined in LIBCMT.lib(sbheap.obj)
LIBCMTD.lib(sbheap.obj) : error LNK2005: ___sbh_heap_check already defined in LIBCMT.lib(sbheap.obj)
LIBCMTD.lib(sbheap.obj) : error LNK2005: ___sbh_pHeaderDefer already defined in LIBCMT.lib(sbheap.obj)
LIBCMTD.lib(isctype.obj) : error LNK2005: __isctype_l already defined in LIBCMT.lib(isctype.obj)
LIBCMTD.lib(isctype.obj) : error LNK2005: __isctype already defined in LIBCMT.lib(isctype.obj)

The solution is to add option "/NODEFAULTLIB:LIBCMT" to the linker; this can (only) be done via manually editing linking options in project configuration:

Google gadgets for Linux

2008-06-05T22:20:00.006-04:00

Google released Google gadgets for Linux, as an open-source project. After a few trials-and-errors, I managed to set it up on my Ubuntu 8.04 64-bit.

Install the following packages: libgtk2.0-dev libmozjs-dev libxul-dev libcurl4-openssl-dev libgstreamer0.10-dev libgstreamer-plugins-base0.10-dev libdbus-1-dev libxml2-dev librsvg2-dev spidermonkey-bin libxcomposite1 xcompmgr (Not sure all are really required, but just in case) Also, some qt-related (development) packages are required if you want QT support.
Download google-gadgets-for-linux-0.9.1.tar.gz .
Untar, configure and make will all the default options. Install additional packages if required.
Enable "composite mode" in X-server, by adding these lines to /etc/X11/xorg.conf : Section "Extensions"
Option "Composite" "Enable"
EndSection
Re-login to force server restart.
In a separate window, start: xcompmgr -o 10
And finally, in another shell start Google Gadgets as ggl-gtk or ggl-qt. You may need to add /usr/local/bin to you $PATH and /usr/local/lib to you $LD_LIBRARY_PATH.
You can now add gadgets through UI or by passing .gg file directly to the command line, e.g.: ggl-gtk ~/Download/Tetris-1.3.gg

Java without Java, part I

2008-05-30T20:07:00.002-04:00

Java language has been there for a while, and various "alternative" implementations, proprietary and open source, flourished. Microsoft announced recently that it is dropping its Visual J++ implementation (which is part of Visual Studio 2005 but not 2008), but in open source world, nothing could be "dropped" as such, so using Java without Sun's implementation was and is an option.

In this 2-part post, I am going to review some options to run relatively simple Java programs on Windows without using Sun's Java VM. First part is devoted to using gcj to compile and run Java code.

It is of course relatively straightforward to actually compile Java files (to .class files or natively), and Java front-end to Gnu Compiler Collection has been known for a while as "gcj"; it is much more demanding and challenging task to implement substantial piece of Java library; this activity has been ongoing as "classpath project". Standard GNU gcc distribution includes a slightly modified version of classpath, which is compiled into library "libgcj.a/.so"

Cygwin installation gladly offers gcj version 3.4.6, along with corresponding library. Furthermore, this program passes standard "Hello, World!" test:

public class Hello {
  static public void  main (String[] argv) {
    System.out.println ( "Hello, World!" );
  }
}

here is how to compile and run :

> /usr/bin/gcj --main=Hello Hello.java -o Hello
> ls -l Hello.exe
-rwxrwxrwx 1 user mkpasswd 6359272 May 30 16:45 Hello.exe*
> ./Hello.exe
Hello, World!

Unfortunately, any attempt to move substantially beyond "Hello, Word!" kind of tests result in run-time failure like this:

> ./Test.exe 
Exception in thread "main" java.lang.Error: Not implemented
   <<No stacktrace available>>

without even a clear indication of what is not implemented.

Worse, resulting Cygwin executables while being statically linked against "libgcj.a" (thus hefty 6MB for a trivial program) still depend on Cygwin DLL "cygwin1.dll"; compile option "-mno-cygwin" which is meant to generate executable independent of Cygwin DLL and does so for C and C++ files (so-called MinGW mode) fails even for aforementioned "Hello, World!" example (compilation succeeds, but executable crashes with no error messages).

Well, of course, version 3.4.6 is already quite old. What if one tries to download latest GCJ and compile under Cygwin?

In fact, one will quickly discover that Cygwin support is all but dropped from GCJ. You can build and install a compiler itself, but Java library is specifically marked as "not supported" in GCC configuration script:

  *-*-cygwin*)
    target_configdirs="$target_configdirs target-libtermcap target-winsup"
    noconfigdirs="$noconfigdirs target-gperf target-libgloss ${libgcj}"

If one tries to remove this restriction and force compilation, it proceeds for a while and then hangs indefinitely while trying to execute

gcj-dbtool -n classmap.db

Another option to try is MinGW distribution. However, they currently support gcc-3.4.5, with GCC 4 being in "alpha" testing. So this option is not yet viable.

Situation is not completely hopeless, though: there is some nice guy Mohan Embar who makes his own distributions of MinCW/GCJ, including latest versions. This kind of works, but not without some caveats.

For the purpose of this slightly-more-complex-than-Hello-World testing, I used the following Java program "Test.java":

import java.util.*;
import java.util.regex.*;
public class Test {
  static private Pattern p=Pattern.compile ("\\s*(\\d+)\\s*\\+\\s*(\\d+)\\s*$");
  static public void  main (String[] argv) {
    String arg = join ( Arrays.asList(argv), " " );
    Matcher m = p.matcher ( arg );
    if (m.lookingAt()) {
      int a = (new Integer(m.group(1))).intValue();
      int b = (new Integer(m.group(2))).intValue();
      System.out.println ( a + " + " + b + " = " + (a + b) );
    }
    else
      System.out.println ( "Could not parse!" );
  }
  public static String join(Collection s, String delimiter) {
    StringBuffer buffer = new StringBuffer();
    Iterator iter = s.iterator();
    while (iter.hasNext()) {
      buffer.append(iter.next());
      if (iter.hasNext())
 buffer.append(delimiter);
    }
    return buffer.toString();
  }
}

If you download this distribution, unzip it, set your $PATH and compile Test.exe as before, you will get the following failure trying to run it:

Exception in thread "main" java.lang.ExceptionInInitializerError
   at java.lang.Class.initializeClass(/datal/gcc/gcc/libjava/java/lang/Object.java:513)
   at gnu.java.util.regex.RE.getLocalizedMessage(/datal/gcc/gcc/libjava/classpath/gnu/java/util/regex/RE.java:262)
   at gnu.java.util.regex.RESyntax.(/datal/gcc/gcc/libjava/classpath/gnu/java/util/regex/RESyntax.java:345)
   at java.lang.Class.initializeClass(/datal/gcc/gcc/libjava/java/lang/Object.java:513)
   at java.util.regex.Pattern.(/datal/gcc/gcc/libjava/classpath/java/util/regex/Pattern.java:76)
   at java.util.regex.Pattern.compile(/datal/gcc/gcc/libjava/classpath/java/util/regex/Pattern.java:153)
   at java.util.regex.Pattern.compile(/datal/gcc/gcc/libjava/classpath/java/util/regex/Pattern.java:135)
   at Test.(ccZgp08ajx:0)
   at java.lang.Class.initializeClass(/datal/gcc/gcc/libjava/java/lang/Object.java:513)
   at Test.main(ccZgp08ajx:0)
Caused by: java.util.MissingResourceException: Bundle gnu/java/util/regex/MessagesBundle not found
   at java.util.ResourceBundle.getBundle(/datal/gcc/gcc/libjava/java/util/ResourceBundle.java:372)
   at java.util.ResourceBundle.getBundle(/datal/gcc/gcc/libjava/java/util/ResourceBundle.java:243)
   at gnu.java.util.regex.RE.(/datal/gcc/gcc/libjava/classpath/gnu/java/util/regex/RE.java:133)
   at java.lang.Class.initializeClass(/datal/gcc/gcc/libjava/java/lang/Object.java:513)
   ...9 more

The problem appears to be in static linking (that is, linking with libgcj.a rather than loading libgcj.dll runtime). Java is capable of loading classes dynamically, while static linking can only link-in objects (classes) which are referenced compile-time; hence any implementation which is using dynamic loading is doomed to fail. Therefore, for the time being "static" linking is not "officially" supported by GCJ team, and on Unix shared linking is the default. On Windows, however, most of GCJ distributions don't support DLL linking (Mohan Embar claims that his GCC/GCJ 3.4 distribution supports DLL linking, but it didn't work for me, and even if it did, it useless in this old version anyway)

In this case we are loading not a class, but so-called "resources", specifically message file with various messages associated with "regular expression" errors. If you have access to this file (which is included in any GCJ source distribution), you can link in this resource manually like this:

> gcj --resource gnu/java/util/regex/MessagesBundle.properties -c \
 path\gcc-version\libjava\classpath\resource\gnu\java\util\regex\MessagesBundle.properties \
 -o MessagesBundle.properties.o
> gcj --main=Test MessagesBundle.properties.o Test.java -o Test
> ./Test.exe 2+2
2 + 2 = 4

As you could see this actually works (!!!), though size of Test.exe is 38 Mbytes (!!!) (you can use utility "strip" to get it down to approx. 12M)

Unfortunately, attempt to compile and run any of Swing UI demos results in failure:

> .\DynamicTreeDemo.exe
Exception in thread "main" java.awt.AWTError: Cannot load AWT toolkit: gnu.java.awt.peer.gtk.GtkToolkit
   at java.awt.Toolkit.getDefaultToolkit(/datal/gcc/gcc/libjava/classpath/java/awt/Toolkit.java:544)
   at java.awt.EventQueue.invokeLater(/datal/gcc/gcc/libjava/classpath/java/awt/EventQueue.java:316)
   at javax.swing.SwingUtilities.invokeLater(/datal/gcc/gcc/libjava/classpath/javax/swing/SwingUtilities.java:950)
   at DynamicTreeDemo.main(D:/USERPR~1/IGNATI~1.PTC/LOCALS~1/Temp/ccCwbaaajx:0)
Caused by: java.lang.UnsatisfiedLinkError: gtkpeer: can't open the module
   at java.lang.Runtime._load(/datal/gcc/gcc/libjava/classpath/java/awt/event/WindowEvent.java:309)
   at java.lang.Runtime.loadLibrary(/datal/gcc/gcc/libjava/java/lang/Runtime.java:656)
   at java.lang.System.loadLibrary(/datal/gcc/gcc/libjava/java/lang/System.java:515)
   at gnu.java.awt.peer.gtk.GtkToolkit.(/datal/gcc/build/wingcc_build/i686-pc-mingw32/libjava/gnu/java/awt/peer/
gtk/GtkToolkit.java:145)
   at java.lang.Class.initializeClass(/datal/gcc/gcc/libjava/classpath/java/awt/event/WindowEvent.java:309)
   at java.lang.Class.forName(/datal/gcc/gcc/libjava/classpath/java/awt/event/WindowEvent.java:309)
   at java.awt.Toolkit.getDefaultToolkit(/datal/gcc/gcc/libjava/classpath/java/awt/Toolkit.java:561)
   ...3 more

Don't know if this is fixable with GCJ, but in fact it is possible to run some Swing UI demos "natively" on Windows; we will consider how to do this in Part II.

32-bit MPlayer on 64-bit Linux, again

2008-05-20T21:15:00.002-04:00

Almost exactly one year ago, I published here an essay on building 32-bit mplayer on 64-bit RHE Linux (took me a lot of time when, actually). Reading this over a year later, I can hardly understand how I really managed to do it and how the process worked; it must have been some magic which mplayer configuration did. Anyway, here is much more straightforward and comprehensive approach on building not just mplayer, but whole 32-bit Linux (sub-) system within existing 64-bit OS. It is based on Ubuntu 8.04 "Hardy Heron" , but should work with small modifications on any Debian-based system.

The following largely based on this forum post, which in turn references this tutorial. I will assume that code name of your release is "hardy" and 32-bit root is "/sys32"; make sure to replace with values suitable for your system.

sudo apt-get install dchroot debootstrap
sudo mkdir /sys32/
sudo vi /etc/dchroot.conf
- Add this line: hardy /sys32
sudo debootstrap --arch i386 hardy /sys32/ http://archive.ubuntu.com/ubuntu
sudo cp /etc/{passwd,shadow,group,sudoers,hosts} /sys32/etc/
sudo cp /etc/apt/sources.list /sys32/etc/apt/sources.list

(At this moment you might want to edit your sources.list file to your taste, or remove from there things that are specific to 64-bit system, not that there should be any)

Now comes some controversial step. Full-blown debian installation could take gigabytes of disk space, and only small piece of that is binary data which is different for 32-bit. You can reduce the required disk space by "sharing" certain folders with architecture-independent data between the main system and sub-system, but that has also a site-effect of making certain actions in the sub-system break how things work in your main system. Forum post that I referenced above recommend you share /usr/share/fonts this way, but I decided against it; it costs me about 200Mb more, but adds certain piece of mind. Some directories though are worth sharing.

sudo vi /etc/fstab
Add the following lines:
- /home /chroot/home none bind 0 0
- /tmp /chroot/tmp none bind 0 0
- /dev /chroot/dev none bind 0 0
- /proc /chroot/proc proc defaults 0 0
- /media/cdrom0 /chroot/media/cdrom0 none bind 0 0
sudo mkdir /chroot/media/cdrom0
sudo mount -a

OK, by now you should have a basic Debian-style 32-bit system under /sys32; this command

sudo chroot /sys32/

would switch you to operate from "within" this sub-system. From this point on, we continue setup from within (note that "chroot" automatically puts us into root-privileged shell, so "sudo" isn't needed)

locale-gen "en_US.UTF-8"
dpkg-reconfigure locales
apt-get update
apt-get upgrade
apt-get install libgtk2.0-dev gcc g++ make

The last command is just an example; feel free to install whatever you feel like you need, including software built from source, or anything else. MPlayer, among other things, could be built this way without any problems.

Note that when installing from source, it may be useful to do everything but the last installation step from your regular user account, like that:

[Download all source you might want]
sudo chroot /sys32/
su - your_user_account
cd /tmp
tar -xvzf your_sources.tag.gz
cd your_sources
./configure
make
make test
exit
cd /tmp/your_sources
make install

One last remark: you can install some 32-bit support as part of your 64-bit system, including compiler and run-time libraries (you can't however get build libraries and headers other than building them yourself following steps similar to outlined above). Here it is:

apt-get install build-essential lib32gcc1 libc6-dev-i386 libc6-i386 lib32stdc++6 ia32-libs gcc-multilib

This makes it possible to run gmplayer as /sys32/usr/local/bin/gmplayer, without any change in environment or any dependencies on the 32-bit subsystem.

FAT32 file system in Solaris 10

2008-05-19T12:23:00.001-04:00

Despite many Unix-type file system in existence, it is still problematic to have a single file system which is well compatible with both traditional BSD-based Unix systems, like OpenSolaris (which use older ufs or newer ZFS), and modern Linux distributions (using ext2 or ext3). Though it is technically possible to both mount ufs under Linux and ext3 under Solaris, in neither case support is enabled by default and it may not be very reliable.

That leaves the only file system type which is shared by all Unix/Linux computers without need of any hacking or kernel recompilation as FAT32 from Microsoft. All of its inherit limitations non-withstanding, FAT32-formatted USB flash cards are usually immediately understood and mounted with full read/write access by all modern Unix and Linux distributions.

The only things is, creating such file systems is not exactly simple. For starters, since Windows 2000 operating system, Microsoft does not provide an option to (re-) format driver as FAT32 if its size exceeds 32 GB, despite the fact and the file system itself has no such limit, so, funny as it is, to create new FAT32 drives one needs either older Windows OS, like Windows 98, or, a Unix/Linux computer.

Debian (and I guess all other modern Linux systems) allows you to create FAT32 file system with one simple command like that (you need to have dosfstools installed first):

mkdosfs -F 32 /dev/sdb1

(you can also use 'fdisk' prior to that to create DOS partition table, but I am not sure it is necessary). Drives so formatted can be then accessed from Windows computer with not trouble as any other FAT32 drive.

With Solaris though, things aren't exactly that simple. I assume that it is being somewhat too picky about existence of "valid" DOS-style "primary" partition table (remember this old DOS logic of four "primary" partitions C,D,E,F and then the rest, if any, being "extended" partitions?). While I am sure it must be somehow possible to convince fdisk and mkdosfs under Linux to create something Solaris will understand, here I present an opposite approach of creating FAT32 system under Solaris.

All operations are executed under "root".

First off, Solaris has so-called "volume management" which (I guess) is charged with auto-mounting user disks. All instructions related to any manipulation with disk devices and mounts outside of services provided by "volume management" insist that you shut down volume management server upfront with this command:

/etc/init.d/volmgt stop

and then restart it with "/etc/init.d/volmgt start" when you're done.

Next, after you plugged it/installed your drive to be formatted, this command:

iostat -En

will help you to identify correct device name for your disk. My external USB hard drive "Free Agent" from Seagate, for example, is reported like that:

c2t0d0           Soft Errors: 86 Hard Errors: 0 Transport Errors: 0 
Vendor: Seagate  Product: FreeAgent Pro Revision: 400D Serial No:  Size: 750.16GB <750156374016 bytes>
Media Error: 0 Device Not Ready: 0 No Device: 0 Recoverable: 0 
Illegal Request: 86 Predictive Failure Analysis: 0

Next, you run Sun's fdisk to (interactively) create DOS partition table, like that:

fdisk /dev/rdsk/c2t0d0p0

(note that this has to be "character special" file from /dev/rdsk, and p0 (meaning "first primary partition") appended to the name)

Finally, this command generates file system:

mkfs -F pcfs -o 'fat=32' /dev/rdsk/c2t0d0p0:c

(Note ":c" appended to the name, corresponding to the traditions DOS assignment of drive letter "C" to the 1-st partition)

You can now mount your system with this command:

mount -F pcfs /dev/dsk/c2t0d0p0:c /myusbdrive

(Note that this time we use "block special" file from /dev/dsk)

Using NFS mounts under Windows

2008-05-15T23:13:00.002-04:00

When one needs to access files on remote Linux workstation from a windows computer, there are two obvious ways in which this can be done: run Samba server on Linux or NFS client on Windows. While former approach is I guess by far more popular, here I will consider later one: sharing files via NFS client/server.

First, we need to install a NFS client. I don't know of any free one, and perhaps the best known and reliable commercial solution is DiskAccess from Javvin Technologies. Installation is straightforward.

Problems begin though when it turns out that in order to access NFS file system with DiskAccess one needs certain authentication configuration setup. This could be and is done in a corporate environments with NIS (formerly Yellow Page server).

However, when we only have two machines that need to talk to each other, this might be an overkill and a disaster to setup. This is perhaps why there is another option in aforementioned dialog, "PCNFSD Server". What is that?

As the name suggests, this is perhaps mini-daemon intended to facilitate communication between "PC" and NFS server. This sounds good, except that this utility is barely known even to Google, isn't in any Linux distribution, and as a matter of a fact "canonical" version from SUN is perhaps older than Linux itself.

Fortunately, one kind soul invested necessary effort to port this to Linux

Get it here: http://ftp.linux.org.uk/pub/linux/Networking/attic/Other/pcnfsd/linux_pcnfsd2.tgz, untar and unzip into a new directory;
Edit file common.h to uncomment "#define SHADOW_SUPPORT"
Make other changes necessary to build successfully. On RHE4, I had to define "LIBS= -lcrypt" in Makefile.linux;
make -f Makefile.linux
Run linux/rpc.pcnfsd as root. No configuration is required.

Fast, simple, and keeps DiskAccess happy.

Of course, what remains to be done is to "export" a directory and "mount" it under Windows.

To export directory /ext/user on RHE4:

Add line like this:
/ext/user 192.168.2.1/255.255.252.0(sync,rw)
to file /etc/exports (provided that this IP subnet is an accurate description of your local net, this will export directory in read-write mode to the local file system only. You can substitute '*' for IP range if security is not on the top of your priorities list);
Reset list of exported directories with /usr/sbin/exportfs -r or /usr/sbin/exportfs -a ;
While this is supposed to be enough, you may need to restart NFS server with command like this: /etc/rc.d/init.d/nfs restart

To mount exported NFS directory under Windows,

Once after DiskAccess installation, go to control panel, select DiskAccess item and enter your credentials, and set other options as you see fit;
You can now mount with regular Windows UI or with command like this:
net use R: \\nfs_host\ext\user
You shouldn't be forced to use drive letter, but for some reason it didn't work for me without that.

Using rsync for backup, part I

2008-05-14T11:37:00.006-04:00

Huge hard drives are cheap these days, and almost always they come bundled with some fancy back-up solution; not at all surprising, given that it is precisely what majority of customers are going to use new disk space for.

Interestingly enough, there aren't that many good backup utilities for Linux, or open source tools of this sort in general. Most likely this is because people still use "rsync" for Linux, Unix, and multi-platform backups.

"rsync" is very far from being "perfect" backup tool (more on that below), but at least is is old, stable, reliable, simple to use and available for all kinds of platforms, including Cygwin and native Windows port.

Therefore, we will begin by reviewing the most basic ways to use rsync for backups.

Note that it seems like these blatant deficiencies caused rsync maintainers in the last few years to introduce new options which are supposed to help properly implement backup operation. At this moment though, I don't want to get into this; the following tips are based on rsync version 2.6.3 (protocol version 28, released in 2004) or higher.

First, we need to start and configure rsync server on a dedicated server. Here is the plan.

Server configuration (everything is executed as "root"):

Decide whether you want to run rsync daemon through inet (most of servers are used this way) or as a stand-alone server (like apache for example). The advantages of either approach are outlines in this passage:

If you start off the rsync daemon through your inet daemon, then you incur much more overhead with each rsync call. You basically restart the rsync daemon for every connection your server machine gets! It's the same reasoning as starting Apache in standalone mode rather than through the inet daemon. It's quicker and more efficient to start rsync in standalone mode if you anticipate a lot of rsync traffic. Otherwise, for the occasional transfer follow the procedure to fire off rsync via the inet daemon. This way the rsync daemon, as small as it is, doesn't sit in memory if you only use it once a day or whatever. Your call.
Let's assume for the following that you like myself are going to run rsync via inet. All modern Linux distributions have necessary hookup already done for you, and all you need to do is to open appropriate UI and enable rsync server. However, just in case, here is command-line configuration instructions from rsyncd.conf manual page:
When run via inetd you should add a line like this to /etc/services:
rsync 873/tcp
and a single line something like this to /etc/inetd.conf:
rsync stream tcp nowait root /usr/bin/rsync rsyncd --daemon

Create config file /etc/rsyncd.conf like that (it is assumed that "/ext/BACKUP" is the root of your backup area):

log file = /ext/BACKUP/log/rsyncd.log
pid file = /var/run/rsyncd.pid
lock file = /var/run/rsync.lock
secrets file = /etc/rsyncd.scrt
auth users = rsync
read only = no
transfer logging = yes
list = yes
[MyModule]
   path = /ext/BACKUP/mymoduledir
   comment = Description of what this backup location is for

Create all necessary sub-directories of /ext/BACKUP (including "log") and change ownership of all "modules" directories like /ext/BACKUP/mymoduledir to 'nobody':

#chown nobody /ext/BACKUP/mymoduledir

Create password file /etc/rsyncd.scrt with one line only which looks like that:
rsync:password
('rsync' can be any 'user' name, see 'auth user' configuration option above).
Change access mode of this file to 0600 :
#chmod 0600 /etc/rsyncd.scrt
If external access (from outside of your local network) is required, you can change default port 873 to something else and/or open this port in your firewall or router.

Client configuration (executed by any user who has read access to files being backed up)

Create file /etc/rsync.p (if acting as root) or ~/.rsync.p (if regular user) with just one line which is password from server configuration section. Change its mode to 0600.
You can now start backup job by executing this command:
```
rsync -az path_1 ... path_N --password-file /etc/rsync.p rsync@server::MyModule
```
Where path_1 ... path_N are all directories you want to copy to MyModule backup location.
Note however that (a) rsync attaches a special meaning to paths which end with a slash, like '/home/user/', basically interpreting them as '/home/user/*'; this is most useful when backing up just one path, and (b) for every path in a list, special sub-directory will be created under /ext/BACKUP/mymoduledir, which corresponds to the last component of the path (after special treatment of end-slashes is taken into account, of course). It is your responsibility to make sure these do not overlap.
If you want to automate regular backups, create a cron job like that:
30 2 * * * /usr/bin/rsync options as above

As another example, under Cygwin you use command like this to back up all files in all drives while excluding some Windows directories:

rsync -az /cygdrive/c /cygdrive/d \
   --exclude 'System Volume Information/' --exclude /c/WINNT/ \
   --delete-excluded rsync@rsync.myhomeserver.com::laptop

Here is why this approach, while useful, need certain rework to provide true backup solution:

There is no way to save to the server file owner information. All files are saved as created by "nobody" (you can change this default, but you cannot force server to save original file ownership data);
If directory (in the original location) has restricted access, server might not be able to back up files within this directory (you can force server to relax permissions, but then you would have lost original permissions info);
While detailed log file can be saved (to this end we suggested option 'transfer logging' above), there are no (good) tools to analyze it and report backup failures, especially for backups which are run in the background by cron.

Update (22-06-08). It seems like I rushes to judgment too fast when I claimed above that "all modern Linux distributions have necessary hookup already done for you, and all you need to do is to open appropriate UI and enable rsync server". In fact, Ubuntu has no such configuration pre-installed; you have to install inetutils-inetd or xinetd and: - in case of inetd, configure it manually as specified above; - in case of xinetd, create new file "rsync" in directory /etc/xinetd.d :

service rsync
{
        disable         = no
        socket_type     = stream
        wait            = no
        user            = root
        server          = /usr/bin/rsync
        server_args     = --daemon
        log_on_failure  += HOST
        instances       = 2
}

(from here) and do "/etc/init.d/xinetd restart".

Fixing Perl installation under Cygwin

2008-05-12T16:02:00.003-04:00

I discussed on multiple occasions before various issues arising from Cygwin upgrade, but there is one problem I fixed multiple times but never cared to document properly.

Perl built-in extension installation tools rely on existence and/or executability of certain files; this is tested with Perl test operators like 'if -r <filename>". As specified in the documentation, this operator tests that "File is readable by effective uid/gid.". No doubt that this test simply matches file permissions as reported by 'stat'' system call with file owner and current user id.

This seems sane enough, but the trouble is, Windows-centric notions of "Administrative rights" cannot be reliably mapped to Cygwin world. I always work in Windows as "myself," but with full Administrative rights, but Cygwin cannot possibly appreciate that I can access each file as if I were "Administrator" even though I am not one. As a result, multiple files which are created with permissions like that:

-rwxrwx---+ 1 Administrators SYSTEM 470528 May  1  2007 /usr/bin/bash.exe*

fail Perl's "-r" test, unless of course one wants to actually login as "Administrator", though I can still read and execute them.

While it is beyond me why Cygwin should create such peculiar permissions, here is a quick patch which should resolve most, if not all, problems related to installing Perl extensions.

chmod a+x /usr/bin/perl.exe
cd /usr/lib/perl5/5.8/ExtUtils
chmod a+x,a+r typemap xsubpp .
cd /usr/lib/perl5/vendor_perl/5.8/ExtUtils
chmod a+x,a+r xsubpp .

Best DVD player

2008-05-03T14:57:00.001-04:00

DVD players are cheap these days, but not all of them are born equal. The best player is my humble opinion is DVP3960, which can play virtually all known types of video files, and nearly all DVD writable media.

The only think which is worth to know (and this is the reason I am publishing it here) is so-called "region-free" hack. Copying from http://forum.videohelp.com/topic331294.html,

Power Up the unit with NO Disc in the tray.
Open the tray
Press the SETUP Button on the remote control
Navigate to the PREFERENCES page using the Right Arrow Key
Press the DOWN ARROW one time to select
Press the 1 button on your remote control
Press the 3 button on your remote control
Press the 8 button on your remote control
Press the 9 button on your remote control
Press the 3 button on your remote control
Press the 1 button on your remote control
The current Region Code Setting will display
Use the UP/DOWN Arrow Keys to select the region required or '0' for All Regions
Press the PLAY Button on the remote control

Dealing with "native" Windows encoding

2008-04-28T10:17:00.001-04:00

Microsoft Windows and other Microsoft utilities, like Microsoft Office, use encoding "UTF-16LE" by default; if they offer you multiple choices of encoding, they call it simply "Unicode". If the goal is to generate Unicode files which could be opened by all Microsoft applications, these better be in UTF-16LE.

Multiple language and libraries offer built-in conversion to UTF-16LE; however, one must be aware of two potential problems with that: (1) standard 4-byte header that Windows expects (and writes on output), and (2) potential problem with built-in DOS line ending mode ("text mode"); files must be written in "binary" mode.

Proper way to create UTF-16LE file in Python would be this:

fh = open ( "Test.txt", "wb" )
fh.write ( "\xff\xfe")
fh.write ( u"Проверка\r\n".encode("UTF-16LE" ) )
fh.close()

My first bugzilla bug report

2008-04-21T19:25:00.001-04:00

Just submitted my first ever Firefox bug report: bug 430140

"HTML source as simple as that:
<html>
<body style="direction: rtl;">
<img src="http://www.freeiconsweb.com/Icons/16x16_File_icons/Computer_File_001.gif"> First<br>
<img src="http://www.freeiconsweb.com/Icons/16x16_File_icons/Computer_File_001.gif"> Second<br>
</body>
</html>
(use any image you want) causes two lines to be inconsistent: in the 1-st line
image is on the right side of text (correct RTL behavior), but on the 2-nd line
it is on the left size (seems to be inconsistent with RTL requested) while line
itself is right-aligned.

Attempt to separate lines with techniques other than "<BR>" solves the problem
and restores consistent RTL behavior.

Even if I am wrong about correct behavior, there is no reason these two lines
should display inconsistently.

Note: while MSIE and Safari display this correctly, Opera mysteriously
(mis-)behaves consistently with Firefox.

ImageMagick and RMagick

2008-04-12T13:53:00.003-04:00

There is now a "standard" (free, BSD-style) image manipulation library ImageMagick. As it is pretty customary these days, it provides bindings for many popular higher-level languages, including Java, .NET, PHP, Perl, Python and Ruby. Ruby binding is known as RMagick.

One problem with ImageMagick is that it is notoriously difficult to install. The root of this problem may be that library insists on integration with X-server libraries (and indeed provides some X-client capabilities, though interestingly enough not supported on Windows).

(As a result, people are coming up with various ways to make a smaller library with restricted functionality, e.g. mini-magick. Unfortunately, there seems to be no documentation available, other than this tutorial, so I didn't want to spend time researching this)

Cygwin does include ImageMagic (you'll need at least 2 cygwin packages for ImageMagic: image-magick and libMagick-devel and various dependents for some reason not picked up by cygwin setup utility, e.g. libXft-devel and libbz2-devel), but all my attempts to first install ImageMagic and then RMagic failed (perhaps bundled version was too old, though judging from version numbers it was ok). ImageMagick web site includes pre-built version of all libraries for cygwin, but it appears to be very poorly built, includes many hardcoded paths from original build machine, etc.

So, the only option was to install ImageMagick from the sources, and it succeeded, after I explicitly excluded Perl bindings

./configure --without-perl

(be aware that built could take several hours to complete). After that, RMagick can be installed with command as simple as:

gem install rmagick

To try it out, you can use this program; try to uncomment line "canvas.display" near the end and look at the result. This didn't work well under Cygwin at all (though displaying same image just read from file worked fine).

Library itself is very powerful indeed. Documentation is good and detailed. One interesting thing that I discovered is that library has no less than 3 different methods to resize an image: "to sample", "to scale", and "to resize". You can compare all three by picking some random picture (sufficiently large) and running on it program like that:

#! /usr/local/bin/ruby -w

require 'rubygems'
require 'RMagick'
include Magick

imgfile = ARGV[0]
img = (Image.read imgfile)[0]

factor = 0.25
osize = File.size imgfile
puts "Original file #{imgfile} : #{osize} bytes, factor=#{factor}"

[:sample, :resize, :scale].each { |resize_alg|
  t = Time.now
  img_resized = img.method(resize_alg).call factor
  dt = Time.now - t
  fname = "test_#{resize_alg}.jpg"
  img_resized.write( fname );
  size = File.size fname
  puts "#{sprintf("%-6s",resize_alg)}: #{dt.to_f} secs, #{size} bytes =\
 #{sprintf("%.2f",100*size.to_f/osize)}%"
}

On my machine, it generated this output:

sample: 0.012 secs, 258985 bytes= 11.08%
resize: 1.287 secs, 227424 bytes= 9.73%
scale : 0.469 secs, 222147 bytes= 9.51%

Which is mathematically interesting: apparently, "resize" and "scale" provide some smoothing, therefore reducing size of the compressed image; "sample" is by far the fastest way to resize, but by just picking some more or less random pixels from the original image it generated result which is not only visually worse, but is harder to compress.

Global $PATH with Debian

2008-04-04T17:17:00.005-04:00

Well, one of the constant sources of trouble for nowadays UNIX users is continuous contradiction between UNIX command-line-based roots and modern "Windows-like" desktop environments.

Let's say I want to add new directory to "global" system $PATH. That is, I want all applications, no matter how started, from the command line or from GUI, no matter by what user, is to be able to find executables in certain directory, among others like "/usr" or "/usr/local/bin".

Interestingly, there is no simple or at least standard way of accomplishing this simple task.

On a very basic level, there are two main "sources" of the environment

1. Shell initialization file, system-wide or per-user. For "bash", these are:

/etc/bash.bashrc
/etc/profile
~/.bashrc
~/.profile
~/.bash_profile

It is important that all files from this list which have "profile" in their names are only invoked by "login" shells; that is, only for shells which are invoked when user logins remotely to the system or for shells started with "--login" (or just "-l") command line options.

2. Various "display managers", like GDM = "Gnome Display Manager".

Now, here is a tricky part: I could not figure it out how GDM sets it environments. Environment itself which comes out from GDM is actually identical to that set up in file /etc/login.defs, but changing said file appears to have no effect.

However, there is another file which GDM appears to honor: /etc/environment (note that this is NOT a bash script, but a simple configuration file). And yes, you can add PATH definition there and GDM will use it with no interference.

The only problem is that GNOME (and all other modern desktops) have a feature "run as root", and that requires different environment (again, I believe default one is identical to one defined in /etc/login.defs).

Given this situation, there are two basic strategies to define your system $PATH. You can of course customize both to fit your needs.

Approach 1 is described here.

The basic idea is that if there is a local user-defined executable file ~/.xsession, GDM will invoke it when firing up default user session. Therefore, this ~/.xsession file

#! /bin/bash --login
exec x-session-manager

makes sure windows environment is consistent with login environment.

Of course, you can change header to simply "#! /bin/sh" and set any environment you want in the same file before invoking x-session-manager.

Approach 2

Set your global $PATH in /etc/environment, but then make sure it is good both for regular login and applications requiring super-user privileges. Here is mine:

PATH=/usr/local/bin:/opt/jdk/bin:/usr/bin:/bin:/usr/bin/X11:/usr/games:/sbin:/usr/sbin

You can and probably should customize your bash configuration files to set local $PATH for user (by default, file ~/.bash_profile prepends "~/bin" to $PATH)

Volume Control in WIndows XP

2008-03-22T07:48:00.003-04:00

This sounds like a trivial advice, but if you see volume control icon disappear from your taskbar notification area in Windows XP, you can start volume control application by using command,

$SYSTEMROOT\system32\sndvol32.exe

Installing Mono with yum

2008-03-18T16:59:00.006-04:00

Creators of Mono so far haven't made it very simple to install; precompiled binaries exist only for few not the most popular distributions like Red Hat Enterprise 4 or OpenSuSe 10.3. But even there, it is a bit of a pain to download, verify dependencies and install all required packages.

One tool suggested to help somewhat with with problem is yum. Today we will talk about what it takes to use yam on Red Hat Enterprise 4 to install latest Mono release.

Note that at any point during that process it may turn out that your system is missing some requirements which mine didn't; it is assumed you know to to locate missing packages on whatever media you are using and install them.

We begin by installing certain prerequisites of yum. All of them are python packages, and all of them should be compatible with python 2.3.4 which comes with RHE4; upgrading to a newer python version may be possible, but not recommended since yum also depends on python package rpm, which is part of RedHat distribution.

urlgrabber. cd /tmp
wget http://linux.duke.edu/projects/urlgrabber/download/urlgrabber-3.1.0.tar.gz
tar -xvzf urlgrabber-3.1.0.tar.gz
cd urlgrabber-3.1.0
python setup.py build
python setup.py install
ElementTree. Using http://effbot.org/media/downloads/elementtree-1.2-20040618.tar.gz, same procedure as above;
cElementTree.
Using http://effbot.org/media/downloads/cElementTree-1.0.5-20051216.tar.gz, same procedure as above.

Now we are ready to install yum. I used version 2.6.1, since according to this, it is the latest version compatible with python 2.3.X.

cd /tmp
wget http://linux.duke.edu/projects/yum/download/2.6/yum-2.6.1.tar.gz
tar -xvzf yum-2.6.1.tar.gz
cd yum-2.6.1
make DESTDIR=/
make DESTDIR=/ install

Make sure yum module loads properly, and fix problems if there are any :

[root@myRHE4 ~]# python
Python 2.3.4 (#1, Jan  9 2007, 16:40:18)
[GCC 3.4.6 20060404 (Red Hat 3.4.6-3)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import yum
>>>

Create info about mono repository:

cd /etc/yum.repos.d
wget http://www.go-mono.com/download-stable/rhel-4-i386/mono.repo

You're ready to begin installation! Here is what I installed:

yum install mono-complete
yum install gtk-sharp2-complete
yum install mono-basic
yum install mono-debugger

One last item - optional, but nice - it is to make sure Linux kernel can run your Mono applications "natively" (like Windows), without using "mono" every time. Following these directions, create script like that:

#! /bin/bash
if [ ! -e /proc/sys/fs/binfmt_misc/register ]; then
    /sbin/modprobe binfmt_misc
    # Some distributions, like Fedora Core, perform
    # the following command automatically when the
    # binfmt_misc module is loaded into the kernel.
    # Thus, it is possible that the following line
    # is not needed at all. Look at /etc/modprobe.conf
    # to check whether this is applicable or not.
    mount -t binfmt_misc none /proc/sys/fs/binfmt_misc
fi

# Register support for .NET CLR binaries
if [ -e /proc/sys/fs/binfmt_misc/register ]; then
    # Replace /usr/bin/mono with the correct pathname to
    # the Mono CLR runtime (usually /usr/local/bin/mono
    # when compiling from sources or CVS).
    echo ':CLR:M::MZ::/usr/bin/mono:' > /proc/sys/fs/binfmt_misc/register
else
    echo "No binfmt_misc support"
    exit 1
fi

and run it as root (or append to file /etc/rc.local as suggested) To verify your installation, use program "Hello, World!" from here :

using Gtk;
using System;

class Hello {

static void Main()
{
Application.Init ();

// Set up a button object.
Button btn = new Button ("Hello World");
// when this button is clicked, it'll run hello()
btn.Clicked += new EventHandler (hello);

Window window = new Window ("helloworld");
// when this window is deleted, it'll run delete_event()
window.DeleteEvent += delete_event;
               
// Add the button to the window and display everything
window.Add (btn);
window.ShowAll ();

Application.Run ();
}

// runs when the user deletes the window using the "close
// window" widget in the window frame.
static void delete_event (object obj, DeleteEventArgs args)
{
Application.Quit ();
}

// runs when the button is clicked.
static void hello (object obj, EventArgs args)
{
Console.WriteLine("Hello World");
Application.Quit ();
}
}

To compile and run:

gmcs helloworld.cs -pkg:gtk-sharp-2.0
./helloworld.exe

Installing VMWare tools under Debian

2008-03-07T09:38:00.006-05:00

VMWare tools depend on specific kernel version, and thus upon upgrading the kernel it is necessary to upgrade them as well. This process is a bit painful and not without its caveats, so here is a brief description of steps involved.

Make sure you have correct version of kernel headers installed. Under Debian, this is done with the command:
# apt-get install linux-headers-2.6.18-6-686
where version string is reported by "uname -r".
While traditional location for linux header files was under /usr/src/linux, Debian puts them under /usr/src/linux-headers-2.6.18-6-686/include (you will need this directory later in the installation process)
Note: Aforementioned apt-get command would also create directory /usr/src/linux-headers-2.6.18-6 with platform-independent files.
Go to virtual machine settings and specifically enable "shared folders", if not already enabled (you do not need to add any such folders)
Now select in VMWare menus: VM => Install VMware tools.
In a few seconds, VMWare will mount VMWare tool installation "CD" (you will probably see CD image on the desktop and top-level folder opened). Ignore '.rpm' file and unpack '.tar.gz' file under '/tmp':
# cd /tmp
# tar -xvzf /media/cdrom0/VMwareTools-6.0.2-59824.tar.gz
Start installation process:
# cd vmware-tools-distrib
# ./vmware-install.pl
It will proceed for a while before beginning to ask questions about where to install files. Accept all of the suggested defaults.
After all the files are installed, installer asks if you want to launch VMWare configuration tool /usr/bin/vmware-config-tools.pl . Say "yes" (the default)
Here comes the only time when you (might) need to override default. When asked about "C header files that match your running kernel", specify directory "/usr/src/linux-headers-2.6.18-6-686/include" (or similar) from step 1.
Agree to install "experimental" feature "Virtual Machine Communication Interface".
When selecting graphics mode, you screen might re-size and flicker for a few seconds.
And the end, configuration utility shall start the services and report successful installation. Make sure that status line at the bottom of VMWare application window says "VMware tools installed successfully".

Configuration utility might suggest to switch to vmxnet for network support, like that:

/etc/init.d/network stop
/sbin/rmmod pcnet32
/sbin/rmmod vmxnet
# /sbin/depmod -a (may be unnecessary)
/sbin/modprobe vmxnet
/etc/init.d/network start

As part of the installation, installer overrides your existing X.org configuration file. You have previously configured your scroll mouse, these changes are now lost (though there should be a backup file /etc/X11/xorg.conf.old.0 available).
To make scroll mouse work again, restore these changes to /etc/X11/xorg.conf :
Option "Protocol" "ImPS/2"
Option "Emulate3Buttons" "false"
Option "Buttons" "5"
Option "ZAxisMapping" "4 5"
(from http://communities.vmware.com/thread/107816)
Shut down and them power up viertual machine to enable changes to xconf.org to take effect and to make sure VMware services are properly restarted (do not just 'reboot')
Upon successful startup, test scroll mouse and copy/paste between host and guest (if enabled in virtual machine properties, of course), and use /usr/bin/vmware-toolbox to make further adjustments to the configuration.

If everything went fine, congratulations! You have successfully installed/upgraded VMware tools. Till next kernel upgrade! IMPORTANT: If something didn't work and you must try again, do not just re-run vmware-config-tools.pl or vmware-install.pl . Begin from step 3 !
Update (24-Jun-2008). If when invoking "install VMware tools" you get CD mounted with some garbage (instead of two installer files it is supposed to contain), then: (1) "Eject" this CD (e.g., with desktop CD icon using RMB); (2) Go to VM => "Cancel VMware tools install"; (3) Go to VM => "Install VMware Tools" again. This time it should work.

cal.exe under Cygwin

2008-02-01T10:54:00.001-05:00

Upgrading Cygwin is never free of surprises, big or small. This time, following an upgrade, I noticed that decades-old utility "cal" (display a Gregorian calendar) suddenly disappeared.

Searching Cygwin-related sites took me a while, since quite obviously queries like "Cygwin cal" or "cygwin Gregorian calendar" bring up thousands of false positives.

Finally, I found a posting on Cygwin mailing list saying that for whatever reason cal.exe, along with some other utilities which used to be part of package "cygutils" (installed by default) now moved to (supposedly new) package "util-linux", so it has to be installed (I don't actually think it has any other useful utilities except "cal").

OK, one more problem solved... What is awaiting us at the next upgrade?

Russian subtitles in mplayer

2007-12-17T10:57:00.002-05:00

After a few attempts, I successfully made Russian subtitles to work correctly in mplayer. Here how it works:

Download Cyrillic bitmap fonts from mplayer site (there are other collections there that might work as well);
Unpack anywhere you want;
Edit font.desc file, removing references to non-existing font files;
Download (for instance, from here) or prepare subtitles (I used .srt file) in Windows (1251) encoding;
Run mplayer like that:
gmplayer -font <full path to font.desc> -sub <full path to .srt file> ...
Don't forger that mplayer saves its options in ~/.mplayer directory and re-uses them by default, so on your next invocation either use your "default" parameter like -font /usr/local32/share/mplayer/font/font.desc, or remove relevant files from ~/.mplayer .

TRAMP in Emacs 22.1

2007-10-26T13:59:00.001-04:00

It always feels kind of bad when after upgrading beta-version to a next stable release something stops working.

For nearly 6 years after release of Emacs 21.1 I used to compile my own "beta" version of upcoming Emacs 22; this is primarily because new Emacs finally promised to implement proper support of Unicode. For about 4 or 5 years these releases always were sufficiently stable to use in everyday work; one of the most exciting features of Emacs 22 (unrelated to Unicode) was TRAMP, which came to replace outdated ange-ftp to provide transport-independent support for remote file editing.

I must say that in my situation when I almost always work with 2-3 (or more) computers with different OS's available through different means (ftp, ssh, telnet) this feature is extremely useful for me; no other utility I am aware of provides such extensible and configurable transparent access to remote file systems....

So you can understand my disappointment when after installing now-"stable" version Emacs 22.1 (released in June 2007), I found TRAMP to be seriously broken, in more than one way.

First problem I have encountered was lack of file completion; normally, pressing TAB button (once or twice) while entering file name shows possible completions or completes file name right away (if completion is unambiguous). This stopped working for good in Emacs 22.1 / TRAMP 2.0.55 .

Here is what I found out...

The gist of this specific problem is in fact Windows-specific (this is probably why no one noticed). Lisp function tramp-file-name-handler (file tramp.el) has the following logic:

     ;; When we are in completion mode, some operations shouldn't be
    ;; handled by backend.
    ((and completion (memq operation '(expand-file-name)))
     (tramp-run-real-handler operation args))
    ((and completion (zerop (length localname))
          (memq operation '(file-exists-p file-directory-p)))
     t)

This says that when we invoke tramp-file-name-handler with method = expand-file-name, TRAMP delegates to "default" expand-file-name. This was perhaps done as an innocent attempt to improve performance and it does not as and of itself create problems under Unix; however, under Windows "regular" TRAMP names like "/user@machine:path" get "expanded" to "Windows" names like "c:/user@machine:path" (or other drive letter). This "expanded" non-existent path then gets passed to subsequent calls blocking successful file completion.

One solution is simply to block this "delegation":

     ((and nil completion (memq operation '(expand-file-name)))
     (tramp-run-real-handler operation args))
    ((and completion (zerop (length localname))
          (memq operation '(file-exists-p file-directory-p)))
     t)

This makes TRAMP to work, but it is indeed quite slow.

Better solution is to modify file completion. We create new function

(defun expand-file-name-noprefix (name &optional default-directory)
 (let ((std-result (tramp-run-real-handler 'expand-file-name (list name default-directory)))
   (win-prefix-re "^[a-zA-Z]:"))
   (if (or (not (string-match "^/" name))
       (not (string-match win-prefix-re std-result)))
   std-result
     (substring std-result 2))))

which deletes "<drive_letter>:" prefix from "expanded" name, if it seems unwarranted; then we replace code in tramp-file-name-handler with :

     ((and completion (memq operation '(expand-file-name)))
     (tramp-run-real-handler 'expand-file-name-noprefix args))
    ((and completion (zerop (length localname))
          (memq operation '(file-exists-p file-directory-p)))
     t)

This solves problem #1, and makes Unix and Windows versions behave consistently.

However, there is another problem.

File "handlers" can receive third argument (first two are file name and directory name), which is a "filter" to apply (e.g. to feasible completions). Previously in beta versions of Emacs 22 this argument was empty, but in 22.1 file-name-completion method gets passed a filter "file-exists-p", which seems quite meaningless in my opinion: when we are asked to generate list of "completions", doesn't it go without saying we are only talking about existing files?

The result of this "enhancement" in Emacs 22.1 is that after TRAMP backend generates list of possible completions, it is obliged to separately test each of these files for "existence". This substantially slows down the performance, most noticeably if trying to complete in a directory with many entries.

The good solution for this perhaps would be to keep "cache" of "known" files so that test for "existence" wouldn't involve any communication to the server. A quick but working patch is to simply suppress this third argument (code is given below).

One more item of interest (though not technically a bug) is that previously TRAMP could not "complete" very first name after host name (that is, in a name "/user@host:toplevel/nextlevel", we could complete nextlevel but not firstlevel). In Emacs 22.1, this has been fixed, but interestingly enough this fix is not enabled by default, but rather depends on minor mode "partial-completion". This mode is otherwise harmless, so it is a good idea to always use "(setq partial-completion-mode t)" with TRAMP.

With all these points taken into account, this is how my local .emacs file looks like now to make TRAMP work in Emacs 22.1 :

  (require 'tramp)
 (if (equal tramp-version "2.0.55") ;; bundled with Emacs 22.1.1
     (progn
   (setq partial-completion-mode t)
(defun tramp-file-name-handler (operation &rest args)
 "Invoke Tramp file name handler.
Falls back to normal file name handler if no tramp file name handler exists."
;;  (setq edebug-trace t)
;;  (edebug-trace "%s" (with-output-to-string (backtrace)))

 (let ((args
    (if (memq operation '(file-name-completion))
        (list (car args) (car (cdr args)))
      args)))
 (save-match-data
   (let* ((filename (apply 'tramp-file-name-for-operation operation args))
      (completion (tramp-completion-mode filename))
      (foreign (tramp-find-foreign-file-name-handler filename)))
     (with-parsed-tramp-file-name filename nil
   (cond
    ;; When we are in completion mode, some operations shouldn' be
    ;; handled by backend.
    ((and completion (memq operation '(expand-file-name)))
     (tramp-run-real-handler 'expand-file-name-noprefix args))
    ((and completion (zerop (length localname))
          (memq operation '(file-exists-p file-directory-p)))
     t)
    ;; Call the backend function.
    (foreign (apply foreign operation args))
    ;; Nothing to do for us.
    (t (tramp-run-real-handler operation args))))))))
))  ;; if tramp-version == "2.0.55"

(don't forget to also include somewhere function expand-file-name-noprefix as given above)

Appendix 1. The reason why the first problem was not caught in time might have something to do with TRAMP not working by default in Windows at all, not without some Windows-specific customization (this is unchanged from previous versions); even if Cygwin ssh is available, this won't work (not sure why).

Proper approach is to use free Windows utility plink (distributed along with popular free Windows SSH/telnet client PuTTy). Put it in a directory from Emacs's exec-path, and use this code in your .emacs :

(defvar w32
 (not (not (string-match "w32\\|win32\\|mswindows"
             (symbol-name window-system))))
"t if running under Windows GUI, nil otherwise"
)

(when w32
 (setq tramp-methods
   '(
     ("ssh"
      (tramp-connection-function  tramp-open-connection-rsh)
      (tramp-login-program        "plink")
      (tramp-copy-program         nil)
      (tramp-remote-sh            "/bin/sh")
      (tramp-login-args           ("-ssh")) ;optionally add "-v"
      (tramp-copy-args            nil)
      (tramp-copy-keep-date-arg   nil)
      (tramp-password-end-of-line "xy"))
     ("telnet" ;; Sorry, this is NOT working yet....
      (tramp-connection-function  tramp-open-connection-rsh)
      (tramp-login-program        "plink")
      (tramp-copy-program         nil)
      (tramp-remote-sh            "/bin/sh")
      (tramp-login-args           ("-telnet")) ;optionally add "-v"
      (tramp-copy-args            nil)
      (tramp-copy-keep-date-arg   nil)
      (tramp-password-end-of-line "xy")) ;; this is questionable
     ("rlogin"
      (tramp-connection-function  tramp-open-connection-rsh)
      (tramp-login-program        "plink")
      (tramp-copy-program         nil)
      (tramp-remote-sh            "/bin/sh")
      (tramp-login-args           ("-rlogin")) ;optionally add "-v"
      (tramp-copy-args            nil)
      (tramp-copy-keep-date-arg   nil)
      (tramp-password-end-of-line nil))))
         ;; (setq  tramp-debug-buffer t)
          (setq tramp-default-method "rlogin"))

Appendix 2. TRAMP Manual given extensive information on how to debug TRAMP. The bottom line is this: create a special Lisp file, let's say, "debugtramp.el" as follows:

(setq debug-on-error t)
(setq debug-on-signal t)

(require 'tramp)
(require 'trace)

(mapcar 'trace-function-background
 (mapcar 'intern
  (all-completions "tramp-" obarray 'functionp)))
(untrace-function 'tramp-read-passwd)
(untrace-function 'tramp-gw-basic-authentication)

(setq tramp-verbose 10)

And load it before problem might occur; afterward, save buffer output-trace (huge!) to a new file and investigate the problem.

Hebrew fonts

2007-10-24T07:05:00.002-04:00

The Culmus project aims to provide "Hebrew-speaking GNU/Linux and Unix community with a basic collection of Hebrew fonts for X Windows". In fact, Windows TTF version is also available.

Kdiff3 for cygwin

2007-10-07T19:38:00.003-04:00

Kdiff3 is an excellent utility for GUI-based comparison and merge. However, it is a bit cumbersome to build it properly. Default configuration assumes presence of KDE, so if you are not working from within KDE desktop, you are out of luck. However, there is also "QT only version" which only relies on QT libraries from Trolltech; it is primarily used to build Windows-version of the utility (also very good and highly recommended). If you want to build QT-only version of kdiff3 on a Unix-like system (including Cygwin), here is what to do (based on Kdiff3 version 0.9.92, released in April'07)

Make sure you have X development libraries, headers, and fonts installed (included in Cygwin);
Make sure you have QT3 installed (>= 3.3) (also included in Cygwin); set environment variable QTDIR, e.g. "setenv QTDIR /usr/lib/qt3"
Download Kdiff 0.9.92 source distribution;
Untar and go to sub-directory "src"
Edit Makefile.qt, primarily compilation option and include directories (you might want to replace -I$(QTDIR)/include with -I/usr/include/qt3)
make -f Makefile.qt
make -f Makefile.qt install

Enjoy!
UPD (11-March-08). Under Debian (kernel 2.6.18-6-686):

apt-get install libqt3-mt-dev qt++
export QTDIR=/usr/share/qt3
No need to change Makefile.qt

File I/O in Ruby

2007-09-24T15:31:00.000-04:00

Ruby FAQ which seems to constantly move around and is currently not available (Google cache is though) has a very useful section 10.6 Should I feel uneasy if I don't close a file :

There are at least four good ways of ensuring that you do close a file:
Use close (remembering to catch exceptions).
f = open "file"
begin
  f.each {|l| print l}
ensure
  f.close
end
Use the block form of open
File.open("file") do |f|
  f.readlines.each { |l| print l }
end
Use foreach
IO.foreach("file") {|l| print l}
Use readlines.
IO.readlines("file").each {|l| print l}

Copy DVD

2007-09-14T18:41:00.003-04:00

Well, the next best thing after using gmplayer is copying (protected) DVDs. Number of good utilities exist under Windows, e.g. free DVD Decrypter and non-free but very good Magic DVD Ripper. Here we will cover what is available under Linux.

First, very good review, and only slightly outdated, is available in Gentoo Wiki. In fact, from all the tools mentioned there I only tried the simplest one: dvdbackup.

Here is what to do:

Install libdvdcss ;
Install libdvdread . Run configure command like that: ./configure --with-libdvdcss-libs=/usr/local/lib
Get dvdbackup;
Goto "src" subdirectory;
(optional) Consider applying this patch;
Compile dvdbackup like that: gcc -I/usr/local/include dvdbackup.c -o dvdbackup /usr/local/lib/libdvdread.a /usr/local/lib/libdvdcss.a -ldl (Note: linking libdvdread.so does not work for some reason, but feel free to use libdvdcss.so)
Copy dvdbackup to /usr/local/bin or whatever
Use like that:
(to copy DVD)
dvdbackup -v 4 -M -i /media/cdrecorder -o ~/mytopdir -n my_name
(to get info)
dvdbackup -i /media/cdrecorder -I

P.S. To use gmplayer to watch a movie dumped to hard drive use command like that:

gmplayer dvd://1 -dvd-device ~/mytopdir/my_name

Unicode for Ruby

2007-06-11T17:38:00.002-04:00

I am trying to get comfortable using Ruby (having already written a fair amount of code in Perl, Python, and PHP). It is apparent that one of the serious problems with Ruby is lack of support for Unicode (in the current 1.8.x Ruby version). Class String in Ruby is just a byte string and nothing else.

A few things of interest:

Standard Ruby distribution does support Iconv library. Fox example, to convert a string of text from UTF-16LE (used by Windows) to UTF8, one can use:
```
require 'iconv'
utf8_str = Iconv.iconv( "UTF-8", "UTF-16LE", win_str ).join
```
(obviously, this is merely a wrapper around "standard" GNU iconv library. To see full list of currently installed encodings, use `iconv -l')
Also, there is a special option 'U' to standard String 'unpack' function, which effectively treats string as having UTF-8 encoding and then splits it into array of Unicode integers;
Using the above, there is an attempt made on rubyforge to create well-behaved Unicode String class purely in Ruby (the only way to install above seems to be to download the only source file directly). However, this project was obviously not finished and is barely used, given that last modification occurred 18 months ago and there seems to be no activity on project public forums. While many features of "regular" strings are supported and others could be added, it is not clear whether for example full regular expression support is feasible;
Interestingly, Ruby, being originally developed in Japan, was designed to deal with non-ascii encodings from its day one. In particular, Ruby supports the notion of "default encoding", which corresponds to "global variable" $KCODE . It can be set with command line option -K, e.g. `-Ku' for UTF-8. This variable can be assigned to at any moment to overwrite default value. The only trouble is, in my (Cygwin) Ruby and iconv installation, actual value of $KCODE which corresponds to utf8 is "UTF8", whereas iconv only has "UTF-8" (note the dash) and refuses to understand "UTF8". Assigning explicitly `$KCODE = "UTF-8"' does not help, as Ruby still resets $KCODE to "UTF8". It means that to use Ruby unicode library mentioned above I had to make changes to have it pass to Iconv "UTF-8" whenever previously it wanted to pass "UTF8";
In this e-mail thread, someone explains why he thinks Ruby is fine without any (more) Unicode support;
Here some Unicode-related changes planned for Ruby 2.0 are described;
However, here is the detailed description of what has already been done in Ruby 1.9, with an understanding that 2.0 will be a subset of 1.9; I cannot see Unicode ever mentioned there;

In a nutshell, all of the above means that at least before Ruby 2.0 is released, the best option to deal with Unicode strings is to make all String's (=byte arrays) used share same Unicode-compatible encoding, e.g. UTF-16LE . Then, we will need to always keep that in mind while dealing with such strings. Fox example, to split TAB-separated string, we would use:

words = line.split( "\t\0" )

Appendix 1. Some reference material on Ruby:

Two similarly brief introductions to Ruby: Programming Ruby: The Pragmatic Programmer's Guide and Ruby User's Guide (by language creator);
As part of Pragmatic Guide mentioned above there is a library reference and built-in reference; also, seemingly same documents are also available from here; I am not why if there is any difference or not;
The Ruby Language FAQ;
Ruby Cheatsheet;
Ruby-doc.org server features, among other things, standard Ruby documentation and customized Google search;
Ruby QuickRef;
Book Programming Ruby (2-nd edition) is not available for free (you can buy it as a regular book or download PDF file for $25), but number of useful excerpts are available, and judging from these excerpts book appears to be well-done and useful;
Server rubylearning.com offers some on-line tutorial, as well as (huge) PDF file download (link is only provided via e-mail, but see this for book and this for accompanying Ruby programs)
One more PDF book from the Scribd collection.

Appendix 2. On main Ruby web site, there is a special section Ruby From Other Languages. Especially interesting to us are possible confusion and incompatibilities compared to Python and Perl. A few things, however, are missing from the list of incompatibilities.

Ranges in Ruby are inclusive while in Python they are not; that is, in Python `range(2,10)' consists of all numbers from 2 to 9 while in Ruby similar n spirit notation `2..10' is used for all numbers from 2 to 10 inclusively. As a result, to, for example, drop the last character from a string in Python, we use `str[:-1]' while in Ruby this translates to `str[0..-2]';
Including function from a file in Ruby does not imply any consequences in terms on namespace, whatsoever. To take advantage of namespaces, you must use `module' explicitly;
Apparently Ruby does not use popular Perl regular expression library, so its regular expression syntax, while very similar to that from Perl, might not be fully compatible. E.g., Ruby does not support look-behind zero-width assertions;
On a positive side, Ruby does implement Python-style '%' operator for parameters substitution, though it is not used anywhere in the standard documentation; instead, Ruby-own parameter substitution ("x = '#{my_var + 57}'") is used. I prefer Python style, besides, it allows for formatted output;

(Happy Ruby programming! :-)

UPD [18-Jun-07]. Slashdot reviews new book "Practical Ruby Gems"

Compiling MPlayer on linux-amd64

2007-05-18T00:19:00.002-04:00

I now have a new desktop computer (DELL Precision Workstation 390n, RH Enterprise 4) and already spent a few days setting it up in various ways. Most of the time, it has been straightforward, but setting up MPlayer was much less than trivial.

1. 64-bit vs. 32-bit problems

(This section will not apply to you if your OS is 32-bit)

MPlayer must be compiled in 32-bit mode since it invokes proprietary Windows codecs and therefore some basic 32-bit environment must be available.

Make sure gcc -m32 can be used to compile 32-bit programs (you might also want to setup gcc4, see below; note that you will not need C++ support, plain gcc will suffice);
Create special new directory for 32-bit stuff, e.g. /usr/local32, set your $PATH and $LD_LIBRARY_PATH accordingly;
Install 32-bit version of libpng library, by setting export CFLAGS="-m32 -O2" and executing ./configure --prefix="/usr/local32";
Now you need 32-bit version of gtk-2 libraries. You can compile them from scratch, but in my case I only had to setup symbolic links (in /usr/local32/lib): libatk-1.0.so -> /usr/lib/libatk-1.0.so.0 libgdk_pixbuf-2.0.so -> /usr/lib/libgdk_pixbuf-2.0.so.0 libgdk-x11-2.0.so -> /usr/lib/libgdk-x11-2.0.so.0 libglib-2.0.so -> /usr/lib/libglib-2.0.so.0 libgmodule-2.0.so -> /usr/lib/libgmodule-2.0.so.0 libgobject-2.0.so -> /usr/lib/libgobject-2.0.so.0 libgtk-x11-2.0.so -> /usr/lib/libgtk-x11-2.0.so.0 libpango-1.0.so -> /usr/lib/libpango-1.0.so.0 libpangox-1.0.so -> /usr/lib/libpangox-1.0.so.0 libpangoxft-1.0.so -> /usr/lib/libpangoxft-1.0.so.0

2. Setting up MPlayer problems

Download appropriate "Binary Codec Packages", e.g. "essential" and "win32codecs" from http://www.mplayerhq.hu/MPlayer/releases/codecs/ , and install them under /usr/local32/lib/codecs;
Download "Skins" (at least one) from main download page http://www.mplayerhq.hu/design7/dload.html, and install under /usr/local32/share/mplayer/skins, and make symbolic link "default" to default skin, e.g. default -> Blue
Download one bitmap font from here and install per documentation, which is to say, unpack it under /usr/local32/share/mplayer and make symbolic link "font": /usr/local32/share/mplayer/font -> font-arial-14-iso-8859-1
Install a true type font in directory ~/.mplayer and make a symbolic link subfont.ttf, e.g: ~/.mplayer/subfont.ttf -> tahoma.ttf

3. Mysterious problems Following a successful compilation, mplayer was able to start and to display movie files, however, any attempt to change playback speed was causing a crash in function vo_update_osd, file libvo/sub.c. Here is what I did to solve the problem:

unset CFLAGS (or open a new shell); otherwise, you will disregard special compilation flags used by mplayer;
configure with the command: ./configure --enable-gui --cc='gcc4 -m32' --target=i386-linux --with-extraincdir=/usr/local32/include --with-extralibdir=/usr/local32/lib --prefix="/usr/local32" (using gcc4 is nor relevant to the crash, but was helpful with some other problems which appeared with gcc3. Versions used: gcc3 = 3.4.6, gcc4 = 4.1.1)
in a separate directory, configure as above but with additional option --enable-debug. and compile only one object: libvo/sub.o (you can run make to compile everything, there is no harm)
Copy libvo/sub.o from 2-nd directory to the 1-st and re-build MPlayer in the 1-st directory;
Now execute "make install" in the 1-st directory (with appropriate permissions) and enjoy!

4. Bonus track

Standard mplayer help suggests to use Command Line Option -slang to select subtitles, e.g. -slang en. In fact, for some reason this is not very reliable. It is in fact much more convenient to use option -sid . This way, trying sequentially -sid 0, -sid 1, -sid 2, ... you will find what you need. E.g., most often this is what you'd use to play DVD: gmplayer -sid 0 -fs dvd://1
Perhaps it is worth looking at new project Democracy (to be renamed to Miro), which is a multi-platform universal media player, implemented in Python and using Pyrex, xine and some other packages.

My favorite Firefox extensions

2007-05-15T23:10:00.004-04:00

	Adblock Plus	Ads blocker
	Dictionary Tooltip	Brings dictionary lookup results in instantaneous pop-up frame
	Fission	Progress bar in the address bar (Safari Style)
	Forecastfox l10n	International weather forecasts
	Gmail Notifier	Notifies you about new GMail
	Google Notebook	Direct access to Google Notebook while browsing
	Google Toolbar	Everyone knows what it is :-)
	IE View	View current page in IE
	Sage	A lightweight RSS aggregator
	Save As Image	Adds the ability to save a page, frame, or part of either as an image.
	User Agent Switcher	Lie to a web server about your browser :-)
	Web Developer	Adds a menu and a toolbar to the browser with various web developer tools
	Stylish	Allows for page-specific style customization
	Dictionary Search	Very basic tool that allows submission of selected word to any search engine
	Deepest Sender	Extension for blog posting, supports LJ/WP/Blogger

Also Firefox "buttons" :

BOThread for FireFox expand all threads in LiveJournal post
LJ/Mine Append &style=mine to LiveJournal URLs: this will cause the page to display in your LJ style instead of in whatever eyeball-searing layout the page's author has chosen.

P.S. [9-Jul-07] Two more useful extensions: LiveJournal Addons (see also here) and Locationbar².

P.S. [9-Jul-07]. Also add to the list of useful extensions Google Browser Sync.

P.S. [13-Sep-07]. Hebrew Calendar

P.S [18-Mar-08]. dragdropupload

P.S [16-Apr-08]. Multiple Tab Handler

P.S [05-May-08]. FireStox

Using getmail/procmail to peek into IMAP server

2007-05-08T10:37:00.001-04:00

I am using IMAP (in fact, Microsoft Exchange) server to read my mail. Now, I want some automated tool to take a sneak peek at all new e-mails which come my way and if these e-mails deemed "important", let's say, forward them to another e-mail account.

This could be easily done on a UNIX platform using getmail/procmail combination, but there is a fair amount of configuration required to achieve this goal, including patching getmail source in the process, so let me for the future reference underline the steps required and give some examples of configuration files I used.

1. Install procmail.,/p>

Currently, the latest version is 3.22 and source code can be copied directly from here (September 2001)

To install, you can do

make
make install

but before you do, make sure that your /bin/sh is in fact bash (install it if necessary) and not regular Unix sh; if not, replace first line of file initmake with #! /bin/bash.

After installation, make sure "man procmailrc" works for you (you will need it!) and if not, adjust your $MANPATH and/or copy all *.[15] files from "new" to an appropriate location.

2. Install getmail.

Currently, the latest version 4.7.4 is available here (April 2007). Note that you must have Python installed as getmail is Python-based.

Before you build/install getmail, make one change to file getmailcore/_retrieverbases.py : replace line

return self._getmsgpartbyid(msgid, '(RFC822)')

with line

return self._getmsgpartbyid(msgid, '(BODY.PEEK[])')

as suggested in this e-mail thread, in order to prevent your new messages from being marked as "read' ('seen' in IMAP language).

As a side note, alternative approach (instead of using 'PEEK' command) could have been to manually mark messages fetched as 'unseen'. Unfortunately, Python imaplib does not have an interface to do so like Perl module Mail::IMAPClient does, where there is a special function deny_seeing. We can however look at the implementation of this function:

sub deny_seeing {
  my($self, @msgs) = @_;
  if ( ref($msgs[0]) =~ /ARRAY/ ) { @msgs = @{$msgs[0]} };
  $self->unset_flag('\\Seen', @msgs);
}
sub unset_flag {
  my($self, $flag, @msgs) = @_;
  if ( ref($msgs[0]) =~ /ARRAY/ ) { @msgs = @{$msgs[0]} };
  $flag =~ /^\\/ or $flag = "\\" . $flag
     if $flag =~ /^(Answered|Flagged|Deleted|Seen|Draft)$/i;
  $self->store( join(",",@msgs), "-FLAGS.SILENT (" . $flag . ")" );
}

to understand that in Python this probably could have been done by calling doing something like that (untested):

imap.store (message_set, "-FLAGS.SILENT (\\Seen)" )

getmail, like any Python module, is installed in 2 steps:

python setup.py build
python setup.py install

3. Make new directory ~/.getmail and create there file getmailrc like that:

[retriever]
type = SimpleIMAPRetriever
server =  imap.myservername.com
username = myusername
port = 143
password = mypassword
mailboxes = ("Inbox",)
[destination]
type = MDA_external
path = /usr/bin/procmail
arguments = ('~/.getmail/procmailrc',)

(fill out proper IMAP server credentials). If you need help with getmail configuration file, see here.

4. In the same directory ~/.getmail, create new file procmailrc like that :

PMDIR=$HOME/.getmail
# Put ## before LOGFILE if you want no logging (not recommended)
LOGFILE=$PMDIR/log
# Set to yes when debugging
VERBOSE=yes
# Remove ## when debugging; set to no if you want minimal logging
LOGABSTRACT=all

:0: procmail.lock
* ^From:.+(Petya|Vasya|Kolyan|president@kremlin.ru)
!myemailaddress@gmail.com

:0
/dev/null

(Turn off debugging as per comments in the above file when satisfied). If you need help with procmail configuration files, use "man procmailrc".

5. Test your new setup as needed by using command:

getmail  --dont-delete --new

and then satisfied, add something like this in your crontable:

1,6,11,16,21,26,31,36,41,46,51,56 * * * * /usr/local/bin/getmail -q -l -n

You are all set!

Extracting MIME content from e-mail

2007-02-26T10:19:00.001-05:00

Even the best Web-based and standalone e-mail clients, like gmail or thunderburd, do not allow users to directly access MIME parts of an e-mail message (though some simpler clients like KMail do).

To solve this problem, many special utilities exist. One such utility is mimedump from Benjamin Elijah Griffin.

Get it here: http://www.cpan.org/authors/id/E/EL/ELIJAH/mimedump.75

Install all required CPAN packages, especially File::NFSLock;

and use it like that

mimedump.pl /tmp/output_dir message_file

Portable thread library

2006-12-21T04:12:00.001-05:00

Given how important multithreading programming has become, it is surprisingly difficult to find simple and usable portable C/C++ thead library.

In fact, this page claims there are only two such C++ libraries in existence: zthread and Boost.Threads.

zthread offers a clean, well-thought C++ design (claimed to be modeled after Java threads). Despite the fact that Cygwin was not mentioned as one of the supported platforms, it does compile and run under Cygwin with no visible (Cygwin-specific) problems. However,

There is a complete absence of any examples, demos, unit tests, tutorials, etc. The only documentation provided is doxygen-generated.
It appears that the implementation, rather than establishing a set of common wrappers over win32 and POSIX APIs (and perhaps others), is trying to more or less provide its own alternative implementation, perhaps having maximum platform-independence as one of the goals. As a result, library is quite volatile and does need a very active maintenance and user base, which does not seem to be there.
As a typical example, some users report apparent problems under VC++ 6.0 which no one is able to diagnose or to comment on (thus one needs Visual .NET to use it).
I spent long time trying to understand why my test program (under both Cygwin and Linux) was mysteriously crashing before I realized that zthread library is probably trying to do some very "intelligent" memory management, and as a result, all zthread-objects like mutex'es or threads must be dynamically allocated (via new); but then, you never need to release them...

Speaking of Boost.Threads, this is apparently just one piece of a huge collection of libraries - the latest release takes around 11M of source files - there is no documented way to build a single library separate from others.

So, is there a better alternative?

It appears that there is: it is POSIX Threads for Win32 project. They provide ready-to use libraries that can be (dynamically) linked with any Win32 application for (almost) complete POSIX thread support. See FAQ for long and interesting discussion on various ways to handle exceptions that arise from inside the library (in my experiments, I used pthreadVC1.lib)

Finally, a good pthread tutorial and documentation is available here.

Appendix. Sample C++ application implemented with zthread:

#include <string>
#include <iostream>
#include <zthread/Thread.h>
#include <zthread/Mutex.h>
  
ZThread::Mutex * output = new ZThread::Mutex ();

class ThreadExample : public ZThread::Runnable
{
 public:
   ThreadExample(std::string thread_name, size_t iterations)
      :   name_(thread_name), num_times_to_loop_(iterations)
   {}
  
   void run()
   {
      for (size_t i = 0; i < num_times_to_loop_; i++) 
      {
         output->acquire();
         std::cerr << i << " " << name_ << "\n";
         output->release();
      }
      output->acquire();
      std::cerr << name_ << " finished! " << std::endl;
      output->release();
   }
  
 private:
   std::string name_;
   size_t num_times_to_loop_;
};

int main()
{
   using namespace std;
   using namespace ZThread;
  
   try
   {
      Thread t1(new ThreadExample("Thread-1", 50));
      Thread t2(new ThreadExample("Thread-2", 50));
   }
   catch (const Synchronization_Exception& e)
   {
      cerr << e.what() << "\n";
   }
}

and pthread:

#include <string>
#include <iostream>
#include <pthread.h>

pthread_mutex_t output = PTHREAD_MUTEX_INITIALIZER;

class ThreadExample
{
 public:
   ThreadExample(std::string thread_name, size_t iterations)
      :   name_(thread_name), num_times_to_loop_(iterations)
   {}
  
   void run()
   {
      for (size_t i = 0; i < num_times_to_loop_; i++) 
      {
         pthread_mutex_lock (&output);
         std::cerr << i << " " << name_ << std::endl;
         pthread_mutex_unlock (&output);
      }
      pthread_mutex_lock (&output);
      std::cerr << name_ << " finished! " << std::endl;
      pthread_mutex_unlock (&output);
   }

   static void * prun(void * self) 
   {
      ((ThreadExample *)self)->run();
      return NULL;
   }
  
 private:
   std::string name_;
   size_t num_times_to_loop_;
};

int main()
{
   pthread_t  p1, p2;

   pthread_create(&p1, NULL, ThreadExample::prun, new ThreadExample("Thread-1", 50));
   pthread_create(&p2, NULL, ThreadExample::prun, new ThreadExample("Thread-2", 50));

   pthread_exit(NULL);
   return 0;
}

double to float conversion

2006-12-10T10:47:00.001-05:00

When we must convert float number to integer in C, we have two convenient functions at our disposal: floor and ceil. For some reason however, there are no natural counterparts to these functions when we deal with double -> float conversion.

I wrote a simple implementation of these utilities using existing C library functions frexp and ldexp:

float fdround ( double x, int isfloor )
{
  const int    Nf = 23; /* IEEE 754: http://en.wikipedia.org/wiki/IEEE_754 */
  double    m;
  int        exp, sx;
  if (0 == (sx = ((x < 0.0) ? (-1) : ((x > 0.0) ? 1 : 0)))) return 0.0;
  m = frexp ( x * sx, &exp );
  return (float)(ldexp ( (double)sx * ((isfloor ^ (sx == -1))? floor : ceil)
           (m * (1 << (Nf + 1))), exp - Nf - 1));  
}  
#define fceil(x) (fdround ( (x), 0 ))  
#define ffloor(x) (fdround ( (x), 1 ))

This implementation of course depends on the correct knowledge of the number of bits for mantissa as per IEEE standard, which is 23 for 32-bit floating-point numbers.

Here is a simple utility to test whether the above function generates correct numbers:

void test(double x)
{
  float f = ffloor(x), c = fceil(x);
  float a = 0.4 * f + 0.6 * c, b = 0.6 * f + 0.4 * c, y = (f + c)/2;
  assert ( f <= x && c >= x );
  assert ( a == c && b == f );
  assert ( y == f || y == c );
}

Enjoy!

Cygwin shell hangs upon upgrade

2006-12-09T12:56:00.001-05:00

I just updated my cygwin installation and it stopped working: on startup any cygwin shell hangs.

Quick Internet search revealed nothing, so a little research of my own was in order...

It appears that cygwin shell startup files insist that file /bin/sh be a copy of /bin/bash. This logic is coded into file /etc/profile.d/00bash.sh :

# Get here if missing, broken, ash, or old bash, so an update is needed.
# Use copy, not hard or symlink, since symlinks won't work from Windows cmd
# and a hardlink to a running shell can't be broken. Try in-place copy
# first, but fall back to --remove-destination in case /bin/sh has different
# ACLs than /bin/bash. Record the attempt in /var/log/setup.log.full.
echo "`date '+%Y/%m/%d %T'` /etc/profile.d/00bash.sh:" "Attempting to update /bin/sh.exe" >> /var/log/setup.log.full 2>&1
{ /bin/cp -fpuv /bin/bash.exe /bin/sh.exe ||
/bin/cp -puv --remove-destination /bin/bash.exe /bin/sh.exe
} >> /var/log/setup.log.full 2>&1

On the other hand, cygwin upgrade only changes /bin/bash and not /bin/sh, thus triggering the above code (and more) to execute, and this was apparently causing hang (I didn't look into where exactly).

Therefore, it this happens to you, reboot your machine, and immediately afterwards, before any cygwin applications can initiate, do this (or similar depending on where your cygwin installation is)

copy /B c:\cygwin\bin\bash.exe c:\cygwin\bin\sh.exe

do-while loops in Python

2006-10-16T06:12:00.002-04:00

It just occurred to me today that apparently Python does not have a "do-while" type of loop, or any equivalent. The most obvious usage for this would be :

f = open ( filename )
do 
   buf = f.read(blocksize)
   do_something_with_buf(buf)
while buf

suggestions on how to handle situation like this in Python vary, from using "break" in traditional "while" loop to avoiding (any form of) while altogether, using Python-style "for" instead.

Apparently, there is PEP (Python Enhancement Proposal) 315 pending since 2003 that would request just that, a "do-while" loop implementation. This was considered at some point as a possible candidate for Python 2.5, but was rejected as not that important, and this comment and recent discussion at python-dev mailing list revealed lack of consensus on simple, implementable, Python-style syntax for "do-while" loops.

It seems like therefore this will have to wait (at best) till "Python 3000" version of Python (a.k.a. "Py3k" or sometimes even "p3yk"), which is kind of sad.

BTW, this presentation has a nice overview of Py3k plans and ideas.

Ports blocked by RCN

2006-08-09T18:12:00.001-04:00

I recently inquired RCN (my Internet provider) about their policy on blocking port. I didn't really expect an answer, but after a period of silence they responded:

Port 80 inbound
Port 25 inbound
Port 25 outbound
Port 135
Port 137
Port 138
Port 139
Port 445

In fact, the reason I posted this inquiry to begin with was that I noticed that port 443 (https) is now being blocked, or so it seems. However, from their reply it seems like no. Well, maybe I am wrong somewhere. In any case, it is good to al least have an official list from RCN.

How to format numbers in Java

2006-07-26T17:57:00.001-04:00

It appears that formatting numbers in Java is quite a challenge. Here is what I found out so far.

If you simply need to output a real value with given number of digits, the simplest way to do so would be this (make sure to import java.text.*):

DecimalFormat form = new DecimalFormat ( "0.000" ); System.out.println ( form.format(-Math.sqrt(2)));

This will print

-1.414

The most important feature of DecimalFormat() is that you can take advantage of pre-defined "factory methods", by calling functions

form = NumberFormat.getXXXInstance(locale);

where XXX is ether empty, or one of "Integer", "Currency", or "Percent".

You can also use form.parse() to parse strings in corresponding format, and form.toPattern() to access pre-defined locale-specific patterns.

See this help page for more on formats and locales.

Alternate Data Streams (ADS)

2006-07-13T14:34:00.000-04:00

I just learned about "Alternate Data Streams" in NTFS, a feature which has apparently been available from day one, and I am shocked that I never, ever heard about this! Links:

via /.

Files and pictures hosting

2006-06-24T02:05:00.001-04:00

There is an interesting startup company Streamload that implements files, pictures and various media hosting in a way that could be potentially correct.

Namely, they allow you to both store files with a simple web-based file manager and selectively open access to these files for the public. The way it is done, essentially two completely different (Web) UIs are implemented; as of today, internal UI isn't bad (though does need a lot of polishing) while external UI is outright terrible.

Their pricing model is based on providing "unlimited" storage while restricting bandwidth which is also a sound business model. Storage space nowadays is so cheap that it practically does not count, while various other services guaranteeing "unlimited access" must compensate for it somehow (one popular "solution" is of course to limit your storage in a hope that it will indirectly limit bandwidth which is outright stupid). As for me personally, everything I might ever want publish online will be accessed very rarely so I am absolutely happy with bandwidth restriction.

That said, number of problems facing this service (and similar) is enormous; the most important question of course being whether it will survive in the more and more competitive marketplace. But, the good ideas outlined above make it worth checking periodically for the progress...

At the end, a few links of interest. I originally came across this site from Slashdot article referencing a PC World review of a few online file storage services. On the Internet, you can also enjoy older (February) review from PCMag, demo.com business overview of the company, both old and new blogs of company programmer(s), and perhaps most invaluably example of a user publishing many pictures and other media with this service.

Finally, one unrelated remark that could be more important than all of of the above: there are unconfirmed, but reliable reports that Google is preparing its own Web hosting platform.

UPD. Streamload is reviewed on digg.com.

UPD (10-Dec-06). Another interesting attempt at file hosting, www.bluetie.com .

Google Summer of Code 2006

2006-06-23T06:08:00.001-04:00

Google Summer of Code is another interesting initiative from Google which I haven't heard about before or at least haven't mentioned it in this blog. Very interesting list of FOSS projects which Google deems important enough for the community and ideas from the authors of each respective project on various ways to improve it.

Check, for example, ideas from GNU Compiler Collection makers or from Haskell.

printf() and 64-bit integers

2006-06-22T22:30:00.001-04:00

In the last post we talked about scanf(), so perhaps it would be proper now to discuss printf().

Good and reasonably brief manual page could be found here; then make sure to check Microsoft-specific formats here.

Some interesting features are there, e.g. you can use <'> format (single quote) to enable locale-specific number output (has no effect in C locale); you can also refer to specific argument by number, specify format width as a separate argument, and so on.

Let's consider one specific task to print some (large) 64-bit value. The basic problem is that you must know (1) what (integer) type to use to represent (at least) 64-bit integers, and (2) what printf() format does this type correspond to, if any.

On UNIX platforms long usually correspond to "machine word" (that is to say, has same size as pointer) so on 64-bit UNIX long is 64-bit and the following code:

printf ( "2^63 = %lu\n", 1L << 63 );

will print 2^63 = 9223372036854775808.

On 32-bit UNIX systems long is 32-bit but long long is 64-bit, and then the following code will work as expected:

printf ( "2^63 = %llu\n", (long long)1 << 63 )

(Gnu gcc understands 1LL, but I am not sure if other compilers do).

On Windows, however (with Microsoft compiler) approach is different and size of basic types does not change with CPU and so long is always 32-bit (same as int). Thus, in older Visual C++ systems you have to write:

printf ( "2^63 = %I64u\n", (__int64)1 << 63 );

... while in Visual .NET the above example with "long long" will work.

scanf

2006-06-06T11:56:00.001-04:00

For the first time since I learned C 18 years ago I noticed that the standard library function scanf (and friends) has been enhanced with many new formats and features.

E.g., the following code

scanf ("%s %[^\n]\n", name, args);

reads a "word" from the input (and assigns it to name) and then reads the rest of the line till end-of-line, irrespective of spaces and other symbols, and assigns it to args.

Very detailed description of scanf formats is available at Solaris 10 Reference Manual Collection pages.

There is also a brief introduction to the standard C library here. Wikipedia, as usual, has a worthy collection of links as well.

C++

2006-05-29T00:42:00.002-04:00

Well, I am now trying to learn a very new language for me... C++.

Let's make this clear: the notion of programming language has acquired a very new meaning in the last 10-15 years and now means much more than merely a language (which is more or less set of rules how to formally lay down the algorithm, like a human language is a way to semi-formally express one's thoughts). Rather, programming language must also have certain standard libraries, or standard bindings for number of important things such that:

Input/Output;
Charsets/Unicode;
Regular expressions;
Strings/Formats;
Network/Sockets;

and many more items like that.

Moreover, a language is to include some minimal native datatypes, such that arrays, lists, associative lists, etc.

In this sense, neither plain C or C++ are languages till we attach to them required libraries; one such library is GNU libc, which includes most, if not all, of items listed above.

Still, C+GNU libc is not a language since it does not possess all required datatypes which are either native or standard. That means that virtually any not-completely-trivial programming job in C must begin with writing some "utility" functions (such as expandable arrays) or at least adopting some 3rd-party external library. Good explanation of this problem with C (suggesting C++ as a better choice) could be seen here.

In this respect, C++ does possess a uniques feature that C does not: STL. (C++)+GNU libc+STL can be qualified as full-featured language and as such compared with Python, OCaml, Java, .NET, Haskell and other popular systems.

One immediate trouble that must be mentioned is the lack of central Internet resource documenting this (complete) language. Here is a list of most important resources available:

C++ language structure: The C++ resources network; good explanation of basic language constructs with source code examples. Look, for instance, at their page explaining templates;
GNU libc has official documentation .While it isn't bad, it's not perfect either. Many pieces of GNU libc can be better documented somewhere else; see for example very detailed UNIX sockets in C FAQ; there is also a good UNIX reference here.
STL is the most difficult thing to find a good documentation for. C/C++ Reference is good but very very brief; there is also somewhat outdated but still very much useful SGI STL documentation; See also A Tour of the Standard Library from Bjarne Stroustrup.
There is also GNU libstd++ documentation which is not really a documentation but a collection of pretty much random remarks that deal with some specifics of GNU implementation of ISO C++ standard as well as with other issues of interest;
The official C++ page of the language creator and standardizer Bjarne Stroustrup must be mentioned here;
Very classical C++ FAQ is very much up-to-date and has a bit of everything.

So, what do I think of C++ as a language?

Well, I am not impressed.

Very brief list of main problems I see is:

There is no simple and safe way to call C functions. Most libraries that C++ programmer is using are C libraries, and while calling C functions isn't difficult it often presents challenges copying between std::string and char* "strings", dealing with const, etc.;
Code tends to be tedious and hard to read. Things like that: std::vector<std::string>::iterator ii=my_arr.begin() do not cause me any pleasure to read or to write;
Lack of Java-style interfaces is killing me. Hasn't polymorphism always been considered part of OO programming paradigm?
At least with GNU g++ error messages are often meaningless and extremely long with plenty of reference to internal include files that I have no interest in;
In principle, wring in C++ you might not worry about memory allocations (unless you are explicitly calling new, which is rarely needed in small programs, why would you call delete?) but this might create a false sense of security; unlike all other relatively high-level language, you are writing programs without a protection of a garbage collector, and I am reasonably sure many programs are in fact leaking memory;
STL is often a poor replacement of complex data types from other languages. E.g., functors do in principle exist but are rarely usable; number of functions is very limited. Try, for example, to print a map sorted by value (trivial in Perl or Python) and see how much code you'll have to write to execute this quite trivial and not at all uncommon task. Or, how do you like testing whether map a has key key with this code: a.find(key)==a.end() ?
I may be stupid, but I fail to see how iostream-style formatted output is better that printf. How is this std::cout << std::hex << std::setw(8) << std::setfill('0') << ii << std::endl; easier than this: printf ( "%08X\n", ii ); (Well, actually C++ FAQ mentioned above has a direct response to that)

Of course, C++ does have certain advantages over other systems (e.g. usable debugger and calling C functions with ease) but that perhaps should be a subject of another post...

UPD. Found more C++ documentation, including the ISO standard as 776-page PDF file, here.

Bittorrent client software

2006-05-25T14:26:00.001-04:00

Dug a little bit on reach selection of Bittorrent clients to choose a good one for my Linux box; finally decided that QTorrent will suit me best for the time being. It is written in Python (making it simple to install and very much platform-independent) and UI is detailed while not congested. On the negative side, it appears not to be actively developed since about 2004, so this might be a temporary stop.

Among available alternatives are :

Opera 9 with built-in Torrent client; arguably this could be the most convenient solution except that Opera 9 is still in beta and not very stable; white it works (mostly) when you browse torrent client hangs it in a few hours because of "too many open files";

KTorrent has a lot of promise, but unlike QTorrent it is too actively developed, so it bypassed my old SuSe 9.0 system (KDE 3.1) long, long ago; even much older version 0.9 is still too new for it;

Azureus is Java-based so it can be installed and it works, but it is way too advanced for a regular user like me, with a lot of pop-up notifications, red-blue-green signs, and even automatic updates! Interestingly, this applications serves as an example how well-behaved FOSS Java-based application should be written; it can work both with SUN Java as well as with free GCC-based Java implementations.

UPD(22-March-07). There is an interesting command-line bittorrent client ctorrent. It can be configured under OpenBSD, though to do so I had to use "export LDFLAGS=-lcrypto". Do not forget to also open ports 2106-2706 which it is using by default.

UPD(15-May-07). In fact, ctorrent does not work well, it apparently has some disk access problems, perhaps only if saving torrent with too many files, perhaps only on OpenBSD. In the meantime, Opera 9 is long out of "beta" and though some problems were addressed its torrent client is I think still unusable. At the same time, my prediction above on QTorrent materialized : now most trackers refuse to talk to it citing too old version of protocol. So, I am back to using Azureus, which appeared to be not all that bad, and though I have a new machine now and supposedly may try using KTorrent, I just may stay with Azureus for the time being.

Acrobat 7

2006-05-23T07:12:00.001-04:00

I almost exclusively use Acrobat (editor) to nicely fill out PDF-based forms; in Acrobat, this is called "Text Box tool" and it often seems to me I am the only one using it for it is often very difficult to find it in otherwise exploding Acrobat menus.

It is even more difficult though to customize colors and fonts for inserted text. It used to be (in Acrobat 5) that font and color were part of "text box properties" (makes some sense, doesn't it?); since Acrobat 6 this is part of special "properties" toolbar; since Acrobat 7 this toolbar is no longer activated by default; you can search all you want for "font" and "color" in the menus but you will not see it anywhere till you activate this "properties" toolbar and select the text.

Very intuitive UI, isn't it? But then again, maybe I am the only one actually using it.

Installing Python + ZSI on Windows

2006-05-10T19:13:00.001-04:00

Now, who could ever think that in setting up my new laptop the most difficult thing would be to set up Python?

To be sure, Python is a very nice language, completely dynamic and OO, which allows you to implement a fast prototype for a complex object interaction. Then, you might want to re-implement in more "static" language like Java or just leave it Python if speed and reliability isn't among your first priorities.

However, another very nice feature of Python is a very well-done and well-supported Windows port, including a native Windows installer. There is a price to pay for this beauty: each Python release is implemented in specific Visual Studio C++/.NET version; e.g. all 2.4.* releases (latest stable release at this moment) are done in Visual Studio .NET 2003 (internal version = "7.1"); whereas 2.3.* releases are done in Visual C++ 6.0 (internal version = "6.0"). That said, the latest suite from Microsoft (right now) is Visual Studio .NET 2005 (internal version = "8.0"), and this is exactly what I have (by default) installed on my new laptop.

Praises to Python Windows port above notwithstanding, file msvccompiler.py, part of standard Python distribution, does not do the best possible job at detecting user's Visual Studio environment. It has not occurred to the author that the latest version it knows about (7.1) will sooner or later be superseded with a newer one; as as result, on my laptop an attempt to install any Python extension that contains C code fails with dubious message "The .NET Framework SDK needs to be installed before building extensions for Python"; message sure to puzzle someone who knows damn well .NET SDK is installed on his machine...

As a final remark, I must say that I am using Python for (effectively) RPC calls via TCP/IP using SOAP and Python extension called ZSI (along with mod-python on the server). I was using version 1.7 of ZSI, which only worked for me after applying simple patch to the client code.

Anyway, let me without further adieu present my sequence of actions:

Installed binary distribution of Python for Windows; latest stable release 2.4.3;
Downloaded and installed latest ZSI build 2.0rc2 (no C code so installed flawlessly); I noticed that client code has changed dramatically since 1.7 so that my patch may be no longer required;
Run test script. It appears to fail because the API (specifically function ZSI.client.Binding) changed in an incompatible way. What's more, there is no API to tell me version number, so there is no simple way to write client code compatible with both 1.7 and post-1.7 ZSI API. After a while, I solve this problem by parsing function documentation string Binding.__init__.__doc__;
Run test script. It fails complaining that it cannot load "xml.dom.ext.reader";
This is actually very peculiar, since Python is of course well-equipped with XML DOM parsers; but yes, I vaguely remember that indeed for some mysterious reasons ZSI depends on an external expat-based XML parser;
OK, I go ahead and download the latest source release of PyXML (I mistakenly think binaries are not available for Python 2.4 since this version of PyXML is rather old, but in fact they are);
Build fails with message "The .NET Framework SDK needs to be installed before building extensions for Python" (see above);
I try to modify file msvccompiler.py to convince it to use my installed version of Visual Studio. After a while, it does work and installation of PyXML succeeds;
Test script now crashes Python executable. This is perhaps related to incompatibilities of two dynamic runtimes that Python itself (7.1) and PyXML are trying to load;
I download source distribution of Python (2.4) and try to compile it from source using Visual Studio 2005. It builds simple python.exe and it crashes on startup, invoking debugger and stalling build;
I remove all previous installations of Python and install older Python version 2.3 from the scratch (binaries) along with Visual C++ 6.0 environment;
Following the steps described above, both ZSI (2.0-rc2) and PyXML now install successfully;
Test script fails somewhere in ZSI client code. An attempt to debug it reveals that function Binding::RPC is called from _Caller with (default) argument replytype=None, which then fails in parsing. An attempt to fix this (TC.Any()) improves the result a little bit, but not all that much. It appears that ZSI changed XML marshalling logic and thus I cannot have post-1.7 client and 1.7 server;
I try to install older version of ZSI (1.7 + my patch) above the previously installed (Python extension installation mechanism does not give me any simple way to uninstall); this results in empty SOAP message being passed to the server;
Desperate, I simply erase the sub-directory d:\Python23\Lib\site-packages\ZSI and reinstall ZSI 1.7;
Run the test script; finally it works.

Clock screensavers for Windows

2006-05-06T22:07:00.000-04:00

Spent a while today trying to find a good screensaver for my new (Windows XP) laptop (Dell Precision M60). My requirements are essentially very basic:

Screensaver must show time, preferably date, and as little else as possible;
Must be free to use and contain no AdWare of any kind;
Majority of screen estate must remain black during work of screensaver (it is screen saver after all);
Should take only minuscule amount of Windows resources.

Not all that much, ah? Yet hours of search only yielded two more or less reasonable options: Analog Clock Screensaver by Lunarmedia and Digital Clock Screen Saver by Xequte Software. Is there anything else I missed?

Useful FF download links

2006-04-30T22:09:00.000-04:00

This blog entry is useful for listing canonical locations to download all latest, Beta- and Alpha- Firefox releases.

Perl-alike "open2" in Haskell

2006-03-04T07:12:00.000-05:00

import System.Posix
import IO

-- similar to Perl open2
open2 :: String -> [String] -> IO (Fd,Fd)
open2 fileName cmdArgs = do
          (to_out,to_in) <- createPipe
          (from_out,from_in) <- createPipe
          pid <- forkProcess (do
              dupTo to_out stdInput
              dupTo from_in stdOutput
              closeFd to_in
              closeFd from_out
              closeFd to_out
              closeFd from_in
              executeFile fileName True cmdArgs Nothing)
          closeFd to_out
          closeFd from_in
          return (to_in,from_out)

Sun Studio 11, dbx, glibc and my old SuSe

2006-02-17T22:49:00.003-05:00

OK, after having heard SUN now offers Sun Studio 11 on Linux, I tried to download it.

This is of course a pathetic excuse for a software. Simply NetBeans-based IDE which merely invokes GNU compilers. There is also Motif-based "X-designer" ... Br... And yes, there is "sample" C++ GUI "project". Nonsense.

The only remotely useful piece of whole crap could be dbx debugger, but of course it fails to start on my system asking for newer version of GNU libc :

dbx: /lib/i686/libc.so.6: version `GLIBC_2.3.4' not found (required by dbx)

I even risked downloading latest GNU libc (2.3.6 right now), hoping to install it into a separate directory (not really sure whether this is possible in the first place); this hope however died in vain after this error:

cannot set up thread-local storage: set_thread_area failed when setting up thread-local storage

It seems like something on my system is too old, either kernel (2.4.21), or compiler (3.3.1), or both.

Found one interesting page on the subject of upgrading libc.so which supposedly describes a (presumably) safe way to do so. It follows from it that I might have better luck trying 2.3.4, though I still don't understand whether I can simply install it in another directory to a avoid de-stabilizing the system or not.

May be I try it one day.

My SuSe 9.0 is rapidly becoming too old and outdated, so that few pre-compiled binaries can be installed. Unfortunately.

Audacity vs. Audio Record Wizard

2006-02-05T00:46:00.000-05:00

Just found an excellent free program for Linux/Windows, Audacity. It is included in SuSe 9.0, and under Windows is a perfect free replacement for (otherwise good) (shareware) program Audio Record Wizard.

Spellbound in Mozilla 1.5+

2006-02-04T21:55:00.000-05:00

To install spellbound in Mozilla 1.5+, follow this links: Installer (found here, and here there is a generic spellbound discussion)

external USB devices with SuSe 9.0

2006-02-04T19:56:00.000-05:00

Nice introduction on how to install external USB devices with SuSe 9.0!

Controlling ssh from (python) script

2006-02-02T23:01:00.000-05:00

It appears ssh only reads its password from the TTY making it difficult to supply one via a script. Here is a possible solution in python:

The trick is, of course, to create pseudo-TTY:

--
#!/usr/bin/env python
#Remote command through SSH using user/password
import os, time

def pause(d=0.2):
 time.sleep(d)

def rcmd(user, rhost, pw, cmd):
 #Fork a child process, using a new pseudo-terminal as the child's 
controlling terminal.
 pid, fd = os.forkpty()
 # If Child; execute external process
 if pid == 0:
  os.execv("/bin/ssh", ["/bin/ssh", "-l", user, rhost] + cmd)
 #if parent, read/write with child through file descriptor
 else:
  pause()
  #Get password prompt; ignore
  os.read(fd, 1000)
  pause()
  #write password
  os.write(fd, pw + "\n")
  pause()
  res = ''
  #read response from child process
  s = os.read(fd,1 )
  while s:
   res += s
   s = os.read(fd, 1)
  return res

#Example: execute ls on server 'serverdomain.com'
print rcmd('username', 'serverdomain.com', 'Password', ['ls -l'])
---