Strings on UberMongoose

Unicode on Linux

Sun, 26 May 2013 14:52:45 +0530

At first the place of Unicode in Linux looks simple. It is deceptively so. Linux C/C++ has support for both char and wchar_t. Let’s take a look at the char version:

#include <cstring>
#include <iostream>
 
int main(int argc, char* argv[])
{
    const char text[] = "Άλφα";
    std::cout << text << std::endl;
    std::cout << "sizeof(char): " << sizeof(char) << std::endl;
    std::cout << "sizeof(text): " << sizeof(text) << std::endl;
}

This results in the output:

Άλφα
sizeof(char): 1
sizeof(text): 9

Notice that the console prints out the characters correctly and that the size of a char is 1 byte. On the other hand the size of the text buffer is 9 bytes, though manifestly it only contains 4 (+1 for the null terminator). The reason for this is that each character is encoded as a couple of UTF-8 characters. This is important. All char buffers in Linux supports Unicode by default because GCC by default encodes literal text as UTF-8 streams. On the down side now some of the old string handling functions – for example strlen – which are not UTF-8 aware start returning bovine excrement as results and require replacement but we will return to that.

Unicode and Windows

Tue, 06 Nov 2012 13:46:55 +0530

Unicode string support in Windows is a strange beast.

When Microsoft hired Dave Cutler and others to write a new OS for them they were thinking about writing a new version of OS/2. In fact the new Microsoft OS was a combined effort by Microsoft and IBM that would have resulted in OS/2 version 3.0 if not for the success of Microsoft’s own Windows 3 which made them reconsider that approach. At the eleventh hour they shifted their API from the extended OS/2 API that was to be the way things got done on the new OS to an extended version of the Windows API. This last minute change was possible due to some clever trickery on the part of the programmers who wrote the new Windows OS but it resulted in a strange API, especially when it came to string handling in the new OS known to history as Windows NT 3.1.

Unicode

Fri, 12 Oct 2012 09:32:22 +0530

For many years programmers and strings existed in a state of balance, as much balance as could have been possible under the constant threat of undefined behaviour that sociopathic string functions threatened in any case. It was not to be however for their lives of relative stability was forever changed with the advent of Other Languages into string buffers. In this part of our continual study of strings we delve into intriguing world of internationalisation, multi byte character sets and Unicode.

The Private Lives of Strings

Sat, 15 Sep 2012 16:14:33 +0530

Ah, Strings. Surprisingly simple for most C# and Java programmers yet so complex for those of us left behind in the world of C and C++, Strings are perhaps one of the strangest beasts in the programming landscape. Yet they began so simple, perhaps deceptively so like a chameleon on a plain coloured leaf waiting to deceive the observer with its next background. Yet I have tracked this beast down through the ages, yea, verily I have identified its habits and its lair and here I present to you in true Attenboroughish fashion the Private Life of Strings. To understand the habitat of strings one must go back to their distant ancestors. The typewriters.