Skip to main content

Posts

Showing posts with the label utf8

Importing the Unicode Character Database in MySQL

In Python it is easily possible to findout the name of a Unicode character and findout some properties about that character. The module which does that is called unicodedata . An example: >>> import unicodedata >>> unicodedata.name( '☺' ) 'WHITE SMILING FACE' This module uses the data as released in the UnicodeData.txt file from the unicode.org website. So if UnicodeData.txt is a 'database', then we should be able to import it into MySQL and use it! I wrote a small Python script to automate this. The basic steps are: Download UnicodeData.txt Create a unicodedata.ucd table Use LOAD DATA LOCAL INFILE to load the data This isn't difficult especially because the file doesn't have the actual characters in it. It is just an ASCII file with codepoints and the data related to it. But it would be useful to have those characters in the database. So what I did is to create those with CONVERT(UNHEX(value) USING utf32 . What that does...

That's not my name! A story about character sets

When computers were still using large black text oriented screens or no screens at all, a computer only knew how to store a limited set of characters. Then it was normal to store a name with the more complicated characters replaced by more basic characters. The ASCII standard was used to make communication between multiple systems (or applications) easier. Storing characters as ASCII needs little space and is quite strait forward. Then DOS used CP850 and CP437 and so on to make it possible to use language /location specific characters. Then ISO8859-1 , ISO8859-15 and more of these character sets were defined as standard. And now there is Unicode: UTF-8, UTF-16, UCS2, etc. which allow you to store many different kinds of characters in the same character set. But all those character sets only work correctly if you configure all applications correctly. Many of the character sets are very similar and seem to work correctly even if one of the systems is not correctly configured. If...