Johannes Schlüter

Yesterday was a quite thrilling day for the PHP development team and led to some imprecise news articles so let's take a look at what happened: Over the last months many of the core contributors agreed that the current approach to bring Unicode into PHP's engine wasn't the right approach and a good thing would be to rethink it from the start. By a provocative move of one contributor the stalled situation got some more movement and Rasmus declared the current implementation to be discontinued to restart.

The past

When the foundation of what should have become PHP 6 was created a decision was made to use UTF-16 as internal encoding for "everything" inside the engine. The choice for UTF-16 was made due to the fact that PHP 6 would use the ICU library which is focused on offering UTF-16 string functions. By using UTF-16 as default encoding we'd have to convert the script code and all data passed from or to the script (request data, database results, output, ...) from another encoding, usually UTF-8, to UTF-16 or back. The need for conversion doesn't only require CPU time and more memory (a UTF-16 string takes double memory of a UTF-8 string in many cases) but makes the implementation rather complex as we always have to figure out which encoding was the right one for a given situation. From the userspace point of view the implementation brought some backwards compatibility breaks which would require manual review of the code. These all are pains for a very small gain for many users where many would be happy about a tighter integration of some mbstring-like functionality. This all led to a situation for many contributors not willing to use "trunk" as their main development tree but either develop using the stable 5.2/5.3 trees or refuse to do development at all.

The present

Yesterday the stagnation created by the situation has been resolved and it was decided that our trunk in svn will be based on 5.3 and we'll merge features from the old trunk and new features there so that 5.3 will be a true stable branch. The EOL for 5.2 has not yet been defined but I suggest you to really migrate over to 5.3, which usually can be done with very little work, as soon as possible.

The future

Right now we're starting different discussions to see what kind of Unicode support we really want. Many contributors react positive on a proposed "string class" which wraps string operations in Unicode and binary forms without going deep in the engine. In my opinion such an approach might also be a way to solve some of the often criticized inconsistencies in PHP's string APIs without the need to break old code. (new code uses the new class, old code the old functions) But that idea is far from a proper proposal or even the implementation, current status is about refocusing the development and get the requirement and design discussions going. By that it's a bit early to decide whether the next version of PHP will be called PHP 5.4, PHP 6 or maybe even PHP 7.

PHP is alive and kicking!

When I see people talking about Unicode and PHP 6 I often see them mentioning one fact as a big change: PHP 6 allows (mostly any) arbitrary Unicode character as (part of an) identifier. So you can have code like this:

function 新日本石油() {
    echo "Let's hope this isn't an offensive function name... ";
    echo "it's copied from some news site";
}

新日本石油();

Well yes, that's funny, at first but serves a purpose: Consider you have an application tied to an environment with a special terminology, then translating this terms to English might be extremely confusing (especially as programmers often don't really know the correct terminology of that domain) and it's good to call the thing by it's name - while that can be quite complicated, too, in a previous job we had such a case and often used the German terms which produced quite funny names for getters and setters which didn't satisfy us ... but that's not what I wanted to talk about.

The purpose of this were some bad news: That's nothing new. The relevant scanner rule hasn't changed since 4.0 - the only change is that PHP 6 doesn't treat it as random set of bytes anymore but knows about Unicode codepoints and interprets is as such.

Out of interest I did some little digging into the PHP repository's history:

$ svn annotate trunk/Zend/zend_language_scanner.l
...
34779 zeev LABEL [a-zA-Z_\x7f-\xff][a-zA-Z0-9_\x7f-\xff]*
...
$ svn log -r 34779
------------------------------------------------------------------------
r34779 | zeev | 2000-10-29 15:35:34 +0100 (Sun, 29 Oct 2000) | 2 lines

Unify the names of these last 3 files...

------------------------------------------------------------------------


The result? - The rule, as it is in PHP 6 wasn't changed since 2000, really nothing new there with PHP 6, and even then the only change was that the scanner file was renamed...

Ilia recently published the probably final release candidate of the current stable tree: PHP 5.2.4RC2. The stable release can be expected soon. This release fixes quite some bugs from older versions and does a few minor adjustments. Please test the RC release so unexpected regressions can be detected and fixed before it's marked stable. If regressions and other problems aren't detected now it will take a few months till the next version will be released, so do your best to make sure it's worth being called stable! Please file found issues, after checking for other reports and the documentation, into the bug tracker. The recent trend to report bugs using blogs isn't working well!

In other news one could read that there were also some major improvements in the development of PHP 6 made: Next to all that Unicode related work which was made over the last months PHP got, due to Andrey, mysqlnd as PHP-optimized replacement for libmysql and, thanks to Dmitry's work, namespace support. So this weekend might be the perfect time for sending you're family visting other parts of the family, letting them sweat on some beach or having fun at some theme park - then you can sit down, relax, and, after testing PHP 5.2.4RC2, give that new stuff in PHP 6 a test.

For getting started with PHP 6, which you can get, as you're hopefully used to, from the snaps site, you should first read Andrei's slides about Unicode and PHP 6. The most important Unicde+PHP facts are also collected inside a README file. For namespaces there's, again, a README file, David Coallier has written a nice summary and Stanislav wrote a short FAQ. If you are curious what mysqlng is: simply read Ulf's blog postings or Andrey's slides.