Johannes Schlüter

For the last few months I had quite a few MySQL blog posts and didn't have anything from my "new features in PHP [trunk|5.4]" series. This article is a bridge between those two. The PHP 5.4 NEWS file has a small entry:

MySQLi: Added iterator support in MySQLi. mysqli_result implements Traversable. (Andrey, Johannes)

From the outside it is a really small change and easy to miss. The mentioned class, mysqli_result, implements an interface which adds no new methods. What once can't see is that this relates to some internal C-level functions which can be called by the engine for doing a foreach iteration on objects of this class. So with PHP 5.4 you don't have to use an "ugly" while construct anymore to fetch rows from a mysqli result but can simply do a foreach:

mysqli_report(MYSQLI_REPORT_STRICT);
try {
    $mysqli = new mysqli(/* ... */);
    foreach ($myslqi->query("SELECT a, b, c FROM t") as $row) {
        /* Process $row which is an associative array */
    }
} catch (mysqli_sql_exception $e) {
    /* an error happened ... */
}

I'm configuring mysqli in a way to throw exceptions on error. This is useful in this case as mysqli::query() might return false in the case of an error. Passing false to a foreach will give a fatal error, so I'd need a temporary variable and a check in front of the foreach loop, with exceptions I simply do the error handling in the catch block.

One thing to note is that mysqli is using buffered results ("store result") by default. If you want to use unbuffered result sets ("use result") you can easily do that by setting the flag accordingly:

foreach ($myslqi->query("SELECT a, b, c FROM t", MYSQLI_USE_RESULT) as $row) {
    /* ... */
}

People who are advanced with iterators in PHP might ask "Why did you implement Traversable only, not Iterator?" - the main reason is that we simply didn't want to. The mysqli_result class already has quite a few methods and we didn't want to make the interface confusing. If you need an Iterator class for some purpose you can simply wrap mysqli_result in an IteratoIterator.

I was writing about PHP.next before, many things improved there meanwhile. Most notably we have a committed version number: The next PHP release will be called PHP 5.4. The topic I want to talk about today is "Improved application portability" which covers multiple small changes which aim at making it simpler for developers to write applications working on any PHP setup.

Separating <?= from short_open_tags

PHP knows quite a few ways to separate PHP code from surrounding text (usually HTML), most applications use <?php as that works on every system. There is a short form of this, <?, which can be disabled using php.ini's short_open_tags setting. Being able to disable this is important when embedding PHP code into XML documents containing XML processing instructions. Now we also have <?= which, basically, is a shortcut for <?php echo. This tag is useful when using PHP as templating language as it prevents cluttered code. The issue in current version of PHP is that this is bound to short_open_tags, so portable applications can't rely on it. But PHP 5.4 will bring the solution: <?= will always be there, independently from short_open_tags. Yay!

No more magic_quotes

In the old times it was easy to write code using PHP.

<?php
$q = mysql_query("SELECT * FROM t WHERE name = '$name' ");
?>

And you had, thanks to register_globals, some data to work on and this was mostly secure as PHP automatically escaped request data. But well this escaping worked only in a few cases acceptable good. Besides not knowing anything about other encodings or DBMS-specific escape sequences it also failed for non-string values as in

<?php
$q = mysql_query("SELECT * FROM t WHERE id = $id ");
?>

Where the external value wasn't escaped. So portable applications, which aim at being secure nowadays have to check whether magic_quote_gpc is enabled, then remove the "bad" quotes and then finally escape again using the appropriate way. That's quite  an annoyance and doing this the wrong way can cause bad bugs (like forcing such a replacement logic in an endless recursion by providing arrays) So nobody really likes magic_quotes. So with PHP 5.4 they are gone. No more need to worry about them. Use the proper escaping and you're done. Wonderful. Only issue: Legacy applications might rely on magic_quotes so when upgrading PHP make sure the application does the required escaping itself so almost-secure applications won't become insecure.

Dropped explicit --enable-zend-multibyte compile-time option

Especially in Asia people use multi-byte encodings which aren't ASCII-compatible so mixing them with PHP code might be hard. In current versions of PHP there is a compile-time option to enable a special multibyte mode for the engine which will handle this in the engine so PHP code can be provided using these encodings. By this portable applications had a hard time due to this conversion (not) being done. Thanks to the work by Dmitry and Moriyoshi this mode is now always enabled whithout penalty for people not depending on it and the extended functionality from mbstring can be provided as a shared module. By this distributors can provide a single build which will work for everybody.

Closing remarks

As always in this series: Be aware that things discussed here might change. Please try out the current snapshot of PHP 5.4 and test it with your applications. No we can still fix backwards compatibility breaks. fixing them after a release will possibly break it for people depending on the new behavior. Happy coding!

File uploads via HTTP are an annoyance. Web Browser know quite a lot but still the give little feedback to the users. Some a bit more, most close to no feedback. Now over the years this led to man unhappy users. Over the last year, with all these AJAX things, solutions emerged so that one can periodically poll the web server on a second connection for the status. For implementing this we have one architectural problem: PHP implements, for very good reasons, a shared nothing architecture. So one request from connection has no insight into another request/connection - but this is needed for the upload progress. Different people thought about this and implemented solutions. One of the first things was implemented in APC, another one by a special uploadprogress extension. They are nice and found quite some adoption but they have two problems. For one they are not fully native to PHP, so they have to be installed additionally, and the use a local storage to transport the status. APC uses the system's shared memory. upload_progress the filesystem (yes, file systems could be shared, I do know that). not very satisfying for having PHP as the language for solving the web problem.

The obvious solution, of course, would be to use PHP's session handling system for this. The PHP session system is an integral part of PHP and can be configured to use different storage handlers, like the local file system or memcache, which can be useful to share session data in a load-balanced cluster. Now there were some technical issues why this wasn't done at first ... but then Arnaud Le Blanc sat down and created a proper implementation of an upload progress storage handler which has been commit to PHP trunk.

Long story short: In the next version of PHP (5.4?) you will, mot likely, have an Upload Progress mechanism built-in.

Arnaud wrote a nice RFC explaining this functionality. So we configure our PHP to enable this feature, by making sure we have the default values:

session.upload_progress.enabled = 1 
session.upload_progress.prefix = upload_progress_
session.upload_progress.name = PHP_SESSION_UPLOAD_PROGRESS

And we are set to go. Then we obviously need an HTML file upload form:

<form action="upload.php" method="POST" enctype="multipart/form-data">
 <input type="hidden" name="<?php echo ini_get("session.upload_progress.name"); ?>" value="johannesupload" />
 <input type="file" name="file1" />
 <input type="file" name="file2" />
 <input type="submit" />
</form>

And we can upload a file. If the file is big enough (and the connection slow enough) we can then periodically poll the server and read and read data about the progress from the $_SESSION["upload_progress_johannesupload"] variable. The full contents is mentioned in the RFC, so I won't quote it here.

So far so good. But how to poll? Well, take the sample by Rasmus about the above mentioned APC-based solution and adopt it. I let this to the experienced reader as exercise

Disclaimer: This article is describing features in not released software. Things may change without notice till it is release. Feel free to read other blog postings from my series of new features!

P.S. I still think the browser should give better feedback by itself. As should it offer better integration of HTTP-Auth, like a simple logout button ...

Some time ago I wrote an article about the implementation of type hints for non-object types for PHP. Meanwhile many things happened and that implementation was replaced by a different one. Readers of my previous post might know that I have doubts about type hints in PHP. People who met me in person and asked me about it know for sure

So what's the status now? - Well type hints, for non-object types, exist and they don't. There is a valid syntax which looks like this:

function foo(int $i) {
    echo $i;
}

What's the consequence of this code? - Well the type hint is simply ignored. This means that

 foo("Hello world");

Will run without any error and print Hello world. So there is an syntax looking like another part of the language which throws errors but behaves completely different.

function foo(bar $b) {  }

foo("bar");'

Catchable fatal error: Argument 1 passed to foo() must be an instance of bar, string given [...]

The int hint is just one of them, there are a few more:

• bool, boolean
• string, binary
• scalar
• numeric
• int, integer, long
• real, double, float
• resource
• object

So why is there a syntax added which is ignored? Well there are two things one can do: The hint can be read via the tokenizer to do some magic or one might create a PHP extension which hooks into the engine to do stricter checks on this, like for a testing system. Does it make sense? - Well, I let it to you to decide whether it make sense to have two syntaxes which look the same but do very different things (being ignored vs. throwing a terminating error) and whether it makes sense to push a system where the core language behaves differently depending on the system (is an extension using this hook loaded? which of them? - one which casts or one which throws an error?) ... I seriously hoped PHP was out of the era of introducing new inconsistencies ...

I plan to continue my series about new stuff in PHP's trunk but for now just a short note about something which was removed: PHP 5.3 has different ways to access SQLite databases of all kinds. Two of them are provided by the sqlite extension: The sqlite_ group of functions and the pdo_sqlite2 driver. The issue there is that this depends on the SQLite 2 library which isn't supported by upstream anymore for a few years. It was a logical step therefore to remove this extension from PHP trunk. The support for the sqlite3 extension and the PDO_sqlite driver (same link as above, read it carefully), which use version 3 of the library, are continued. Please note that the disk format changed from version 2 to 3 so you might not only have to change the application but also to recreate the database file. This change will most likely appear in the feature version of PHP, which will most likely be called PHP 5.4.

So in my blog series I try to cover all additions to PHP trunk so I have to mention scalar type hints.

<?php
function print_float(float $f) {
    echo $f."\n";
}

for ($i = 1; $i < 5; $i++) {
    print_float( $i / 3 );
}
?>
	

0.33333333333333
0.66666666666667

Catchable fatal error: Argument 1 passed to print_float() must be of the type double, integer given, called in typehints.php on line 7 and defined in typehints.php on line 2

Is expected behavior in PHP's trunk. If you want such a thing to work please use the numeric type hint.

In case that wasn't enought fun: There's more!

<?php
function handle_result(int $i) {
    echo $i."\n";
}

$pdo = new PDO("mysql:host=localhost;dbname=test", "user", "pass");
$pdo->setAttribute(PDO::MYSQL_ATTR_DIRECT_QUERY, false);
$result = $pdo->query("SELECT 42 AS id");
$row = $result->fetch();
handle_result($row['id']);

$pdo = new PDO("mysql:host=localhost;dbname=test", "user", "pass");
$pdo->setAttribute(PDO::MYSQL_ATTR_DIRECT_QUERY, true);
$result = $pdo->query("SELECT 42 AS id");
$row = $result->fetch();
handle_result($row['id']);
?>
	

42

Catchable fatal error: Argument 1 passed to handle_result() must be of the type integer, string given, called in typehints.php on line 16 and defined in typehints.php on line 2

So what happens here? - Depending on the PDO::MYSQL_ATTR_DIRECT_QUERY option PDO will either emulate prepared statements (which would be irrelevant here, but that's another story) or use native prepared statements. When using prepared statements MySQL switches over to its "binary" protocol which returns the native types, else it always returns strings. When using PDO_mysql linked against mysqlnd the native type is returned to PHP. If I would use libmysql both times a string would be returned. In other words: The behavior is system dependent. The default behavior for other drivers isn't defined either. As PHP is a dynamically typed language this shouldn't matter, so depending on your driver the results vary. Great. Oh and did I mention that the types aren't specified, so a later version of a driver might change it. And PDO is just a simple example for this ...

The key point of this all is that as soon as you mix strong typing - by using type hints - with PHP's weak type system you open a can of worms.

And to be overly clear: I can understand the need for strict type systems and see where such a thing helps. But adding a strong type system, stricter than most other languages (hey, you can't pass an integer where a float is expected!), makes little sense to me. But that's just me.

As in my previous posts in the series of blog posts about features in PHP trunk. Please try the snapshots. Mind that these features might (not) end up in the next release of PHP, feedback is welcome.

I was writing about new features in the upcoming PHP version (5.4, 6.0?) before. Today's topic reads like this in the NEWS file:

- Added array dereferencing support. (Felipe)

Now you might wonder what this typical short entry means. when doing OO-style PHP you might make use of a sntax feature which one might call "object dereferencing" which looks like this:

<?php
class Foo {
    public function bar() { }
}

function func() {
    return new Foo();
}

func()->bar();
?>

So one can chain method calls or property access. Now for a long time people requested the same thing for array offset access. This was often rejected due to uncertainties about memory issues, as we don't like memory leaks. But after proper evaluation Felipe committed the patch which allows you to do things like

<?php
function foo() {
    return array(1, 2, 3);
}
echo foo()[2]; // prints 3
?>

Of course this also works with closures:

<?php
$func = function() { return array('a', 'b', 'c'); };
echo $func()[0]; // prints a
?>

And even though the following example is stupid I might accept this feature as one of the few places where it is ok to use references in PHP:

<?php
$data = array('me', 'myself', 'you');
function &get_data() {
    return $GLOBALS['data'];
}
get_data()[2] = 'I'; // $data will now contain 'me', 'myself' and 'I'
?>

Wonderful, isn't it? If you want to test it please take a look at the recent snapshots for PHP trunk and send us your feedback! Please mind that all features in PHP trunk may or may not appear in the next major PHP release.

In a few previous blog posts I was already talking about changes in PHP's trunk, like some BC break or some new features, time to continue:

JSON is the modern data format used in "AJAX" applications. As the leading language for the Web PHP course has support for handling JSON data: You can pass any data to json_encode() and the function will create a JSON representation of this data:

<?php
echo json_encode(array(1,2,3,4));
?>

[1,2,3,4]

This also works for objects:

<?php
$o = new stdclass;
$o->a = 42;
echo json_encode($o);
?>

{"a":42}

Wonderful. But the world isn't that easy. For many PHP objects the JSON-representation of the data is a bit more complex.for instance what about private properties or maybe you want to calculate some inner values? - In PHP 5.3 you were on your own. but thanks to Sara there's hope in sight: the new interface JsonSerializable. Classes implementing this interface have to provide a method jsonSerialize() which will be called by json_encode() and has to return a JSON-compatible representation of the data by doing whatever you want. So let's take a look:

<?php
class JsonTest implements JsonSerializable {
    private $a, $b;

    public function __construct($a, $b) {
        $this->a = $a;
        $this->b = $b;
    }

    public function jsonSerialize() {
        return $this->a + $this->b;
    }
}

echo json_encode(new JsonTest(23, 42));
?>

65

Now this example in itself is of course useless, but let's create a bigger structure, which includes a few of these objects:

<?php
$data = array(
    new stdClass();
    new JsonTest(1,2),
    new JsonTest(3,4),
    array(5,6)
);
echo json_encode($data);
?>

[{},3,7,[5,6]]

Of course you'd usually have more complex serialization logic, but that's left to you.

Now almost certainly somebody will ask "and what about the other way round?" - The only answer there is: Sorry there we can't do much. JSON doesn't encode and meta-information so our generic parser in json_decode() can't do anything special. But anyways: The new interface will certainly be useful.

At last a short disclaimer: PHP trunk is a development tree. Features in their may be changed at anytime and might not be released. Feedback is welcome.

A few years ago I used another blog to write about "More PHP power on the command line" almost 5 years later the PHP interactive shell got a major update which went in PHP's trunk. The commit message tells a lot about the improvements:

- Improved CLI Interactive readline shell (Johannes)
  . Added cli.pager ini setting to set a pager for output.
  . Added cli.prompt ini settingto configure the shell prompt.
  . Added shortcut #inisetting=value to change ini settings at run-time.
  . Don't terminate shell on fatal errors.

A pager can be a an shell command which will receive the command output on its
STDIN channel

php > #cli.pager=less
php > phpinfo();
(output will appear in the pager)
php > #cli.pager=grep -i readline
php > phpcredits();
Readline => Thies C. Arntzen
php > #cli.pager=
(output appears again direct on the terminal)

A prompt can contain a few escape sequences like

php > #cli.prompt=\e[032m\v \e[031m\b \e[34m\> \e[0m
5.3.99-dev php > //Colorful prompt with version number

A prompt can also contaian PHP code in backticks

php > #cli.prompt=`echo gethostname();` \b \>
guybrush php >

But I assume a screenshot is quite useful, so here it goes:

So what doyou need to use this feature? - PHP trunk compiled using --with-readline or --with-libedit.The PHP Documentation has information about the interactive mode, the new features will be added once trunk is closer to a release and it's clear in what release this will appear.

Dealing with character sets and encodings is tough. As long as you're dealing only with English texts you in a luxury situation and can mix utf-8 and iso-8859-1 encoded texts and most (all?) of your tests will work. Some of your users, like me, with strange names ("Schlüter") will be annoyed as your application breaks them ("Schlüter"), but these will be edge cases. There are bigger issues with mixing encodings but that's not what I wanted to tell now.

Handling these encodings in PHP correctly is tough. PHP, currently, has a quite simple approach to the problem in general: PHP doesn't care about encodings. A string is simply a sequence of bytes. Well in general. The details are difficult. Core features like JSON-handling or XML processing expect that your PHP strings are encoded using utf-8, which is a sane choice, for JSON it is part of the JSON specification, for XML it's the only way to be able to work with all documents without additional information from you along every XML operation.

On the other side we have browsers. A browser has to read documents from all over the world which can be encoded in any encoding you like. For whatever reason browser developers decided that iso-8859-1 would make a great default encoding, which means that if a response to a browser's request doesn't specify anything else the browser assumes the document is encoded using iso-8859-1. The document's encoding will also be used when sending form data back to the server. To handle this, PHP has a php.ini setting default_charset which, if set, will set the selected encoding in the HTTP header. The default value of this is setting is empty - so browsers fall back on their default, iso-8859-1.

Recently Rasmus made a commit to PHP trunk which changes the default to utf-8.

In the long-run this changes is good as it works for all languages, iso-8859-1 only works for a limited set of European languages, and works better with the outside environment, where utf-8 adoption is growing (JSON and XML were examples).

In the short-term this might cause trouble for applications depending on the default.The good thing is that the development in trunk has just started and the release of PHP.next is still sometime away and you can easily prepare your application - which is a good thing anyways to protect from administrators making mistakes with current versions already.

To set the encoding from within your application you can for example call ini_set('default_charset', $enc); or header("Content-type: text/html; charset=$enc"); at the beginning of your script, where $enc is your preferred encoding. Please mind that this has no effect on the script itself, which, for instance, means you have to configure your database connection accordingly, too!

Yesterday was a quite thrilling day for the PHP development team and led to some imprecise news articles so let's take a look at what happened: Over the last months many of the core contributors agreed that the current approach to bring Unicode into PHP's engine wasn't the right approach and a good thing would be to rethink it from the start. By a provocative move of one contributor the stalled situation got some more movement and Rasmus declared the current implementation to be discontinued to restart.

The past

When the foundation of what should have become PHP 6 was created a decision was made to use UTF-16 as internal encoding for "everything" inside the engine. The choice for UTF-16 was made due to the fact that PHP 6 would use the ICU library which is focused on offering UTF-16 string functions. By using UTF-16 as default encoding we'd have to convert the script code and all data passed from or to the script (request data, database results, output, ...) from another encoding, usually UTF-8, to UTF-16 or back. The need for conversion doesn't only require CPU time and more memory (a UTF-16 string takes double memory of a UTF-8 string in many cases) but makes the implementation rather complex as we always have to figure out which encoding was the right one for a given situation. From the userspace point of view the implementation brought some backwards compatibility breaks which would require manual review of the code. These all are pains for a very small gain for many users where many would be happy about a tighter integration of some mbstring-like functionality. This all led to a situation for many contributors not willing to use "trunk" as their main development tree but either develop using the stable 5.2/5.3 trees or refuse to do development at all.

The present

Yesterday the stagnation created by the situation has been resolved and it was decided that our trunk in svn will be based on 5.3 and we'll merge features from the old trunk and new features there so that 5.3 will be a true stable branch. The EOL for 5.2 has not yet been defined but I suggest you to really migrate over to 5.3, which usually can be done with very little work, as soon as possible.

The future

Right now we're starting different discussions to see what kind of Unicode support we really want. Many contributors react positive on a proposed "string class" which wraps string operations in Unicode and binary forms without going deep in the engine. In my opinion such an approach might also be a way to solve some of the often criticized inconsistencies in PHP's string APIs without the need to break old code. (new code uses the new class, old code the old functions) But that idea is far from a proper proposal or even the implementation, current status is about refocusing the development and get the requirement and design discussions going. By that it's a bit early to decide whether the next version of PHP will be called PHP 5.4, PHP 6 or maybe even PHP 7.

PHP is alive and kicking!