Johannes Schlüter

iSCSI devices in VirtualBox

nospam@example.com (Johannes Schlüter) — Thu, 02 Sep 2010 12:40:18 +0200

While playing with my disc-less environment, booting via gPXE from an iSCSI device, I found out that VirtualBox has direct support for iSCSI while this isn't shown in the GUI. To register an iSCSI device as virtual disc one has to use VBoxManage on the command line:

$ VBoxManage addiscsidisk \
       --server iscsi.home.schlueters.de \
       --target iqn.1986-03.com.sun:02:f4635770-f010-61be-e8f7-83403206bec5

The result can be checked:

$ VBoxManage list hdds
[...]
UUID:        292b93ef-0fef-41c4-ad9c-700c38d5537e
Parent UUID: base
Format:      iSCSI
Location:    iscsi.home.schlueters.de|iqn.1986-03.com.sun:02:f4635770-f010-61be-e8f7-83403206bec5
State:       created
Type:        normal
Usage:       test (UUID: 0e9dac71-1a33-42f3-8320-1f157582e931)

Once the disc is registered it can be selected in the GUI (or from command line) while configuring a VM. In my test I once lost the network connection to the iSCSI target, VBox handled that in a safe way by pausing the VM. Quite handy.

HashTables

nospam@example.com (Johannes Schlüter) — Sun, 22 Aug 2010 23:52:00 +0200

In case you ever heard me talking about PHP internals I certainly mentioned something along the lines of "Everything in PHP is a HashTable" that's not true, but next to a zval the HashTable is one of the most important structures in PHP. While preparing my "PHP Under The Hood" talk for the Dutch PHP Conference there was a question on IRC about extension_loaded() being faster than function_exists(), which might be strange as both of them are simple hash lookups and a hash lookup is said to be O(1). I started to write some slides for it but figured out that I won't have the time to go through it during that presentation, so I'm doing this now, here:

You all should know PHP arrays. They allow you to create a list of elements where every element may be identified by a key. This key may either be an integer or a string. Now we need a way to store this association in an effective way in memory. An efficient way to store a collection of data in memory is a "real" array, an array of elements indexed by a sequence of numbers, starting at 0, up to n. As memory is essentially a sequence of numbered storage blocks (this obviously is simplified, there might be segments and offsets, there might be virtual pages, etc.) we can efficiently access an element by knowing the address of the first element, the size of the elements (we assume all have the same size, which is the case here), and the index: The address of the n-th element is start_address + (n * size_of_element). That's basically what C arrays are.

Now we're not dealing with C and C arrays but also want to use string offsets so we need a function to calculate a numeric value, a hash, for each key. An hash function you most likely know is MD5, MD5 is creating a 128 bit numeric value which is often represented using 32 hexadecimal characters. For our purpose 128 bit is a bit much and MD5 is too slow so the PHP developers have chosen the "DJBX33A (Daniel J. Bernstein, Times 33 with Addition)" algorithm. This hash function is relatively fast and gives us an integer of the value, the trouble with this algorithm is that it is more likely to produce conflicts, that means to string values which create the same numeric value.

Now back to our C array: For being able to safely read any element, to see whether it is used, we need to pre-allocate the entire array with space for all possible elements. Given our hash function returning a system dependent (32 bit or 64 bit) integer this is quite a lot (size of an element multiplied wit the max int value), so PHP does another trick: It throws some digits away. For this a table mask is calculated. The a table mask is a number which is a power of two and then one subtracted and ideally higher than the number elements in the hash table. If one looks at this in binary representation this is a number where all bits are set. Doing a binary AND operation of our hash value and the table this gives us a number which is smaller than our table mask. Let's look at an example: The hash value of "foobar" equals, in decimal digits, to 3127933054. We assume a table mask of 2047 (2¹¹-1).

  3127933054    10111010011100000111100001111110
&       2047    00000000000000000000011111111111
=        126    00000000000000000000000001111110

Wonderful - we have an array Index, 126, for this string and can set the value in memory!

If life were that easy. Remember: We used a hashing function which is by far not collision free and then dropped almost two thirds of the binary digits by using the table mask. This makes it rather likely that some collisions appear. These collisions are handled by storing values with the same hash in a linked list. So for accessing the an element one has to

Calculate the hash
Apply the table mask
locate the memory offset
check whether this is the element we need, traverse the linked list.

Now the question initially asked was why extension_loaded() might be faster than function_exists() and we can tell: For many random reasons and since you have chosen a value which probably conflicts in the first, not in the second. So now the question is how often such collisions happen for this I did a quick analysis: Given the function table of a random PHP build of mine with 1106 functions listed I have 634 unique hash values and 210 hash values calculated from different functions. Worst is the value of 471 which represents 6 functions.

Full results are online but please mind: These results are very specific to my environment. Also mind that the code actually works on a copy of my function table so the table mask might be different, which changes results. Also note that the given PHP code won't work for you as I added special functions exporting the table mask and hash function to mz version of PHP. And then yet another note: doing performance optimizations on this level is the by far worst place as to many unknown factors go in. And you don't have any measurable performance win. Mind readability. Clear code is worth way more than the 2 CPU cycles you probably gain! But you may have learned how hash tables work.

References and foreach

nospam@example.com (Johannes Schlüter) — Fri, 20 Aug 2010 13:21:05 +0200

References in PHP are bad, as I mentioned before, and you certainly should avoid using them. Now there is one use case which leads to an, at first, unexpected behavior which I didn't see as a real live issue when I stumbled over it at first, but then there were a few bug reports about it and recently a friend asked me about it ... so here it goes:

What is the output of this code:

<?php
$a = array('a', 'b', 'c', 'd');

foreach ($a as &$v) { }
foreach ($a as $v) { }

print_r($a);
?>

We are iterating two times over an array without doing anything. So the result should be no change. Right? - Wrong! The actual result looks like this:

Array
(
    [0] => a
    [1] => b
    [2] => c
    [3] => c
)

For understanding why this happens let's take a step back and look at the way PHP variables are implemented and what references are:

A PHP variable basically consists out of two things: A "label" and a "container." The label is the entry in a hash table (there are a few optimizations in the engine so it is not always in a hashtable but well) which may represents a symbol table of a function, and array or an object's property table. So we have a name and a pointer to the container. The container, internally called "zval", stores the value and some meta information, this container can also be a new hashtable with a set of labels pointing to other containers if we now create an reference this will cause a second label to point to the same container as another label. Both label from then on have the same powers of the container.

Now let's look at the situation from above. In a picture it looks like this:

So we have six containers (the global symbol table on the top, a container holding the array called $a on the left and one container for each element on the right) Now we start the first iteration. So the global symbol gets a new entry for $v and v is made a reference to the container of the first array element.

So an change to either $a[0] or $v goes to the same container and therefore has an effect to the other. When the iteration continues the reference is broken and $v is made a reference to the different elements. So after the iteration ends $v is a reference to the last element.

Remember: $v being a reference means that any change to $v effects the other references, in this situation $a[3]. Up till now nothing special happened. but now the second iteration begins. This one assigns the value of the current element to $v for each step. Now $v is a reference to the same element as $a[3] so by assigning a value to $v $a[3] is changed, too:

This continues for he next steps, too.

And now we can easily guess what will happen at the last step: $v is being assigned the value of the last element, $a[3], and as $a[3] is a reference to $v it therefore assignees itself to itself so effectively nothing happens.

And this is the result we saw above.

So to make this story full of pictures short: Be careful about references! They can have really strange effects.

MySQL at FrOSCon

nospam@example.com (Johannes Schlüter) — Thu, 19 Aug 2010 00:19:39 +0200

Oh time is flying! - This weekend it is already time for FrOSCon, the Free and Open source Conference in St. Augustin close to Western Germany's former capitol Bonn. The conference consists out of a main track and different side tracks, like the PHP developer room and the OpenSQL sub-conference.

In the PHP developer room I will give an overview over things that happened at MySQL, especially in regards to PHP in recent times. My colleague Ulf Wendel will then go and talk about plugins to mysqlnd - the MySQL native driver for PHP - in detail.

In the OpenSQL Camp track you can find other interesting MySQL related talks which will, unfortunately, not leave you with enough time to watch all the interesting talks of the other tracks. And that all for an entrance fee of just 5€! So if you have a chance: Go there and say hi!

Scalar type hints in PHP trunk

nospam@example.com (Johannes Schlüter) — Sat, 07 Aug 2010 23:00:00 +0200

So in my blog series I try to cover all additions to PHP trunk so I have to mention scalar type hints.

<?php
function print_float(float $f) {
    echo $f."\n";
}

for ($i = 1; $i < 5; $i++) {
    print_float( $i / 3 );
}
?>
	
0.33333333333333
0.66666666666667

Catchable fatal error: Argument 1 passed to print_float() must be of the type double, integer given, called in typehints.php on line 7 and defined in typehints.php on line 2

Is expected behavior in PHP's trunk. If you want such a thing to work please use the numeric type hint.

In case that wasn't enought fun: There's more!

<?php
function handle_result(int $i) {
    echo $i."\n";
}

$pdo = new PDO("mysql:host=localhost;dbname=test", "user", "pass");
$pdo->setAttribute(PDO::MYSQL_ATTR_DIRECT_QUERY, false);
$result = $pdo->query("SELECT 42 AS id");
$row = $result->fetch();
handle_result($row['id']);

$pdo = new PDO("mysql:host=localhost;dbname=test", "user", "pass");
$pdo->setAttribute(PDO::MYSQL_ATTR_DIRECT_QUERY, true);
$result = $pdo->query("SELECT 42 AS id");
$row = $result->fetch();
handle_result($row['id']);
?>

42

Catchable fatal error: Argument 1 passed to handle_result() must be of the type integer, string given, called in typehints.php on line 16 and defined in typehints.php on line 2

So what happens here? - Depending on the PDO::MYSQL_ATTR_DIRECT_QUERY option PDO will either emulate prepared statements (which would be irrelevant here, but that's another story) or use native prepared statements. When using prepared statements MySQL switches over to its "binary" protocol which returns the native types, else it always returns strings. When using PDO_mysql linked against mysqlnd the native type is returned to PHP. If I would use libmysql both times a string would be returned. In other words: The behavior is system dependent. The default behavior for other drivers isn't defined either. As PHP is a dynamically typed language this shouldn't matter, so depending on your driver the results vary. Great. Oh and did I mention that the types aren't specified, so a later version of a driver might change it. And PDO is just a simple example for this ...

The key point of this all is that as soon as you mix strong typing - by using type hints - with PHP's weak type system you open a can of worms.

And to be overly clear: I can understand the need for strict type systems and see where such a thing helps. But adding a strong type system, stricter than most other languages (hey, you can't pass an integer where a float is expected!), makes little sense to me. But that's just me.

As in my previous posts in the series of blog posts about features in PHP trunk. Please try the snapshots. Mind that these features might (not) end up in the next release of PHP, feedback is welcome.

Features in PHP trunk: Array dereferencing

nospam@example.com (Johannes Schlüter) — Sat, 31 Jul 2010 13:11:39 +0200

I was writing about new features in the upcoming PHP version (5.4, 6.0?) before. Today's topic reads like this in the NEWS file:

- Added array dereferencing support. (Felipe)

Now you might wonder what this typical short entry means. when doing OO-style PHP you might make use of a sntax feature which one might call "object dereferencing" which looks like this:

<?php
class Foo {
    public function bar() { }
}

function func() {
    return new Foo();
}

func()->bar();
?>

So one can chain method calls or property access. Now for a long time people requested the same thing for array offset access. This was often rejected due to uncertainties about memory issues, as we don't like memory leaks. But after proper evaluation Felipe committed the patch which allows you to do things like

<?php
function foo() {
    return array(1, 2, 3);
}
echo foo()[2]; // prints 3
?>

Of course this also works with closures:

<?php
$func = function() { return array('a', 'b', 'c'); };
echo $func()[0]; // prints a
?>

And even though the following example is stupid I might accept this feature as one of the few places where it is ok to use references in PHP:

<?php
$data = array('me', 'myself', 'you');
function &get_data() {
    return $GLOBALS['data'];
}
get_data()[2] = 'I'; // $data will now contain 'me', 'myself' and 'I'
?>

Wonderful, isn't it? If you want to test it please take a look at the recent snapshots for PHP trunk and send us your feedback! Please mind that all features in PHP trunk may or may not appear in the next major PHP release.

ZFS and VirtualBox

nospam@example.com (Johannes Schlüter) — Sun, 25 Jul 2010 15:34:05 +0200

With ZFS you may not only do "file system" stuff but ZFS may also provide raw block devices ("zvol") which benefit from ZFS space accounting, snapshotting, checksumming, etc. A purpose of these is to use these zvols and exporting them via iSCSI or give them to applications which can store data on them. One application for this I'm using s VirtualBox and as I always forget the exact commands needed to create a zvol and making it available to VBox I decided to write it down.

Reasons for me for using zvols instead of regular VBox disks are that I can easily snapshot them (every 15 minutes a snapshot ...) individually and can easily clone them (around one second and barely any disk space needed to get a clone of a VM to do some experimental stuff...) and incremental backups using snapshots and zfs send. That said there's at least one - possibly - negative factor: A regular virtual disk file can be shared with other people and other operating systems, a zvol has to be dumped into a regular vmdk first.

Anyways here are the steps needed:

# zfs create -V 10G tank/some_name
# chown johannes /dev/zvol/rdsk/tank/some_name
# VBoxManage internalcommands createrawvmdk \
   -filename /home/johannes/VBoxdisks/some_name.vmdk \
   -rawdisk /dev/zvol/rdsk/tank/some_name
# VBoxManage registerimage disk /home/johannes/VBoxdisks/some_name.vmdk

So first we create the zvol with a size of 10G. This won't be allocated but everybody asking for the size of the device will get this information back and this is the maximum size that will be used - as one can use compression and dedup there this often might be way less usage. Then, as I'm running VBox under my user account, I give my user all the rights needed by making the regular user the owner. The third step creates a vmdk file pointing to the raw device location which is then registered with VirtualBox so a VM can be configured for using it.

Works nicely.

15 years of PHP

nospam@example.com (Johannes Schlüter) — Tue, 08 Jun 2010 00:00:00 +0200

Wow, 15 years ago, on June 8th 1995, Rasmus announced PHP 1.0. Time for a personal look back:

I can't remember when I first went online. Out from the local BBS systems into the wide open net. It must have been around the same time. But soon I figured out that I needed my own homepage, so I created one, using black fonts on a green (#00ff00) background. I was proud. The only issue: I had no idea how to make it accessible to others, I had the HTML file local on the PC, but well, I still was proud. I was proud since the effect of this HTML with a little bit of JavaScript was way stronger than most of my BASIC stuff I did before.

A bit later, in 1998, a cousin brought a magazine about writing Perl CGI applications which caught my attention so I started learning about CGI and Perl and all the related stuff. Back then I always used my own,custom data formats for storing data. But my brother had mercy and introduced me to MySQL. I knew Paradox (and BerkeleyDB) from my Delphi programming but the powers of SQL were amazing. So I was happily building my stuff Perl+MySQL till ... yeah till the server broke down. CGI didn't work anymore for some reason.

My brother, who setup our Linux box at home but didn't know much about Apache, had the pragmatic solution: PHP worked. So what did I do? - Learn PHP. And I loved it. That was in 1999. After that many tings happened. PHP evolved and became a strong player in the web market. I worked for companies like Mayflower, MySQL, Sun Microsystems and, most likely, soon Oracle. And even though I'm using way more C and C++ these days most of it is still directly related to PHP and this all due to some Greenlandic guy who published hist Personal Home Page Tools and my brother who couldn't configure CGI.

15 years and still strong. Congratulations PHP!

Jason, let me help you!

nospam@example.com (Johannes Schlüter) — Thu, 03 Jun 2010 19:01:00 +0200

In a few previous blog posts I was already talking about changes in PHP's trunk, like some BC break or some new features, time to continue:

JSON is the modern data format used in "AJAX" applications. As the leading language for the Web PHP course has support for handling JSON data: You can pass any data to json_encode() and the function will create a JSON representation of this data:

<?php
echo json_encode(array(1,2,3,4));
?>

[1,2,3,4]

This also works for objects:

<?php
$o = new stdclass;
$o->a = 42;
echo json_encode($o);
?>

{"a":42}

Wonderful. But the world isn't that easy. For many PHP objects the JSON-representation of the data is a bit more complex.for instance what about private properties or maybe you want to calculate some inner values? - In PHP 5.3 you were on your own. but thanks to Sara there's hope in sight: the new interface JsonSerializable. Classes implementing this interface have to provide a method jsonSerialize() which will be called by json_encode() and has to return a JSON-compatible representation of the data by doing whatever you want. So let's take a look:

<?php
class JsonTest implements JsonSerializable {
    private $a, $b;

    public function __construct($a, $b) {
        $this->a = $a;
        $this->b = $b;
    }

    public function jsonSerialize() {
        return $this->a + $this->b;
    }
}

echo json_encode(new JsonTest(23, 42));
?>

65

Now this example in itself is of course useless, but let's create a bigger structure, which includes a few of these objects:

<?php
$data = array(
    new stdClass();
    new JsonTest(1,2),
    new JsonTest(3,4),
    array(5,6)
);
echo json_encode($data);
?>

[{},3,7,[5,6]]

Of course you'd usually have more complex serialization logic, but that's left to you.

Now almost certainly somebody will ask "and what about the other way round?" - The only answer there is: Sorry there we can't do much. JSON doesn't encode and meta-information so our generic parser in json_decode() can't do anything special. But anyways: The new interface will certainly be useful.

At last a short disclaimer: PHP trunk is a development tree. Features in their may be changed at anytime and might not be released. Feedback is welcome.

Berkeley DB 5 and PHP

nospam@example.com (Johannes Schlüter) — Thu, 03 Jun 2010 15:54:00 +0200

Everybody working on Unix or in the database world stumbles over Oracle Berkeley DB every now and then. DB is an Open Source embedded database used by applications like OpenLDAP or Postfix. Traditionally it followed mostly a key-value access pattern. Now what caught my attention was the fact that the recently released DB 5.0 provides an SQLite-like C API with the promise of providing better concurrency and performance than regular SQLite. Time to give it a shot.

So I grabbed the source distribution, checked the documentation and saw that I shall use the --enable-sql_compat configure option so DB creates a libsqlite3.so which can be used by PHP's build system. On my system compiling this worked like a charm and after linking PHP against it the SQLite3 extension as well as th SQLite PDO driver worked as before. I didn't do any benchmarks, yet, but I would be interested to read some results. Oh and yes, according to the documentation there are a few minor changes in behavior, this should only affect very few PHP users though.

So the whole process after extracting DB and PHP sources looked like this:

$ cd db-5.0.21/build_unix
$ ../dist/configure --prefix=/opt/db-5.0.21 --enable-sql_compat --enable-cxx
$ make
# make install
$ cd ../..
$ cd php-src
$ ./configure --with-sqlite3=/opt/db-5.0.21 --with-pdo-sqlite=/opt/db-5.0.21 --with-other-php-options
$ make
# make install

Now you don't have to use the SQLite interface for using DB but can also use PHP's dba extension. In current releases of PHP PHP expects at most DB version number 4 for the DBA driver, but a patch was committed to PHP's svn repository so upcoming PHP releases can be build using the latest DB version by adding --with-db4=/opt/db-5.0.21 to PHP's configure line.

Now I can hear you complain: "three ways to access a database isn't enough!" - which is true as the SQLite3 extension as well as the PDO SQLite driver are limited to SQLite capabilities and DBA, as an abstraction layer for Berkeley DB-like databases, only offers some basic operations, too. But there's hope:

While browsing through the DB source distribution I stumbled over a directory called php_db4 which obviously caught my further attention, so the DB developers are bundling a DB-specific PHP extension wrapping DB's native C++ API in an PHP OO API. To try it simply follow the steps from the PHP manual for compiling custom extensions:

$ cd db-5.0.21/php_db4
$ phpize
$ ./configure --with-db4=/opt/db-5.0.21
$ make
# make install

And then a quick check using reflection to see what functionality is provided by this extension:

$ php -dextension=db4.so --re db4
Extension [  extension #45 db4 version 0.9 ] {

  - Constants [210] {
    Constant [ integer DB_VERSION_MAJOR ] { 5 }
    Constant [ integer DB_VERSION_MINOR ] { 0 }
    Constant [ integer DB_VERSION_PATCH ] { 21 }
    Constant [ string DB_VERSION_STRING ] { Berkeley DB 5.0.21: (March 30, 2010) }
    Constant [ integer DB_MAX_PAGES ] { 4294967295 }
    Constant [ integer DB_MAX_RECORDS ] { 4294967295 }
    Constant [ integer DB_DBT_APPMALLOC ] { 1 }
    Constant [ integer DB_DBT_ISSET ] { 8 }
    Constant [ integer DB_DBT_MALLOC ] { 16 }
    Constant [ integer DB_DBT_PARTIAL ] { 64 }
    Constant [ integer DB_DBT_REALLOC ] { 128 }
    Constant [ integer DB_DBT_USERMEM ] { 1024 }
    Constant [ integer DB_DBT_DUPOK ] { 4 }
    Constant [ integer DB_CREATE ] { 1 }
    Constant [ integer DB_CXX_NO_EXCEPTIONS ] { 2 }
    Constant [ integer DB_FORCE ] { 1 }
    Constant [ integer DB_NOMMAP ] { 8 }
    Constant [ integer DB_RDONLY ] { 1024 }
    Constant [ integer DB_RECOVER ] { 2 }
    Constant [ integer DB_MULTIVERSION ] { 4 }
    Constant [ integer DB_TXN_SNAPSHOT ] { 2 }
    Constant [ integer DB_THREAD ] { 16 }
    Constant [ integer DB_TRUNCATE ] { 32768 }
    Constant [ integer DB_TXN_NOSYNC ] { 1 }
    Constant [ integer DB_TXN_NOT_DURABLE ] { 2 }
    Constant [ integer DB_USE_ENVIRON ] { 4 }
    Constant [ integer DB_USE_ENVIRON_ROOT ] { 8 }
    Constant [ integer DB_AUTO_COMMIT ] { 256 }
    Constant [ integer DB_DIRTY_READ ] { 512 }
    Constant [ integer DB_DEGREE_2 ] { 1024 }
    Constant [ integer DB_READ_COMMITTED ] { 1024 }
    Constant [ integer DB_READ_UNCOMMITTED ] { 512 }
    Constant [ integer DB_NO_AUTO_COMMIT ] { 8192 }
    Constant [ integer DB_RPCCLIENT ] { 1 }
    Constant [ integer DB_INIT_CDB ] { 64 }
    Constant [ integer DB_INIT_LOCK ] { 128 }
    Constant [ integer DB_INIT_LOG ] { 256 }
    Constant [ integer DB_INIT_MPOOL ] { 512 }
    Constant [ integer DB_INIT_REP ] { 1024 }
    Constant [ integer DB_INIT_TXN ] { 2048 }
    Constant [ integer DB_JOINENV ] { 0 }
    Constant [ integer DB_LOCKDOWN ] { 4096 }
    Constant [ integer DB_PRIVATE ] { 8192 }
    Constant [ integer DB_RECOVER_FATAL ] { 16384 }
    Constant [ integer DB_SYSTEM_MEM ] { 65536 }
    Constant [ integer DB_EXCL ] { 64 }
    Constant [ integer DB_FCNTL_LOCKING ] { 2048 }
    Constant [ integer DB_RDWRMASTER ] { 16384 }
    Constant [ integer DB_WRITEOPEN ] { 65536 }
    Constant [ integer DB_TXN_NOWAIT ] { 16 }
    Constant [ integer DB_TXN_SYNC ] { 4 }
    Constant [ integer DB_ENCRYPT_AES ] { 1 }
    Constant [ integer DB_CDB_ALLDB ] { 64 }
    Constant [ integer DB_DIRECT_DB ] { 128 }
    Constant [ integer DB_NOLOCKING ] { 1024 }
    Constant [ integer DB_NOPANIC ] { 2048 }
    Constant [ integer DB_OVERWRITE ] { 4096 }
    Constant [ integer DB_PANIC_ENVIRONMENT ] { 8192 }
    Constant [ integer DB_REGION_INIT ] { 16384 }
    Constant [ integer DB_TIME_NOTGRANTED ] { 32768 }
    Constant [ integer DB_TXN_WRITE_NOSYNC ] { 32 }
    Constant [ integer DB_YIELDCPU ] { 65536 }
    Constant [ integer DB_UPGRADE ] { 1 }
    Constant [ integer DB_VERIFY ] { 2 }
    Constant [ integer DB_DIRECT ] { 16 }
    Constant [ integer DB_EXTENT ] { 64 }
    Constant [ integer DB_ODDFILESIZE ] { 128 }
    Constant [ integer DB_CHKSUM ] { 8 }
    Constant [ integer DB_DUP ] { 16 }
    Constant [ integer DB_DUPSORT ] { 4 }
    Constant [ integer DB_ENCRYPT ] { 1 }
    Constant [ integer DB_RECNUM ] { 64 }
    Constant [ integer DB_RENUMBER ] { 128 }
    Constant [ integer DB_REVSPLITOFF ] { 256 }
    Constant [ integer DB_SNAPSHOT ] { 512 }
    Constant [ integer DB_STAT_CLEAR ] { 1 }
    Constant [ integer DB_JOIN_NOSORT ] { 1 }
    Constant [ integer DB_AGGRESSIVE ] { 1 }
    Constant [ integer DB_NOORDERCHK ] { 2 }
    Constant [ integer DB_ORDERCHKONLY ] { 4 }
    Constant [ integer DB_PR_PAGE ] { 16 }
    Constant [ integer DB_PR_RECOVERYTEST ] { 32 }
    Constant [ integer DB_PRINTABLE ] { 8 }
    Constant [ integer DB_SALVAGE ] { 64 }
    Constant [ integer DB_REP_NOBUFFER ] { 2 }
    Constant [ integer DB_REP_PERMANENT ] { 4 }
    Constant [ integer DB_LOCKVERSION ] { 1 }
    Constant [ integer DB_FILE_ID_LEN ] { 20 }
    Constant [ integer DB_LOCK_NORUN ] { 0 }
    Constant [ integer DB_LOCK_DEFAULT ] { 1 }
    Constant [ integer DB_LOCK_EXPIRE ] { 2 }
    Constant [ integer DB_LOCK_MAXLOCKS ] { 3 }
    Constant [ integer DB_LOCK_MINLOCKS ] { 5 }
    Constant [ integer DB_LOCK_MINWRITE ] { 6 }
    Constant [ integer DB_LOCK_OLDEST ] { 7 }
    Constant [ integer DB_LOCK_RANDOM ] { 8 }
    Constant [ integer DB_LOCK_YOUNGEST ] { 9 }
    Constant [ integer DB_LOCK_NOWAIT ] { 1 }
    Constant [ integer DB_LOCK_RECORD ] { 2 }
    Constant [ integer DB_LOCK_SET_TIMEOUT ] { 4 }
    Constant [ integer DB_LOCK_SWITCH ] { 8 }
    Constant [ integer DB_LOCK_UPGRADE ] { 16 }
    Constant [ integer DB_HANDLE_LOCK ] { 1 }
    Constant [ integer DB_RECORD_LOCK ] { 2 }
    Constant [ integer DB_PAGE_LOCK ] { 3 }
    Constant [ integer DB_LOGVERSION ] { 17 }
    Constant [ integer DB_LOGOLDVER ] { 8 }
    Constant [ integer DB_LOGMAGIC ] { 264584 }
    Constant [ integer DB_ARCH_ABS ] { 1 }
    Constant [ integer DB_ARCH_DATA ] { 2 }
    Constant [ integer DB_ARCH_LOG ] { 4 }
    Constant [ integer DB_ARCH_REMOVE ] { 8 }
    Constant [ integer DB_FLUSH ] { 1 }
    Constant [ integer DB_LOG_CHKPNT ] { 2 }
    Constant [ integer DB_LOG_COMMIT ] { 4 }
    Constant [ integer DB_LOG_NOCOPY ] { 8 }
    Constant [ integer DB_LOG_NOT_DURABLE ] { 16 }
    Constant [ integer DB_LOG_WRNOSYNC ] { 32 }
    Constant [ integer DB_user_BEGIN ] { 10000 }
    Constant [ integer DB_debug_FLAG ] { 2147483648 }
    Constant [ integer DB_LOG_DISK ] { 1 }
    Constant [ integer DB_LOG_LOCKED ] { 2 }
    Constant [ integer DB_LOG_SILENT_ERR ] { 4 }
    Constant [ integer DB_MPOOL_CREATE ] { 1 }
    Constant [ integer DB_MPOOL_LAST ] { 16 }
    Constant [ integer DB_MPOOL_NEW ] { 32 }
    Constant [ integer DB_MPOOL_DIRTY ] { 2 }
    Constant [ integer DB_MPOOL_DISCARD ] { 1 }
    Constant [ integer DB_MPOOL_NOFILE ] { 1 }
    Constant [ integer DB_MPOOL_UNLINK ] { 2 }
    Constant [ integer DB_TXNVERSION ] { 1 }
    Constant [ integer DB_GID_SIZE ] { 128 }
    Constant [ integer DB_EID_BROADCAST ] { -1 }
    Constant [ integer DB_EID_INVALID ] { -2 }
    Constant [ integer DB_REP_CLIENT ] { 1 }
    Constant [ integer DB_REP_MASTER ] { 2 }
    Constant [ integer DB_RENAMEMAGIC ] { 198656 }
    Constant [ integer DB_BTREEVERSION ] { 9 }
    Constant [ integer DB_BTREEOLDVER ] { 8 }
    Constant [ integer DB_BTREEMAGIC ] { 340322 }
    Constant [ integer DB_HASHVERSION ] { 9 }
    Constant [ integer DB_HASHOLDVER ] { 7 }
    Constant [ integer DB_HASHMAGIC ] { 398689 }
    Constant [ integer DB_QAMVERSION ] { 4 }
    Constant [ integer DB_QAMOLDVER ] { 3 }
    Constant [ integer DB_QAMMAGIC ] { 270931 }
    Constant [ integer DB_AFTER ] { 1 }
    Constant [ integer DB_APPEND ] { 2 }
    Constant [ integer DB_BEFORE ] { 3 }
    Constant [ integer DB_CONSUME ] { 4 }
    Constant [ integer DB_CONSUME_WAIT ] { 5 }
    Constant [ integer DB_CURRENT ] { 6 }
    Constant [ integer DB_FAST_STAT ] { 1 }
    Constant [ integer DB_FIRST ] { 7 }
    Constant [ integer DB_GET_BOTH ] { 8 }
    Constant [ integer DB_GET_BOTHC ] { 9 }
    Constant [ integer DB_GET_BOTH_RANGE ] { 10 }
    Constant [ integer DB_GET_RECNO ] { 11 }
    Constant [ integer DB_JOIN_ITEM ] { 12 }
    Constant [ integer DB_KEYFIRST ] { 13 }
    Constant [ integer DB_KEYLAST ] { 14 }
    Constant [ integer DB_LAST ] { 15 }
    Constant [ integer DB_NEXT ] { 16 }
    Constant [ integer DB_NEXT_DUP ] { 17 }
    Constant [ integer DB_NEXT_NODUP ] { 18 }
    Constant [ integer DB_NODUPDATA ] { 19 }
    Constant [ integer DB_NOOVERWRITE ] { 20 }
    Constant [ integer DB_NOSYNC ] { 21 }
    Constant [ integer DB_POSITION ] { 23 }
    Constant [ integer DB_PREV ] { 24 }
    Constant [ integer DB_PREV_NODUP ] { 26 }
    Constant [ integer DB_SET ] { 27 }
    Constant [ integer DB_SET_LOCK_TIMEOUT ] { 1 }
    Constant [ integer DB_SET_RANGE ] { 28 }
    Constant [ integer DB_SET_RECNO ] { 29 }
    Constant [ integer DB_SET_TXN_NOW ] { 8 }
    Constant [ integer DB_SET_TXN_TIMEOUT ] { 2 }
    Constant [ integer DB_UPDATE_SECONDARY ] { 30 }
    Constant [ integer DB_WRITECURSOR ] { 8 }
    Constant [ integer DB_WRITELOCK ] { 16 }
    Constant [ integer DB_OPFLAGS_MASK ] { 255 }
    Constant [ integer DB_MULTIPLE ] { 2048 }
    Constant [ integer DB_MULTIPLE_KEY ] { 16384 }
    Constant [ integer DB_RMW ] { 8192 }
    Constant [ integer DB_DONOTINDEX ] { -30998 }
    Constant [ integer DB_KEYEMPTY ] { -30996 }
    Constant [ integer DB_KEYEXIST ] { -30995 }
    Constant [ integer DB_LOCK_DEADLOCK ] { -30994 }
    Constant [ integer DB_LOCK_NOTGRANTED ] { -30993 }
    Constant [ integer DB_NOSERVER ] { -30990 }
    Constant [ integer DB_NOSERVER_HOME ] { -30989 }
    Constant [ integer DB_NOSERVER_ID ] { -30988 }
    Constant [ integer DB_NOTFOUND ] { -30987 }
    Constant [ integer DB_OLD_VERSION ] { -30986 }
    Constant [ integer DB_PAGE_NOTFOUND ] { -30985 }
    Constant [ integer DB_REP_DUPMASTER ] { -30984 }
    Constant [ integer DB_REP_HANDLE_DEAD ] { -30983 }
    Constant [ integer DB_REP_HOLDELECTION ] { -30982 }
    Constant [ integer DB_REP_ISPERM ] { -30980 }
    Constant [ integer DB_REP_NEWMASTER ] { -30893 }
    Constant [ integer DB_REP_NEWSITE ] { -30976 }
    Constant [ integer DB_REP_NOTPERM ] { -30975 }
    Constant [ integer DB_REP_UNAVAIL ] { -30974 }
    Constant [ integer DB_RUNRECOVERY ] { -30973 }
    Constant [ integer DB_SECONDARY_BAD ] { -30972 }
    Constant [ integer DB_VERIFY_BAD ] { -30970 }
    Constant [ integer DB_VERB_DEADLOCK ] { 1 }
    Constant [ integer DB_VERB_RECOVERY ] { 8 }
    Constant [ integer DB_VERB_REPLICATION ] { 32 }
    Constant [ integer DB_VERB_WAITSFOR ] { 32768 }
  }

  - Classes [4] {
    Class [  class db4txn ] {

      - Constants [0] {
      }

      - Static properties [0] {
      }

      - Static methods [0] {
      }

      - Properties [0] {
      }

      - Methods [8] {
        Method [  public method abort ] {
        }

        Method [  public method commit ] {
        }

        Method [  public method discard ] {
        }

        Method [  public method id ] {
        }

        Method [  public method set_timeout ] {
        }

        Method [  public method set_name ] {
        }

        Method [  public method get_name ] {
        }

        Method [  public method db4txn ] {
        }
      }
    }

    Class [  class db4cursor ] {

      - Constants [0] {
      }

      - Static properties [0] {
      }

      - Static methods [0] {
      }

      - Properties [0] {
      }

      - Methods [7] {
        Method [  public method close ] {
        }

        Method [  public method count ] {
        }

        Method [  public method del ] {
        }

        Method [  public method dup ] {
        }

        Method [  public method get ] {

          - Parameters [2] {
            Parameter #0 [  &$param0 ]
            Parameter #1 [  &$param1 ]
          }
        }

        Method [  public method put ] {
        }

        Method [  public method pget ] {

          - Parameters [2] {
            Parameter #0 [  &$param0 ]
            Parameter #1 [  &$param1 ]
          }
        }
      }
    }

    Class [  class db4 ] {

      - Constants [0] {
      }

      - Static properties [0] {
      }

      - Static methods [0] {
      }

      - Properties [0] {
      }

      - Methods [15] {
        Method [  public method db4 ] {
        }

        Method [  public method open ] {
        }

        Method [  public method close ] {
        }

        Method [  public method cursor ] {
        }

        Method [  public method del ] {
        }

        Method [  public method get ] {
        }

        Method [  public method get_encrypt_flags ] {
        }

        Method [  public method pget ] {
        }

        Method [  public method get_type ] {
        }

        Method [  public method join ] {
        }

        Method [  public method put ] {
        }

        Method [  public method set_encrypt ] {
        }

        Method [  public method stat ] {
        }

        Method [  public method sync ] {
        }

        Method [  public method truncate ] {
        }
      }
    }

    Class [  class db4env ] {

      - Constants [0] {
      }

      - Static properties [0] {
      }

      - Static methods [0] {
      }

      - Properties [0] {
      }

      - Methods [11] {
        Method [  public method db4env ] {
        }

        Method [  public method close ] {
        }

        Method [  public method dbremove ] {
        }

        Method [  public method dbrename ] {
        }

        Method [  public method get_encrypt ] {
        }

        Method [  public method open ] {
        }

        Method [  public method remove ] {
        }

        Method [  public method set_data_dir ] {
        }

        Method [  public method set_encrypt ] {
        }

        Method [  public method txn_begin ] {
        }

        Method [  public method txn_checkpoint ] {
        }
      }
    }
  }
}

The extension bundles some examples to get you started with Berkeley DB.

Even though I'd prefer a relational client-server database for most use cases I can image situations where a simple embedded solutions has its benefits, especially if combined with other Oracle Berkeley DB based applications.

Now in trunk: Improved interactive shell

nospam@example.com (Johannes Schlüter) — Fri, 28 May 2010 20:55:00 +0200

A few years ago I used another blog to write about "More PHP power on the command line" almost 5 years later the PHP interactive shell got a major update which went in PHP's trunk. The commit message tells a lot about the improvements:

- Improved CLI Interactive readline shell (Johannes)
  . Added cli.pager ini setting to set a pager for output.
  . Added cli.prompt ini settingto configure the shell prompt.
  . Added shortcut #inisetting=value to change ini settings at run-time.
  . Don't terminate shell on fatal errors.

A pager can be a an shell command which will receive the command output on its
STDIN channel

php > #cli.pager=less
php > phpinfo();
(output will appear in the pager)
php > #cli.pager=grep -i readline
php > phpcredits();
Readline => Thies C. Arntzen
php > #cli.pager=
(output appears again direct on the terminal)

A prompt can contain a few escape sequences like

php > #cli.prompt=\e[032m\v \e[031m\b \e[34m\> \e[0m
5.3.99-dev php > //Colorful prompt with version number

A prompt can also contaian PHP code in backticks

php > #cli.prompt=`echo gethostname();` \b \>
guybrush php >

But I assume a screenshot is quite useful, so here it goes:

So what doyou need to use this feature? - PHP trunk compiled using --with-readline or --with-libedit.The PHP Documentation has information about the interactive mode, the new features will be added once trunk is closer to a release and it's clear in what release this will appear.

ZFS

nospam@example.com (Johannes Schlüter) — Mon, 10 May 2010 20:10:00 +0200

So, I use OpenSolaris on most of my boxes as native operating system. VMs run other OSs. My choice for OpenSolaris was driven by the availability of DTrace. One of the greatest tools for system/program analysis ever created. By running OpenSolaris I've also got ZFS which is Oracle's über file system. I never really cared about ZFS, at least not until I missed it. So ZFS integrates all the different storage layers in one system - RAID-controller, logical volume manager, POSIX file system layer, ... Really nice to have that integrated, eases management. Now I don't change my disks that often and the file system silently runs underneath. From time to time I looked into my auto snapshot to restore some stuff and got used to snapshot my VMs (running on ZFS-powered "virtual" zvol devices) before updating them which over time became a habit about which I didn't really think.

Then I've got myself a netbook. Some cheap up to date ASUS EeePC. On that system I choose to install Ubuntu - which was troublesome enough (had to compile my owned manually patched wireless driver) so I didn't bother to try OpenSolaris. Works like a charm, even without ZFS. Some time after I configured the netbook a new Ubuntu release came out and since then I'm in trouble. I read on too many sites that things broke with this release so I dare to update the system. On my OpenSolaris boxes updating to a update, even to a dev build, is a no-brainer: The packaging system automatically creates a ZFS snapshot and configures the boot loader in a way that the old as well as the new system can be booted. So I can click the update button, reboot and either it works (typical case) or I can revert. Really nice.

Now back to Ubuntu: If I press the button and something goes wrong I have to reinstall the system (or use a backup) which I don't want. I just want to use the netbook as a mobile browser, presentation system etc. There are other systems I use to play/experiment with...

At the recent PHP Barcamp Salzburg we got to a discussion about ZFS, too. In the discussion there was talk about the auto snapshotting and a claim was "well, I won't need it, I have everything in a version control software and I know what I delete" that might be true but once you have ZFS you change your way to operate and you don't have the whole system in a version control thing. It's so great to be able to clone a VM in less than a second to play with some stuff. It is cool to be able to enable compression with one short shell command. It's fantastic to have a fully checksummed filesystem with RAID-Z. Man how did we live in the old days? Nice to e aware of the luxury I'm used to

P.S. This blog is running on ZFS, too - of course, gave a good feeling to be able to revert during today's update, too.

Mind the encodings!

nospam@example.com (Johannes Schlüter) — Sun, 28 Mar 2010 03:31:00 +0200

Dealing with character sets and encodings is tough. As long as you're dealing only with English texts you in a luxury situation and can mix utf-8 and iso-8859-1 encoded texts and most (all?) of your tests will work. Some of your users, like me, with strange names ("Schlüter") will be annoyed as your application breaks them ("SchlÃ¼ter"), but these will be edge cases. There are bigger issues with mixing encodings but that's not what I wanted to tell now.

Handling these encodings in PHP correctly is tough. PHP, currently, has a quite simple approach to the problem in general: PHP doesn't care about encodings. A string is simply a sequence of bytes. Well in general. The details are difficult. Core features like JSON-handling or XML processing expect that your PHP strings are encoded using utf-8, which is a sane choice, for JSON it is part of the JSON specification, for XML it's the only way to be able to work with all documents without additional information from you along every XML operation.

On the other side we have browsers. A browser has to read documents from all over the world which can be encoded in any encoding you like. For whatever reason browser developers decided that iso-8859-1 would make a great default encoding, which means that if a response to a browser's request doesn't specify anything else the browser assumes the document is encoded using iso-8859-1. The document's encoding will also be used when sending form data back to the server. To handle this, PHP has a php.ini setting default_charset which, if set, will set the selected encoding in the HTTP header. The default value of this is setting is empty - so browsers fall back on their default, iso-8859-1.

Recently Rasmus made a commit to PHP trunk which changes the default to utf-8.

In the long-run this changes is good as it works for all languages, iso-8859-1 only works for a limited set of European languages, and works better with the outside environment, where utf-8 adoption is growing (JSON and XML were examples).

In the short-term this might cause trouble for applications depending on the default.The good thing is that the development in trunk has just started and the release of PHP.next is still sometime away and you can easily prepare your application - which is a good thing anyways to protect from administrators making mistakes with current versions already.

To set the encoding from within your application you can for example call ini_set('default_charset', $enc); or header("Content-type: text/html; charset=$enc"); at the beginning of your script, where $enc is your preferred encoding. Please mind that this has no effect on the script itself, which, for instance, means you have to configure your database connection accordingly, too!

Future of PHP 6

nospam@example.com (Johannes Schlüter) — Fri, 12 Mar 2010 13:15:23 +0100

Yesterday was a quite thrilling day for the PHP development team and led to some imprecise news articles so let's take a look at what happened: Over the last months many of the core contributors agreed that the current approach to bring Unicode into PHP's engine wasn't the right approach and a good thing would be to rethink it from the start. By a provocative move of one contributor the stalled situation got some more movement and Rasmus declared the current implementation to be discontinued to restart.

The past

When the foundation of what should have become PHP 6 was created a decision was made to use UTF-16 as internal encoding for "everything" inside the engine. The choice for UTF-16 was made due to the fact that PHP 6 would use the ICU library which is focused on offering UTF-16 string functions. By using UTF-16 as default encoding we'd have to convert the script code and all data passed from or to the script (request data, database results, output, ...) from another encoding, usually UTF-8, to UTF-16 or back. The need for conversion doesn't only require CPU time and more memory (a UTF-16 string takes double memory of a UTF-8 string in many cases) but makes the implementation rather complex as we always have to figure out which encoding was the right one for a given situation. From the userspace point of view the implementation brought some backwards compatibility breaks which would require manual review of the code. These all are pains for a very small gain for many users where many would be happy about a tighter integration of some mbstring-like functionality. This all led to a situation for many contributors not willing to use "trunk" as their main development tree but either develop using the stable 5.2/5.3 trees or refuse to do development at all.

The present

Yesterday the stagnation created by the situation has been resolved and it was decided that our trunk in svn will be based on 5.3 and we'll merge features from the old trunk and new features there so that 5.3 will be a true stable branch. The EOL for 5.2 has not yet been defined but I suggest you to really migrate over to 5.3, which usually can be done with very little work, as soon as possible.

The future

Right now we're starting different discussions to see what kind of Unicode support we really want. Many contributors react positive on a proposed "string class" which wraps string operations in Unicode and binary forms without going deep in the engine. In my opinion such an approach might also be a way to solve some of the often criticized inconsistencies in PHP's string APIs without the need to break old code. (new code uses the new class, old code the old functions) But that idea is far from a proper proposal or even the implementation, current status is about refocusing the development and get the requirement and design discussions going. By that it's a bit early to decide whether the next version of PHP will be called PHP 5.4, PHP 6 or maybe even PHP 7.

PHP is alive and kicking!

Ob ich es je verstehen werde?

nospam@example.com (Johannes Schlüter) — Tue, 02 Mar 2010 14:03:00 +0100

Heute feiert die deutsche Internet-Community und ja das Urteil des BVerG ist zu begrÃ¼ÃŸen. Aber noch immer frage ich mich warum man einem Privatunternehmen deutlich mehr Daten freiwillig gibt.

Ich schrieb es ja zuvor schon: Ich verstehe nicht wie Leute all Ihre Daten einem Unternehmen kÃ¶nnen und zugleich dagegen sind wenn der Staat ein Bruchteil der Daten per Gesetz bekommt. Ich weiÃŸ nicht ob Google heute selber irgendwas bÃ¶ses mit den Daten macht, nach allem was man so hÃ¶rt wohl nicht, aber zum einen haben staatliche Stellen schon heute Zugang zu den Daten bei Google zum anderen weiÃŸ man nicht was Google macht, wenn das WerbegeschÃ¤ft mal was schlechter lÃ¤uft, inzwischen haben die nen groÃŸen Stapel Mitarbeiter zu fÃ¼ttern und AktionÃ¤re zufrieden zustellen.

Ja, Google gibt Daten an BehÃ¶rden raus. In den USA ist das im Rahmen des USA PATRIOT Act von 2001 geregelt und Eric Schmidt, CEO von Google, hat das ja unlÃ¤ngst in einem Interview betont. In Deutschland haben wir zum GlÃ¼ck (noch?) kein vergleichbares Gesetz, dass Geheimdiensten und PolizeibehÃ¶rden derart umfangreiche Rechte einrÃ¤umt, wie dies im Patriot Act der Fall ist aber auch hier gibt es MÃ¶glichkeiten, die zur Beschlagnahme von Daten von Google fÃ¼hren kÃ¶nnen .. oder auch sowas wie Geheimdienstliche Zusammenarbeit, die einen Zugriff deutscher BehÃ¶rden nicht unmÃ¶glich machen (obei dann die Frage ist ob die CIA oder das BKA schlimmer ist )

Aber nur um nochmal zu erinnern was fÃ¼r Daten Google hat, dass sind ja nicht nur die Suchanfragen, die man mal so stellt sondern auch die Information welche Websites man so aufruft (Danke Google Ads und Google Analytics Ã¼berall) das sind Mails die man so schreibt (und sei es nur weil der EmpfÃ¤nger GMail nutzt) das sind Informationen mit wem man wann so telefoniert (Android des angerufenen is ja an GMail&Co. angebunden) In den USA sind das auch Inhalte von Mailboxnachrichten (die per Spracherkennung als Mail verschickt werden). Dank Picassa und Photoerkennung kennen die auch das Gesicht und Aufenthaltsorte (das is eigentlich der spannende Teil bei der Streetview-Diskussion - Google hat die unzensierten Bilder und automatische Gesichtserkennung ...) und noch viel mehr an Daten. Das kombiniert mit den Suchanfragen bei der Google Suche, der Produktsuche, Google Maps, .... gibt ein extrem detailliertes PersÃ¶nlichkeitsbild. Nein. Ich verstehe nicht warum man sowas in einer einzigen Datenbank, die man nicht kontrolliert, haben wollen kÃ¶nnte.