Frames of Reference - PHP

MSN Live Search API V2.0 in PHP

yeager — Sat, 03 Jan 2009 21:59:45 +0000

At the end of 2008, Microsoft abandoned their old search API. While older applications continue to work, new AppID’s only work with the new V2.0 API. If you’ve created a new AppID, and wonder why none of the code samples you find on the Web work with it, now you know. It’s not hard to update code to use the newer (still in beta) V2.0 API, and we’ll do exactly that by updating our previous API search code. If you’ve used the old code, it’s quite simple to update to the new code. Just copy the file, and change one line in your PHP page — the new code requires PHP5, so be sure your server has PHP5 installed.

Quick Update

Get the new Search API V2.0 PHP class and copy it to the same location as the older code. Find the line in your PHP script reading include ("MSNSearch.php") and replace it with include ("MSNSearchV2.php"). You’ll then need to get a new AppID from Microsoft and update your page (V1.0 AppID’s don’t work with V2.0 API, and vice-versa).

How it works

Examine the source code for the class yourself — it formats a HTTP URL and sends it to Microsoft’s server, receiving the search response as XML. It then uses PHP’s SimpleXML to parse the results (only in PHP5, which is why PHP5 is required). The class has a few helper functions to assist so you don’t have to write search forms or format results, and CSS classes tag all HTML code for easy formatting.

Microsoft’s new V2.0 API’s don’t require SOAP (although you can use it if you want), so the previous requirement of a SOAP library disappears. The new V2.0 API’s have the option to send requests as HTTP (returning results formatted as XML) without using SOAP.

To get a functioning search page, place this PHP code snippet in a script and you’ll have a ready-made search page, with a few customizations for your specific site.

Place the Search API V2.0 PHP class in the same directory as your script (or change the include line to point it to the location of where you placed it).
Place your AppID key instead of “INSERTAPIKEYHERE”.
If you don’t have your script on your webserver as /search-msn.php, change the definition of SEARCH_URL.

<?php
define("SEARCH_URL","/search-msn.php");
include("MSNSearchV2.php");

# Values from HTTP GET, or reasonable defaults
$start = isset($_GET['start']) ? intval($_GET['start']) : 1;
$q = isset($_GET['q']) ? $_GET['q'] : "";
if ($start < 1)
    $start = 1;

print searchform($q,"form_top",SEARCH_URL);
if (strlen($q) > 1) {
  $msnsearch = new MSNSearch('INSERTAPIKEYHERE');
  $sresult = false;
  $msnsearch->setQuery($q);
  $msnsearch->setPage($start);
  $sresult = $msnsearch->search();
  if (($sresult === true) && ($msnsearch->totalRecords > 0)) {
    print $msnsearch->search_header($q);
    print $msnsearch->search_results();
    print $msnsearch->search_navagation($q);
    print searchform($q,"form_bottom",SEARCH_URL);
  } else 
    print "<p>Sorry, no results found for <b>$q</b>.</p>\n";
}
?>

All PHP code on this page licensed under a BSD license.

References:

MSN API V1.0 Search in PHP
MS Live API V2.0 Documentation
MS Live API V1 Documentation (Deprecated)

PHP, XHTML MIME type and Caching

yeager — Thu, 01 Jan 2009 08:00:00 +0000

May 2005 — Updated Dec 2005, July 2006

Introduction

Using PHP for your XHTML web site is great, but if you’re not considering possible effects you could be violating Internet standards and reducing the usability for your users. You might not know your validated and tested XHTML is treated as “tag soup” by browsers, or that without sending the proper headers your pages caching ability is severely reduced, or perhaps some browsers (due to bugs) won’t cache your page at all. When using PHP with XHTML, several issues need to be considered.

XHTML and MIME type. You should be serving XHTML as application/xhtml+xml instead of text/html. But not all clients support it. How to serve it properly to clients following standards, without making the site unusable to outdated clients?
PHP and Caching. A small amount of work is required to improve the caching ability of your site, reduce bandwidth and improve response time.
Default Caching of other static items (CSS, images, etc). This requires mod_expires and a few htaccess lines.
Problems caused by buggy browsers. Internet Explorer has some nasty bugs existing from 4.x all the way to 6.x. This complicates your job as you’ll have to work-around IE’s bugs. Yes, it’s true we shouldn’t have to do this (as it’s the clients fault, not ours) if they fail to follow standards, but in the real world IE is extremely common, and we don’t want to penalize those people just because a company in Washington can’t (or won’t) follow Internet standards.

This article assumes you’ve got a background on Apache and some basic HTTP protocol knowledge; if you don’t know what etag, 304, gzip, If-Modified-Since, If-None-Match, HTTP headers and MIME types are see the references at the end of this article (As always, Google is your friend).

We’ll also assume you’re using Apache on *nix with a recent PHP (4.3.x) version, are interested in writing valid XHTML, following web standards, and improving bandwidth usage of your site.

Caching

Caching makes your site more responsive to the user, but caching involves two different functions. First, avoiding trips to the server if possible (using cache-control and expires), and second, if we must go to the server, avoid transferring the document (using validation and 304 responses).

Unfortunately with PHP caching will generally not work “out of the box”. We’ve got to do a little work to get browsers to optimally cache our documents. Then Apache should be configured such that static documents (CSS, JS, images, etc) will be cached. This requires mod_expires and a few simple htaccess lines.

XHTML and mimetype

W3C says to serve XHTML as application/xhtml+xml, but IE can’t handle it, while modern browsers (Firefox, Opera, etc) can. We’ve got to change the MIME type for modern browsers, while falling back gracefully to outdated browsers like IE. However, we don’t want to “browser sniff” the User-Agent as this is unreliable at best.

Fortunately, the HTTP protocol has a means to do exactly what we want: the Accept header. Each request by a client has the option to specify to the server exactly what it wants. A few examples make this clear.

Firefox 1.0.3 
text/xml,application/xml,application/xhtml+xml,text/html;q=0.9,
text/plain;q=0.8,image/png,*/*;q=0.5

IE 6.0 SP1
image/gif, image/x-xbitmap, image/jpeg, image/pjpeg, application/vnd.ms-excel, 
application/vnd.ms-powerpoint, application/msword, application/x-shockwave-flash, */*

Warning!

You should *never* try out code on your production server. Always have a test or development box for trying out new code. Experimenting on your live server is asking for trouble!

Its easy to determine Firefox prefers application/xhtml+xml, while IE can’t handle it. Notice the line in Firefox saying q=0.9? That’s a Q-Value, and it’s used so a client can say what they prefer, while also saying what the can accept. Q values range from 0.000 to 1.000, although most only use one decimal.

For firefox, it has application/xhtml+xml with no q-value (assume 1), while text/html has q-value of 0.9. Thus Firefox prefers application/xhtml+xml (since 1.0 > .9), although it can accept text/html. By using the Accept header the client sends us, we avoid problematic browser-sniffing and allow the client to tell us what MIME type they prefer.

Solution (The Code)

Here’s the code; we’ll explain it after you’ve looked at it. Save this as include-mime.php and later we’ll have an example that uses it. You can also grab the php_xhtml.tar.gz file available for download.

<?php
##############################################################################
#     XHTML and mimetype script for PHP                                      #
#     Copyright (C) 2005  Darrin Yeager                                      #
#     All rights reserved.                                                   #
#     http://www.dyeager.org                                                 #
#                                                                            #
# Redistribution and use in source and binary forms, with or without         #
# modification, are permitted provided that the following conditions         #
# are met:                                                                   #
#                                                                            #
#   1. Redistributions of source code must retain the above copyright        #
#      notice, this list of conditions and the following disclaimer.         #
#                                                                            #
#   2. Redistributions in binary form must reproduce the above copyright     #
#      notice, this list of conditions and the following disclaimer in the   #
#      documentation and/or other materials provided with the distribution.  #
#                                                                            #
#   3. Redistributions of modified versions must carry prominent notices     #
#      stating that you changed the files and the date of any change.        #
#                                                                            #
#   4. Neither the name of Darrin Yeager nor the names of any contributors   #
#      may be used to endorse or promote products derived from this software #
#      without specific prior written permission.                            #
#                                                                            #
# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS        #
# "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT          #
# LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR      # 
# A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT       #
# OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,      #
# SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED   #
# TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR     #
# PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF     #
# LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING       #
# NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS         #
# SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.               #
#                                                                            #
##############################################################################

$charset = "utf-8";
$mime = "text/html";
$is304 = false;

# NOTE: To allow for q-values with one space (text/html; q=0.5), 
# use the following regex:
# "/text\/html;[\ ]{0,1}q=([0-1]{0,1}\.\d{0,4})/i"
if((isset($_SERVER["HTTP_ACCEPT"])) && (stristr($_SERVER["HTTP_ACCEPT"],"application/xhtml+xml")))  {
   if(preg_match("/application\/xhtml\+xml;q=([0-1]{0,1}\.\d{0,4})/i",$_SERVER["HTTP_ACCEPT"],$matches)) {
      $xhtml_q = $matches[1];
      if(preg_match("/text\/html;q=([0-1]{0,1}\.\d{0,4})/i",$_SERVER["HTTP_ACCEPT"],$matches)) {
         $html_q = $matches[1];
         if((float)$xhtml_q >= (float)$html_q)
            $mime = "application/xhtml+xml";
      }
   }
   else
	  $mime = "application/xhtml+xml";
}

# Get the file stats and compute last-modified time.
$filestats = @stat($_SERVER["SCRIPT_FILENAME"]);
$lastmod = $filestats[9] - date('Z');  #Convert Local time -> GMT

# ETag is "inode-lastmodtime-filesize" - See PHP stat function for more detail
$etag = '"' . dechex($filestats[1]) . "-" . dechex($lastmod) . "-" . dechex($filestats[7]) . '"';

# Check HTTP_IF_NONE_MATCH
# and report a 304 Not Modified header if they match.
if (isset ($_SERVER["HTTP_IF_NONE_MATCH"])) {
	if ($etag === stripslashes($_SERVER["HTTP_IF_NONE_MATCH"])) 
		$is304 = true;
}

if ($is304) {
	if (isset($_SERVER["SERVER_PROTOCOL"]) && $_SERVER["SERVER_PROTOCOL"] == "HTTP/1.1") 
		header("HTTP/1.1 304 Not Modified");
	else
		header("HTTP/1.0 304 Not Modified");
	header("ETag: " . $etag);
	header("Vary: Accept");
	header("Connection: close");
	exit;
}

header("Content-Type: $mime;charset=$charset");
header("Cache-Control: max-age=86400, s-maxage=86400");
header("Vary: Accept");
# If for some reason we didn't get a valid file modification time
# from the stat function, or it errored out, DO NOT send the ETag
# header as it will not be valid. Valid in this since is defined
# as modified AFTER Dec 24, 1999.
if ($lastmod > 946080000) {        # 946080000 = Dec 24, 1999 4PM
	header("ETag: " . $etag);
}

if (DOCTYPE == "strict") { ?>
<!DOCTYPE html 
     PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN"
     "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
<?php } else if (DOCTYPE == "math") { ?>
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.1 plus MathML 2.0//EN"
               "http://www.w3.org/TR/MathML2/dtd/xhtml-math11-f.dtd">
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en">
<?php } else { ?>
<!DOCTYPE html
   PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"
   "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
<?php } ?>
<head>
<meta http-equiv="Content-Type" content="<?php echo $mime ?>;charset=<?php echo $charset ?>" />

Now that you’ve reviewed the code, let’s examine it line by line and see how it works.

$charset = "utf-8";
$mime = "text/html";
$is304 = false;

Set a few default variables. The default MIME type is text/html — unless it’s been changed by this script later to application/xhtml+xml. The default character set is utf-8, but if you use some other character set, change the variable here. The last variable is just a flag to indicate if we’re going to return a 304 Not Modified header. By default we won’t.

# NOTE: To allow for q-values with one space (text/html; q=0.5), 
# use the following regex:
# "/text\/html;[\ ]{0,1}q=([0-1]{0,1}\.\d{0,4})/i"
if((isset($_SERVER["HTTP_ACCEPT"])) && (stristr($_SERVER["HTTP_ACCEPT"],"application/xhtml+xml")))  {
   if(preg_match("/application\/xhtml\+xml;q=([0-1]{0,1}\.\d{0,4})/i",$_SERVER["HTTP_ACCEPT"],$matches)) {
      $xhtml_q = $matches[1];
      if(preg_match("/text\/html;q=([0-1]{0,1}\.\d{0,4})/i",$_SERVER["HTTP_ACCEPT"],$matches)) {
         $html_q = $matches[1];
         if((float)$xhtml_q >= (float)$html_q)
            $mime = "application/xhtml+xml";
      }
   }
   else
	  $mime = "application/xhtml+xml";
}

We’ve already discussed the Accept header above and what it does, so now just determine what the Accept header contains, and if the client wants application/xhtml+xml or just text/html. The regex code is similar to other code on the web, but this regex checks the entire float decimal, not just the integer value after the decimal, and looks for up to 4 decimal points of precision. It then compares as float, not integer. The code isn’t as bad as it looks — if you strip out the regex parts you get the following algorithm:

If Accept header was sent and application/xhtml+xml exists in it
     If q-value exists for application/xhtml+xml
          get qvalue for text/html (if it exists)
          if q-application/xhtml+xml >= q-text/html
               mimetype = application/xhtml+xml
     else
          mimetype = application/xhtml+xml
else
     mimetype = text/html

# Get the file stats and compute last-modified time.
$filestats = @stat($_SERVER["SCRIPT_FILENAME"]);
$lastmod = $filestats[9] - date('Z');  #Convert Local time -> GMT

# ETag is "inode-lastmodtime-filesize" - See PHP stat function for more detail
$etag = '"' . dechex($filestats[1]) . "-" . dechex($lastmod) . "-" . dechex($filestats[7]) . '"';

Calculate the ETag the HTTP protocol returns to the client. If any of the document changes, this ETag must also. The ETag is a string, so you’re free to determine it any way you want. For this use, we use the *nix INode, Last modified time (converted to GMT), and the file size in bytes, all converted to hex, which is similar to the method Apache uses for static HTML files.

However, if this is not specific for your needs, you’ll need to find another way to generate a unique ETag. For example, sites mainly using databases may use one script for many different pages. If so, you must find a unique ETag algorithm. This ETag will be returned to us if a client has a copy of the page, but doesn’t know if it’s current or not. That’s why it must be unique.

# Check HTTP_IF_NONE_MATCH
# and report a 304 Not Modified header if they match.
if (isset ($_SERVER["HTTP_IF_NONE_MATCH"])) {
	if ($etag === stripslashes($_SERVER["HTTP_IF_NONE_MATCH"])) 
		$is304 = true;
}
if ($is304) {
	if (isset($_SERVER["SERVER_PROTOCOL"]) && $_SERVER["SERVER_PROTOCOL"] == "HTTP/1.1") 
		header("HTTP/1.1 304 Not Modified");
	else
		header("HTTP/1.0 304 Not Modified");
	header("ETag: " . $etag);
	header("Vary: Accept");
	header("Connection: close");
	exit;
}

If the client sent us back an ETag for verification, check and see if it’s the same one we just calculated. If it is, we will not need to send the document, just a 304 Not Modified header, and exit the script. No body will be (or needs to be) sent. However, you should send the Vary header so the client knows the response depends on the original Accept header.

header("Content-Type: $mime;charset=$charset");
header("Cache-Control: max-age=86400, s-maxage=86400");
header("Vary: Accept");
# If for some reason we didn't get a valid file modification time
# from the stat function, or it errored out, DO NOT send the ETag
# header as it will not be valid. Valid in this since is defined
# as modified AFTER Dec 24, 1999.
if ($lastmod > 946080000) { 				# 946080000 = Dec 24, 1999 4PM
	header("ETag: " . $etag);
}

We’ve now obtained a valid ETag, determined the client needs the full body of the page, and are ready to send it. We set a header with the correct MIME type we’ve determined earlier and a header for caching. The cache-control header allows clients and proxies to store this page for 1 day and use it as fresh, without checking with the server again.

The Vary header tells the client and any proxy the content can vary depending on what the client can Accept. Unfortunately IE has a long-standing bug with regards to the Vary header which prevents caching any content with a Vary header other than User-Agent. We’ll workaround this bug by enabling gzip compression of the page later in Apache which forces IE to work as it should. If you don’t enable compression, IE will never locally cache your page. This bug has existed from IE 4.x until 6.x. Some day Microsoft might fix it. (July 2006 — see update at end for IE7)

if (DOCTYPE == "strict") { ?>
<!DOCTYPE html 
     PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN"
     "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
<?php } else if (DOCTYPE == "math") { ?>
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.1 plus MathML 2.0//EN"
               "http://www.w3.org/TR/MathML2/dtd/xhtml-math11-f.dtd">
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en">
<?php } else { ?>
<!DOCTYPE html
   PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"
   "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
<?php } ?>
<head>
<meta http-equiv="Content-Type" content="<?php echo $mime ?>;charset=<?php echo $charset ?>" />

Now we send a correct DOCTYPE and output a META tag with the MIME type and the correct character set. Which DOCTYPE is used depends on the DOCTYPE variable in your PHP page. The remainder of your script follows as normal.

Configure Apache

Our PHP page is now setup to be cached for a period of time of our choosing, and to respond to validation requests. But we still must deal with static content. Apache will do the heavy lifting for us, as long as we tell it what we want it to do. This is simple — to configure Apache for cache-control of static content, add lines like the following in .htaccess or put them in your httpd.conf.

ExpiresActive On
ExpiresByType text/css A1209600
ExpiresByType image/png A2592000
ExpiresByType image/gif A2592000
ExpiresByType image/x-icon A5184000

php_flag zlib.output_compression On
php_value zlib.output_compression_level 1

You must have mod_expires in your Apache (most do) for this to work. Using mod_expires you can set default cache retention times for various types of static files. The numbers are the expiration in seconds since the client requested it. So A86400 means the content is valid for one day — the client can just use it from the cache without checking the server again. Nothing needs to be sent over the network for one day. Naturally, you’ll want to fine-tune these numbers for your own use.

For the compression level, just like gzip it can be a level from 1-9. Level 1 is the lowest, while 9 is the highest level of compression. However, going from 1 to 9 dramatically increases the CPU time to compress, while not yielding much size improvement. For example, on level 1, you might get 60% compression, while on level 9 it might increase to 67% or so, but at the cost of much more CPU time. Since web servers can be busy, leave it at level one to make life easy for your CPU.

It’s worth mentioning again, IE has a bug with regards to the Vary header. We MUST enable GZIP or IE will never cache our page. Other browsers (Firefox, Opera, etc) handle this correctly — it’s just IE with the difficulty. GZIP is expressed in Accept-Encoding header sent by client. If that header indicates the client can handle gzip compressed content, PHP does all the work, including sending the appropriate Vary:Accept-Encoding header. If the client indicates they can’t handle compressed content, PHP sends the file uncompressed.

Example

<?php
# to use XHTML 1.0 Strict DTD
define("DOCTYPE","strict");

# to use XHTML 1.1 MathML DTD
#define("DOCTYPE","math");

# if DOCTYPE is not defined, the transitional
# XHTML 1.0 DTD will be used.
require_once("include-mime.php");
?>
<title>Test Page</title>
</head>
<body>

<h1>Test Page</h1>

<h2>Server Variables</h2>
<pre>
<?php
foreach ($_SERVER as $key => $value)
  print "$key: $value\n";
?>
</pre>


</body>
</html>

Save this file off and try it! By changing the DOCTYPE variable, you can get strict, transitional or the new MathML DTD.

Conclusion

That’s it. You’ll need to modify the cache times for your site, but with GZIP compression, appropriate cache-control and 304 Not Modified responses, you should see bandwidth reduction, and your users will see a snappier responding web site. We’ve also worked around Microsoft’s crippled browser.

Update on If-Modified-Since (Dec 2005)

Someone noticed I don’t mention If-Modified-Since in this code. The reason is two-fold. First, it’s the older protocol, and most modern browsers should support ETags. Second, it’s not as flexible. For example, suppose you’re dealing with a blog — how do you handle last modified date? You could use the last article date (which is reasonable), but that presents you with a problem. How do you deal with different pages being served depending on whether the user is logged in or not? The article dates are the same, yet the page served is different. This situation is difficult to deal with using only dates, but with ETags it’s easy — just add some reference (like user ID) to your ETag, and the browser cache will work correctly.

If you want to modify the code on this page to handle If-Modified-Since it’s not really that hard, and is left as an exercise for the interested reader.

Update on IE7 (July 2006)

Unfortunately, testing with IE7 beta 3 reveals Microsoft still hasn’t fixed the caching bug (at least in the betas, maybe in the final release?). IE 7 won't have application/xhtml+xml support either. Or a truly fixed CSS implementation. Sigh. It appears IE7, instead of truly being a step forward (after 5 years since IE 6), should really be considered IE6 SP2.

We can only hope IE8 fixes these bugs and truly supports XHTML and CSS. After all, the standards have been around for a decade, and Microsoft certainly has the resources (if not the desire) to fully implement web standards.

License Change (September 2006)

Previously released under GPL, this code is now under a License similar to BSD. The actual license is in the code on this page.

The download file contains the GPL license. You many choose either the GPL license in the download file or the BSD-style license contained in the code on this page. It’s your option.

References

http://www.mnot.net/cache_docs/
http://lists.over.net/pipermail/mod_gzip/2002-December/006826.html
http://www.sitepoint.com/forums/printthread.php?t=158442
http://www.w3.org/TR/xhtml-media-types/
http://hixie.ch/advocacy/xhtml
http://www.xml.com/pub/a/2003/03/19/dive-into-xml.html
http://keystonewebsites.com/articles/mime_type.php
http://simon.incutio.com/archive/2003/04/23/conditionalGet
http://alexandre.alapetite.net/doc-alex/php-http-304/index.en.html

Automatic Web Page Translation with Google

yeager — Fri, 26 Dec 2008 19:37:29 +0000

Previously, we wrote about getting the browser default language in PHP. But so what? What can you do with it? In this article, we’ll use the previous code, and expand on it to create a page which can be automatically translated into many languages — with the help of the Google translator.

Method

The method is simple. Create a drop down box and fill it with the names of languages you want to translate to, then call Google’s translator with the selected language. Not really that hard at all, if you take it piece by piece.

But how good is Google’s translator? If you understand multiple languages, check it out for yourself. Naturally, it’s not as good as a manual translation, but when faced with a choice of nothing at all, Google’s machine translation isn’t really so bad.

But how can a person know you offer such a service, if they can’t read the message about selecting alternative languages? That’s where the previous detection of the browser’s default language comes in, as we’ll create a message in whatever the default language is, prompting them to choose a translation. Thus, even if they can’t read anything on your site, they at least can read a prompt (in their native language) allowing them to translate your page into their language.

We’ll tackle the job in several steps.

Create a list of drop down languages.
Create javascript event on the drop down list so when it changes, automatic translation is performed.
Tie it all up to load when the page loads, using unobtrusive javascript.

The Code

The following is BSD-Licensed Javascript and PHP code.

First, create a standard HTML option box, with one snippet of PHP code to get a prompt in whatever the users default language is.

<select id="gtrans">
    <option><?php print getLanguage();?></option>
    <option value="fr">Français (French)</option>
    <option value="de">Deutsch (German)</option>
    <option value="el">Ελληνικά (Greek)</option>
    <option value="iw">עברית (Hebrew)</option>
    <option value="it">Italiano (Italian)</option>
    <option value="ru">Русский (Russian)</option>
    <option value="es">Español (Spanish)</option>
    <option value="vi">Việt (Vietnamese)</option>
</select>

The getLanguage() code is simple, and guarantees even if a user can’t read anything on your page, they can at least read a prompt for translation. It uses the getDefaultLanguage() from our previous article on PHP browser language detection, so we won’t discuss it further here — you can refer to the previous article for more.

function getLanguage() {
   # Drop any region from the default language.
   #   i.e. en-us becomes just en.
   $lang = getDefaultLanguage();
   $x = explode("-",$lang);
   switch ($x[0]) {
      case "de": return "Wählen Sie eine Sprache";
      case "el": return "Επιλέξτε Γλώσσα";
      case "es": return "Seleccione Idioma";
      case "fr": return "Sélection de la langue";
      case "it": return "Seleziona Lingua";
      case "iw": return "בחר שפה";
      case "ru": return "Выбор языка";
      case "vi": return "Chọn ngôn ngữ";
      default: return "Select Language";
   }
}

So now we’ve got a list of languages, along with a prompt in the users own language so they can read the prompt, even if they can’t read the rest of the page. We now need to attach a javascript onchange event to the box, and create the code to call Google’s translator when the language selection changes.

<script type="text/javascript">
function init() {
   var e = document.getElementById("gtrans");
   if (e)
      e.onchange = function() {
         gtranslate(this.options[this.selectedIndex].value)
      };
}
//
function gtranslate(lang) {
   var url = "http://translate.google.com/translate?u=" +
      encodeURIComponent(location.host + location.pathname) +
      "&hl=en&ie=UTF-8&sl=en&tl=" + lang;
   //window.location opens in current window, window.open is new window
   //window.location = url;
   window.open(url);
}
//Be careful if you integrate this into other code so it won't drop the 
// other code's window.onload event.
window.onload = init;
</script>

In the window.onload event, we add an event handler to the combo box, and create the event handler to call Google’s translator.

And that’s it! Note: Your web server has to be a server available to Google for the translation to work, as Google will retrieve the page in order to translate it.

Caveats

If a user has javascript disabled, the box will appear, but won’t work. To solve this, you can use DOM methods and create the drop down entirely in javascript, thus if a user has javascript disabled, they’ll see nothing. That solution is just straight javascript, and isn’t included here as it would complicate the solution. It’s left to the interested reader.

Be sure to use UTF-8 as your character encoding. If you’re not using UTF-8 right now, convert all your documents to it — you’ll be glad you did later.

Second, if you’re sending different language-specific content at the same URL, be sure to send the appropriate Vary header. If you don’t, intermediate proxy caches might be confused and serve the wrong language to some people. To do that, just use the following first in your PHP code: header("Vary: Accept-Language"). But be warned Internet Explorer has some bugs with the Vary header you should be aware of.

For more on q-values, IE bugs, and more explanation on the regular expressions and headers in general, read our previous article Serving XHTML With the Correct MIMETYPE for discussions of similar issues. You are serving your XHTML correctly as application/xhtml+xml aren’t you?

References

Getting the Browser Default Language in PHP

yeager — Sat, 18 Oct 2008 18:41:55 +0000

If you’re doing international (i18n or Iñtërnâtiônàlizætiøn) work (or just want to make your site available in several languages), you’ll likely need to determine the users default language in your PHP code to determine which language to serve up. Searching the web yields one common code piece frequently; unfortunately as you’ll soon see it may not give you the results you need as it ignores parts of the HTTP spec which may or may not be critical to the accuracy of the results.

HTTP Language Headers

The interchange between browser and server transfers information about the client and its capabilities in headers — user agent, what it will accept, and (what we’re interested in) language. The browser sends language information in a header called HTTP_ACCEPT_LANGUAGE, which looks something like this:

 es,en-us;q=0.3,de;q=0.1

Those values state the browser accepts Spanish (es), US English (en-us), and German (de). Obviously, most browsers don’t send so many possibilities, but you get the idea. Most of the code you can find to determine default language simply searches the header for the first 2-letter language code and returns the first it finds. But looking at the example, you’ll note some additional information q=0.3 — what’s that?

HTTP Header Q-Values for HTTP_ACCEPT_LANGUAGE

As part of the HTTP spec, those are Q-Values, and must be a number between 0 and 1 (if no number appears, you can assume the value as 1). Q-Values provide not only information to what a browser supports, but what it prefers. In the previous example, es has no q-value, so it’s 1.0, while en-us is 0.3 and de is 0.1 so that means this client can handle Spanish, US English, or German — but prefers Spanish if it’s available. If it’s not, the server is free to send any of the other supported choices.

Now you see the problem — if you only search the HTTP_ACCEPT_LANGUAGE header for a match and ignore the q-value, you have no way to determine what language the client prefers — you’ll only get a match for support. Or at worse, if a q-value is 0 (meaning no support at all), you’ll get a language the client specifically tells you not to send. Why simple reg-ex solutions work becomes obvious after examining some actual HTTP_ACCEPT_LANGUAGE headers sent by popular browsers:

en-us,en;q=0.5 (Mozilla)
en-US,en;q=0.9 (Opera)
en-us (Internet Explorer)
en (Lynx)

In those cases a simple string match works, even if q-values are ignored. But if the actual HTTP_ACCEPT_LANGUAGE HTTP header contains multiple languages with differing q-values like "en,de;q=0.9" (a person whose primary language is English, but knows German) simple string searches fail spectacularly. Obviously, we must consider q-values if our results are to be correct.

Algorithm

The solution is simple. Break apart the string into it’s language components (they’re separated by commas), and then pick the one with the highest q-value to use (assume any language lacking q-values have a value of 1.0). In our example, we’ll split the string and get the following array back:

es — Spanish, assume q-value = 1.0
en-us;q=0.3 — US English, with q-value of 0.3
de;q=0.1 — German, with q-value of 0.1

Now with the languages identified,use regular expressions to extract the q-value, if it exists. Once all the q-values are assigned, select the one with the highest q-value, if it exists. If multiple languages have the same q-value, it’s safe to use any of them equally.

The Code

The following is BSD-Licensed PHP code.

#########################################################
# Copyright © 2008 Darrin Yeager                        #
# http://www.dyeager.org/                               #
# Licensed under BSD license.                           #
#   http://www.dyeager.org/downloads/license-bsd.php    #
#########################################################

function getDefaultLanguage() {
   if (isset($_SERVER["HTTP_ACCEPT_LANGUAGE"]))
      return parseDefaultLanguage($_SERVER["HTTP_ACCEPT_LANGUAGE"]);
   else
      return parseDefaultLanguage(NULL);
   }

function parseDefaultLanguage($http_accept, $deflang = "en") {
   if(isset($http_accept) && strlen($http_accept) > 1)  {
      # Split possible languages into array
      $x = explode(",",$http_accept);
      foreach ($x as $val) {
         #check for q-value and create associative array. No q-value means 1 by rule
         if(preg_match("/(.*);q=([0-1]{0,1}\.\d{0,4})/i",$val,$matches))
            $lang[$matches[1]] = (float)$matches[2];
         else
            $lang[$val] = 1.0;
      }

      #return default language (highest q-value)
      $qval = 0.0;
      foreach ($lang as $key => $value) {
         if ($value > $qval) {
            $qval = (float)$value;
            $deflang = $key;
         }
      }
   }
   return strtolower($deflang);
}

Then in your code, just call getDefaultLanguage() and you’ll get a string back with the highest q-value language sent by the browser in the HTTP_ACCEPT_LANGUAGE header.

Caveats

First, be sure to use UTF-8 as your character encoding. If you’re not using UTF-8 right now, convert all your documents to it — you’ll be glad you did later.

So what?

What’s this good for? In a future article, we’ll demonstrate how to use this method to get instant translations of your web pages into many different languages — automatically.

References

Serving XHTML With Correct MIMETYPE (More on q-values)
HTTP 1.1 Specification (Section 14.4 Accept-Language header)
Language Tags in HTML and XHTML (w3c)
RFC 4646
List of Language Tags

Optimizing PHP Code for Caching with Movable Type

yeager — Sat, 01 Mar 2008 08:25:02 +0000

One of the advantages of Movable Type over Wordpress comes from publishing static HTML files (fetching data from the database for each page request doesn't scale well). By creating static HTML files, round trips to the database are eliminated and your blog is more likely to handle sudden bursts of high traffic. How many times have you gone to a popular Wordpress blog and seen database errors like "too many database connections"?

Additionally, Apache generates ETag and Last-Modified headers for you on static HTML files, and automatically sends 304 Not Modified headers as well. If you don't know why this is important, read the definitive introduction to caching -- we'll assume you have a solid foundation in the basics and won't review that material here. Suffice it to say, caching and proper headers are important to save server CPU and bandwidth.

However, static HTML files are, well, static. Converting Movable Type blogs to PHP is simple, but if you've already published as HTML you'll have all new URL's after converting to PHP -- a situation we want to avoid. But the big problem for scripting languages like PHP, ASP or CGI is Apache won't automatically handle ETag's or 304 Not Modified headers. Every time a page is requested, the full page is sent -- even if it's not needed. That wastes bandwidth.

We want to avoid that, and it's simple -- we'll tackle it in two parts.

Configure Apache to parse HTML files with PHP

First, in your .htaccess file, add the line

AddType application/x-httpd-php .php .html

This configures Apache to treat both php AND html files as PHP. If you don't have any PHP code in your HTML files, the PHP interpreter simply passes the HTML along. However, by doing this you'll notice the ETag and Last-Modified headers are gone. And thus your ability to properly cache the page as well. That's bad, but not hard to fix.

NOTE: NEVER EVER test on your production server. Always have a test environment to play with. Testing on your production server is begging for problems. You should read and understand the code before using it.

To get proper headers back when using PHP, use the following script and place it at the top of your PHP files with a line like the following: <?php require($_SERVER['DOCUMENT_ROOT'] . "/php-caching.php"); ?> (assuming you've placed the script in your web servers root directory).

That's it. You'll now have appropriate headers and caching is enabled. Read the script and note you can control the Expires header (default expiration is 1 minute). Also note the script uses the file date of the page as the Last-Modified time. Normally this is what you want, but if it's not you'll have to change the code.

Movable Type and PHP

One more thing for Movable Type needs to be addressed. Some of Movable Type uses Perl CGI, and we don't want to inject PHP on those pages. Dynamic CGI isn't cachable anyway, so it's not a big issue, but PHP code won't work on Perl pages. Depending on how modified your Movable Type installation is (and what version), it could be as simple as placing the following in your header template:

<MTUnless name="system_template">
< ?php include($_SERVER['DOCUMENT_ROOT'] . "/php-caching.php"); ?>
</MTUnless>

(Remove the space between the < and ?php)

Of course, you'll need to experiment with your specific setup to be sure if works for you. But you'll notice the Perl CGI's are marked as system templates. Of course, YMMV.

As always, TEST TEST TEST! Every installation is different and requires testing to be sure it all works in your unique situation -- MT is highly customizable so it's not a one size fits all proposition. The script is only about 50 lines so should be easy to follow. Read the code before using it and be sure you understand what it's doing.

References

PHP XHTML and MIME type Another article on this site with introduction to caching, and includes XHTML mime type detection to properly serve XHTML as application/xhtml+xml.
PHP XHTML and MIME type Part II Builds on the previous article with improvements.
Caching Introduction The best caching information on the Internet. Period. Read it -- it's really that good.