Ensuring correct character set handling

To ensure that the UP.Link server correctly transcodes data exchanged between your service and an UP.Phone, your service must do the following:

• Set the charset parameter in the HTTP header of each entity that contains text that will be displayed by the UP.Phone--that is, for each HDML deck and each alert.

• Use the METHOD=POST and POSTDATA options in all tasks that return data.

• Use the ACCEPT-CHARSET option in tasks that return data from applications that support different character sets.

The following sections describe these items in detail.

---

Setting the charset parameter in HTTP headers

The charset parameter, an extension of the HTTP Content-type header, specifies the character set used in the content of a message entity. For example, an HTTP response containing an HDML deck that uses the UTF-8 character set should appear as follows:

Content-type: text/x-hdml; charset=utf-8

<HDML VERSION=3.0>
    <DISPLAY>
        This is a utf-8 deck.
    <DISPLAY>
<HDML>

Configuring your Web server to set the charset parameter

To set the charset parameter in static HDML decks served by your Web server, you configure it as follows:

where charset_name is the name of the character set you are using (for example, utf-8). For a list of the supported character set names, see Character set names.

If you need to provide static decks that use different character sets, you might consider adding a unique filename extension to the decks that use each character set and configuring your Web server to handle the different extensions. For example, if you want to serve decks that use UTF-8 and Shift-JIS, you could add hdml-utf8 extensions to the UTF-8 decks and hdml-sjis extensions to the Shift-JIS decks, and then configure your Web server as follows:

Setting the charset parameter using C and Perl utility libraries

The UP.SDK includes sample Perl and C utility libraries that help your service create HTTP responses. The library functions allow you to specify the charset parameter, that they append to the Content-Type HTTP header. By default, these functions set the charset parameter to UTF-8.¹ If you use any other character set, you should specify it when you call the functions.

For example, suppose you want to use the utility libraries to generate a digest containing the following HDML deck:

Content-type: text/x-hdml; charset=ISO-8859-1

<HDML VERSION=3.0>
    <DISPLAY>
        This is a Latin-1 deck.
    <DISPLAY>
<HDML>

To do this, you would use C code similar to the following:

#include <stdlib.h>
#include <stdio.h>
#include <string.h>
#include "digest.h" /* from examples/source/digests */

main ()
{
  Digest d;
  char *output;

  /* Create digest object, add deck, and output it. */
  d = DigestConstruct();
  DigestAddDeck(d, "?NS=DECK1", "<HDML VERSION=3.0>"
    "<DISPLAY>This is a Latin-1 deck</DISPLAY></HDML>",
    "charset=ISO-8859-1");
  output = DigestSerialize(d);
  printf("%s",output);

  /* Clean up */
  free(output);
  DigestDestruct(d);
}

To generate the same digest using the Perl library, you would use code similar to the following:

#! /usr/local/bin/perl5.001
# Sample libraries path (sdk_dir/examples/apputils)
BEGIN { push (@INC, "../apputils"); }
require Digest;
require 'HDMLUtils.pl';

# Define deck, add to digest, and output.
$deck1 = '<HDML VERSION=3.0>'.
    '<DISPLAY>'.
    'This is a Latin-1 deck'.
    '</DISPLAY>'.
    '</HDML>';

my $digest = new Digest;
$digest->addHDMLDeck("?NS=DECK1", $deck1,
    "charset=ISO-8859-1");
&AppUtils::OutputDigest($digest->asString());

---

Using the POST method in HDML tasks

The UP.Link server does not transcode data passed in CGI query strings. For this reason, you should always use the POST method (METHOD=POST and POSTDATA options) in HDML tasks that pass data the user has entered on the UP.Phone. The METHOD and POSTDATA options are allowed in all tasks--that is, they are allowed in <ACTION>, <CE>, and <A> elements.

For example, the following entry card demonstrates the correct way to return data entered by the user:

<HDML VERSION=3.0>
    <ENTRY KEY=company>
        <ACTION TYPE=ACCEPT TASK=GO
            METHOD=POST POSTDATA="name=$(company)"
            DEST="http://foo.com/news.cgi">
            Enter company name:
    </ENTRY>
</HDML>

To specify multiple arguments for the POSTDATA option, delimit them with ampersands. For more information on the METHOD and POSTDATA options, see the <ACTION>, <CE>, and <A> sections of the HDML Language Reference.

Ensuring correct bookmarking and caching behavior

The UP.Phone uses URLs to identify both bookmarked cards and cached decks. For example, suppose the deck in the previous section were used in the interface depicted in Figure 9-1.

FIGURE  9-1.     HTTP request and response that don't allow bookmarks or unique caching

Note that the URL of the display card shown at the bottom of Figure 9-1 is simply http://foo.com/news.cgi. If a user bookmarked the card, this is the URL that would be stored with the bookmark. If the user subsequently invoked the bookmark, the UP.Phone would simply request http://foo.com/news.cgi, without the POST data. There would be no way for the service to know that the user wanted to revisit the card about ABC Company.

Likewise, the URL identifying the deck in the UP.Phone cache is simply http://foo.com/news.cgi. If the user subsequently makes another request--for example, for information about DEF Company--it would have the same URL and would replace the deck for ABC Company in the cache. In many cases, this behavior would be undesirable.

To ensure that the user can bookmark a card and return to it, you must specify the card's BOOKMARK option so that it allows your service to recreate the POST data--that is, you must "alias" the request that originally generated the card. Typically, it is easiest to create the bookmark URL by simply duplicating the POST data as CGI variables. For example, instead of returning the HDML deck shown at the bottom of Figure 9-1, the application should return HDML like the following:

<HDML VERSION=3.0>
<DISPLAY MARKABLE=TRUE
    BOOKMARK="http://foo.com/news.cgi?name=ABC">
    ABC Company reported excellent earnings last quarter.
</DISPLAY>
</HDML>

Of course, the application must include logic to generate the same result when the CGI variables are set to name=ABC as it does when the POST data is set to name=ABC. Note that, because the CGI variables are never transcoded, they might be in a character set that the UP.Phone does not understand. However, that is not an issue for users, because bookmark URLs are transparent to them.

To ensure correct caching behavior, it is recommended that you repeat the POST data in the CGI query string of the original request. Although the data in the query string does not get transcoded and is useless for most purposes, it can uniquely identify the resulting deck. For example, the deck that generates the request should be rewritten as follows:

<HDML VERSION=3.0>
    <ENTRY KEY=company>
        <ACTION TYPE=ACCEPT TASK=GO
           METHOD=POST POSTDATA="name=$(company)"
           DEST="http://foo.com/news.cgi?cache_id=$(company)">
           Enter company name:
    </ENTRY>
</HDML>

This allows the UP.Phone to cache the deck separately from other decks generated by the same CGI script. It also allows the service to distinguish the deck from other decks and selectively remove it from the cache with a cache operation. Note that the name for the query string argument (cache_id) is different from the name of the POST data argument (name). This prevents the service from confusing the request with one of the bookmark URLs described above.

---

Using the ACCEPT-CHARSET task option

When the UP.Link server transcodes "post" data for your application, it does so based on the HTTP headers you specify. For example, if the HTTP header for a deck with an entry card specifies the UTF-8 character set, the UP.Link server returns the user entry in UTF-8 characters.

If the HTTP deck header does not specify a character set, the UP.Link server uses the ACCEPT-CHARSET option for the task to determine the character set your application expects. You should use the ACCEPT-CHARSET option whenever you pass user data to another application that supports a different character set than your own. You do not need to use this option when you pass data to your own application, since your content-type header already provides this information.

For example, if the foo.bar service uses the UTF-8 character set, the HDML deck in the previous section would be more correctly written as follows:

<HDML VERSION=3.0>
    <ENTRY KEY=company>
        <ACTION TYPE=ACCEPT TASK=GO ACCEPT-CHARSET=UTF-8
           METHOD=POST POSTDATA="name=$(company)"
           DEST="http://foo.com/news.cgi?cache_id=$(company)">
           Enter company name:
    </ENTRY>
</HDML>

In this case, the UP.Link server transcodes the user-entered company value for the name argument but not for the cache_id argument since it is passed as a CGI variable rather than a POSTDATA value.

---

Testing your service with different character sets

If you run the UP.Simulator in UP.Link mode, you can simulate UP.Phones that use a wide variety of character sets and languages. If you run the UP.Simulator in HTTP Direct mode, you must make sure that the UP.Simulator is configured to use the same character set as your service. Because the UP.Simulator does not use an UP.Link server when you run it in HTTP Direct mode, it can't take advantage of the UP.Link server's transcoding capabilities. For more information about configuring the UP.Simulator for different locales and running the UP.Simulator in different modes, see the UP.SDK Tools and APIs Reference.

---

Character set names

The following table lists character set names that you can specify for the charset parameter in HTTP header and the ACCEPT-CHARSET option in HDML tasks. The UP.Link server does support a number of aliases for each of these names. However, to ensure correct character set handling, it is recommended that you use only these names.