Showing posts with label localization. Show all posts
Showing posts with label localization. Show all posts

Monday, September 3, 2018

A Country Without a Language: Constructing and using language tags

Tír gan teanga, tír gan anam (a country without a langauge is a country without a soul).
One of the tools everyone who develops web applications or writes web pages uses is Best Current Practice 47 (BCP-47) for what's commonly called a language tag (langtag) or locale code. The BCP 47 practice specification, defined by the Internet Engineering Task Force (IETF), sets out how to specify a language using internationally standard codes.

Since every engineer working with web technology should be using this specification, it would be helpful to have a solid understanding of this somewhat arcane and sometimes confusing specification. What follows is a simplified explanation that leads us to just that.

The Basics

The langtag is constructed by combining a language code from ISO639-1:2002, Codes for the representation of names of languages -- Part 1: Alpha-2 code, called a "language subtag" and optionally, more specific restrictions, also called subtags, for the script used, the region in which the localization is used, and the variant, each separated by a dash. Although there are additional subtags, these four are more than enough to denote a language used. In fact, the common practice within BCP 47 is to compose a langtag that is only verbose enough to uniquely identify the language used, and typically, langtags are composed of only the language and the region subtags. Let's see how this works using a few examples.

A Basic Langtag

Although the most basic langtag is one that consists solely of the language code, e.g., "en" for English, the most basic common langtag pattern is a language separated by regions. Since George Bernard Shaw famously said that "England and America are two countries divided by a common language", we'll use this as an example to construct basic langtags that identify that although the script used is the same, English written and spoken in the US is different than that used in England.

The "common language" is identified by its ISO language code, "en" and each of the regions, the US and England are identified by their respective ISO country codes, US and GB. Combining these two basic subtags results in two langtags, "en-US" and "en-GB", for the US and England, respectively.

Of course we could have added the script subtag to each of these langtags to identify the alphabet used, but since it's the same in GB English as it is in US English, that would not add meaning to the langtag, so it would not be included in this case. In fact, we find that the script subtag is relatively rare, however, to see how it works we'll look at a language that can use different scripts, or alphabet sets - a language such as Irish (or Gaeilge).

A Different Script

Advert in Gaeilge using the latin gaelic alphabet
There is a relatively famous advertisement for Guinness Stout that carries the slogan "Ní féidir an dubh cur ina bhán air", which is written in Irish, or Gaeilge. The Irish language is often written today using a Latin alphabet, much like English, with a liberal use of vowels that include accents like those in the word "Ní"; however, prior to the middle of the 20th century, there was no "h" in the Irish alphabet as the lenition was identified by a dot above a letter. This practice means the same phrase in the original Irish script, LATG, would be "Ní féidir an duḃ cur ina ḃán air", as shown in the advertisement pictured.

This difference would give us two langtags - "ga-latn-IE" and "ga-latg-IE". Since the formal language specification for Gaeilge now uses the LATN alphabet, and we're only verbose enough to identify the language, the "ga-latn-IE" langtag would commonly be shortened to "ga-IE".

A Variant or Two

Now let's turn our attention to the variant subtag.

Variants are seldom used in common practice - there are only 100 registered with the IANA and there are typically few who regularly use any single variant. Variants are often used to denote archaic uses and intermingled languages like the mix of English and Spanish commonly called "Spanglish".

If we turn our attention back to the langtag for the US, we might also want to include a regional variant for the Northeast or Southern US, especially given the differences in third-person word choice (where the common choice for third-person pronouns is "you" and "you" for singular and plural, respectively, in the southern vernacular the singular and plural are "y'all" and "all y'all"). Subregion variants such as this are quite common, even if they do not reach the status of a dialect. Although variants are common, they are not often registered with the IANA, which is a requirement for the variant to be used as a subtag.

One exception to the pattern of unregistered variants is Boontling, a variant of English that is tied to Boonville, California. Since a variant subtag for Boontling - BOONT - is listed in the IANA language subtag registry as a variant of (US) English, its langtag would be "en-boont" or "en-US-boont" or "en-latn-US-boont" if you wished to use the more verbose, which we don't.

It's also possible for a langtag to have multiple variants. The only example I know of for this would be a variant of English spoken in Scotland (typically referred to as Braid or Ullans) that uses the variant subtag SCOTLAND and a variant of this variant that is spoken in Ulster, Northern Ireland, which would make the langtag "en-scotland-ulster" or "en-GB-scotland-ulster".

BCP 47 Implementation

The way in which the specification has been written implementing langtags can be a little confusing. Sometimes variants are widely used enough that they become regional and sometimes variants even become recognized as their own language. One instance of this is the two primary variants of Norwegian, Norwegian Bokmål and ‎Norwegian Nynorsk. Although linguistically these are two variants or dialects, the ISO considers them languages in themselves, which means there are three valid language subtags that can be used to construct langtags for Norwegian in Norway: "no-NO", representing Norwegian in Norway; "nb-NO", representing Norwegian Bokmål in Norway; and "nn-NO", representing Norwegian Nynorsk in Norway.

Difficulties like this aside, however, one of the rules of accessibility (a11y) under the "robust" principle, requires us to include a langtag for documents using the lang attribute. The inclusion of the language allows assistive technology, like screen readers, to announce words and phrases properly and allows user agents to offer dynamic translation.

As anyone who has read authors that sprinkle phrases in multiple languages throughout their work knows, even though a root document has a language specified, there may be portions in other languages. Those portions also need to be spoken correctly and the user may benefit from dynamic translation of them as well. To help with this process, the folks writing the HTML spec made the lang attribute a global attribute, not just an attribute on the document, meaning it can be applied to any HTML element.

So, if you're concerned about the usability of your pages, include the langtag on the document (e.g., <html lang="en-US">) and anywhere else it's appropriate...and even if you're not concerned about general usability, adding the langtag to the document will help you meet the accessibility guidelines (WCAG 2.1, Guideline 3.1, Success Criteria 3.1.1 and 3.1.2) - and we all want that.

Happy coding.

Monday, April 10, 2017

How much is that doggie in the window

(How Much Is) That Doggie in the Window?
How much is that doggie in the window?
The one with the waggly tail
How much is that doggie in the window?
I do hope that doggie's for sale

Patti Page

One of the problems you're likely to face when dealing with user input of currency in an international market is the difference between number formats and the ambiguity it may introduce.

If we ignore currencies that don't allow fractional values (like the Yen), there's a big difference when trying to convert either "123,456,789.01" and "123.456.789,01" even though when a human looks at them we can tell they're the same.

For UI engineers, you'll likely use either parseInt or parseFloat - unless you give up entirely and pass the value unmodified. This poses a problem, because although there is a method to convert a number to a string that considers locale, the functions that convert strings to numbers do not.

So, what are we to do? First, let's look at what's native in JavaScript to help us find the underlying assumptions. The parseInt method is pretty self-explanatory, aside from what happens on corner cases where you're trying to convert without a radix or the like - and the parseFloat method is only slightly less self-explanatory.

The parseFloat method does not consider locale, even though the methods provided to convert a floating point to a string do. If, for example, a comma is used as a fractional separator, everything to the left of the comma will be taken as an integer. If the user - let's say one who lives in Germany - enters a number as 1.234,56 then our friendly method will convert that to 1 and 234 thousandths. If you're expecting one thousand two hundred thirty four and fifty-six hundredths, getting one and twenty-three hundredths is probably going to be disappointing. If the user - let's say one who lives in the US - enters a number as 1,234.56 then our friendly method will convert that to one (you won't even get the 234 thousandths).

By looking at the behavior of parseFloat, we learn that a dot is always interpreted as a fractional separator. We (UI engineers) just need to determine what to do when there are multiple separators and, more importantly, both types of separators (comma and dot) in a number string.

When writing numbers using separators, the rightmost separator is the fractional separator - everything else is a grouping separator - unless there are only grouping separators. How do we know if the separator is a grouping separator and not a fractional separator? We have to make an educated guess, based on the number of "groups".

Following these assumptions makes our enhanced parsing much simpler than what we might have thought at the beginning. All we need to do is split the string on the fractional separator, reconstruct the string in a format parseFloat always understands, and let parseFloat do the heavy lifting - a little task that can easily be handled by creative use of the indexOf method, giving us something like the code below.

JavaScript
function parseLocalizedFloat(numstring) {   var comma = numstring.lastIndexOf(','),       dot = numstring.lastIndexOf('.'),       sp = numstring.lastIndexOf(' '),       rightmost = Math.max(comma, dot, sp),       seps = /[\s\,\.]/g,       grps = numstring.split(seps).length,       normalized = numstring;    /* if there is a group separator and a decimal separator */    if ((comma > -1 && dot > -1) ||        (comma > -1 && sp > -1) ||        (dot > -1 && sp > -1)) {      normalized = numstring.substr(0, rightmost).replace(seps, '') +          '.' +          numstring.substr(rightmost).replace(seps, '');    /* if there are only group separators */    } else if (grps > 2) {      normalized = numstring.replace(seps, '');    /* if there is only one separator, assume it's a decimal separator */    } else if (grps === 2) {      normalized = numstring.replace(seps, '.');    }    return parseFloat(normalized); }

Note that if you cannot use lastIndexOf, you'll need to reverse the string, use indexOf and subtract that from the length (and don't forget to subtract 1 because it's a zero-based index). Lines 2-7 change to those show below, but everything beyond the variable declaration and assignment block will remain the same.
JavaScript
var reversed = numstring.split('').reverse().join(''),     comma = Math.max(reversed.length - reversed.indexOf(','), 0) - 1,     dot = Math.max(reversed.length - reversed.indexOf('.'), 0) - 1,     sp = Math.max(reversed.length - reversed.indexOf('.'), 0) - 1,     rightmost = Math.max(comma, dot, sp),     seps = /[\s\,\.]/g,     grps = numstring.split(seps).length,     normalized = numstring;


This method will accurately parse arabic numerals for the primary locale-based formats (shown below) much more reliably than parseInt or parseFloat alone. As a demo, click on any of the amounts and a JavaScript alert message will appear that will show you the string as it is in the table cell, the value you would get if you used parseFloat, and the value my internationalized version returns.


Number format examples for multiple locales
LocaleExample
Danish4 294 967 295,123
English (CA)4 294 967 295,123
English (GB)4,294,967,295.12
English (US)4,294,967,295.12
Finnish4 294 967 295,123
French4 294 967 295,123
French (CA)4 294 967 295,123
German4 294 967.295,123
Italian4.294.967.295,123
Norwegian4.294.967.295,123
Spanish4.294.967.295,123
Swedish4 294 967 295,123
Thai 4,294,967,295.12


Update

If you are coding specifically for next generation browsers, you can use the JavaScript Intl object to simplify your code, assuming you also know the user's locale (you can check browser support at http://caniuse.com/#feat=internationalization). If you can use the Intl object, your code becomes something like this...

function parseLocalizedFloat(numstring) {   var intl = new Intl.NumberFormat(userLocale),       dot = /^\d(\D)\d{1,}$/.exec(intl.format(9.99)),       pos = numstring.lastIndexOf(dot);    /* if there is a decimal separator */    if (pos > -1) {      normalized = numstring.substr(0, pos).replace(/\D/g, '') +          '.' +        numstring.substr(pos).replace(/\D/g, '');    } else {      normalized = numstring.replace(/\D/g, '');    }    return parseFloat(normalized); }


Happy coding.