TIP #412 Version 1.1: Dynamic Locale Changing for msgcat with On-Demand File Load

This is not necessarily the current version of this TIP.

TIP:	412
Title:	Dynamic Locale Changing for msgcat with On-Demand File Load
Version:	$Revision: 1.1 $
Author:	Harald Oehlmann <harald dot oehlmann at elmicron dot de>
State:	Draft
Type:	Project
Tcl-Version:	8.6
Vote:	Done
Created:	Tuesday, 27 March 2012
Obsoletes:	TIP #399
Keywords:	Tcl, localization, msgcat

Abstract

This TIP adds dynamic locale switching capabilities to the msgcat package.

Rationale

Dynamic Locale Switching

Within a multi-language application like a web-server, one may change the locale quite frequently, for example if users with different locales are requesting pages. Unfortunately, this does not fit well with the model adopted by the msgcat package, which assumes that all code follows this sequence:

Set locale list: mclocale locale
Load language files with other package load: mcload msg-folder
Translate strings: mc key args...

Note that if the locale should be changed after other packages are loaded, one must restart at step 2. This requires reloading all packages which is mostly not practical.

The aim of this TIP is to extend the package by dynamic locale change capabilities.

msgcat will reload any missing message catalog files of all currently loaded packages on a locale change. In addition, any package may register to get informed to a locale change. Other packages may do changes to reflect the locale change like rebuilding the GUI.

This TIP compares to TIP #399 that the package is able to load message catalog files on demand, e.g. specially on a locale change.

package locale

If the clock command gets called with the argument "-locale", the locale is changed using msgcat::mclocale. After processing, the initial value is restored. The package keeps track, which locales where already used and calls msgcat::mcload for any new locale. The locale is restored after processing.

This is an implementation of dynamic locales but conflicts with the new features described above. Other packages may be informed to change the locale and may trigger expensive operations like a rebuild of the GUI.

In consequence, each package may define a package locale which is independent of the default locale.

Specification

Package Equals Client Namespace

A client package is a package which uses msgcat. A unique namespace is required for each client package. Within msgcat, namespace and package is always connected.

Up to now, the msgcat package used this namespace as an identifier to store the catalog data of a certain package.

This is now extended to additional properties which are stored for a package.

Package locale

A package locale may be used by a package instead the default locale msgcat::mclocale. A package locale may be set or unset by a package.

Default and Package State

All State values (like the current locale) are available, once as default and once per package, if a package sets a package locale.

The used naming is:

default state: valid for all packages which do not set a package locale.
package state: only valid for one package if it has set a package locale.

The following state values are present as default state and may be set individually per package.

The locale state value is the value from mclocale.
The preferences state value is the value from mcpreferences.
The loadedlocales state value is a list of currently loaded locales. In particular, an mcload was issued for those locales and they were not cleared in-between.

Default State

A new command supports operations on the default state:

msgcat::mcloadedlocales subcommand

The default configuration may be get set by:

locale: set by the command mclocale
preferences: set by the command mcpreferences
loadedlocales: set through the action of the new command msgcat::mcloadedlocales load

The default configuration may be set by:

All those state values are set by a change of the default locale by mclocale locale. This command does the following operations:

Build and set the preferences.
Load all message catalog files for all packages without a package locale for all locales, which are in the new preferences but not in loadedlocales.
Invoke the callbacks of all packages which have registered for a locale change callback.

After a change of the default locale, the user may decide to clear all now unneeded loaded locales. This is possible using the new command:

msgcat::mcloadedlocales clear

The list of currently loaded locales is set to mcpreferences and all message catalog keys of packages without a package locale set and with locales not in mcpreferences are unset.

Package Configuration

The package configuration of the calling package may be changed using the following new command:

msgcat::mcpackagelocale subcommand ?locale?

The parameter locale may only be specified with the subcommand set.

Available subcommands are:

Subcommand "set"

Set or change the package locale.

The global state values are copied, if there were no package locale set before.

The package locale is changed to the optional given new package locale.

Subcommand "get"

Return the package locale or the default locale, if no package locale set.

Subcommand "preferences"

Return the package preferences or the default preferences, if no package locale set.

Subcommand "loaded"

The list of locales loaded for this package is returned.

Subcommand "isset"

Returns true, if a package locale is set.

Subcommand "unset"

Unset the package locale and use the default state for the package. Load all message catalog files of the package for locales, which were not present in the package loadedlocales list and are present in the default list.

Subcommand "clear"

Set the current loaded locales list of the package to mcpreferences and unset all message catalog keys of the package with locales not included in the package preferences.

It is an error to call this subcommand without a package locale set.

Package Configuration Options

Each package may have a set of configuration options set to invoke certain actions. They may be retrieved or changed with the following new command:

msgcat::mcpackageconfig subcommand option ?value?

Available subcommands are:

get: Get the current value of the option or an error if not set.
isset: Returns true if option is set.
set: Set the given value to the option. May have additional consequences and return values as described in the option section.
unset: Unset the option.

Available options are:

Package Option "mcfolder"

This is the message folder of the package. This option is set by mcload and by the subcommand set. Both are identical and both return the number of loaded message catalog files.

Setting or changing this value will load all locales contained in the preferences valid for the package. This implies also to invoke any set loadcmd (see below).

Unsetting this value will disable message file load for the package.

If the locale valid for this package changes, this value is used to eventually load message catalog files.

Message catalog files are always sourced in the namespace of the package registering the value.

Package Option "loadcmd"

This callback is invoked before a set of message catalog files are loaded for the package which has this property set.

This callback may be used to do any preparation work for message file load or to get the message data from another source like a data base. In this case, no message files are used (mcfolder is unset).

See chapter callback invocation below. The parameter list appended to this callback is the list of locales to load.

If this callback is changed, it is called with the preferences valid for the package.

Package Option "changecmd"

This callback is invoked when a default local change was performed. Its purpose is to allow a package to update any dependency on the default locale like showing the GUI in another language.

Tk may be extended to register to this callback and to invoke a virtual event.

See the callback invocation section below. The parameter list appended to this callback is mcpreferences. All registered packages are invoked in no particular order.

Package Option "unknowncmd"

Use a package locale mcunknown procedure instead of the standard version supplied by the msgcat package (msgcat::mcunknown).

The called procedure must return the formatted message which will finally be returned by msgcat::mc.

A generic unknown handler is used if set to the empty string. This consists in returning the key if no arguments are given. With given arguments, format is used to process the arguments.

See chapter callback invocation below. The appended arguments are identical to mcunknown.

Callback Invocation

Callbacks are invoked under the following conditions:

the callback command is set,
the command is not the empty string,
the registration namespace exists.

Any error within the callback stops the operation which invoked the callback. This might be surprising, as the error might be in another package.

Test if Message Key is Set

Message catalog keys may be expensive to calculate and thus may be set on demand.

The following new procedure returns false, if mc would call mcunknown for a key:

msgcat::mcexists src

There are two options, to limit the key search to just the current namespace (don't search in parent namespaces) and just the current locale (don't search the preferences but the first item):

msgcat::mcexists ?-exactnamespace? ?-exactlocale? src

forget package

A package may clear all its keys and state using the new command:

msgcat::mcforgetpackage

Locale and Preferences Format

Locales set by mcset may eventually not correspond to the current preferences, as the preferences are treated as follows:

put to lower case,
remove any multiple "_" and any "_" at the beginning or at the end of the locale.

It is proposed, that:

the locale and the first preferences element is always identical to the lowercase passed locale,
any multiple "_" are seen as one separator.

Example: preferences of locale "sy__cyrl_win"

* current preferences: "sy_cyrl_win sy_cyrl sy"
* proposed preferences: "sy__cyrl_win sy__cyrl sy".

Alternatively, all locales may normalized using the upper algorithm, which felt heavy in computation with little gain.

Example Usage

Example from TIP #399

Imagine an application which supports the current user language and French, German and English. An external package tp is used. The package uses msgcat and installs itself during the package require tp call:

package require msgcat
msgcat::mcload [file join [file dirname [info script]] msgs]

An implementation of the application with the current msgcat 1.5.0 would require the following initialization sequence:

package require msgcat
package require np

and the following code to change the locale to French:

package forget np
msgcat::mclocale fr
package require np

Using the extension of this TIP, one may load as usual:

package require msgcat
package require np

and to change to french locale:

msgcat::mclocale fr

The first time, a locale is required, all corresponding message files of all packages which use msgcat get loaded. This might be a heavy operation.

If a locale is reactivated (and the message catalog data was not cleared), it is a quick operation.

Without this TIP, it is computational expensive (if possible, as many packages are not reloadable or a reload may disturb current processing, e.g., by forcing the closing of sockets, etc.).

Change with No Need to Come Back

If it is certain that a locale is changed and the then obsolete data is of no use, one may clear unused message catalog items:

msgcat::mclocale fr
msgcat::mcloadedlocale clear

Use a Callback to be Notified About a Locale Change

Packages which display a GUI may update their widgets when the locale changes. To register to a callback, use:

namespace eval gui {
    msgcat::mcpackageconfig changecmd updateGUI

    proc updateGui args {
        puts "New locale is '[lindex $args 0]'."
    }
}

% msgcat::mclocale fr
fr
% New locale is 'fr'.

To Use Another Locale Source than Message Catalog Files

If locales (or additional locales) are contained in another source like a data base, a package may use the load callback and not mcload:

namespace eval db {
    msgcat::mcpackageconfig loadcmd loadMessages
    msgcat::mcconfig loadedpackages\
            [concat [msgcat::mcconfig loadedpackages] [namespace current]]

    proc loadMessages args {
        foreach locale $args {
            if {[LocaleInDB $locale]} {
                 msgcat::mcmset $locale [GetLocaleList $locale]
            }
        }
    }
}

Use a package locale

The reference implementation also contains a changed clock command which uses a package locale. Here are some sketches from the implementation.

First, a package locale is initialized and the generic unknown function is activated:

msgcat::mcpackagelocale set
msgcat::mcpackageconfig unknowncmd ""

If the user requires the week day in a certain locale, it is changed:

clock format [clock seconds] -format %A -locale fr

and the code:

msgcat::mcpackagelocale set $locale
return [lindex [msgcat::mc DAYS_OF_WEEK_FULL] $day]
### Returns "mercredi"

Some message-catalog items are heavy in computation and thus are dynamically cached using:

proc ::tcl::clock::LocalizeFormat { locale format } {
    set key FORMAT_$format
    if { [::msgcat::mcexists -exactlocale -exactnamespace $key] } {
	return [mc $key]
    }
    #...expensive computation of format clipped...
    mcset $locale $key $format
    return $format
}

Reference Implementation

See Tcl fossil tag msgcat_dyn_locale [1].

Compatibility

Imagined incompatibilities:

If packages call mcload multiple times with different folders, the data was currently appended. This is still the case, but only the last folder is used for any reload. The property mcfolder may be transformed to a list to cover this case.
The return value of mcload (file count) may be much higher as there may be loaded much more files. I suppose, this value is only used by the test suite to verify functionality and is not for big general use.
Message files may not be aware, that they may be loaded at any moment and not only after their own mcload. I suppose, this is the biggest issue but I think, there is no alternative.
Message files do not get reloaded any more, if a second mcload is issued with the same path argument.
Package which temporary change the default locale trigger any callback and may lead to user visible side effects.

Issues

Known issues:

Packages might not be aware of a locale change and may buffer translations outside of msgcat. Packages should not buffer msgcat messages if they are used in a dynamic locale application (like tklib tooltip does for example).
The clock command currently has a small dynamic patch for msgcat implemented. This must be removed in favor to new msgcat features due to the temporarily change of the default locale.

Extensions

Expose the function to calculate the preference list from a given locale.
Load a message catalog file for a given locale without changing the default/package locale.
Methods isloaded to check if a locale is currently loaded.
Access message catalog with specified namespace, locale and search behavior.

Alternatives

The alternative is the former TIP #399, but that is problematic because the list of locales must be known before any package load. The additional complexity of this TIP is a justifiable trade-off against the greatly improved flexibility in the loading and locale selection order.

Copyright

This document has been placed in the public domain.

This is not necessarily the current version of this TIP.

TIP AutoGenerator - written by Donal K. Fellows