Painless translations with F# and a bit of Magic.

February 6, 2022

A few weeks ago the unavoidable happened, a customer asked for a translated (non English) version of our SPA. We knew this would happen at some point, but as it happens always had more important work in our backlog.

There was no way to postpone this further.

Our requirements:

Translations can be outsourced/ done by non developers
Translations can be changed without releasing a new version

From a developer perspective translated apps usually are a bit of a pain. Whenever you add a string somewhere in the codebase you usually need to go somewhere else and create a translation unit (key, invariant) you can then reference in the code again.

What if we could make it possible to add translations without all that trouble?

IMHO we did a good job at making this as painless as possible. Below are the details on how this is used and how it’s implemented.

Translations

Translations are stored in a git repo as simple JSON files with plural support.

Example:

{
  "common" : {
      "name" : "Name"
      "label" : "Label"
      "label_plural" : "Labels"
  }
}

Exposing JSON files to translators is not really an option. Therefore all translaitons are managed via weblate and changes are synced back to our git repository.

Discovering new Translation Keys

Usually adding new translations is a pain, as you need to register a key and invariant string somewhere. To make this less painful we took a different approach, automagically extracting translation keys and invariant strings from the codebase. This is done by scanning the compiled assembiles for calls that look like the example below.

tr("common.label", "Label", "Labels")
tr("common.name", "Name")

Well, actually we look for IL instructions, not F# code.

IL_0000: ldstr        "common.label"
IL_0005: ldstr        "Label"
IL_000a: ldstr        "Labels"
IL_000f: newobj       instance void Codename.Shared.I18nString::.ctor(string, string, string)

IL_0015: ldstr        "common.name"
IL_001a: ldstr        "Name"
IL_001f: newobj       instance void Codename.Shared.I18nString::.ctor(string, string)

All matching calls are accumulated and compared to the keys we already registered. We also automatically do things like:

Add new translation keys and their invariant (singular + plural) strings to weblate
Mark translations as “need editing” (in weblate) if the invariant string changes for a key.
Remove unused translations in weblate (and git) – if activated.
Detect duplicate keys and notify the developer
Detect invalid translation calls

Translating a 'Thing' in the codebase

Translating something works the same across Codename.Server, Codename.Shared & Codename.Client.

(* Client *)
UI.Btn (
    style = ButtonStyle.Primary,
    label =  tr("common.save", "Save").Localized(),
    onClick = ignore
)
                        
(* Shared *)
type Priority =
    | Low
    | Medium
    | High
    | Critical with

    member this.FriendlyStringValue =
        match this with
        | Low -> tr("Priority.Low", "Low").Localized()
        | Medium -> tr("Priority.Medium", "Medium").Localized()
        | High -> tr("Priority.High", "High").Localized()
        | Critical -> tr("Priority.Critical", "Critical").Localized()

(* Server - Client Remoting API response *)
GreetingTest = requireLoggedIn (fun app _ ->
    async {
        return  tr("common.greeting", "Hello from the server!").Localized()
    }
)

Translations with placeholders

It is possible to use .NET format strings and provide replacement in the call to Localized. There is no restriction on the argument count.

 tr("common.assignedToUser", "Assigned to {0}").Localized("Peter")
 tr("common.fromTo", "from {0} to {1}").Localized("42", "64")

Singular and Plural translations

Translations get complicated fast, to make this all simpler the translation system supports plurals.

tr("common.label", "Label", "Labels").Localized() // -> "Label"
tr("common.label", "Label", "Labels").Localized(usePlural = true) // -> "Labels"

Custom Operators for Translations

To make life a bit easier there are custom operators calling .Localized() and .Localized(usePlural = true).

let (!@) (a: I18nString) : string =
    a.Localized()

let (!@@) (a: I18nString) : string =
    a.Localized(usePlural = true)

Now instead of calling Localized we can do the following:

!@ tr("common.label", "Label", "Labels") // -> "Label"
!@@ tr("common.label", "Label", "Labels") // -> "Labels"

Things to be aware of

Translation calls need to be simple so we can detect them and find their argument values.

module TK =
    let common = "common."

    // 💥 invalid
    let label = tr(common + "label", "Label", "Labels")
    
    // 💥 invalid
    let label = tr(sprintf "%slabel" common, "Label", "Labels")

    // 💥 invalid
    let label = tr($"{common}label", "Label", "Labels")
    
    // ✅ valid
    let label = tr("common.label", "Label", "Labels")

As translations should be resolved lazily (so they are not detached from the users preference).

module SomeModule =

    // 💥 don't do this. Translations will always show the invariant string. 
    // if you want to use a translation in multiple places put the `I18nString`
    // in a shared module.
    let label = !@ tr("common.label", "Label", "Labels")
    
    // ✅ share translations in their untranslated state. 
    module TK =
        let label = tr("common.label", "Label", "Labels")

    // 💥 don't do this. Translations will always show the invariant string. 
    let headers = [
        UI.Table.TableHeaderCell (TK.Actions.actionNumber.Localized())
        UI.Table.TableHeaderCell (TK.Actions.actionTitle.Localized())
        UI.Table.TableHeaderCell (TK.Common.priority.Localized())
    ]

    // 💥 don't do this. Translations are only resolved once.
    // - Client: changing the language will not update this value
    // - Server: requests with different langauge contexts get wrong values
    let headers = lazy [
        UI.Table.TableHeaderCell (TK.Actions.actionNumber.Localized())
        UI.Table.TableHeaderCell (TK.Actions.actionTitle.Localized())
        UI.Table.TableHeaderCell (TK.Common.priority.Localized())
    ]

    // ✅ translations are always resolved when needed
    let headers () = [
        UI.Table.TableHeaderCell (TK.Actions.actionNumber.Localized())
        UI.Table.TableHeaderCell (TK.Actions.actionTitle.Localized())
        UI.Table.TableHeaderCell (TK.Common.priority.Localized())
    ]

Implementation “Details”

Determining the current active language

No matter if we want to lookup a translation on the server (.NET), client (Fable) or in shared (.NET/Fable) we need to know which language to use.

Client – Fable

When a user logs into the app we fetch the preferred language from the backend and store it locally. We now include that language in all http requests to the server.

Server – .NET

On the server we use request localization. This works by extracting the language header the client provided and setting the language for the request context.

#if FABLE_COMPILER
[<RequireQualifiedAccess>]
module ClientLanguage =
    let mutable currentUserLanguage = Language.Invariant
#endif

type Language with
    static member Current: Language =
        #if !FABLE_COMPILER

        (* Get current language from current culture info on the server.
           Explaining why this works even tho tasks are not tied to a specific thread is a bit
           complicated - but here is a very good explanation (It uses AsyncLocal<T>).

           read me -> https://github.com/dotnet/runtime/issues/48077
         *)

        CultureInfo.CurrentCulture.TwoLetterISOLanguageName
        |> Language.OfString
        |> Option.defaultValue Language.Invariant

        #else
        (* Manually set by the client app to match the users preference *)

        ClientLanguage.currentUserLanguage
        #endif

Getting a Translation table

I18nString.Localize() uses TranslationProvider.Shared.Lookup internally to get a localized value. The TranslationProvider has the following public api.

type TranslationProvider =   
    member Lookup: language: Language * key: string -> TranslationLookupUnit option   
    member TranslationTable: language: Language -> TranslationLookupTable   
    
    static member Init: provider: ITranslationStore -> unit   
    static member Shared: TranslationProvide

The Init method needs to be called with a ITranslationStore before we can resolve any translation.

Depending on the context (Server/Client) a different implementation for the ITranslationStore interface needs to be provided.

type ITranslationStore =
    abstract GetTranslationTable: Language -> TranslationLookupTable