Sitecore smart translation tool with SPE and Azure Cognitive Services (AI)

In my previous posts about images cropping, I've used Azure Cognitive Services (Vision) for managing media cropping in a smart way. Now, I'm sharing another usage of Azure Cognitive Services (Language) for building a Powershell tool that makes possible to translate your Sitecore content in a quick and easy way.

Handling item versioning and translation from the Sitecore content editor is a kinda tedious work for editors, especially when it comes to manually creating localized content for your site.

The idea of the PSE tool is to make the editor's life easier, so in several clicks can achieve the language version creation of the item (including subitems and datasources) and also populate the items with translated content!

Azure Translator - An AI service for real-time text translation

Translator is a cloud-based machine translation service you can use to translate text in near real-time through a simple REST API call. The service uses modern neural machine translation technology and offers statistical machine translation technology. Custom Translator is an extension of Translator, which allows you to build neural translation systems. The customized translation system can be used to translate text with Translator or Microsoft Speech Services. For more info please refer to the official documentation.

About the tool

As I mentioned before, this tool is based on SPE, so it's easy to integrate on your Sitecore instance. I'll share the full implementation details but also the code and packages. The service API layer has been implemented on .NET.

Blog post image — The context menu script

The context menu script

https://youtu.be/etbZIlaYEes
Demo

Creating the Azure service

Before proceeding with the implementation, let's see how to create the Translator service in Azure. The steps are very straightforward as usual when creating such resources.

Login to Azure portal (https://portal.azure.com/) and click on create new resource.
Search for Translator and finally click on the create button.

Azure Translator Resource

Fill the required options and choose a plan. For testing purposes there is a free plan!.
Free plan limits: 2M chars of any combination of standard translation and custom training free per month.
More details about the available plans here.

Azure Translator Options

That's it! You have your translator service created, now just take a look at the keys and endopint section, you will need it for updating in your config file:

Keys and Endopint

Service implementation (C#)

TranslatorService.cs

This is the service that communicates with the Azure API, it's quite basic and straightforward, you can also find examples and documentation in the official sites.

csharp

1using System;
2using System.Net.Http;
3using System.Text;
4using System.Threading.Tasks;
5using Newtonsoft.Json;
6using Sitecore.Cognitive.Translator.PSE.Caching;
7using Sitecore.Cognitive.Translator.PSE.Models;
8using Sitecore.Configuration;
9
10namespace Sitecore.Cognitive.Translator.PSE.Services
11{
12    public class TranslatorService : ITranslatorService
13    {
14        private readonly string _cognitiveServicesKey = Settings.GetSetting($"Sitecore.Cognitive.Translator.PSE.TranslateService.ApiKey", "");
15        private readonly string _cognitiveServicesUrl = Settings.GetSetting($"Sitecore.Cognitive.Translator.PSE.TranslateService.ApiUrl", "");
16        private readonly string _cognitiveServicesZone = Settings.GetSetting($"Sitecore.Cognitive.Translator.PSE.TranslateService.ApiZone", "");
17
18        public async Task<TranslationResult[]> GetTranslatation(string textToTranslate, string fromLang, string targetLanguage, string textType)
19        {
20            return await CacheManager.GetCachedObject(textToTranslate + fromLang + targetLanguage + textType, async () =>
21            {
22                var route = $"/translate?api-version=3.0&to={targetLanguage}&suggestedFrom=en";
23
24                if (!string.IsNullOrEmpty(fromLang))
25                {
26                    route += $"&from={fromLang}";
27                }
28
29                if (!string.IsNullOrEmpty(textType) && textType.Equals("Rich Text"))
30                {
31                    route += "&textType=html";
32                }
33
34                var requestUri = _cognitiveServicesUrl + route;
35                var translationResult = await TranslateText(requestUri, textToTranslate);
36
37                return translationResult;
38            });
39        }
40
41        async Task<TranslationResult[]> TranslateText(string requestUri, string inputText)
42        {
43            var body = new object[] { new { Text = inputText } };
44            var requestBody = JsonConvert.SerializeObject(body);
45
46            using (var client = new HttpClient())
47            using (var request = new HttpRequestMessage())
48            {
49                request.Method = HttpMethod.Post;
50                request.RequestUri = new Uri(requestUri);
51                request.Content = new StringContent(requestBody, Encoding.UTF8, "application/json");
52                request.Headers.Add("Ocp-Apim-Subscription-Key", _cognitiveServicesKey);
53                request.Headers.Add("Ocp-Apim-Subscription-Region", _cognitiveServicesZone);
54
55                var response = await client.SendAsync(request).ConfigureAwait(false);
56                var result = await response.Content.ReadAsStringAsync();
57                var deserializedOutput = JsonConvert.DeserializeObject<TranslationResult[]>(result);
58
59                return deserializedOutput;
60            }
61        }
62    }
63}

The code is simple, I'm just adding a caching layer on top to avoid repeated calls to the API.

You can check the full parameters list in the official documentation, but let me just explain the ones I used:

api-version (required): Version of the API requested by the client. Value must be 3.0.
to (required): Specifies the language of the output text. The target language must be one of the supported languages included in the translation scope.
from (optional): Specifies the language of the input text. Find which languages are available to translate from by looking up supported languages using the translation scope. If the from parameter is not specified, automatic language detection is applied to determine the source language.
textType (optional): Defines whether the text being translated is plain text or HTML text. Any HTML needs to be a well-formed, complete element. Possible values are: plain (default) or html. In this case, I'm passing the HTML when is translating from a Rich Text field.

We need also to create the models where the data is parsed into (TranslationResult), I'm not adding the code here to make it simple, but you can check the source code for full details.

TranslationExtensions.cs

using System.Linq;
using System.Threading.Tasks;
using Sitecore.Cognitive.Translator.PSE.Services;
using Microsoft.Extensions.DependencyInjection;
using Sitecore.DependencyInjection;

namespace Sitecore.Cognitive.Translator.PSE.Extensions
{
public class TranslationExtensions
{
private readonly ITranslatorService _translatorService;

public TranslationExtensions(ITranslatorService translatorServices)
{
_translatorService = translatorServices;
}

public TranslationExtensions()
{
_translatorService = ServiceLocator.ServiceProvider.GetService();
}

public async Task TranslateText(string input, string fromLang, string destLang, string textType)
{
var res = await _translatorService.GetTranslatation(input, fromLang, destLang, textType);

if (res != null && res.Any() && res[0].Translations.Any())
{
return res[0].Translations[0].Text;
}

return string.Empty;
}
}
}

Sitecore.Cognitive.Translator.PSE.config

xml

1<?xml version="1.0" encoding="utf-8" ?>
2<configuration xmlns:patch="http://www.sitecore.net/xmlconfig/">
3  <sitecore>
4    <settings>
5      <setting name="Sitecore.Cognitive.Translator.PSE.TranslateService.ApiKey" value="{YOUR_APP_KEY}" />
6      <setting name="Sitecore.Cognitive.Translator.PSE.TranslateService.ApiUrl" value="https://api.cognitive.microsofttranslator.com/" />
7      <setting name="Sitecore.Cognitive.Translator.PSE.TranslateService.ApiZone" value="{YOUR_APP_ZONE}" />
8      <setting name="Sitecore.Cognitive.Translator.PSE.TranslateService.CacheSize" value="10MB" />
9    </settings>
10    <services>
11      <configurator type="Sitecore.Cognitive.Translator.PSE.DI.RegisterContainer, Sitecore.Cognitive.Translator.PSE" />
12    </services>
13    <events>
14      <event name="publish:end:remote">
15        <handler type="Sitecore.Cognitive.Translator.PSE.Caching.CacheManager, Sitecore.Cognitive.Translator.PSE" method="ClearCache" />
16      </event>
17      <event name="customCache:rebuild:remote">
18        <handler type="Sitecore.Cognitive.Translator.PSE.Caching.CacheManager, Sitecore.Cognitive.Translator.PSE" method="ClearCache" />
19      </event>
20    </events>
21  </sitecore>
22</configuration>
23

Powershell Scripts

We need basically one main script to be added in the context menu (Add Language Version and Translate) and then few functions that has been written in this way to make it more readable and modular.

Add Language Version and Translate

Import-Function GetLanguages
Import-Function GetItems
Import-Function ConfirmationMessage
Import-Function Translate
Import-Function GetUserOptions
Import-Function GetUserFieldsToTranslate
Import-Function ConfirmationMessage

# Global variables
$location = get-location
$currentLanguage = [Sitecore.Context]::Language.Name
$langOptions = @{}
$destinationLanguages = @{}
$options = @{}

# Variables from user input - Custom Object
$userOptions = [PSCustomObject]@{
'FromLanguage' = $currentLanguage
'ToLanguages' = @()
'IncludeSubitems' = $false
'IncludeDatasources' = $false
'IfExists' = "Skip"
'FieldsToTranslate' = @()
}

# Get language options
GetLanguages $langOptions $destinationLanguages

# Ask user for options
$result = GetUserOptions $currentLanguage $langOptions $destinationLanguages $userOptions
if($result -ne "ok") {
Write-Host "Canceling"
Exit
}

# Get all items
$items = @()
$items = GetItems $location $userOptions.IncludeSubitems $userOptions.IncludeDatasources

# Ask user for fields to translate
$dialogResult = GetUserFieldsToTranslate $items $options $userOptions
if($dialogResult -ne "OK") {
Write-Host "Canceling"
Exit
}

# Ask user for confirmation
$proceed = ConfirmationMessage $items.Count $options $userOptions
if ($proceed -ne 'yes') {
Write-Host "Canceling"
Exit
}

# Call the translator service
Translate $items $userOptions

GetLanguages

1function GetLanguages {
2    [CmdletBinding()]
3    param($langOptions, $destinationOptions)
4	
5	$user = Get-User -Current
6	$languages = Get-ChildItem "master:\sitecore\system\Languages"
7    $currentLanguage = [Sitecore.Context]::Language.Name
8	
9	# Get list of languages with writting rights and remove the origin language
10    foreach ($lang in $languages) {
11        $langOptions[$lang.Name] = $lang.Name    
12        if (Test-ItemAcl -Identity $user -Path $lang.Paths.Path -AccessRight language:write) {
13            $destinationOptions[$lang.Name] = $lang.Name
14        }
15    }
16    
17    $destinationOptions.Remove($currentLanguage)
18}

GetUserOptions

function GetUserOptions {
[CmdletBinding()]
param($currentLanguage, $langOptions, $destinationLanguages, [PSCustomObject]$userOptions)

# Version overwritting options
$ifExistsOpts = @{};
$ifExistsOpts["Append"] = "Append";
$ifExistsOpts["Skip"] = "Skip";
$ifExistsOpts["Overwrite"] = "OverwriteLatest";

$result = Read-Variable -Parameters `
@{ Name = "fLang"; Value=$currentLanguage; Title="From Language"; Options=$langOptions; },
@{ Name = "tLang"; Title="Destination Languages"; Options=$destinationLanguages; Editor="checklist"; },
@{ Name = "iSubitems"; Value=$false; Title="Include Subitems"; Columns = 4;},
@{ Name = "iDatasources"; Value=$false; Title="Include Datasources"; Columns = 4 },
@{ Name = "iExist"; Value="Skip"; Title="If Language Version Exists"; Options=$ifExistsOpts; Tooltip="Append: Create new language version and translate content." `
+ "Skip: skip it if the target has a language version.Overwrite Latest: overwrite latest language version with translated content."; } `
-Description "Select a the from and target languages with options on how to perform the translation" `
-Title "Add Language and Translate" -Width 650 -Height 660 -OkButtonName "Proceed" -CancelButtonName "Cancel" -ShowHints

$userOptions.FromLanguage = $fLang
$userOptions.ToLanguages += $tLang
$userOptions.IncludeSubitems = $iSubitems
$userOptions.IncludeDatasources = $iDatasources
$userOptions.IfExists = $iExist

return $result
}

GetItems

1function GetItems {
2    [CmdletBinding()]
3    param($location, $includeSubitems, $includeDatasources)
4 
5    Import-Function GetItemDatasources
6    
7    $items = @()
8    $items += Get-Item $location
9 
10    # add subitems
11    if ($includeSubitems) {
12        $items += Get-ChildItem $location -Recurse
13    }
14     
15    # add datasources
16    if ($includeDatasources) {
17        Foreach($item in $items) {
18            $items += GetItemDatasources($item)
19        }
20    }
21     
22    # Remove any duplicates, based on ID
23    $items = $items | Sort-Object -Property 'ID' -Unique
24    
25    return $items
26}

GetFields

function GetFields {
[CmdletBinding()]
param($items, $options)

Import-Function GetTemplatesFields

Foreach($item in $items) {
$fields += GetTemplatesFields($item)
}

# Remove any duplicates, based on ID
$fields = $fields | Sort-Object -Property 'Name' -Unique

# build the hashtable to show as checklist options
ForEach ($field in $fields) {
$options.add($field.Name, $field.ID.ToString())
}

return $fields
}

GetItemDatasources

1function GetItemDatasources {
2    [CmdletBinding()]
3    param([Item]$Item)
4 
5    return Get-Rendering -Item $item -FinalLayout -Device (Get-LayoutDevice -Default) |
6        Where-Object { -not [string]::IsNullOrEmpty($_.Datasource)} |
7        ForEach-Object { Get-Item "$($item.Database):" -ID $_.Datasource }
8}

GetTemplatesFields

function GetTemplatesFields {
[CmdletBinding()]
param([Item]$Item)

$standardTemplate = Get-Item -Path "master:" -ID ([Sitecore.TemplateIDs]::StandardTemplate.ToString())
$standardTemplateTemplateItem = [Sitecore.Data.Items.TemplateItem]$standardTemplate
$standardFields = $standardTemplateTemplateItem.OwnFields + $standardTemplateTemplateItem.Fields | Select-Object -ExpandProperty key -Unique
$itemTemplateTemplateItem = Get-ItemTemplate -Item $Item
$itemTemplateFields = $itemTemplateTemplateItem.OwnFields + $itemTemplateTemplateItem.Fields
$filterFields = $itemTemplateFields | Where-Object { $standardFields -notcontains $_.Name } | Sort-Object

return $filterFields
}

GetUserFieldsToTranslate

1function GetUserFieldsToTranslate {
2    [CmdletBinding()]
3    param($items, $options, [PSCustomObject]$userOptions)
4    Import-Function GetFields
5    
6    # Get all fields from items
7    $fields = @()
8    $fields = GetFields $items $options
9    
10    # Promt the user for selecting the fields for translation
11    $dialogParams = @{
12        Title = "Fields selector"
13        Description = "Select the fields you want to translate"
14        OkButtonName = "OK"
15        CancelButtonName = "Cancel"
16        ShowHints = $true
17        Width = 600
18        Height = 800
19        Parameters = @(
20            @{
21                Name = "fieldsIdToTranslate"
22                Title = "Checklist Selector"
23                Editor = "check"
24                Options = $options
25                Tooltip = "Select one or more fields"
26            }
27        )
28    }
29    
30    $dialogResult = Read-Variable @dialogParams
31    $userOptions.FieldsToTranslate = $fieldsIdToTranslate
32    
33    return $dialogResult
34}

ConfirmationMessage

function ConfirmationMessage {
[CmdletBinding()]
param($itemsCount, $options, [PSCustomObject]$userOptions)

$fieldsToUpdate = ""
$opt = @()

ForEach($ft in $userOptions.FieldsToTranslate) {
$opt = $options.GetEnumerator() | ? { $_.Value -eq $ft }
$fieldsToUpdate += "$($opt.Key), "
}

$fieldsToUpdate = $fieldsToUpdate.Substring(0,$fieldsToUpdate.Length-2)

$message = "Updating $itemsCount item(s)!"
$message += ""
$message += "Origin Language:$($userOptions.FromLanguage)"
$message += "Destination Languages:$($userOptions.ToLanguages)"
$message += "Include Subitems:$($userOptions.IncludeSubitems)"
$message += "Include Datasources:$($userOptions.IncludeDatasources)"
$message += "Copy Method:$($userOptions.IfExists)"
$message += "Fields to Translate:$($fieldsToUpdate)"
$message += ""

return Show-Confirm -Title $message
}

Translate

1function Translate {
2    [CmdletBinding()]
3    param($items, [PSCustomObject]$userOptions)
4    
5    Write-Host "Proceeding with execution..."
6    
7    # Call the translator service
8    $translatorService = New-Object Sitecore.Cognitive.Translator.PSE.Extensions.TranslationExtensions
9    
10    $items | ForEach-Object {
11    	$currentItem = $_
12    	foreach($lang in $userOptions.ToLanguages) {
13    		Add-ItemLanguage $_ -Language $userOptions.FromLanguage -TargetLanguage $lang -IfExist $userOptions.IfExists
14    		
15    		Write-Host "Item : '$($currentItem.Name)' created in language '$lang'"
16    		
17    		Get-ItemField -Item $_ -Language $lang -ReturnType Field -Name "*" | ForEach-Object{ 
18    		    # Only look within Single-line and Rich Text fields that has been choosen in the dialog box
19                if(($_.Type -eq "Single-Line Text" -or $_.Type -eq "Rich Text" -or $_.Type -eq "Multiline Text") -and $userOptions.FieldsToTranslate.Contains($_.ID.ToString())) {
20                    if (-not ([string]::IsNullOrEmpty($_))) {
21                        # Get the item in the target created language
22                        $langItem = Get-Item -Path "master:" -ID $currentItem.ID -Language $lang
23        				
24        				# Get the translated content from the service
25        				$translated = $translatorService.TranslateText($currentItem[$_.Name], $userOptions.FromLanguage, $lang, $_.Type)
26        				
27        				# edit the item with the translated content
28        				$langItem.Editing.BeginEdit()
29        				$langItem[$_.Name] = $translated.Result
30        				$langItem.Editing.EndEdit()
31        				
32        				Write-Host "Field : '$_' translated from '$($userOptions.FromLanguage)'" $currentItem[$_.Name] " to : '$lang'" $translated.Result
33                    }
34                }
35    	    }
36    	}	 
37    }
38}

In the Translate function, I'm doing the call to the API (Sitecore.Cognitive.Translator.PSE.Extensions.TranslationExtensions).

That's very much it, now is time to test it! If everything went well, you will be able to add language versions to your items with also translated content from Azure Cognitive Translation.

Let's see this in action!

For the purpose of this demo, I've created a simple content tree with 3 levels, the items has some content in english (plain and HTML) and I'll be using the tool to create the Spanish-Argentina and French-France versions + translated content.

1- Click on the Home item and choose the Add Language Version and Translate option from the scripts section.

2- Choose the options, in this case I want to translate from the default 'en' language to both 'es-AR' and 'fr-FR'. Also I want to include the subitems, but as for this test the items doesn't have a presentation nor datasources, I'm keeping this disabled. No versions in the target language exist for those items, so I'm keeping the "Skip" option.

3- Click on proceed and choose the fields you want to translate:

I'm selecting all fields, as you can check in the SPE code, I'm removing the standard fields from the items to be translated, normally you don't want that and it will overpopulate the fields list.

4- Click OK, double check the data entered and click the OK button for making the magic to happen:

5- Click on the View script results link to check the output logs:

6- Check that the items have been created in the desired languages and the contents are already translated. Review them, publish and have a cup of coffee :).

fr-FR items version:

es-AR items version:

Voila! After few clicks you have your content items created in the language version with the content translated, I hope you like it us much as I do.

Find the source code in GitHub, download the Sitecore package

here

Download: Sitecore_Cognitive_Translator-1.0.zip

Sitecore_Cognitive_Translator-1.0.zip71.8 KB

or get the asset image from Docker Hub.

Thanks for reading!