'hold my beer' with Sitecore

At the recent London Sitecore User Group meetup Ian Jepp from Lake Solutions gave a talk on AI Images. He outlined a common problem faced these days where it's a compromise between managing a set of tailored versions of the same image differently scaled for multiple devices and page occurrences, and usage of a single HQ image scaled to desired size. While the first comes with a huge resourcing cost to create and manage such collections, the latter severely impacts the page load time harming both UX and SEO results with Google.

The presented solution was to get images cropped on the fly to thumbnails. That's not a big deal when cutting down to a fixed area of an image, e.g. centre of it, however, would that be sufficient from the UX perspective?

Let's have a look at example with this original image:

It's already been scaled down for presentation purpose while a HQ version of it would be several MBs, but still serves the purpose well. Now, let's say we'd like to like to generate a 200x200px thumbnail selecting the central area of the image:

Unfortunately, that doesn't look satisfactory as we've cut out a bit of our 'area of interest' which is the car itself. We could have manually selected the 200x200px frame around the car, however, performing this action in bulk, e.g. adjusting thousands of images in the Media Library to produce thumbnails - that sounds like a lot of tedious, manual work and not an acceptable option without a team of editors.

Here come the Microsoft Cognitive Services with Computer Vision Service offering its advanced features for processing images, including 'smart' image cropping what is exactly what we need. Let's see how the service would handle this task for us using AI to identify the crucial area and cutting the image down to desired size:

That's a spot on result.

As the presentation didn't come with open-source code to try it out, I've decided to spend some time on exploring this topic further.

To get Cognitive Services assistance with this task, we need some code sending an image to Computer Vision Service together with the desired width and height of the result image. Microsoft offers a code sample for a quick start. As the service allows some flexibility on how to provide it with the image source (API Reference), I'll present 2 overloads - one accepting a serialized image, while the other providing URL of one already available on the web (that includes your CDN of course):

namespace Sitecore91.Foundation.CognitiveServices.ComputerVision
{
    public interface IComputerVisionService
    {
        byte[] GetThumbnail(byte[] image, int width, int height, bool isSmartCrop = true);
        byte[] GetThumbnail(string imageUrl, int width, int height, bool isSmartCrop = true);
    }
}

Now, let's get a simple implementation:

using System;
using System.Net.Http;
using System.Net.Http.Headers;
using Sitecore.Diagnostics;

namespace Sitecore91.Foundation.CognitiveServices.ComputerVision
{
    public class ComputerVisionService : IComputerVisionService
    {
        private readonly string _subscriptionKey = "<-SUBSCRIPTION KEY->";
        private readonly string _serviceUrl = "https://<-AZURE REGION->.api.cognitive.microsoft.com/vision/v2.0/";

        public byte[] GetThumbnail(byte[] image, int width, int height, bool isSmartCrop = true)
        {
            var apiMethod = "generateThumbnail";
            var requestUri = _serviceUrl + apiMethod + $"?width={width}&height={height}&smartCropping={isSmartCrop}";

            try
            {
                using (var client = new HttpClient())
                {
                    client.DefaultRequestHeaders.Add("Ocp-Apim-Subscription-Key", _subscriptionKey);
                    
                    using (var content = new ByteArrayContent(image))
                    {
                        content.Headers.ContentType = new MediaTypeHeaderValue("application/octet-stream");
                        using (var response =  client.PostAsync(requestUri, content).GetAwaiter().GetResult())
                        {
                            if (response.IsSuccessStatusCode)
                            {
                                return response.Content.ReadAsByteArrayAsync().GetAwaiter().GetResult();
                            }
                            var errorMessage = response.Content.ReadAsStringAsync().GetAwaiter().GetResult();
                            Log.Error(errorMessage, this);
                            return null;
                        }
                    }
                }
            }
            catch (Exception e)
            {
                Log.Error(e.Message, this);
                return null;
            }
        }

        public byte[] GetThumbnail(string imageUrl, int width, int height, bool isSmartCrop = true)
        {
            var apiMethod = "generateThumbnail";
            var requestUri = _serviceUrl + apiMethod + $"?width={width}&height={height}&smartCropping={isSmartCrop}";

            try
            {
                using (var client = new HttpClient())
                {
                    client.DefaultRequestHeaders.Add("Ocp-Apim-Subscription-Key", _subscriptionKey);

                    using (var content = new StringContent($"{{url:\"{imageUrl}\"}}"))
                    {
                        content.Headers.ContentType = new MediaTypeHeaderValue("application/json");
                        using (var response = client.PostAsync(requestUri, content).GetAwaiter().GetResult())
                        {
                            if (response.IsSuccessStatusCode)
                            {
                                return response.Content.ReadAsByteArrayAsync().GetAwaiter().GetResult();
                            }
                            var errorMessage = response.Content.ReadAsStringAsync().GetAwaiter().GetResult();
                            Log.Error(errorMessage, this);
                            return null;
                        }
                    }
                }
            }
            catch (Exception e)
            {
                Log.Error(e.Message, this);
                return null;
            }
        }
    }
}

Code looks quite straightforward. All you need to do is replacing the 'SUBSCRIPTION KEY' and 'AZURE REGION' variables which you get upon registering for Azure Cognitive Services in a certain region. All the necessary information how to do that is available in the links provided above. What's more, Computer Vision Service offers a free tier sufficient for development and testing purposes, so nothing stops you from giving it a try with your solution.

One of the field types for display with Rendering Variants is Template. It allows you to define your own NVelocity template to embed custom processing of the output. The really cool thing about it that it can be made flexible even further and extended by adding custom renderers.

The templates mentioned above are processed by getVelocityTemplateRenderers SXA pipeline outlined here. That gives us a trace that this could be achieved with extending the pipeline by adding a custom AddTemplateRenderers processor. Taking a closer look with ILSpy at Sitecore.XA.Foundation.Variants.Abstractions.dll reveals that we can do that by implementing the IGetTemplateRenderersPipelineProcessor.

SXA docs for Creating a Rendering Variant available here describe a predefined set of objects and tools provided with SXA to use in the template:

• $item: access to the current item ($item.Name displays current item name).
• $dateTool.Format(date, format): formats date and time values.
• $numberTool.Format(number, format): formats numbers.
• $geospatial: in case of geospatial search ($geospatial.Distance) will show distance to certain point of interest).

What's really cool, you can easily extend this list and implement a custom renderer with business logic of your choice. Some good usage examples could be your company's real-time share price or current weather in your location. However, as Sitecore community is well known for it's outstanding sense of humour we need to keep up - let's create a tool rendering a random joke for display.

First, to locate where to hook our implementation, let's have a look into the getVelocityTemplateRenderers pipeline:

<getVelocityTemplateRenderers patch:source="Sitecore.XA.Foundation.Variants.Abstractions.config">
  <processor type="Sitecore.XA.Foundation.Variants.Abstractions.Pipelines.GetVelocityTemplateRenderers.InitializeVelocityContext, Sitecore.XA.Foundation.Variants.Abstractions" resolve="true"/>
  <processor type="Sitecore.XA.Foundation.Variants.Abstractions.Pipelines.GetVelocityTemplateRenderers.AddTemplateRenderers, Sitecore.XA.Foundation.Variants.Abstractions" resolve="true"/>
</getVelocityTemplateRenderers>

Based on the decompiled implementation of AddTemplateRenderers processor, we just need to inject another entry into the pipeline context. This should do the trick:

using Sitecore.XA.Foundation.Variants.Abstractions.Pipelines.GetVelocityTemplateRenderers;

namespace Sitecore91.Foundation.SXAExtensions.Pipelines.NVelocityTemplateRenderers
{
    public class AddCustomTemplateRenderers : IGetTemplateRenderersPipelineProcessor
    {
        public void Process(GetTemplateRenderersPipelineArgs args)
        {
            args.Context.Put("jokeTool", new JokeTool());
        }
    }
}

Our implementation of JokeTool consumes the Geek Jokes API to provide us with a random joke on each pipeline execution (at least as long as it's not cached):

using System;
using System.Net.Http;

namespace Sitecore91.Foundation.SXAExtensions.Pipelines.NVelocityTemplateRenderers
{
    public class JokeTool
    {
        public string GetRandomJoke()
        {
            var jokeApiUrl = "https://geek-jokes.sameerkumar.website/api";
            try
            {
                using (var httpClient = new HttpClient())
                {
                    return httpClient.GetStringAsync(jokeApiUrl).GetAwaiter().GetResult();
                }
            }
            catch (Exception e)
            {
                return "No jokes today.";
            }
        }
    }
}

Having code in place, let's wire it up with an appropriate patch in the end of pipeline:

<?xml version="1.0" encoding="utf-8" ?>
<configuration xmlns:patch="http://www.sitecore.net/xmlconfig/">
  <sitecore>
    <pipelines>
      <getVelocityTemplateRenderers>
        <processor type="Sitecore91.Foundation.SXAExtensions.Pipelines.NVelocityTemplateRenderers.AddCustomTemplateRenderers, Sitecore91.Foundation.SXAExtensions" 
                   patch:after="processor[@type='Sitecore.XA.Foundation.Variants.Abstractions.Pipelines.GetVelocityTemplateRenderers.AddTemplateRenderers, Sitecore.XA.Foundation.Variants.Abstractions']" />
      </getVelocityTemplateRenderers>
    </pipelines>
  </sitecore>
</configuration>

What should result in amending the pipeline as below:

<getVelocityTemplateRenderers patch:source="Sitecore.XA.Foundation.Variants.Abstractions.config">
  <processor type="Sitecore.XA.Foundation.Variants.Abstractions.Pipelines.GetVelocityTemplateRenderers.InitializeVelocityContext, Sitecore.XA.Foundation.Variants.Abstractions" resolve="true"/>
  <processor type="Sitecore.XA.Foundation.Variants.Abstractions.Pipelines.GetVelocityTemplateRenderers.AddTemplateRenderers, Sitecore.XA.Foundation.Variants.Abstractions" resolve="true"/>
  <processor type="Sitecore91.Foundation.SXAExtensions.Pipelines.NVelocityTemplateRenderers.AddCustomTemplateRenderers, Sitecore91.Foundation.SXAExtensions" patch:source="AddCustomTemplateRenderers.config"/>
</getVelocityTemplateRenderers>

To see in action our newly created tool, we use it with the Template field just like the other already predefined ones (e.g. $dateTool):

Now you can enjoy your day with a fresh portion of wisdom on each page visit:

That was literally fun.

Rendering Variants offer convenient way of customising how SXA components get rendered. One of the most frequently used fields for building rendering variants is Query, which allows you to utilise item(s) out of the component's scope.

What SXA offers out of the box is a decent collection of predefined, routinely used tokens which help to simplify the queries and makes their usage much more convenient. They are listed here in the SXA docs for resolveTokens pipeline:

• $compatibleThemes - path to all themes
• $theme - currently used theme
• $pageDesigns - root of page designs
• $partialDesigns - root of partial designs
• $currenttemplate - name of the current template
• $tenant - path to the current tenant
• $site - path to the current site
• $home - path to the current site start item (Home)
• $templates - path to the current site templates
• $siteMedia - path to Virtual Media folder located under site

As you can see, it contains the most handy tokens we might need for building queries, such as home item ($home) or component template ($currentTemplate). Of course every application, depending on its business specific needs, 'develops' along the time it's own crucial nodes which become frequently referenced and used in the queries. Very good example is the root item for articles, which occurs in every web site offering news to the user.

In the following use case we will create $articles token referring to articles, which will be translated by resolveTokens pipeline as a query returning all items implementing template Article under the Articles root node.

Technically, what we need to do is extending the resolveTokens pipeline:

<resolveTokens patch:source="Sitecore.XA.Foundation.Multisite.config">
  <processor type="Sitecore.XA.Foundation.Multisite.Pipelines.ResolveTokens.ResolveMultisiteTokens, Sitecore.XA.Foundation.Multisite" resolve="true"/>
  <processor type="Sitecore.XA.Foundation.Presentation.Pipelines.ResolveTokens.ResolvePresentationTokens, Sitecore.XA.Foundation.Presentation" resolve="true" patch:source="Sitecore.XA.Foundation.Presentation.config"/>
  <processor type="Sitecore.XA.Foundation.Theming.Pipelines.ResolveTokens.ResolveThemingTokens, Sitecore.XA.Foundation.Theming" resolve="true" patch:source="Sitecore.XA.Foundation.Theming.config"/>
  <processor type="Sitecore.XA.Foundation.TokenResolution.Pipelines.ResolveTokens.CurrentTemplateToken, Sitecore.XA.Foundation.TokenResolution" resolve="true" patch:source="Sitecore.XA.Foundation.TokenResolution.config"/>
  <processor type="Sitecore.XA.Foundation.TokenResolution.Pipelines.ResolveTokens.EscapeQueryTokens, Sitecore.XA.Foundation.TokenResolution" resolve="true" patch:source="Sitecore.XA.Foundation.TokenResolution.config"/>
</resolveTokens>

by adding a new processor handling the custom token.

To figure out what exactly has to be implemented and how are the currently existing processors handling the predefined tokens, we can use ILSpy, dotPeek or any other .NET decompiler to take a closer look at Sitecore.XA.Foundation.TokenResolution.dll. It turns out that to extend the resolveTokens pipeline to handle custom token resolution, we need to implement the Pipelines.ResolveTokens.ResolveTokensProcessor abstract class.

To fulfill the use case mentioned before, I implemented it this way:

using Sitecore.Data.Items;
using Sitecore.XA.Foundation.TokenResolution.Pipelines.ResolveTokens;

namespace Sitecore91.Foundation.SXAExtensions.Pipelines.Tokens
{
    public class ResolveCustomTokens : ResolveTokensProcessor
    {
        public override void Process(ResolveTokensArgs args)
        {
            args.Query = ReplaceTokenWithValue(args.Query, "$articles", () => GetArticles(args.ContextItem));
        }

        private string GetArticles(Item contextItem)
        {
            var articlesRootPath = contextItem.Database.GetItem(Constants.Items.ArticlesHome).Paths.FullPath;
            var templateId = Constants.Templates.Article.ToString();

            return articlesRootPath + "//*[@@templateid='" + templateId + "']";
        }
    }
}

Just to clarify, Constants static class used in the processor is just a standard helper containing IDs of frequently used items:

using Sitecore.Data;

namespace Sitecore91.Foundation.SXAExtensions
{
    public struct Constants
    {
        public struct Items
        {
            public static ID ArticlesHome { get { return new ID("{4109F746-326F-4EF9-9BEB-D14261E8BD83}"); } }
        }

        public struct Templates
        {
            public static ID Article { get { return new ID("{3A98D12C-1911-49A6-8F8E-AAEDBCC9C3FD}"); } }
        }
    }
}

As the implementation code is in place we're almost there, what’s left is just wiring it up in the pipeline. We locate the processor right before the last one in the resolveTokens pipeline:

<?xml version="1.0" encoding="utf-8" ?>
<configuration xmlns:patch="http://www.sitecore.net/xmlconfig/">
  <sitecore>
    <pipelines>
      <resolveTokens>
        <processor type="Sitecore91.Foundation.SXAExtensions.Pipelines.Tokens.ResolveCustomTokens, Sitecore91.Foundation.SXAExtensions" 
                   patch:before="processor[@type='Sitecore.XA.Foundation.TokenResolution.Pipelines.ResolveTokens.EscapeQueryTokens, Sitecore.XA.Foundation.TokenResolution']" />
      </resolveTokens>
    </pipelines>
  </sitecore>
</configuration>

To double check the result, pipeline after the amendments should look like this:

<resolveTokens patch:source="Sitecore.XA.Foundation.Multisite.config">
  <processor type="Sitecore.XA.Foundation.Multisite.Pipelines.ResolveTokens.ResolveMultisiteTokens, Sitecore.XA.Foundation.Multisite" resolve="true"/>
  <processor type="Sitecore.XA.Foundation.Presentation.Pipelines.ResolveTokens.ResolvePresentationTokens, Sitecore.XA.Foundation.Presentation" resolve="true" patch:source="Sitecore.XA.Foundation.Presentation.config"/>
  <processor type="Sitecore.XA.Foundation.Theming.Pipelines.ResolveTokens.ResolveThemingTokens, Sitecore.XA.Foundation.Theming" resolve="true" patch:source="Sitecore.XA.Foundation.Theming.config"/>
  <processor type="Sitecore.XA.Foundation.TokenResolution.Pipelines.ResolveTokens.CurrentTemplateToken, Sitecore.XA.Foundation.TokenResolution" resolve="true" patch:source="Sitecore.XA.Foundation.TokenResolution.config"/>
  <processor type="Sitecore91.Foundation.SXAExtensions.Pipelines.Tokens.ResolveCustomTokens, Sitecore91.Foundation.SXAExtensions" patch:source="ResolveCustomTokens.config"/>
  <processor type="Sitecore.XA.Foundation.TokenResolution.Pipelines.ResolveTokens.EscapeQueryTokens, Sitecore.XA.Foundation.TokenResolution" resolve="true" patch:source="Sitecore.XA.Foundation.TokenResolution.config"/>
</resolveTokens>

OK, backend to support the $articles token is done, so let's start making use of it.

In Content Editor I created the Landing Page rendering variant to display few page values: Title, Published date and a Query referring to the collection of articles:

Of course, as articles are usually quite complex items, let's tailor their display to absolute minimum showing only their names for this example purpose. To achieve that, I added a child field of type Template to the Articles query and called it Name.

Template field thanks to the support of NVelocity templates refers to its context (article) item by usage of $item token. This means that for each returned article we select and display only it's name:

That's it.

Now, after making sure I applied the Landing Page rendering variant to the desired component, I am presented with the fields corresponding to the content of the item: Title, Published date, together with all (4, it's a test) of the articles names:

Hope this helps.