Sitecore smart image cropping, tags and alt text with AI: Azure Computer Vision - Part III. | Miguel Minoldo | Blog Posts | Miguel Minoldo

In my previous post I've shared the custom image field implementation that makes use of the Azure Computer Vision service in order to crop and generate the thumbnails using AI. Please before proceed with this reading, make sure you already went through the previous posts: Part I and Part II.

Now, I'll be sharing the last, but not least part of this topic, how to make it working in the front-end side, the media request flow and so on.

Image request flow

Blog post image — The image request flow

The image request flow

So, the request flow is described in the following graph, basically follows the normal Sitecore flow but with the introduction of the Azure Computer Vision and Image Sharp to generate the proper cropping version of the image.

AICroppingProcessor

This custom processor will be overriding the Sitecore OOTB ThumbnailProcessor. It's basically a copy from the original code with a customization to check the "SmartCropping" parameter from the image request.

csharp

1using Sitecore.Diagnostics;
2using Sitecore.Resources.Media;
3using System;
4using Microsoft.Extensions.DependencyInjection;
5using System.IO;
6using System.Linq;
7using Sitecore.Computer.Vision.CroppingImageField.Services;
8using Sitecore.DependencyInjection;
9
10namespace Sitecore.Computer.Vision.CroppingImageField.Processors
11{
12    public class AICroppingProcessor
13    {
14        private static readonly string[] AllowedExtensions = { "bmp", "jpeg", "jpg", "png", "gif" };
15
16        private readonly ICroppingService _croppingService;
17
18        public AICroppingProcessor(ICroppingService croppingService)
19        {
20            _croppingService = croppingService;
21        }
22
23        public AICroppingProcessor()
24        {
25            _croppingService = ServiceLocator.ServiceProvider.GetService<ICroppingService>();
26        }
27
28        public void Process(GetMediaStreamPipelineArgs args)
29        {
30            Assert.ArgumentNotNull(args, "args");
31
32            var outputStream = args.OutputStream;
33
34            if (outputStream == null)
35            {
36                return;
37            }
38
39            if (!AllowedExtensions.Any(i => i.Equals(args.MediaData.Extension, StringComparison.InvariantCultureIgnoreCase)))
40            {
41                return;
42            }
43
44            var smartCrop = args.Options.CustomOptions[Constants.QueryStringKeys.SmartCropping];
45
46            if (!string.IsNullOrEmpty(smartCrop) && bool.Parse(smartCrop))
47            {
48                Stream outputStrm;
49
50                outputStrm = Stream.Synchronized(_croppingService.GetCroppedImage(args.Options.Width, args.Options.Height, outputStream.MediaItem));
51                args.OutputStream = new MediaStream(outputStrm, args.MediaData.Extension, outputStream.MediaItem);
52            }
53            else if (args.Options.Thumbnail)
54            {
55                var transformationOptions = args.Options.GetTransformationOptions();
56                var thumbnailStream = args.MediaData.GetThumbnailStream(transformationOptions);
57
58                if (thumbnailStream != null)
59                {
60                    args.OutputStream = thumbnailStream;
61                }
62            }
63        }
64    }
65}

We need also to customize the MediaRequest to also take the "SmartCropping" parameter into account:

using Sitecore.Configuration;
using Sitecore.Diagnostics;
using Sitecore.Resources.Media;

using System.Web;

namespace Sitecore.Computer.Vision.CroppingImageField.Requests
{
using System.Collections.Specialized;

public class AICroppingMediaRequest : MediaRequest
{
private HttpRequest _innerRequest;
private MediaUrlOptions _mediaQueryString;
private MediaUri _mediaUri;
private MediaOptions _options;

protected override MediaOptions GetOptions()
{
var queryString = this.InnerRequest.QueryString;

if (queryString == null || queryString.Count == 0)
{
_options = new MediaOptions();
}
else
{
SetMediaOptionsFromMediaQueryString(queryString);

if (!string.IsNullOrEmpty(queryString.Get(Constants.QueryStringKeys.SmartCropping)))
{
SetCustomOptionsFromQueryString(queryString);
}
}

if (!this.IsRawUrlSafe)
{
if (Settings.Media.RequestProtection.LoggingEnabled)
{
string urlReferrer = this.GetUrlReferrer();

Log.SingleError(string.Format("MediaRequestProtection: An invalid/missing hash value was encountered. " +
"The expected hash value: {0}. Media URL: {1}, Referring URL: {2}",
HashingUtils.GetAssetUrlHash(this.InnerRequest.RawUrl), this.InnerRequest.RawUrl,
string.IsNullOrEmpty(urlReferrer) ? "(empty)" : urlReferrer), this);
}

_options = new MediaOptions();
}

return _options;
}

private void SetCustomOptionsFromQueryString(NameValueCollection queryString)
{
this.ProcessCustomParameters(_options);

if (!string.IsNullOrEmpty(queryString.Get(Constants.QueryStringKeys.SmartCropping))
&& !_options.CustomOptions.ContainsKey(Constants.QueryStringKeys.SmartCropping)
&& !string.IsNullOrEmpty(queryString.Get(Constants.QueryStringKeys.SmartCropping)))
{
_options.CustomOptions.Add(Constants.QueryStringKeys.SmartCropping, queryString.Get(Constants.QueryStringKeys.SmartCropping));
}
}

private void SetMediaOptionsFromMediaQueryString(NameValueCollection queryString)
{
MediaUrlOptions mediaQueryString = this.GetMediaQueryString();

_options = new MediaOptions()
{
AllowStretch = mediaQueryString.AllowStretch,
BackgroundColor = mediaQueryString.BackgroundColor,
IgnoreAspectRatio = mediaQueryString.IgnoreAspectRatio,
Scale = mediaQueryString.Scale,
Width = mediaQueryString.Width,
Height = mediaQueryString.Height,
MaxWidth = mediaQueryString.MaxWidth,
MaxHeight = mediaQueryString.MaxHeight,
Thumbnail = mediaQueryString.Thumbnail,
UseDefaultIcon = mediaQueryString.UseDefaultIcon
};

if (mediaQueryString.DisableMediaCache)
{
_options.UseMediaCache = false;
}

foreach (string allKey in queryString.AllKeys)
{
if (allKey != null && queryString[allKey] != null)
{
_options.CustomOptions[allKey] = queryString[allKey];
}
}
}

public override MediaRequest Clone()
{
Assert.IsTrue((base.GetType() == typeof(AICroppingMediaRequest)), "The Clone() method must be overridden to support prototyping.");

return new AICroppingMediaRequest
{
_innerRequest = this._innerRequest,
_mediaUri = this._mediaUri,
_options = this._options,
_mediaQueryString = this._mediaQueryString
};
}
}
}

This code is very straightforward, it will basically check if the "SmartCropping=true" parameter exists in the media request, and then executes the custom code to crop the image.

The "Get Thumbnails" method limitations

As we can see in the official documentation, there are some limitations on the thumbnail generator method.

Image file size must be less than 4MB.
Image dimensions should be greater than 50 x 50.
Width of the thumbnail must be between 1 and 1024.
Height of the thumbnail must be between 1 and 1024.

The most important one is that the width and height cannot exceed the 1024px, this is problematic as sometimes we need to crop on a bigger ratio.

So, in order to make it more flexible, I'm doing the cropping using the Graphics library but getting the focus point coordinates from the "Get Area Of Interest" API method:

csharp

1using Sitecore.Data.Items;
2using Microsoft.Extensions.DependencyInjection;
3using System.IO;
4using Sitecore.DependencyInjection;
5using Sitecore.Resources.Media;
6using System.Drawing;
7using System.Drawing.Imaging;
8using System.Drawing.Drawing2D;
9
10namespace Sitecore.Computer.Vision.CroppingImageField.Services
11{
12    public class CroppingService : ICroppingService
13    {
14        private readonly ICognitiveServices _cognitiveServices;
15
16        public CroppingService(ICognitiveServices cognitiveServices)
17        {
18            _cognitiveServices = cognitiveServices;
19        }
20
21        public CroppingService()
22        {
23            _cognitiveServices = ServiceLocator.ServiceProvider.GetService<ICognitiveServices>();
24        }
25
26        public Stream GetCroppedImage(int width, int height, MediaItem mediaItem)
27        {
28            using (var streamReader = new MemoryStream())
29            {
30                var mediaStrm = mediaItem.GetMediaStream();
31
32                mediaStrm.CopyTo(streamReader);
33                mediaStrm.Position = 0;
34
35                var img = Image.FromStream(mediaStrm);
36
37                // The cropping size shouldn't be higher than the original image
38                if (width > img.Width || height > img.Height)
39                {
40                    Sitecore.Diagnostics.Log.Warn($"Media file is smaller than the requested crop size. " +
41                        $"This can result on a low quality result. Please upload a proper image: " +
42                        $"Min Height:{height}, Min Width:{width}. File: {mediaItem.DisplayName}, Path{mediaItem.MediaPath}", this);
43                }
44
45                // if the cropping size exceeds the cognitive services limits, get the focus point and crop 
46                if (width > 1025 || height > 1024)
47                {
48
49                    var area = _cognitiveServices.GetAreaOfImportance(streamReader.ToArray());
50                    var cropImage = CropImage(img, area.areaOfInterest.X, area.areaOfInterest.Y, width, height);
51
52                    return cropImage;
53                }
54
55                var thumbnailResult = _cognitiveServices.GetThumbnail(streamReader.ToArray(), width, height);
56
57                return new MemoryStream(thumbnailResult);
58            }
59        }
60
61        public string GenerateThumbnailUrl(int width, int height, MediaItem mediaItem)
62        {
63            var streamReader = MediaManager.GetMedia(mediaItem).GetStream();
64            {
65                using (var memStream = new MemoryStream())
66                {
67                    streamReader.Stream.CopyTo(memStream);
68
69                    var thumbnail = _cognitiveServices.GetThumbnail(memStream.ToArray(), width, height);
70                    var imreBase64Data = System.Convert.ToBase64String(thumbnail);
71
72                    return $"data:image/png;base64,{imreBase64Data}";
73                }
74            }
75        }
76
77        private Stream CropImage(Image source, int x, int y, int width, int height)
78        {
79            var bmp = new Bitmap(width, height);
80            var outputStrm = new MemoryStream();
81
82            using (var gr = Graphics.FromImage(bmp))
83            {
84                gr.InterpolationMode = InterpolationMode.HighQualityBicubic;
85                using (var wrapMode = new ImageAttributes())
86                {
87                    wrapMode.SetWrapMode(WrapMode.TileFlipXY);
88                    gr.DrawImage(source, new Rectangle(0, 0, bmp.Width, bmp.Height), x, y, width, height, GraphicsUnit.Pixel, wrapMode);
89                }
90            }
91
92            bmp.Save(outputStrm, source.RawFormat);
93
94            return outputStrm;
95        }
96    }
97}

Let's see this in action!

After picking your picture in the AI Cropping Image field, it gets already cropped and you can see the different thumbnails. You can choose or change the thumbnails by updating the child items here: /sitecore/system/Settings/Foundation/Vision/Thumbnails.

Also note that you get an auto generated Alt text "Diego Maradona holding a ball" and a list of tags.

AI Cropping Image Field

The results

This is how the different cropped images will look like in the front end. Depending on your front end implementation, you will define different cropping sizes per breakpoints.

In this following implementation, I'm setting the image as a background and using the option to render the image URL as follows:

Tablet

Desktop

Mobile

Sitecore Media Cache

As I mentioned before, the cropped images are also stored in the media cache, as we can confirm by checking the media cache folder

Other usages and helpers

Sitecore HTML helper

You can use the @Sitecore.Html helper to render an image tag, as usual, or to generate just the URL of the image (src).

Code

1@Html.Sitecore().AICroppingImageField("AI Image", Model.Item, new AdvancedImageParameters { Width = 600, Height = 600, AutoAltText = true })

Result

Note: Have a look at the AdvancedImageParameters:

OnlyUrl: If true it will just render the image URL (for being used as src in the img tag).
AutoAltText: If true, the alt text will be replaced by the one generated from Azure IA.
Width and Height: int values, to specify the cropping size.
Widths and Sizes: If set, it will generate a srcset image with for the different breakpoints.
SizesTag and SrcSetTag: Those are mandatories if when using the previous settings.

Code

1@Html.Sitecore().AICroppingImageField("AI Image", Model.Item, new 
2AdvancedImageParameters {Widths = "170,233,340,466", Sizes = "50vw,(min-width: 
3999px) 25vw,(min-width: 1200px) 15vw", SizesTag = "data-sizes", SrcSetTag = "data-
4srcset", AutoAltText = true })

Result

GlassMapper extension

At last, an extension method has been added in order to get the media URL from the image field.

Code

jsx

1<img src="@Model.AiImage.GetImageUrl(600, 600)" />

Result

1<img src="https://vision.test.cm/-/media/project/vision/homepage/iatestimage.png?
2w=600&h=600&smartCropping=true&hash=C2E215FE2CF74D4C8142E35619ABB8DE">

Sitecore Package and code

Please find the whole implementation in my GitHub repo, also feel free to contribute :)

You can also download the Sitecore package from

here

Download: Sitecore_Computer_Vision_CroppingImage_Field-1.0.zip

Sitecore_Computer_Vision_CroppingImage_Field-1.0.zip178.0 KB

. Note: It has been tested on Sitecore 8.2, 9.x and 10.x.

You can also get the Docker asset image from Docker Hub!

1docker pull miguelminoldo/sitecore.computer.vision

That's it! I hope you find it interesting and useful! Any feedback is always welcome!

Sitecore smart image cropping, tags and alt text with AI: Azure Computer Vision - Part III.

Image request flow

AICroppingProcessor

The "Get Thumbnails" method limitations

Let's see this in action!

The results

Sitecore Media Cache

Other usages and helpers

Sitecore HTML helper

Code

Result

Code

Result

GlassMapper extension

Code

Result

Sitecore Package and code

here

Share this post

Comments (0)

Share this post

Comments (0)

Sitecore Marketplace Apps - A Text-to-Speech & Poscast Use Case

From Composable to Composed — Why SitecoreAI Signals a New Chapter for DXP

Sitecore User France - November 2024