Retinex

Note: This translation of the "Retinex" French page is a working document.

Generalities

While the eyes are able to see correctly the colours through a poor lighting, a coloured surrounding or a veil of fog, cameras badly manage in theses conditions. It is by copying the eyes biological mechanisms to adapt itself to these conditions that the MSR algorithm (MultiScale Retinex) has been created. In addition to the digital photography, the Retinex algorithm (Retinex is the contraction for Retina + Cortex) is used in astronomy to show up information laying in the astronomical photographs, in medicine to detect not much visible structures in radiography and tomodensitometry. Numerous theories and algorithms have been working out for more than 20 years. The first experimentation has been proposed by Rahman in 1996. The approach follows the human visual perception and the Edwin Land's retinal function. In a way, this approach is pretty similar to CIECAM. This function and more particularly its general form is similar to a DOG (Difference Of Gaussian). The idea consists of characterizing the luminous information of a point from its intensity and the intensity of its neighbours. This said, this approach has no scientific basis and is only lying on experience and various empirical constants. Over the years many improvements have been added, but from my own point of view, no one is fully satisfactory. I relied on two documents :

• "Automatic Image Haze Removal Based on Luminance Component" (Fan Guo, Zixing Cai, Bin Xie, Jin Tang"[1]
• "Retinex Algorithm on Changing Scales for Haze Removal with Depth Map" (Weixing Wang, Lian Xu)"[2]
• and from some programming tricks inspired from "2003 Fabien Pelisson <Fabien.Pelisson@inrialpes.fr>"

The use of Retinex might be beneficial to images processing:

• that are hazy, misty or having a veil of fog
• with important luminance gaps
• where the user looks for special effects (tone mapping …)

Retinex at the beginning of the processing

It is the algorithm described in this page, with its limitations, advantages and drawbacks.

Retinex in "Wavelets"

I installed 2 possibilities to enable Retinex in the Wavelet process: Wavelet levels/fr. Some of the quoted limitations are no more there !

Quick comparison between the both versions " Retinex in Wavelets" and " Retinex at the beginning of the processing": Wavelet_levels/fr#Avantages_.28.2B.29_et_inconv.C3.A9nients_.28-.29_de_Retinex.2Ffr_par_rapport_.C3.A0_Retinex_in_wavelet

Imposed limitations by RawTherapee

The basic algorithm impose a reference to the whole image, and not to a crop or a reduction like in the RawTherapee processing. This limitation imposed by the gaussian function causes several consequences:

• The processing has been shifted from its dedicated place, that should have been close to "Lab Adjustments" - near the beginning of the RawTherapee processing – that is necessarily (except if somebody has an idea to do different) a raw process. As a consequence, non raw files (TIFF, JPG, …) can't be processed with this algorithm? This problem should be solved soon (November 2015 ??).
• The second consequence of this position is that the characteristics of the raw data situated just after demosaicing are very different that the ones situated downstream: no gamma, no gamut limitation, no white balance, no RGB conversion… so we must expect artefacts and poor luminance and colour rendering.
• The third consequence is the system response time which imposes to re-actualize the whole process after each change in the settings.
• The fourth, sometimes, will need a white point modification, in order to avoid colours distortions (for example: a magenta sky).
• But advantage, this process being situated before "Denoise", the noise reduction will be here fully effective.

Nevertheless, despite these handicaps, as we will see it further, results are more than satisfactory as well about processing time that in contrast and colours rendering, particularly by joining Retinex and Wavelets actions.

Algorithm principle

For more information, see the "pdf" documents given in the "Generalities" section.

To elaborate a "Transmission Map" file obtained by

• making the difference, for luminance only, between each pixel logarithm of the input image and the neighbouring pixels logarithm of the matching gaussian image. The standard deviation (gaussian function) used here is very high - usually in RawTherapee sigma values from 0.5 to 5 are common – here the values range from 10 to 280 (Single Scale Retinex), it is the user choice.
• modifying the input image luminance distribution by

• applying a gamma before the "Transmission Map" file creation
• applying an inverse gamma to restore the input image characteristics

This modification of the distribution allows:

• to change the image tonality
• to modify the "Transmission Map" file to take into account for example the under or over exposed areas.

• applying several time (Scale) – 3 times, not modifiable by the user – this algorithm with an addition using empirical coefficients (Multi Scale Retinex). (Low scale values increase the apparent contrast, but give a perspective look to the image, high scale values make the image more natural but they have a tendency to increase the noise).
• a variation of MSR by the user according to the desired effect (Uniform, Low, High, Highlight)
  • Uniform: Strive to process low and high intensities in a balanced way.
  • Low: Improve low intensities areas
  • High: Improve the rendering of the more exposed areas
  • Highlight: Improve the rendering of the highlight areas that may become magenta, but can bring strong artefacts.
• choosing the colorspace
  • logarithmic L*a*b*
  • logarithmic HSL
  • linear HSL – this version doesn't comply with Retinex algorithm but in some cases it allows a more suitable image processing.

So, we get a logarithmic distribution "Transmission Map" – except in linear mode! - that we will be able to "subtract" to the input image, either to get an image theoretically free from mist and from veil fog or to get special effects. This distribution owns approximatively a gaussian distribution with a minima (minT) about, according to the images – higher for images with veil – from -10 to -40, an average close to from -1 to +2, a standard deviation often about 2 to 6 and a maxima (maxT) about 10 to 40. The negative values mean low intensity and positive values mean high intensity. Caution, these values are logarithmic coefficients that stand for very high values (log 1000 = 6.9) (exp 10 # 22000)...(exp 20) # 500.000.000. In theory we should use high "Scale" and gaussian values in the areas closest to the lens (where the veil effect is low) and low values in the distance (where the veil effect is important).

To process the mask issued from the gaussian process

The "Transmission Map" file correspond to a recursive process of:

• The source image – Input image which went through various modifications by the histogram equalizer and the gamma.
• The "mask" files directly issued from the gaussian process (scale, radius, method low.. high…)

"Mask" display type

"Mask" files processing (the idea came to me by studying the Rusell Cottrell's plugin) will allow to decrease flares and artefacts. I added a combo-box to choose the display type. This is both an educational system and also an aid to find the "right" settings.

• Process:
  • Standard: it is the default setting
  • Mask: display the mask obtained by the Retinex algorithm issued from the gaussian process. We will see here the impacts:
    • of the various settings upstream: method (low, uniform, high and highlight), radius, gamma, histogram equalizer. In the other hand, the other settings downstream have no effect on this display: contrast, gain, brightness, threshold and the curve "Transmission Map".
    • of the various settings of the "Mask equalizer".
    • you can export these settings (TIFF/JPG) to use this mask in external software (Gimp, Photoshop).
  • Unsharp mask: you can use this option to subtract the mask to the input image. In this case, an action on "strength" will allow to balance between the input image and the mask. So we get the possibility to have images with very high radius values.
  • Transmission: display an "image" of the "Transmission Map" file.
    • this "image" doesn't match the reality that matches to a logarithmic scale which values are mostly situated between -30 and +30.
    • (fixed) in order to make this values "visible", I applied two arbitrary coefficients – I avoided the automatic computing in order to allow comparisons – which shift and increase the basic values.
    • (auto) I used here the values displayed in the Preview (TM Min Max Mean Sigma) to elaborate a file with values ranged from 0 to 32700. Of course these data become visible with the maximum range but don't match the reality. This configuration can be, depending on the images, preferable to "fixed".
    • the displayed image takes into account each Retinex processing, except: a) "median filter"; b) "Gain"; c) "Brightness" (offset); d) and of course "Strength".
    • you can act on the different settings, particularly on the "Mask equalizer" ones, and also the "Transmission Map" curve, and then see directly the actions on the haze, the artefacts annd contrasts.

Action methods on the masks

I set 3 methods:

• a diagonal curve which acts directly on the recursive contrast of the "mask" files – take care, this curve is very sensitive and can lead to important artefacts. In the other hand, it hasn't, or a few, impact on the processing time. You can use this curve alone or combined with the other methods.
• a gaussian process is applied to the masks. You can act on the shadows, the highlights and the radius. The increase of the processing time is moderate.
• a "wavelet" process (Sharp mask) is applied to the masks. You can act on the shadows and the highlights. The increase of the processing time is important or very important according to the selected method (partial or total).

"Transmission Map" file processing

The main basis processing consist of to apply, using the "Transmission Map" average and standard-deviation, a transformation of the type:

• newT = (oldT – mini) / maxi-mini.
• with mini = average – k * standard-deviation
• with maxi = average + k * standard-deviation

The choice of k - contrast (variance): modifiable by the user – is decisive in the image rendering: low values will increase the seeming contrast, hight values will make the image more natural with less artefacts and hazes.

I added three components to this basis processing:

• a threshold - modifiable by the user – that will allow to reduce the maximum (maxT) and minimum (minT) values by integrating the clipped values to the new distribution.

Example: if the chosen threshold is 10, all the values lower than -10 will became -10, all the values over +10 will become +10. This action is going to increase the contrasts, but can lead to artefacts.

• a curve (flatcurve) - modifiable by the user – to directly act on the distribution, by replacing theory by practical… It is by acting on the distribution that the user will be able for example to reduce the negative values and increase the positive ones (or the opposite) to improve the rendering in real time.
• a median 3x3 - modifiable by the user – that acts on the distribution irregularities, in order to reduce the artefacts.

Elaborating the "Image Haze-free" file

This file is obtained by "difference" between the input image and the "Transmission Map" file.

We are going to use here 2 extra parameters, Gain and Brightness (Offset): These two parameters allow to include the "Transmission Map" file to an exploitable image. Indeed, without precautions, the restored image could be totally out of the luminance usual limits that are between 0 and 32768.

• "Gain" will act on the total amplitude by increasing or decreasing it and for example reduce the numbe of totally black pixels.
• "Brightness" (Offset) will allow to reset the luminance after "Gain" acted.

In order to give some help to the user, I added a display for the "haze-free" image minimum values to the GUI. We can act on "Gain" and "Brightness" (Offset), but also on each other parameters, particularly "contrast" (variance) and the "Transmission Map" curve. The ideal is to set Min values the closest possible to 0 and the Max values the closest possible to 32768. For example, an "original" image with min=-44230 and max=76000 could become min=32, max=32500. Of course no other values are forbidden, but in all cases these values will be clipped and brought back to 0 – 32768, the "Transmission Map" file wealth is lost !!

Keys for acronyms and labels used in the GUI to ease the use:

• Restored haze-free Min=-5681 Max=34568 : display image maximum and minimum values "without veil" before clipping.
• TM Min=-3.8 Max=4.1 Mean=0.1 Sigma=3.3 : Tansmission Map -- Min and Max match to the following computing Min=Mean - Sigma * Variance / 100 ; Max=Mean + Sigma * Variance / 100 *TM Tm=-6.5 TM=11.8 : Transmission Map -- Tm minimum value of the distribution; TM maximum value of the distribution.

It is obvious that that a display in the GUI:

• of the "Transmission Map" distribution and also
• the "Haze-free" image data

should make easier both the system understanding and ergonomics… but this is still to be fixed ???

Retinex versus Tone-mapping

There is several ways (according to specialized sources in this field) to adapt the Retinex algotithm to a "Tone-mapping" rendering. The study of the existing code, my understanding of the problem… led me to the following solution:

• Proceed to several "Retinex" specific code iterations. I name specific code the part excluding
  • the RGB==>L*a*b* or RGB==>HSL conversions (and vice versa).
  • the histogram equalizer.
  • the differential gamma (low, middle, high, free).
  • in the same way, aren't concerned the choices done with the "Low", "Uniform", "High", "Highlight" method
• These iterations obviously increase the time processing, up to 2.5 times for 5 iterations.

You can choose between several ways to carry on the iterations by acting on 3 factors I called "Gradient":

• "Gaussian gradient" that act on: a) the gaussien blur sigma (Radius or Neighboring pixels); b) the "scale" factor (set by default to 3).
• "Transmission gradient" that act on: a) the contrast (variance); b) the threshold that limits the "Transmission Map" file.
• "Strength gradient" that act on "Strength", that means the combination of the "Haze-free" image with the original one.

If you set these 3 gradients to 0, all the iterations will be identical to the basis processes.

Impact of each gradient:

• "Gaussian gradient" : according to the "it" iteration, the Radius value ("Neighboring pixels") is divided by "grad" and "scale" value become:
• Gg 1 :
  • it=1 grad=1; it=2 grad=1.25;it=3 grad=1.5;it=4 grad=1.75;it=5 grad=2
  • it=1 scal=4; it=2 scal=3; it=3 scal=3; it=4 scal=3; it=5 scal=2
• Gg 2 :
  • it=1 grad=1; it=2 grad=1.50;it=3 grad=2.0;it=4 grad=2.50;it=5 grad=3
  • it=1 scal=5; it=2 scal=4; it=3 scal=3; it=4 scal=2; it=5 sca=2
• Gg 3 :
  • it=1 grad=1; it=2 grad=1.66;it=3 grad=2.3;it=4 grad=3.00;it=5 grad=3.6
  • it=1 scal=5; it=2 scal=4; it=3 scal=3; it=4 scal=2; it=5 scal=2
• Gg 4 :
  • it=1 grad=1; it=2 grad=1.80;it=3 grad=2.6;it=4 grad=3.40;it=5 grad=4.2
  • it=1 scal=5; it=2 scal=4; it=3 scal=3; it=4 scal=2; it=5 scal=2
• Gg 5 :
  • it=1 grad=1; it=2 grad=3.50;it=3 grad=6.0;it=4 grad=8.5;it=5 grad=11.0
  • it=1 scal=5; it=2 scal=4; it=3 scal=3; it=4 scal=2; it=5 scal=2
• Gg 6 :
  • it=1 grad=1; it=2 grad=6.00;it=3 grad=11.0;it=4 grad=16.0;it=5 grad=21.0
  • it=1 scal=5; it=2 scal=4; it=3 scal=3; it=4 scal=2; it=5 scal=2
• Gg-1 :
  • it=1 grad=1; it=2 grad=0.875;it=3 grad=0.75;it=4 grad=0.50;it=5 grad=0.37
  • it=1 scal=3; it=2 scal=3; it=3 scal=3; it=4 scal=3; it=5 scal=3

For Gg 5 and Gg 6, the selected method is taken into account, if it is "highlight", 'grad' values are boosted according to "Highlight Threshold"

• "Transmission gradient" : according to the "it" iteration, the contrast (variance) and threshold are multiplicated by "var":
  • Tg 1: it=1 var=1; it=2 var=0.875; it=3 var=0.75; it=4 var=0.625; it=5 var=0.5
  • Tg 2: it=1 var=1; it=2 var=0.800; it=3 var=0.60; it=4 var=0.400; it=5 var=0.2
  • Tg-1: it=1 var=1; it=2 var=1.125; it=3 var=1.25; it=4 var=1.375; it=5 var=1.5
  • Tg-2: it=1 var=1; it=2 var=1.400; it=3 var=1.80; it=4 var=2.200; it=5 var=2.6
• "Strength gradient" : according to the "it" iteration, "strength" is multiplicated by "s":
  • Sg 1 : it=1 s=1.30; it=2 s=1; it=3 s=0.70; it=4 s=0.5; it=5 s=0.5
  • Sg 2 : it=1 s=1.60; it=2 s=1; it=3 s=0.40; it=4 s=0.3; it=5 s=0.3
  • Sg-1 : it=1 s=0.80; it=2 s=1; it=3 s=1.20; it=4 s=1.2; it=5 s=1.2
  • Sg-2 : it=1 s=0.60; it=2 s=1; it=3 s=1.40; it=4 s=1.5; it=5 s=1.5

Note : "Hue equalizer" and "Transmission median filter" are applied only once, whatever the number of iterations is.

The various adjustments

Adjustments always accessible

There are 5 adjustments at your disposal:

• Retinex method : with "Low" "Uniform" "High" "Highlight"
  • Low: improve low intensity areas
  • Uniform: Strive to process low and high intensities in a balanced way.
  • High: Improve the rendering of the more exposed areas
  • Highlight: Improve the rendering of the highlight areas that may become magenta, but can bring strong artefacts.

These 4 methods act directly on the "Transmission Map" file.

• Space
  • L*a*b*
  • HSL Log
  • HSL Lin

The choice between these 3 spaces depends on the image… The 2 firsts "meet" the Retinex algorithms.

• Strength: Combine "Haze-free" image with the original image. Strength=0 brings no Retinex processing to the image. Strength=100 brings a total processing, "Haze-free" image only is displayed. Values under 50 are recommended.
• Chroma (patch): act on the colour component, using the Retinex algorithm, in percentage of the "strength" value relative to the luminance. The slider runs a Retinex process dedicated to the colour component by "simplifying" the process, luminance specific components - gamma, luminance, gaussian mask, ...- aren't executed. Caution, processing times are doubled and the extra memory calls important.
• Radius (Neigboring pixels): take into account the neighbouring pixels using the "Difference gaussian" algorithm, higher are the values, more affected is the foreground. Lower are the values, more affected are the distant areas, act directly on the "Transmission Map" file.
• Highlight threshold: only accessible with the " Highlight" method; set the "Highlight" method threshold, a value of 1 will give about the same effects that the "High" method, high values should allow to reduce the magenta colours in the overexposed areas. Be careful, it will probably be needed to:
  • change some adjustments, in particular "Radius" (Neigboring pixels)
  • increase the "Raw white point" value in case of magenta hues in the overexposed areas
  • use "Hue equalizer" in case of magenta hues in the overexposed areas
  • act on "Gamma Retinex"
• Contrast (Variance): This coefficient balances "Transmission Map" average and standard-deviation. It is a key factor for the image rendering. It acts on the low and high contrast thresholds of the "Transmission Map" file, the lower are the values the more contrasted is the image, the higher are the values the more natural is the image and the artefacts reduced. This Contrast (Variance) doesn't act on the "Transmission Map" file but allow the Haze-free image computing.