Unsharp Anti Aliasing

13/02/2012

While working on trying to implement FXAA I stumbled upon a fairly cool hack for AA which provides some interesting results. It isn't exactly that smart or groundbreaking but it is still useful to know.

The first thing I found was that as it turns out the naive solution of finding the color difference to surrounding pixels and then mixing based upon this difference works surprisingly well. The one main gripe I had with it - which is fairly much the same for all AA aproximations was that it made detailed textures blurry and the whole scene look a bit messy. Even with a good tolerance threshold and some other cheap tricks on top it just wasn't really up to it.

At the same time I was doing this I was playing around with implementing an unsharp mask - a post effect for sharpening the final image. I realised that all of the color difference operations I was doing I seemed to be performing in both shaders. It made sense to combine them into one and see what happens.

Combined into a single shader it sharpens the resultant image and blurs along areas of hard contrast - hopefully edges. Again, the effectiveness of this result surprised me. Sharpening the image ensured that it retained some nice per-pixel details and even though this wasn't all over the image (there were still too many false positives being highlighted as edges), the fact that it had some sections of high detail gave it an overall sharp look. This sharpening also didn't get in the way of the AA effect along edges. Some other unexpected benefits were the removal of the unsharp mask haloing, which you can get around edges, and also the way it works well with cheaper texture sampling methods.

On a good day it can resemble 4x MSAA with an intelligently applied unsharp mask and on a bad day it can create some odd artefacts and blurry texture sections. Overall though - for a very cheap filter it gives some quite impressive results. As always source is free to use/modify/distribute.

vec3 fxaa_unsharp(sampler2D input, vec2 pos, int width, int height) {
  
  const float sharpen = 0.5;
  const float boundry = 0.1;
  const float alias_scale = 2.75;
  
  float kernel = 1.0;
  
  float xoff = kernel / float(width);
  float yoff = kernel / float(height);

  vec3 rgb_ne = texture2D(input, pos + vec2(-xoff,yoff)).rgb;
  vec3 rgb_n = texture2D(input, pos + vec2(0,yoff)).rgb;
  vec3 rgb_nw = texture2D(input, pos + vec2(xoff,yoff)).rgb;
  vec3 rgb_w = texture2D(input, pos + vec2(xoff,0)).rgb;
  vec3 rgb_o = texture2D(input, pos + vec2(0,0)).rgb;
  vec3 rgb_e = texture2D(input, pos + vec2(-xoff,0)).rgb;
  vec3 rgb_sw = texture2D(input, pos + vec2(-xoff,-yoff)).rgb;
  vec3 rgb_s = texture2D(input, pos + vec2(0,-yoff)).rgb;
  vec3 rgb_se = texture2D(input, pos + vec2(xoff,-yoff)).rgb;
  
  vec3 average = (rgb_ne + rgb_n + rgb_nw + rgb_w + rgb_e + rgb_sw + rgb_s + rgb_se) / 8;
  vec3 difference = rgb_o - average;
  
  rgb_o = rgb_o + (difference * sharpen);
  difference = rgb_o - average;
  
  float fdiff = abs(dot(vec3(1,1,1), difference));
  
  if (fdiff > boundry) {
    
    float alias_amount = clamp(fdiff * alias_scale, 0.25, 0.75);
    //return mix(vec3(0,1,0), vec3(1,0,0), alias_amount);
    return mix(rgb_o, average, alias_amount);
    
  } else {
    return rgb_o;
  }

}