[Sticky] When to use which outlier rejection filter

Thank you Marc for this excellent and important question for our data integrations.

Three rules of thumb to follow would be:

• If you have less then 15-20 frames, use median integration without sigma clipping
• If you more than 15-20 frames and less than 40-50 use average integration and outlier rejection with kappa 2.5 & 2 iterations.
• If you have more then 40-50 frames use average integration and outlier rejection with kappa 3 (or larger) with only 1 iteration.

Regarding linear fit clipping:

Linear fit clipping is a solution for a problem, the problem of bad data normalization over the entire field of view.

• so never use it in integration of bias, dark & flat frames because the problem of bad normalization over the entire field of view is non-existent in this case.
  
• And in integration of lights, use LNC instead of linear fit clipping. LNC actually solves this problem, so linear fit clipping is redundant in that case.

Then you are left with the choice of sigma clipping or winsorized sigma clipping.

• If you combine data of the same camera and optics, I would advise to always use winsorized sigma clipping. It works a bit better, especially with more frames.
• Otherwise, be careful with winsorized sigma clipping, it is known to introduce stack artefacts, especially at star borders. It's really bad if you combine data of a refractor and a newton for instance. The diffraction spikes of the newton will show artefacts. These artefacts won't be there with regular sigma clipping.
• In case of doubt, use sigma clipping instead of winsorized clipping, combined with LNC regular sigma clipping in APP works very good.

GENERAL WARNINGs on outlier rejection:

• always try to remove as much artefacts, like bad pixels, in the calibration fase using a Bad Pixel Map
• let integration work for you by dithering your data with plenty of pixels. Dithering with 10-20 pixels is much more efficient to remove calibration problems than 1-2 pixel dithers.
• Outlier rejection must be considered as a filter to be used as a last resort to solve artefacts/problems in your data. The less you need to use it, the better the integration quality will be.
• Outlier rejection removes problems, but also good data. If you set outlier rejection too strong, you will remove a lot of good data as well. So being too aggresively with outlier rejection is very harmfull for the Signal to Noise ratio of your integrated stack. So be carefull with using it.
• If you have more than 50 frames, then 1 iteration is all it should take to remove problems. Only adjust your kappa value.
• Never set the kappa value below 2, this is always really destructive for your data.

And thank you for the answer (you just wrote a chapter for the manual 🙂 ).

Indeed I did 😉

Hi Mabula -

Can you explain what LNC (local normalization correction) is and when to use (as well as when not to use it).  Also, while you're at it, an explanation of MBB (multi-band blending) would be very helpful to me as well.

Thanks in advance.

Hi Scott,

I'll make a note of it to create a FAQ for both features with extensive explanation, okay?

Mabula

Sounds great, Mabula - thanks!

Hi,

Do the three rules of thumb also apply to every kind of frames (lights, darks, bias, flats)? I wonder if outlier rejection filter should be used when creating a bad pixel map? Aren't hot pixels and cold pixels eliminated by outlier rejection mechanism?

Thanx a lot, Stephan

Hi Stephan,

Yes, the three rules of thumb apply to very kind of frame, lights, darks, flats and bias frames 😉

If you enable outlier rejection (for flats and darks) in the creation of a Bad Pixel Map, they will be ignored. Some sensors have bad pixels that behave almost linearly and sometimes show different behaviour per exposure. If you would have outlier rejection enabled in that case, the BPM could miss that bad pixel, so that's the reason for ignoring outlier rejection in the BPM creation.

Yes, outlier rejection would eliminate hot and cold pixels, but if you can remove hot and cold pixels immediately in calibration, before integration you have a big advantage! Having already removed hot and cold pixels will give you the opportunity to use much less aggresive outlier rejection filter parameters ( so less iterations and higher kappa) preserving the quality of your data better.

Be aware that setting outlier rejection parameters higher than needed, will remove good pixels as well and therefore will downgrade the Signal to Noise (SNR) ratio of you integration unnecessarily. 

So outlier rejection filters should always be used with caution ! Only use them if absolutely needed because they will downgrade your data if you apply them too strong.

Setting outlier rejection too strong can reduce SNR of you integration by as much as 5-50 % ! Which is very significant. For instance 3 iterations with kappa 2 will in almost all cases be very destructive to your data. Usually you only need 1 iteration with kappa 3 to get rid of outliers like airplanes/satellites/cosmic rays without harming your data 😉

Kind regards,

Mabula

Hi Mabula, first of all Happy Easter!

Would you mind updating this little guideline with the new rejection algorithms when you have a minute? I am still a bit confused about the LN, MAD and LN MAD combined algorithms. When would it be of advantage to use one of them? What kind of diffraction protection would you suggest to go with these settings? I tried a bit with LN MAD Winsor and 2 and was quite satisfied, however I am not sure if I am losing to much SNR.

Best Frank

Friendly bump as I am just again was pondering about this question. I have 20 subs of Red with lots of satellite trails. Normal Sigma Clip with 2.5, 2 shows none. MAD LN Winsor Sigma clip with 2.5, 2 show all of them (faint). Best Frank

Hi Frank @foschmitz,

Please accept my apologies for the late response. I have been very busy towards the APP 1.061 release.

Yes, the winsor sigma clip seems to underperform a little bit when compared to regular sigma clipping in some cases.

I will need to perform more testing, to see if I can further improve these outlier rejection filters.

To verify if you lose too much SNR by clipping too aggresively, is to simply create an integration without outlier rejection and one with. Then compare the SNR, it can be found in the fits header of the integration.

Another way is too create outlier rejection maps, they will show, how much is rejected.

I will replace this post with an updated one 😉 and will make this one redundant.

Kind regards,

Mabula

I have thought myself into paralysis and have a more basic question on this topic, can anyone help? 

I have over 20 frames each for Flats, Darks, Dark Flats and Bias (80 plus frames), and around 10 frames each for LRGB (40 frames).

So, when we are talking about this approx. 20 frames tipping point to go from Median to Average, does this apply to lights and calibration frames?

I assume it includes lights, in which case in my example am I working on 10 frames per channel or 40 light frames total when choosing Median /Average?

Also, the Calibration step only seems to refer to Calibration frames, not lights. SO, for example, if I am selecting Average or Median for Master Darks am I making the decision on how many Darks I have, or how many Lights?

Thanks

Jon

I have thought myself into paralysis and have a more basic question on this topic, can anyone help? 

I have over 20 frames each for Flats, Darks, Dark Flats and Bias (80 plus frames), and around 10 frames each for LRGB (40 frames).

So, when we are talking about this approx. 20 frames tipping point to go from Median to Average, does this apply to lights and calibration frames?

I assume it includes lights, in which case in my example am I working on 10 frames per channel or 40 light frames total when choosing Median /Average?

Also, the Calibration step only seems to refer to Calibration frames, not lights. SO, for example, if I am selecting Average or Median for Master Darks am I making the decision based on how many Darks I have, or how many Lights?

Thanks

Jon

The rejection is set for each type of frame. So e.g. if you have 20 dark frames and 10 light frames per channel use median without outlier rejection for the lights per channel and use average with outlier rejection for the dark master.

Great, thanks.

The Median/Average settings for calibration frames appear to be in the Calibration module but I'm not sure where the equivalent for lights is - can you help?

Great, thanks.

The Median/Average settings for calibration frames appear to be in the Calibration module but I'm not sure where the equivalent for lights is - can you help?

EDIT. I have found the lights in the Integration Module but have another question.

I have Set Weights to Quality and Integrate to Median, as I only have 12 Light frames (I would have used Average if 20 or more).

Further down I need to select Outlier rejection. I have read that the level of rejection to use is also dependent on the number of frames. I think basically it was saying the more frames you have the less iterations required (e.g. >50 frames use 1 iteration).

So, I am assuming this applies to the number of light frames - is this correct?

Assuming it is correct, are we still counting the number of frames in each channel or the sum of frames across all channels?

Anything else to consider?

Apologies for the picky questions but I'm sure I'm not the only one trying to get this right.

Thanks

Jon

Hi Jon,

When you use median, you should select "no rejection" since median already is kind of outlier rejection.

Actually I use always the default settings when using the outlier rejection filter (in combination with "average" when I have approx. at least 20 frames), which is windsor clip (for my images that is good enough), kappa 3 and iterations 1. I ALWAYS check the "create outlier rejection map" to see what pixels were rejected. If the final stack image still contains "errors" such as airplane or sat tracks I increase the level of outlier rejection, either iterations or lowering kappa to 2.5. I also review all singe frames before stacking and manually delete bad ones.

The number of single frames that is stacked to a result is to be considered. So when you create a stack for one channel it is the number of frames for this channel. When you integrate all frames of all channels in order to get an luminance frame (which I do only recommend when the numer of frames is about the same for each channel and the weights is set to equal), then consider the total number of frames.

My advise: almost every button, dropdown and selection in APP has a help that pops up, when you move the cursor over it. This help text is very good.

Cheers, Stephan

Thanks Stephan, that's really helpful.

In particular I hadn't thought about checking outlier rejection maps but have added that to my workflow.

Cheers

Jon 🙂