Outlier Rejection Question

Anyone has some experience in that ? Or you have never seen any difference between them ?

Thank you

I use the help text in APP as a guideline:

Thanks @kees_scherer. I also use these tooltips which are clear about Sigma and Winsor Sigma clip filters.

I just wanted a more detailed explanation about LN and MAD filters. Is there a rule of thumb depending of the number of frames or we should use them (LN and MAD) everytime ?

Well, thanks again for your answer.

Fred

Here's a post about the number of frames and rejection. Usually with more frames the LN algorithms are better in general. 

Posted by: Fredmt

Hello,

 

 I've been wondering what is the best Outlier Rejection in general in 2) Calibrate ? Winsor or Sigma Clip ?

What about the 6) Integrate window ? Winsor or Sigma Clip  (+ LN ? +MAD ? or even both ? With diffraction protection )

I've seen many different things (Especially on this forum) but some of them are different due to problems that have been corrected along the way. I guess this is simply because of the evolution of APP.

PS : I use an APO Refrator 127/952 (No diffraction pattern) with an OSC ZWO ASI071 MC Pro. I usually stack large data sets (Sometimes 40-60 to more than 200 frames).

 

Thank you very much for your amazing job.

Fred

Hi @fredmt, @kees_scherer, @vincent-mod,

There are 8 different rejection algorithms in integration. In creation of the masters, there are only 4, the Local Normalization (LN) filters are not available in the creation of master frames since that does not make sense.

Let me explain that first, calibation frames don't need to be locally normalized

• flats need to be normalized, but never locally !
• bias and dark frames should never be normalized,

so the pixel values don't need to be adjusted to get better and more robust/reliable statistics when calibration frames are concerned.

This argument holds as well for use of Linear Fit clipping used in other applications, you should not use linear fit clipping/local normalization on calibration frames.

Local Normalization rejection is a much more advanced method than linear fit clipping, because linear fit clipping never attempts to actually calculate the local sky background for each pixel. Linear fit clipping is more naive, since it only studies the relation of pixels in 1 pixelstack, whereas Local Normalization studies the relation of each pixel in the entire pixelstack with the surrounding pixels in the pixelstacks. The range is 32x32 pixels with interpolation even in the latest version. This makes LN rejection much ! more reliable.

So for regular integration, with APP's latest version, I would always recommend to use one of the LN filters, since these are more robust over the entire field of view of your integration.

Some explanation is here as well, seventh bullet:

https://www.astropixelprocessor.com/community/release-information/astro-pixel-processor-1-073-ready-for-download/

Here is an example of integration of H-alpha data from the Rosette Nebula from 5 different telescopes and camera's:

Rejection Map when using non-LN rejection:

Rejection Map when using LN rejection, showing much more robust and very uniform rejection:

In one image:

Then we have the discrimination between sigma clipping and winsor sigma clipping.

Winsor sigma clipping is more robust, but can give ugly artefacts at star borders, especially on diffraction patterns or when data is combined from different telescopes.

You can counter the diffraction artefacts by enabling diffraction protection though. A usefull guideline is to set the diffration protection to 5-10% of the amount of frames that you are integrating. So for 100 frames, set the diffraction pattern to a value of 5-10. This basicially means, that in each pixelstack, the amount of pixels that is rejected, is limited to 5-10 pixels. Which will still reject the outliers, and at the same time, will better preserve good signal and thus maintain a better Signal To Noise Ratio. It also has the effect that star borders are less harmed due to rejection when the stars have different sizes/shapes in your images.

So in this regard, I would recommend winsor sigma clipping in combination with diffraction protection over regular sigma clipping.

Finally, we have a discrimination between MAD and non-MAD fitlers.

MAD = Median Absolute Deviation

MAD is a robust way to calculate the standard deviation in the pixel stacks and is more robust than the regular calculation of the standard deviation (root of the variance).

MAD tends to work bettter than non-MAD filters with a small amount of frames. I would say with less than 100 frames. If you have more than 100 frames, the difference will be very minimal if any.

If you use a MAD filter, you must be carefull with the outlier rejection settings, especially the kappa value. If you set this lower than 3, you will start to reject a lot of data and you run the risk of rejection good data as well. Always check the amount of rejection by enabling rejection maps in integration.

So in this regard, I would recommend MAD rejection in combination with a rejection map for verification over non-MAD clipping when you have less than 100 frames. But keep in mind, you need to be carefull with the kappa value, don't set it much lower than 3 😉

So overall, I would recommend the following:

• always use LN rejection, the latest version of APP is very advanced as descrived in release notes of APP 1.073
• Use winsor clipping in combination with diffraction protection if needed to prevent artefacts
• With less than 100 frames, use MAD but don't set the kappa to low.

In general, always be carefull with outlier rejection, setting the kappa too low and the amount of iteration too high, will always be very harmfull for your data. Yes, all outliers will be gone with aggressive settings, so will be rejected part of the good data . You can reduce the Signal to Noise ratio by 10% to even 50% procent if you are not carefull !

kappa -> lower kappa is more aggressive, a value lower than 2,5 is already very aggresive !

iterations -> I rarely use more than 2 iterations, and in most cases, 1 iteration is all you need. 3 iterations is already very aggressive !

So using more than 3 iterations with kappa 2 should never be used, it will be very bad  for your Signal to Noise ratio of your integration.

Let me know if this has explained some of your questions and if you have any others.

Kind regards,

Mabula

Thanks very much fro the detailed explanation @mabula-admin! Still, one question of the topic poster has not been answered: what is the best outlier rejection in 2) Calibration? In general I do not ever touch the settings in that screen and I get very good results (I think!!!). So if you have advise on how to make APP do a better job then we are all ears 🙂

CS, Wouter

Thank you very much for this very complete answer @mabula-admin.

But as mentionned by @wvreeven, what about the 2) Calibrate window ?

I usually use outlier rejection on all my calibration frames (Flat, DarkFlat, Dark and Bias) and I also get great results. Do I really need to ? (I've seen some people as @wvreeven not using them at all.  Some people do though). Is there a criteria to use them or not to use them ?

Thanks again

Fred

On darks and bias you don't want to use outlier rejection as those frames are used to actually calculate the noise and need all of it without rejection. Flats do need to be calibrated with bias rejection.

Ok, Thanks @vincent-mod for your answer.

Why, then, are these settings available ? Is there a case where it's better to use them ?

I saw videos (even on the APP channel on Vimeo) where these settings are used.

I'm just trying to understand how all these things work to get the best out this great software.

Fred

Hi Fred,

In my view, the main use of (a very mild!) form of outlier rejection for your calibration masters would be to eliminate cosmic ray hits. These are not a property of your sensor, but random hits. These will not be corrected by a bad pixel map (because a BPM is a "description" of the bad behaviour of your sensor, but can not correct random hits). The longer you expose your dark frames, the more chance for a hit during your exposure. So, using rejection filters can be benificial for your darks. Will it always make your masterdarks  better? That depends. Cosmic ray hits are random, so you might not have any and you certainly will not have many. The main thing is to keep the rejection filter very mild, otherwise you filter out the randomness of the thermal dark signal and an average of that is what you want to have in your Masterdark. 

Regards,

Anne

I think you might just try it to see if you indeed have cosmic ray hits and see if they pose a problem in a BPM for instance. I never really saw this mentioned as being a potential problem either. Rejection should be kept at a minimum or even off in most cases for bias and darks, it depends a bit on the sensor characteristics I guess too. Not all data is always the same unfortunately so to have more options available is a nice thing, though can be confusing.

Hello @annehouw and @vincent-mod. Thank you for your answers.

I've tired already a few times these settings. I'll do more tests in the next few days to see what works best for me.

Thanks again

Fred