Outlier Rejection Q...
 
Share:

The problem with the data upload limit for attachments has been fixed. I have restored it to 30MegaBytes. A recent forum software upgrade was responsible for the changed limit. Please accept my apologies for missing this when upgrading the forum sofware, Mabula.

Outlier Rejection Question  

  RSS

(@fredmt)
Brown Dwarf Customer
Joined: 7 months ago
Posts: 8
18/03/2019 6:24 pm  

Hello,

 

 I've been wondering what is the best Outlier Rejection in general in 2) Calibrate ? Winsor or Sigma Clip ?

What about the 6) Integrate window ? Winsor or Sigma Clip  (+ LN ? +MAD ? or even both ? With diffraction protection )

I've seen many different things (Especially on this forum) but some of them are different due to problems that have been corrected along the way. I guess this is simply because of the evolution of APP.

PS : I use an APO Refrator 127/952 (No diffraction pattern) with an OSC ZWO ASI071 MC Pro. I usually stack large data sets (Sometimes 40-60 to more than 200 frames).

 

Thank you very much for your amazing job.

Fred

This topic was modified 5 months ago by Fredmt
This topic was modified 5 months ago by Mabula Haverkamp - Admin

ReplyQuote
(@fredmt)
Brown Dwarf Customer
Joined: 7 months ago
Posts: 8
30/03/2019 1:32 pm  

Anyone has some experience in that ? Or you have never seen any difference between them ?

Thank you


ReplyQuote
(@kees_scherer)
Main Sequence Star Customer
Joined: 2 years ago
Posts: 35
01/04/2019 10:51 am  

I use the help text in APP as a guideline:

APP outlier rejection


ReplyQuote
(@fredmt)
Brown Dwarf Customer
Joined: 7 months ago
Posts: 8
03/04/2019 8:28 pm  

Thanks @kees_scherer. I also use these tooltips which are clear about Sigma and Winsor Sigma clip filters.

I just wanted a more detailed explanation about LN and MAD filters. Is there a rule of thumb depending of the number of frames or we should use them (LN and MAD) everytime ?

Well, thanks again for your answer.

 

Fred

This post was modified 5 months ago by Fredmt

ReplyQuote
(@vincent-mod)
Galaxy Moderator
Joined: 2 years ago
Posts: 580
04/04/2019 11:04 am  

Here's a post about the number of frames and rejection. Usually with more frames the LN algorithms are better in general. 


ReplyQuote
(@mabula-admin)
Quasar Admin
Joined: 2 years ago
Posts: 2113
04/04/2019 1:05 pm  
Posted by: Fredmt

Hello,

 

 I've been wondering what is the best Outlier Rejection in general in 2) Calibrate ? Winsor or Sigma Clip ?

What about the 6) Integrate window ? Winsor or Sigma Clip  (+ LN ? +MAD ? or even both ? With diffraction protection )

I've seen many different things (Especially on this forum) but some of them are different due to problems that have been corrected along the way. I guess this is simply because of the evolution of APP.

PS : I use an APO Refrator 127/952 (No diffraction pattern) with an OSC ZWO ASI071 MC Pro. I usually stack large data sets (Sometimes 40-60 to more than 200 frames).

 

Thank you very much for your amazing job.

Fred

Hi @fredmt, @kees_scherer, @vincent-mod,

There are 8 different rejection algorithms in integration. In creation of the masters, there are only 4, the Local Normalization (LN) filters are not available in the creation of master frames since that does not make sense.

Let me explain that first, calibation frames don't need to be locally normalized

  • flats need to be normalized, but never locally !
  • bias and dark frames should never be normalized,

 

so the pixel values don't need to be adjusted to get better and more robust/reliable statistics when calibration frames are concerned.

This argument holds as well for use of Linear Fit clipping used in other applications, you should not use linear fit clipping/local normalization on calibration frames.

Local Normalization rejection is a much more advanced method than linear fit clipping, because linear fit clipping never attempts to actually calculate the local sky background for each pixel. Linear fit clipping is more naive, since it only studies the relation of pixels in 1 pixelstack, whereas Local Normalization studies the relation of each pixel in the entire pixelstack with the surrounding pixels in the pixelstacks. The range is 32x32 pixels with interpolation even in the latest version. This makes LN rejection much ! more reliable.

So for regular integration, with APP's latest version, I would always recommend to use one of the LN filters, since these are more robust over the entire field of view of your integration.

Some explanation is here as well, seventh bullet:

https://www.astropixelprocessor.com/community/release-information/astro-pixel-processor-1-073-ready-for-download/

Here is an example of integration of H-alpha data from the Rosette Nebula from 5 different telescopes and camera's:

Actual Ha Data Integration Without MBB Without LNC of 5 optical setups RosetteNebula

Rejection Map when using non-LN rejection:

Non Local Normalization Outlier Rejection Non Uniform Rejection

Rejection Map when using LN rejection, showing much more robust and very uniform rejection:

Local Normalization Outlier Rejection Uniform Robust Rejection

In one image:

LN Rejection Showing Uniform Rejection When Compared to nonLN rejection

 


 

Then we have the discrimination between sigma clipping and winsor sigma clipping.

Winsor sigma clipping is more robust, but can give ugly artefacts at star borders, especially on diffraction patterns or when data is combined from different telescopes.

You can counter the diffraction artefacts by enabling diffraction protection though. A usefull guideline is to set the diffration protection to 5-10% of the amount of frames that you are integrating. So for 100 frames, set the diffraction pattern to a value of 5-10. This basicially means, that in each pixelstack, the amount of pixels that is rejected, is limited to 5-10 pixels. Which will still reject the outliers, and at the same time, will better preserve good signal and thus maintain a better Signal To Noise Ratio. It also has the effect that star borders are less harmed due to rejection when the stars have different sizes/shapes in your images.

So in this regard, I would recommend winsor sigma clipping in combination with diffraction protection over regular sigma clipping.

 


Finally, we have a discrimination between MAD and non-MAD fitlers.

MAD = Median Absolute Deviation

MAD is a robust way to calculate the standard deviation in the pixel stacks and is more robust than the regular calculation of the standard deviation (root of the variance).

MAD tends to work bettter than non-MAD filters with a small amount of frames. I would say with less than 100 frames. If you have more than 100 frames, the difference will be very minimal if any.

If you use a MAD filter, you must be carefull with the outlier rejection settings, especially the kappa value. If you set this lower than 3, you will start to reject a lot of data and you run the risk of rejection good data as well. Always check the amount of rejection by enabling rejection maps in integration.

So in this regard, I would recommend MAD rejection in combination with a rejection map for verification over non-MAD clipping when you have less than 100 frames. But keep in mind, you need to be carefull with the kappa value, don't set it much lower than 3 😉

 


 

So overall, I would recommend the following:

  • always use LN rejection, the latest version of APP is very advanced as descrived in release notes of APP 1.073
  • Use winsor clipping in combination with diffraction protection if needed to prevent artefacts
  • With less than 100 frames, use MAD but don't set the kappa to low.

 


 

In general, always be carefull with outlier rejection, setting the kappa too low and the amount of iteration too high, will always be very harmfull for your data. Yes, all outliers will be gone with aggressive settings, so will be rejected part of the good data . You can reduce the Signal to Noise ratio by 10% to even 50% procent if you are not carefull !

kappa -> lower kappa is more aggressive, a value lower than 2,5 is already very aggresive !

iterations -> I rarely use more than 2 iterations, and in most cases, 1 iteration is all you need. 3 iterations is already very aggressive !

 

So using more than 3 iterations with kappa 2 should never be used, it will be very bad  for your Signal to Noise ratio of your integration.

 


 

Let me know if this has explained some of your questions and if you have any others.

Kind regards,

Mabula

This post was modified 5 months ago 4 times by Mabula Haverkamp - Admin

Main developer of Astro Pixel Processor and owner of Aries Productions


(@wvreeven)
Red Giant Customer
Joined: 11 months ago
Posts: 51
04/04/2019 2:35 pm  

Thanks very much fro the detailed explanation @mabula-admin! Still, one question of the topic poster has not been answered: what is the best outlier rejection in 2) Calibration? In general I do not ever touch the settings in that screen and I get very good results (I think!!!). So if you have advise on how to make APP do a better job then we are all ears 🙂

 

CS, Wouter


Fredmt liked
ReplyQuote
(@fredmt)
Brown Dwarf Customer
Joined: 7 months ago
Posts: 8
04/04/2019 6:06 pm  

Thank you very much for this very complete answer @mabula-admin.

But as mentionned by @wvreeven, what about the 2) Calibrate window ?

I usually use outlier rejection on all my calibration frames (Flat, DarkFlat, Dark and Bias) and I also get great results. Do I really need to ? (I've seen some people as @wvreeven not using them at all.  Some people do though). Is there a criteria to use them or not to use them ?

 

Thanks again

Fred


wvreeven liked
ReplyQuote
(@vincent-mod)
Galaxy Moderator
Joined: 2 years ago
Posts: 580
12/04/2019 9:45 am  

On darks and bias you don't want to use outlier rejection as those frames are used to actually calculate the noise and need all of it without rejection. Flats do need to be calibrated with bias rejection.


ReplyQuote
(@fredmt)
Brown Dwarf Customer
Joined: 7 months ago
Posts: 8
13/04/2019 7:21 pm  

Ok, Thanks @vincent-mod for your answer.

Why, then, are these settings available ? Is there a case where it's better to use them ?

I saw videos (even on the APP channel on Vimeo) where these settings are used.

I'm just trying to understand how all these things work to get the best out this great software.

 

Fred


ReplyQuote
(@annehouw)
Main Sequence Star Customer
Joined: 2 years ago
Posts: 31
14/04/2019 5:13 pm  

Hi Fred,

In my view, the main use of (a very mild!) form of outlier rejection for your calibration masters would be to eliminate cosmic ray hits. These are not a property of your sensor, but random hits. These will not be corrected by a bad pixel map (because a BPM is a "description" of the bad behaviour of your sensor, but can not correct random hits). The longer you expose your dark frames, the more chance for a hit during your exposure. So, using rejection filters can be benificial for your darks. Will it always make your masterdarks  better? That depends. Cosmic ray hits are random, so you might not have any and you certainly will not have many. The main thing is to keep the rejection filter very mild, otherwise you filter out the randomness of the thermal dark signal and an average of that is what you want to have in your Masterdark. 

 

Regards,

Anne

This post was modified 4 months ago by Annehouw

ReplyQuote
(@vincent-mod)
Galaxy Moderator
Joined: 2 years ago
Posts: 580
16/04/2019 10:45 am  

I think you might just try it to see if you indeed have cosmic ray hits and see if they pose a problem in a BPM for instance. I never really saw this mentioned as being a potential problem either. Rejection should be kept at a minimum or even off in most cases for bias and darks, it depends a bit on the sensor characteristics I guess too. Not all data is always the same unfortunately so to have more options available is a nice thing, though can be confusing.


Fredmt liked
ReplyQuote
(@fredmt)
Brown Dwarf Customer
Joined: 7 months ago
Posts: 8
16/04/2019 10:15 pm  

Hello @annehouw and @vincent-mod. Thank you for your answers.

I've tired already a few times these settings. I'll do more tests in the next few days to see what works best for me.

 

Thanks again

Fred

This post was modified 4 months ago 3 times by Fredmt

ReplyQuote
Share: