Lone Exposure Noise Reduction

Hi Brian,

So you never should use the built-in camera features, they are not really designed for astrophotography and not very good either. You save the raw data from the camera (.raw) which has no processing done to it. You create darks, bias and flats for your camera and use those in APP and other software to calibrate the raw data.

@vincent-mod it's possible I don't understand how this works in my camera, but my understanding is that it gives me dark subtracted raw images, which in theory should be the highest quality for a camera with no sensor temperature control.  There is a separate noise reduction setting which does process the image, I have that shut off.

If I have a few mights of pre-dark subtracted raw images, how would I process those with a flat in APP?  Are there options for that?

@eebclee For processing of the flats, darks with the same exposure time are needed as well. Do you have those? If yes, load the flats and the darks and create a master flat. Then load only the lights and the master flat, go to tab 6 and click the Integrate button. Let us know if APP complains about anything and we will try to help you solve that. 

@wvreeven I haven't tried that approach yet, I'll give it a try.

@eebclee Let me add that the reason why you should not load the lights, darks and flats at the same time, is that in that case APP will try to apply the darks to the lights as well, which you want to avoid. So first create the master flat and then apply the master flat to the lights.

Edit: got it, I'd loaded the darks as "Dark" instead of "DarkFlat"

@wvreeven I just took flats and darks at the same exposure and tried "calibrate" and I get this warning:

@eebclee The ISO value was the same as well, correct? Was the dark scaling disabled in the camera?

Adding to this, are you now using the raw data files?

@vincent-mod I’ve always been using the raw files. All same ISO, and for the flats there’s no dark subtraction in camera — it will only do that for exposures over 1sec, the flats and darks for the flats I took yesterday are 1/500th sec.

I haven’t stacked yet but it looks like things are going fine this time, creating a master flat first.  I’ll update when it’s done.

@eebclee Hi Brian, with a DSLR camera you should aim at flats of at least 1 second. The reason is the shutter which takes a relatively large time to open and close when compared to 1/500 sec. That may cause an unevenly lit flat which introduces all kinds of nasty gradients.

@wvreeven I'll try some new longer flats and flat darks then.

My latest attempt with the short flats finished just as I got this message, and looks a lot like what I got when I used longer flats (w/o separate darks) before.  If I do the same stacking with no flats things look much better.  I am using an L-Pro light solution filter and custom white balance, do I need to do something other than select camera white balance in tab (0) for that to work?

Iterations take me a while -- there are 200-some frames, start to finish appears to take >12 hours on my iMac Pro.  The end results (when I use no refs) look better than other programs I've tried, but something like DSS takes about 1/10th the time.  Is this expected or do I likely have some setting wrong?  I've changed very little from the defaults.

Thanks for all your help.

I've tried this a few times and I'm still getting strange results.  With LENR off I collected 8 second flats and darks, and created master flats in APP (both using the camera white balance and not).  Then I loaded only that master flat and my LENR lights and tried processing.  Camera white balance in both cases results in very strange colors.  Camera white balance off gets rid of the strange colors but still isn't flat fielding.

Camera white balance:

 Camera white balance off:

Edit: even with no dark, when I stack all 400-some light frames and choose camera white balance I get this kind of strange pink to turquoise result.  I'm using an L-Pro light pollution filter and custom white balance.

Hi Brian @eebclee,

@vincent-mod it's possible I don't understand how this works in my camera, but my understanding is that it gives me dark subtracted raw images, which in theory should be the highest quality for a camera with no sensor temperature control. 

Well, that is not entirely correct... allthough the camera makers will tell you differently normally.

Actually and this might surprise many, but the in-camera dark subtraction is for sure the worst dark subtracted data that you will ever get 😑. Let me explain what happens. The in-camera dark that is made after the regular exposure is just 1 dark frame. That single dark frame is subtracted from the light frame in camera, right? Now 1 single dark frame also has variable noise. That noise is injected in your light frame, leading to a more noisy light frame. Yes, the bad/hot pixels are gone, giving the impression that there is less noise, but that is fixed pattern noise, not the gaussian/normally distributed noise from the dark current of your camera's sensor which you want to keep in check for dark subtraction.

If you create darks yourself, you would create 10,20, maybe 50 darks. The masterdark created from 10 darks is already square root of 10 = 3,2x better for subtraction than that in-camera dark that is subtracted. This is because the variable/gaussian noise will drop with the square root of the number of darks. That means that such a masterdark will inject far less noise than an in-camera dark. If you create a masterdark of only 1 dark, you inject so much noise, it really is a waste of your light frame exposure to be honest ;-). So in reality, the in-camera single dark frame subtraction gives you the worst dark subtracted data possible... It is definitely not of the highest quality, but rather the opposite I would argue.

In addition, the in-camera dark subtract operation can behave unexpected/odd for different camera models and brands including further not advertised noise reduction algortihms, leading to non-ideal or in the worst case incompatibel data for further calibration steps.

So it is really hard for me to advise on how to calibrate in-camera dark subtracted data for flat-field correction, because I will never know what exactly happened to the pure RAW exposure and thus I have no clue how to correctly solve that.

I can only advise to never use in-camera dark subtraction if you want to get good or preferably optimally calibrated data 😉

Let me know if this is clear and if you have any additional questions about all of this. It is a complicated problem and like Vincent argued earlier, that in-camera dark subtraction is something that is usefull only for regular photographers. For astrophotography, I would argue it is a no-go, it really damages your exposure actually.

Mabula

Hi Mabula,

The flip side of the is that with CMOS sensors, unlike CCDs, the noise is non-Gaussian and some noise sources are highly time dependent.  Depending on the sensor, short darks near the exposure can be much better than long darks separated by more time.  For a camera I'm working on right now, the best data is taken in CDS mode which takes this to the extreme, each pixel is read out at both the beginning and end of the exposure to get around reset noise.  For the Canon, I don't know.  I did hope to be able to do the comparison myself.  If you don't know exactly what the camera is doing in LENR though, if it's anything more than simple dark subtraction, I can understand that even figuring out the correct bias level could be troublesome.

Do you have any idea what might be going wrong with stacking when I use a custom white balance for the light pollution filter?

Thanks,

Brian

Posted by: @eebclee

Hi Mabula,

The flip side of the is that with CMOS sensors, unlike CCDs, the noise is non-Gaussian and some noise sources are highly time dependent.  Depending on the sensor, short darks near the exposure can be much better than long darks separated by more time.  For a camera I'm working on right now, the best data is taken in CDS mode which takes this to the extreme, each pixel is read out at both the beginning and end of the exposure to get around reset noise.  For the Canon, I don't know.  I did hope to be able to do the comparison myself.  If you don't know exactly what the camera is doing in LENR though, if it's anything more than simple dark subtraction, I can understand that even figuring out the correct bias level could be troublesome.

Do you have any idea what might be going wrong with stacking when I use a custom white balance for the light pollution filter?

Thanks,

Brian

 

 

Dear Brian,

Well, even for CCD camera's the noise sources can be strongly non-linear (in time) like severe amp-glow. CMOS is not much worse these days I would argue. CCD's do have read noise for the whole sensor or parts of the sensor depending on the electronics where CMOS has read noise per pixel.

But... as long as you create calibration frames that match the lights and flats for exposure, gain/iso, temperature all these sensors can be perfectly calibrated and really much ! better than with camera LENR, both in the real world as in theory 😉

"Do you have any idea what might be going wrong with stacking when I use a custom white balance for the light pollution filter?"

No not really, what are you doing exactly? With which camera? Do you mean custom whitebalance by setting it with the RGB sliders in APP?

Mabula

@mabula-admin

It's a Canon 80D with a light pollution filter.  I'm using a custom in camera white balance for that.  If I stack those frames (even with no references) in APP and check on the option to use the camera white balance, I get these very strange color results.  If I don't check the camera white balance, it looks ok although with someone muted colors.