Lone Exposure Noise...
 
Share:
Notifications
Clear all

19 June 2021: Our upload server https://upload.astropixelprocessor.com/ has been migrated successfully to our new office with higher upload and download speeds (nearly 10MByte/sec up/down ) ! We now have 1 general upload user called: upload with password: upload. The users upload1 - upload5 have been disabled.

31 May 2021: APP 1.083-beta2 has been released ! APP 1.083 stable will follow soon afterwards. It includes a completely new Star Reducer Tool, New File Saver Module, Improved Comet registration and much more, check the release notes here!

DOWNLOADS are available HERE!

 

Lone Exposure Noise Reduction  

  RSS

(@eebclee)
White Dwarf Customer
Joined: 2 months ago
Posts: 13
July 18, 2021 02:42  

I'm trying out APP and some other apps, my camera is a Canon 80D.  When taking images I use "Long Exposure Noise Reduction" meaning every light frame is dark subtracted in camera.  I'm not clear how I should them process these frames in APP (or most other programs I've tried so far!).  Last round I tried taking my flats in the same manner (~8-15 sec flats, 15 sec light exposures) in which case no dark or bias frames should be needed.  My final integrated image came out kind of turquoise with a halo of pink -- something went really wrong.

What settings should I use to process this already dark subtracted data?  If I use short exposure flats that do need dark and bias frames, does APP know not to apply them to the light images?  If there's an answer to all this that my search didn't turn up, please point me to it.  Thanks!

 

Brian


ReplyQuote
(@vincent-mod)
Universe Admin
Joined: 4 years ago
Posts: 3694
July 18, 2021 12:59  

Hi Brian,

So you never should use the built-in camera features, they are not really designed for astrophotography and not very good either. You save the raw data from the camera (.raw) which has no processing done to it. You create darks, bias and flats for your camera and use those in APP and other software to calibrate the raw data.


ReplyQuote
(@eebclee)
White Dwarf Customer
Joined: 2 months ago
Posts: 13
July 18, 2021 21:55  

@vincent-mod it's possible I don't understand how this works in my camera, but my understanding is that it gives me dark subtracted raw images, which in theory should be the highest quality for a camera with no sensor temperature control.  There is a separate noise reduction setting which does process the image, I have that shut off.

If I have a few mights of pre-dark subtracted raw images, how would I process those with a flat in APP?  Are there options for that?


ReplyQuote
(@wvreeven)
Quasar Admin
Joined: 3 years ago
Posts: 1110
July 18, 2021 22:37  

@eebclee For processing of the flats, darks with the same exposure time are needed as well. Do you have those? If yes, load the flats and the darks and create a master flat. Then load only the lights and the master flat, go to tab 6 and click the Integrate button. Let us know if APP complains about anything and we will try to help you solve that. 


Brian Lee liked
ReplyQuote
(@eebclee)
White Dwarf Customer
Joined: 2 months ago
Posts: 13
July 18, 2021 22:50  

@wvreeven I haven't tried that approach yet, I'll give it a try.


ReplyQuote
(@wvreeven)
Quasar Admin
Joined: 3 years ago
Posts: 1110
July 18, 2021 23:04  

@eebclee Let me add that the reason why you should not load the lights, darks and flats at the same time, is that in that case APP will try to apply the darks to the lights as well, which you want to avoid. So first create the master flat and then apply the master flat to the lights.


ReplyQuote
(@eebclee)
White Dwarf Customer
Joined: 2 months ago
Posts: 13
July 19, 2021 01:03  

Edit: got it, I'd loaded the darks as "Dark" instead of "DarkFlat"

@wvreeven I just took flats and darks at the same exposure and tried "calibrate" and I get this warning:

Screen Shot 2021 07 18 at 4.01.19 PM
This post was modified 2 months ago by Brian Lee

ReplyQuote
(@wvreeven)
Quasar Admin
Joined: 3 years ago
Posts: 1110
July 19, 2021 02:43  

@eebclee The ISO value was the same as well, correct? Was the dark scaling disabled in the camera?


ReplyQuote
(@vincent-mod)
Universe Admin
Joined: 4 years ago
Posts: 3694
July 19, 2021 09:09  

Adding to this, are you now using the raw data files?


ReplyQuote
(@eebclee)
White Dwarf Customer
Joined: 2 months ago
Posts: 13
July 19, 2021 18:23  

@vincent-mod I’ve always been using the raw files. All same ISO, and for the flats there’s no dark subtraction in camera — it will only do that for exposures over 1sec, the flats and darks for the flats I took yesterday are 1/500th sec.

I haven’t stacked yet but it looks like things are going fine this time, creating a master flat first.  I’ll update when it’s done.


ReplyQuote
(@wvreeven)
Quasar Admin
Joined: 3 years ago
Posts: 1110
July 19, 2021 22:18  

@eebclee Hi Brian, with a DSLR camera you should aim at flats of at least 1 second. The reason is the shutter which takes a relatively large time to open and close when compared to 1/500 sec. That may cause an unevenly lit flat which introduces all kinds of nasty gradients.


ReplyQuote
(@eebclee)
White Dwarf Customer
Joined: 2 months ago
Posts: 13
July 19, 2021 22:33  

@wvreeven I'll try some new longer flats and flat darks then.

My latest attempt with the short flats finished just as I got this message, and looks a lot like what I got when I used longer flats (w/o separate darks) before.  If I do the same stacking with no flats things look much better.  I am using an L-Pro light solution filter and custom white balance, do I need to do something other than select camera white balance in tab (0) for that to work?

Iterations take me a while -- there are 200-some frames, start to finish appears to take >12 hours on my iMac Pro.  The end results (when I use no refs) look better than other programs I've tried, but something like DSS takes about 1/10th the time.  Is this expected or do I likely have some setting wrong?  I've changed very little from the defaults.

Thanks for all your help.

M51 RGB session 1 St test

ReplyQuote
(@eebclee)
White Dwarf Customer
Joined: 2 months ago
Posts: 13
July 23, 2021 05:50  

I've tried this a few times and I'm still getting strange results.  With LENR off I collected 8 second flats and darks, and created master flats in APP (both using the camera white balance and not).  Then I loaded only that master flat and my LENR lights and tried processing.  Camera white balance in both cases results in very strange colors.  Camera white balance off gets rid of the strange colors but still isn't flat fielding.

Camera white balance:

M51 Test RGB session 1 St

 Camera white balance off:

M51 Test3 RGB session 1 St

Edit: even with no dark, when I stack all 400-some light frames and choose camera white balance I get this kind of strange pink to turquoise result.  I'm using an L-Pro light pollution filter and custom white balance.

 

This post was modified 2 months ago by Brian Lee

ReplyQuote
(@mabula-admin)
Universe Admin
Joined: 4 years ago
Posts: 2692
July 23, 2021 22:57  

Hi Brian @eebclee,

@vincent-mod it's possible I don't understand how this works in my camera, but my understanding is that it gives me dark subtracted raw images, which in theory should be the highest quality for a camera with no sensor temperature control. 

Well, that is not entirely correct... allthough the camera makers will tell you differently normally.

Actually and this might surprise many, but the in-camera dark subtraction is for sure the worst dark subtracted data that you will ever get 😑. Let me explain what happens. The in-camera dark that is made after the regular exposure is just 1 dark frame. That single dark frame is subtracted from the light frame in camera, right? Now 1 single dark frame also has variable noise. That noise is injected in your light frame, leading to a more noisy light frame. Yes, the bad/hot pixels are gone, giving the impression that there is less noise, but that is fixed pattern noise, not the gaussian/normally distributed noise from the dark current of your camera's sensor which you want to keep in check for dark subtraction.

If you create darks yourself, you would create 10,20, maybe 50 darks. The masterdark created from 10 darks is already square root of 10 = 3,2x better for subtraction than that in-camera dark that is subtracted. This is because the variable/gaussian noise will drop with the square root of the number of darks. That means that such a masterdark will inject far less noise than an in-camera dark. If you create a masterdark of only 1 dark, you inject so much noise, it really is a waste of your light frame exposure to be honest ;-). So in reality, the in-camera single dark frame subtraction gives you the worst dark subtracted data possible... It is definitely not of the highest quality, but rather the opposite I would argue.

In addition, the in-camera dark subtract operation can behave unexpected/odd for different camera models and brands including further not advertised noise reduction algortihms, leading to non-ideal or in the worst case incompatibel data for further calibration steps.

So it is really hard for me to advise on how to calibrate in-camera dark subtracted data for flat-field correction, because I will never know what exactly happened to the pure RAW exposure and thus I have no clue how to correctly solve that.

I can only advise to never use in-camera dark subtraction if you want to get good or preferably optimally calibrated data 😉

Let me know if this is clear and if you have any additional questions about all of this. It is a complicated problem and like Vincent argued earlier, that in-camera dark subtraction is something that is usefull only for regular photographers. For astrophotography, I would argue it is a no-go, it really damages your exposure actually.

Mabula

 


ReplyQuote
(@eebclee)
White Dwarf Customer
Joined: 2 months ago
Posts: 13
July 24, 2021 02:42  

Hi Mabula,

The flip side of the is that with CMOS sensors, unlike CCDs, the noise is non-Gaussian and some noise sources are highly time dependent.  Depending on the sensor, short darks near the exposure can be much better than long darks separated by more time.  For a camera I'm working on right now, the best data is taken in CDS mode which takes this to the extreme, each pixel is read out at both the beginning and end of the exposure to get around reset noise.  For the Canon, I don't know.  I did hope to be able to do the comparison myself.  If you don't know exactly what the camera is doing in LENR though, if it's anything more than simple dark subtraction, I can understand that even figuring out the correct bias level could be troublesome.

Do you have any idea what might be going wrong with stacking when I use a custom white balance for the light pollution filter?

Thanks,

Brian

 

 

This post was modified 2 months ago by Brian Lee

ReplyQuote
(@mabula-admin)
Universe Admin
Joined: 4 years ago
Posts: 2692
July 27, 2021 22:44  
Posted by: @eebclee

Hi Mabula,

The flip side of the is that with CMOS sensors, unlike CCDs, the noise is non-Gaussian and some noise sources are highly time dependent.  Depending on the sensor, short darks near the exposure can be much better than long darks separated by more time.  For a camera I'm working on right now, the best data is taken in CDS mode which takes this to the extreme, each pixel is read out at both the beginning and end of the exposure to get around reset noise.  For the Canon, I don't know.  I did hope to be able to do the comparison myself.  If you don't know exactly what the camera is doing in LENR though, if it's anything more than simple dark subtraction, I can understand that even figuring out the correct bias level could be troublesome.

Do you have any idea what might be going wrong with stacking when I use a custom white balance for the light pollution filter?

Thanks,

Brian

 

 

Dear Brian,

Well, even for CCD camera's the noise sources can be strongly non-linear (in time) like severe amp-glow. CMOS is not much worse these days I would argue. CCD's do have read noise for the whole sensor or parts of the sensor depending on the electronics where CMOS has read noise per pixel.

But... as long as you create calibration frames that match the lights and flats for exposure, gain/iso, temperature all these sensors can be perfectly calibrated and really much ! better than with camera LENR, both in the real world as in theory 😉

"Do you have any idea what might be going wrong with stacking when I use a custom white balance for the light pollution filter?"

No not really, what are you doing exactly? With which camera? Do you mean custom whitebalance by setting it with the RGB sliders in APP?

Mabula

 

 

 


ReplyQuote
(@eebclee)
White Dwarf Customer
Joined: 2 months ago
Posts: 13
August 5, 2021 23:06  

@mabula-admin

 

It's a Canon 80D with a light pollution filter.  I'm using a custom in camera white balance for that.  If I stack those frames (even with no references) in APP and check on the option to use the camera white balance, I get these very strange color results.  If I don't check the camera white balance, it looks ok although with someone muted colors.


ReplyQuote
Share: