Masterfiles overcorrect (again)

Hello Chris,

Can you put a sample of your files (say 5 lights, 5 dark, 5 flat, 5 darkflat, 5 bias) on dropbox, so I (and others) can have a look at your data?

Bias (yes..) I never use bias at all when I have darks (and flatdarks) of the same duration and temperature as the light counterparts. 

Furthermore, there could be non-linearities in the flat frames because of a too high or low light intensity, but difficult to say without being able to see and measure the flat frames...

Regards,

Anne

Hi, 

I have the same camera and have had the same problems. So I took a new set of darks, flats and bias files, and my overcorrection disappeared. Some setting must have been wrong, I guess. I now have a very colorful background, but that may be just another mistery of astronomical image processing (or APP😉).

Do you use both bias and darkflats? I thought you could either use one or the other.

And, please,could you say a bit more about ASI1600MCC not dealing well with very short flats? Mine are taken at 0.0004s, if I remember correctly.

Regards

Stefan

Hello Stefan,

I try to keep flat exposures between 1 and 3 seconds. There are a few reasons why:

- Very short flats can be noisy if the signal level is low. Ideal is somewhere around 1/2 of max ADU. Using these flats will introduce this noise in all your subs. You want your flats to be as noise free as possible.

- The sensor is not read out all at once (unless you have a camera with a global shutter). Using ultra short exposures can introduce the patter of that read sequence in the image. If this is the case with your camera, I do not know.

- I use a led light box with a dimmer. LEDs are normally dimmed on a frequency basis (so they flicker). As a result, I need to average that flicker. Hence the multiple second exposures for the flats.

- You can use either bias or darkflats (but not both, because you doe not want to subtract the bias pedestal twice). With short exposures (few seconds), it does not matter a lot in my experience if you use bias or darkflats. I want to be sure, so I take darkflats anyway.

Regards,

Anne

If I understand correctly then APP takes the dark flat RAWs and calibrates them with the master bias before creating the master dark flat. That way you still need a master bias when applying the dark flat and there is no risk of subtracting the bias pedestal twice.

Clear skies, Wouter

Hello Wouter

I guess you are right (see scenario 3 below), but I like to keep it simple.

If you have darks (or flatdarks) that are matched to your lights (or flats) with exposure length, temperature, gain, offset etc, in that case you do not need bias at all.

This is scenario 2 below.  So, my answer to foucault is: yes, you can use bias or darkflats, provided the darks are matched to the lights. With supershort flats, bias and darkflats are practically the same because the thermal signal is so low in that case, it does not matter which one you use.

In case of calibration issues like the OP mentioned, my way forward would be to keep it as simple as possible (no scaling of bias or darks) and inspect the calibration files, especially the flats...

Regards,

Anne

FLAT FRAMES

Flat frame calibration has several options, you need to choose 1 option from 1-3:

1. can and should be calibrated by a master bias of the same ISO or gain + offset as the flat frames
2. can and should be calibrated by a master dark of the same ISO or gain + offset and the same exposure length and temperature as the flat frames
3. can and should be calibrated by a master dark of the same ISO or gain + offset and the same exposure length and temperature as the flat frames, made of darks which were calibrated by a masterbias of of the same ISO or gain + offset as the dark frames  + the master bias of the same ISO or gain + offset as the dark frames. So in this case, you use 2 master calibration frames to perform flat frame calibration.

 4) Additionaly, Flat frames can always be calibrated by a Bad Pixel Map besides any of the above 3 methods, which I strongly recommend.

Hello Wouter

I guess you are right (see scenario 3 below), but I like to keep it simple.

If you have darks (or flatdarks) that are matched to your lights (or flats) with exposure length, temperature, gain, offset etc, in that case you do not need bias at all.

This is scenario 2 below.  So, my answer to foucault is: yes, you can use bias or darkflats, provided the darks are matched to the lights. With supershort flats, bias and darkflats are practically the same because the thermal signal is so low in that case, it does not matter which one you use.

In case of calibration issues like the OP mentioned, my way forward would be to keep it as simple as possible (no scaling of bias or darks) and inspect the calibration files, especially the flats...

Regards,

Anne

FLAT FRAMES

Flat frame calibration has several options, you need to choose 1 option from 1-3:

1. can and should be calibrated by a master bias of the same ISO or gain + offset as the flat frames
2. can and should be calibrated by a master dark of the same ISO or gain + offset and the same exposure length and temperature as the flat frames
3. can and should be calibrated by a master dark of the same ISO or gain + offset and the same exposure length and temperature as the flat frames, made of darks which were calibrated by a masterbias of of the same ISO or gain + offset as the dark frames  + the master bias of the same ISO or gain + offset as the dark frames. So in this case, you use 2 master calibration frames to perform flat frame calibration.

 4) Additionaly, Flat frames can always be calibrated by a Bad Pixel Map besides any of the above 3 methods, which I strongly recommend.

Hi,

I’d like to keep it simple, too. Would the following process make sense?

1.       Match the dark frame to the light frame with equal temperature, exposure time, gain an offset

2.       Match the bias frame to the flat frame with equal temperature, gain an offset

Where temperature, exposure  time, gain and offset in 2) can be different from temperature, exposure  time, gain and offset in 1)

Regards

Stefan

Hi Anne,
thanks for your reply. Here are my files:
https://faubox.rrze.uni-erlangen.de/dl/fi6tatp1oLo2hJBjCJNCv8dN/overflat.zip

Yes, I use bias AND flatdarks...
Have to check without bias on the big workstation at work (think I tested this already, but I am totally lost in testing and waiting for results).
Cheers,
Chris

Hi Stefan,

In case of doubt, it is always good to consult the sticky in the tutorials segment on the forum about the calibration rules. It is a bit long by now, but it contains all the principles of data calibration.

I can answer your question by a direct quote from that sticky: 😉 

Can I use a different ISO or gain+offset than my Light frames, for my flat frames?

Yes, that is no problem and actually preferred in most cases. Make your flats with a low ISO or gain+offset to be able to increase the exposure length of the flats. They will be of better quality then. 

Regards,

Anne

Hi Chris,

I did some experiments with your data. Results were a lot like your screenshot. Apparently an overcorrection of the flat frame. I tried several variations, but this is what came back.

Then I zoomed in and found that the BGGR CFA setting could not bethe right one. I did see a grid pattern in the data. Subsequently, I did some experimentation and found GRBG to give better results. No more grid pattern, just random noise.

I recreated all the calibration masters with this setting and did an integration. Below the result:

As you can see, the symmetry has gone. This looks to me more like complex gradients than a symmetrical overcorrection of flat. Here is the flat, it is symmetrical:

Never mind the red color, that is not relevant for the end result.

Finally, I did a real quick light pollution correction and got this:

Note that I generated all images on the highest BG DDP level and that it is only 5 subs.

I do not know what the CFA pattern is that you used, but this is what my experimentation turned up. I did not use bias in any of the integrations, since your darks and lights have the same integration times.

I hope this gives some extra food for thought. 

Anne

P.S: I found some discussions to avoid very short integration times with your camera (shorter than 1 sec). I have no opinion on this, but here is a thread describing it: https://stargazerslounge.com/topic/308419-flats-problem-or-something-else/

Hi,

Thanks for that advice. The rules are very helpful. Actually I've never been to the tutorial section, always googled for them.

And as for your experimentation with the CFA: at least my 1600MCC has a GRBG array.

Regards

Stefan

Hi Anne,
many thanks for your support!
Well, I forgot to tell the Bayer pattern, it is GRBG, sorry about that. Has to be forced in the zero-tab.
I found this stargazers thread also and I tried everything, the OP was adviced, like finding stray light and using UV/IR cutting filters (I also used the filter for darks). Well his odyssey resembles mine nearly exactly. Finally, as I understand, it was the problem of the camera... 😰 
The only thing I can imagine the problem might also come from is my flatbox. I use a Lacerta flatbox with LEDs which I upgraded with a dimmer in order to get these <0.2 sec exposure times. As you have seen, my debayered flats are red. I have redone flats (again) and they appear to be blue. Which confuses me even further.
So, let's see, if I continue to shoot astrophotos, if this kind of effect will stay with me. If so, I will contact ZWO about that.

Cheers,
Chris

Hi Chris,

Sorry I could not help more with this data. On the other hand, this is a very popular sensor, so fundamentally it should be a good instrument.

There is one thing that I did not find in your text: Did you try to make a flat with a long exposure time (say 1 to a few seconds)? 

You might want to try that.

If you cannot get longer exposure times by using the dimmer, use a white T-shirt or sheets of paper until you have at least more than 1 second to expose your histogram to about halfway*. Of course you also need to take darkflats with that same exposure. 

Try integration and see if this is better. If you want to be a bit more more methodical, you can use photoshop and load your "short" exposure flat and you "long" exposure flat as layers and use the subtract blend mode. The result should be uniform. If not, the sensor response is different at shorter exposure times. This could be on-chip processing for instance. 

That is all I can think of.

I hope you can resolve this issue and continue to shoot astrophotos!

Regards,

Anne

* According to this post on the ZWO forum the ADU in the flats for this camera should be around 15.000, not higher https://bbs.astronomy-imaging-camera.com/viewtopic.php?f=21&t=6912&p=12566&hilit=flat#p12566

@annehouw

Hi Anne,

just wanted to reply after half a year. I found, that, if you increase the LNC interations in Intergration to at least 3, the effect of overcorrection the flats gets minimized. Still there, but this could be easily treated by the remove light pollution function in Tools.

Thanks an clear skies!

Chris