[Sticky] Data calibration principles/rules - must read !

Posted by: Rowland F Archer Jr

Hi Mabula,

I think adding that to the FITS header would certainly not hurt - and might even save you some support questions 🙂 !

The process of calibrating with FlatDarks is different enough from the process documented all over the Internet that maybe a short video tutorial would be useful too, with so many people using the ASI1600 and similar CMOS cameras that benefit from the FlatDark / no bias processing.  Especially with image acquisition products like ACP, Starkeeper.it Voyager, and CCD Commander among others that change the exposure length on the fly to reach a desired ADU for flats.  

I think I'm finally getting it straight!

Thanks,

Rowland

 

Hi Rowland @rowland-f-archer-jr,

I will add the information to the masters and integrations 😉 a.s.a.p.

I have done a big rebuild of the calibration engine last year, to make this very easy, at least, I think... 

Technically speaking, calibration of flats with flatdarks is not different than calibration of lights with darks, right? The same rules apply.

APP will automatically calibrate your flats with bias and/or flat darks if these are provided, since without them, flat calibration can't work because the bias pedestal always needs to be subtracted to get proper flat-field calibration. APP makes sure that this happens. If bias or flatdarks aren't there, the user is ever warned, that he will probably face a problem. And then there is always the console, you can see exactly what is happening with your flats or lights in the console:

15:31:07 - IMAGE VIEWER: loading into Linear Calibrated Image Loader: C:\Users\Aries Productions\Documents\AllTypesOfImages\astroShots\CocoonNebula Lights\Lights\DSC_0026.NEF
15:31:07 - 2) CALIBRATE: MATCH CALIBRATION MASTERS: found calibration masters for light frame: DSC_0026.NEF
15:31:07 - 2) CALIBRATE: frame: DSC_0026.NEF MasterDark :DSC_0050.NEF
15:31:07 -
15:31:07 - FRAME DETAILS UPDATER: starting...
15:31:07 - FRAME DETAILS UPDATER: no new frames to add
15:31:07 - FRAME DETAILS UPDATER: rebuilding all frame details...
15:31:07 - FRAME DETAILS UPDATER: checking if frames were identified earlier...
15:31:07 - FRAME DETAILS UPDATER: adding frame marks...
15:31:07 - FRAME DETAILS UPDATER: updated succesfully
15:31:07 - 2) CALIBRATE: trying to find calibration details of light frame: C:\Users\Aries Productions\Documents\AllTypesOfImages\astroShots\CocoonNebula Lights\Lights\DSC_0026.NEF
15:31:07 - 2) CALIBRATE: found calibration masters for light frame: C:\Users\Aries Productions\Documents\AllTypesOfImages\astroShots\CocoonNebula Lights\Lights\DSC_0026.NEF
15:31:07 - 2) CALIBRATE: Master Dark : C:\Users\Aries Productions\Documents\AllTypesOfImages\astroShots\Cocoon Darks\DSC_0050.NEF
15:31:07 -
15:31:07 - GENERAL IMAGE LOADER: loading frame C:\Users\Aries Productions\Documents\AllTypesOfImages\astroShots\CocoonNebula Lights\Lights\DSC_0026.NEF
15:31:08 - 2) CALIBRATE: Adaptive Data Pedestal: disabled
15:31:08 - GENERAL IMAGE LOADER: loading frame C:\Users\Aries Productions\Documents\AllTypesOfImages\astroShots\Cocoon Darks\DSC_0050.NEF
15:31:10 - GENERAL IMAGE LOADER: frame C:\Users\Aries Productions\Documents\AllTypesOfImages\astroShots\Cocoon Darks\DSC_0050.NEF was loaded successfully
15:31:10 - 2) CALIBRATE: analysing MasterDark...
15:31:10 - DATA ANALYSER TOOLS: re-instantiated multi-core analytical memory blocks, size 511 MBs
15:31:13 - 2) CALIBRATE: MasterDark analysed
15:31:15 - PREVIEW FILTER: filtering new frame:DSC_0026.NEF
15:31:15 - PREVIEW FILTER: creating clone of frame...
15:31:15 - GENERAL IMAGE LOADER: frame C:\Users\Aries Productions\Documents\AllTypesOfImages\astroShots\CocoonNebula Lights\Lights\DSC_0026.NEF was loaded successfully

Simply said, if you load either bias and/or flat darks, these will be applied to your flats as integrated masters and this is done completely automatically. The option to scale the MasterFlatDark is optional as well.

If you think that a video tutorial in this regard is required/warrented to take away some confusion, we can off course make one a.s.a.p. Perhaps you can send me a nice dataset so I can create one?

Kind regards,

Mabula

Posted by: Heno

Hi Mabula

I have read this thread a few times trying to get all into my head. I made myself a step by step procedure, but I would like to run it with you so see if I have got it all. For ASI 1600 MM-C it goes like this:

1. Take flat frames for each filter.
2. Take dark flats.(Match flats time, bias, offset, (and temperature ?)).
3. Take Bias frames, 0,1 sek (?), 400 of.

4. Generate Master flats using 1, 2 and 3, one for each filter.

5. Take lights
6. Take darks (Match lights time, bias, offset and temperature)
7. Create Master darks as required by 6.

8. Calibrate lights using 4, 5 and 7.

I'm not sure how much temperature matters for flats, dark flats and bias frames, but keeping it the same cannot hurt.

I read on Cloudy Nights that bias frame length should be 0,2 sec for ASI 1600. (Jon Rista). And then, if your flats and then your dark flats are of similar length you could use the dark flat as bias frames also. I may have misunderstood the last bit.

Comments to this please?

Helge

Edit: A bad piksel map could be useful, but should it be used with both flat/dark flat process and lights calibration?

Hi Helge @heno,

Yes, it's all good 😉

Temperature is not a big concern for the flats, flat darks and bias. Simply make sure you don't make them in completely different circumstances 😉

Keep in mind that the exposure time of bias frames is so short, that temperature normally does not enter the equation. Temperature has to do with the dark signal. So if you create darks of more than 1 second, temperature starts to play a rol due to build up of dark current/signal.

These cmos sensors like the asi1600 really make bad bias frames if you use the shortest exposure time possible, as you know, therefore, to get proper results you need to make bias frames with slightly longer exposure times (0,1 second-0,5 seconds) to get proper bias signal in your bias frames. If you would expose these longer than 1 second, dark current and temperature starts to influence things...

And then, if your flats and then your dark flats are of similar length you could use the dark flat as bias frames also. I may have misunderstood the last bit.

if you only use dark flats, then it would not matter how you load them into the application, if you load them as bias or flatdarks it will not make a difference. Both will be subtracted from the flats 😉

A bad piksel map could be useful, but should it be used with both flat/dark flat process and lights calibration?

There is no reason not to use a bad pixel map, whatever calibration workflow you use, a bad pixel map will never hurt your data. It will never inject noise 😉 I use it on both lights and flats all the time.

Let me know if you further questions 😉

Mabula

Mabula

Thank you for confirming my understanding of the process. I will use my 0,5 sek dark flats also as bias frames, i.e. no need to create separate ones. I will need to cover my flats light panel to shoot flats in the same range, but that's no problem. 

Thanks again for your reply.

Now I will sit down and make a "process for dummies" (that's me 😀) sheet, so I don't forget it all once the season is over.

Posted by: Mabula Haverkamp - Admin

These cmos sensors like the asi1600 really make bad bias frames if you use the shortest exposure time possible, as you know, therefore, to get proper results you need to make bias frames with slightly longer exposure times (0,1 second-0,5 seconds) to get proper bias signal in your bias frames. If you would expose these longer than 1 second, dark current and temperature starts to influence things...

Hi Mabula @mabula-admin

What do you think about bias frames from DSLRs CMOS sensor with time as short as it's possible, e.g. FujiFilm X-Trans with 1/4000s or Canon at 1/8000s? Or should I make Bias frames with longer exposure times, e.g. 0,1s?

Mariusz

It depends on the camera, apparently the ASI mentioned above has an issue to produce decent bias signal. With DSLR's my experience has been fine setting that to the shortest possible setting. Normally that is also what you want as it's all about the read-out noise of the sensor. That should be a fixed, random signal just from reading it out and having longer exposures can introduce other noise which isn't part of the bias signal.

Hi @mariusz and @vincent-mod,

From what I have seen from cmos sensors in general is that shooting bias frames with the shortest exposure time possible can give significant problems. I would advice not to use the shortest exposure time possible, but work on the safe side by using bias frames with exposure times between 1/100th - 1/10th of a second. That is still very short, so any dark signal will be very small. I think that is the safest and most robust way to shoot bias frames with a camera with a cmos sensor. So I would advice this for both astronomical cmos camera's as well as for DSLR cmos camera's 😉

To add to this, apparently, even some ccd sensors fail to produce a good bias signal with the shortest exposure time possible. I have seen very strange bias frames of both Atik and QHY CCD camera's with exposure times of 1/1000-1/4000th of a second... Calibration was greatly improved when bias frames were used of 1/10th of a second in those cases. I must add, that in these cases, always an old CCD camera was involved (age > 5 year), so I suspect strong degradation of the sensor and perhaps electronics surrounding the sensor to play a role here...

It's also very easy to test: simply shoot some frames with the shortest exposure time possible, then some frames with 1/100th - 1/10th of a second and shoot some darks of, let's say, 1 minute, so the dark current will a have significant contribution to the frame.

Then, study these frames with the same stretch parameters on a very strong stretch. If the shortest exposed frames look the same as the others, then, very likely, the frames can be used properly. If something is not right with the shortest exposed bias frames, you should see this with a strong stretch.

Cheers,

Mabula

Cheers,

Mabula

Save calibrate, register, normalise files or not ? For... OR  only time and space consuming ?

For cooled CMOS cameras, is there any reason to use a bias frame at all?

If we assume that one has a dark library of  (fully) matched darks, and that flat darks are taken to match flats, does a master bias make any difference at all?

-Jim

It's best to simply try it out, you can check the noise levels afterwards. I think it's still useful as a bias signal will always be there, but how much it contributes depends on the sensor. Some sensors do seem to like bias frames which aren't taken at the fastest speed.

I guess my question is more of a theoretical one. Is there any advantage in theory to separately calibrating using a bias and a bias-subtracted master dark, over simply calibrating with a  masterdark that has not had it's bias removed?

Trying it shows no difference in noise level, so I'm left wondering why the bias frame is a thing, and specifically why it's called out above as being important. 

Ah, like that. No there is no difference if you have the darks with bias added. I do have to note here that I never tried this approach in APP.

Hi

I am new in processing by APP. I have read this thread and have done an abstract for me.

Can somebody check my layout in the attachment and shows me possible errors?

Thanks a lot

Ed

Hi Ed,

Welcome to the forum. In principles your layout looks ok but it is a bit hard to comment on it without info about the telescope you use and the sky conditions of your imaging site. This is especially true for the number of Light images and their exposure times. Perhaps the tips that I wrote down in this thread may be of help to you:

https://www.astropixelprocessor.com/community/main-forum/newbie-lrgb-processing/

To answer the two questions in your layout:

1. FlatDarks can be shot inside since they do not require any light coming through the telescope, just as with Darks.
2. Bias images shot at the same temperature as Lights, Darks and FlatDarks cannot be wrong

Wouter

Hi Wouter

Thanks for the answer. I know that the exposure time and the sky conditions play an important role.

My Scope is a Pentax 75SDHF (FL 500mm) and the sky condition are Bortle 4 and 20.6 mag/arsec2.

The number of the images with 15 is a minimum and can or should be greater depending of the sky pollution.

Thanks for the newest thread about lrgb processing, i will read it.

Ed

16 bit masters or 32 bit masters? I could not find the reason for having the 32bit master option so i did a test. I made a set of masters (based on 300 bias, 250 darks and 25 flats) with 16 bit and a set with 32 bit and made 2 integrations with 75 luminance subs, 1 with 16bit and 1 with 32 bit masters. The luminance lights used are 16 bit fit files. The result is that there is no difference in the end result, visually or in noise level / SNR. So i still do not know why there is the 32 bit master option...........

So I asked Mabula about it and indeed, it doesn't change anything in the calibration phase. It seems more of a compatibility option as other software packages have 32-bit options as well.. even though it shouldn't matter there either.

@mabula-admin

Hello Mabula,

I'm new in astrophotography and in this forum as well.

But i purchased APP because i like want i read about you and your product; so thank you for the work you have done.

I'm waiting for my telescope and i'm approaching theoretically to the things that will come; and one of the important things i want to understand is what i have to do when i'm out in the field and making photos.

And it helps a lot to read the things you are describing in this post; very good explanations.

But now i'm wondering a little bit about your statements:

"I know that quite a number of astrophotographers are struggling with APP's calibration engine, so I know I need to improve APP in this department, which I will do with high priority" and

"As I said, an upcoming upgrade of APP will remedy these strict rules to make the process much easier, I know this is needed "

What does that mean for the version i purchased (1.075)?

Are the things different in this version; or which version are you talking about?

I hope you understand what i want to ask you because my english is not so good; sorry.

with kind regards 

Didi_sangye

Hi Didi,

Thanks for purchasing APP! So, even though the calibration process in APP is pretty straightforward, some situations can cause some confusion still. In general it'll be fine, some cases you need to know a bit more to make it work nicely. These situations is what Mabula wants to address and make even more automatic.

Hi,

Still learning about all this, but still somewhat confused. Can I confirm whether the following is ok please:

Session 1

Lights taken at ISO800, 60 sec exposures
Darks taken at ISO800, 60 sec exposures
Bias taken at ISO800, 1/4000 sec exposures

Flats taken at another time - ISO800, about 1 sec exposures (using flats wizard in NINA)

If I load all these up in the load tab, leave everything as default and basically click on 'Integrate' in the integrate tab then appropriate master Dark, master Bias and master Flat and bad pixel map get generated automatically as part of APPs processing??

If so, can the resulting masters be used for subsequent sessions as long as the temperature during those sessions is roughly the same? (I have a DSLR so cooling not an option). ie I would have no need to take darks and bias frames in those subsequent sessions??

Thanks
Ed

That seems ok yes (flats don't have to be at the same ISO) and masters would be automatically created. The resulting masters can be used for other sessions, both bias and darks are fine for that. I would advise to take flats after each session though. You can create a BPM first with a few nice, long darks (like 300s) and that you can use for every session and a long time. If you don't have any amp-glow, you could just use the BPM for hot pixel correction. Otherwise you still need the darks. I usually create a dark library, when it's cloudy for instance, making darks for all exposures I want. And I create a new one when seasons change as indeed a DSLR is not cooled.

Thanks Vincent, glad I understand it better than I thought! Can I ask why you would recommend flats after every session though? That's a new one on me and would like to understand the reasons why. 

Thanks

As far as i understood flats should fit to the camera position and the focus and in my case that changes from session to session; so i‘m taking flats after every session with a self-made flat-box (a3 light pad about 70€)

so for me another question comes up: how can i check the quality of the flats because it is sayed that one can reach much better result with a more expensive el-foil (special flat-box about 250€ for my 10“ newton); so how to check the result of the flats?

thanks

You don't necessarily need expensive hardware to do this. What comes out is what matters, sky flats can work fine as well, but it needs to be a good flat in the end. A good flat is exposed to about half of the linear range... histogram of the sensor (e.g. an ADU of 30.000 with a 16-bit representation of the histogram, something Sequence Generator Pro does for any sensor which I like). The light source for the flat should be as even as possible, when taking sky-flats, you don't want to expose it to a patchy cloudy sky as these produce variations in light across the image. So it has to be fully clouded, evenly or clear). A flat is there to only correct for the light fall-off effect of your scope, other effects should be taken out as these won't represent the fall-off of the scope during normal imaging. Another important bit is to have the focus to be exactly the same as when you took the lights, this is because any dust can be corrected for as well, but those need to be exactly the same in both lights and flats, so the exact same focus.

And when you have rotation going on for your camera and filters, you also need flats in those positions indeed.

Posted by: @edarter

Thanks Vincent, glad I understand it better than I thought! Can I ask why you would recommend flats after every session though? That's a new one on me and would like to understand the reasons why. 

 

Thanks

Yes, it's not required, especially not when your setup is always standing in an observatorium for instance, but even then I would propably recommend it. When you have the workflow for good flats, it becomes easier and I once didn't take new flats as I didn't feel like doing it and a new little dust speckle ruined all my frames of that night. So it's more because of the dust possibility to do that just in case.

Hi,

sorry, if this was addressed already.

If I want to integrate some lights, I add BPM, Master-Dark and Master-Flat. Do I include the Master-darkflat I got from flats calibration? Or is this already subtracted from the Master-flat during calib?

Cheers,

Chris

No need for a Master DarkFlat when using MasterFlats. You only need to use a Master DarkFlat when using unprocessed Flat frames so a Master Flat can be created.

HTH, Wouter

@wvreeven Thanks!

I have been reading through the first part of this thread and I wonder if it should be rewritten to match the current state of APP plus all the good advice in the thread. I seem to understand that there is a difference between CCD sensors and CMOS sensors and then also a difference between CMOS cameras that do and CMOS cameras that do not produce amp glow. And then there are DSLR cameras that seem to have their own rules altogether.

In my case, I now own an ASI1600MM-C Pro and an ASI6200. Both cameras have CMOS sensors and I do not use bias frames anymore at all. Only dark flats, flats, darks and lights.

For the ASI1600MM-C Pro I use a dark library with master darks for several exposure times (30, 40, 60, 90, 120, etc seconds) to match with the LRGBH lights of the same exposure times. For the ASI6200 (which doesn’t have amp glow) I only use one master dark for 60 seconds exposure time and that seems to work veey well for the lights of various exposure times that I take with the LRGBHOS filters that I use with the camera.

So far I seem to get good results in the sense that my images seem to represent well the objects that I wanted to image. No strange artifacts and post-processing gives decent colors and such. However, I am still not 100% sure that I am doing this in the right way. And that’s just the two cameras I use. I understand that writing comprehensive instructions that cover all cases is very difficult at best and I feel that this thread probably contains most of the relevant bits but it is quite hard to dig them out. So perhaps a rewrite summarizing all the essential/relevant info would create a great reference for most, if not all, of us.

Wouter

Yes, totally true Wouter and these things will likely be described very detailed in the (still) upcoming manual. Because there are so many different sensors, a general rule is quite challenging to write and will also require the user to experiment like you do. I'd say that in general CMOS users can still use Bias, but to be sure it's best to get the short exposures up to something like 0.05 seconds or even using dark flats. These are still usefull for calibrating flats I assume. The bias signal is ofcourse also present in darks so for that you can use them for lights without using specific bias frames.

CCD en DSLR camera's are usually fine with bias frames that are taken as fast as possible, upping the times there can however also be wise just because it would make this "rule" more general.

From reading through this 5 page thread, it appears the program has steadily improved to the point that all you need to do is input the required L, D, F, and DF frames and APP will pretty much do the rest. BPM's will be generated based on the supplied data, and Bias Frames really need not be taken.

Am I reading this correctly?