Share:

We have updated the forum software 🙂 ! We are now working on upgrading the whole site and the Forum to a dark theme...  so please refresh your internet browser twice to properly update the forum appearance AND functionality... Check your internet browser for the refresh function, on windows, this usually is the F5 button, but it depends on your internet browser.

[Sticky] Data calibration principles/rules - must read !  

Page 1 / 4
  RSS

(@mabula-admin)
Quasar Admin
Joined: 2 years ago
Posts: 2071
June 5, 2017 22:12  

This post will explain the basic data calibration principles, which are needed to get accurate and good data calibration.

Data calibration plays an essential rol in data integration and further post-processing.

The quality of your images starts with collecting the data. After having acquired the data, the data processing starts with the data calibration. Don't proceed with processing until you have visually verified that data calibration is working correctly (using the l-calibrated image viewer mode), because it can really amount to substantial differences in the end quality of your images.

Astro Pixel Processor has strict calibration rules which when applied properly should give you optimal data calibration. If flat frame calibration is failing for instance, it's very likely you haven't followed the rules below.
 

BIAS FRAMES

Simple, for BIAS frames there are no rules, since they can’t be calibrated,. Bias frames show the read-out noise and patterns of your sensor. They are the most basic calibration frames and are always contained in all of your light, dark & flat frames.

what-is-the-bias-pedestal-and-why-does-it-matter?

 

DARK FRAMES

  • can and should only by calibrated by a master bias of the same ISO or gain + offset as the dark frames

 

FLAT FRAMES

Flat frame calibration has several options, you need to choose 1 option from 1-3:

  1. can and should be calibrated by a master bias of the same ISO or gain + offset as the flat frames
  2. can and should be calibrated by a master dark of the same ISO or gain + offset and the same exposure length and temperature as the flat frames
  3. can and should be calibrated by a master dark of the same ISO or gain + offset and the same exposure length and temperature as the flat frames, made of darks which were calibrated by a masterbias of of the same ISO or gain + offset as the dark frames  + the master bias of the same ISO or gain + offset as the dark frames. So in this case, you use 2 master calibration frames to perform flat frame calibration.

 

 4) Additionaly, Flat frames can always be calibrated by a Bad Pixel Map besides any of the above 3 methods, which I strongly recommend.

 

LIGHT FRAMES

We have several options, like the flat frame calibration, to get good data calibration, again choose 1 from 1-3:

  1. can and should be calibrated by a master bias of the same ISO or gain + offset as the light frames
  2. can and should be calibrated by a master dark of the same ISO or gain+offset and the same exposure length and temperature as the light frames
  3. can and should be calibrated by a master dark of the same ISO or gain + offset and the same exposure length and temperature as the light frames, made of darks which were calibrated by a masterbias of of the same ISO or gain + offset as the dark frames  + the master bias of the same ISO or gain + offset as the dark frames. So in this case, you use 2 master calibration frames to perform light frame calibration.

 

4) additionally , can and  should be calibrated by a master flat. There is no restriction on ISO or gain+offset of the master flat or it's exposure length. But the Master Flat needs to be calibrated following the rules for FLAT FRAMES.

 5) additionaly, LIGHT frames can always be calibrated by a Bad Pixel Map besides any of the above 4 methods, which I strongly recommend.

 

-----------------------------------------------------------------------

 

Further explanation about the strict calibration rules:

APP doesn't allow for the dark frames to be calibrated by a master bias of another ISO or gain + offset, because, if allowed, you will get subtoptimal calibration for the read noise patterns on your sensor. As a side note: other applications do allow this, so those applications simply allow sub optimal calibration.

APP doesn't have dark frame adaption to your flats or lights yet:

You should realize that dark frame adaption for temperature, exposure lenght and possibly ISO or gain is always less preferable to taking the time to create good matching dark frames. The reason being, dark frame adaption is very hard and error prone, since most sensors have signals in the dark current that don't behave linearly, like amp glow and hot/cold pixels for instance. That being said, dark frame adaption is on the RFC list to be implemented.

Can I use a different ISO or gain+offset than my Light frames, for my flat frames?

Yes, that is no problem and actually preferred in most cases. Make your flats with a low ISO or gain+offset to be able to increase the exposure length of the flats. They will be of better quality then. Be aware of the rules you need to follow for flat & light frames. Both need to have the bias pedestal subtracted only once using any of the available options. So if you only use bias frames and no darks. You need 2 master bias frames of the 2 different ISO or gain+offset value to make sure that the bias pedestal is removed correctly (only once) from both lights and flats.

Can I use master calibration frames created by another application?

Yes, you can, but it depends on the data type and it can cause problems due to different calibration algorithms and image format conversion routines. If you want, I can help and assist. The batch modify tool in 9) can do everything to get it working correcly.

For RAW DSLR data this is really tricky, since most other applications use DCRAW for raw conversion and this has the downside of not being able to use the whole sensor for data calibration. APP uses the whole sensor and has no dependency on DCRAW. So master calibration frames created in APP have larger image dimensions than those of any other program that uses DCRAW.

 

This topic was modified 1 month ago by Mabula Haverkamp - Admin

Main developer of Astro Pixel Processor and owner of Aries Productions


ReplyQuote
(@singding)
Red Giant Customer
Joined: 1 year ago
Posts: 63
February 7, 2018 13:28  

So just to confirm, we cannot input flat-darks to calibrate the flats? ( In the bias section...thats what I did....)


ReplyQuote
(@mabula-admin)
Quasar Admin
Joined: 2 years ago
Posts: 2071
February 7, 2018 16:47  
Posted by: singding

So just to confirm, we cannot input flat-darks to calibrate the flats? ( In the bias section...thats what I did....)

Hi @singding,

Thank you for your question and welcome to the forum.

No, you can actually, I do it myself all the time 🙂 

A flat dark is just a dark for a flat, load them as darks. And from the above:

FLAT FRAMES

Flat frame calibration has several options, you need to choose 1 option from 1-3:

  1. can and should be calibrated by a master bias of the same ISO or gain + offset as the flat frames
  2. can and should be calibrated by a master dark of the same ISO or gain + offset and the same exposure length and temperature as the flat frames
  3. can and should be calibrated by a master dark of the same ISO or gain + offset and the same exposure length and temperature as the flat frames, made of darks which were calibrated by a masterbias of of the same ISO or gain + offset as the dark frames  + the master bias of the same ISO or gain + offset as the dark frames. So in this case, you use 2 master calibration frames to perform flat frame calibration.

 4) Additionaly, Flat frames can always be calibrated by a Bad Pixel Map besides any of the above 3 methods, which I strongly recommend.

Rule 2 basically is the rule for using (flat) darks to calibrate your flat frames.

Let me know if this is clear and if you have any other question.

Kind regards,

Mabula

 

Main developer of Astro Pixel Processor and owner of Aries Productions


ReplyQuote
(@singding)
Red Giant Customer
Joined: 1 year ago
Posts: 63
February 10, 2018 01:04  

Sorry....it doesn't lol...if I want to use flats ( and flat darks for the flats) and darks (for the lightz) how would I enter all of them if I am supposed to put the flat-darks where the darks go....? Do I simply add them all into the dark spot and the program figures it out?


ReplyQuote
(@singding)
Red Giant Customer
Joined: 1 year ago
Posts: 63
February 10, 2018 01:09  

Basically, where do I load the flat-darks if I am using regular darks...do I add the flat-darks into the flats section?


ReplyQuote
(@gregwrca)
Neutron Star Customer
Joined: 2 years ago
Posts: 223
February 10, 2018 05:22  

Tab 1) LOAD

GW


ReplyQuote
(@mabula-admin)
Quasar Admin
Joined: 2 years ago
Posts: 2071
February 10, 2018 16:39  
Posted by: singding

Sorry....it doesn't lol...if I want to use flats ( and flat darks for the flats) and darks (for the lightz) how would I enter all of them if I am supposed to put the flat-darks where the darks go....? Do I simply add them all into the dark spot and the program figures it out?

Yes, you can simply load the flat darks using the darks load button. APP will use the right master(flat)dark for the flats since it matches the darks and the flats for ISO/gain and exposure time.

Usually what I do, i first create the masterflat so I load, the dark flats (using darks button) and I load the flats (using flats button). Then I create the masterflat. Then I clean the frame list panel and I create the masterdark for the lights (and BPM if you haven't done so).

When finished, I clean the entire list again, I load my lights, my masterdark (for the lights), my masterflat (which is corrected calibrated by the dark flats) and my BPM.

This should work 😉 let me know if this is clear.

To supplement: loading the darks for the lights and the flat darks for the flats at the same time and clicking on calibrate, will actually make two different masterdarks 😉 and APP will match them correctly following the ISO/gain and exposure values.

Mabula

Main developer of Astro Pixel Processor and owner of Aries Productions


ReplyQuote
(@1cm69)
Red Giant Customer
Joined: 1 year ago
Posts: 94
February 15, 2018 18:14  

OK, I am really confused.

I took some images the other night at ISO6400 x 40secs each - Lights

I followed normal procedure & captured Darks at the same ISO & Exposure length as the Lights

Then, as I have always done before using APP, I captured some Flats using my lightbox & the AV-Flat option in BYEOS. These Flats were captured at ISO6400, again, as I have always done prior to working with APP and the AV setting on my DSLR automatically set the Exposure time to 1/2000th Sec.

I did not capture any Bias, never have done in the past either when using DSS etc...

When adding all files to APP, creating MasterDark, MasterFlat & BPM then clearing the 'Load' tab, reloading Lights along with MD, MF & BPM I run through all the steps only to find that the resulting Integration has not used the MF at all, vignette is still apparent.

I now find this thread and see that APP seems to use/require calibration files totally differently to any other processing software  🙄 

I have read & tried to understand APP's requirements from the above thread but am hugely confused.

Firstly my understanding is:

Bias frames is that they are images taken at the fastest possible EXP time available but at the same ISO as the Lights with cap on & are not TEMP dependent.

Darks need to be the same ISO & EXP time as Lights with cap on & are TEMP dependent.

Flats should be same ISO as lights but EXP time variable depending upon position (1/3 to 1/2) histogram from an even light source.

 

Having said this, it shows in the information in post 1 above that:

DARK FRAMES can & should only be calibrated by a master bias of the same ISO.... as the Dark frames

I DO NOT TAKE BIAS, DOES THIS MATTER?

Then the section on Flat frames...as above in post 1...

As stated, I do not take Bias frames so all I can see is that point 2 which relates to me but & here's the big but...

in point 2 it states that the Flats need to be calibrated by a MD of the same ISO - not an issue, this is normal...

but also the same EXP time & TEMP - so here's my issue. Am I supposed to now be taking Flats the SAME EXP length as my Darks?

My Darks EXP length already matches my Lights EXP length, so carrying this logic through, my Flats EXP length will also need to match my Lights EXP length.

This cannot be correct, sometime Lights are 20 or 30mins in length, if I attempted to capture a Flat at that EXP length the image would be hugely overexposed!

Even if I followed the tip that the Flat & Light frames ISO can vary & dropped the ISO for my Flat capturing to my lowest setting of 100, I would still be fighting overexposure on multiple minute exposures but then doing this would mean the point 2 of the Flat Frames requirements would fail because I'd be using ISO100 for Flats but my standard ISO that I use for my Lights & Darks that need to match.

There seems to be a total over complication of the process here & it's contrary to any other available information on collecting calibration frames.

The info in post 1 needs rewriting to make it more understandable, or APP needs re-coding to make it simpler.

So, for example I captured Lights at ISO6400 x 40secs each

If Bias files are required, OK I'll have to take them but anyway...

What are the setting required for my Darks, are they not meant to match the Lights??

I apologise for the rant but I am pulling my hair out trying to understand the strict rules that seem as though they contradict themselves.

 

Regards..,

 

 

 


ReplyQuote
(@gregwrca)
Neutron Star Customer
Joined: 2 years ago
Posts: 223
February 15, 2018 18:59  

Just take 100 bias and be done with it for the year. Set your camera on it as fast as possible shutter speed, and in your case I guess ISO6400, and have it take a hundred frames, temperature doesn't matter. App will make a master bias of this and you can use it forever essentially. Same with that bad pixel map just make it once and use it on subsequent sessions. I don't do all the removing files from list that Mabula goes through, I just load everything up and hit calibrate and go from there leaving all subs in the list

GW


ReplyQuote
(@1cm69)
Red Giant Customer
Joined: 1 year ago
Posts: 94
February 15, 2018 21:31  

OK, think I am getting somewhere in understanding this procedure, getting confused between ISO & EXP.

ISO being light sensitivity & EXP being shutter speed.

Boiling down everything I come to this:

ALL - BIAS, DARK, FLAT & LIGHT frames need to have a matching ISO but they can vary in EXP length and some are dependent on TEMPERATURE, this then marries correctly with post 1 in Flat Frames point 1 & Light Frames point 1

In my case I just ran through a stack which prior to this was not using the Flat frames even though I had added them, this was my data.

100 x 1/4000th sec ISO6400 BIAS - Lens cap on, not TEMP dependent

10 x 40 sec ISO6400 DARKS - Lens cap on, TEMP dependent (try to match Light TEMP)

10 x 1/2000th sec ISO6400 FLATS - using light panel, camera on AV mode, histogram centered.

9 x 40 sec ISO6400 LIGHTS

 

All added to Loaded in APP & run through each step as normal.

Didn't bother unloading after Master file creation, just continued through.

 


ReplyQuote
(@gregwrca)
Neutron Star Customer
Joined: 2 years ago
Posts: 223
February 15, 2018 23:31  

Sounds sound. One thing I do after star analysis or registration is go ahead and sort the subs by star shape (right click a sub) and remove any really bad ones then sort by quality and remove any bad ones and then proceed to normalize and integrate

GW


ReplyQuote
(@1cm69)
Red Giant Customer
Joined: 1 year ago
Posts: 94
February 16, 2018 10:19  
Posted by: gdwats@comcast.net

Sounds sound. One thing I do after star analysis or registration is go ahead and sort the subs by star shape (right click a sub) and remove any really bad ones then sort by quality and remove any bad ones and then proceed to normalize and integrate

Great tip, thanks


ReplyQuote
(@mabula-admin)
Quasar Admin
Joined: 2 years ago
Posts: 2071
February 26, 2018 23:36  
Hi @1cm69,
 
I will respond below to your questions:
Posted by: 1CM69

OK, I am really confused.

I took some images the other night at ISO6400 x 40secs each - Lights

I followed normal procedure & captured Darks at the same ISO & Exposure length as the Lights

Then, as I have always done before using APP, I captured some Flats using my lightbox & the AV-Flat option in BYEOS. These Flats were captured at ISO6400, again, as I have always done prior to working with APP and the AV setting on my DSLR automatically set the Exposure time to 1/2000th Sec.

I did not capture any Bias, never have done in the past either when using DSS etc...

When adding all files to APP, creating MasterDark, MasterFlat & BPM then clearing the 'Load' tab, reloading Lights along with MD, MF & BPM I run through all the steps only to find that the resulting Integration has not used the MF at all, vignette is still apparent.

I now find this thread and see that APP seems to use/require calibration files totally differently to any other processing software  🙄 

I have read & tried to understand APP's requirements from the above thread but am hugely confused.

Firstly my understanding is:

Bias frames is that they are images taken at the fastest possible EXP time available but at the same ISO as the Lights with cap on & are not TEMP dependent.

Darks need to be the same ISO & EXP time as Lights with cap on & are TEMP dependent.

Flats should be same ISO as lights but EXP time variable depending upon position (1/3 to 1/2) histogram from an even light source.

Having said this, it shows in the information in post 1 above that:

DARK FRAMES can & should only be calibrated by a master bias of the same ISO.... as the Dark frames

I DO NOT TAKE BIAS, DOES THIS MATTER?

Then the section on Flat frames...as above in post 1...

As stated, I do not take Bias frames so all I can see is that point 2 which relates to me but & here's the big but...

in point 2 it states that the Flats need to be calibrated by a MD of the same ISO - not an issue, this is normal...

but also the same EXP time & TEMP - so here's my issue. Am I supposed to now be taking Flats the SAME EXP length as my Darks?

My Darks EXP length already matches my Lights EXP length, so carrying this logic through, my Flats EXP length will also need to match my Lights EXP length.

This cannot be correct, sometime Lights are 20 or 30mins in length, if I attempted to capture a Flat at that EXP length the image would be hugely overexposed!

Even if I followed the tip that the Flat & Light frames ISO can vary & dropped the ISO for my Flat capturing to my lowest setting of 100, I would still be fighting overexposure on multiple minute exposures but then doing this would mean the point 2 of the Flat Frames requirements would fail because I'd be using ISO100 for Flats but my standard ISO that I use for my Lights & Darks that need to match.

There seems to be a total over complication of the process here & it's contrary to any other available information on collecting calibration frames.

The info in post 1 needs rewriting to make it more understandable, or APP needs re-coding to make it simpler.

So, for example I captured Lights at ISO6400 x 40secs each

If Bias files are required, OK I'll have to take them but anyway...

What are the setting required for my Darks, are they not meant to match the Lights??

I apologise for the rant but I am pulling my hair out trying to understand the strict rules that seem as though they contradict themselves.

Regards..,

Okay... let me start by indicating that the calibration engine will be upgraded soon. I will work to make it smarter and easier.

But for now... APP isn't using other calibration rules than other applications, but the rules are stricter than for instance an application like DSS uses. The biggest problem here is confusion about general rules of data calibration, I think.

A mistake a lot of astrophotographers make is not being aware that for achieving accurate flat-field calibration, both light frames and flat frames need to be calibrated properly. APP's calibration rules are strict in the sense that APP forces you to do it correctly. If you don't follow it correctly, flat-field calibration will not work. So you are forced to think about why it isn't working.

I know that quite a number of astrophotographers are struggling with APP's calibration engine, so I know I need to improve APP in this department, which I will do with high priority 😉

Let me try to take away the confusion using the documentation already provided in the first post of this topic.

  • It's essential that both light frames and flat frames are calibrated by either a masterbias or a masterdark. Doing so will subtract the bias offset/pedestal from both light frames and flat frames and only that will ensure correct flat-field calibration.

 

Now if you use a masterdark for both light frames and flat frames, you will need to make 2 masterdarks. The masterdark for the flats is sometimes called a masterDarkFlat (or is it masterFlatDark?).

  1. The first masterdark needs to have the same iso & exposure & temperature as your lights.
  2. The second masterdark needs to have the same iso & exposure & temperature as your flats.

 

On the other hand, if you use a masterbias for both light frames and flat frames, you will need to make 2 masterbias frames.

  1. The first masterbias needs to have the same iso as your lights.
  2. The second masterbias needs to have the same iso as your flats.

 

If you shoot the lights and flats with the same ISO, then all you need is 1 single masterbias frame. Other applications do allow you to use 1 single masterbias for lights and flats of different iso values. Think about this. Obviously, it's not something that is optimal. The bias signal will never be calibrated properly if there is some fixed pattern signal in your bias frames.

So if you make flats of 1 second exposure, your darks for your flats need to be of 1 second exposure. You want to calibrate the dark current, bias signal and bias offset/pedestal from your flats so logically, the masterdark used for flat calibration needs to match the flats for iso & exposure & temperature. They certainly don't need to match the iso & exposure & temperature of your light frames.

And vice versa, the same principle applies to the darks for your lights, they need to match the light frames for iso & exposure & temperature.

You need to look at light frame and flat frame calibration as 2 different calibrations but following exactly the same logical rules if you wish to correct for dark current, bias signal and bias offset/pedestal.

I hope that this clarifies how data calibration works and that APP is doing nothing special or strange in this sense when compared to other applications. But you need to apply the rules in APP because APP is strict about this at the moment. As I said, an upcoming upgrade of APP will remedy these strict rules to make the process much easier, I know this is needed 😉

Kind regards,

Mabula

Main developer of Astro Pixel Processor and owner of Aries Productions


ReplyQuote
(@1cm69)
Red Giant Customer
Joined: 1 year ago
Posts: 94
March 1, 2018 11:58  
Posted by: Mabula Haverkamp - Admin
Hi @1cm69,
 
I will respond below to your questions:
Posted by: 1CM69

OK, I am really confused.

I took some images the other night at ISO6400 x 40secs each - Lights

I followed normal procedure & captured Darks at the same ISO & Exposure length as the Lights

Then, as I have always done before using APP, I captured some Flats using my lightbox & the AV-Flat option in BYEOS. These Flats were captured at ISO6400, again, as I have always done prior to working with APP and the AV setting on my DSLR automatically set the Exposure time to 1/2000th Sec.

I did not capture any Bias, never have done in the past either when using DSS etc...

When adding all files to APP, creating MasterDark, MasterFlat & BPM then clearing the 'Load' tab, reloading Lights along with MD, MF & BPM I run through all the steps only to find that the resulting Integration has not used the MF at all, vignette is still apparent.

I now find this thread and see that APP seems to use/require calibration files totally differently to any other processing software  🙄 

I have read & tried to understand APP's requirements from the above thread but am hugely confused.

Firstly my understanding is:

Bias frames is that they are images taken at the fastest possible EXP time available but at the same ISO as the Lights with cap on & are not TEMP dependent.

Darks need to be the same ISO & EXP time as Lights with cap on & are TEMP dependent.

Flats should be same ISO as lights but EXP time variable depending upon position (1/3 to 1/2) histogram from an even light source.

Having said this, it shows in the information in post 1 above that:

DARK FRAMES can & should only be calibrated by a master bias of the same ISO.... as the Dark frames

I DO NOT TAKE BIAS, DOES THIS MATTER?

Then the section on Flat frames...as above in post 1...

As stated, I do not take Bias frames so all I can see is that point 2 which relates to me but & here's the big but...

in point 2 it states that the Flats need to be calibrated by a MD of the same ISO - not an issue, this is normal...

but also the same EXP time & TEMP - so here's my issue. Am I supposed to now be taking Flats the SAME EXP length as my Darks?

My Darks EXP length already matches my Lights EXP length, so carrying this logic through, my Flats EXP length will also need to match my Lights EXP length.

This cannot be correct, sometime Lights are 20 or 30mins in length, if I attempted to capture a Flat at that EXP length the image would be hugely overexposed!

Even if I followed the tip that the Flat & Light frames ISO can vary & dropped the ISO for my Flat capturing to my lowest setting of 100, I would still be fighting overexposure on multiple minute exposures but then doing this would mean the point 2 of the Flat Frames requirements would fail because I'd be using ISO100 for Flats but my standard ISO that I use for my Lights & Darks that need to match.

There seems to be a total over complication of the process here & it's contrary to any other available information on collecting calibration frames.

The info in post 1 needs rewriting to make it more understandable, or APP needs re-coding to make it simpler.

So, for example I captured Lights at ISO6400 x 40secs each

If Bias files are required, OK I'll have to take them but anyway...

What are the setting required for my Darks, are they not meant to match the Lights??

I apologise for the rant but I am pulling my hair out trying to understand the strict rules that seem as though they contradict themselves.

Regards..,

Okay... let me start by indicating that the calibration engine will be upgraded soon. I will work to make it smarter and easier.

But for now... APP isn't using other calibration rules than other applications, but the rules are stricter than for instance an application like DSS uses. The biggest problem here is confusion about general rules of data calibration, I think.

A mistake a lot of astrophotographers make is not being aware that for achieving accurate flat-field calibration, both light frames and flat frames need to be calibrated properly. APP's calibration rules are strict in the sense that APP forces you to do it correctly. If you don't follow it correctly, flat-field calibration will not work. So you are forced to think about why it isn't working.

I know that quite a number of astrophotographers are struggling with APP's calibration engine, so I know I need to improve APP in this department, which I will do with high priority 😉

Let me try to take away the confusion using the documentation already provided in the first post of this topic.

  • It's essential that both light frames and flat frames are calibrated by either a masterbias or a masterdark. Doing so will subtract the bias offset/pedestal from both light frames and flat frames and only that will ensure correct flat-field calibration.

 

Now if you use a masterdark for both light frames and flat frames, you will need to make 2 masterdarks. The masterdark for the flats is sometimes called a masterDarkFlat (or is it masterFlatDark?).

  1. The first masterdark needs to have the same iso & exposure & temperature as your lights.
  2. The second masterdark needs to have the same iso & exposure & temperature as your flats.

 

On the other hand, if you use a masterbias for both light frames and flat frames, you will need to make 2 masterbias frames.

  1. The first masterbias needs to have the same iso as your lights.
  2. The second masterbias needs to have the same iso as your flats.

 

If you shoot the lights and flats with the same ISO, then all you need is 1 single masterbias frame. Other applications do allow you to use 1 single masterbias for lights and flats of different iso values. Think about this. Obviously, it's not something that is optimal. The bias signal will never be calibrated properly if there is some fixed pattern signal in your bias frames.

So if you make flats of 1 second exposure, your darks for your flats need to be of 1 second exposure. You want to calibrate the dark current, bias signal and bias offset/pedestal from your flats so logically, the masterdark used for flat calibration needs to match the flats for iso & exposure & temperature. They certainly don't need to match the iso & exposure & temperature of your light frames.

And vice versa, the same principle applies to the darks for your lights, they need to match the light frames for iso & exposure & temperature.

You need to look at light frame and flat frame calibration as 2 different calibrations but following exactly the same logical rules if you wish to correct for dark current, bias signal and bias offset/pedestal.

I hope that this clarifies how data calibration works and that APP is doing nothing special or strange in this sense when compared to other applications. But you need to apply the rules in APP because APP is strict about this at the moment. As I said, an upcoming upgrade of APP will remedy these strict rules to make the process much easier, I know this is needed 😉

Kind regards,

Mabula

Hi Mabula,

thanks for replying.

I need to clarify this a little more if you don't mind.

an example:

Lights - 900sec @ ISO800 (delta temp)

Darks - 900sec @ ISO800 (same as delta temp)

Flats - 1/2000th sec @ ISO800 (DSLR in AV mode, not temp dependent)

Bias - 1/4000th sec @ ISO800 (Fastest shutter speed, not temp dependent)

In the above example of data collection:

                                                                       do I need any more calibration frames?

                                                                       is it necessary to create multiple MasterDark or MasterBias files?

The above would be my normal data collection process, ISO may vary from 800 but L, D, F & B will always have a matching ISO, plus L & D will always match in exposure length & as close as possible in TEMP.

 

Can I make a suggestion that in the new iteration of the calibration engine that there is a way that APP points out to the end user which calibration frames go together with which others & warn users of missing calibration frames after loading from the folder & before hitting the CALIBRATE button

i.e. WARNING: Dark frames not loaded - please load Dark frames or continue without subtracting Dark signal.

       WARNING: Loaded Dark frames do not match ISO of loaded Light frames & will not be used for Calibration purposes.

 

Also, just a question about the section of your reply, below:

Now if you use a masterdark for both light frames and flat frames, you will need to make 2 masterdarks. The masterdark for the flats is sometimes called a masterDarkFlat (or is it masterFlatDark?).

  1. The first masterdark needs to have the same iso & exposure & temperature as your lights.
  2. The second masterdark needs to have the same iso & exposure & temperature as your flats.

given my example data above, should I always be collecting Darks to match Light frames & Darks to match Flat frames:

i.e. 

Darks - 900sec @ ISO800 (same as delta temp) for creation of MasterDark for Lights

&

Darks - 1/2000th sec @ ISO800 for creation of MasterDark for Flats

or can just my usual routine suffice, Darks that match Lights

 

From the rules you have posted, my understanding is this, again as per my normal data collection as above.

 

My Darks will be calibrated by a MasterBias created from my Bias - 1/4000th sec @ ISO800 (Fastest shutter speed, not temp dependent)

My Flats will be calibrated by a MasterBias created from my Bias - 1/4000th sec @ ISO800 (Fastest shutter speed, not temp dependent)

My Lights will be calibrated by a MasterDarks created from my Darks - 900sec @ ISO800 (same as delta temp)

is this correct?

Am I missing anything?

 

Regards..,

 

Kirk


ReplyQuote
(@mabula-admin)
Quasar Admin
Joined: 2 years ago
Posts: 2071
March 1, 2018 12:33  

@1cm69,

Yes, that should work 😉 Most is correct.

My Darks will be calibrated by a MasterBias created from my Bias - 1/4000th sec @ ISO800 (Fastest shutter speed, not temp dependent)

My Flats will be calibrated by a MasterBias created from my Bias - 1/4000th sec @ ISO800 (Fastest shutter speed, not temp dependent)

My Lights will be calibrated by a MasterDarks created from my Darks - 900sec @ ISO800 (same as delta temp)

Indeed In your example the flats will be calibrated by your masterbias.

The lights will be calibrated by the masterdark which is made from darks calibrated by the masterbias. So the masterdark has the Bias signal + offset/pedestal subtracted as well.

So when you calibrate the lights, you now need to load your masterdark + masterbias and masterflat (& BPM) and then the lights should be propely calibrated. If you don't load the materbias, the bias offset/pedestal won't be subtracted from the lights because the offest/pedestal is also subtracted from the masterdark. If you don't subtract the Masterbias from the darks, then you won't need to use the masterbias for your lights (the bias signal + offset/pedestal then is still part of the masterdark, so if you subtract the masterdark from the lights, you will also subtract the bias signal + offset/pedestal.)

 

Yes, that's exactly what I intend to do. The new calibration engine will be less strict, but it will start warning the user if the supplied frames don't guarantee an optimal calibration path (with the option to disable the warnings off course).

 

Using darks to calibrate flats is something that is not necessary in most cases. But if you want optimal calibration, for some camera's (like the newer CMOS camera's like an asi-1600mm ) it is to be advised to subtract the dark current (with fixed pattern noise and amp glow) even from the flats. So it's a bit camera dependent.

 

Regarding your flats, do you get good flat calibration with flats of such short exposure times? I know that such short flat exposure times can give all sorts of problems, so my advice would be to dim the light source somehow so the flats can be exposed at least in the order of 0.5 to 5 seconds. But if it works, it works off course 😉

Kind regards,

Mabula

 

Main developer of Astro Pixel Processor and owner of Aries Productions


ReplyQuote
(@1cm69)
Red Giant Customer
Joined: 1 year ago
Posts: 94
March 1, 2018 13:01  

Hi Mabula,

OK, I am now much clearer in the process.

In regards to my Flats, I think that is something I will look in to.

I have read so many different processes for taking Flats & haven't really narrowed down to a consistent process as yet.

Regards..,

Kirk

 


ReplyQuote
(@1cm69)
Red Giant Customer
Joined: 1 year ago
Posts: 94
March 4, 2018 14:05  

Hi Mabula - @mabula-haverkamp-administrator

I have been looking more into my Flat collection procedure & have read vast amounts of differing accounts of the best way to capture Flats.

So, as you pointed out my current Flats taken at 1/2000th sec are possibly too short but this was because I was attempting to take them at the same ISO as my Lights, Darks etc... which is generally ISO800. Because this ISO is fairly high & sensitive the AV setting in my DSLR is setting the EXP to this very quick shutter speed.

Now having re-read your rules for APP & understanding it all a little more, I can see a way around this & this is to reduce the ISO of my Flats to achieve a longer EXP time. Yes this would mean taking another set of Bias frames also to match these Flat frames, I know this.

So I set upon experimenting a little with various different configurations of my light panel & DSLR ISO settings.

I have found through a process of elimination that an ISO setting of 100 with my light panel on it's lowest setting & using 2 t-shirts between my scope and the light panel, gives me the best looking histogram.

This however is where I am now stuck.

I have read many paces not to trust the histogram in BYEOS as it is not the true RAW histogram but more a JPEG one. I have tried using various pieces of software such as RAWdigger, IRIS etc... in a failed attempt to find the so called MEDIAN.

Everywhere seems to rave about Pixinsight but I am not willing to pay the high price for the software, I have chosen APP for my processing choice.

I have uploaded a selection of my RAW .cr2 files to DropBox, here:  https://www.dropbox.com/sh/4vryrk1m1vq3q27/AAC9x5w912vMqtZKFFquZW_Qa?dl=0

Please could you possibly look at these & let me know which you consider to be the closest to the MEDIAN or could you point me in the direction of any software other than Pixinsight that I can use to find the info myself.

Kind regards..,

Kirk


ReplyQuote
(@1cm69)
Red Giant Customer
Joined: 1 year ago
Posts: 94
March 15, 2018 18:16  

Anyone??


ReplyQuote
 Tim
(@tim)
Main Sequence Star Customer
Joined: 2 years ago
Posts: 36
March 15, 2018 19:21  

Leave the iso for your flats the same as your lights and darks. This is a MUST. Just increase the time for your flats without changing the iso. I use around  1 second but it can vary a lot. If necessary use a few sheets of white paper or white T-shirts to reduce the glare and increase the time for your flats. Using the histogram in BYEOS is fine for this purpose. Keep it around the center. 


ReplyQuote
(@1cm69)
Red Giant Customer
Joined: 1 year ago
Posts: 94
March 15, 2018 20:22  

Hi Tim, thanks for replying. I’ve tried many ways to take longer Flats using the same ISO as my Lights, generally ISO800 but cannot get anywhere near even 0.5sec, high ISO just far too sensitive. 

This is why I was following Mabula’s optional advice as above in reply to a previous post of mine, to capture Flats at a different ISO but calibrating them with their own MasterBias taken at the same ISO then using this produced MasterFlat in the further processing steps. 

i.e.

Lights, Darks & Bias ISO800 (Lights & Darks same Exp length, Bias at fastest Exp length)

Flats & Bias ISO100 (Flats Exp length as required, Bias at fastest Exp length)

Clear as mud 😉

also, I found this informative page, some of it not relevant as I do not own Pixinsight:

http://www.myastroscience.com/proper-flats-with-dslr

 


ReplyQuote
Page 1 / 4
Share: