[Sticky] Data calibration principles/rules - must read !

So just to confirm, we cannot input flat-darks to calibrate the flats? ( In the bias section...thats what I did....)

Posted by: singding

So just to confirm, we cannot input flat-darks to calibrate the flats? ( In the bias section...thats what I did....)

Hi @singding,

Thank you for your question and welcome to the forum.

No, you can actually, I do it myself all the time 🙂 

A flat dark is just a dark for a flat, load them as darks. And from the above:

FLAT FRAMES

Flat frame calibration has several options, you need to choose 1 option from 1-3:

1. can and should be calibrated by a master bias of the same ISO or gain + offset as the flat frames
2. can and should be calibrated by a master dark of the same ISO or gain + offset and the same exposure length and temperature as the flat frames
3. can and should be calibrated by a master dark of the same ISO or gain + offset and the same exposure length and temperature as the flat frames, made of darks which were calibrated by a masterbias of of the same ISO or gain + offset as the dark frames  + the master bias of the same ISO or gain + offset as the dark frames. So in this case, you use 2 master calibration frames to perform flat frame calibration.

 4) Additionaly, Flat frames can always be calibrated by a Bad Pixel Map besides any of the above 3 methods, which I strongly recommend.

Rule 2 basically is the rule for using (flat) darks to calibrate your flat frames.

Let me know if this is clear and if you have any other question.

Kind regards,

Mabula

Sorry....it doesn't lol...if I want to use flats ( and flat darks for the flats) and darks (for the lightz) how would I enter all of them if I am supposed to put the flat-darks where the darks go....? Do I simply add them all into the dark spot and the program figures it out?

Basically, where do I load the flat-darks if I am using regular darks...do I add the flat-darks into the flats section?

Tab 1) LOAD

Posted by: singding

Sorry....it doesn't lol...if I want to use flats ( and flat darks for the flats) and darks (for the lightz) how would I enter all of them if I am supposed to put the flat-darks where the darks go....? Do I simply add them all into the dark spot and the program figures it out?

Yes, you can simply load the flat darks using the darks load button. APP will use the right master(flat)dark for the flats since it matches the darks and the flats for ISO/gain and exposure time.

Usually what I do, i first create the masterflat so I load, the dark flats (using darks button) and I load the flats (using flats button). Then I create the masterflat. Then I clean the frame list panel and I create the masterdark for the lights (and BPM if you haven't done so).

When finished, I clean the entire list again, I load my lights, my masterdark (for the lights), my masterflat (which is corrected calibrated by the dark flats) and my BPM.

This should work 😉 let me know if this is clear.

To supplement: loading the darks for the lights and the flat darks for the flats at the same time and clicking on calibrate, will actually make two different masterdarks 😉 and APP will match them correctly following the ISO/gain and exposure values.

Mabula

OK, I am really confused.

I took some images the other night at ISO6400 x 40secs each - Lights

I followed normal procedure & captured Darks at the same ISO & Exposure length as the Lights

Then, as I have always done before using APP, I captured some Flats using my lightbox & the AV-Flat option in BYEOS. These Flats were captured at ISO6400, again, as I have always done prior to working with APP and the AV setting on my DSLR automatically set the Exposure time to 1/2000th Sec.

I did not capture any Bias, never have done in the past either when using DSS etc...

When adding all files to APP, creating MasterDark, MasterFlat & BPM then clearing the 'Load' tab, reloading Lights along with MD, MF & BPM I run through all the steps only to find that the resulting Integration has not used the MF at all, vignette is still apparent.

I now find this thread and see that APP seems to use/require calibration files totally differently to any other processing software  🙄 

I have read & tried to understand APP's requirements from the above thread but am hugely confused.

Firstly my understanding is:

Bias frames is that they are images taken at the fastest possible EXP time available but at the same ISO as the Lights with cap on & are not TEMP dependent.

Darks need to be the same ISO & EXP time as Lights with cap on & are TEMP dependent.

Flats should be same ISO as lights but EXP time variable depending upon position (1/3 to 1/2) histogram from an even light source.

Having said this, it shows in the information in post 1 above that:

DARK FRAMES can & should only be calibrated by a master bias of the same ISO.... as the Dark frames

I DO NOT TAKE BIAS, DOES THIS MATTER?

Then the section on Flat frames...as above in post 1...

As stated, I do not take Bias frames so all I can see is that point 2 which relates to me but & here's the big but...

in point 2 it states that the Flats need to be calibrated by a MD of the same ISO - not an issue, this is normal...

but also the same EXP time & TEMP - so here's my issue. Am I supposed to now be taking Flats the SAME EXP length as my Darks?

My Darks EXP length already matches my Lights EXP length, so carrying this logic through, my Flats EXP length will also need to match my Lights EXP length.

This cannot be correct, sometime Lights are 20 or 30mins in length, if I attempted to capture a Flat at that EXP length the image would be hugely overexposed!

Even if I followed the tip that the Flat & Light frames ISO can vary & dropped the ISO for my Flat capturing to my lowest setting of 100, I would still be fighting overexposure on multiple minute exposures but then doing this would mean the point 2 of the Flat Frames requirements would fail because I'd be using ISO100 for Flats but my standard ISO that I use for my Lights & Darks that need to match.

There seems to be a total over complication of the process here & it's contrary to any other available information on collecting calibration frames.

The info in post 1 needs rewriting to make it more understandable, or APP needs re-coding to make it simpler.

So, for example I captured Lights at ISO6400 x 40secs each

If Bias files are required, OK I'll have to take them but anyway...

What are the setting required for my Darks, are they not meant to match the Lights??

I apologise for the rant but I am pulling my hair out trying to understand the strict rules that seem as though they contradict themselves.

Regards..,

Just take 100 bias and be done with it for the year. Set your camera on it as fast as possible shutter speed, and in your case I guess ISO6400, and have it take a hundred frames, temperature doesn't matter. App will make a master bias of this and you can use it forever essentially. Same with that bad pixel map just make it once and use it on subsequent sessions. I don't do all the removing files from list that Mabula goes through, I just load everything up and hit calibrate and go from there leaving all subs in the list

OK, think I am getting somewhere in understanding this procedure, getting confused between ISO & EXP.

ISO being light sensitivity & EXP being shutter speed.

Boiling down everything I come to this:

ALL - BIAS, DARK, FLAT & LIGHT frames need to have a matching ISO but they can vary in EXP length and some are dependent on TEMPERATURE, this then marries correctly with post 1 in Flat Frames point 1 & Light Frames point 1

In my case I just ran through a stack which prior to this was not using the Flat frames even though I had added them, this was my data.

100 x 1/4000th sec ISO6400 BIAS - Lens cap on, not TEMP dependent

10 x 40 sec ISO6400 DARKS - Lens cap on, TEMP dependent (try to match Light TEMP)

10 x 1/2000th sec ISO6400 FLATS - using light panel, camera on AV mode, histogram centered.

9 x 40 sec ISO6400 LIGHTS

All added to Loaded in APP & run through each step as normal.

Didn't bother unloading after Master file creation, just continued through.

Sounds sound. One thing I do after star analysis or registration is go ahead and sort the subs by star shape (right click a sub) and remove any really bad ones then sort by quality and remove any bad ones and then proceed to normalize and integrate

Posted by: gdwats@comcast.net

Sounds sound. One thing I do after star analysis or registration is go ahead and sort the subs by star shape (right click a sub) and remove any really bad ones then sort by quality and remove any bad ones and then proceed to normalize and integrate

Great tip, thanks

Posted by: 1CM69

OK, I am really confused.

I took some images the other night at ISO6400 x 40secs each - Lights

I followed normal procedure & captured Darks at the same ISO & Exposure length as the Lights

Then, as I have always done before using APP, I captured some Flats using my lightbox & the AV-Flat option in BYEOS. These Flats were captured at ISO6400, again, as I have always done prior to working with APP and the AV setting on my DSLR automatically set the Exposure time to 1/2000th Sec.

I did not capture any Bias, never have done in the past either when using DSS etc...

When adding all files to APP, creating MasterDark, MasterFlat & BPM then clearing the 'Load' tab, reloading Lights along with MD, MF & BPM I run through all the steps only to find that the resulting Integration has not used the MF at all, vignette is still apparent.

I now find this thread and see that APP seems to use/require calibration files totally differently to any other processing software  🙄 

I have read & tried to understand APP's requirements from the above thread but am hugely confused.

Firstly my understanding is:

Bias frames is that they are images taken at the fastest possible EXP time available but at the same ISO as the Lights with cap on & are not TEMP dependent.

Darks need to be the same ISO & EXP time as Lights with cap on & are TEMP dependent.

Flats should be same ISO as lights but EXP time variable depending upon position (1/3 to 1/2) histogram from an even light source.

Having said this, it shows in the information in post 1 above that:

DARK FRAMES can & should only be calibrated by a master bias of the same ISO.... as the Dark frames

I DO NOT TAKE BIAS, DOES THIS MATTER?

Then the section on Flat frames...as above in post 1...

As stated, I do not take Bias frames so all I can see is that point 2 which relates to me but & here's the big but...

in point 2 it states that the Flats need to be calibrated by a MD of the same ISO - not an issue, this is normal...

but also the same EXP time & TEMP - so here's my issue. Am I supposed to now be taking Flats the SAME EXP length as my Darks?

My Darks EXP length already matches my Lights EXP length, so carrying this logic through, my Flats EXP length will also need to match my Lights EXP length.

This cannot be correct, sometime Lights are 20 or 30mins in length, if I attempted to capture a Flat at that EXP length the image would be hugely overexposed!

Even if I followed the tip that the Flat & Light frames ISO can vary & dropped the ISO for my Flat capturing to my lowest setting of 100, I would still be fighting overexposure on multiple minute exposures but then doing this would mean the point 2 of the Flat Frames requirements would fail because I'd be using ISO100 for Flats but my standard ISO that I use for my Lights & Darks that need to match.

There seems to be a total over complication of the process here & it's contrary to any other available information on collecting calibration frames.

The info in post 1 needs rewriting to make it more understandable, or APP needs re-coding to make it simpler.

So, for example I captured Lights at ISO6400 x 40secs each

If Bias files are required, OK I'll have to take them but anyway...

What are the setting required for my Darks, are they not meant to match the Lights??

I apologise for the rant but I am pulling my hair out trying to understand the strict rules that seem as though they contradict themselves.

Regards..,

Okay... let me start by indicating that the calibration engine will be upgraded soon. I will work to make it smarter and easier.

But for now... APP isn't using other calibration rules than other applications, but the rules are stricter than for instance an application like DSS uses. The biggest problem here is confusion about general rules of data calibration, I think.

A mistake a lot of astrophotographers make is not being aware that for achieving accurate flat-field calibration, both light frames and flat frames need to be calibrated properly. APP's calibration rules are strict in the sense that APP forces you to do it correctly. If you don't follow it correctly, flat-field calibration will not work. So you are forced to think about why it isn't working.

I know that quite a number of astrophotographers are struggling with APP's calibration engine, so I know I need to improve APP in this department, which I will do with high priority 😉

Let me try to take away the confusion using the documentation already provided in the first post of this topic.

• It's essential that both light frames and flat frames are calibrated by either a masterbias or a masterdark. Doing so will subtract the bias offset/pedestal from both light frames and flat frames and only that will ensure correct flat-field calibration.

Now if you use a masterdark for both light frames and flat frames, you will need to make 2 masterdarks. The masterdark for the flats is sometimes called a masterDarkFlat (or is it masterFlatDark?).

1. The first masterdark needs to have the same iso & exposure & temperature as your lights.
2. The second masterdark needs to have the same iso & exposure & temperature as your flats.

On the other hand, if you use a masterbias for both light frames and flat frames, you will need to make 2 masterbias frames.

1. The first masterbias needs to have the same iso as your lights.
2. The second masterbias needs to have the same iso as your flats.

If you shoot the lights and flats with the same ISO, then all you need is 1 single masterbias frame. Other applications do allow you to use 1 single masterbias for lights and flats of different iso values. Think about this. Obviously, it's not something that is optimal. The bias signal will never be calibrated properly if there is some fixed pattern signal in your bias frames.

So if you make flats of 1 second exposure, your darks for your flats need to be of 1 second exposure. You want to calibrate the dark current, bias signal and bias offset/pedestal from your flats so logically, the masterdark used for flat calibration needs to match the flats for iso & exposure & temperature. They certainly don't need to match the iso & exposure & temperature of your light frames.

And vice versa, the same principle applies to the darks for your lights, they need to match the light frames for iso & exposure & temperature.

You need to look at light frame and flat frame calibration as 2 different calibrations but following exactly the same logical rules if you wish to correct for dark current, bias signal and bias offset/pedestal.

I hope that this clarifies how data calibration works and that APP is doing nothing special or strange in this sense when compared to other applications. But you need to apply the rules in APP because APP is strict about this at the moment. As I said, an upcoming upgrade of APP will remedy these strict rules to make the process much easier, I know this is needed 😉

Kind regards,

Mabula

Posted by: Mabula Haverkamp - Admin

Hi @1cm69,

 

I will respond below to your questions:

Posted by: 1CM69

OK, I am really confused.

I took some images the other night at ISO6400 x 40secs each - Lights

I followed normal procedure & captured Darks at the same ISO & Exposure length as the Lights

Then, as I have always done before using APP, I captured some Flats using my lightbox & the AV-Flat option in BYEOS. These Flats were captured at ISO6400, again, as I have always done prior to working with APP and the AV setting on my DSLR automatically set the Exposure time to 1/2000th Sec.

I did not capture any Bias, never have done in the past either when using DSS etc...

When adding all files to APP, creating MasterDark, MasterFlat & BPM then clearing the 'Load' tab, reloading Lights along with MD, MF & BPM I run through all the steps only to find that the resulting Integration has not used the MF at all, vignette is still apparent.

I now find this thread and see that APP seems to use/require calibration files totally differently to any other processing software  🙄 

I have read & tried to understand APP's requirements from the above thread but am hugely confused.

Firstly my understanding is:

Bias frames is that they are images taken at the fastest possible EXP time available but at the same ISO as the Lights with cap on & are not TEMP dependent.

Darks need to be the same ISO & EXP time as Lights with cap on & are TEMP dependent.

Flats should be same ISO as lights but EXP time variable depending upon position (1/3 to 1/2) histogram from an even light source.

Having said this, it shows in the information in post 1 above that:

DARK FRAMES can & should only be calibrated by a master bias of the same ISO.... as the Dark frames

I DO NOT TAKE BIAS, DOES THIS MATTER?

Then the section on Flat frames...as above in post 1...

As stated, I do not take Bias frames so all I can see is that point 2 which relates to me but & here's the big but...

in point 2 it states that the Flats need to be calibrated by a MD of the same ISO - not an issue, this is normal...

but also the same EXP time & TEMP - so here's my issue. Am I supposed to now be taking Flats the SAME EXP length as my Darks?

My Darks EXP length already matches my Lights EXP length, so carrying this logic through, my Flats EXP length will also need to match my Lights EXP length.

This cannot be correct, sometime Lights are 20 or 30mins in length, if I attempted to capture a Flat at that EXP length the image would be hugely overexposed!

Even if I followed the tip that the Flat & Light frames ISO can vary & dropped the ISO for my Flat capturing to my lowest setting of 100, I would still be fighting overexposure on multiple minute exposures but then doing this would mean the point 2 of the Flat Frames requirements would fail because I'd be using ISO100 for Flats but my standard ISO that I use for my Lights & Darks that need to match.

There seems to be a total over complication of the process here & it's contrary to any other available information on collecting calibration frames.

The info in post 1 needs rewriting to make it more understandable, or APP needs re-coding to make it simpler.

So, for example I captured Lights at ISO6400 x 40secs each

If Bias files are required, OK I'll have to take them but anyway...

What are the setting required for my Darks, are they not meant to match the Lights??

I apologise for the rant but I am pulling my hair out trying to understand the strict rules that seem as though they contradict themselves.

Regards..,

Okay... let me start by indicating that the calibration engine will be upgraded soon. I will work to make it smarter and easier.

But for now... APP isn't using other calibration rules than other applications, but the rules are stricter than for instance an application like DSS uses. The biggest problem here is confusion about general rules of data calibration, I think.

A mistake a lot of astrophotographers make is not being aware that for achieving accurate flat-field calibration, both light frames and flat frames need to be calibrated properly. APP's calibration rules are strict in the sense that APP forces you to do it correctly. If you don't follow it correctly, flat-field calibration will not work. So you are forced to think about why it isn't working.

I know that quite a number of astrophotographers are struggling with APP's calibration engine, so I know I need to improve APP in this department, which I will do with high priority 😉

Let me try to take away the confusion using the documentation already provided in the first post of this topic.

• It's essential that both light frames and flat frames are calibrated by either a masterbias or a masterdark. Doing so will subtract the bias offset/pedestal from both light frames and flat frames and only that will ensure correct flat-field calibration.

 

Now if you use a masterdark for both light frames and flat frames, you will need to make 2 masterdarks. The masterdark for the flats is sometimes called a masterDarkFlat (or is it masterFlatDark?).

1. The first masterdark needs to have the same iso & exposure & temperature as your lights.
2. The second masterdark needs to have the same iso & exposure & temperature as your flats.

 

On the other hand, if you use a masterbias for both light frames and flat frames, you will need to make 2 masterbias frames.

1. The first masterbias needs to have the same iso as your lights.
2. The second masterbias needs to have the same iso as your flats.

 

If you shoot the lights and flats with the same ISO, then all you need is 1 single masterbias frame. Other applications do allow you to use 1 single masterbias for lights and flats of different iso values. Think about this. Obviously, it's not something that is optimal. The bias signal will never be calibrated properly if there is some fixed pattern signal in your bias frames.

So if you make flats of 1 second exposure, your darks for your flats need to be of 1 second exposure. You want to calibrate the dark current, bias signal and bias offset/pedestal from your flats so logically, the masterdark used for flat calibration needs to match the flats for iso & exposure & temperature. They certainly don't need to match the iso & exposure & temperature of your light frames.

And vice versa, the same principle applies to the darks for your lights, they need to match the light frames for iso & exposure & temperature.

You need to look at light frame and flat frame calibration as 2 different calibrations but following exactly the same logical rules if you wish to correct for dark current, bias signal and bias offset/pedestal.

I hope that this clarifies how data calibration works and that APP is doing nothing special or strange in this sense when compared to other applications. But you need to apply the rules in APP because APP is strict about this at the moment. As I said, an upcoming upgrade of APP will remedy these strict rules to make the process much easier, I know this is needed 😉

Kind regards,

Mabula

Hi Mabula,

thanks for replying.

I need to clarify this a little more if you don't mind.

an example:

Lights - 900sec @ ISO800 (delta temp)

Darks - 900sec @ ISO800 (same as delta temp)

Flats - 1/2000th sec @ ISO800 (DSLR in AV mode, not temp dependent)

Bias - 1/4000th sec @ ISO800 (Fastest shutter speed, not temp dependent)

In the above example of data collection:

                                                                       do I need any more calibration frames?

                                                                       is it necessary to create multiple MasterDark or MasterBias files?

The above would be my normal data collection process, ISO may vary from 800 but L, D, F & B will always have a matching ISO, plus L & D will always match in exposure length & as close as possible in TEMP.

Can I make a suggestion that in the new iteration of the calibration engine that there is a way that APP points out to the end user which calibration frames go together with which others & warn users of missing calibration frames after loading from the folder & before hitting the CALIBRATE button

i.e. WARNING: Dark frames not loaded - please load Dark frames or continue without subtracting Dark signal.

       WARNING: Loaded Dark frames do not match ISO of loaded Light frames & will not be used for Calibration purposes.

Also, just a question about the section of your reply, below:

Now if you use a masterdark for both light frames and flat frames, you will need to make 2 masterdarks. The masterdark for the flats is sometimes called a masterDarkFlat (or is it masterFlatDark?).

1. The first masterdark needs to have the same iso & exposure & temperature as your lights.
2. The second masterdark needs to have the same iso & exposure & temperature as your flats.

given my example data above, should I always be collecting Darks to match Light frames & Darks to match Flat frames:

i.e. 

Darks - 900sec @ ISO800 (same as delta temp) for creation of MasterDark for Lights

&

Darks - 1/2000th sec @ ISO800 for creation of MasterDark for Flats

or can just my usual routine suffice, Darks that match Lights

From the rules you have posted, my understanding is this, again as per my normal data collection as above.

My Darks will be calibrated by a MasterBias created from my Bias - 1/4000th sec @ ISO800 (Fastest shutter speed, not temp dependent)

My Flats will be calibrated by a MasterBias created from my Bias - 1/4000th sec @ ISO800 (Fastest shutter speed, not temp dependent)

My Lights will be calibrated by a MasterDarks created from my Darks - 900sec @ ISO800 (same as delta temp)

is this correct?

Am I missing anything?

Regards..,

Kirk

@1cm69,

Yes, that should work 😉 Most is correct.

My Darks will be calibrated by a MasterBias created from my Bias - 1/4000th sec @ ISO800 (Fastest shutter speed, not temp dependent)

My Flats will be calibrated by a MasterBias created from my Bias - 1/4000th sec @ ISO800 (Fastest shutter speed, not temp dependent)

My Lights will be calibrated by a MasterDarks created from my Darks - 900sec @ ISO800 (same as delta temp)

Indeed In your example the flats will be calibrated by your masterbias.

The lights will be calibrated by the masterdark which is made from darks calibrated by the masterbias. So the masterdark has the Bias signal + offset/pedestal subtracted as well.

So when you calibrate the lights, you now need to load your masterdark + masterbias and masterflat (& BPM) and then the lights should be propely calibrated. If you don't load the materbias, the bias offset/pedestal won't be subtracted from the lights because the offest/pedestal is also subtracted from the masterdark. If you don't subtract the Masterbias from the darks, then you won't need to use the masterbias for your lights (the bias signal + offset/pedestal then is still part of the masterdark, so if you subtract the masterdark from the lights, you will also subtract the bias signal + offset/pedestal.)

Yes, that's exactly what I intend to do. The new calibration engine will be less strict, but it will start warning the user if the supplied frames don't guarantee an optimal calibration path (with the option to disable the warnings off course).

Using darks to calibrate flats is something that is not necessary in most cases. But if you want optimal calibration, for some camera's (like the newer CMOS camera's like an asi-1600mm ) it is to be advised to subtract the dark current (with fixed pattern noise and amp glow) even from the flats. So it's a bit camera dependent.

Regarding your flats, do you get good flat calibration with flats of such short exposure times? I know that such short flat exposure times can give all sorts of problems, so my advice would be to dim the light source somehow so the flats can be exposed at least in the order of 0.5 to 5 seconds. But if it works, it works off course 😉

Kind regards,

Mabula

Hi Mabula,

OK, I am now much clearer in the process.

In regards to my Flats, I think that is something I will look in to.

I have read so many different processes for taking Flats & haven't really narrowed down to a consistent process as yet.

Regards..,

Kirk

Hi Mabula - @mabula-haverkamp-administrator

I have been looking more into my Flat collection procedure & have read vast amounts of differing accounts of the best way to capture Flats.

So, as you pointed out my current Flats taken at 1/2000th sec are possibly too short but this was because I was attempting to take them at the same ISO as my Lights, Darks etc... which is generally ISO800. Because this ISO is fairly high & sensitive the AV setting in my DSLR is setting the EXP to this very quick shutter speed.

Now having re-read your rules for APP & understanding it all a little more, I can see a way around this & this is to reduce the ISO of my Flats to achieve a longer EXP time. Yes this would mean taking another set of Bias frames also to match these Flat frames, I know this.

So I set upon experimenting a little with various different configurations of my light panel & DSLR ISO settings.

I have found through a process of elimination that an ISO setting of 100 with my light panel on it's lowest setting & using 2 t-shirts between my scope and the light panel, gives me the best looking histogram.

This however is where I am now stuck.

I have read many paces not to trust the histogram in BYEOS as it is not the true RAW histogram but more a JPEG one. I have tried using various pieces of software such as RAWdigger, IRIS etc... in a failed attempt to find the so called MEDIAN.

Everywhere seems to rave about Pixinsight but I am not willing to pay the high price for the software, I have chosen APP for my processing choice.

I have uploaded a selection of my RAW .cr2 files to DropBox, here:  https://www.dropbox.com/sh/4vryrk1m1vq3q27/AAC9x5w912vMqtZKFFquZW_Qa?dl=0

Please could you possibly look at these & let me know which you consider to be the closest to the MEDIAN or could you point me in the direction of any software other than Pixinsight that I can use to find the info myself.

Kind regards..,

Kirk

Anyone??

Leave the iso for your flats the same as your lights and darks. This is a MUST. Just increase the time for your flats without changing the iso. I use around  1 second but it can vary a lot. If necessary use a few sheets of white paper or white T-shirts to reduce the glare and increase the time for your flats. Using the histogram in BYEOS is fine for this purpose. Keep it around the center. 

Hi Tim, thanks for replying. I’ve tried many ways to take longer Flats using the same ISO as my Lights, generally ISO800 but cannot get anywhere near even 0.5sec, high ISO just far too sensitive. 

This is why I was following Mabula’s optional advice as above in reply to a previous post of mine, to capture Flats at a different ISO but calibrating them with their own MasterBias taken at the same ISO then using this produced MasterFlat in the further processing steps. 

i.e.

Lights, Darks & Bias ISO800 (Lights & Darks same Exp length, Bias at fastest Exp length)

Flats & Bias ISO100 (Flats Exp length as required, Bias at fastest Exp length)

Clear as mud 😉

also, I found this informative page, some of it not relevant as I do not own Pixinsight:

http://www.myastroscience.com/proper-flats-with-dslr

I have a fairly bright LED that I use for flats. I just add sheets of white paper between the screen and scope until I get it dim enough to take longer flats. Its just a matter of how many sheets of paper.

I don't use bias frames. I use dark flats which are slightly more accurate than bias frames - especially for longer flats. Here is an easy way to look at it.

Take lights and darks at the same iso, exposure and temperature.

Take your flats somewhere between say .5 and 5 seconds. This is just an approximation. Then, put the cap on your scope and take the same number of images at the same iso and time that you just used for your flats. These are called flat darks or dark flats. So flat darks are to calibrate flats like darks calibrate lights.

So what APP does is create a master dark flat to calibrate and create a master flat. This master flat is used with the master dark  to calibrate the lights.

There are many references on the net saying the iso of the lights and darks should match the iso of the flats and flat darks. There are also different references saying they don't have to match. Its easy to match them so I do it. I hope this makes sense -if not let me know.

My Flats panel has 3 light levels, I use the lowest with a diffuser sheet & 2 t-shirts & at ISO100 exposure length is 0.6sec for a mid histogram. At ISO800 I just get a blown overexposed white screen with histogram at the far right. 

As I mentioned I've read some articles saying the the iso from the flats should be the same as the darks and lights and I've also read that it didn't matter. I've never tested using different iso's so I can't comment from personal experience. Nothing is wrong with trying to see if it works.

It sounds like I have a similar panel as yours. I used to use the t-shirt method but when I needed to reduce the glare even more, it was suggested to me to try adding individual sheets of white paper (computer paper?) so I would have maximum control over the brightness. This worked for me.

" nothing is wrong with trying to reinvent the wheel". 
Why would you waste your time even trying? Are you going out 
of your way to avoid matching ISO?

Posted by: gdwats@comcast.net

" nothing is wrong with trying to reinvent the wheel". 
Why would you waste your time even trying? Are you going out 
of your way to avoid matching ISO?

Look mate, I assume you’re the type that skim reads & then jumps to conclusions and then posts a useless comment. 

Reread the whole thread & absorb the information, then if you have worthwhile input to share, go ahead and let us all know. 

Cheerio

Can I use a different ISO or gain+offset than my Light frames, for my flat frames?

Yes, that is no problem and actually preferred in most cases. Make your flats with a low ISO or gain+offset to be able to increase the exposure length of the flats. They will be of better quality then. Be aware of the rules you need to follow for flat & light frames. Both need to have the bias pedestal subtracted only once using any of the available options. So if you only use bias frames and no darks. You need 2 master bias frames of the 2 different ISO or gain+offset value to make sure that the bias pedestal is removed correctly (only once) from both lights and flats.

above excerpt taken directly from the first post in this thread.

Posted by: 1CM69

Hi Mabula - @mabula-haverkamp-administrator

I have been looking more into my Flat collection procedure & have read vast amounts of differing accounts of the best way to capture Flats.

So, as you pointed out my current Flats taken at 1/2000th sec are possibly too short but this was because I was attempting to take them at the same ISO as my Lights, Darks etc... which is generally ISO800. Because this ISO is fairly high & sensitive the AV setting in my DSLR is setting the EXP to this very quick shutter speed.

Now having re-read your rules for APP & understanding it all a little more, I can see a way around this & this is to reduce the ISO of my Flats to achieve a longer EXP time. Yes this would mean taking another set of Bias frames also to match these Flat frames, I know this.

So I set upon experimenting a little with various different configurations of my light panel & DSLR ISO settings.

I have found through a process of elimination that an ISO setting of 100 with my light panel on it's lowest setting & using 2 t-shirts between my scope and the light panel, gives me the best looking histogram.

This however is where I am now stuck.

I have read many paces not to trust the histogram in BYEOS as it is not the true RAW histogram but more a JPEG one. I have tried using various pieces of software such as RAWdigger, IRIS etc... in a failed attempt to find the so called MEDIAN.

Everywhere seems to rave about Pixinsight but I am not willing to pay the high price for the software, I have chosen APP for my processing choice.

I have uploaded a selection of my RAW .cr2 files to DropBox, here:  https://www.dropbox.com/sh/4vryrk1m1vq3q27/AAC9x5w912vMqtZKFFquZW_Qa?dl=0

Please could you possibly look at these & let me know which you consider to be the closest to the MEDIAN or could you point me in the direction of any software other than Pixinsight that I can use to find the info myself.

Kind regards..,

Kirk

@1cm69 , Will check them now and I will show you how you can verify yourself in APP 😉

Hang on...

Mabula

Posted by: Tim

Leave the iso for your flats the same as your lights and darks. This is a MUST. Just increase the time for your flats without changing the iso. I use around  1 second but it can vary a lot. If necessary use a few sheets of white paper or white T-shirts to reduce the glare and increase the time for your flats. Using the histogram in BYEOS is fine for this purpose. Keep it around the center. 

@tim, no it actually isn't a must I believe ;-). I never do it on both an Asi1600mm-c and Nikon D5100 and D610 DLSRs.

What would be the explanation that you can't do this? If you have a topic/website/source that says otherwise, do share it 😉

(I always create flats with ISO 100 on a DLSR and gain 0 with the asi1600mm-c and never had any calibration problem with the flats...)

Flats have the purpose of catching the vignetting profile and dust bunnies of your optical setup. To get the vignetting profile, the flats need to be corrected with either a masterbias or masterdarkflat of the same iso as the flats if you want to do it properly.

I have used Backyard Nikon myself, but that also doesn't give a linear histogram, or does it ? You will be amazed at how much longer you can expose your flats 😉 !

Mabula

OK, thanks

Posted by: Tim

I have a fairly bright LED that I use for flats. I just add sheets of white paper between the screen and scope until I get it dim enough to take longer flats. Its just a matter of how many sheets of paper.

I don't use bias frames. I use dark flats which are slightly more accurate than bias frames - especially for longer flats. Here is an easy way to look at it.

Take lights and darks at the same iso, exposure and temperature.

Take your flats somewhere between say .5 and 5 seconds. This is just an approximation. Then, put the cap on your scope and take the same number of images at the same iso and time that you just used for your flats. These are called flat darks or dark flats. So flat darks are to calibrate flats like darks calibrate lights.

So what APP does is create a master dark flat to calibrate and create a master flat. This master flat is used with the master dark  to calibrate the lights.

There are many references on the net saying the iso of the lights and darks should match the iso of the flats and flat darks. There are also different references saying they don't have to match. Its easy to match them so I do it. I hope this makes sense -if not let me know.

Hi @tim,

Okay, since I never had a problem with using flats of other ISO/gain values as my lights, I wouldn't have a clue about the technical explanation why it wouldn't be okay. But I agree, it's easy to match the light frames and in case of any doubt in this hobby, it's better to take the safe route 😉

I know of some of the Beta testers that use APP (some are very experienced and APOD published) that most use different ISO/gain values for their flats for the mentioned reason. You can increase exposure time of your flats more easily and that will simpy give better quality flats, most problems in flat calibration occur due to too short exposures.

Other ISO/gain for flats works equally well with slow and very fast focal ratios as well. If you are using a fast telescope like a Takashashi Epsilon F/2.8 then you will be happy this simply works. For fast focal ratios things need to be very precise. For instance, this 2 panel mosaic is made with an Takahashi Epsilon at F/2.8 using flats that weren't equal to the light frames by Maurice Toet:

http://www.mauricetoet.nl/Deep-sky/i-qFS6F3J/A

worked perfectly 😉

The problem in this case is, how to make proper flats with an Epsilon (or other fast optics), that is not so easy though.

Mabula

Posted by: 1CM69

Posted by: gdwats@comcast.net

" nothing is wrong with trying to reinvent the wheel". 
Why would you waste your time even trying? Are you going out 
of your way to avoid matching ISO?

Look mate, I assume you’re the type that skim reads & then jumps to conclusions and then posts a useless comment. 

Reread the whole thread & absorb the information, then if you have worthwhile input to share, go ahead and let us all know. 

Cheerio

Can I use a different ISO or gain+offset than my Light frames, for my flat frames?

Yes, that is no problem and actually preferred in most cases. Make your flats with a low ISO or gain+offset to be able to increase the exposure length of the flats. They will be of better quality then. Be aware of the rules you need to follow for flat & light frames. Both need to have the bias pedestal subtracted only once using any of the available options. So if you only use bias frames and no darks. You need 2 master bias frames of the 2 different ISO or gain+offset value to make sure that the bias pedestal is removed correctly (only once) from both lights and flats.

above excerpt taken directly from the first post in this thread.

Hi @gregwrca & @1cm69, yes, do read the whole topic, to get the right context, before posting please;-) and stay nice and respectfull to each other please.

Astrophotography is a difficult and demanding passion 😉 and going too fast usually is not helpfull.

Mabula