15th Feb 2024: Astro Pixel Processor 2.0.0-beta29 released - macOS native File Chooser, macOS CMD-Q fixed, read-only Fits on network fixed and other bug fixes
7th December 2023: added payment option Alipay to purchase Astro Pixel Processor from China, Hong Kong, Macau, Taiwan, Korea, Japan and other countries where Alipay is used.
[Sticky] Data calibration principles/rules - must read !
Mabula, I have no personal experience to show that the flats iso has to be the same as the lights iso. However, when I started in this hobby I used DSS and the DSS FAQ says "The flat frames should be created with the ISO speed of the light frames." http://deepskystacker.free.fr/english/index.html
I've also read this from other people I consider good imagers. I've also read from good imagers that they don't have to match. Based on what DSS says and the difference of opinions among some imagers, I thought it safe to always make them match.
Since this is easy for me to do, I was trying to show the OP an easy way to control his LED output to accomplish the same thing.
Kirk @1cm69 in response to your question on how to know if a flat is well illuminated, or how to see the unstretched linear data in your CR2 frames
I have uploaded the CR2 file of ISO100 with 2,5s exposure time. It's very well illuminated linearly ;-).
Everyone that reads the FAQ can download the CR2 and follow the steps and learn how this works in APP.
I think it's of great use and value that you can see for yourself what the differences are between linear data and the non-linear sRGB data with camera white balance that your camera's display shows 😉
Mabula, I have no personal experience to show that the flats iso has to be the same as the lights iso. However, when I started in this hobby I used DSS and the DSS FAQ says "The flat frames should be created with the ISO speed of the light frames." http://deepskystacker.free.fr/english/index.html
I've also read this from other people I consider good imagers. I've also read from good imagers that they don't have to match. Based on what DSS says and the difference of opinions among some imagers, I thought it safe to always make them match.
Since this is easy for me to do, I was trying to show the OP an easy way to control his LED output to accomplish the same thing.
Yes, I fully understand and I do recall that I have seen it on the DSS site and other places. I think it's very wise to follow the safe road in case of doubt, especially in astrophotography 😉
Perfect and thank you for your assistance in this thread 😉 it's very well appreciated.
You can quote me 😉 on saying: the ISO or gain of the flats does not have to be the same as the iso or gain of the lights.
With a lot of things, technical arguments aside, if it works, it works ! But honestly, I am not aware of any sound technical argument why it shouldn't be like this 😉
( If there actually is one, I need to know so please share ! )
I am in the midst of redoing my Master Calibration frames ready for the up-coming winter months and have a question regarding the preparation of Master Darks.
I have captured a number (minimum of 35) dark frames for each of the exposure times that I normally use when capturing Lights, e.g 60s, 120s, 300s and have loaded these into APP along with Bias and Flat Frames.
I was expecting APP to provide me with a Master Dark for each of these different exposure times as this was what I have used in the past. However APP v1.062 is loading all the available Darks and producing only a single consolidated Master Dark with an exposure time shown as equal to the longest of the Darks that I loaded into the run.
I thought this result might have depended on whether the 'scale MasterDark' option was checked but I've tried both possibilities and still get the same result.
Having observed this behaviour, my questions are:
a) Do you still recommend (where time permits) the preparation of separate MasterDarks for each different exposure time used when capturing Lights (OR does MasterDark scaling take care of this)?
b) If use of separate MasterDark per exposure time is still recommended, how do I do this? Have I missed a setting when initiating my Master Calibration frames preparation run, or do I need to load Darks for only a single exposure interval at a time.
c) Lastly, while experimenting, I've noticed that I get a somewhat different Bad Pixel Map result depending on the duration of the Dark Frames I use during preparation - I say this after looking at only a top corner of two BPMs and seeing different hot pixels, so not an in-depth analysis. But this difference being so, it occurs to me to ask what would be the best darks exposure time to use in the preparation of my Bad Pixel Map, e.g one equal to the longest or shortest exposure times I normally used when capturing Lights? Or perhaps a MasterDark that is produced by amalgamating Darks with several different exposure times, which is the Master Dark result I seem to be getting at the moment?
Ok... so if I follow 'the rules' for creating the calibration files, let's assume now have: Lights, Darks, Flats, Dark Flats, Bias & BPM
Are ALL of these needed, for the 'best' result?
Assuming I do need them all: Do I just LOAD each type and then (for simplicity's sake skip over all the options) go straight to INTEGRATE? Will APP will create the all masters and properly apply them?
I'm hoping the answer is YES. This is what I've been doing for months now...
I am in the midst of redoing my Master Calibration frames ready for the up-coming winter months and have a question regarding the preparation of Master Darks.
I have captured a number (minimum of 35) dark frames for each of the exposure times that I normally use when capturing Lights, e.g 60s, 120s, 300s and have loaded these into APP along with Bias and Flat Frames.
I was expecting APP to provide me with a Master Dark for each of these different exposure times as this was what I have used in the past. However APP v1.062 is loading all the available Darks and producing only a single consolidated Master Dark with an exposure time shown as equal to the longest of the Darks that I loaded into the run.
I thought this result might have depended on whether the 'scale MasterDark' option was checked but I've tried both possibilities and still get the same result.
Having observed this behaviour, my questions are:
a) Do you still recommend (where time permits) the preparation of separate MasterDarks for each different exposure time used when capturing Lights (OR does MasterDark scaling take care of this)?
b) If use of separate MasterDark per exposure time is still recommended, how do I do this? Have I missed a setting when initiating my Master Calibration frames preparation run, or do I need to load Darks for only a single exposure interval at a time.
c) Lastly, while experimenting, I've noticed that I get a somewhat different Bad Pixel Map result depending on the duration of the Dark Frames I use during preparation - I say this after looking at only a top corner of two BPMs and seeing different hot pixels, so not an in-depth analysis. But this difference being so, it occurs to me to ask what would be the best darks exposure time to use in the preparation of my Bad Pixel Map, e.g one equal to the longest or shortest exposure times I normally used when capturing Lights? Or perhaps a MasterDark that is produced by amalgamating Darks with several different exposure times, which is the Master Dark result I seem to be getting at the moment?
First of all, please accept my apologies for my delayed response ;-(
Do you still recommend (where time permits) the preparation of separate MasterDarks for each different exposure time used when capturing Lights (OR does MasterDark scaling take care of this)?
If you are able practically due to time constraints and temperature matching to prepare good masters of a fixed exposure duration, by all means do that 😉
On the other hand, dark-frame scaling is rather robust now in APP and I have used it myself on lots of data now and it works very well. With Dark frame scaling, you can simply create 1 very good masterdark of let's say 200 darks with a long exposure time and use that always besides a good Masterbias. A masterbias is required for dark frame scaling to work.
It also depends on the sensor if you can use dark frame scaling to good effect. For instance, the zwo asi 183 cmos camera has severe amp-glow, in that case, dark frame scaling will lead to worse results since the amp-glow can't be properly scaled. This is due to it's non-linear behaviour with respect to exposure time and temperature. So with such a camera, you will need to create proper darks of fixed temperature and exposure time.
"I was expecting APP to provide me with a Master Dark for each of these different exposure times as this was what I have used in the past. However APP v1.062 is loading all the available Darks and producing only a single consolidated Master Dark with an exposure time shown as equal to the longest of the Darks that I loaded into the run."
Yes, this used to be the case, it has changed since the introduction of dark-frame scaling. I (me, myself), for instance, load 100s of darks of different exposures and temperatures for my dslr data and use that combined Masterdark with dark frame scaling to very good effect.
If use of separate MasterDark per exposure time is still recommended, how do I do this? Have I missed a setting when initiating my Master Calibration frames preparation run, or do I need to load Darks for only a single exposure interval at a time.
If you still want to create masters that are separated in exposure time, load them as different sessions in the multi-session mode and do the same for the lights ;-). Or simply make the masters one at at time.When loading the lights, APP will match the right Masterdark to the correct light frames based on both ISO/gain and exposure time.
By the way, Darks will still be discriminated if they have a different ISO or gain value.
Lastly, while experimenting, I've noticed that I get a somewhat different Bad Pixel Map result depending on the duration of the Dark Frames I use during preparation - I say this after looking at only a top corner of two BPMs and seeing different hot pixels, so not an in-depth analysis. But this difference being so, it occurs to me to ask what would be the best darks exposure time to use in the preparation of my Bad Pixel Map, e.g one equal to the longest or shortest exposure times I normally used when capturing Lights? Or perhaps a MasterDark that is produced by amalgamating Darks with several different exposure times, which is the Master Dark result I seem to be getting at the moment?
For the creation of a good Bad Pixel Map, I would recommend to use very long dark frames. Mixing darks of different exposure times should not downgrade the result. Try to have a lot of darks with good exposure times, at least a couple of minutes 😉
The purpose here is to have a good measurement of all the non-linear artefacts on the sensor, like the hot pixels. By using more darks that have longer exposure times, the bad pixels will be more apparent. You can even create the darks at much higher temperatures for this purpose, since higher temperatures will show non-linear beahviour of pixels faster...
Ok... so if I follow 'the rules' for creating the calibration files, let's assume now have: Lights, Darks, Flats, Dark Flats, Bias & BPM
Are ALL of these needed, for the 'best' result?
Assuming I do need them all: Do I just LOAD each type and then (for simplicity's sake skip over all the options) go straight to INTEGRATE? Will APP will create the all masters and properly apply them?
I'm hoping the answer is YES. This is what I've been doing for months now...
Hi @cheetah, this is actually a non-trivial question. And very hard 😉 to give a correct answer to, because it really depends on the sensor characteristics in your camera.
For some camera's it will be optimal, for others it won't. What kind of camera or camera's are you using?
Assuming I do need them all: Do I just LOAD each type and then (for simplicity's sake skip over all the options) go straight to INTEGRATE? Will APP will create the all masters and properly apply them?
I'm hoping the answer is YES. This is what I've been doing for months now...
Yes, with APP's new calibration engine, since APP 1.062, APP will do everything on auto-pilot and correctly. All masters will be used and properly applied 😉
If it's non-trivial, perhaps it would be better to create a master list of cameras/sensors and the best way to calibrate for each? So 1,000 users don't write you, all requesting the same thing. 🙂
It seems the more I read about calibration images, the more confused I get. Would be great to get your expert opinion, for (idiot) users like me...
I am going to be dipping my toe in to imaging using my ZWOASI120MM-S camera & will be capturing data via SharpCap.
What I have noticed with my test images that I have made is that SharpCap does not write any information to the head of the file i.e. there is no Gain or Temperature etc... the limit seems to be image dimension and exposure length that is written.
Having previously captured data with a DSLR where all the necessary info is written to the file, how does APP cope when this information is lacking?
How does APP know which Darks match which Lights for example if Temperature & Gain is not present in the file?
If it's non-trivial, perhaps it would be better to create a master list of cameras/sensors and the best way to calibrate for each? So 1,000 users don't write you, all requesting the same thing. 🙂
It seems the more I read about calibration images, the more confused I get. Would be great to get your expert opinion, for (idiot) users like me...
An entire list with camera sensors regarding calibration workflow/paths is not a solution I think for several reasons.
camera's of the same model could behave differently, especially CCD camera's.
the age of a sensor could be a factor, especially for CCD's
some camera models actually have different versions (camera's sometimes get a firmware upgrade which could change amp-glow present or not )
the exposure time of your lights could warrent a different workflow, longer exposures could show non-linear problems like amp-glow which then warrents a different workflow.
I think it's good that a photographer learns about the sensor in his camera and thinks about how to exploit it 😉 as best as possible
probably there are more arguments
So a list would make the illusion that for a fixed camera model the most optimal workflow would always be the same. Unfortunately, it doesn't work like that.
Your camera, the Zwo Asi1600mc-p, has a pretty new CMOS sensor. I have the asi1600mm-c first version myself.
This camera, dus to it's cmos technology seems to give the best calibration in most cases if you use:
darks and flats for lights. Try to match the darks for gain and exposure time as good as you can to your lights.
flat darks for the flats, again try to match the darkFlats for gain and exposure time as good as you can to your flats.
No BIAS ! it seems that bias frames are not very good and this has to do with the sensor readout technology. Omitting bias and using dark flats for the flat calibration is giving better results for most asi-1600 camera's as far as I know.
The asi-1600 does have a little amp-glow, so therefore you don't want to use dark-frame scaling if you can prevent it. Therefore, create darks and flatdarks that properly match the frames to calibrate 😉
Mabula
This post was modified 6 years ago by Mabula-Admin
I am going to be dipping my toe in to imaging using my ZWOASI120MM-S camera & will be capturing data via SharpCap.
What I have noticed with my test images that I have made is that SharpCap does not write any information to the head of the file i.e. there is no Gain or Temperature etc... the limit seems to be image dimension and exposure length that is written.
Having previously captured data with a DSLR where all the necessary info is written to the file, how does APP cope when this information is lacking?
How does APP know which Darks match which Lights for example if Temperature & Gain is not present in the file?
I don't use SharpCap myself, but I have seen some data captured with SharpCap. I think I did see gain and exposure information in the fits header. Maybe ShaprCap was upgraded to have it in the header?
If there is no information, APP will still be able to process it, since APP will automatically use Masters if present allthough gain and exposure time is not known. You will be resposible for knowing how to match it all then if you start to create frames with different exposure times and gain values.
Please let me know if you are able to process the files correctly in APP 😉
This camera, due to it's cmos technology seems to give the best calibration in most cases if you use:
darks and flats for lights. Try to match the darks for gain and exposure time as good as you can to your lights.
flat darks for the flats, again try to match the darkFlats for gain and exposure time as good as you can to your flats.
No BIAS ! it seems that bias frames are not very good and this has to do with the sensor readout technology. Omitting bias and using dark flats for the flat calibration is giving better results for most asi-1600 camera's as far as I know.
Thank you! I'm quite surprised by the -no bias- portion of that.
Do I need to calibrate the flats with the dark flats first... and THEN load the lights and calibrate with flats and darks? Or can I simply load everything (lights, darks, dark flats and flats) and have APP do it all in 1 step?
I have been playing around with SharpCap & also APT (Astro Photography Tool) saving images as .fits files in both programs.
APT seems to add more info to the header but I think that I have figured out what to do.
Instead of being able to load all raw calibration files and lights in to APP at a single time and letting APP work it's magic with a single button press, it looks as though I need to process each matching batch of calibration files individually to produce their own Masters.
Then I add the Lights along with the newly generated Masters and let APP take over from there.
What I have set up when capturing all RAW data is to add strings to the filenames such as Gain, Exposure Length etc.. just so as I know what matches to what.
This is probably how others do it as well I would assume.
I use ACP to take dawn sky flats, so my flats vary between 0.5 and 15 seconds (not for the same filter - L is at the low end of this and S at the high end). Flats for a single filter may vary by as much as 5 to 10 seconds.
To create a dark library to use flat-darks that match closely would require a lot of darks! This is for an ASI1600MM Pro.
Any idea how much variation would be acceptable between the flat time and the "matching" dark? Could I use dark scaling?
Indeed, that would require a lot of darks. I know that for the asi1600mm-pro and most other CMOS camera's you get better results with darkflats.
My adivce would be to create 1 MasterDarkFlat with a suitable exposure time with respect to the mean exposure time of all of our flats. And then apply dark scaling on the masterdarkflat. To be able to do this, you will need to add a suitable Masterbias as well, because scaling can only be done on the dark current signal and noise, so the bias signal must be known.
Now, since a real bias of the asi1600mm-pro (real being, a bias shot with the shortest exposure time possible) is not good. I advice you to create a Masterbias with a bit longer exposure time. Use 0.1 seconds instead of 0.001 seconds. The resulting Masterbias will be better and it will hardly be affected by dark current since this is stil a very short exposure time.
In 2) calibrate, you can enable darkflat scaling. Check the integration options for a MasterDarkFlat.
Thanks for the great suggestions! I will give this a try and let you know how it works.
You actually triggered one question. Does APP's dark scaling work both for larger and smaller exposure lengths? I seem to reading about some other software (not APP) where dark scaling only worked for shorter exposures, and the recommendation was to use the longest exposure you needed if you were going to dark scale to smaller values.
The implementation in APP is probably rather differently than most other applications, since APP is using more than 1 factor in scaling ! Other applications only use 1 factor.
To be able to do this, the algorithm that finds the scaling factor(s) must be very robust and it must work in lots of different cases.
Does APP's dark scaling work both for larger and smaller exposure lengths? I seem to reading about some other software (not APP) where dark scaling only worked for shorter exposures, and the recommendation was to use the longest exposure you needed if you were going to dark scale to smaller values.
I am not aware of this issue and I have never encountered such a problem with the dark scaling in APP or read about such a problem in literature. So I think it is very likely an implementation issue of that particular application. However, if you think that scaling is not working well in some cases, please do let me know so I can test it 😉
Thanks for the quick replay! It answers it mostly. When I load the MasterDarkFlat do I assign it to all channels then? And the same question for when I load the MasterBaus?
Thanks for the quick replay! It answers it mostly. When I load the MasterDarkFlat do I assign it to all channels then? And the same question for when I load the MasterBaus?
Well the MasterBias would normally be loaded for all channels, so Yes, the bias is a characteristic of the sensor of your camera only 😉
And Yes, if you want the MasterDarkFlat to be used to calibrate all flats of all channels, you will need to assign it to all channels. I think that is in most cases what you want to do 😉
My adivce would be to create 1 MasterDarkFlat with a suitable exposure time with respect to the mean exposure time of all of our flats. And then apply dark scaling on the masterdarkflat. To be able to do this, you will need to add a suitable Masterbias as well, because scaling can only be done on the dark current signal and noise, so the bias signal must be known.
Now, since a real bias of the asi1600mm-pro (real being, a bias shot with the shortest exposure time possible) is not good. I advice you to create a Masterbias with a bit longer exposure time. Use 0.1 seconds instead of 0.001 seconds. The resulting Masterbias will be better and it will hardly be affected by dark current since this is stil a very short exposure time.
In 2) calibrate, you can enable darkflat scaling. Check the integration options for a MasterDarkFlat.
Hi Mabula -
I created a 5 second MasterDark to use as a Dark Flat and a 0.1 second MasterBias.
I loaded my Flats, 300 second Lights, a 300 second MasterDark, the 0.1 second MasterBias, and the 5 second MasterDarkFlat.
I set dark scaling on for the MasterDarkFlat.
I ran Calibrate and the results look fine, but I noticed when it finished, the Lights were showing that they were calibrated with the MasterBias.
Here's what the Master section of the file list looks like after running Calibration:
Is this OK, or am I not doing something correctly?
My adivce would be to create 1 MasterDarkFlat with a suitable exposure time with respect to the mean exposure time of all of our flats. And then apply dark scaling on the masterdarkflat. To be able to do this, you will need to add a suitable Masterbias as well, because scaling can only be done on the dark current signal and noise, so the bias signal must be known.
Now, since a real bias of the asi1600mm-pro (real being, a bias shot with the shortest exposure time possible) is not good. I advice you to create a Masterbias with a bit longer exposure time. Use 0.1 seconds instead of 0.001 seconds. The resulting Masterbias will be better and it will hardly be affected by dark current since this is stil a very short exposure time.
In 2) calibrate, you can enable darkflat scaling. Check the integration options for a MasterDarkFlat.
Hi Mabula -
I created a 5 second MasterDark to use as a Dark Flat and a 0.1 second MasterBias.
I loaded my Flats, 300 second Lights, a 300 second MasterDark, the 0.1 second MasterBias, and the 5 second MasterDarkFlat.
I set dark scaling on for the MasterDarkFlat.
I ran Calibrate and the results look fine, but I noticed when it finished, the Lights were showing that they were calibrated with the MasterBias.
Here's what the Master section of the file list looks like after running Calibration:
Is this OK, or am I not doing something correctly?
Regarding the MasterBias, if the masterbias of 0.1 seconds that you created for the dark frame scaling has the same offset and gain as the lights then that MasterBias should be fine for using with your lights as well. If the offset and/or gain are different than the lights then it could create suboptimal calibration off course.
To be sure, what you could do in this case. First create the master flats separately with the MasterBias and the MasterDarkFlat, then you won't need to load the MasterBias for the light frame calibration 😉
OK, I will use that. I was surprised that the MasterFlat did not show the MasterDarkFlat as being used to calibrate it in the file list. It only showed Bias. That's what made me think I was doing something wrong.
The marks of which Masters are used, will not show next to a MasterFlat, because the MasterFlat itself is not calibrated by any master ;-).
The flats are calibrated with the appropriate masters internally and APP will warn if there is a problem with flat calibration.
It's probably best I think to add this information(which Masters were used to calibrate the individual flats?) in the fits header of the Masters. Would you agree?
I think adding that to the FITS header would certainly not hurt - and might even save you some support questions 🙂 !
The process of calibrating with FlatDarks is different enough from the process documented all over the Internet that maybe a short video tutorial would be useful too, with so many people using the ASI1600 and similar CMOS cameras that benefit from the FlatDark / no bias processing. Especially with image acquisition products like ACP, Starkeeper.it Voyager, and CCD Commander among others that change the exposure length on the fly to reach a desired ADU for flats.
I have read this thread a few times trying to get all into my head. I made myself a step by step procedure, but I would like to run it with you so see if I have got it all. For ASI 1600 MM-C it goes like this:
1. Take flat frames for each filter. 2. Take dark flats.(Match flats time, bias, offset, (and temperature ?)). 3. Take Bias frames, 0,1 sek (?), 400 of.
4. Generate Master flats using 1, 2 and 3, one for each filter.
5. Take lights 6. Take darks (Match lights time, bias, offset and temperature) 7. Create Master darks as required by 6.
8. Calibrate lights using 4, 5 and 7.
I'm not sure how much temperature matters for flats, dark flats and bias frames, but keeping it the same cannot hurt.
I read on Cloudy Nights that bias frame length should be 0,2 sec for ASI 1600. (Jon Rista). And then, if your flats and then your dark flats are of similar length you could use the dark flat as bias frames also. I may have misunderstood the last bit.
Comments to this please?
Helge
Edit: A bad piksel map could be useful, but should it be used with both flat/dark flat process and lights calibration?
