Share:

[Sticky] Data calibration principles/rules - must read !  

Page 3 / 4
  RSS

(@cheetah)
Main Sequence Star Customer
Joined: 2 years ago
Posts: 30
October 2, 2018 19:04  

If it's non-trivial, perhaps it would be better to create a master list of cameras/sensors and the best way to calibrate for each?  So 1,000 users don't write you, all requesting the same thing.  🙂 

It seems the more I read about calibration images, the more confused I get.  Would be great to get your expert opinion, for (idiot) users like me...

In my case, I use an Zwo Asi1600mc-p.  

 

 

 


ReplyQuote
(@1cm69)
Neutron Star Customer
Joined: 2 years ago
Posts: 132
October 4, 2018 16:59  

Hi Mabula,

I am going to be dipping my toe in to imaging using my ZWOASI120MM-S camera & will be capturing data via SharpCap.

What I have noticed with my test images that I have made is that SharpCap does not write any information to the head of the file i.e. there is no Gain or Temperature etc... the limit seems to be image dimension and exposure length that is written.

Having previously captured data with a DSLR where all the necessary info is written to the file, how does APP cope when this information is lacking?

How does APP know which Darks match which Lights for example if Temperature & Gain is not present in the file?

 

Regards..,

Kirk


ReplyQuote
(@mabula-admin)
Quasar Admin
Joined: 2 years ago
Posts: 2128
October 14, 2018 21:13  
Posted by: Cheetah

If it's non-trivial, perhaps it would be better to create a master list of cameras/sensors and the best way to calibrate for each?  So 1,000 users don't write you, all requesting the same thing.  🙂 

It seems the more I read about calibration images, the more confused I get.  Would be great to get your expert opinion, for (idiot) users like me...

In my case, I use an Zwo Asi1600mc-p.  

Hi @cheetah,

An entire list with camera sensors regarding calibration workflow/paths is not a solution I think for several reasons.

  • camera's of the same model could behave differently, especially CCD camera's.
  • the age of a sensor could be a factor, especially for CCD's
  • some camera models actually have different versions (camera's sometimes get a firmware upgrade which could change amp-glow present or not )
  • the exposure time of your lights could warrent a different workflow, longer exposures could show non-linear problems like amp-glow which then warrents a different workflow.
  • I think it's good that a photographer learns about the sensor in his camera and thinks about how to exploit it 😉 as best as possible
  • probably there are more arguments

So a list would make the illusion that for a fixed camera model the most optimal workflow would always be the same. Unfortunately, it doesn't work like that.

Your camera, the Zwo Asi1600mc-p, has a pretty new CMOS sensor. I have the asi1600mm-c first version myself.

This camera, dus to it's cmos technology seems to give the best calibration in most cases if you use:

  • darks and flats for lights. Try to match the darks for gain and exposure time as good as you can to your lights.
  • flat darks for the flats, again try to match the darkFlats for gain and exposure time as good as you can to your flats.
  • No BIAS ! it seems that bias frames are not very good and this has to do with the sensor readout technology. Omitting bias and using dark flats for the flat calibration is giving better results for most asi-1600 camera's as far as I know.

The asi-1600 does have a little amp-glow, so therefore you don't want to use dark-frame scaling if you can prevent it. Therefore, create darks and flatdarks that properly match the frames to calibrate 😉

Mabula

This post was modified 11 months ago by Mabula Haverkamp - Admin

Main developer of Astro Pixel Processor and owner of Aries Productions


ReplyQuote
(@mabula-admin)
Quasar Admin
Joined: 2 years ago
Posts: 2128
October 14, 2018 21:19  
Posted by: 1CM69

Hi Mabula,

I am going to be dipping my toe in to imaging using my ZWOASI120MM-S camera & will be capturing data via SharpCap.

What I have noticed with my test images that I have made is that SharpCap does not write any information to the head of the file i.e. there is no Gain or Temperature etc... the limit seems to be image dimension and exposure length that is written.

Having previously captured data with a DSLR where all the necessary info is written to the file, how does APP cope when this information is lacking?

How does APP know which Darks match which Lights for example if Temperature & Gain is not present in the file?

 

Regards..,

Kirk

Hi Kirk @1cm69,

I don't use SharpCap myself, but I have seen some data captured with SharpCap. I think I did see gain and exposure information in the fits header. Maybe ShaprCap was upgraded to have it in the header?

If there is no information, APP will still be able to process it, since APP will automatically use Masters if present allthough gain and exposure time is not known. You will be resposible for knowing how to match it all then if you start to create frames with different exposure times and gain values.

Please let me know if you are able to process the files correctly in APP 😉

Mabula

Main developer of Astro Pixel Processor and owner of Aries Productions


ReplyQuote
(@cheetah)
Main Sequence Star Customer
Joined: 2 years ago
Posts: 30
October 14, 2018 22:21  

This camera, due to it's cmos technology seems to give the best calibration in most cases if you use:

  • darks and flats for lights. Try to match the darks for gain and exposure time as good as you can to your lights.
  • flat darks for the flats, again try to match the darkFlats for gain and exposure time as good as you can to your flats.
  • No BIAS ! it seems that bias frames are not very good and this has to do with the sensor readout technology. Omitting bias and using dark flats for the flat calibration is giving better results for most asi-1600 camera's as far as I know.

 

Thank you!  I'm quite surprised by the -no bias- portion of that.

Do I need to calibrate the flats with the dark flats first...  and THEN load the lights and calibrate with flats and darks?  Or can I simply load everything (lights, darks, dark flats and flats) and have APP do it all in 1 step?

 

 

 


ReplyQuote
(@1cm69)
Neutron Star Customer
Joined: 2 years ago
Posts: 132
October 15, 2018 19:59  

Hi Mabula,

thanks for replying.

I have been playing around with SharpCap & also APT (Astro Photography Tool) saving images as .fits files in both programs.

APT seems to add more info to the header but I think that I have figured out what to do.

Instead of being able to load all raw calibration files and lights in to APP at a single time and letting APP work it's magic with a single button press, it looks as though I need to process each matching batch of calibration files individually to produce their own Masters.

Then I add the Lights along with the newly generated Masters and let APP take over from there.

What I have set up when capturing all RAW data is to add strings to the filenames such as Gain, Exposure Length etc.. just so as I know what matches to what.

This is probably how others do it as well I would assume.

Regards..,

Kirk  😉 


ReplyQuote
(@rowland-f-archer-jr)
Red Giant Customer
Joined: 2 years ago
Posts: 71
January 16, 2019 00:07  

I use ACP to take dawn sky flats, so my flats vary between 0.5 and 15 seconds (not for the same filter - L is at the low end of this and S at the high end).  Flats for a single filter may vary by as much as 5 to 10 seconds.

To create a dark library to use flat-darks that match closely would require a lot of darks!  This is for an ASI1600MM Pro.

Any idea how much variation would be acceptable between the flat time and the "matching" dark?  Could I use dark scaling?

Thanks!

Rowland


ReplyQuote
(@mabula-admin)
Quasar Admin
Joined: 2 years ago
Posts: 2128
January 17, 2019 12:53  

Dear Rownland @rowland-f-archer-jr,

Indeed, that would require a lot of darks. I know that for the asi1600mm-pro and most other CMOS camera's you get better results with darkflats.

My adivce would be to create 1 MasterDarkFlat with a suitable exposure time with respect to the mean exposure time of all of our flats. And then apply dark scaling on the masterdarkflat. To be able to do this, you will need to add a suitable Masterbias as well, because scaling can only be done on the dark current signal and noise, so the bias signal must be known.

Now, since a real bias of the asi1600mm-pro (real being, a bias shot with the shortest exposure time possible) is not good. I advice you to create a Masterbias with a bit longer exposure time. Use 0.1 seconds instead of 0.001 seconds. The resulting Masterbias will be better and it will hardly be affected by dark current since this is stil a very short exposure time.

In 2) calibrate, you can enable darkflat scaling. Check the integration options for a MasterDarkFlat.

Please let me know if this helps 😉

Kind regards,

Mabula

Main developer of Astro Pixel Processor and owner of Aries Productions


ReplyQuote
(@rowland-f-archer-jr)
Red Giant Customer
Joined: 2 years ago
Posts: 71
January 17, 2019 13:05  

Hi Mabula,

Thanks for the great suggestions!  I will give this a try and let you know how it works.

You actually triggered one question.  Does APP's dark scaling work both for larger and smaller exposure lengths?  I seem to reading about some other software (not APP) where dark scaling only worked for shorter exposures, and the recommendation was to use the longest exposure you needed if you were going to dark scale to smaller values.

Thanks again,

Rowland


ReplyQuote
(@mabula-admin)
Quasar Admin
Joined: 2 years ago
Posts: 2128
January 17, 2019 18:49  

Hi Rowland @rowland-f-archer-jr,

Dark scaling is very tricky to implement well.

The implementation in APP is probably rather differently than most other applications, since APP is using more than 1 factor in scaling ! Other applications only use 1 factor.

To be able to do this, the algorithm that finds the scaling factor(s) must be very robust and it must work in lots of different cases.

Does APP's dark scaling work both for larger and smaller exposure lengths? I seem to reading about some other software (not APP) where dark scaling only worked for shorter exposures, and the recommendation was to use the longest exposure you needed if you were going to dark scale to smaller values.

I am not aware of this issue and I have never encountered such a problem with the dark scaling in APP or read about such a problem in literature. So I think it is very likely an implementation issue of that particular application. However, if you think that scaling is not working well in some cases, please do let me know so I can test it 😉

Mabula

Main developer of Astro Pixel Processor and owner of Aries Productions


ReplyQuote
(@whixson)
Neutron Star Customer
Joined: 12 months ago
Posts: 100
January 17, 2019 18:58  

Thanks for the quick replay! It answers it mostly. When I load the MasterDarkFlat do I assign it to all channels then? And the same question for when I load the MasterBaus?


ReplyQuote
(@rowland-f-archer-jr)
Red Giant Customer
Joined: 2 years ago
Posts: 71
January 17, 2019 20:24  

Hi Mabula -- OK, I haven't seen the problem, just recall reading it.  I'll get some 5 second darks and 0.1 second bias frames and give it a try.

 

Cheers,

Rowland


ReplyQuote
(@mabula-admin)
Quasar Admin
Joined: 2 years ago
Posts: 2128
January 18, 2019 13:41  
Posted by: Whixson

Thanks for the quick replay! It answers it mostly. When I load the MasterDarkFlat do I assign it to all channels then? And the same question for when I load the MasterBaus?

@whixson,

Well the MasterBias would normally be loaded for all channels, so Yes, the bias is a characteristic of the sensor of your camera only 😉

And Yes, if you want the MasterDarkFlat to be used to calibrate all flats of all channels, you will need to assign it to all channels. I think that is in most cases what you want to do 😉

Cheers,

Mabula

Main developer of Astro Pixel Processor and owner of Aries Productions


ReplyQuote
(@mabula-admin)
Quasar Admin
Joined: 2 years ago
Posts: 2128
January 18, 2019 13:42  
Posted by: Rowland F Archer Jr

Hi Mabula -- OK, I haven't seen the problem, just recall reading it.  I'll get some 5 second darks and 0.1 second bias frames and give it a try.

 

Cheers,

Rowland

Okay @rowland-f-archer-jr , thanks !

Main developer of Astro Pixel Processor and owner of Aries Productions


ReplyQuote
(@rowland-f-archer-jr)
Red Giant Customer
Joined: 2 years ago
Posts: 71
January 25, 2019 22:08  
Posted by: Mabula Haverkamp - Admin

My adivce would be to create 1 MasterDarkFlat with a suitable exposure time with respect to the mean exposure time of all of our flats. And then apply dark scaling on the masterdarkflat. To be able to do this, you will need to add a suitable Masterbias as well, because scaling can only be done on the dark current signal and noise, so the bias signal must be known.

Now, since a real bias of the asi1600mm-pro (real being, a bias shot with the shortest exposure time possible) is not good. I advice you to create a Masterbias with a bit longer exposure time. Use 0.1 seconds instead of 0.001 seconds. The resulting Masterbias will be better and it will hardly be affected by dark current since this is stil a very short exposure time.

In 2) calibrate, you can enable darkflat scaling. Check the integration options for a MasterDarkFlat.

Hi Mabula - 

I created a 5 second MasterDark to use as a Dark Flat and a 0.1 second MasterBias.  

I loaded my Flats, 300 second Lights, a 300 second MasterDark, the 0.1 second MasterBias, and the 5 second MasterDarkFlat.

I set dark scaling on for the MasterDarkFlat.  

I ran Calibrate and the results look fine, but I noticed when it finished, the Lights were showing that they were calibrated with the MasterBias.

light calibration

Here's what the Master section of the file list looks like after running Calibration:

master dark flat

Is this OK, or am I not doing something correctly?

Thanks,

Rowland

 


ReplyQuote
(@mabula-admin)
Quasar Admin
Joined: 2 years ago
Posts: 2128
January 27, 2019 21:44  
Posted by: Rowland F Archer Jr
Posted by: Mabula Haverkamp - Admin

My adivce would be to create 1 MasterDarkFlat with a suitable exposure time with respect to the mean exposure time of all of our flats. And then apply dark scaling on the masterdarkflat. To be able to do this, you will need to add a suitable Masterbias as well, because scaling can only be done on the dark current signal and noise, so the bias signal must be known.

Now, since a real bias of the asi1600mm-pro (real being, a bias shot with the shortest exposure time possible) is not good. I advice you to create a Masterbias with a bit longer exposure time. Use 0.1 seconds instead of 0.001 seconds. The resulting Masterbias will be better and it will hardly be affected by dark current since this is stil a very short exposure time.

In 2) calibrate, you can enable darkflat scaling. Check the integration options for a MasterDarkFlat.

Hi Mabula - 

I created a 5 second MasterDark to use as a Dark Flat and a 0.1 second MasterBias.  

I loaded my Flats, 300 second Lights, a 300 second MasterDark, the 0.1 second MasterBias, and the 5 second MasterDarkFlat.

I set dark scaling on for the MasterDarkFlat.  

I ran Calibrate and the results look fine, but I noticed when it finished, the Lights were showing that they were calibrated with the MasterBias.

light calibration

Here's what the Master section of the file list looks like after running Calibration:

master dark flat

Is this OK, or am I not doing something correctly?

Thanks,

Rowland

 

Hi Rowland @rowland-f-archer-jr,

I think you are okay here 😉

Regarding the MasterBias, if the masterbias of 0.1 seconds that you created for the dark frame scaling has the same offset and gain as the lights then that MasterBias should be fine for using with your lights as well. If the offset and/or gain are different than the lights then it could create suboptimal calibration off course.

To be sure, what you could do in this case. First create the master flats separately with the MasterBias and the MasterDarkFlat, then you won't need to load the MasterBias for the light frame calibration 😉

Let me know if this is clear 😉

Mabula

 

Main developer of Astro Pixel Processor and owner of Aries Productions


ReplyQuote
(@rowland-f-archer-jr)
Red Giant Customer
Joined: 2 years ago
Posts: 71
January 28, 2019 11:30  

Hi Mabula,

OK, I will use that.  I was surprised that the MasterFlat did not show the MasterDarkFlat as being used to calibrate it in the file list.  It only showed Bias.  That's what made me think I was doing something wrong.

Cheers,

Rowland


ReplyQuote
(@mabula-admin)
Quasar Admin
Joined: 2 years ago
Posts: 2128
January 28, 2019 22:55  

Hi Rowland @rowland-f-archer-jr,

The marks of which Masters are used, will not show next to a MasterFlat, because the MasterFlat itself is not calibrated by any master ;-).

The flats are calibrated with the appropriate masters internally and APP will warn if there is a problem with flat calibration.

It's probably best I think to add this information(which Masters were used to calibrate the individual flats?) in the fits header of the Masters. Would you agree?

Mabula

Main developer of Astro Pixel Processor and owner of Aries Productions


ReplyQuote
(@rowland-f-archer-jr)
Red Giant Customer
Joined: 2 years ago
Posts: 71
January 28, 2019 23:18  

Hi Mabula,

I think adding that to the FITS header would certainly not hurt - and might even save you some support questions 🙂 !

The process of calibrating with FlatDarks is different enough from the process documented all over the Internet that maybe a short video tutorial would be useful too, with so many people using the ASI1600 and similar CMOS cameras that benefit from the FlatDark / no bias processing.  Especially with image acquisition products like ACP, Starkeeper.it Voyager, and CCD Commander among others that change the exposure length on the fly to reach a desired ADU for flats.  

I think I'm finally getting it straight!

Thanks,

Rowland

 


ReplyQuote
 Heno
(@heno)
White Dwarf Customer
Joined: 2 years ago
Posts: 19
February 2, 2019 18:23  

Hi Mabula

I have read this thread a few times trying to get all into my head. I made myself a step by step procedure, but I would like to run it with you so see if I have got it all. For ASI 1600 MM-C it goes like this:

1. Take flat frames for each filter.
2. Take dark flats.(Match flats time, bias, offset, (and temperature ?)).
3. Take Bias frames, 0,1 sek (?), 400 of.

4. Generate Master flats using 1, 2 and 3, one for each filter.

5. Take lights
6. Take darks (Match lights time, bias, offset and temperature)
7. Create Master darks as required by 6.

8. Calibrate lights using 4, 5 and 7.

I'm not sure how much temperature matters for flats, dark flats and bias frames, but keeping it the same cannot hurt.

I read on Cloudy Nights that bias frame length should be 0,2 sec for ASI 1600. (Jon Rista). And then, if your flats and then your dark flats are of similar length you could use the dark flat as bias frames also. I may have misunderstood the last bit.

Comments to this please?

Helge

Edit: A bad piksel map could be useful, but should it be used with both flat/dark flat process and lights calibration?

This post was modified 8 months ago 2 times by Heno

ReplyQuote
Page 3 / 4
Share: