Share:

[Sticky] Data calibration principles/rules - must read !  

Page 4 / 4
  RSS

(@mabula-admin)
Quasar Admin
Joined: 2 years ago
Posts: 2081
February 5, 2019 15:39  
Posted by: Rowland F Archer Jr

Hi Mabula,

I think adding that to the FITS header would certainly not hurt - and might even save you some support questions 🙂 !

The process of calibrating with FlatDarks is different enough from the process documented all over the Internet that maybe a short video tutorial would be useful too, with so many people using the ASI1600 and similar CMOS cameras that benefit from the FlatDark / no bias processing.  Especially with image acquisition products like ACP, Starkeeper.it Voyager, and CCD Commander among others that change the exposure length on the fly to reach a desired ADU for flats.  

I think I'm finally getting it straight!

Thanks,

Rowland

 

Hi Rowland @rowland-f-archer-jr,

I will add the information to the masters and integrations 😉 a.s.a.p.

I have done a big rebuild of the calibration engine last year, to make this very easy, at least, I think... 

Technically speaking, calibration of flats with flatdarks is not different than calibration of lights with darks, right? The same rules apply.

APP will automatically calibrate your flats with bias and/or flat darks if these are provided, since without them, flat calibration can't work because the bias pedestal always needs to be subtracted to get proper flat-field calibration. APP makes sure that this happens. If bias or flatdarks aren't there, the user is ever warned, that he will probably face a problem. And then there is always the console, you can see exactly what is happening with your flats or lights in the console:

15:31:07 - IMAGE VIEWER: loading into Linear Calibrated Image Loader: C:\Users\Aries Productions\Documents\AllTypesOfImages\astroShots\CocoonNebula Lights\Lights\DSC_0026.NEF
15:31:07 - 2) CALIBRATE: MATCH CALIBRATION MASTERS: found calibration masters for light frame: DSC_0026.NEF
15:31:07 - 2) CALIBRATE: frame: DSC_0026.NEF MasterDark :DSC_0050.NEF
15:31:07 -
15:31:07 - FRAME DETAILS UPDATER: starting...
15:31:07 - FRAME DETAILS UPDATER: no new frames to add
15:31:07 - FRAME DETAILS UPDATER: rebuilding all frame details...
15:31:07 - FRAME DETAILS UPDATER: checking if frames were identified earlier...
15:31:07 - FRAME DETAILS UPDATER: adding frame marks...
15:31:07 - FRAME DETAILS UPDATER: updated succesfully
15:31:07 - 2) CALIBRATE: trying to find calibration details of light frame: C:\Users\Aries Productions\Documents\AllTypesOfImages\astroShots\CocoonNebula Lights\Lights\DSC_0026.NEF
15:31:07 - 2) CALIBRATE: found calibration masters for light frame: C:\Users\Aries Productions\Documents\AllTypesOfImages\astroShots\CocoonNebula Lights\Lights\DSC_0026.NEF
15:31:07 - 2) CALIBRATE: Master Dark : C:\Users\Aries Productions\Documents\AllTypesOfImages\astroShots\Cocoon Darks\DSC_0050.NEF
15:31:07 -
15:31:07 - GENERAL IMAGE LOADER: loading frame C:\Users\Aries Productions\Documents\AllTypesOfImages\astroShots\CocoonNebula Lights\Lights\DSC_0026.NEF
15:31:08 - 2) CALIBRATE: Adaptive Data Pedestal: disabled
15:31:08 - GENERAL IMAGE LOADER: loading frame C:\Users\Aries Productions\Documents\AllTypesOfImages\astroShots\Cocoon Darks\DSC_0050.NEF
15:31:10 - GENERAL IMAGE LOADER: frame C:\Users\Aries Productions\Documents\AllTypesOfImages\astroShots\Cocoon Darks\DSC_0050.NEF was loaded successfully
15:31:10 - 2) CALIBRATE: analysing MasterDark...
15:31:10 - DATA ANALYSER TOOLS: re-instantiated multi-core analytical memory blocks, size 511 MBs
15:31:13 - 2) CALIBRATE: MasterDark analysed
15:31:15 - PREVIEW FILTER: filtering new frame:DSC_0026.NEF
15:31:15 - PREVIEW FILTER: creating clone of frame...
15:31:15 - GENERAL IMAGE LOADER: frame C:\Users\Aries Productions\Documents\AllTypesOfImages\astroShots\CocoonNebula Lights\Lights\DSC_0026.NEF was loaded successfully

Simply said, if you load either bias and/or flat darks, these will be applied to your flats as integrated masters and this is done completely automatically. The option to scale the MasterFlatDark is optional as well.

If you think that a video tutorial in this regard is required/warrented to take away some confusion, we can off course make one a.s.a.p. Perhaps you can send me a nice dataset so I can create one?

Kind regards,

Mabula

 

Main developer of Astro Pixel Processor and owner of Aries Productions


ReplyQuote
(@mabula-admin)
Quasar Admin
Joined: 2 years ago
Posts: 2081
February 5, 2019 15:57  
Posted by: Heno

Hi Mabula

I have read this thread a few times trying to get all into my head. I made myself a step by step procedure, but I would like to run it with you so see if I have got it all. For ASI 1600 MM-C it goes like this:

1. Take flat frames for each filter.
2. Take dark flats.(Match flats time, bias, offset, (and temperature ?)).
3. Take Bias frames, 0,1 sek (?), 400 of.

4. Generate Master flats using 1, 2 and 3, one for each filter.

5. Take lights
6. Take darks (Match lights time, bias, offset and temperature)
7. Create Master darks as required by 6.

8. Calibrate lights using 4, 5 and 7.

I'm not sure how much temperature matters for flats, dark flats and bias frames, but keeping it the same cannot hurt.

I read on Cloudy Nights that bias frame length should be 0,2 sec for ASI 1600. (Jon Rista). And then, if your flats and then your dark flats are of similar length you could use the dark flat as bias frames also. I may have misunderstood the last bit.

Comments to this please?

Helge

Edit: A bad piksel map could be useful, but should it be used with both flat/dark flat process and lights calibration?

Hi Helge @heno,

Yes, it's all good 😉

Temperature is not a big concern for the flats, flat darks and bias. Simply make sure you don't make them in completely different circumstances 😉

Keep in mind that the exposure time of bias frames is so short, that temperature normally does not enter the equation. Temperature has to do with the dark signal. So if you create darks of more than 1 second, temperature starts to play a rol due to build up of dark current/signal.

These cmos sensors like the asi1600 really make bad bias frames if you use the shortest exposure time possible, as you know, therefore, to get proper results you need to make bias frames with slightly longer exposure times (0,1 second-0,5 seconds) to get proper bias signal in your bias frames. If you would expose these longer than 1 second, dark current and temperature starts to influence things...

And then, if your flats and then your dark flats are of similar length you could use the dark flat as bias frames also. I may have misunderstood the last bit.

if you only use dark flats, then it would not matter how you load them into the application, if you load them as bias or flatdarks it will not make a difference. Both will be subtracted from the flats 😉

A bad piksel map could be useful, but should it be used with both flat/dark flat process and lights calibration?

There is no reason not to use a bad pixel map, whatever calibration workflow you use, a bad pixel map will never hurt your data. It will never inject noise 😉 I use it on both lights and flats all the time.

Let me know if you further questions 😉

Mabula

This post was modified 5 months ago by Mabula Haverkamp - Admin

Main developer of Astro Pixel Processor and owner of Aries Productions


ReplyQuote
 Heno
(@heno)
White Dwarf Customer
Joined: 2 years ago
Posts: 17
February 5, 2019 18:40  

Mabula

Thank you for confirming my understanding of the process. I will use my 0,5 sek dark flats also as bias frames, i.e. no need to create separate ones. I will need to cover my flats light panel to shoot flats in the same range, but that's no problem. 

Thanks again for your reply.

Now I will sit down and make a "process for dummies" (that's me 😀) sheet, so I don't forget it all once the season is over.


ReplyQuote
(@mariusz)
Molecular Cloud Customer
Joined: 8 months ago
Posts: 4
March 6, 2019 21:40  
Posted by: Mabula Haverkamp - Admin

These cmos sensors like the asi1600 really make bad bias frames if you use the shortest exposure time possible, as you know, therefore, to get proper results you need to make bias frames with slightly longer exposure times (0,1 second-0,5 seconds) to get proper bias signal in your bias frames. If you would expose these longer than 1 second, dark current and temperature starts to influence things...

Hi Mabula @mabula-admin

 

What do you think about bias frames from DSLRs CMOS sensor with time as short as it's possible, e.g. FujiFilm X-Trans with 1/4000s or Canon at 1/8000s? Or should I make Bias frames with longer exposure times, e.g. 0,1s?

 

Mariusz


ReplyQuote
(@vincent-mod)
Black Hole Moderator
Joined: 2 years ago
Posts: 383
March 8, 2019 01:36  

It depends on the camera, apparently the ASI mentioned above has an issue to produce decent bias signal. With DSLR's my experience has been fine setting that to the shortest possible setting. Normally that is also what you want as it's all about the read-out noise of the sensor. That should be a fixed, random signal just from reading it out and having longer exposures can introduce other noise which isn't part of the bias signal.


mariusz liked
ReplyQuote
(@mabula-admin)
Quasar Admin
Joined: 2 years ago
Posts: 2081
March 10, 2019 17:22  

Hi @mariusz and @vincent-mod,

From what I have seen from cmos sensors in general is that shooting bias frames with the shortest exposure time possible can give significant problems. I would advice not to use the shortest exposure time possible, but work on the safe side by using bias frames with exposure times between 1/100th - 1/10th of a second. That is still very short, so any dark signal will be very small. I think that is the safest and most robust way to shoot bias frames with a camera with a cmos sensor. So I would advice this for both astronomical cmos camera's as well as for DSLR cmos camera's 😉

To add to this, apparently, even some ccd sensors fail to produce a good bias signal with the shortest exposure time possible. I have seen very strange bias frames of both Atik and QHY CCD camera's with exposure times of 1/1000-1/4000th of a second... Calibration was greatly improved when bias frames were used of 1/10th of a second in those cases. I must add, that in these cases, always an old CCD camera was involved (age > 5 year), so I suspect strong degradation of the sensor and perhaps electronics surrounding the sensor to play a role here...

It's also very easy to test: simply shoot some frames with the shortest exposure time possible, then some frames with 1/100th - 1/10th of a second and shoot some darks of, let's say, 1 minute, so the dark current will a have significant contribution to the frame.

Then, study these frames with the same stretch parameters on a very strong stretch. If the shortest exposed frames look the same as the others, then, very likely, the frames can be used properly. If something is not right with the shortest exposed bias frames, you should see this with a strong stretch.

Cheers,

Mabula

Cheers,

Mabula

Main developer of Astro Pixel Processor and owner of Aries Productions


mariusz liked
ReplyQuote
(@stojang)
Hydrogen Atom Customer
Joined: 6 months ago
Posts: 1
May 4, 2019 17:30  

Save calibrate, register, normalise files or not ? For... OR  only time and space consuming ?


ReplyQuote
Page 4 / 4
Share: