Share:

APP 1.075 has been released !

Forum support from Vincent will be limited between now and  Oktober the 30th, 2019. Mabula will be available for support on the forum though.

2019 September: Astro Pixel Processor and iTelescope.net celebrate a new Partnership!

[Sticky] Data calibration principles/rules - must read !  

Page 4 / 4
  RSS

(@mabula-admin)
Quasar Admin
Joined: 3 years ago
Posts: 2148
February 5, 2019 15:39  
Posted by: Rowland F Archer Jr

Hi Mabula,

I think adding that to the FITS header would certainly not hurt - and might even save you some support questions 🙂 !

The process of calibrating with FlatDarks is different enough from the process documented all over the Internet that maybe a short video tutorial would be useful too, with so many people using the ASI1600 and similar CMOS cameras that benefit from the FlatDark / no bias processing.  Especially with image acquisition products like ACP, Starkeeper.it Voyager, and CCD Commander among others that change the exposure length on the fly to reach a desired ADU for flats.  

I think I'm finally getting it straight!

Thanks,

Rowland

 

Hi Rowland @rowland-f-archer-jr,

I will add the information to the masters and integrations 😉 a.s.a.p.

I have done a big rebuild of the calibration engine last year, to make this very easy, at least, I think... 

Technically speaking, calibration of flats with flatdarks is not different than calibration of lights with darks, right? The same rules apply.

APP will automatically calibrate your flats with bias and/or flat darks if these are provided, since without them, flat calibration can't work because the bias pedestal always needs to be subtracted to get proper flat-field calibration. APP makes sure that this happens. If bias or flatdarks aren't there, the user is ever warned, that he will probably face a problem. And then there is always the console, you can see exactly what is happening with your flats or lights in the console:

15:31:07 - IMAGE VIEWER: loading into Linear Calibrated Image Loader: C:\Users\Aries Productions\Documents\AllTypesOfImages\astroShots\CocoonNebula Lights\Lights\DSC_0026.NEF
15:31:07 - 2) CALIBRATE: MATCH CALIBRATION MASTERS: found calibration masters for light frame: DSC_0026.NEF
15:31:07 - 2) CALIBRATE: frame: DSC_0026.NEF MasterDark :DSC_0050.NEF
15:31:07 -
15:31:07 - FRAME DETAILS UPDATER: starting...
15:31:07 - FRAME DETAILS UPDATER: no new frames to add
15:31:07 - FRAME DETAILS UPDATER: rebuilding all frame details...
15:31:07 - FRAME DETAILS UPDATER: checking if frames were identified earlier...
15:31:07 - FRAME DETAILS UPDATER: adding frame marks...
15:31:07 - FRAME DETAILS UPDATER: updated succesfully
15:31:07 - 2) CALIBRATE: trying to find calibration details of light frame: C:\Users\Aries Productions\Documents\AllTypesOfImages\astroShots\CocoonNebula Lights\Lights\DSC_0026.NEF
15:31:07 - 2) CALIBRATE: found calibration masters for light frame: C:\Users\Aries Productions\Documents\AllTypesOfImages\astroShots\CocoonNebula Lights\Lights\DSC_0026.NEF
15:31:07 - 2) CALIBRATE: Master Dark : C:\Users\Aries Productions\Documents\AllTypesOfImages\astroShots\Cocoon Darks\DSC_0050.NEF
15:31:07 -
15:31:07 - GENERAL IMAGE LOADER: loading frame C:\Users\Aries Productions\Documents\AllTypesOfImages\astroShots\CocoonNebula Lights\Lights\DSC_0026.NEF
15:31:08 - 2) CALIBRATE: Adaptive Data Pedestal: disabled
15:31:08 - GENERAL IMAGE LOADER: loading frame C:\Users\Aries Productions\Documents\AllTypesOfImages\astroShots\Cocoon Darks\DSC_0050.NEF
15:31:10 - GENERAL IMAGE LOADER: frame C:\Users\Aries Productions\Documents\AllTypesOfImages\astroShots\Cocoon Darks\DSC_0050.NEF was loaded successfully
15:31:10 - 2) CALIBRATE: analysing MasterDark...
15:31:10 - DATA ANALYSER TOOLS: re-instantiated multi-core analytical memory blocks, size 511 MBs
15:31:13 - 2) CALIBRATE: MasterDark analysed
15:31:15 - PREVIEW FILTER: filtering new frame:DSC_0026.NEF
15:31:15 - PREVIEW FILTER: creating clone of frame...
15:31:15 - GENERAL IMAGE LOADER: frame C:\Users\Aries Productions\Documents\AllTypesOfImages\astroShots\CocoonNebula Lights\Lights\DSC_0026.NEF was loaded successfully

Simply said, if you load either bias and/or flat darks, these will be applied to your flats as integrated masters and this is done completely automatically. The option to scale the MasterFlatDark is optional as well.

If you think that a video tutorial in this regard is required/warrented to take away some confusion, we can off course make one a.s.a.p. Perhaps you can send me a nice dataset so I can create one?

Kind regards,

Mabula

 

Main developer of Astro Pixel Processor and owner of Aries Productions


ReplyQuote
(@mabula-admin)
Quasar Admin
Joined: 3 years ago
Posts: 2148
February 5, 2019 15:57  
Posted by: Heno

Hi Mabula

I have read this thread a few times trying to get all into my head. I made myself a step by step procedure, but I would like to run it with you so see if I have got it all. For ASI 1600 MM-C it goes like this:

1. Take flat frames for each filter.
2. Take dark flats.(Match flats time, bias, offset, (and temperature ?)).
3. Take Bias frames, 0,1 sek (?), 400 of.

4. Generate Master flats using 1, 2 and 3, one for each filter.

5. Take lights
6. Take darks (Match lights time, bias, offset and temperature)
7. Create Master darks as required by 6.

8. Calibrate lights using 4, 5 and 7.

I'm not sure how much temperature matters for flats, dark flats and bias frames, but keeping it the same cannot hurt.

I read on Cloudy Nights that bias frame length should be 0,2 sec for ASI 1600. (Jon Rista). And then, if your flats and then your dark flats are of similar length you could use the dark flat as bias frames also. I may have misunderstood the last bit.

Comments to this please?

Helge

Edit: A bad piksel map could be useful, but should it be used with both flat/dark flat process and lights calibration?

Hi Helge @heno,

Yes, it's all good 😉

Temperature is not a big concern for the flats, flat darks and bias. Simply make sure you don't make them in completely different circumstances 😉

Keep in mind that the exposure time of bias frames is so short, that temperature normally does not enter the equation. Temperature has to do with the dark signal. So if you create darks of more than 1 second, temperature starts to play a rol due to build up of dark current/signal.

These cmos sensors like the asi1600 really make bad bias frames if you use the shortest exposure time possible, as you know, therefore, to get proper results you need to make bias frames with slightly longer exposure times (0,1 second-0,5 seconds) to get proper bias signal in your bias frames. If you would expose these longer than 1 second, dark current and temperature starts to influence things...

And then, if your flats and then your dark flats are of similar length you could use the dark flat as bias frames also. I may have misunderstood the last bit.

if you only use dark flats, then it would not matter how you load them into the application, if you load them as bias or flatdarks it will not make a difference. Both will be subtracted from the flats 😉

A bad piksel map could be useful, but should it be used with both flat/dark flat process and lights calibration?

There is no reason not to use a bad pixel map, whatever calibration workflow you use, a bad pixel map will never hurt your data. It will never inject noise 😉 I use it on both lights and flats all the time.

Let me know if you further questions 😉

Mabula

This post was modified 9 months ago by Mabula Haverkamp - Admin

Main developer of Astro Pixel Processor and owner of Aries Productions


ReplyQuote
 Heno
(@heno)
White Dwarf Customer
Joined: 2 years ago
Posts: 22
February 5, 2019 18:40  

Mabula

Thank you for confirming my understanding of the process. I will use my 0,5 sek dark flats also as bias frames, i.e. no need to create separate ones. I will need to cover my flats light panel to shoot flats in the same range, but that's no problem. 

Thanks again for your reply.

Now I will sit down and make a "process for dummies" (that's me 😀) sheet, so I don't forget it all once the season is over.


ReplyQuote
(@mariusz)
Brown Dwarf Customer
Joined: 12 months ago
Posts: 6
March 6, 2019 21:40  
Posted by: Mabula Haverkamp - Admin

These cmos sensors like the asi1600 really make bad bias frames if you use the shortest exposure time possible, as you know, therefore, to get proper results you need to make bias frames with slightly longer exposure times (0,1 second-0,5 seconds) to get proper bias signal in your bias frames. If you would expose these longer than 1 second, dark current and temperature starts to influence things...

Hi Mabula @mabula-admin

 

What do you think about bias frames from DSLRs CMOS sensor with time as short as it's possible, e.g. FujiFilm X-Trans with 1/4000s or Canon at 1/8000s? Or should I make Bias frames with longer exposure times, e.g. 0,1s?

 

Mariusz


ReplyQuote
(@vincent-mod)
Galaxy Moderator
Joined: 2 years ago
Posts: 871
March 8, 2019 01:36  

It depends on the camera, apparently the ASI mentioned above has an issue to produce decent bias signal. With DSLR's my experience has been fine setting that to the shortest possible setting. Normally that is also what you want as it's all about the read-out noise of the sensor. That should be a fixed, random signal just from reading it out and having longer exposures can introduce other noise which isn't part of the bias signal.


mariusz liked
ReplyQuote
(@mabula-admin)
Quasar Admin
Joined: 3 years ago
Posts: 2148
March 10, 2019 17:22  

Hi @mariusz and @vincent-mod,

From what I have seen from cmos sensors in general is that shooting bias frames with the shortest exposure time possible can give significant problems. I would advice not to use the shortest exposure time possible, but work on the safe side by using bias frames with exposure times between 1/100th - 1/10th of a second. That is still very short, so any dark signal will be very small. I think that is the safest and most robust way to shoot bias frames with a camera with a cmos sensor. So I would advice this for both astronomical cmos camera's as well as for DSLR cmos camera's 😉

To add to this, apparently, even some ccd sensors fail to produce a good bias signal with the shortest exposure time possible. I have seen very strange bias frames of both Atik and QHY CCD camera's with exposure times of 1/1000-1/4000th of a second... Calibration was greatly improved when bias frames were used of 1/10th of a second in those cases. I must add, that in these cases, always an old CCD camera was involved (age > 5 year), so I suspect strong degradation of the sensor and perhaps electronics surrounding the sensor to play a role here...

It's also very easy to test: simply shoot some frames with the shortest exposure time possible, then some frames with 1/100th - 1/10th of a second and shoot some darks of, let's say, 1 minute, so the dark current will a have significant contribution to the frame.

Then, study these frames with the same stretch parameters on a very strong stretch. If the shortest exposed frames look the same as the others, then, very likely, the frames can be used properly. If something is not right with the shortest exposed bias frames, you should see this with a strong stretch.

Cheers,

Mabula

Cheers,

Mabula

Main developer of Astro Pixel Processor and owner of Aries Productions


mariusz liked
ReplyQuote
(@stojang)
Hydrogen Atom Customer
Joined: 10 months ago
Posts: 1
May 4, 2019 17:30  

Save calibrate, register, normalise files or not ? For... OR  only time and space consuming ?


ReplyQuote
(@kingjamez)
White Dwarf Customer
Joined: 2 years ago
Posts: 17
September 17, 2019 20:39  

For cooled CMOS cameras, is there any reason to use a bias frame at all?

If we assume that one has a dark library of  (fully) matched darks, and that flat darks are taken to match flats, does a master bias make any difference at all?

-Jim


ReplyQuote
(@vincent-mod)
Galaxy Moderator
Joined: 2 years ago
Posts: 871
September 18, 2019 20:58  

It's best to simply try it out, you can check the noise levels afterwards. I think it's still useful as a bias signal will always be there, but how much it contributes depends on the sensor. Some sensors do seem to like bias frames which aren't taken at the fastest speed.


ReplyQuote
(@kingjamez)
White Dwarf Customer
Joined: 2 years ago
Posts: 17
September 18, 2019 21:04  

I guess my question is more of a theoretical one. Is there any advantage in theory to separately calibrating using a bias and a bias-subtracted master dark, over simply calibrating with a  masterdark that has not had it's bias removed?

Trying it shows no difference in noise level, so I'm left wondering why the bias frame is a thing, and specifically why it's called out above as being important. 

 


ReplyQuote
(@vincent-mod)
Galaxy Moderator
Joined: 2 years ago
Posts: 871
September 18, 2019 21:08  

Ah, like that. No there is no difference if you have the darks with bias added. I do have to note here that I never tried this approach in APP.


kingjamez liked
ReplyQuote
 Ed
(@lewis)
Hydrogen Atom Customer
Joined: 2 years ago
Posts: 2
September 27, 2019 14:58  

Hi

I am new in processing by APP. I have read this thread and have done an abstract for me.

Can somebody check my layout in the attachment and shows me possible errors?

Thanks a lot

Ed


ReplyQuote
(@wvreeven)
Red Giant Customer
Joined: 1 year ago
Posts: 72
September 27, 2019 16:07  

Hi Ed,

Welcome to the forum. In principles your layout looks ok but it is a bit hard to comment on it without info about the telescope you use and the sky conditions of your imaging site. This is especially true for the number of Light images and their exposure times. Perhaps the tips that I wrote down in this thread may be of help to you:

https://www.astropixelprocessor.com/community/main-forum/newbie-lrgb-processing/

To answer the two questions in your layout:

  1. FlatDarks can be shot inside since they do not require any light coming through the telescope, just as with Darks.
  2. Bias images shot at the same temperature as Lights, Darks and FlatDarks cannot be wrong

 

Wouter


ReplyQuote
 Ed
(@lewis)
Hydrogen Atom Customer
Joined: 2 years ago
Posts: 2
September 27, 2019 17:11  

Hi Wouter

Thanks for the answer. I know that the exposure time and the sky conditions play an important role.

My Scope is a Pentax 75SDHF (FL 500mm) and the sky condition are Bortle 4 and 20.6 mag/arsec2.

The number of the images with 15 is a minimum and can or should be greater depending of the sky pollution.

Thanks for the newest thread about lrgb processing, i will read it.

Ed


ReplyQuote
(@kees_scherer)
Main Sequence Star Customer
Joined: 2 years ago
Posts: 37
October 22, 2019 18:26  

16 bit masters or 32 bit masters? I could not find the reason for having the 32bit master option so i did a test. I made a set of masters (based on 300 bias, 250 darks and 25 flats) with 16 bit and a set with 32 bit and made 2 integrations with 75 luminance subs, 1 with 16bit and 1 with 32 bit masters. The luminance lights used are 16 bit fit files. The result is that there is no difference in the end result, visually or in noise level / SNR. So i still do not know why there is the 32 bit master option...........

Stack made with 16 and 32bit masters


ReplyQuote
(@vincent-mod)
Galaxy Moderator
Joined: 2 years ago
Posts: 871
October 22, 2019 19:33  

So I asked Mabula about it and indeed, it doesn't change anything in the calibration phase. It seems more of a compatibility option as other software packages have 32-bit options as well.. even though it shouldn't matter there either.


ReplyQuote
Page 4 / 4
Share: