Share:

APP 1.075 has been released !

2019 November: Complete LRGB Tutorial of NGC292, The Small Magellanic Cloud by Christian Sasse, Astronomer in Charge of iTelescope.net

2019 September: Astro Pixel Processor and iTelescope.net celebrate a new Partnership!

[Sticky] Data calibration principles/rules - must read !  

Page 2 / 4
  RSS

 Tim
(@tim)
Main Sequence Star Customer
Joined: 2 years ago
Posts: 36
March 15, 2018 21:37  

I have a fairly bright LED that I use for flats. I just add sheets of white paper between the screen and scope until I get it dim enough to take longer flats. Its just a matter of how many sheets of paper.

I don't use bias frames. I use dark flats which are slightly more accurate than bias frames - especially for longer flats. Here is an easy way to look at it.

Take lights and darks at the same iso, exposure and temperature.

Take your flats somewhere between say .5 and 5 seconds. This is just an approximation. Then, put the cap on your scope and take the same number of images at the same iso and time that you just used for your flats. These are called flat darks or dark flats. So flat darks are to calibrate flats like darks calibrate lights.

So what APP does is create a master dark flat to calibrate and create a master flat. This master flat is used with the master dark  to calibrate the lights.

There are many references on the net saying the iso of the lights and darks should match the iso of the flats and flat darks. There are also different references saying they don't have to match. Its easy to match them so I do it. I hope this makes sense -if not let me know.


ReplyQuote
(@1cm69)
Neutron Star Customer
Joined: 2 years ago
Posts: 132
March 15, 2018 22:53  

My Flats panel has 3 light levels, I use the lowest with a diffuser sheet & 2 t-shirts & at ISO100 exposure length is 0.6sec for a mid histogram. At ISO800 I just get a blown overexposed white screen with histogram at the far right. 


ReplyQuote
 Tim
(@tim)
Main Sequence Star Customer
Joined: 2 years ago
Posts: 36
March 15, 2018 23:57  

As I mentioned I've read some articles saying the the iso from the flats should be the same as the darks and lights and I've also read that it didn't matter. I've never tested using different iso's so I can't comment from personal experience. Nothing is wrong with trying to see if it works.

It sounds like I have a similar panel as yours. I used to use the t-shirt method but when I needed to reduce the glare even more, it was suggested to me to try adding individual sheets of white paper (computer paper?) so I would have maximum control over the brightness. This worked for me.


1CM69 liked
ReplyQuote
(@gregwrca)
Neutron Star Customer
Joined: 3 years ago
Posts: 227
March 16, 2018 04:50  
" nothing is wrong with trying to reinvent the wheel". 
Why would you waste your time even trying? Are you going out
of your way to avoid matching ISO?

GW


ReplyQuote
(@1cm69)
Neutron Star Customer
Joined: 2 years ago
Posts: 132
March 16, 2018 07:52  
Posted by: gdwats@comcast.net
" nothing is wrong with trying to reinvent the wheel". 
Why would you waste your time even trying? Are you going out
of your way to avoid matching ISO?

Look mate, I assume you’re the type that skim reads & then jumps to conclusions and then posts a useless comment. 

Reread the whole thread & absorb the information, then if you have worthwhile input to share, go ahead and let us all know. 

Cheerio

Can I use a different ISO or gain+offset than my Light frames, for my flat frames?

Yes, that is no problem and actually preferred in most cases. Make your flats with a low ISO or gain+offset to be able to increase the exposure length of the flats. They will be of better quality then. Be aware of the rules you need to follow for flat & light frames. Both need to have the bias pedestal subtracted only once using any of the available options. So if you only use bias frames and no darks. You need 2 master bias frames of the 2 different ISO or gain+offset value to make sure that the bias pedestal is removed correctly (only once) from both lights and flats.

above excerpt taken directly from the first post in this thread.


ReplyQuote
(@mabula-admin)
Quasar Admin
Joined: 3 years ago
Posts: 2168
March 16, 2018 17:59  
Posted by: 1CM69

Hi Mabula - @mabula-haverkamp-administrator

I have been looking more into my Flat collection procedure & have read vast amounts of differing accounts of the best way to capture Flats.

So, as you pointed out my current Flats taken at 1/2000th sec are possibly too short but this was because I was attempting to take them at the same ISO as my Lights, Darks etc... which is generally ISO800. Because this ISO is fairly high & sensitive the AV setting in my DSLR is setting the EXP to this very quick shutter speed.

Now having re-read your rules for APP & understanding it all a little more, I can see a way around this & this is to reduce the ISO of my Flats to achieve a longer EXP time. Yes this would mean taking another set of Bias frames also to match these Flat frames, I know this.

So I set upon experimenting a little with various different configurations of my light panel & DSLR ISO settings.

I have found through a process of elimination that an ISO setting of 100 with my light panel on it's lowest setting & using 2 t-shirts between my scope and the light panel, gives me the best looking histogram.

This however is where I am now stuck.

I have read many paces not to trust the histogram in BYEOS as it is not the true RAW histogram but more a JPEG one. I have tried using various pieces of software such as RAWdigger, IRIS etc... in a failed attempt to find the so called MEDIAN.

Everywhere seems to rave about Pixinsight but I am not willing to pay the high price for the software, I have chosen APP for my processing choice.

I have uploaded a selection of my RAW .cr2 files to DropBox, here:  https://www.dropbox.com/sh/4vryrk1m1vq3q27/AAC9x5w912vMqtZKFFquZW_Qa?dl=0

Please could you possibly look at these & let me know which you consider to be the closest to the MEDIAN or could you point me in the direction of any software other than Pixinsight that I can use to find the info myself.

Kind regards..,

Kirk

@1cm69 , Will check them now and I will show you how you can verify yourself in APP 😉

Hang on...

Mabula

Main developer of Astro Pixel Processor and owner of Aries Productions


ReplyQuote
(@mabula-admin)
Quasar Admin
Joined: 3 years ago
Posts: 2168
March 16, 2018 18:11  
Posted by: Tim

Leave the iso for your flats the same as your lights and darks. This is a MUST. Just increase the time for your flats without changing the iso. I use around  1 second but it can vary a lot. If necessary use a few sheets of white paper or white T-shirts to reduce the glare and increase the time for your flats. Using the histogram in BYEOS is fine for this purpose. Keep it around the center. 

@tim, no it actually isn't a must I believe ;-). I never do it on both an Asi1600mm-c and Nikon D5100 and D610 DLSRs.

What would be the explanation that you can't do this? If you have a topic/website/source that says otherwise, do share it 😉

(I always create flats with ISO 100 on a DLSR and gain 0 with the asi1600mm-c and never had any calibration problem with the flats...)

Flats have the purpose of catching the vignetting profile and dust bunnies of your optical setup. To get the vignetting profile, the flats need to be corrected with either a masterbias or masterdarkflat of the same iso as the flats if you want to do it properly.

I have used Backyard Nikon myself, but that also doesn't give a linear histogram, or does it ? You will be amazed at how much longer you can expose your flats 😉 !

Mabula

This post was modified 1 year ago by Mabula Haverkamp - Admin

Main developer of Astro Pixel Processor and owner of Aries Productions


ReplyQuote
(@1cm69)
Neutron Star Customer
Joined: 2 years ago
Posts: 132
March 16, 2018 18:13  

OK, thanks


ReplyQuote
(@mabula-admin)
Quasar Admin
Joined: 3 years ago
Posts: 2168
March 16, 2018 18:33  
Posted by: Tim

I have a fairly bright LED that I use for flats. I just add sheets of white paper between the screen and scope until I get it dim enough to take longer flats. Its just a matter of how many sheets of paper.

I don't use bias frames. I use dark flats which are slightly more accurate than bias frames - especially for longer flats. Here is an easy way to look at it.

Take lights and darks at the same iso, exposure and temperature.

Take your flats somewhere between say .5 and 5 seconds. This is just an approximation. Then, put the cap on your scope and take the same number of images at the same iso and time that you just used for your flats. These are called flat darks or dark flats. So flat darks are to calibrate flats like darks calibrate lights.

So what APP does is create a master dark flat to calibrate and create a master flat. This master flat is used with the master dark  to calibrate the lights.

There are many references on the net saying the iso of the lights and darks should match the iso of the flats and flat darks. There are also different references saying they don't have to match. Its easy to match them so I do it. I hope this makes sense -if not let me know.

Hi @tim,

Okay, since I never had a problem with using flats of other ISO/gain values as my lights, I wouldn't have a clue about the technical explanation why it wouldn't be okay. But I agree, it's easy to match the light frames and in case of any doubt in this hobby, it's better to take the safe route 😉

I know of some of the Beta testers that use APP (some are very experienced and APOD published) that most use different ISO/gain values for their flats for the mentioned reason. You can increase exposure time of your flats more easily and that will simpy give better quality flats, most problems in flat calibration occur due to too short exposures.

Other ISO/gain for flats works equally well with slow and very fast focal ratios as well. If you are using a fast telescope like a Takashashi Epsilon F/2.8 then you will be happy this simply works. For fast focal ratios things need to be very precise. For instance, this 2 panel mosaic is made with an Takahashi Epsilon at F/2.8 using flats that weren't equal to the light frames by Maurice Toet:

http://www.mauricetoet.nl/Deep-sky/i-qFS6F3J/A

worked perfectly 😉

The problem in this case is, how to make proper flats with an Epsilon (or other fast optics), that is not so easy though.

Mabula

Main developer of Astro Pixel Processor and owner of Aries Productions


ReplyQuote
(@mabula-admin)
Quasar Admin
Joined: 3 years ago
Posts: 2168
March 16, 2018 18:42  
Posted by: 1CM69
Posted by: gdwats@comcast.net
" nothing is wrong with trying to reinvent the wheel". 
Why would you waste your time even trying? Are you going out
of your way to avoid matching ISO?

Look mate, I assume you’re the type that skim reads & then jumps to conclusions and then posts a useless comment. 

Reread the whole thread & absorb the information, then if you have worthwhile input to share, go ahead and let us all know. 

Cheerio

Can I use a different ISO or gain+offset than my Light frames, for my flat frames?

Yes, that is no problem and actually preferred in most cases. Make your flats with a low ISO or gain+offset to be able to increase the exposure length of the flats. They will be of better quality then. Be aware of the rules you need to follow for flat & light frames. Both need to have the bias pedestal subtracted only once using any of the available options. So if you only use bias frames and no darks. You need 2 master bias frames of the 2 different ISO or gain+offset value to make sure that the bias pedestal is removed correctly (only once) from both lights and flats.

above excerpt taken directly from the first post in this thread.

Hi @gregwrca & @1cm69, yes, do read the whole topic, to get the right context, before posting please;-) and stay nice and respectfull to each other please.

Astrophotography is a difficult and demanding passion 😉 and going too fast usually is not helpfull.

Mabula

Main developer of Astro Pixel Processor and owner of Aries Productions


ReplyQuote
(@mabula-admin)
Quasar Admin
Joined: 3 years ago
Posts: 2168
March 16, 2018 18:47  

@1cm69, I'll make a FAQ sticky for your flat exposure question, since others might have the same question. Hang on..

Main developer of Astro Pixel Processor and owner of Aries Productions


ReplyQuote
(@1cm69)
Neutron Star Customer
Joined: 2 years ago
Posts: 132
March 16, 2018 18:49  

Brilliant, I appreciate all your help Mabula 


ReplyQuote
 Tim
(@tim)
Main Sequence Star Customer
Joined: 2 years ago
Posts: 36
March 16, 2018 19:04  

Mabula, I have no personal experience to show that the flats iso has to be the same as the lights iso. However, when I started in this hobby I used DSS and the DSS FAQ says  "The flat frames should be created with the ISO speed of the light frames."    http://deepskystacker.free.fr/english/index.html

I've also read this from other people I consider good imagers. I've also read from good imagers that they don't have to match. Based on what DSS says and the difference of opinions among some imagers, I thought it safe to always make them match. 

Since this is easy for me to do, I was trying to show the OP an easy way to control his LED output to accomplish the same thing.


ReplyQuote
(@mabula-admin)
Quasar Admin
Joined: 3 years ago
Posts: 2168
March 16, 2018 21:12  

Kirk @1cm69 in response to your question on how to know if a flat is well illuminated, or how to see the unstretched linear data in your CR2 frames

I have uploaded the CR2 file of ISO100 with 2,5s exposure time. It's very well illuminated linearly ;-).

Everyone that reads the FAQ can download the CR2 and follow the steps and learn how this works in APP.

I think it's of great use and value that you can see for yourself what the differences are between linear data and the non-linear sRGB data with camera white balance that your camera's display shows 😉

https://www.astropixelprocessor.com/community/faq/dslr-how-to-check-the-linear-histogram-of-your-data-or-how-long-can-i-expose-my-flats/

Let me know, if my explanation is clear enough 😉

Mabula

Main developer of Astro Pixel Processor and owner of Aries Productions


ReplyQuote
(@1cm69)
Neutron Star Customer
Joined: 2 years ago
Posts: 132
March 16, 2018 21:17  

OK, thank you Mabula  🙂  


ReplyQuote
(@mabula-admin)
Quasar Admin
Joined: 3 years ago
Posts: 2168
March 16, 2018 21:24  
Posted by: Tim

Mabula, I have no personal experience to show that the flats iso has to be the same as the lights iso. However, when I started in this hobby I used DSS and the DSS FAQ says  "The flat frames should be created with the ISO speed of the light frames."    http://deepskystacker.free.fr/english/index.html

I've also read this from other people I consider good imagers. I've also read from good imagers that they don't have to match. Based on what DSS says and the difference of opinions among some imagers, I thought it safe to always make them match. 

Since this is easy for me to do, I was trying to show the OP an easy way to control his LED output to accomplish the same thing.

Hi @Tim,

Yes, I fully understand and I do recall that I have seen it on the DSS site and other places. I think it's very wise to follow the safe road in case of doubt, especially in astrophotography 😉

Perfect and thank you for your assistance in this thread 😉 it's very well appreciated.

You can quote me 😉 on saying: the ISO or gain of the flats does not have to be the same as the iso or gain of the lights. 

With a lot of things, technical arguments aside, if it works, it works ! But honestly, I am not aware of any sound technical argument why it shouldn't be like this 😉

( If there actually is one, I need to know so please share ! )

Mabula

Main developer of Astro Pixel Processor and owner of Aries Productions


kingjamez liked
ReplyQuote
(@mestutters)
Red Giant Customer
Joined: 2 years ago
Posts: 65
August 29, 2018 12:16  

Hi Mabula,

I am in the midst of redoing my Master Calibration frames ready for the up-coming winter months and have a question regarding the preparation of Master Darks.

I have captured a number (minimum of 35) dark frames for each of the exposure times that I normally use when capturing Lights, e.g  60s, 120s, 300s and have loaded these into APP along with Bias and Flat Frames.

I was expecting APP to provide me with a Master Dark for each of these different exposure times as this was what I have used in the past.  However APP v1.062  is loading all the available Darks and producing only a single consolidated Master Dark with an exposure time shown as equal to the longest of the Darks that I loaded into the run.

I thought this result might have depended on whether the 'scale MasterDark' option was checked but I've tried both possibilities and still get the same result.

Having observed this behaviour, my questions are:

a) Do you still recommend (where time permits) the preparation of separate MasterDarks for each different exposure time used when capturing Lights (OR does MasterDark scaling take care of this)?

b) If use of separate MasterDark per exposure time is still recommended, how do I do this? Have I missed a setting when initiating my Master Calibration frames preparation run, or do I need to load Darks for only a single exposure interval at a time.

c)  Lastly, while experimenting,  I've noticed that I get a somewhat different Bad Pixel Map result depending on the duration of the Dark Frames I use during preparation - I say this after looking at only a top corner of two BPMs and seeing different hot pixels, so not an in-depth analysis.  But this difference being so,  it occurs to me to ask what would be the best darks exposure time to use in the preparation of my Bad Pixel Map,  e.g one equal to the longest or shortest exposure times I normally used when capturing Lights? Or perhaps a MasterDark that is produced by amalgamating Darks with several different exposure times, which is the Master Dark result I seem to be getting at the moment?

Regards

Mike

 


ReplyQuote
(@cheetah)
Main Sequence Star Customer
Joined: 2 years ago
Posts: 34
September 22, 2018 18:17  

Ok... so if I follow 'the rules' for creating the calibration files, let's assume now have:  Lights, Darks, Flats, Dark Flats, Bias & BPM

Are ALL of these needed, for the 'best' result?

Assuming I do need them all:  Do I just LOAD each type and then (for simplicity's sake skip over all the options)  go straight to INTEGRATE?   Will APP will create the all masters and properly apply them?

I'm hoping the answer is YES.  This is what I've been doing for months now...  

 


ReplyQuote
(@mabula-admin)
Quasar Admin
Joined: 3 years ago
Posts: 2168
October 2, 2018 18:44  
Posted by: mestutters

Hi Mabula,

I am in the midst of redoing my Master Calibration frames ready for the up-coming winter months and have a question regarding the preparation of Master Darks.

I have captured a number (minimum of 35) dark frames for each of the exposure times that I normally use when capturing Lights, e.g  60s, 120s, 300s and have loaded these into APP along with Bias and Flat Frames.

I was expecting APP to provide me with a Master Dark for each of these different exposure times as this was what I have used in the past.  However APP v1.062  is loading all the available Darks and producing only a single consolidated Master Dark with an exposure time shown as equal to the longest of the Darks that I loaded into the run.

I thought this result might have depended on whether the 'scale MasterDark' option was checked but I've tried both possibilities and still get the same result.

Having observed this behaviour, my questions are:

a) Do you still recommend (where time permits) the preparation of separate MasterDarks for each different exposure time used when capturing Lights (OR does MasterDark scaling take care of this)?

b) If use of separate MasterDark per exposure time is still recommended, how do I do this? Have I missed a setting when initiating my Master Calibration frames preparation run, or do I need to load Darks for only a single exposure interval at a time.

c)  Lastly, while experimenting,  I've noticed that I get a somewhat different Bad Pixel Map result depending on the duration of the Dark Frames I use during preparation - I say this after looking at only a top corner of two BPMs and seeing different hot pixels, so not an in-depth analysis.  But this difference being so,  it occurs to me to ask what would be the best darks exposure time to use in the preparation of my Bad Pixel Map,  e.g one equal to the longest or shortest exposure times I normally used when capturing Lights? Or perhaps a MasterDark that is produced by amalgamating Darks with several different exposure times, which is the Master Dark result I seem to be getting at the moment?

Regards

Mike

 

Hi Mike @mestutters,

First of all, please accept my apologies for my delayed response ;-(

Do you still recommend (where time permits) the preparation of separate MasterDarks for each different exposure time used when capturing Lights (OR does MasterDark scaling take care of this)?

If you are able practically due to time constraints and temperature matching to prepare good masters of a fixed exposure duration, by all means do that 😉

On the other hand, dark-frame scaling is rather robust now in APP and I have used it myself on lots of data now and it works very well. With Dark frame scaling, you can simply create 1 very good masterdark of let's say 200 darks with a long exposure time and use that always besides a good Masterbias. A masterbias is required for dark frame scaling to work.

It also depends on the sensor if you can use dark frame scaling to good effect. For instance, the zwo asi 183 cmos camera has severe amp-glow, in that case, dark frame scaling will lead to worse results since the amp-glow can't be properly scaled. This is due to it's non-linear behaviour with respect to exposure time and temperature. So with such a camera, you will need to create proper darks of fixed temperature and exposure time.

"I was expecting APP to provide me with a Master Dark for each of these different exposure times as this was what I have used in the past. However APP v1.062 is loading all the available Darks and producing only a single consolidated Master Dark with an exposure time shown as equal to the longest of the Darks that I loaded into the run."

Yes, this used to be the case, it has changed since the introduction of dark-frame scaling. I (me, myself), for instance, load 100s of darks of different exposures and temperatures for my dslr data and use that combined Masterdark with dark frame scaling to very good effect.

If use of separate MasterDark per exposure time is still recommended, how do I do this? Have I missed a setting when initiating my Master Calibration frames preparation run, or do I need to load Darks for only a single exposure interval at a time.

If you still want to create masters that are separated in exposure time, load them as different sessions in the multi-session mode and do the same for the lights ;-). Or simply make the masters one at at time.When loading the lights, APP will match the right Masterdark to the correct light frames based on both ISO/gain and exposure time.

By the way, Darks will still be discriminated if they have a different ISO or gain value.

Lastly, while experimenting, I've noticed that I get a somewhat different Bad Pixel Map result depending on the duration of the Dark Frames I use during preparation - I say this after looking at only a top corner of two BPMs and seeing different hot pixels, so not an in-depth analysis. But this difference being so, it occurs to me to ask what would be the best darks exposure time to use in the preparation of my Bad Pixel Map, e.g one equal to the longest or shortest exposure times I normally used when capturing Lights? Or perhaps a MasterDark that is produced by amalgamating Darks with several different exposure times, which is the Master Dark result I seem to be getting at the moment?

For the creation of a good Bad Pixel Map, I would recommend to use very long dark frames. Mixing darks of different exposure times should not downgrade the result. Try to have a lot of darks with good exposure times, at least a couple of minutes 😉

The purpose here is to have a good measurement of all the non-linear artefacts on the sensor, like the hot pixels. By using more darks that have longer exposure times, the bad pixels will be more apparent. You can even create the darks at much higher temperatures for this purpose, since higher temperatures will show non-linear beahviour of pixels faster...

Kind regards,

Mabula

 

Main developer of Astro Pixel Processor and owner of Aries Productions


ReplyQuote
(@mabula-admin)
Quasar Admin
Joined: 3 years ago
Posts: 2168
October 2, 2018 18:50  
Posted by: Cheetah

Ok... so if I follow 'the rules' for creating the calibration files, let's assume now have:  Lights, Darks, Flats, Dark Flats, Bias & BPM

Are ALL of these needed, for the 'best' result?

Assuming I do need them all:  Do I just LOAD each type and then (for simplicity's sake skip over all the options)  go straight to INTEGRATE?   Will APP will create the all masters and properly apply them?

I'm hoping the answer is YES.  This is what I've been doing for months now...  

 

Hi @cheetah, this is actually a non-trivial question. And very hard 😉 to give a correct answer to, because it really depends on the sensor characteristics in your camera.

For some camera's it will be optimal, for others it won't. What kind of camera or camera's are you using?

Assuming I do need them all:  Do I just LOAD each type and then (for simplicity's sake skip over all the options)  go straight to INTEGRATE?   Will APP will create the all masters and properly apply them?

I'm hoping the answer is YES.  This is what I've been doing for months now...  

Yes, with APP's new calibration engine, since APP 1.062, APP will do everything on auto-pilot and correctly. All masters will be used and properly applied 😉

Kind regards,

Mabula

Main developer of Astro Pixel Processor and owner of Aries Productions


ReplyQuote
Page 2 / 4
Share: