[Solved] Calibration not working ........ - it did work after some guidance !

Hi Sara,

Most likely, the calibration workflow in APP wasn't done correctly. I'll be most happy to help you out with this. (I am currently writing the calibration documentation).

What did you use for the flat calibration, only bias, or darks as well?

And did you use both bias and darks for the light frame calibration?

(Most errors are due to failing to subtract the bias pedestal or subtracting it twice, from both the light and flat frames)

You could send me a small subset of lights, bias, darks, flats and I'll show how it's done correctly.

You already have created a BPM?

Cheers,

Mabula

I used a full set of flats, bias and darks and I did create a BPM as well. I left all calibration settings on default as well. 

Perhaps it would be easier to TV my PC instead of me trying to send a load of files across to you?

How would you propose to TV ?

Only 1 light frame, 5 bias, 5 flats, 5 darks would suffice to illustrate the needed workflow, (it would help me understand why initially the calibration goes wrong in your case)

(I am off walking the dog, but I'll be back 😉 ) I could setup a shared dropbox folder if you want?

Mabula

Hi Sara,

Explanation about the calibration rules is here:

data-calibration-priciples-must-read

I have created a shared dropbox folder 😉

Mabula

Thanks Mabula - I had to go out suddenly sorry! I'm just uploading 5x bias, darks, flats and 1x light - It will take about 3 hours.

Thanks Sara, in the mean time I have added more information about calibration here:

https://www.astropixelprocessor.com/community/tutorials-workflows/

I'll have a look once the files are received 😉

Thanks! I've had a read of all the info and I'm not clear why any of it would prevent my flats from working at all - But don't try to explain it to me as I'm not technologically aware and big words and science goes over my head. I'm better just being told what buttons to press rather than left to decide what and why as then I have no idea.

Posted by: Sara

Thanks! I've had a read of all the info and I'm not clear why any of it would prevent my flats from working at all - But don't try to explain it to me as I'm not technologically aware and big words and science goes over my head. I'm better just being told what buttons to press rather than left to decide what and why as then I have no idea.

I am curious as well, to see why it initially did not work and if I can find a way to make it work like it should. If I have an answer, I'll try to be as clear as possible why it did fail.

What I don't understand is why Maxim calibrated perfectly - as it always does and APP didn't work at all.... I look forward to your findings 🙂 I can assure you now that the SAME calibration frames were being used for both Maxim and APP 🙂 

Not to be beaten I tried again and this time it all worked... sort of ...... I used the same settings as before. 

I do have a satellite trail that doesn't seem to want to go....... I usually use some form of sigma algorithm and that deals with them nicely, but I'm not sure what I should be doing where in APP?

Also the overall calibration in APP is FAR worse than in Maxim as you can see in the attached screen grab. I'm sure it's only a settings issue... I hope so.

*edit* Ah got it... medium and not average in the integration setting... Interestingly as well, I am not trying all sorts of settings, but the Maxim integration is much better regarding noise and very much lower than anything I can get with APP.

Hi Sara,

Your single sub, calibrated  with the 5 bias, 5 darks and 5 flats... I don't think this is too shabby 😉

Very strongly stretched for illustration purpose of the quality of calibration.

I'll upload the masters to our dropbox 😉

Regarding the integration result that you posted, I think that if you feed badly calibrated lights into the registration engine, the result will never be very good. (Best is to first focus on good calibration before starting with integration.)

"Ah got it... medium and not average in the integration setting... Interestingly as well, I am not trying all sorts of settings, but the Maxim integration is much better regarding noise and very much lower than anything I can get with APP."

I think that if you start with good calibration, APP could possibly amaze you on the positive side ; -)

Mabula

Can you tell me the settings you used? I am very much struggling to match the Maxim noise level on integration

Posted by: Sara

Can you tell me the settings you used? I am very much struggling to match the Maxim noise level on integration

Hang on, I'll write down exactly what I did in APP.

1) First of all I created a Bad Pixel Map using only the 5 darks and 5 flats.

you need to load only the darks and flats for the creation of the BPM.

go to 2) CALIBRATE, scroll down and select create Bad Pixel Map:

I used the defaults, hot kappa 2 and cold percentage 50%. If you feed more darks, you can get away with a higher kappa (between 2 and 3 usually).

This created the following bad pixel map, zoomed in screenshot, don't be alarmed by the amount of bad pixels 😉 :

(turn off DPP and set black point at 0, on my RFC is to let APP do this automatically for a BPM)

According to the FITS header (check with details selectbox above image viewer when the BPM is loaded into the image viewer):

HDU1 - NPIX    =              8328304 / raw number of pixels                          
HDU1 - HOTKAPPA= '2,00    '           / kappa value used for hot pixel determination  
HDU1 - COLDFRAC= '0,50    '           / percentage used for cold pixel determination  
HDU1 - NBADPIX =               233946 / number of bad pixels                          
HDU1 - PBADPIX = '2,809   '           / percentage of bad pixels                      
HDU1 - NHOTPIX =               233945 / number of hot pixels                          
HDU1 - PHOTPIX = '2,809   '           / percentage of hot pixels                      
HDU1 - NCOLDPIX=                    1 / number of cold pixels                         
HDU1 - PCOLDPIX= '0,000   '           / percentage of cold pixels                     
HDU1 - NLINPIX =              8094358 / number of linear pixels                       
HDU1 - PLINPIX = '97,191  '           / percentage of linear pixels      

No cold/defect pixels, 3% hot pixels, maybe the hot kappa was a bit too high, but it's effective like you can see in the calirbated stretched result I posted.

2) The next step is to make sure that the bias pedestal is removed from the lights and the flats. Since you supplied dark frames of 300seconds (which don't match the 600sec lights and APP doesn't scale the darks yet 😉 which will come... but scaling is full of problems, it's always better to supply exposure and temperature matched darks if it's possible ) I only performed Bias calibration on the lights and the flats.

So I loaded the 5 bias and 5 flats and clicked calibrate. No outlier rejection in integration since I have only 5 frames to work with (median or average gives almost identical results in this case, if you use more than 20 frames, use average integration and regular sigma or winsorized clipping, only 1 iteration kappa 3 should suffice. Never use linear fit clipping for calibration frames 😉 ! )

The flats will be automatically calibrated by the masterbias which is created first. So we are left with a master bias and a master flat created out of bias subtracted flats.

I dont's use the darks for the light frame calibration, since they don't match for exposure. I only used the darks for the BPM. Which isn't bad, I don't inject noise from a masterdark in the light frame now and your sensor doesn't seem to have a troublesome dark current which would warrent dark current subtraction. If it's not needed, don't do it!

The light frame will be calibrated now with the masterbias, masterflat & the BPM:

left: uncalibrated   right: calibrated

Notice in the screenshots the marks at the light frame in the frame column at the bottom panel : B F BPM

So your light frame is calibrated using a masterbias, masterflat of bias subtracted flats, and a Bad Pixel Map. No dark subtraction 😉 which is better if your sensor doesn't have Fixed Pattern Noise or amp glow in the dark current. I don't know if this calibration will hold up if you integrate 100 subs, but I think it will work very well since I can stretch the calibrated light very strongly without seeing calirbation problems.

one more screenshot of the noise in the calibrated sub by zooming in pretty far, left uncalibrated, right calibrated:

The BadPixelMap is working very good I think 😉 only from 5 darks and 5 flats... with more darks and flats, it will be more robust 😉

Let me know if you can duplicate this calibration path 😉 and how the integration then turns out?

I am uploading all results with your data to our dropbox 😉

OK - I created a BPM as per your reply.. all good so far.

I followed your calibration above, but what you haven't said is how you best create the integrated stack....... I guess that following your post I now have various useful masters, but how is it best to then integrate and stack them with the lights?

At what stage do I add the lights? 

Sorry for so many questions, but without documentation for me, I am lost 🙂 

So clean the frame list panel first, now load the calibration masters, in this case the masterbias, masterflat and the BPM, and load the lights. You can now directly proceed to 3) and further.

Verify visually with the image viewer mode l-calibrated that the calibration is actually good now, if you haven't done so already. How many hot pixels do see reported in the FITS header of the BPM?

For registration, normalization and integration, I would suggest mostly defaults.

If you have more than 20 lights frames use average integration and sigma clipping 1 iteration kappa 3 to start with.

I guess all your lights have almost the same identical field of view? So no need to turn on dynamic distortion correction I think in 4) REGISTER

5) use defaults

6) pixel interpolation, use Lanczos3 with no under/over shoot enabled

6) after a first stack, you can try a second with MBB at 5% to reduce stack artefacts at the borders, but I guess you won't need it if there is vey little movement between the frames.

Finally try a stack with LNC 3 iteration first degree to see what that does?

Let me know it the integration becomes more satisfactory now 😉

Mabula

OK it all works now!! Thanks!

So I did a comparison with the Maxim calibration and the APP calibration with the batch selector in PI.... Interesting to see that the noise level on the APP stacked and calibrated data was higher than the Maxim data. I've attached a PI screen grab. The APP data also looked noisier to the eye.

Thank you very much for the help with this... its an interesting comparison between 2 different programs.

Here's another question too.... When I reopen APP and load up the same lights and masters, as soon as I load the masters the lights get an F B etc beside them..... This doesn't mean that they are already calibrated does it? As I've not enabled the calibration tab.

😉

Sara, are you aware that noise comparison is useless as a metric if the data to be compared isn't normalised?

If you divide a dataset with a factor of 2, then the noise metric is divided by  2 as well 😉 (it's relative not absolute)

So noise isn't something you can compare in an absolute sense, you need to normalise.

A difference in flat calibration implementation can easily create a scale/dispersion difference clarifying the differences in noise values. 

Furthermore outlier rejection can have a huge impact on the found noise value for a frame.

I would love to compare the app stack to the maxim stack after normalisation though 😉

Posted by: Sara

 

Here's another question too.... When I reopen APP and load up the same lights and masters, as soon as I load the masters the lights get an F B etc beside them..... This doesn't mean that they are already calibrated does it? As I've not enabled the calibration tab.

And another question that I don't understand ...... Sorry 

Posted by: Sara

Here's another question too.... When I reopen APP and load up the same lights and masters, as soon as I load the masters the lights get an F B etc beside them..... This doesn't mean that they are already calibrated does it? As I've not enabled the calibration tab.

No, APP hasn't changed your original light frames. APP just shows with F B BPM or D marks, that a suitable calibration frame can be applied. From process 3) on APP will automatically apply the masters to the data internally in the image loaders.

So once I load the files (Lights, BPM, Masterbias and masterdarks) I don't need to touch calibrate again and just go to register? As calibrate is just creating the calibration masters and not actually calibrating the lights? Is that right? 

Posted by: Sara

So once I load the files (Lights, BPM, Masterbias and masterdarks) I don't need to touch calibrate again and just go to register? As calibrate is just creating the calibration masters and not actually calibrating the lights? Is that right? 

the "calibrate"button in 2)CALIBRATE will indeed only create the master frames.

For automatic workflow in APP, you will always pass 2) when you go to 3) then CA is added as a mark in the frame column to show that that process is completed. APP will always scan the loaded frames and check if frames can be calibrated by loaded master frames 😉

"is that right?" - so yes! You can see the calibration as  something that is done internally in the image loader in each process that needs calibrated frames. The frames will only be calibrated when you explicitly push save calibrated frames in 2). And then, new fits frames are created containing the calibrated data. Your original data will never be altered 😉 !

Brilliant - I've written it all down and will do a video for my website soon 🙂

Do you understand my noise comparison argument? I think it's a big mistake made by a lot of photographers worldwide, to treat noise as an absolute metric. It simply isn't.

Maybe I should write a special FAQ for this with clear examples. It's a bit hard to explain and this mistake is being made on all fora worldwide I am afraid.

Posted by: Mabula Haverkamp

Do you understand my noise comparison argument? I think it's a big mistake made by a lot of photographers worldwide, to treat noise as an absolute metric. It simply isn't.

Maybe I should write a special FAQ for this with clear examples. It's a bit hard to explain and this mistake is being made on all fora worldwide I am afraid.

Nope - don't understand it at all.... all that sort of thing goes way over my head and I don't understand it. I can't do figures at all.... sorry

Ok, I'll make a separate post tomorrow about it, since I think it should help everyone to be able to compare their images with a meaningful metric 😉

Sure it will be great to be able to compare the outputs from different programs - Unless it's a one button press I won't be able to do it LOL!