15th Feb 2024: Astro Pixel Processor 2.0.0-beta29 released - macOS native File Chooser, macOS CMD-Q fixed, read-only Fits on network fixed and other bug fixes
7th December 2023: added payment option Alipay to purchase Astro Pixel Processor from China, Hong Kong, Macau, Taiwan, Korea, Japan and other countries where Alipay is used.
Today I have had all sorts of calibration warnings which I don't understand at all.
I took 8 sets (8 filters) of flats and dark flats using NINA's Flats wizard. I loaded them into APP to create master files. APP created the MDF as it should. But when APP started to create the MF, I got this critical warning. APP did not seem to recognize the MDFs it just created.
But they were all there, image below. (This has in fact happened to me before!)
I removed the dark flats, kept the MDFs and rerun the calibration. No warning this time. So I thought all was OK. (MFs were created in both runs).
Now I want to use the MFs and MDFs to calibrate the lights. Immediately I had this warning.
This time I had these calibration files loaded:
I pressed OK to the warning and kept going, but I have absolutely no idea if the calibration works OK or not, or if the calibration warning is incorrect. I see no reason for this warning.
1. Why is this happening at all? 2. How can I see (numbers or otherwise) if the MDFs have been used when creating the MFs? 2. How can I find out if APP is using the MDFs as it should when stacking lights?
1. In order to help you find out, we'll need more info. How many flats and dark flats are you loading per session? Are you loading any flats or dark flats in more than one session? Do all flats and dark flats in a single session have the same exposure time or do they vary because of the use of a flat wizard?
2. You can see that information in the console log of APP.
3. APP should not use the MDFs on the lights. The MDFs are only used to calibrate the flats. Once the MFs have been created, the MDFs are no longer used.
Now I want to use the MFs and MDFs to calibrate the lights. Immediately I had this warning.
This time I had these calibration files loaded:
Hi Heno @heno, If you only load MasterFlats to calibrate lights, then this warning should always appear. This is expected. The warning clearly states why the warning is thrown:
you need to add either bias and/or darks to calibrate the lights to have the flats as expected. Both Flats and Lights need to have the sensor offset subtracted for flat-field calibration to work 😉
I took 8 sets (8 filters) of flats and dark flats using NINA's Flats wizard. I loaded them into APP to create master files. APP created the MDF as it should. But when APP started to create the MF, I got this critical warning. APP did not seem to recognize the MDFs it just created.
But they were all there, image below. (This has in fact happened to me before!)
@heno, if all frames (flats and darkflats) were loaded correctly with the filter assignments then the warning should not happen provided that you use the latest APP version. Do you want me to test it? If so, upload a subset of frames of all 8 filters using the upload instructions on the top left of the forum and let me know once uploaded 😉
@wvreeven 1. There are 20 flats/dark flats pr. filter. The exposure time (and gain used) may be different from filter to filter, but the same for each set (filter). 2. I did not check that, is the log saved? 3. I did not know that. This means that if I use MF there is no need to load the MDF for light calibration? I can delete them then. (I always take flats and dark flats together)
@mabula-admin Thanks for your reply. This part (your first answer) is totally my mistake. Either I have forgotten to load the dark file or accidentally removed it. Not sure. And I did not read the error message carefully enough. Sorry for wasting your time like this.
@mabula-admin Regarding your second answer about creating the MF and MDF, I restarted my PC this morning just to be sure that everything was fresh and tried again with only one filter this time. I had no warnings. When comparing the meta data for the MF I created yesterday and the one created today, they are identical down to the last digit. I can not tell them apart. I would have thought there had to be some difference if one MF was created without the use of an MDF? How would the numbers change, if at all? I will create MFs the rest of the filters as soon as the current integration finishes. I will come back with the result.
3. I did not know that. This means that if I use MF there is no need to load the MDF for light calibration? I can delete them then. (I always take flats and dark flats together)
Let me check with Mabula to be absolutely sure, but once the MF has been created you should be able to delete the MDF. On the other hand, if you often shoot flats at the same exposure time, temperature and gain you may want to store the MDF so you don't have to reshoot the dark flats.
@heno I have checked and verified that the MDF indeed only is used for calibration of the flats when the MF is created. As soon as the MF has been created, APP no longer needs the MDF and you may do with it as you please, including deleting it.
I just tried again, loaded up flats and dark flats + BPM for the remaining seven filters. It started out well as MF for blue and green were created without warnings. Then on Ha, Lum, OIII, Red and SII I had the same calibration error warning as yesterday. The MDF were all there. If I'm doing something wrong, I have no clue what it is. This I have done a hundred times and I'm not doing anything different. And the MDFs looks fine to me. I will upload the files, 10 of each for each filter. The file name do not indicate which color the filters are, so here is an index: Blue - 1.00s Grn - 2,08s Ha - 2,64s Lum - 1,35s OIII - 2,14s Red - 3,36s SII - 4,10s The dark flats are the ones with (1) in the file name. @mabula-admin For your info.
I just tried again, loaded up flats and dark flats + BPM for the remaining seven filters. It started out well as MF for blue and green were created without warnings. Then on Ha, Lum, OIII, Red and SII I had the same calibration error warning as yesterday. The MDF were all there. If I'm doing something wrong, I have no clue what it is. This I have done a hundred times and I'm not doing anything different. And the MDFs looks fine to me. I will upload the files, 10 of each for each filter. The file name do not indicate which color the filters are, so here is an index: Blue - 1.00s Grn - 2,08s Ha - 2,64s Lum - 1,35s OIII - 2,14s Red - 3,36s SII - 4,10s The dark flats are the ones with (1) in the file name. @mabula-admin For your info.
I have used the latest version available at the top of the forum.
I have made the Masterflats twice without any issues.
1) first try, I have loaded the darkflats and used te filter tag to assign the filter name. All worked perfectly. In the console windows I verified that the correct MasterDarkFlats were used to calibrate the flats.
2) second try, I have loaded the darkflats for all channels. Still all worked perfectly... but there is a slight issue then with the exposure times. The exposure times of the O3 = 2,18s and Grn = 2,14 are so close, that APP will consider it the same exposure time. So in this case the O3 and Grn darkflats are combined into 1 MDF. But then that MDF is used to calibrate the O3 and Grn flats, which is perfectly okay since the gain matches and exposure times so close to each other will not affect the results. Dark current signal with those exposure time differences is simply the same...
So I can only assume that you might not be using the latest version?
Hi Mabula @mabula-admin Thanks for doing the testing and your response. I checked the PC download folder and the I actually had downloaded 1.083.3. But I cannot be 100% sure if it was installed. So I have now reinstalled 1.083.3 just to be absolutely sure. (I also noticed that I can press the license button to see which version is running. Learning something new every day. 😀 ) So to your other responses: 1. This is what I always do. I did not know it could be done any other way. 2. If I understand this correctly, I can actually load dark flats and select "all channels" and APP will sort out which belongs which based on exposure time. That will save some time in the future.
I'll go on with the calibration and let you know how it went.
EDIT:
Calibration finished without warnings, all good. Thanks!
As you see I have an MDF for each of Ha and OIII with correct Gain and exposure in each case but still get the warning about it not being possible to calibrate flats. I always create Flats and Darkflats in same session - same temperature etc, so they should be completely compatible.
In this example I have 24 Flats and 24 DarkFlats for each filter.
I am using APP 1.083.2 Linux x64.
All frames taken using kstars 3.6.2.
I always ignore the warning but still worried I could be messing things up.
@nickscosmos This is a known issue in APP 1.083 when mixing flats and darkflats of different exposure times. Please either try to process the Ha and OIII files separately or use 2.0.0-beta7.
Not sure what the problem is. The exposure times and Gains for flats and darkflats are exactly same to all figures so it should be able to match Ha flats with Ha darkflats etc.
Until I find what the issue is, am I correct in assuming that if I create an MF for each filter separately then integrate all the lights together (Ha and OIII) then I should get fully registered integrations - one for each filter ?
@nickscosmos Yes, you should get fully registered integrations, one for each filter, that way. If you process them separately then you can load the integration results as lights (no calibration files) and then go to tab 5 and normalize them. That will also register them. When done you can save the normalized files and process those as you normally would.
ok thanks for your help. Yes I need to register integration together if done separately, otherwise will play it safe and create masters first if doing it all together. Still learning I am afraid.
This is not a calibration warning per say, but it sort of belongs here, I think. Tonight I noticed that I have pixels clipping on some images, and I don't quite undersand why. When capturing images I'm nowhere near 0 ADU, usually the lowest pixel value is around 1000 ADU, 800 minimum. Can anyone explain why this is happening?
@heno The pixels of the lights may not be near zero but those of the calibrated lights apparently are. This means that the background level of the lights is close to the mean value of the darks. Since the value of the darks fluctuate around that mean value, calibrating the lights with them makes some pixels get a value of zero or even a negative value. The solution is to take longer exposures for the lights.
Well, I still do not understand to be honest. The mean value of the master dark is 9,78e-003. The mean value of a (randomly picked) light is 1958 and the lowest pixel value is 1564. Explain to me how subtracting the dark from the light can give negative pixel values. Obviously there is something going on that I am missing, and I hate when that happens. But it just don't make sense to me, yet.
Well, I still do not understand to be honest. The mean value of the master dark is 9,78e-003. The mean value of a (randomly picked) light is 1958 and the lowest pixel value is 1564. Explain to me how subtracting the dark from the light can give negative pixel values. Obviously there is something going on that I am missing, and I hate when that happens. But it just don't make sense to me, yet.
Hi @heno, both the(uncalibrated) light frames and the masterDark (or Bias) will have a mean/median value and a spread of noise (dispersion/scale/standard deviation) surrounding that median value. If the median value of the (uncalibrated) light is less then the median value of the MasterDark + kappa*standard deviation, some pixels will simply clip after dark subtraction. The kappa can be as high as 20! If the integrated files look okay, you have nothing to worry. But the warning indicates that you will probably benefit from longer exposures, or that the masterdark has an issue like being composed of too little dark frames for example.
You need to realize that both the light and the masterdark have a dispersion surrounding their median (or average) value. On specific pixels that dispersion can be that large that the uncalibrated pixel will clip after being subtracted by a masterdark pixel that would be median + Kappa*dispersion where kappa can be large.
@Mabula First of all, thanks for taking time to answer this. Secondly, I'm not asking this to start an argument or make myself interesting. I really want to understand this and why it is happening.
Is kappa the same as standard deviation? If so, in said light frame above it is 180. In the dark frame it is 3,98e-004. (I tried to google Kappa, but I could not find anything that made sense to me.) Or is Kappa something else entirely?
Usually I take 20 dark frames, I can easily increase that. Would higher offset affect/prevent such clipping? I see no adverse effect on the images so I will probably just let it slide for now and see if it happens again. But I know it has happened before.
