15th Feb 2024: Astro Pixel Processor 2.0.0-beta29 released - macOS native File Chooser, macOS CMD-Q fixed, read-only Fits on network fixed and other bug fixes
7th December 2023: added payment option Alipay to purchase Astro Pixel Processor from China, Hong Kong, Macau, Taiwan, Korea, Japan and other countries where Alipay is used.
First time processing RGB-picture from QHY8pro camera was perfect. Now no other is going well again. What's wrong?
I processed IC434 picture that I took last month with my QHY8pro cooled color camera and all went well. I was surprised by the good quality picture. But it seems to be a one time lucky processing or something, because I am not able to get it done on other pictures I made last weeks e.g. of M81-M82. I'm really getting frustrated about this. I did everything the same as I did the first time but al the results are rubbish.
Horehead nebula went well. Processing M81-M82 went bad, despite using exactly same parameters. I hope someone is able to give me the hint about what I'm overlooking.
This topic was modified 5 years ago 2 times by jheuser
This topic was modified 5 years ago by Mabula-Admin
Looks like you didn't take care of the LP using the LP tool (and maybe lacking some flats?). Or are there specific things that isn't working in your image?
I did try light pollution removal but that didn't work out very well either. But I expected a better result before trying anything like light pollution removal.
What I noticed is that the flats of M82-M81 are very blue, while the flats of the IC434 were neutral. I did same processing but different results.
I processed IC434 picture that I took last month with my QHY8pro cooled color camera and all went well. I was surprised by the good quality picture. But it seems to be a one time lucky processing or something, because I am not able to get it done on other pictures I made last weeks e.g. of M81-M82. I'm really getting frustrated about this. I did everything the same as I did the first time but al the results are rubbish.
Horehead nebula went well. Processing M81-M82 went bad, despite using exactly same parameters. I hope someone is able to give me the hint about what I'm overlooking.
If I look at your M81,82 image, I think there is a data calibration issue here. I can clearly see that vignetting is still present which suggests that either flats are not used or the flat-field calibration was incomplete due to missing bias/dark/flatdarks..
Can you provide more information so we can solve this?
what kind of frames did you use and how many?
For each type, which iso/gain and exposure time did you use ?
Is it with the same camera or another?
If you used flats, did you create new flats for this project?
Looks like you didn't take care of the LP using the LP tool (and maybe lacking some flats?). Or are there specific things that isn't working in your image?
Illumination problems like this should probably not be solved with the LP correction tool, since it indicates improper flat-field calibration. First try to get good flat-field calibration of your lights. Then any remaining gradients are to be corrected with the Remove Light Pollution tool.
I did try light pollution removal but that didn't work out very well either. But I expected a better result before trying anything like light pollution removal.
What I noticed is that the flats of M82-M81 are very blue, while the flats of the IC434 were neutral. I did same processing but different results.
So that might explain the problem then 😉 If the flats are very blue, perhaps the Red and Green channels in your flats are not sufficiently illuminated.
Can you share a flat perhaps ? Or show a screenshot of a flat in APP with the dpp preset at no stretch(data as is). We want to see how that looks including the histogram 😉
It might be that RGB are indeed not equally illuminated in the flats of M81-M82, but strangly the same was with the flats of IC434 and that turned out good.
As an anwser to your other questions: I always use to make flats for each individual session, and always 21 subs.
same 50 darks used on both pictures (10 min exposures)
same 99 bias frames on both pictures. (1 ms exposure)
all my light frames are 10 minutes exposures, -15 deg. C. 22 exposures for IC434, 44 exposures for M81-M82.
No ISO values, it's a cooled RGB-CCD camera, both pictures same QHY8pro camera.
I make the pictures with Nebulosity 3 with dithering on.
Is there a way of showing the FITS-header info?
This post was modified 5 years ago 2 times by jheuser
It might be that RGB are indeed not equally illuminated in the flats of M81-M82, but strangly the same was with the flats of IC434 and that turned out good.
As an anwser to your other questions: I always use to make flats for each individual session, and always 21 subs.
same 50 darks used on both pictures (10 min exposures)
same 99 bias frames on both pictures. (1 ms exposure)
all my light frames are 10 minutes exposures, -15 deg. C. 22 exposures for IC434, 44 exposures for M81-M82.
No ISO values, it's a cooled RGB-CCD camera, both pictures same QHY8pro camera.
I make the pictures with Nebulosity 3 with dithering on.
I did not say that the channels of the flats need to be equally illuminated, I said they need to be illuminated sufficiently 😉 different illumination levels should never be a problem as long as the histograms peaks are clear of both left and right side of the histogram and still in the linear domain of your camera's sensor, which it will for the more recent camera's.
Can you please show the screenshots with "Or show a screenshot of a flat in APP with the dpp preset at no stretch(data as is). " ? The masterflats are now stretched which is not interesting here.
You are also welcome to send me some light frames and the masters that you have right now. I am quite sure something is not right here judging from your integration result 😉
you can send it to support@astropixelprocessor.com using wetransfer.com
Your master calibration frames are not correct, they should be CFA masters and they are debayered already which should not be done. How did you create these? Did you create these from already debayered bias and dark frames?
Mabula
The new masterbias and masterdark are clearly different from the masters of the project IC434 where all was okay 😉
Check the lines in bold :
Fits header MasterBias of IC434:
SIMPLE = T / Java FITS: Sun Feb 10 16:50:31 CET 2019 BITPIX = -32 / bits per data value NAXIS = 2 / number of axes NAXIS1 = 3040 / size of the n'th axis NAXIS2 = 2016 / size of the n'th axis EXTEND = T / Extensions are permitted BSCALE = 1.0 / scale factor BZERO = 0.0 / no offset DATE = '2019-02-10T15:56:01' / creation date of MasterBias SOFTWARE= 'Astro Pixel Processor by Aries Productions' / software VERSION = '1.067 ' / Astro Pixel Processor version CALFRAME= 'MasterBias' / master bias frame INSTRUME= 'QHY8Pro ' / instrument name CFAIMAGE= 'RGGB ' / Color Filter Array pattern GAIN = 1.0 / gain or ISO depending on instrument EXPTIME = 0.001 / exposure time (s) MEAN-R = ' 6,60E-03' / mean of channel-R MEAN-G1 = ' 6,64E-03' / mean of channel-G1 MEAN-G2 = ' 6,64E-03' / mean of channel-G2 MEAN-B = ' 6,60E-03' / mean of channel-B MED-R = ' 6,60E-03' / median of channel-R MED-G1 = ' 6,63E-03' / median of channel-G1 MED-G2 = ' 6,63E-03' / median of channel-G2 MED-B = ' 6,59E-03' / median of channel-B SIGMA-R = ' 8,68E-05' / standard deviation of channel-R SIGMA-G1= ' 8,68E-05' / standard deviation of channel-G1 SIGMA-G2= ' 8,83E-05' / standard deviation of channel-G2 SIGMA-B = ' 8,76E-05' / standard deviation of channel-B NOISE-R = ' 4,56E-05' / MRS gaussian noise estimate of channel-R NOISE-G1= ' 4,54E-05' / MRS gaussian noise estimate of channel-G1 NOISE-G2= ' 4,61E-05' / MRS gaussian noise estimate of channel-G2 NOISE-B = ' 4,62E-05' / MRS gaussian noise estimate of channel-B NUMFRAME= 99 / # number of frames used in MasterBias creation
Now see your new masterbias of M81,M82:
Image #1 SIMPLE = T / Java FITS: Fri Feb 15 16:32:40 CET 2019 BITPIX = -32 / bits per data value NAXIS = 3 / number of axes NAXIS1 = 3038 / size of the n'th axis NAXIS2 = 2014 / size of the n'th axis NAXIS3 = 3 / size of the n'th axis EXTEND = T / Extensions are permitted BSCALE = 1.0 / scale factor BZERO = 0.0 / no offset DATE = '2019-02-15T15:39:22' / creation date of MasterBias SOFTWARE= 'Astro Pixel Processor by Aries Productions' / software VERSION = '1.067 ' / Astro Pixel Processor version CALFRAME= 'MasterBias' / master bias frame INSTRUME= 'QHY8Pro ' / instrument name CFAIMAGE= 'no ' / Color Filter Array pattern GAIN = 1.0 / gain or ISO depending on instrument EXPTIME = 0.001 / exposure time (s) MEAN-R = ' 6,60E-03' / mean of channel-R MEAN-G = ' 6,64E-03' / mean of channel-G MEAN-B = ' 6,60E-03' / mean of channel-B MED-R = ' 6,60E-03' / median of channel-R MED-G = ' 6,63E-03' / median of channel-G MED-B = ' 6,59E-03' / median of channel-B SIGMA-R = ' 8,53E-05' / standard deviation of channel-R SIGMA-G = ' 8,52E-05' / standard deviation of channel-G SIGMA-B = ' 8,63E-05' / standard deviation of channel-B NOISE-R = ' 2,63E-05' / MRS gaussian noise estimate of channel-R NOISE-G = ' 2,51E-05' / MRS gaussian noise estimate of channel-G NOISE-B = ' 2,63E-05' / MRS gaussian noise estimate of channel-B NUMFRAME= 99 / # number of frames used in MasterBias creation
So the problem is that you did not create these masters from undebayered raw data in this case. The dimensions are also not correct which suggest that some of the borders were clipped, which usually happens when software debayers the data. It might be done by Nebulosity or APP. Do you recall debayering the bias and darks ?
My suspicion is that you need to use your old masterbias and masterdark and create a new masterflat from the flats for M81,M82. Before making the masters, make sure that you set the debayer pattern correcly in 0) Raw/Fits and enable Force CFA 😉
I will try as well now.
Mabula
This post was modified 5 years ago 2 times by Mabula-Admin
Your master calibration frames are not correct, they should be CFA masters and they are debayered already which should not be done. How did you create these? Did you create these from already debayered bias and dark frames?
This is the way this QHY8pro creates pictures: debayered.
make sure that you set the debayer pattern correcly in 0) Raw/Fits and enable Force CFA
At least I thougth I did: force CFA.
You might be right that Nebulosity does already some cropping. Because when I uses the capture software that was included with the camera, it gave a different pixel-frame. (bigger) I didn't realize that might be important. I like to capture with Nebulosity because of the possibility of dithering. Maybe I should shoot everything with the software that came with the camera and retry all over again.
But it was because of the good first result in APP with IC434 Horsehead nebula, that I thought it was had noting to do with the capture software.
Okay, apart from the debayering issue, I can also verify that your flats are bad unfortunately. The blue channel is way too strong for your camera's sensor. Apparently the Histogram peak has entered the non-linear domain of your sensor.
To illustrate this I have created 6 screenshots,
The Masterflat from the undebayered flats, unstretched, with and withour neutralize BG in the preview filter:
We would expect that the neutralized masterflat is gray, not colored ! We don't see a lot now, so we do this again, but with stretching of the masterflat:
Now, this does not look good at all. The corners are blue in the neutralized masterflat and we can also from the histogram that blue clearly deviates from red and green 😉 A clear indication that something is wrong with the blue channel in the flats...
And this is confirmed when we apply this to a light, again stretched and with/without neutralize-BG:
The blue corners of the neutralized masterflat show as yellow in the calibrated light due to the divide operation in flat-field calibration.
So, unfortunately, the flats need to be recreated if possible. I will try what the old MasterFlat gives as welll..
Mabula
This post was modified 5 years ago 3 times by Mabula-Admin
Your master calibration frames are not correct, they should be CFA masters and they are debayered already which should not be done. How did you create these? Did you create these from already debayered bias and dark frames?
This is the way this QHY8pro creates pictures: debayered.
make sure that you set the debayer pattern correcly in 0) Raw/Fits and enable Force CFA
At least I thougth I did: force CFA.
You might be right that Nebulosity does already some cropping. Because when I uses the capture software that was included with the camera, it gave a different pixel-frame. (bigger) I didn't realize that might be important. I like to capture with Nebulosity because of the possibility of dithering. Maybe I should shoot everything with the software that came with the camera and retry all over again.
But it was because of the good first result in APP with IC434 Horsehead nebula, that I thought it was had noting to do with the capture software.
You should be able to use Nebulosity, dithering is much better than no dithering so please try 😉
Can't you choose in Nebulosity, not to debayer the frames when captured ? I would think that should be possible.
If data is cropped it gives a significant problem when you want to calibrate it with data that wasn't cropped. You can't tell how to calibrate pixel by pixel then... we can't tell for sure which pixel in the lights is which pixel in dark...
So it is always important that if the capture software gives a different image dimension, that you create all data, so all lights and all calibration frames, with that same capture software 😉 Otherwise things will go wrong for sure...
Thanks a lot for your investigation. I've just tried to calibrate with the old master-flat. Picture is slightly different but still far from good. I'll see how to get along with all of it. Especially how to make better flats with this camera.
I think the first thing I'm going to try is making all pictures with he software that came with the camera and use them for processing. See how that works out.
One last thing: I just did the whole processing in Nebulosity. That seemed to have worked out better. Probably because capturing and calibration/stacking in the same program is better aligned inside the software? But star registration was very
I think it is vital to shoot all data with the same capture software 😉 and the same software version as well.
This holds for both astronomical CCD & cmos camera's, but also for DLSR camera's. There are differences in how the data is stored in the files, especially when it concerns Bayer CFA sensors like yours.
So to get consistent and controllable results, please shoot your calibration frames with the exact same software as your lights ;-). Even different software versions can give trouble, I know from experience with SGP.
And do let me know how your next target turns out 😉
Also, when things like flat frames are indeed bad it's better not to use them at all. You will get vignetting, but then it might be possible to do something with LP correction for instance.
Last night I did a new session, this time of M51 and different type of capturing. Not with the EZcap software that came with the QHY8pro camera because this SW keeps crashing, gives error messages and hangs up. But with an Ascom QHY8pro driver in Nebulosity. Did all the bias, darks, flats as well with same driver en SW.
Stacked in APP, same result as before. Stacked in DSS and noticed nothing, but when opening in APP it gave same picture. Now I noticed dat DDP auto stretching in APP is very high. I could however manage to get a better picture using the 'remove light pollution' tool in APP which gave a far better image. Little tweaking with the other tools like calibrate star colors, saturation and crop resulted in a reasonable picture:
Still not sure how to do better, but I can manage to get an acceptable result.
I wanted to add that the problem is solved. Apparently I didn't use the right setting for this camera when shooting. Gain must be set to 0 and flats needed to be made with no more that 30 % of the max. exposure. According to the manual of this camera. I should have read the manual first. 🙂
Processing the images with APP is no problem anymore.
