15th Feb 2024: Astro Pixel Processor 2.0.0-beta29 released - macOS native File Chooser, macOS CMD-Q fixed, read-only Fits on network fixed and other bug fixes
7th December 2023: added payment option Alipay to purchase Astro Pixel Processor from China, Hong Kong, Macau, Taiwan, Korea, Japan and other countries where Alipay is used.
I have processed the Nikon D800 M42-A7 data using the files as mentioned in Wei-Hao's log file.
Indeed, APP had trouble with flat-field correction on this dataset. So I tested and have adjusted several things in the APP flat-field modules, the problem was due to 2 factors:
a technical problem in calculating the location and scale of each flat using multiple cores.
the reference lokation and scale used for normalizing the flats between each other is adjusted. The flat is chosen that has median location of all flats.
To compare the result APP gives now, I also ran the data through PI, like Wei-Hao did as well:
These are the 2 stacked results, only stretched with auto DPP in APP with saturation enabled. Very strong stretch with much saturation to clearly see everything.
Left is APP (AAD debayer, corrected bad pixels with BPM), right is PI (VNG debayer, still shows hot pixels..):
So APP does perform the flat-calibration now correctly 😉 I will run some more tests on more data of Wei-Hao and others before releasing APP 1.060.
The Nikon D800 sensor has a shutter technique problem showing of in the masterflat's rejection map, and to top it off, Wei-Hao's Nikon D800 apparently has different sensor dimensions as the original D800 model. Nikon or the sensor manufacturer has changed the sensor dimensions, but the camera is sold under the same name. (perhaps the sensor manufacturer has changed the sensor model?)
I have caught this exception in APP for D800 (and D800E) camera's for APP 1.060, so APP will correctly remove the border. PI (and thus dcraw) is still showing the much thicker top raw raw border on the top side as you can see in the PI integration. The new sensor dimensions are 64 larger in height, and 64 more pixels need to be removed from the top border which dcraw isn't doing yet.
Both issues are shown in the comparison, the dark stripe shown in the masterflat's rejection map reveals uneven illumination on parts of the sensor. Without data rejection on the masterflat this issue is shown identically on the integration, so it's no issue caused by data rejection.:
Kirk, thank you again for pointing out the flat-fielding issue. Flat-fielding worked correctly on a lot of other datasets, but it clearly wasn't robust enough for other datasets as this dataset showed.
Update: Some user report conflict with existing tethering software. In most cases, it's due to the software not supporting dynamic RAW data format, as capture itself usually succeed before program crash when downloading the overscan NEF data. It's possible that such as image dimension, or compression mode was hard coded into the software. DigiCamControl (latest 1.2) will support dark current enabled DSLRs.
I have added the following in the release Notes of APP 1.060:
FIXED, NIKON D800, D800ENEF overscan support, the RAW image dimensions of Nikon DSLRs hacked by Nikon Hacker firmware's can be different. This is due to loading NEF overscan data with the hacked firmware. This is handled correctly for these models now, thanks to data from Wei-Hao Wang @wei-hao_wang.
(It is easy to support other hacked Nikon models correctly, I just need to see a single raw of such an hacked model.)
Okay, back to calibrating Wei-Hao's data... I have gone a step further now. The flat analysis is now more sophisticated and can even reject the bad flats that are present using robust statistics to compare the flats for location and dispersion of the flat histograms 😉 .
About 50% of the flats (15 out of 32) are rejected in this case automatically, so the D800 sensor shutter is a problem to take into account. Left is with the bad flats removed, right is just processed with all provided flats:
3 examples of flats: 1 good and 2 bad:
Clearly by looking at the flats, the problem is apparent. In the integration it caused a dark band without removing the bad flats.
I'll now proceed with other data of Wei-Hao and other's to further test the robustness of all of this..
It's interesting to see those flat images. I think the horizontal bands are a combined effect of the camera shutter and the flickering of the illumination source (a flat LED panel). What I usually did is to average enough of them (more than 20 or even 30), so the effect can be averaged out. To achieve this, I specifically set to have no frame/pixel rejection for averaging the flats.
That being said, I do not recall seeing flat images this bad in a regular basis. On the camera's LCD screen, most of them look OK, if not all. At some point I need to check whether this is a particularly bad set of flats, or indeed most of my flats taken with the LED panel are like this.
Yes, it's the first time I see it like this.. 50% of the flats of the set have these problems. Indeed, must be combination of both shutter and the illumination source I think. Longer exposures should make it much less problematic (which tends to be the case for a lot of flat-field problems.)
I agree, it might be better to not reject pixels in the pixelstacks and average the frames. But in addition, I managed to add a robust detection method for these deviating frames and it works very well. The deviating frames are not integrated in the masterflat.
The way I accomplish this is by assuming that any regular and correct flat will have a certain/fixed ratio (within a certain error window) between
a central value of a specific corner area of the flat
to the central value of the center area of the flat
I calculate this ratio for two opposite corners. Then the deviating frames are easily detected by comparing the 2 ratios per flat to the ratio's of other flats.
In a flat set without deviating flats , only 2.5-5% of the flats are not used. In this flat set, 50% is rejected, I did a visual verification and it was 100% correct.
I understand your method. It can even be generalized to divide the image into a few large grids, which is equivalently shrinking the image to very low resolution. Then the "low-res" images can be quickly compared with each other to find outliers.
However, no matter in this grid method, or your center vs corner method, there is a danger. In relatively wide-angle imaging (with focal length less than 300 mm or so), sometimes we do twilight sky flat. However, there could be gradient in the twilight, no matter where we point our lens to. The standard way to solve this problem is to point the camera at many different parts of the sky as well as to rotate the camera angle. Then the many flats can be averaged and hopefully the sky gradient will be averaged out. In this situation, it is expected that every flat looks somewhat different from each other, and every flat is still needed. The rejection method you mentioned may remove some of the flats in this situation.
A potential solution is to allow the users to turn off this option. However, adding too many options to the UI will confuse the users and make the program less user friendly.
Yes, I fully agree, this separate rejection method will be off by default 😉 for the mentioned reason. I will think about leaving it as an option in the GUI or not. Probably this issue is best dealt by the photographer himself understanding the issue with the sensor and flatpanel combined so I might not include it in APP's next release.
This is my processing of Wei-Hao's Nikon D800 NGC253 data of 2016-11-04 with all of the supplied flats processed with APP 1.060-beta up until star color calibration. The flat-field correction is working well (no under correction in the corners) except for part of the sensor chamber showing at the bottom. Left is the whole Field Of View of the data, right is a crop 😉 : One of the dust spots isn't calibrated out, since this apparently isn't in all subs..
This is only 10 x 488seconds, so I should probably integrate with median integration since I do see a satellite stripe... this one is with average integration and no outlier rejection.
I have implemented Multi-factor (3 factors) dark frame scaling as well, which works very robust. I can see from the factors that the temperature was dropping several degrees during the imaging session and that the temperature was a couple of degrees higher when compared to taking the darks themselves. The factors are dropping each next frame towards 1.25 and the actual darks were shot 90 minutes after capturing the lights according to Wei-Hao's logs, so that makes sense as well.
Here's another result from Wei-Hao's M42-A7 dataset with the improved dark-frame scaling and I removed the bad flats from the flats to make a better masterflat.
I have implemented Multi-factor (3 factors) dark frame scaling as well, which works very robust. I can see from the factors that the temperature was dropping several degrees during the imaging session and that the temperature was a couple of degrees higher when compared to taking the darks themselves. The factors are dropping each next frame towards 1.25 and the actual darks were shot 90 minutes after capturing the lights according to Wei-Hao's logs, so that makes sense as well.
Here's another result from Wei-Hao's M42-A7 dataset with the improved dark-frame scaling and I removed the bad flats from the flats to make a better masterflat.
Cheers,
Mabula
That's fantastic Mabula, great work.
I have just loaded the 3 main images from this thread in separate tabs, (1059, 1065 & 1096), clicking on each in sequence and there is definitely a noticeable progression to them.
Image 1096 looks very much even to the eye as well as better colour saturation, mainly the blues are clearer. Plus the stars & nebulosity looks more 3 dimensional.
Looking forward to the next release but willing to wait for you to iron out all the bugs first.
Finally had some spare time to go back over this data using the new version of APP, what a massive difference.
I took on board what Mabula @mabula-haverkamp-administrator had said about the Flats and I manually went through each one and rejected any that showed any form of hard/straight edge & out of the 32 Flats I only used 15.
Anyway, here is my take on the data
One thing that I have noticed Mabula is that when using the 'remove light pollution' tool it automatically adds 'cbg' to the end of the filename as well, this is before I have performed a 'calibrate background', ending as '-lpc-cbg.fits', shouldn't this just be '-lpc.fits'?
Then after doing a 'calibrate background' the filename ending is '-lpc-cbg-cbg.fits' and shouldn't this now be '-lpc-cbg.fits'?
Indeed, I threw away the same amount of the flats (I believe).
It's a great image by the way 😉 the subtle colors in the nebula are beautiful. Thanks to @wei-hao_wang for shooting and providing the data, it turned very helpfull for me as well to improve significantly in calibration in APP.
Regarding Light Pollution correction, the tool performs background calibration as well now. This changed a couple of versions ago.
So if you have done LPC, you immediately do CBG (Calibrate BackGround) as well ;-).
The selected areas to perform the light pollution correction can also be used to perform the background calibration, since these areas should be representative of the actual sky background.
If the LPC version looks good (due to good LP correction using area selectboxes on the sky background), then the background calibration must be good as well, since it used the LP corrected data from those area selectboxes 😉
I apologize for bringing this old thread back. People seem to reach this thread via internet search from time to time, and used the Google drive link that Mabula provided to access to my files. Unfortunately Google changed the security policy and the old link no longer works. (It still leads to the right location, but visitors no longer get permission automatically.) This is the new link:
