[Closed] Astro Pixel Processor 1.058

Astro Pixel Processor 1.058 - quick update with 1 GUI fix

• FIXED, GUI, Fixed a small GUI problem in 2) Calibrate and updated tooltips for outlier rejection filters.

release notes below are of 1.057 which was released the day before 1.058:

- improvement, outlier rejection filters, thanks to extensive testing by Yves van den Broek ( @yves_van_den_broek ) it became apparant that outlier rejection wasn't as efficient as other applications. This has been improved greatly by using more robust statistics in both sigma and winsor sigma clipping. The statistical method to calculate the standard deviation of the pixel stacks used to be the square root of the weighted variance which is very sensitive to outliers.  Weighted, because of the integration weights that can be chosen in integration of light frames. The old statistical method has now been replaced by a Weighted Median Absolute Deviation (MAD) multiplied by a correction factor to have the found value correspond to the expected square root of the variance without being affected by clear outliers in the pixelstacks. The end result is that outlier rejection is now much more robust with even a little amount of frames like 10-20 frames for both calibration frames and light frames.

Example, integration of 96 frames from 5 different telescopes on H-alpha data of the Rosette Nebula (courtesy of Dutch Astroforum.nl forumpower). Rejection is set to sigma clip 1 x 3,5 kappa.

First image is the rejection map of the old rejection efficiency, second is the new efficiency. Note that with the new calculations, rejection is much better and stronger. This is also means that you can suffice with higher kappa values than you were used to previously to get rid of all outliers. This should improve the overal SNR of the integration while rejecting more or all outliers :

This is the actual integration without having enabled LNC, only MBB at 25% :

- innovation/upgrade, new outlier rejection filters, linear fit clipping has been made obsolete and is replaced by 2 new and innovative rejection filters:

1. Local Normalization Sigma clipping
2. Local Normalization Winsor Sigma clipping

Both filters are improvements upon regular sigma and winsor sigma clipping using Local Normalization corrections in the pixel stacks before they are fed to the outlier rejection filters or more specifically:

A local normalization rejection filter is an additional technique to improve both sigma and winsor sigma clipping. Locally, in your integration, all data will be corrected when fed to the outlier rejection filters for more efficient and robust rejection by calculating the local median value of each layer in the stack in a radius of 32 pixels. The pixel values in each pixel stack are corrected using these median values and will therefore correct each pixel for local deviations in intensity in each layer of the stack over the whole field of view. The pixels that survive the rejection filter will be used uncorrected for the actual integration.

This technique makes linear fit clipping obsolete by correctly solving the root problem of local intensity differences between the layers/lights of the stack to be integrated. Since no fitting needs to be done anymore, it is much faster as well.

The result is that you will get very uniform, and robust rejection over the entire field of view.

First image is the new sigma clip rejection efficiency, second is the new Local Normalization Sigma clipping. Notice that outlier rejection of the data is much more uniform now in the areas were the integration layers had deviating illumination levels. This is a very clear proof of having much more reliable outlier rejection at work:

- - innovation/upgrade, new outlier rejection option, a new outlier rejection filter option is introduced called diffraction protection.  This is a new concept with the purpose to prevent rejection artefacts at the star borders. Rejection artefacts can occur if you

1. integrate data with varying star sizes and/or
2. different diffraction patterns.

For instance if you want to

1. combine data of separate nights with big differences in atmospheric seeing or if you want to
2. combine data shot with different telescopes like an APO and a Newton telescope.

You can leave this function disabled by setting it to none. Any other setting will enable this feauture on all rejection filters. If enabled, you can prevent artefacts at star borders by choosing a suitable value. The value correspons to how many pixels are allowed to be rejected in a pixelstack. Realize, that zero pixel values in each pixel stack are not part of pixel stacks. These are removed earlier. An example, I am going to integrate 20 frames of night 1 and 30 frames of night 2. Normally, each pixel stack will consist of 50 pixels. And out of those 50 pixels, normally only a couple of pixels will be real outliers. It's very reasonable to assume that if more than 10% of the pixels are rejected we are not simply rejecting real outliers. In that case we are probably rejecting pixels at star borders, introducing possible ugly artefacts. In this instance, setting the diffraction protection at 5-10 pixels will now start to prevent a lot of artefacts around star borders. An extra benefit is that this feature compensates for the well-known artefacts of winsor sigma clipping. So with diffraction protection enabled, winsor sigma clipping can be used perfectly if you want to combine data of different telescopes with different diffraction patterns. Depending on the amount of frames used in the integration, artefacts can become non-existent while achieving very optimal outlier rejection in pixel stacks where it's really needed. 

First image is the new LN sigma clipping, second with diffraction protection at 10 pixels. 10 pixels is chosen since, I have 5x20 frames of 5 different optical setups. 10 pixels correspons to 10% of each pixel stack. So any pixel stack where more than 10 pixels are rejected are protected now for diffraction rejection. Notice how some outlier rejection hot spots are now gone from the rejection map. These are actual locations of bright stars that have strong diffraction patterns, and since we are combining data with different diffraction patterns, these are now protected and will not show artefacts. Very well known artefacts are from newtonian telescopes giving crosses in the stars due to the secondary mirror construction. You can see this is not visible anymore in the rejection map.

Let's have a look by zooming in on one bright star in the center of the field, left is without diffraction protection, right with. Bottom row are the cooresponding rejection maps. By looking at the bottom right rejection data, we can possible drop the diffraction protection a little bit more. If we look at the diffraction protected data, we have a hard time seeing any artefact due to outlier rejection.

 A summary of the new outlier rejection filter improvements is shown in the image below. All rejection maps are created with 1 iteration and kappa=3.5

TopLeft : old sigma clipping

TopRight: new sigma clipping

BottomLeft: new LN sigma clipping

BottomRight: new LN sigma clipping with diffraction protection

• Fixed, multi-channel processing, a bug was reported by Steve (@gnomus) that he got an error in multi-channel processing with regular normalization. See this topic for details: https://www.astropixelprocessor.com/community/main-forum/integrating-error-is-it-me/   The bug has been fixed as are several others bugs that could pop-up in the case of multi-channel processing and selecting/deselecting groups of images to process them.
• Fixed, backup module, for the purpose of automatic workflow from 1) to 6) APP needs to store process details between each step from 1) to 6). The actual backups did not make hard copies of all relevant information, some information was stored just as references which is not a real backup. Now all data is hard copied, so the backup should restore all previous analytical details when a process like registration is cancelled.
• Fixed, backup module, for the purpose of automatic workflow from 1) to 6) APP needs to store process details between each step from 1) to 6). Previously, if you started integration directly after star analysis, and you cancelled the final integration process, the backup module restored a backup made after star analysis. This was incorrect. Now the correct backup after normalization is restored.
• Fixed, star color calibration tool, sometimes the title of the B-G vs G-R scatterplot was wrong and showed B-R vs. G-R like the other scatterplot.
• IMPROVED/FIXED, RGB Combine tool, the RGB combine tool previously required data layers with the same data type (byte, short, float, double ) and the same data range to be able to make a composite out of compatible data. Now all data is transparantly converted to normalised (data range of 0-1 ) 32bits floats, so data type conversion goes automatically now and the user doesn't need to worry about it.
• FIXED, Integration, in APP 1.056 the integration composition crop mode did not work properly. Works properly again.
• FIXED, Preview Filter, if auto DDP was applied on a rejection/weight or normalization map, sometimes the DDP stretch parameter ST was automatically set to 0 giving an infinite stretch. The ST slider is now limited to a minimum value of 1 in 8bits/16bits data and for 32bits data it's set to 0.0001 (which is extremely low though) in auto DDP mode.
• FIXED, Preview Filter, saving an analytical chart in the image viewer gave a null pointer exception when the stretch select box wasn't selected.
• FIXED, BATCH rotate/resize/modify tools, if you chose to save the results in TIFF format, for each image the bitdepth was asked. Now, it's only asked for the first image and then applied to the whole batch.
• FIXED, RGB Combine tool, if you created a composite but cancelled while saving, then a separate panel that indicated progress wasn't closed.
• UPGRADED/FIXED, Selective Color tool, the tool is now fully integrated into the left control panel, so the tool doesn't have it's own pop-up window anymore. Furthermore, the selective color sliders now show which way you need to slide for more Red or Cyan, Green or Magenta, Blue or Yellow. Adding extra Saturation and Luminance on a specific color now works properly on both linear and stretched data. Finally, you can protect the sky background from injecting chromatic noise by having presets for LOW and HIGH range. The background value is calculated when the tool starts, so the tool reports the sky background value. And there is also a preset to only select the sky background, this will enable you to kill all chromatic noise in the low lights near the sky background if you wish so. The tool can be started directly on an image in the image viewer.

Screenshot showing the new Selective Color tool in action on data of Sara Wager, @swag72 (SHO combination made in the RGB combine tool from H-alpha, SII, OIII narrowband data).

• IMPROVED, Application license file, the license file has been renamed to:

APP-License.lic

to make the license file more compatible with different Operating Systems. If you install APP 1.057, your old license file will be automatically renamed.

• IMPROVED, Application license, entry of your license details is now case insensitive. It used to be case sensitive and in some cases, users had trouble activating their license due to this, so I removed this requirement.
• IMPROVED, Application license, users that need to upgrade/adjust their license code that is expired can now do this more easily. If license verification fails, the user will be asked if he want to enter a new code.
• FIXED, frame list right mouse click menu, remove or delete all files now works.
• FIXED, Bayer Drizzle integration, if the integration scale was set lower than 1.0x and/or the droplet size was set higher than 1.0 pixel, in some cases, a bug could be triggered due to an incomplete calculation of the drizzle weight per target pixel. These settings apply to Bayer Drizzle integration, not regular drizzle integration, because then the scale should be larger than 1.0 and the droplet less than 1 pixel to increase resolution and sharpness.
• IMPROVED/FIXED, Multi-Resolution Support (MRS) Gaussian Noise estimate, the MRS gaussian noise estimate was in error in cases of very high quality or if the image was just full of stars, like in short focal length widefield data. The noise calculation has been improved by better calculation of the noise in the first wavelet layer and by a final verification of comparing the MRS result with the first noise estimate, which is a k-sigma noise estimate of that first wavelet layer. The overall noise calculation is implemented using multiple cores, so the actual wavelet layers used for the calculation can be different across the whole field of view.
• UPGRADE, Calibrated Projective Mosaic algorithms with same camera and optics disabled, these mosaic algorithms are needed if you are constucting a mosaic that has a Field Of View (FOV) larger than 120 degrees. The calibrated projective mosaic registration mode enables you to project the data differently than the default rectilinear projection which is needed in the case of FOVs larger than 120 degrees. The upgrade is substantial. The speed of the mosaic calculation is much faster and is now able to calculate huge mosaics (bigger than 50 panels) with high precision. The upgrade consists of smarter formulation of the mosaic problem in the algebra and this upgrade was already introduced in Astro Pixel Processor 1.054 with settings projective registration model and same camera and optics disabled. So now this algebra upgrade is also applied on the calibrated projective mosaics.

2x 30-panel widefield mosaics  (without and with LNC + MBB) shot at 50mm focal length, courtesy of Scott Rosen (@scott_rosen), without camera calibration are shown:

Same 2x mosaic, but with camera calibration and equirectangular projection instead of rectilinear projection :

Notice how the Field of View is now less distorted due to absence of perspective distortion inherent to the rectilinear projection. The perspective distortion from rectilinear projection becomes a problem with Field Of Views larger than 120 degrees.

• IMPROVED, 4) Register, for mosaic registration there was a setting of number of overlaps, this setting is now made redundant due to the mosaic upgrades.
• IMPROVED, 3) ANALYSE STARS, star detection is now fully automatic, the kappa for detect above noise level is fully automatic for each frame. The value will be lowered if too little stars are detected compared to the automatic minimum #stars target, and increased if it's much higher than the automatic minimum #stars target.
• FIXED, image viewer star map, if a star map was selected but while loading/calculating the frame's star map, another image viewer mode was selected like linear, than the star map loading/calculation wasn't properly shutdown.
• Camera Support, full support is added for Canon EOS M5 (mirrorless)

• IMPROVED/FIXED, Star Color Calibration, the star color calibration tool has been improved and the GUI has been changed so the sliders and their labels are more clear. The Black Body model with modelling for atmospheric extinction coefficients has been improved. The old algorithm was killing the colors in some cases. By adding 2 constraints in the calculation the color saturation is preserved now. The Black Body Model without atmospheric extinction coefficients has been removed since in testing it never gave better results. The Balance RGB model is still an option for calibration. The labels for the constants of the Black Body model now also indicates that adjusting these values has influence on the magenta/green balance.

• IMPROVED, Remove Light Pollution + Correct Vignetting + Calibrate Background + Calibrate Star Colors, these tools have adjusted button sizes and spaces between the buttons and colors are added to discriminate between the buttons.

• FIXED, Frame list panel, @gnomus reported an error in APP where it reported progress marks like Star, Reg, Norm when the frames weren't analysed at all. Internal tracking of calibration, star analysis, registration and normalization parameters was off for frames that had no parameters at all. The marks showed up nevertheless when other frames had parameters due to a bookkeeping error in the code. See the bug report here: https://www.astropixelprocessor.com/community/main-forum/subs-instantly-registered-and-normalised/
• FIXED, Preview Filter, added some auto DDP tweaks that should help in giving good auto stretches on calibration frames like dark and bias frames.