[Closed] Astro Pixel Processor 1.055

Astro Pixel Processor 1.055

• FIXED, saving calibrated frames, if you tried to save calibrated frames without any lights loaded, APP would warn the user, but all buttons would stay disabled...
• FIXED, Mosaic registration, same camera and optics disabled, projective registration model, if a frame couldn't be registered that would be part of the mosaic, the mosaic algorithm could break down and give weird results like increasing RMS while performing the regression.
• IMPROVED, star analysis, detection of stars has been improved significantly for high quality/SNR frames with lots of nebulosity, the old method didn't work optimally. (triggered by testing on high quality frames of the Wide-field Infrared Survey Explorer (WISE) that images in mid infrared bandpasses, pointed to me by Judy Schmidt). 
• ADJUSTED, star analysis, the detect above noise kappa setting now has a range of 6 - 250. A value of 80 now corresponds, more or less, to the old value of 8. The reason for this is, that APP now uses Multi-Resolution Support Gaussian noise for dispersion calculation in the starmap. Previously APP used the root of the variance (standard deviation) for the dispersion, which was subject to outliers from the stars themselves.  The data is flatfielded using a 9x9 median filtered plane that is subtracted from the data to find the stars. The gaussian noise is then calculated from this flat-fielded data. For high quality data, you can now find lots more stars by lowering the kappa value. Only lower the kappa value from the initial 80 value, if APP detects little stars and registration fails. Normally, you won't need more stars then 2500-5000 per frame for mosaics and 1000-2500 for regular registration.
• ADDED CAMERA SUPPORT, full support is added for Canon EOS 5Ds, EOS 5Ds R and EOS 5D Mark IV
• IMPROVED, LIGHT FRAME integration & outlier rejection filters, light frames are always integrated with weights per pixel in the pixel stack, that means that the average and median integration methods are actually weighted average and  weighted median integrations. The average and median values are also calles central values. This improvement uses improved statistics to calculate the standard deviations in the pixel stacks relative to these central values. The standard deviations are needed in all outlier rejection filters. The standard deviations are now so-called "corrected weighted standard deviations" relative to the central values. "corrected" indicates that a correction factor is applied to take into account that we are never able to calculate the exact standard deviation from an unlimited amount of samples. (Or.. we can only sample part of the distribution, so we need to correct for that.)
• IMPROVED, INTEGRATION SPEED, APP was really slow when integrating several hundred frames, especially on conventional harddrives. This has been improved significantly using 2 upgrades in the integration engine.

1. First, reading from the file mapper on the harddisk, (containing all data/frames/layers of the integration) is now simultaneous with calculating the pixel stacks. This increases speed immediately with a factor of 1.5-1.75x depending on hardware configuration.
2. Furthermore, because the integration is strongly limited by harddrive performance in the old integration engine,APP will now greatly reduce the amount of IO calls needed for integration by dynamically adjusting the Read and Write buffers for the file mapper. APP used to have fixed read/write buffers of only 8KiloBytes. Depending on the amount of system memory, these buffers can now become as large 1024 KiloByte or 1 MegaByte.  If you integrate hundreds of frames, integration speed will be increased now with a factor of 2-10x on both SSD and conventional SATA/PATA drives from this improvement alone.

These two upgrades combined will give a speed increase with a factor of 2-20x largely depending on the hardware, which is a huge performance increase. The actual speed increase depends on lots of factors, like the amount and speed of CPU cores, the amount of memory available, harddisk latency and, the actual maximum read/write speeds of your harddisk. This improvement holds for integration of all types of frames. Check the link below for more information and test results:

https://www.astropixelprocessor.com/community/faq/integration-speed-improvements-in-version-1-055-and-the-influence-of-cpu-cores-memory-and-hdd-on-performance/

• FIXED, FITS CFA debayering, super pixel modus wasn't working correctly, image dimensions weren't reduced with a factor of 2, this is now fixed.
• FIXED, saving calibrated frames, if monochrome fits CFA frames were saved with the "force CFA" setting enabled and "CFA pattern" at  "supported" the files were saved with metadata:

CFAIMAGE= 'supported'          / Color Filter Array pattern

This has been corrected to show

CFAIMAGE= 'RGGB'          / Color Filter Array pattern

Because RGGB is the defualt CFA pattern in APP.

• FIXED, INTEGRATION outlier rejection filters, once in a while, an error could occur in the outlier rejection filters, especially in the winsor sigma clip filter. If a filter, for some reason, did find a sigma value of 0 then all pixels were rejected, giving an error in calculation of the average or median value of the pixel stack. This is now properly caught.
• FIXED, INTEGRATION, processing of float/double precision data with NaN pixel values gave an error exception.  These NaN pixels are now removed from the pixelstacks in integration. Thanks to Judy Schmidt that provided me with WISE data that has some pixels with NaN values. All Statistical methods (which were a lot !) needed to be checked for possible NaN values. By enabling fast detection of these values before starting calculation of the pixelstack and possible outlier rejection, these pixels can be removed quickly and won't slow down the other statistical methods.
• FIXED, auto DDP stretch, in case of astroscapes (images with part sky and landscape), the auto DDP function didn't function properly. This is now fixed:

Data courtesy of Rudy Pohl:

old auto DDP stretch:

New auto DDP stretch:

Data courtesy of Frank Sackenheim:

old auto DDP stretch:

new auto DDP stretch:

• FIXED, BAD PIXEL MAP, in case of very low gaussian noise values in the dark signal, the calculation of the lokation of the Hot pixels was sometimes wrong, due to an approximation of the noise calculation. This is now fixed by using the most robust noise metric, the Multi-Resolution Support gaussian noise metric. This was reported on 4 minute ISO 200 darks with a Nikon D750, see https://www.astropixelprocessor.com/community/faq/bad-pixel-map-2/#post-1477 for more information.
• ADDED, BAD PIXEL MAP, the FITS metadata of the Bad Pixel Map now contains information on the number of darks and flats used in the creation of the Bad Pixel Map.

FITS HDUs: 1
HDU1 - SIMPLE  =                    T / Java FITS: Sun Oct 22 20:09:49 CEST 2017      
HDU1 - BITPIX  =                    8 / bits per data value                           
HDU1 - NAXIS   =                    2 / number of axes                                
HDU1 - NAXIS1  =                 6032 / size of the n'th axis                         
HDU1 - NAXIS2  =                 4032 / size of the n'th axis                         
HDU1 - EXTEND  =                    T / Extensions are permitted                      
HDU1 - DATE    = '2017-10-22T18:09:49' / creation date of bad pixel map               
HDU1 - SOFTWARE= 'Astro Pixel Processor by Aries Productions' / software              
HDU1 - VERSION = '1.054   '           / Astro Pixel Processor version                 
HDU1 - CALFRAME= 'bad pixel map'      / bad pixel map for instrument NIKON D750       
HDU1 - INSTRUME= 'NIKON D750'         / instrument name                               
HDU1 - CFAIMAGE= 'no      '           / Color Filter Array pattern                    
HDU1 - NPIX    =             24321024 / raw number of pixels                          
HDU1 - HOTKAPPA= '2,00    '           / kappa value used for hot pixel determination  
HDU1 - COLDFRAC= '0,50    '           / percentage used for cold pixel determination  
HDU1 - NBADPIX =               201827 / number of bad pixels                          
HDU1 - PBADPIX = '0,830   '           / percentage of bad pixels                      
HDU1 - NHOTPIX =               201827 / number of hot pixels                          
HDU1 - PHOTPIX = '0,830   '           / percentage of hot pixels                      
HDU1 - NCOLDPIX=                    0 / number of cold pixels                         
HDU1 - PCOLDPIX= '0,000   '           / percentage of cold pixels                     
HDU1 - NLINPIX =             24119197 / number of linear pixels                       
HDU1 - PLINPIX = '99,170  '           / percentage of linear pixels                   
HDU1 - NUMDARKS=                    6 / # of dark frames used in creation of the BPM  
HDU1 - NUMFLATS=                    0 / # of flat frames used in creation of the BPM  
HDU1 - END                                                                            

•  ADDED, MasterBias, MasterDark, MasterFlat, the fits metadata now contains the number of frames that were used in the creation of the master frame.
• IMPROVED,FIXED, creation of calibration masters while creating the Bad Pixel Map, if you create a Bad Pixel Map, you now have the choice to create the masterbias, masterdark and/or masterbias as well if the required frames are loaded. You can also choose to only create the Bad Pixel Map.

• FIXED, calibration frame zero gain, if a calibration frame was taken with zero gain, the metadata of the resulting master calibration frame, didn't report the zero gain. Now it does report zero gain. Non-zero gains were already shown correctly.
• FIXED, processing already registered frames, if frames were already registered, then in 4) REGISTER, you would process them in the "no registration" mode. In this mode, the star density wasn't calculated properly and therefore the quality calculation wasn't correct. Furthermore, this error caused integration based on quality weights to fail, the integration was done with zero weights resulting in a totally black integration. Error was reported by Steve in this thread: see this topic : https://www.astropixelprocessor.com/community/main-forum/integration-weights/
• FIXED, registration, in some cases, with distortion correction enabled, frames that had practically no translation (less than 1 pixel) and rotation (less than 0.1 degree) relative to the reference frame could fail to register due to instable distortion correction. These execeptions are caught by studying the homography between the frames and for these frames distortion correction is initially disabled to find all the initial star pairs between the frames. The multiple view distortion correction algorithms will still correct these frames for optical distortion though.
• IMPROVED, registration, multiple view dynamic distortion correction algorithms have better adjusted convergence criteria. Previously, some algorithms converged already, but would still continue for some time...
• UPGRADED, Star Color Calibration, the upgraded tool will give better and less green results. You will need to select a star group not influenced by nebulosity or a background galaxy. The star color calibration is based on assuming that the average star of the selected stars is a white star, this means that you will need to select a large enough part or parts of the sky to have enough stars to be used in the calibration. I am still working on further improving this tool, so work is not finished yet. Some examples, first image is uncalibrated, second is calibrated using the stars in the selected area select boxes:

M51 LHaRGB composite (courtesy of astroforum.nl forumpower), before and after calibration:

Widefield Milkyway (courtesy of Frank Sackenheim):

 M31, the Andromeda Galaxy (courtesy of Michel Swerissen):