[Closed] Astro Pixel Processor 1.076

Astro Pixel Processor 1.076

1.076 is a very big update with many improvements, most notably:

• IMPROVED LICENSE ACTIVATION, on computers where the amount of cpu threads is variable (some netbooks), the user is no longer forced to keep re-activating his license.

• MUCH IMPROVED RAW FILE TYPE SUPPORT, this is support for raw file types of regular consumer camera's. The currently supported RAW file types are
  • CANON CR2 & CR3
  • NIKON NEF
  • SONY ARW
  • PENTAX PEF
  • OLYMPUS ORF
  • PANASONIC RW2/RAW
  • LEICA LRW/RAW
  • HASSELBLAD 3FR/FFF

• FULLY AUTOMATIC INTEGRATION SETTING that will control average/median integration and outlier rejection filter settings.

• lots of improvements for MULTI-CHANNEL & MULTI-SESSION processing.

• BIG UPDATE of the RGB COMBINE TOOL including composite presets and an important bug fix. The tool is much faster now and supports up to 9 channels.

• Significant update of the NORMALIZATION engine which is now adaptive pedestal aware and much faster...

• Significant update of the STAR ANALYSIS module which is almost twice as fast now.

• Significant improvements in algebra resulting that multiple view optical distortion correction in Registration is performed much faster !

• MANY KNOWN ISSUES AND BUGS ARE SOLVED 🙂

• MANY CODE OPTIMIZATIONS in several vital modules of APP now show a greatly improved performance of APP 1.076 versus APP 1.075. Especially the star analysis module , multiple view optical distortion correction, and the normalization engine are much faster now. Depending on your data, you will notice that APP 1.076 will be 20% to possibly even 40% faster from calibration to integration. I think that users with camera's with big sensors will see very big speed improvements in the normalization engine especially. The next image shows a performance comparison of APP 1.075 versus APP 1.076 on LRGB multi-channel processing of nearly 200 light frames, APP 1.076 is 33% faster in this test:

CAMERA SUPPORT IMPROVEMENTS:

• IMPROVED, LIBRAW UPDATED TO LATEST VERSION 201910 SNAPSHOT, we have updated the LibRaw library to the latest version.
• IMPROVED, RAW CONVERSION DETAILS ARE NOW REPORTED, you can now find all the relevant Raw Conversion details in the Console Window:

• IMPROVED, CAMERA SUPPORT, ADDED SUPPORT FOR CANON CR3 using LibRaw, the following camera models have been tested and are now supported: Canon EOS M50, EOS 250D/ REBEL SL3, EOS R, EOS Ra, EOS RP, EOS M6 Mark II, EOS 90D
• IMPROVED, CAMERA SUPPORT, ADDED SUPPORT FOR PENTAX PEF using LibRaw, supported camera list can be found here: https://www.astropixelprocessor.com/community/camerasupport/pentax-pef-tested-camera-models/
• IMPROVED, CAMERA SUPPORT, ADDED SUPPORT FOR OLYMPUS ORF using LibRaw, supported camera list can be found here: https://www.astropixelprocessor.com/community/camerasupport/olympus-orf-tested-camera-models/
• IMPROVED, CAMERA SUPPORT, ADDED SUPPORT FOR PANASONIC RW2/RAW using LibRaw, supported camera list can be found here: https://www.astropixelprocessor.com/community/camerasupport/panasonic-rw2-raw-tested-camera-models/
• IMPROVED, CAMERA SUPPORT, ADDED SUPPORT FOR LEICA LRW/RAW using LibRaw, supported camera list can be found here: https://www.astropixelprocessor.com/community/camerasupport/leica-rwl-tested-camera-models/
• IMPROVED, CAMERA SUPPORT, ADDED SUPPORT FOR HASSELBLAD 3FR/FFF using LibRaw, supported camera list can be found here: https://www.astropixelprocessor.com/community/camerasupport/hasselblad-3fr-fff-tested-camera-models/

• IMPROVED, CANON CR2 & NIKON NEF CONVERSION, APP will now use LibRaw as well for conversion of CR2 and NEF files. This means that the raws will load faster and also that the newest NIKON raw compression is supported as included in camera's like the Nikon Z7. Your pre-existing calibration Master frames should still work the same, because the calibration data in APP has always been based on the whole sensor and the LibRaw conversion still allows us to use the whole sensor for calibrating the data.

• IMPROVED, RGB COMBINE TOOL, the tool has received a big update.
  • FIrst, the code is now optimized and so the composite creation will be much faster.
  • In addition, you can now process a composite consisting of 9 different channels (previously, 6 channels was the maximum).
  • If you load an image/channel twice, because you want those to be added with different factors, APP will not load the data twice, consuming double memory, instead it will simply reference the first loaded image.
  • The tool now has quick presets for composites like the Hubble palette, S2HaO3 -> RGB , LRGB, LHaRGB, HaO3O3 -> RGB etc... you can choose the composite formula while loading the data. Once loaded, you can still quickly try different formula and study their influence on the resulting quality and colors. Off course, you can still manipulate all multipliers as before, the new presets should help you to get started quickly with a usefull formula for your particular data. The best looking formula will always depend on the data, so the presets are there to point you quickly in the right direction. The following images show the following composite formula presets SHO 1 & 2, HSO 1 &2, HOO 1&2 on exactly the same data set of the Rosette nebula in S2, Ha & O3 narrowband filters:

• • there was a bug in the RGB Combine tool as demonstrated in this topic: https://www.astropixelprocessor.com/community/main-forum/good-lrgb-stacks-show-up-as-very-dark-in-rgb-combine-tool-even-at-30-stretch/ This problem has been fixed now.

• • the tool has been further improved with the option to choose how all channels in your composite are normalized with each other, just like in 5) Normalize. Previously, the frames were only normalized additively. Now, by default they are normalized with Multiply-Scale which is the recommend normalization when you are creating a composite using different filter data and of different sessions. The first image shows a bad composite result in APP 1.075 due to the bug, the second image shows the same composite in APP 1.076 where the bug is fixed and where better normalization has been applied:

• IMPROVED, SELECTING REFERENCE FRAME FOR REGISTRATION & NORMALIZATION, at the end of Star Analysis a reference frame is automatically chosen. If the multi-channel mode is enabled and/or different instruments/camera's have been used or binned and unbinned data is to be processed, APP will now make a smarter choice for the initial reference frame. The instrument is chosen with the most pixels, an unbinned frame is chosen over a binned frame. And a Luminance channel is chosen if available, otherwise the green channel and otherwise the H-alpha channel to provide the reference. If during registration (normal mode), APP is not able to register all frames, it will automatically switch the reference frame to another camera or channel until all data can be registered.
• FIXED, 4) REGISTER, PROJECTION ROTATE, due to an error in the algebra, the pre-rotation of the projection of the Field Of View (FOV) did not work correctly for regular integration. It did work properly for drizzle integration. The field of view was not properly shown in the integration Field Of View. This is now fixed. First image shows the problem that is fixed, second image shows the correct result after the fix:

• IMPROVED, ADAPTIVE PEDESTAL in METADATA, the adaptive pedestal is now always stored in the metadata of your frames. So you will find this value in the calibrated frames, registered frames, normalized frames and also in integrations:

• IMPROVED, BACKGROUND NEUTRALIZER IS NOW ADAPTIVE PEDESTAL AWARE AND CODE OPTIMIZED FOR MORE SPEED, the Background neutralizer module is used in the preview filter and also in several tools like the Remove Light Pollution Tool and the Background Calibration tool for an initial neutralized view of the sky background. This tool is now Adaptive Pedestal aware which means that the multiplicative sky background correction is done with an additive correction for the adaptive pedestal if present in the metadata of the image to be processed. The effect is that you will get much better colors and correction if an adaptive pedestal is present. Furthermore the code of the module has been optimized, which means it will be roughly 2x as fast now than before.
• FIXED, MULTI-CHANNEL/SESSION, if a channel or session was assigned as "Light" APP would get confused and start processing flats with that assigned as lights. This will no longer occur.
• IMPROVED, STAR COLOR CALIBRATION, a problem in star color analysis has been fixed and the code for the application of the calibration parameters to the data has been optimized, so it will run faster now.
• FIXED, CALIBRATION WITH AN ARTIFICIAL MASTERFLAT, applying an artifical masterflat created with the Vignetting Tool on Bayer CFA data did not work. This is now fixed, reported here: https://www.astropixelprocessor.com/community/main-forum/artificial-flats-wont-load/
• FIXED, PREVIEWFILTER & IMAGEVIEWER PANEL, a bug has been located and solved which caused the image viewer panel not to respond any more. If you would load a 16/32bits image and would set the zoom on on one of the B,W,ST, BA, CON sliders of the preview filter to 4 and then load an 8bits image, the previewfilter and imageviewer panel would stop working. This is now solved.
• FIXED, MULTI-CHANNEL/SESSION, a bug has been solved that caused multi-channel/session processing to fail when you would use filter assignments and session assignments of only 1 letter, like R or H. The bug was reported here https://www.astropixelprocessor.com/community/main-forum/1-076-beta-3-integrate-slider-100-says-0-of-72-frames-will-be-stacked/
• IMPROVED, MULTI-CHANNEL, if filter assignments are used of only 1 letter, like L, R, G, B, H, S, O they are automatically assigned with their full filter name like Luminance, Red, Green, Blue, Hydrogen-alpha, Sulphur II and Oxygen III.
• FIXED, REDRAWING OF MAIN ASTROPIXELPROCESSOR WINDOW, on some computers the drawing of the main application window can sometimes get corrupted. I have implemented additional window/window state/window focus listeners that will force a redraw of the main application window to prevent corruption from happening. The issue was reported here: https://www.astropixelprocessor.com/community/main-forum/very-basic-problem-with-interface/
• IMPROVED, LICENSE ACTIVATION IDENTITY, we were using the amount of cpu threads in the computer for creating a computer identity for license management over 3 different computers. Since some new laptops/notebooks will dynamically alter the amount of cpu threads as an energy management feature (I assume), we have changed the computer identity to include the amount of RAM in GBs of the computer instead. The old computer identity caused APP users with such a laptop/notebook to be forced to keep re-activating their license...

• FIXED, 32bits NORMALIZATION OF BAYER DATA, this was a regression bug that came to life while optimizing code for APP 1.076. Thank you @minusman for reporting the issue directly when encountered in the APP 1.076 beta. The issue was reported here: https://www.astropixelprocessor.com/community/rfcs-request-for-changes/preview-and-saving-calibrated-images-errors/
• FIXED, RAW CONVERSION WHITE BALANCE, the multiplication of White Balance Multipliers was not working correctly when no MasterBias, or MasterDark was supplied. The reason being, that the sensor black level will not be removed for linear deepsky astrophotography processing. This problem has been fixed now by correcting the multiplication of White Balance for the unsubtracted sensor black level. Thank you Daniel @v4mp1 for bringing this to my attention. 3 images are shown, linear image without WhiteBalance (WB), linear image with WB in APP 1.075 (wrong!), linear image with WB in APP 1.076:

• IMPORTANT IMPROVEMENT FOR COLOR DATA, NORMALIZATION CORRECTION FOR ADAPTIVE PEDESTAL, if your color data needs to be processed with an adaptive pedestal due to otherwise possible black clipping of data after calibration, the normalization of data needs to be corrected for the additive application of the adaptive pedestal. The adaptive pedestal is an additive ADU value that is added to the data range and this can upset color ratios if you don't correct for this in data normalization. The whole normalization engine has been updated to correct data normalization for an adaptive pedestal. Please look at this example that clearly shows the effect that this will have. Your color data will now get proper color ratios (R/G, B/G) despite any application of an adaptive pedestal in the calibration engine. First image shows an integration of Nikon NEF frames (courtesy of @wei-hao_wang) where the adaptive pedestal has been purposely de-activated. The data of some of the frames is however partially clipped on the black point after calibration resulting in less optimal data integration. The color ratios though are not affected by a possible artificial data pedestal. The second image is the same data processed in APP 1.075 with the adaptive pedestal enabled, resulting in other and thus bad color ratios as a result of the adaptive pedestal that is needed to properly calibrate the whole dataset. The final/bottom image shows the same data processed in APP 1.076 with the adaptive pedestal enabled with now again correct color ratios due to the correction in the normalization engine for the adaptive pedestal:

• BIG IMPROVEMENT, NORMALIZATION ENGINE, data normalization code has been greatly optimized, resulting in much faster data normalization, especially noticeable on RGB images shot with camera with big sensors! Normalization of big RGB frames  already took several seconds on fast computers, and now it will only take a fraction of a second. This will result in both noticeable faster normalization and integration times.
• BIG IMPROVEMENT, STAR ANALYSIS, extensive work has been done on improving star detection and analysis, due to some users reporting issues with Star Analysis in APP 1,075 (  https://www.astropixelprocessor.com/community/rfcs-request-for-changes/problem-with-star-analysis-1-075/ ).  I am happy to announce that all reported data that gave issues, now works flawlessly. The Star Analysis module is also twice as fast now while becoming much more robust and memory consumption is kept very low as well. Users will notice that APP will detect more stars on more datasets as well. Here are some performance graphs that show Star Analysis of 100 H-alpha frames of 5 different optical setups, for 4 CPU threads, 8 CPU threads and 16 CPU threads, APP 1.075 versus APP 1.076:

• BIG IMPROVEMENT, OPTICAL DISTORTION CORRECTION, due to optimizing several big matrix calculations (non-linear algebra) and making several algebra operations multi-threaded, optical distortion correction calculations are much !!! faster now. You will see speed increases of 10-100x ! This means that all registration tasks with optical distortion correction enabled will go much faster.
• ADDED 2) Calibrate & 6) Integrate, MAXIMUM INTEGRATION METHOD, 2 integration methods have been added in addition to average and median integration. The first is maximum. Use maximum to create a star trails image or to show all outliers/artefacts (with too high ADU values) in the data like hot pixels, satellites, airplanes, cosmic rays etc...
• ADDED 2) Calibrate & 6) Integrate, AUTOMATIC INTEGRATION METHOD, 2 integration methods have been added in addition to average and median integration. The second new method is automatic. With the automatic integration method, APP will automatically set either median or average integration, set suitable integration weights (either equal or quality weights) and it will enable an outlier rejection filter with suitable kappa low and high values, all depending on the number of frames in the integration/stack. With light frames, it will even enable Local Normalization rejection if preferred.

• IMPROVED, OUTLIER REJECTION FILTERS, mad winsor clip has been made redundant and has been replaced by a new outlier rejection filter: adaptive rejection. This is a more advanced outlier rejection filter in which sigma, winsorized and MAD rejection are combined to further improve the outlier detection in the pixel stacks. This filter should work as the best filter if you integrate more than 20 frames.
• IMPROVED/CHANGED, OUTLIER REJECTION FILTERS, you can no longer select the amount of rejection iterations, APP will automatically iterate as much as needed for the given kappa values. In addition, you can now perform outlier rejection on both sides of the pixelstacks. You can reject outliers on the high and on the low side. In almost all cases, you will have very little outliers on the low side, so you can safely set the kappa low to a value of 4 - 8, which means very little outlier rejection will take place on the low side of the pixelstacks. On the other hand, most outliers will be present on the high side (like hot pixels, satellites, airplanes, cosmic rays etc...) so you want to set the kappa high value clearly lower than the kappa low value. A reasonable value for kappa high is between 2.2 - 4. The end result is that more data will now be preserved on the low side and that the real outliers on the high side are more efficiently removed. Finally, the Local Normalization rejection feature of all filters is now enabled if you select local normalization rejection.

• IMPROVED, OUTLIER REJECTION DIFFRACTION PROTECTION, the algorithm for diffraction protection has been improved further so less artefacts will be seen around star borders due to the outlier rejection filters. Artefacts can occur due to different star sizes and shapes in your light frames or due to different diffraction patterns in your light frames.

• IMPROVED, FILE DELETER, if you make a selection of files/frames in the bottom frame list panel, you can delete these files permanently with a right-mouse click menu. The warning has been improved to indicate to the user the consequences of deleting the files, because the files will be permanently removed from the harddrive. So you need to be sure that this is correct. This new warning is much more clear than the previous one:

• FIXED, 4) REGISTER MENU, there was a user interface error at the bottom where you would save the registered frames. This is fixed.
• FIXED, Processing of Bayer CFA data where both the height and width of the sensor is uneven. In some cases this failed in the analyse Flats stage in calibration.
• FIXED, 6) INTEGRATE, there was a minor bug with the integration buffer which would manifest itself when the field of view of the integration/stack was very small. This can happen if you use the 6) Integrate, crop composition mode and would select a very small part of the field of view. This bug is fixed.
• IMPROVED, 6) INTEGRATE, the integration buffers will be larger now depending on the available RAM memory in the computer, which should result in faster integration on most systems. And image loading before the actual integration starts will be more efficient, especially with larger images. Several images (2/4) will be loaded while they are being written to the file mapper. This should also result in faster integration times on most systems, especially with larger images.
• IMPROVED, FITS BAYER CFA, the read speed of Bayer CFA Fits files has been improved greatly ! Image processing on FITS Bayer CFA frames will be much faster now !
• IMPROVED, FRAME LIST PANEL, the frame numbers per frame type (lights, flats etc) are now always ascending no matter how you sort the frames. The first image showed incorrect frame numbering from top to bottom:

• FIXED, 2) CALIBRATE in MULTI-SESSION/FILTER modes, when multiple masterframes were created of a single type, like multiple MasterFlats for different sessions/filters, then due to a concurrency issue, not all masters were directly assigned to the correct light frames when finishing the Create Master Calibration frames task. This is now fixed.
• IMPROVED/FIXED, FRAME LIST PANEL, the column with the file name of the frame that was loaded has been adjusted. If the file name contains the selected work directory, only the file path without the work directory is shown. Furthermore, a bug has been fixed, if you would change the column order by dragging with your mouse, the panel could crash.. this will no longer happen.
• IMPROVED, FRAME LIST PANEL, MULTI-CHANNEL/SESSION, to improve the readability of the frame list panel when processing in multi-channel and/or multi-session modes, all frames per filter are always grouped. Within each filtergroup, the different sessions are grouped. Furthermore, the filter groups will be shown with their own color to further improve readability. Finally, each frame type (lights, bias, dark, BPM etc) will have their own color in the frame list panel.

• FIXED, CALIBRATION MASTERS MULTI-SESSION, in multi-session mode, masters created for specific sessions did not show the correct sessions in the file naming. This is correct now:

• IMPROVED, FRAME QUALITY SELECTION IN MULTI-CHANNEL/FILTER MODE, when selecting a % of frames in 6) INTEGRATE, (lights to stack % ) in the multi-channel mode, the frame % is applied per fitlergroup now. This is a big improvement, because previously, too many frames would be removed of the filter with the least quality instead of removing the worst frames per filter. The selection is not done per session, because you simply want the best frames per filter normally. In the following screenshot, you see a frame % applied of 75% to a selection of Red, Green, Blue and H-alpha frames over 5 different sessions, notice that 75% is set for integration per fitler now. You can see this from the INTEGRATE mark in the quality score column:

• IMPROVED, STAR ANALYSIS QUALITY, the quality calculation for frames after Star Analysis is now much more robust, the formula is now: numberOfStars * median star size * median star roundness. The star size and roundness are based on the median star profile of all stars analysed. Previously, the calculation was much more subject to outliers, because the calculation was a summation of star size and roundness per individual analysed star.
• IMPROVED, TIFF & JPEG Bayer CFA, processing speed has been greatly improved for TIFF and JPEG frames that still need to be debayered.
• FIXED, OPENGL 3 IMAGE VIEWER, in some cases, the OpenGL 3 implementation did not work due to APP trying to use an unsupported Shading Languag version. This should now be fixed. APP will fall back to Shading Language version 1.3 if needed.