[Closed] Astro Pixel Processor 1.083 beta 1-4 detailed release notes

Astro Pixel Processor 1.083 stable

• IMPROVED, STAR REDUCER TOOL, further improvements are made for the background reconstruction of the reduced stars. And a Null Pointer Exception has been fixed which would occur if you pull the sliders while a calculation is in progress.
• FIXED, 4) REGISTER TOOLTIP for distortion margin now shows the correct default margin of 0,005
• FIXED, TOOLTIPS FOR DISABLING 3) ANALYSE STARS, 4) REGISTER & 5) NORMALIZE

• FIXED, MACOS KEYSTROKES, macOS users can now use the Command key to do copy/paste/cut/select all in the whole application. For instance, in the File Chooser, you can use Command-A to select all files. In the console window, you can copy the console text and output it to a txt file. And, you can use copy/paste to enter your license details. Previously, these keystrokes were only possible with the multi-platform CTRL key. Now it also works with the MacOS command key.
• FIXED, FITSREADER, SUPPORT FOR FITS KEYWORD ROWORDER, we can now correctly read and apply the bayer patterns of images that have the ROWORDER keyword combined with the BAYERPAT, COLORTYP keywords.
• IMPROVED, ADAPTIVE PEDESTAL, the console will now show how many pixels clip to black leading to use of the artificial pedestal. And if despite the pedestal still pixels clip to black, the warning will also show it:

Astro Pixel Processor 1.083 beta4

• 1.083 beta4.2 for MacOS has fixed loading of LibRaw, also on the new M1 computers.
• IMPROVED/FIXED, STAR REDUCER TOOL, we have fixed the negativeArraySizeException that occured on some datasets due to border artefacts from stacking. The tool is again improved a bit for star background reconstruction and to prevent certain artefacts from appearing. And it is again a bit faster.

• IMPROVED, HSL SELECTIVE COLOR TOOL, we have added much more data range presets so it will be much easier to select background including or excluding the noise floor. And the data range labels now show 1 more decimal, 5 instead of 4, because this is needed on most datasets.

• FIXED, BAD PIXEL MAP, will show correctly again
• FIXED, ANALYTICAL GRAPH, will show correctly again.

Astro Pixel Processor 1.083 beta3

• IMPROVED (a lot!), STAR REDUCER TOOL, we have fixed all the issues of the tool in the beta2 release. The huge memory consumption is gone and no more black areas in the result. Furthermore, the star reduction and background reconstruction has been improved a lot, you will see very little artefacts on most images. The tool is also much faster now 🙂 ! One slider has been removed which was the area correction slider, it was made redundant. We added one slider to better control possible Halo's on bright stars. So the interface has changed quite a bit and it has a proper tooltip now. This new version is producing great results on a wide variety of data. Images with very dense star populations are very difficult to star reduce or star remove, but the tool is handling it very well now we think. Images with dense star fields were very problematic in the beta2 release and we greatly appreciate all the problem data that we received to be able to address all the reported issues. Thanks a lot!

• IMPROVED, HSL Selective Color Tool, the tool has been improved greatly. Previously, the tool had issues with objects with a high dynamic range in the center of the field of view. This has been solved. Furthermore, the sliders work more directly. Previously, if you would set of the sliders to 0,5 you would hardly see a change in the result. Now you will see changes much more quickly making the tool easier to work with. We have removed the luminance slider because it had no use really and would only lead to odd results. Finally, the user interface has been improved with buttons to see the color selection: "show color selection" so you can see the pixels that will be affected clearly before you make changes. This means that you can actually see the color channel mask for the selective color to be applied. There is a button to see the original image and there is a button to see the actual image on which you are working on the tool at that moment. Finally, the tool now has a proper tooltip.

• IMPROVED, 2)-6) MANY CODE OPTIMIZATIONS, due to optimizations in the modules 2)-6) overall speed increase has been accomplished of about 15-30% depending on the data and the computer which is being used. 3)Analyse stars and 5)Normalize are a lot faster now. 4) Is faster as well, but was already fast. 6)Integrate speed has been clearly improved as well. This shows CPU and memory usage for processing of a typical dataset using 31 CPU threads and 16GB of RAM:

• IMPROVED, MOSAIC REGISTRATION, the code to register all frames into the mosaic has been optimized leading to quite a bit faster mosaic registration.
• ADDED CAMERA SUPPORT USING LATEST LIBRAW 2021-10 snapshot: Canon EOS 1DX Mark III, R5, R6, M50, M2, 850D,M200. Nikon D780, Z50, Z5, Z6 II, Z7 II, Z fc. Sony DSC-HX99, DSC-RX0M2, DSC-RX100M7, ILCE-7RM4, ILCE-6600, ILCE-6100, ILCE-9M2, ZV-1, ILCE-7C, ILCE-7SM3, ILCE-1, ILCE-7RM3A, ILCE-7RM4A, ZV-E10. Panasonic DC-TZ91, DC-Z96, DC-GX880, DC-S1H, DC-S5, DC-GX7M3, DC-G90, DMC-FZ100, DMS-ZS60, DC-G110, DC-G100, DMC-GF2, DC-FZ10002, DC-TZ95, DC-GH5M2. Hasselblad H4D-50, L1D-20C. Olympus E-M1 Mark III, E-M10 Mark IV, E-PL10. Leica X2, SL (TYP 601), M9, Q (TYP 116), M (TYP 240), CL, M8, Q2, M10, DIGILUX3, D-LUX 7. Fujifilm GFX 100S, X-E4, X-S10, GFX 50S II. Pentax MX-1, K-3 Mark III.
• IMPROVED, BATCH SAVE TOOL, we have added the option that you can save a batch of images with the preview filter stretch.
• FIXED, FRAME LIST PANEL, you can now again scroll through the images with the up/down arrow keys while moving with your mouse in the image.
• FIXED, PREVIEW FILTER, we have fixed the white-out stretch issue as reported here: https://www.astropixelprocessor.com/community/main-forum/remove-lp-white-image/ and the stretch DPP preset, show data-as-is should now always work. Sometimes it did not work properly.
• IMPROVED, FILE SAVE DIRECTORIES, in 2)CALIBRATE, 4) REGISTER, save, 5)NORMALIZE,save and 6)INTEGRATE you can now set the save directory with a Set Save Directory button. If you don't use it, the results are saved in the Working Directory by default.
• IMPROVED, NEW FILE SAVER MODULE, by default, all saving is offered in 32bits FLOATS FITS, this is needed and recommend on all data that is still linear. If you save stretched data, you are offered 16bits TIFF. Of course, you can still save however you would like to.
• IMPROVED, RGB COMBINE TOOL, better file name for saving which is now offered as the object naam + the composite formula that is chosen.
• FIXED, 5)NORMALIZE, we fixed the issue which could produce a complete black integration. This happened on multi-channel problem data where Background Neutralization did not work. This issue also fixes possible 0 noise values in analysis leading to infinit Signal to Noise Ratios.
• FIXED & OPTIMIZED, 2)CALIBRATE, the implementation of the adaptive pedestal is now faster and the calibration engine has a new warning system against data that has issues with sensor offset, ligh leakage, incompatible calibration data.
• FIXED, USER INTERFACE, scrolling through the file list with the image viewer set to l-calibrated now works smoothly, user interface does not flickr or get scrambled.
• FIXED, 3) ANALYSE STARS, the issues of star analysis in beta2 were fixed by rolling back to star analysis of 1.082 stable.
• FIXED/IMPROVED, 2)Calibrate adaptive pedestal, if you have done 5)Normalize and then enable/disable the adaptive pedestal, you will need to redo 5)Normalize. You will now be warned and asked if you want to enable/disable the adaptive pedestal and thus redo 5)Normalize.
• FIXED, 4)REGISTER, deselected frames will no longer be registered.
• FIXED, 4)REGISTER, APP will no longer automatically set a new reference frame if not all frames could be registered. APP will now first ask the user if you want to continue trying with a new reference frame. It will report how many frames were registered and what the new reference frame will be.
• IMPROVED, Comet registration, the code to detect the comet nucleus has been improved much from the beta2 code. It has also been optimized in terms of memory usage so it works better and more efficiently now. Furthermore, you can now skip an image if visual comet detection seems to fail, in that case the detection is done based on the detection in the other frames combined with calculating the speed of the comet relative to the stars if the time shot metadata is being used. By default you will need to visually select the comet in the final frame to make all calculations more robust and reliable. In the 1.083-beta2 version, the comet detection would sometimes lead to an endless loop which will no longer occur as well so that has been solved. Once the comet has been detected properly in 3 frames, the comet speed is calculated which leads to better detection and prediction of the comet in the following frames as well.

Astro Pixel Processor 1.083 beta 2 - will be released before June 1st 2021 !

• NEW STAR REDUCER TOOL, APP will now have a tool that will help you to quickly reduce or remove the bright stars in your images. The purpose of this is to have other features in your images to stand out more, like a bright and beautiful nebula or a galaxy. By reducing the stars, the nebula details will become more clear. The tool enables you to reduce the star size, manipulate the overall star brightness and you will be able to switch quickly between the reduced version and the original of your data. To be clear, this tool is not based on, nor using Starnet++. It's 100% our code and implementation. Some examples:

• IMPROVED COMET REGISTRATION/ALIGNMENT, the comet registration has been improved over the module from beta1. In 1.083beta1, the alignment could suffer on comet nuclei that were saturated so it was a bit difficult to find a sub-pixel comet position. In beta2 we have fixed this by considering the whole intensity profile of the comet nucleus and comparing this to other frames. A comet is an extended object, not a point-source like a star, so fitting to a Gauss profile or using Center Of Mass method will not work to get subpixel precision, so we found another way to still do this we think. In addition, we are now using the time shot metadata to further refine the comet movement between frames, so the final registration parameters will be very precise, like 0.1-0.3 pixel precision should be expected for alignment of the comet nucleus, which in turn should make the fainter details of the comet (like in the tail if present) be more sharp in the final integration. Finally, the new method of alignment for the comet, could be used to align images with very little stars as well, so you can use it as a one-star alignment if needed.

• NEW FILE SAVER MODULE INCLUDING PNG WRITE SUPPORT, DPI AND ADDITIONAL ICC COLOR PROFILES, file saving in APP has been completely upgraded. This upgrade is applied in the whole application where frames are saved and it should solve many requests regarding file saving in the application. It is used when masters are created, your light frame integration/stack, when you save with the preview filter and in all the batch tools. Especially in the batch tools, the file saver is new much more user friendly. You only need to set the save settings once for whatever file type that you choose and the batch will be saved in the same way, without ever asking file save settings again. The file saver now is just one panel that allows you to set all file save options which are, directory of choice (no more saving in only the Working Directory), file name of choice, file type (FITS/TIFF/PNG/JPG), bit depth (depending on the file type, 8/16/32 bits), ICC profile for TIFF/PNG/JPG, JPEG quality. The new file saver also includes the option to set Dots Per Inch (DPI, for printing purposes) for TIFF/PNG/JPG frames. Finally, the file saver includes many more ICC color profiles (no profile for linear data, sRGB v2, v4, ColorMatch RGB, Adobe RGB 1998, Apple RGB, Wide-gamut RGB, ProPhoto RGB) to use which can be needed for printing purposes, or for moving the non-linear data to other applications like Photoshop. Be aware that if your data is still linear, simply save your data without an ICC color profile. So you can also choose to not include an ICC color profile for TIFF/PNG/JPG when the data is still linear. (if you do save with an ICC profile on dark linear data, you will get issues with your data! ICC color profiles like sRGB, Adobe1988, ProPhoto etc... are not to be applied on linear data at all, they are for the non-linear color spaces).

• FIXED, USER INTERFACE GLITCHES ON WINDOWS SYSTEMS, as reported in several forum topics, some Windows users, especially with newer hardware, have reported crashes/glitches of the APP Graphical User Interface. One of these topics is: https://www.astropixelprocessor.com/community/main-forum/very-basic-problem-with-interface/ . We have fixed the problem by including the fix in the application installer to disable both Direct3D & OpenGL for the native Java JDK components. The solution was proposed by @bcolyn and it does seem to solve the issue completely we are happy to say.
• ADDED READ/WRITE SUPPORT FOR PNG FILES, you can now load and save PNG files including saving with DPI (Dots per Inch) and choice of ICC color profiles.
• FIXED, FITS READER, APP's fits reader will now read FITS frames that hold multiple FITS HDUs where only the first HDU is the actual image data. This applies to professional astronomy data like from the Hubble Archive...
• WORK-AROUND IN FITS READER, FOR APPLICATIONS THAT USE A FAULTY FITS WRITER: some users reported problems with some FITS files where APP could not read these, but other applications could. The topic is : https://www.astropixelprocessor.com/community/camerasupport/invalid-fits-header-on-zwo-asi-178mc/ Some of the problem FITS files come from the Voyager capture software developed by Leonardo Orazi. Together with Leonardo, we have found the root cause of the problem. It turned out that the FITS writer code that is used in Voyager is a badly ported version of the nasa java fits library that we use in Astro Pixel Processor. The ported version is not following the FITS specification for how the FITS Header bytes should be filled with byte values once all metadata has been stored. So the root cause is not in APP, the root cause of the problem is in the capture software or the ascom driver that such a capture package uses. Leonardo has fixed this issue in Voyager and in APP, I have implemented a work-around so that this particular FITS issue is detected and the files are still read properly.
• IMPROVED, WORKFLOW in 3) ANALYSE STARS, 4) REGISTER, 5) NORMALIZE : you can now directly skip tabs 3), 4) en 5). This is usefull in the following cases:
  • Comet registration, by skipping 3) star analysis, you can directly perform comet registration using only the comet nucleus. You can also perform 1 star registration in this way, if you have data with very little stars. If you do have stars in your data, we recommend to perform star analysis, since APP will then perform comet registration after star registration, and this should and will give you better results in terms of precision.
  • Proceeding with already registered frames, skip 3) and 4) and proceed directly with normalization.
  • Proceeding with already registered and normalized frames, skip 3), 4) and 5) and proceed directly with integration.
  • Proceeding with alread normalized frames, perform star analysis and registration and then skip normalization to directly go to integration.

• FIXED, CALIBRATED PROJECTIVE REGISTRATION, the calibrated projective registration mode did not work in 1.083 beta 1. This has been fixed. The bug caused a null pointer exception when registration was finished because the camera matrices were not produced which was caused by APP failing to ask the focal length and pixel size of the reference frame.
• IMPROVED, ANALYSE STARS, the star detection algorithm has been improved once again because we received some data where it failed. This concerned data of long narrowband exposures where a lot of cosmic rays were present in high numbers... which confused APP. It thought that the cosmic rays were stars... We have made sure that the new algorithm works properly on all of our test data. In addition some optimizations were applied as well, so module 3) Analyse stars is now 10-20% faster 🙂 Please check this example were APP 1.082 failed and 1.083 works, a 30 minute H-alpha exposure with many cosmic rays.

• BATCH MODIFY TOOL HAS BEEN REPLACED BY 4 SEPARATE BATCH TOOLS, we have replaced it finally because it was a multi-function complicated user unfriendly tool. The different functionalities have now been made possible with the introduction of 4 new Batch tools: Batch Crop, Batch modify metadata/CFA, Batch add/multiply, Batch save which will be introduced below:
• NEW BATCH MODIFY FITS METADATA/CFA TOOL, the Batch Modify FITS METADATA/CFA tool allows you to save copies of your frames with corrected metadata for
  • iso/gain
  • exposure
  • instrument (your camera name)
  • filter used
  • capture session
  • add Bayer/X-Trans CFA tag
  • remove Bayer/X-Trans CFA tag / undemosaic already demosaiced data

The following screenshots show an example where you can add the correct CFA tag for data that is missing the tag because the capture software did not include it...

•  NEW BATCH CROP TOOL, the Batch Crop tool enables you to make a crop of your data. You can draw the crop/rectangle and/or use crop offsets to finetune the desired crop. The crop dimensions and aspect ratio are shown as well.

•  NEW BATCH ADD/MULTIPLY TOOL, the Batch Add/Multiply Tool enables you to add/subtract and multiply/divide the data in your images with a certain offset or factor.

• NEW BATCH SAVE TOOL, the batch tool enables you to save a batch of files to a different filetype and possibly change the bitdepth of the data as well.

• IMPROVED BATCH ROTATE/RESIZE TOOL, we have updated the user interface of the tool and it now provides a quick preview when you touch the rotate, resize sliders or touch the flip selectboxes.

• IMPROVED BATCH TOOL FILE SELECTOR, you can choose whether to show only the file name or the complete file path of the files to be loaded into the batch tools. This will also apply if you want de delete multiple frames from the file list with the right mouse-click popup menu.

• FIXED, ADVANCED NORMALIZATION, a regression bug has been fixed that caused advanced normalization not to start in some cases.
• FIXED, BAYER DRIZZLE WITH THE HA-OIII debayer algorithms, enabling bayer drizzle with the Ha-OIII extract Ha or extract OIII algorithms did not work correctly. It showed the exact same result which was similar to the Ha-OIII mono debayer algorithm. It now works correctly as is show in the next screenshots:

Bayer Drizzle with Ha-OIII mono (so mix of the Ha and OIII signals):

Bayer Drizzle with Ha-OIII extract Ha (only Ha):

Bayer Drizzle with Ha-OIII extract OIII (only OIII):

A HOO composite of these Ha and O3 signals made with the RGB Combine and Selective Color tools (data courtesy of Ian Barredo @tracer, asi533 Optolong L-Extreme filter):

• FIXED, SAVING CALIBRATED X-TRANS FILES IN 2) CALIBRATE, as mentioned in this topic: https://www.astropixelprocessor.com/community/main-forum/problem-saving-loading-calibrated-lights-from-fuji-x-trans-sensor/ , there was a problem with the X-trans CFA pattern when the calibrated X-Trans files were saved as fits files in the calibration engine. The problem has now been fixed.
• FIXED/IMPROVED, CALIBRATION MASTERS, by default, APP will now create 32bits Calibration Masters. On the forum a problem was reported with Flat-Field calibration causing rings in the light frame integration. It turned out the solution was to create 32bits calibration masters instead of 16bits calibration masters. With 16bits, severe posterization occured in the masters leading to bad data calibration. The issue is reported here: https://www.astropixelprocessor.com/community/main-forum/integration-causing-nasty-rings/

Astro Pixel Processor 1.083 beta 1 - 30 July 2020:

ADDED COMET REGISTRATION: Astro Pixel Processor 1.083 stable will have full comet procession support, this first beta release introduces registration/alignment on the comet nucleus.

To perform comet registration, go to 4) Register, and set the registration mode to...: comet  🙂

You will need to select the comet in the first image, then on the other light frames, APP will try to auto-detect the comet nucleus. If it fails the auto-detection, due to

• a big change of the comet brightness or
• a big move of the comet between 2 successive image

then you will simply be asked to select the comet again in that image.

I have tested it on a wide range of datasets of different comets as well, and so far all is working nicely with very precise results. Thank you all for uploading all your comet datasets, we received many datasets which really helped in getting this working robustly on a wide variety of datasets.

Here are the download links for the 1st beta release:

• Windows 1.083-beta1
• MacOS 1.083-beta1
  
• Linux DEB 1.083-beta1 
• Linux RPM 1.083-beta1

 We have created a special subforum for all your questions and feedback related to comet processing in APP and in general:

https://www.astropixelprocessor.com/community/comet-registration-and-integration/

Please share your results, questions and any problems that you may encounter.

Thanks,

Mabula