[Sticky] Astro Pixel Processor 2.0.0-beta20 release notes

Astro Pixel Processor 2.0.0-beta20 release notes

This is a big beta release with some major work again on different parts of the application, notably:

• All settings in menu's 0) RAW/FITS to 6) INTEGRATE are now saved and thus restored after an Astro Pixel Processor restart. Thank you Vincent for major work on this improvement!

  In the next beta or the actual 2.0 stable version, we finally intend to release the long expected feature to save/load complete projects including all these settings.

• Improved comet registration

  Several issues have been fixed and we have improved comet registration which makes the process easier and faster on all comet data. You will have 3 options now to process your comet data, which will ensure that you can successfully process your comet data

  1. analyse stars and use timeshot metadata, this is the fastest and most accurate way for most comet datasets.
  2. analyse stars and do not use timeshot metadata this method can be used if method 1 does not work somehow, but it will be a bit less accurate.
  3. skip star analysis and timeshot metadata will also not be used then this method is the way to go if you can not register on the stars for some reason.

  Depending on the comet dataset, you can try each method and see what works best, but for most datasets, method 1 is the preferred way. It is not yet possible to create an integration with both sharp stars and sharp comet out of the box, but this will be the next work now on the comet processing part of APP. 

• Improved concurrency and memory management

  As you can see in the next image, 2.0.0-beta20 will overall use much less memory when compared to beta19 while maintaining performance and it will keep releasing memory back to the Operating System when possible during processing. The graphs illustrate processing of about 150 light frames for 6 filters from data calibration up to and including light frame integration.

Please see below for more details on all the changes and improvements in this release.

• Current DEVELOPMENT PLATFORM

  Astro Pixel Processor now runs on  GraalVM CE 22.3 based on JDK 19. Both memory management and concurrency is improved in this version using new API features of this development platform as detailed in the next 2 notes for this release.

• IMPROVED CONCURRENCY/MULTI-THREADING

  Up until now, we were using Operating System threads for multi-threading. Using a new API from Java, we can now use so called Virtual Threads for this. The main upside is that APP will use less memory resources because creating new threads is now incredibly cheap. In cases where we need and use many threads, like in the registration engine, we see a performance gain, especially for calculation of big mosaics. Calculating a big mosaic should now use 90+% of your cpu power all the time until finished and it should not slow nor stall like sometimes happened in previous APP versions when many 2-view calculations needed to be done.

• IMPROVED MEMORY MANAGEMENT

  We have adjusted the internal memory management to make sure that the application releases memory back to the Operating System much faster. We received several reports that APP would keep consuming more and more memory, not releasing it in due time until the application would crash due to being Out Of Memory. This adjustment has fixed this problem. Simply said, at selected moments, like after each processing step of 2) Calibrate to 6) Integrate, APP will now release all memory possible. It will also do this after each Tool and after each calculation in a Tool. And APP will check if it is possible to release memory every 3 seconds now. Soon we will upgrade the development platform again where graalVM will be based on JDK 20 or JDK21 and we expect more improvements for memory management and the performance going forward.

• IMPROVED settings of menu's 0) RAW/FITS,  1) LOAD, 2) CALIBRATE, 3) ANALYSE STARS, 4) REGISTER, 5) NORMALIZE, 6) INTEGRATE

  All settings in menus 0) to 6) are now saved during each Astro Pixel Processor session, so the next application restart will restore your used settings. You can restore the defaults with a new option in the CFG menu:

• IMPROVED COMET REGISTRATION

  Several issues have been fixed and we have improved comet registration which makes the process easier and faster on all comet data. You will have 3 options now to process your comet data, which will ensure that you can successfully process your comet data 1) analyse stars and use timeshot metadata , 2) analyse stars and do not use timeshot metadata , 3) skip star analysis and timeshot metadata will also not be used then. Additionally:

• • if you use timeshot metadata, you only need to select the comet centroid in 2 images, 1 image at the start of the imaging session and 1 image at the end of the session. The comet speed relative to the stars is then calculated and the comet is then aligned automatically in the remaining images.
  • the comet registration is now much faster on big and saturated comets
  • several nullpointer exceptions could occur, these have been found and fixed.
  • we have added tooltips.
  • comet registration now works on low resolution screens like 1280x800. We have changed the layout of the comet registration settings as you can see in the following screenshot, you can also show/hide the preview filter panel for automatic stretching:

• • you can click the space bar or the enter key to confirm the comet centroid for easier and thus faster manual comet centroid confirmation
  • a problem with the openGL image viewer in comet registration mode is fixed which caused weird behaviour of the image viewer
  • the registration process console is automatically hidden now to show the comet registration as good as possible
  • we removed the button for visual check of the last image, because you need to select the comet always at the end of the imaging session for a good calculation of the comet speed relative to the stars.
  • the comet centroid can be selected manually now which is usefull for weak comets where the automatic comet centroid detection is failling. During comet registration you can enable/disable automatic pinpointing of the nucleus with the new auto pinpoint selectbox:

• • the actual automatic comet nucleus detection has been improved significantly
  • sorting of the light frame data on time shot before the comet registration starts was no longer working, this is now fixed
  • several users shared data of weak comets where APP had trouble with the comet registration, all these datasets work well now

• IMPROVED FITS SAVER

  The new fits keyword ROWORDER is now added to all saved FITS images with value TOP-DOWN.

• FIXED Sorting on star FWHM metric

  Best to worst was actually showing worst to best, so this is now fixed.

• IMPROVED Sorting on Registration RMS

  This is now shown more logical in the analytical plots by having adjusted the normalized registration RMS value for the plots. A registration RMS error of 0,10 pixel will now show much higher, which is logical since you can not register much better on monochrome data. And registration values larger than 1 pixel are shown much lower which is also logical because this causes visible problems in the integrations. The following 2 screenshots show the difference on typical registration RMS values on monochrome data:

• IMPROVED assignment of calibration masters to lights

  If you select or deselect a master calibration frame, APP will now directly perform re-assignment of the calibration masters to the lights. This is indicated as follows: if you deselect a Bad Pixel Map for instance, the BPM mark in the frame column for the lights is now directly removed.

• FIXED Super Pixel detection of data clipping on black value for all image types with OSC/Color data

  If you would use Super Pixel processing from the demosaic algorithms in 0) RAW/FITS, there was a bug where APP would not work correctly with detecting data clipping on the black/0 value. It could warn that many pixels were clipping to 0 where in fact they weren't. This is fixed now.

• FIXED processing of FITS images with ROWORDER = BottomUp

  Sometimes an error was thrown when normalization or integration would start because image dimensions were not correctly put in the image metadata internally in APP. Also fixed now.

• FIXED reading of 32bits float images which should have 1.0 as the white point value

  Some applications like Siril will produce results where max values exist of 1.000009847 for instance instead of having it clipped internally correctly to 1.0. This could cause APP to think that these images were not normalized in a way where white is 1.0 and thus APP would not handle these files well. This is fixed.

• FIXED tooltip appearance of outlier rejection

  settings in 2) CALIBRATE and 6) INTEGRATE.

• FIXED normalization of pixels that are actually under the black point value after normalization

  Could happen on data with short exposure times. These pixels are now correctly clipped to 0.