[Closed] Astro Pixel Processor 1.061

Video Tutorial showing the new Multi-Channel processing feature including explanation of the new calibration engine:

Astro Pixel Processor 1.061

This release is a quick and single bug fix for 1.060. On Linux it was reported that GUI drawing we terribly slow. The Xorg process responsible for drawing in Linux required all (or almost all) CPU resources. This happened due to an adjustment in GUI rendering code. This problem is fixed and applied for all platforms, since the problem relied on the graphics driver's 2D capabilities and the current implementation and thus fix doesn't rely on this.

Astro Pixel Processor 1.060

This release is a major upgrade:

1. the data calibration engine has been rebuilt from scratch and now includes robust 3-factor dark frame scaling.
2. Full Multi-Channel/Filter & Multi-Session, which means, processing of multi-channel and multi-session data can now be done on auto-pilot after all the frames are loaded and assigned to the right channels and/or sessions.
   
3. Speed increasements in star analyis, integration tasks and the creation of the Master calibration frames including Bad Pixel Maps.
   
4. Added Load buttons for darkflats and all masters. And also a button to load frames that doens't need calibration - integration at the moment, we call them Other/Processed frames.
5. Lots of reported Bug Fixes

Release notes applicable to the major upgrade of the calibration engine:

• Improvement/FIXED, flat frame normalization, in creating a master flat, all flats need to be normalized so the histograms of all flats match as good as possible. This means that the flats need to be corrected for both location and dispersion of the pixel value distributions. The actual calculation has been adjusted so it will be much more robust now. In APP 1.059, the flat normalization calculation was not robust, causing visible over correction of the data's vignettting profile clearly visible in the next 2 screen shots, data is courtesy of @ALEXx360, left image shows overcorrection in APP 1.059, right image shows correct processing in APP 1.060:

Another example on data courtesy of @wei-hao_wang, problem was reported by Kirk @1CM69

At this stage I took a screenshot of the Linear data & also the Calibrated data:

-- attachment is not available -- -- attachment is not available --

 

You can easily see the vignette in the Linear data but this should disappear in the Calibrated data but it doesn't, it just changes colour.

Wei-Hao's data was processed in APP 1.060 and the integration turns out like this now, no problem anymore with flat-field over-correction.

To make sure that APP's flat-field correction is now working properly on all datasets, it was tested on a lot of different datasets, monochrome and color, and the results were then verified/compared with the output given by other applications.

• Added, MasterDark scaling, APP now has the ability to scale your masterdarks so the results are minimised for noise. This corrects the MasterDarks for deviations in both exposure time and temperature between the frames to be calibrated. Only requirement to be able to let APP do this, is that you need to supply bias frames or a previously created MasterBias as well.
• Improved, Calibrate, removed the strict calibration rules for exposure time of darks and ISO/gain value of darks & bias frames for calibration of flats and lights.
• Improved, Calibrate, the calibration enigne now includes a warning system that will warn users if they follow a possible suboptimal calibration path. The warning system can be turned off as well.
• Improved, Calibrate, APP will now automatically match master calibration frames to lights based on ISO/gain and exposure values. If an automatic match is not possible and if there are several master frames to choose from, like 3 different masterflats, APP will ask which master needs to be matched with which light. If there is only 1 master, it will be automatically matched with frames that need to be calibrated.
• Improved/FIxed, Calibrate, in APP 1.059, if you calibrated lights or flats with both a masterbias and masterdark, the results weren't optimal due to clipping on the black point between bias and dark calibration. In APP 1.060, this will no longer happen. The totally rewritten calibration engine, will calibrate all frames with all masters and will not clip in between, only after having applied the last masterframe, the black clipping will be done. Furthermore, the calibration is no longer done per calibration type, like bias or dark calibration. Depending on the available masters, the calibration is performed in 1 module which increases speed and reduces memory consumption. All calibration tasks are implemented using multi-core enabled code.
• Improved, Calibrate, the assignment of masterframes to light frames is now 100% transparant. You can verify per light frame which master is matched to it.

• Improved, Bad Pixel Map, the data interpolation for the bad pixels and columns has been improved. Some sensors can have a cluster of bad pixels causing some bad pixels to be surrounded completely by bad pixels. Previously, these bad pixels weren't corrected then, now they are corrected. Data interpolation is done bilinearly using the non-bad pixels nearest to the bad pixel to be corrected.
• Upgrade, Bad Pixel Map, hot column detection has been changed into bad column detection. APP will now also detect cold sensor columns. A new Bad Column Kappa value is introduced to help the user in how strong the bad column detection is needed. This value is also reported in the metadata of the BPM.

Data shown is M1, the Crab Nebula, courtesy of Jean-Brice Gayet (@jb-gayet):

Remaining release notes:

• FIXED, Star Color Calibration & Background Calibration, when these tools are started with monochrome images, then the tools give a proper warning now that these tools require color (RGB) data.

• ADJUSTED, File Chooser window size, the size of all the file chooser windows in APP will now be dependent on the size of the image viewer with a minimum for smaller screen resolutions. Minimum size is set at 800x400 pixels. Otherwise, its 0.75x image viewer width and 0,75x image viewer height. Examples of desktop resolutions 1366x768 and 2560x1440 :
  

  

  

• Added, Preview Filter, added another DDP preset called : no stretch (data as is) , simply select this to view the loaded image data in the image viewer "as is", unaffected by the previewfilter. And another which represents and extreme preview stretch on the data: 30% BG,2 sigma, 0,0% base, usefull for an extreme stretch while working in the Remove Light Pollution Tool for example.

• Improved, GUI 6) INTEGRATE, adjusted the method to create the crop composition mode. Just set mode to crop and APP will ask you to draw the crop of the reference for the Field Of View of the integration. See these images:

• Improved, GUI Progress Monitors, APP 1.059 and lower were using progress monitors as part of Oracle's Java JDK 1.8. These progress monitors caused issues on Linux distributions by claiming focus on the cancel button of the progress monitor whenever a new process started. If the user was working in another application this could cause the user to accidentally cancel the current task  in APP. So to solve this, I have written my own progress monitor to have full control over this. Now some progress monitors still will show a pop-up window (while saving an image for instance) but focus isn't claimed. Other tasks, like the main processes of 2)-6) will show progress monitors fully integrated into the image viewer and will show more information while the task progresses.

Example of new progress monitor for 3) Analyse Stars & 6) Integrate & saving an image :

• FIXED, STAR ANALYSIS, if you loaded a frame without any star, like a flat, APP would show an error instead of properly catching the problem. This is now solved. Problem was reported by Rowland F.Archer Jr. @rowland-f-archer-jr in topic https://www.astropixelprocessor.com/community/rfcs-request-for-changes/blank-subframe-caused-java-null-pointer-exception-in-analyse-stars/

• FIXED, NIKON D800, D800E NEF overscan support,  the RAW image dimensions of Nikon DSLRs hacked by Nikon Hacker firmware's can be different. This is due to loading NEF overscan data with the hacked firmware. This is handled correctly for these models, thanks to data from Wei-Hao Wang @wei-hao_wang.
• IMPROVED, clear Button 1)LOAD, if you click on the clear button in 1) LOAD, you will now be warned first before all memory is cleared. The clear button will release all memory and internal  data processing details.
• FIXED, FITS image loader, fitswork is storing metadata of ISO values not according to fits specification, it's stored as a string comment which is not good:

COMMENT ISO 200

It should be stored at least as a numerical value like ISO: 200     // ISO value of capture. I have caught this exception now in the fits metadata interpreter, so APP will now find the correct ISO value here. Thanks to Raik Erdmann @meinerde for reporting this problem.

• FIXED, FITS image loader, another fix for FitsWork data, this time a data clipping problem. An APP user subtracted a bias master from a frame in FitsWork. FitsWork forgot to clip the data on the zero black point in this case. APP will now automatically correct all FITS data that isn't stored according to FITS specification. The FITS specification states that 8bits, 16bits and 32bits integer data needs to be stored in a certain way using BZERO and BSCALE keywords from the FITS specification. After application of these keywords on the raw fits data, the converted data values MUST be in the data range of either 8bits,16bit or 32bits integers. In FitsWork the data is stored without verification of this. So APP will compensate for this. This is an example of what happened and what the fix does. A moon shot that was bias subtracted.  A lot of data values in the FitsWork fits frame were below 0... giving the noise that you see in the first image. After internal correction with a correct clipping to zero for these negative values, the image looks fine 😉 Thanks to Raik Erdmann @meinerde for reporting this problem. This fix applies to all supported FITS bitdepths, (8,16,32 bits integers and 32,64 bits float/double ).

• FIXED, 5)Normalize, the console output showed lokation instead of location, it now shows location.
• FIXED, 6) Integrate, APP 1.059 showed an error when you tried to integrate with weights: star density. This is now fixed.
• SPEED INCREASE, 3) Analyse Stars, APP is very robust in detecting stars in very high quality data with a lot of stars and/or nebulosity. The implementation to make this possible requires applying a 32bits/64bits median filter with a large radius. Implementing a faster and still exact median filter on float/double data is difficult, but the median filter now has been optimized and is roughly 10x faster. The filter was already implemented using multiple cores. On machines with 16 or less cpu threads, this improvement will speed up star analyis significantly.
• SPEED INCREASE, 3) Analyse Stars, APP by defaults removed detected stars from the detected list if they had very close neighbours. This is done to prevent worse registration precision in high quality data with dense star fields. If less than 1000 stars are detected, this removal is not done anymore, since it will not harm registration precision. This change has a slight positive influence on the speed of analysing stars.

BOTH improvements combined in Analyse Stars will yield a significant speed increase. I have run 3 test to illustrate this. I have analyzed 20 monochrome 10MegaPixel frames with default settings in 3) Analyse Stars with 4 CPU threads enabled of my 1950x Threadripper CPU.

1. APP 1.059
2. APP 1.060 without close neighbour removal
3. APP 1.060 without close neighbour removal and the optimized 32/64 bits median filter to create a robust starmap.

The following image shows the memory consumption and time it took to analyse the stars in the 3 tests:

We can see that in APP 1.060, less memory is consumed and the total time to analyse the stars with default settings is almost 3x faster. The tests were done with only 4 CPU threads and 4GB memory allocated to APP.

• Improvement, Star Analyis, APP will now be able to properly register out-of-focus data. Out-of-focus data was problematic for APP because the default star analysis module assumes the data is well in focus with star intensity profiles similar to an Airy Disc. The intensity profiles are even used to discriminate between stars and other features in the data like hot pixels or very small faint galaxies for instance. To be able to now properly handle out-of-focus data, there is a separate option in star analysis so the default behaviour of the module is changed to assume out-of-focus data, which in turn, is then able to properly register out-of-focus data 😉 The option is called: out of focus? FWHM > 12 pixels? The next 2 images show APP 1.059 on out-of-focus data and APP 1.060:

• FIXED, Bayer Drizzle Integration with DSLR, a combination of applying bayer drizzle and DSLR data with raw borders caused a bug with Nikon D810a data. In normalization, the datatype is changed to 32bits floats and the raw borders of the DLSR data weren't dealt with properly for Bayer Drizzle, causing the bug. Reported by  Maurice Toet (@maurice_toet) here: https://www.astropixelprocessor.com/community/main-forum/java-lang-arrayindexoutofboundsexception-invalid-coordinates/ Maurice used the date H-alpha debayer algorithm in combination with Bayer Drizzle:

• Improvement, Frame List Panel image loading, to load an image into the image viewer, you need to double-click the image in the bottom frame list panel. Once an image is loaded, you can now scroll through the image list with the up and down arrow keys and thereby load succesive images into the image viewer.
• Adjustment, Frame List Panel, the color scheme has been adjusted a little bit to better discriminate between the Reference frame, the frame(s) selected and the frame loaded into the image viewer. 

• Fixed, Batch Modify Tool crop, in some cases, a crop could have a border artefact showing as a white border, usually on the bottom of the cropped image. This will no longer happen.
• Fixed, Preview Filter and Frame Analysis, performing analysis of a frame smaller than 128x128 pixels gave an error. This is now fixed.
• Improved, Integration, possible less IO read/writes which increases integration speed. While integrating data, a memory buffer is created to store the pixel stacks that are read from the memory maps. In APP 1.060, this memory buffer will be bigger if you have enough memory assigned to APP. This buffer can grow to 0.3-0.5x the amount of memory assigned to APP, so if you have 16GBs assigned, this buffer can become 5-8GBs large. In APP 1.059 only blocks of 1MB size were read or written at a time in the most optimal case (per IO call). This can now be as large as 4MB at at time. This reduces the amount of IO calls, which will speed up integration, especially on non-SSD drives.
• Improved, 0) LOAD, the load menu had been improved significantly. You can now load all frame types: Lights, Flats, Darks, DarkFlats, Bias and their Master frames. Off course, Bad Pixel Map, and you can also load Other/Processed frames, these are frames that don't need to be calibrated or integrated if you load them into APP. Furthermore, you can enable Multi-Channel/Filter and/or Multi-Session processing which will enable you to automatically process data of different channels/filters and/or sessions. You can let APP automatically recognize Masterframes and old Integrations. You can also load an old integration as a light frame to be included into a new integration. Lastly, you can now directly load single calibration frames as Master frames, so if you load a flat as a MasterFlat, it will automatically be used as such.

• Improved, Frame List Panel, Integration Output Maps, the different output maps now have their own frame type reported in the frame list panel. So a rejection map, is shown as a rejection map, previously, it was reported as an integration, which was confusing.