15th Feb 2024: Astro Pixel Processor 2.0.0-beta29 released - macOS native File Chooser, macOS CMD-Q fixed, read-only Fits on network fixed and other bug fixes
7th December 2023: added payment option Alipay to purchase Astro Pixel Processor from China, Hong Kong, Macau, Taiwan, Korea, Japan and other countries where Alipay is used.
I would like to suggest an improvement to the mosaic processing process. I'm at the moment processing a mosaic of 4 panels, which were all taken during a single night via NINA. Usually when I do mosaics, I noticed that it doesn't quite work with APP if you try to register and integrate all subs from the 4 panels at once: you need to process each panel individually, then do a registration of the resulting panel masters, which works pretty well.
However, I wondered if using "sessions" would actually change the process in APP and do exactly that. I was wrong 🙂 So I assigned each panel to a session, and I started integration. My laptop has been burning for literally more than 15 hours straight, now, on a 6 core i9-8950HK with 32GB of RAM and it's still processing... To give you an idea, integration requires 710GB of free space and I'm using an SSD 🙂 I didn't want to stop the process because I didn't expect it to take that long, so I'm waiting for the result but I guess it's not going to work well.
So here would be my suggestions:
1. add a "multi-panel" integration mode, where you assign lights to panels, and each panel is processed independently, then registered as a whole. I guess because not everyone would shot every panel in a single night you'd also need to assume that the optics differ, but this shouldn't really be an issue since it's a different process, not a technical difference
2. add an option to "pause" the process instead of cancel: as I explained it takes a LOT of hours to process, and sometimes I need to take video calls for example, which are quite CPU intensive. I made the mistake to cancel registration, which made me lost several hours
3. I'd like to reiterate support for XISF format: I'm the lucky owner of a ASI 6200MC Pro now, and each FITS file alone uses 122MB vs 59MB in XISF. I often have more than 120 subs (192 today) so it's starting to use a lot. I know it wouldn't change the fact that APP would still require GBs of free space for its own processing, but it would definitely help to make sure I have enough free space for processing.
I noticed that it doesn't quite work with APP if you try to register and integrate all subs from the 4 panels at once: you need to process each panel individually, then do a registration of the resulting panel masters, which works pretty well.
One of the key features of APP is the capabilities of creating mosaics and it is possible to load all raw frames and create a mosaic. So if it doesn't work for you then you probably need to tweak the settings of APP. Usually the following settings work.
Tab 1: load all frames in one session if you have one set of calibration files otherwise use multiple sessions
Tab 3: select a high number of stars. Usually 2500 or more is sufficient
Tab 4: set "scale stop" to at least 10, enable "use dynamic distortion correction", disable "same camera and optics" and set "registration mode" to mosaic
Tab 5: set "mode" to advanced
Tab 6: enable "enable MBB" and set the slider to at least 10% or even 15%
Click integrate and inspect the result. Please let us know here if this works for you.
This is exactly the process that I'm using today and that I'm explaining doesn't quite work here: it's been now 15h+ hours that it's running and I'm still waiting for the result. If I have to tweak the numbers of stars again, you can understand how frustrating it becomes. That's why I generally get much better results, and much faster, by processing each panel independently _then_ combining them. It works really well.
Duh. Another reason for having another process and/or adding more resume safe points: after hours, my drive suddenly decided to disconnect.
14:27:58 - MEMORY TO FILE MAPPER: encountered IO exception while reading from /media/cchampeau/T7/work/Light-session_1_session_2_session_3_session_4_session_5-1.dat 14:27:58 - MEMORY TO FILE MAPPER: encountered IO exception while reading from /media/cchampeau/T7/work/Light-session_1_session_2_session_3_session_4_session_5-1.dat 14:27:58 - MEMORY TO FILE MAPPER: encountered IO exception while reading from /media/cchampeau/T7/work/Light-session_1_session_2_session_3_session_4_session_5-1.dat 14:27:58 - MEMORY TO FILE MAPPER: encountered IO exception while reading from /media/cchampeau/T7/work/Light-session_1_session_2_session_3_session_4_session_5-1.dat 14:27:58 - 6) INTEGRATE: integrate light frames: integration task was cancelled 14:27:58 - MEMORY TO FILE MAPPER: closing memory to file mapper.../media/cchampeau/T7/work/Light-session_1_session_2_session_3_session_4_session_5-1.dat 14:27:58 - MEMORY TO FILE MAPPER: removing mapped file 14:27:58 - MEMORY TO FILE MAPPER: closed succesfully 14:27:58 - MEMORY TO FILE MAPPER: closing memory to file mapper.../media/cchampeau/T7/work/Light-weights-session_1_session_2_session_3_session_4_session_5-1.dat 14:27:58 - null
The integration task was still not integrating pixels, but applying LNC. If I want to redo this, I need to go for 15+ hours of intensive I/O and CPU work. So no, I'll go the other way instead and do what I described above: I will integrate each panel separately _then_ do the mosaic.
@melix I understand your frustration. Yes, the processing time becomes increasingly longer if more subs are added and even more so if those subs are part of a mosaic. I merely wanted to point out that it is possible to do a mosaic in one go in APP.
However, you have a point that processing the individual frames first and then combining them into a mosaic in much faster.
In any case, I informed Mabula about your requests.
So I'm happy to tell you that processing each panel independently was much faster (~4hours). You can see the result here: https://www.astrobin.com/8qpcdq/
I would like to suggest an improvement to the mosaic processing process. I'm at the moment processing a mosaic of 4 panels, which were all taken during a single night via NINA. Usually when I do mosaics, I noticed that it doesn't quite work with APP if you try to register and integrate all subs from the 4 panels at once: you need to process each panel individually, then do a registration of the resulting panel masters, which works pretty well.
However, I wondered if using "sessions" would actually change the process in APP and do exactly that. I was wrong 🙂 So I assigned each panel to a session, and I started integration. My laptop has been burning for literally more than 15 hours straight, now, on a 6 core i9-8950HK with 32GB of RAM and it's still processing... To give you an idea, integration requires 710GB of free space and I'm using an SSD 🙂 I didn't want to stop the process because I didn't expect it to take that long, so I'm waiting for the result but I guess it's not going to work well.
So here would be my suggestions:
1. add a "multi-panel" integration mode, where you assign lights to panels, and each panel is processed independently, then registered as a whole. I guess because not everyone would shot every panel in a single night you'd also need to assume that the optics differ, but this shouldn't really be an issue since it's a different process, not a technical difference
2. add an option to "pause" the process instead of cancel: as I explained it takes a LOT of hours to process, and sometimes I need to take video calls for example, which are quite CPU intensive. I made the mistake to cancel registration, which made me lost several hours
3. I'd like to reiterate support for XISF format: I'm the lucky owner of a ASI 6200MC Pro now, and each FITS file alone uses 122MB vs 59MB in XISF. I often have more than 120 subs (192 today) so it's starting to use a lot. I know it wouldn't change the fact that APP would still require GBs of free space for its own processing, but it would definitely help to make sure I have enough free space for processing.
"I would like to suggest an improvement to the mosaic processing process. I'm at the moment processing a mosaic of 4 panels, which were all taken during a single night via NINA. Usually when I do mosaics, I noticed that it doesn't quite work with APP if you try to register and integrate all subs from the 4 panels at once: you need to process each panel individually, then do a registration of the resulting panel masters, which works pretty well.
However, I wondered if using "sessions" would actually change the process in APP and do exactly that. I was wrong 🙂 So I assigned each panel to a session, and I started integration. My laptop has been burning for literally more than 15 hours straight, now, on a 6 core i9-8950HK with 32GB of RAM and it's still processing... To give you an idea, integration requires 710GB of free space and I'm using an SSD 🙂 I didn't want to stop the process because I didn't expect it to take that long, so I'm waiting for the result but I guess it's not going to work well."
Yes, the reason is simply that in the current mosaic algorithms, each frame will need to find connections to all the other frames... and if you try to make a mosaic from the single subs, it will take forever, a 3x3 mosaic, becomes a 300x300 mosaic ... . I am already developing new algorithms that will not have this issue and thus will be smarter 😉 For now, you really need to first integrate the individual panels and make the mosaic from those panels.
Once the improved algortihms are there, the workflow as it is now, will still be preferred though, because making a mosaic using the stacked mosaic panels is way more robust and efficient,.... because the mosaic panels are of much higher quality than the individual frames/subs 😉 So I would always first make the individual panels 😉 and then create the mosaic.
1. add a "multi-panel" integration mode, where you assign lights to panels, and each panel is processed independently, then registered as a whole. I guess because not everyone would shot every panel in a single night you'd also need to assume that the optics differ, but this shouldn't really be an issue since it's a different process, not a technical difference
So this option will not be needed, because of new mosaic algorithms coming...
2. add an option to "pause" the process instead of cancel: as I explained it takes a LOT of hours to process, and sometimes I need to take video calls for example, which are quite CPU intensive. I made the mistake to cancel registration, which made me lost several hours
Great suggestion, I have added it to my ToDo list.
I'd like to reiterate support for XISF format: I'm the lucky owner of a ASI 6200MC Pro now, and each FITS file alone uses 122MB vs 59MB in XISF. I often have more than 120 subs (192 today) so it's starting to use a lot. I know it wouldn't change the fact that APP would still require GBs of free space for its own processing, but it would definitely help to make sure I have enough free space for processing.
XISF support is on our Todo List for sure, but it is quite low on the list because other things are simply more pressing at the moment for developing and expanding Astro Pixel Processor I think. The space concern will be solved by another thing on my ToDo list. Implementing Lossless compression in FITS files, which can be done quickly. So I can say for now that XISF support can and will be implemented at some stage.
So I'm happy to tell you that processing each panel independently was much faster (~4hours). You can see the result here: https://www.astrobin.com/8qpcdq/
