Improved mosaic pro...
 
Share:
Notifications
Clear all

19 June 2021: Our upload server https://upload.astropixelprocessor.com/ has been migrated successfully to our new office with higher upload and download speeds (nearly 10MByte/sec up/down ) ! We now have 1 general upload user called: upload with password: upload. The users upload1 - upload5 have been disabled.

31 May 2021: APP 1.083-beta2 has been released ! APP 1.083 stable will follow soon afterwards. It includes a completely new Star Reducer Tool, New File Saver Module, Improved Comet registration and much more, check the release notes here!

DOWNLOADS are available HERE!

 

Improved mosaic processing  

  RSS

(@melix)
Brown Dwarf Customer
Joined: 1 year ago
Posts: 7
September 3, 2021 09:07  

Hi!

I would like to suggest an improvement to the mosaic processing process. I'm at the moment processing a mosaic of 4 panels, which were all taken during a single night via NINA. Usually when I do mosaics, I noticed that it doesn't quite work with APP if you try to register and integrate all subs from the 4 panels at once: you need to process each panel individually, then do a registration of the resulting panel masters, which works pretty well.

However, I wondered if using "sessions" would actually change the process in APP and do exactly that. I was wrong 🙂 So I assigned each panel to a session, and I started integration. My laptop has been burning for literally more than 15 hours straight, now,  on a 6 core i9-8950HK with 32GB of RAM and it's still processing... To give you an idea, integration requires 710GB of free space and I'm using an SSD 🙂 I didn't want to stop the process because I didn't expect it to take that long, so I'm waiting for the result but I guess it's not going to work well.

So here would be my suggestions:

1. add a "multi-panel" integration mode, where you assign lights to panels, and each panel is processed independently, then registered as a whole. I guess because not everyone would shot every panel in a single night you'd also need to assume that the optics differ, but this shouldn't really be an issue since it's a different process, not a technical difference

2. add an option to "pause" the process instead of cancel: as I explained it takes a LOT of hours to process, and sometimes I need to take video calls for example, which are quite CPU intensive. I made the mistake to cancel registration, which made me lost several hours

3. I'd like to reiterate support for XISF format: I'm the lucky owner of a ASI 6200MC Pro now, and each FITS file alone uses 122MB vs 59MB in XISF. I often have more than 120 subs (192 today) so it's starting to use a lot. I know it wouldn't change the fact that APP would still require GBs of free space for its own processing, but it would definitely help to make sure I have enough free space for processing.

Thanks again!


chris_16 liked
ReplyQuote
(@wvreeven)
Quasar Admin
Joined: 3 years ago
Posts: 1146
September 3, 2021 10:39  
Posted by: @melix

I noticed that it doesn't quite work with APP if you try to register and integrate all subs from the 4 panels at once: you need to process each panel individually, then do a registration of the resulting panel masters, which works pretty well.

One of the key features of APP is the capabilities of creating mosaics and it is possible to load all raw frames and create a mosaic. So if it doesn't work for you then you probably need to tweak the settings of APP. Usually the following settings work.

Tab 1: load all frames in one session if you have one set of calibration files otherwise use multiple sessions

Tab 3: select a high number of stars. Usually 2500 or more is sufficient

Tab 4: set "scale stop" to at least 10, enable "use dynamic distortion correction", disable "same camera and optics" and set "registration mode" to mosaic

Tab 5: set "mode" to advanced

Tab 6: enable "enable MBB" and set the slider to at least 10% or even 15%

Click integrate and inspect the result. Please let us know here if this works for you.


ReplyQuote
(@melix)
Brown Dwarf Customer
Joined: 1 year ago
Posts: 7
September 3, 2021 10:56  

This is exactly the process that I'm using today and that I'm explaining doesn't quite work here: it's been now 15h+ hours that it's running and I'm still waiting for the result. If I have to tweak the numbers of stars again, you can understand how frustrating it becomes. That's why I generally get much better results, and much faster, by processing each panel independently _then_ combining them. It works really well.


ReplyQuote
(@melix)
Brown Dwarf Customer
Joined: 1 year ago
Posts: 7
September 3, 2021 14:34  

Duh. Another reason for having another process and/or adding more resume safe points: after hours, my drive suddenly decided to disconnect.

 

14:27:58 - MEMORY TO FILE MAPPER: encountered IO exception while reading from /media/cchampeau/T7/work/Light-session_1_session_2_session_3_session_4_session_5-1.dat
14:27:58 - MEMORY TO FILE MAPPER: encountered IO exception while reading from /media/cchampeau/T7/work/Light-session_1_session_2_session_3_session_4_session_5-1.dat
14:27:58 - MEMORY TO FILE MAPPER: encountered IO exception while reading from /media/cchampeau/T7/work/Light-session_1_session_2_session_3_session_4_session_5-1.dat
14:27:58 - MEMORY TO FILE MAPPER: encountered IO exception while reading from /media/cchampeau/T7/work/Light-session_1_session_2_session_3_session_4_session_5-1.dat
14:27:58 - 6) INTEGRATE: integrate light frames: integration task was cancelled
14:27:58 - MEMORY TO FILE MAPPER: closing memory to file mapper.../media/cchampeau/T7/work/Light-session_1_session_2_session_3_session_4_session_5-1.dat
14:27:58 - MEMORY TO FILE MAPPER: removing mapped file
14:27:58 - MEMORY TO FILE MAPPER: closed succesfully
14:27:58 - MEMORY TO FILE MAPPER: closing memory to file mapper.../media/cchampeau/T7/work/Light-weights-session_1_session_2_session_3_session_4_session_5-1.dat
14:27:58 - null

 

The integration task was still not integrating pixels, but applying LNC. If I want to redo this, I need to go for 15+ hours of intensive I/O and CPU work. So no, I'll go the other way instead and do what I described above: I will integrate each panel separately _then_ do the mosaic.


ReplyQuote
(@wvreeven)
Quasar Admin
Joined: 3 years ago
Posts: 1146
September 3, 2021 15:00  

@melix I understand your frustration. Yes, the processing time becomes increasingly longer if more subs are added and even more so if those subs are part of a mosaic. I merely wanted to point out that it is possible to do a mosaic in one go in APP.

However, you have a point that processing the individual frames first and then combining them into a mosaic in much faster.

In any case, I informed Mabula about your requests.


ReplyQuote
(@melix)
Brown Dwarf Customer
Joined: 1 year ago
Posts: 7
September 4, 2021 16:02  

So I'm happy to tell you that processing each panel independently was much faster (~4hours). You can see the result here: https://www.astrobin.com/8qpcdq/

 

2021 09 01 heart soul nebula resized

ReplyQuote
(@wvreeven)
Quasar Admin
Joined: 3 years ago
Posts: 1146
September 4, 2021 16:09  

@melix Great result. Well done!


ReplyQuote
(@mabula-admin)
Universe Admin
Joined: 5 years ago
Posts: 2728
September 6, 2021 22:40  
Posted by: @melix

Hi!

I would like to suggest an improvement to the mosaic processing process. I'm at the moment processing a mosaic of 4 panels, which were all taken during a single night via NINA. Usually when I do mosaics, I noticed that it doesn't quite work with APP if you try to register and integrate all subs from the 4 panels at once: you need to process each panel individually, then do a registration of the resulting panel masters, which works pretty well.

However, I wondered if using "sessions" would actually change the process in APP and do exactly that. I was wrong 🙂 So I assigned each panel to a session, and I started integration. My laptop has been burning for literally more than 15 hours straight, now,  on a 6 core i9-8950HK with 32GB of RAM and it's still processing... To give you an idea, integration requires 710GB of free space and I'm using an SSD 🙂 I didn't want to stop the process because I didn't expect it to take that long, so I'm waiting for the result but I guess it's not going to work well.

So here would be my suggestions:

1. add a "multi-panel" integration mode, where you assign lights to panels, and each panel is processed independently, then registered as a whole. I guess because not everyone would shot every panel in a single night you'd also need to assume that the optics differ, but this shouldn't really be an issue since it's a different process, not a technical difference

2. add an option to "pause" the process instead of cancel: as I explained it takes a LOT of hours to process, and sometimes I need to take video calls for example, which are quite CPU intensive. I made the mistake to cancel registration, which made me lost several hours

3. I'd like to reiterate support for XISF format: I'm the lucky owner of a ASI 6200MC Pro now, and each FITS file alone uses 122MB vs 59MB in XISF. I often have more than 120 subs (192 today) so it's starting to use a lot. I know it wouldn't change the fact that APP would still require GBs of free space for its own processing, but it would definitely help to make sure I have enough free space for processing.

Thanks again!

Hi Cédric @melix and @wvreeven,

"I would like to suggest an improvement to the mosaic processing process. I'm at the moment processing a mosaic of 4 panels, which were all taken during a single night via NINA. Usually when I do mosaics, I noticed that it doesn't quite work with APP if you try to register and integrate all subs from the 4 panels at once: you need to process each panel individually, then do a registration of the resulting panel masters, which works pretty well.

However, I wondered if using "sessions" would actually change the process in APP and do exactly that. I was wrong 🙂 So I assigned each panel to a session, and I started integration. My laptop has been burning for literally more than 15 hours straight, now,  on a 6 core i9-8950HK with 32GB of RAM and it's still processing... To give you an idea, integration requires 710GB of free space and I'm using an SSD 🙂 I didn't want to stop the process because I didn't expect it to take that long, so I'm waiting for the result but I guess it's not going to work well."

Yes, the reason is simply that in the current mosaic algorithms, each frame will need to find connections to all the other frames... and if you try to make a mosaic from the single subs, it will take forever, a 3x3 mosaic, becomes a 300x300 mosaic ... . I am already developing new algorithms that will not have this issue and thus will be smarter 😉 For now, you really need to first integrate the individual panels and make the mosaic from those panels.

Once the improved algortihms are there, the workflow as it is now, will still be preferred though, because making a mosaic using the stacked mosaic panels is way more robust and efficient,.... because the mosaic panels are of much higher quality than the individual frames/subs 😉 So I would always first make the individual panels 😉 and then create the mosaic.

1. add a "multi-panel" integration mode, where you assign lights to panels, and each panel is processed independently, then registered as a whole. I guess because not everyone would shot every panel in a single night you'd also need to assume that the optics differ, but this shouldn't really be an issue since it's a different process, not a technical difference

So this option will not be needed, because of new mosaic algorithms coming...

2. add an option to "pause" the process instead of cancel: as I explained it takes a LOT of hours to process, and sometimes I need to take video calls for example, which are quite CPU intensive. I made the mistake to cancel registration, which made me lost several hours

Great suggestion, I have added it to my ToDo list.

I'd like to reiterate support for XISF format: I'm the lucky owner of a ASI 6200MC Pro now, and each FITS file alone uses 122MB vs 59MB in XISF. I often have more than 120 subs (192 today) so it's starting to use a lot. I know it wouldn't change the fact that APP would still require GBs of free space for its own processing, but it would definitely help to make sure I have enough free space for processing.

XISF support is on our Todo List for sure, but it is quite low on the list because other things are simply more pressing at the moment for developing and expanding Astro Pixel Processor I think. The space concern will be solved by another thing on my ToDo list. Implementing Lossless compression in FITS files, which can be done quickly. So I can say for now that XISF support can and will be implemented at some stage.

Mabula


ReplyQuote
(@mabula-admin)
Universe Admin
Joined: 5 years ago
Posts: 2728
September 6, 2021 22:41  
Posted by: @melix

So I'm happy to tell you that processing each panel independently was much faster (~4hours). You can see the result here: https://www.astrobin.com/8qpcdq/

 

2021 09 01 heart soul nebula resized

Excellent result @melix 🙂 ! Thanks for sharing 🙂

 


ReplyQuote
(@melix)
Brown Dwarf Customer
Joined: 1 year ago
Posts: 7
September 6, 2021 22:47  

Thanks a lot for your answer, @mabula-admin !


ReplyQuote
Share: