Improved mosaic pro...
 
Share:
Notifications
Clear all

Improved mosaic processing


(@melix)
Molecular Cloud Customer
Joined: 2 years ago
Posts: 7
Topic starter  

Hi!

I would like to suggest an improvement to the mosaic processing process. I'm at the moment processing a mosaic of 4 panels, which were all taken during a single night via NINA. Usually when I do mosaics, I noticed that it doesn't quite work with APP if you try to register and integrate all subs from the 4 panels at once: you need to process each panel individually, then do a registration of the resulting panel masters, which works pretty well.

However, I wondered if using "sessions" would actually change the process in APP and do exactly that. I was wrong 🙂 So I assigned each panel to a session, and I started integration. My laptop has been burning for literally more than 15 hours straight, now,  on a 6 core i9-8950HK with 32GB of RAM and it's still processing... To give you an idea, integration requires 710GB of free space and I'm using an SSD 🙂 I didn't want to stop the process because I didn't expect it to take that long, so I'm waiting for the result but I guess it's not going to work well.

So here would be my suggestions:

1. add a "multi-panel" integration mode, where you assign lights to panels, and each panel is processed independently, then registered as a whole. I guess because not everyone would shot every panel in a single night you'd also need to assume that the optics differ, but this shouldn't really be an issue since it's a different process, not a technical difference

2. add an option to "pause" the process instead of cancel: as I explained it takes a LOT of hours to process, and sometimes I need to take video calls for example, which are quite CPU intensive. I made the mistake to cancel registration, which made me lost several hours

3. I'd like to reiterate support for XISF format: I'm the lucky owner of a ASI 6200MC Pro now, and each FITS file alone uses 122MB vs 59MB in XISF. I often have more than 120 subs (192 today) so it's starting to use a lot. I know it wouldn't change the fact that APP would still require GBs of free space for its own processing, but it would definitely help to make sure I have enough free space for processing.

Thanks again!


chris_16 liked
ReplyQuote
(@wvreeven)
Galaxy Admin
Joined: 4 years ago
Posts: 1651
 
Posted by: @melix

I noticed that it doesn't quite work with APP if you try to register and integrate all subs from the 4 panels at once: you need to process each panel individually, then do a registration of the resulting panel masters, which works pretty well.

One of the key features of APP is the capabilities of creating mosaics and it is possible to load all raw frames and create a mosaic. So if it doesn't work for you then you probably need to tweak the settings of APP. Usually the following settings work.

Tab 1: load all frames in one session if you have one set of calibration files otherwise use multiple sessions

Tab 3: select a high number of stars. Usually 2500 or more is sufficient

Tab 4: set "scale stop" to at least 10, enable "use dynamic distortion correction", disable "same camera and optics" and set "registration mode" to mosaic

Tab 5: set "mode" to advanced

Tab 6: enable "enable MBB" and set the slider to at least 10% or even 15%

Click integrate and inspect the result. Please let us know here if this works for you.


ReplyQuote
(@melix)
Molecular Cloud Customer
Joined: 2 years ago
Posts: 7
Topic starter  

This is exactly the process that I'm using today and that I'm explaining doesn't quite work here: it's been now 15h+ hours that it's running and I'm still waiting for the result. If I have to tweak the numbers of stars again, you can understand how frustrating it becomes. That's why I generally get much better results, and much faster, by processing each panel independently _then_ combining them. It works really well.


ReplyQuote
(@melix)
Molecular Cloud Customer
Joined: 2 years ago
Posts: 7
Topic starter  

Duh. Another reason for having another process and/or adding more resume safe points: after hours, my drive suddenly decided to disconnect.

 

1458 - MEMORY TO FILE MAPPER: encountered IO exception while reading from /media/cchampeau/T7/work/Light-session_1_session_2_session_3_session_4_session_5-1.dat
1458 - MEMORY TO FILE MAPPER: encountered IO exception while reading from /media/cchampeau/T7/work/Light-session_1_session_2_session_3_session_4_session_5-1.dat
1458 - MEMORY TO FILE MAPPER: encountered IO exception while reading from /media/cchampeau/T7/work/Light-session_1_session_2_session_3_session_4_session_5-1.dat
1458 - MEMORY TO FILE MAPPER: encountered IO exception while reading from /media/cchampeau/T7/work/Light-session_1_session_2_session_3_session_4_session_5-1.dat
1458 - 6) INTEGRATE: integrate light frames: integration task was cancelled
1458 - MEMORY TO FILE MAPPER: closing memory to file mapper.../media/cchampeau/T7/work/Light-session_1_session_2_session_3_session_4_session_5-1.dat
1458 - MEMORY TO FILE MAPPER: removing mapped file
1458 - MEMORY TO FILE MAPPER: closed succesfully
1458 - MEMORY TO FILE MAPPER: closing memory to file mapper.../media/cchampeau/T7/work/Light-weights-session_1_session_2_session_3_session_4_session_5-1.dat
1458 - null

 

The integration task was still not integrating pixels, but applying LNC. If I want to redo this, I need to go for 15+ hours of intensive I/O and CPU work. So no, I'll go the other way instead and do what I described above: I will integrate each panel separately _then_ do the mosaic.


ReplyQuote
(@wvreeven)
Galaxy Admin
Joined: 4 years ago
Posts: 1651
 

@melix I understand your frustration. Yes, the processing time becomes increasingly longer if more subs are added and even more so if those subs are part of a mosaic. I merely wanted to point out that it is possible to do a mosaic in one go in APP.

However, you have a point that processing the individual frames first and then combining them into a mosaic in much faster.

In any case, I informed Mabula about your requests.


ReplyQuote
(@melix)
Molecular Cloud Customer
Joined: 2 years ago
Posts: 7
Topic starter  

So I'm happy to tell you that processing each panel independently was much faster (~4hours). You can see the result here: https://www.astrobin.com/8qpcdq/

 

2021 09 01 heart soul nebula resized

ReplyQuote
(@wvreeven)
Galaxy Admin
Joined: 4 years ago
Posts: 1651
 

@melix Great result. Well done!


ReplyQuote
(@mabula-admin)
Quasar Admin
Joined: 5 years ago
Posts: 3122
 
Posted by: @melix

Hi!

I would like to suggest an improvement to the mosaic processing process. I'm at the moment processing a mosaic of 4 panels, which were all taken during a single night via NINA. Usually when I do mosaics, I noticed that it doesn't quite work with APP if you try to register and integrate all subs from the 4 panels at once: you need to process each panel individually, then do a registration of the resulting panel masters, which works pretty well.

However, I wondered if using "sessions" would actually change the process in APP and do exactly that. I was wrong 🙂 So I assigned each panel to a session, and I started integration. My laptop has been burning for literally more than 15 hours straight, now,  on a 6 core i9-8950HK with 32GB of RAM and it's still processing... To give you an idea, integration requires 710GB of free space and I'm using an SSD 🙂 I didn't want to stop the process because I didn't expect it to take that long, so I'm waiting for the result but I guess it's not going to work well.

So here would be my suggestions:

1. add a "multi-panel" integration mode, where you assign lights to panels, and each panel is processed independently, then registered as a whole. I guess because not everyone would shot every panel in a single night you'd also need to assume that the optics differ, but this shouldn't really be an issue since it's a different process, not a technical difference

2. add an option to "pause" the process instead of cancel: as I explained it takes a LOT of hours to process, and sometimes I need to take video calls for example, which are quite CPU intensive. I made the mistake to cancel registration, which made me lost several hours

3. I'd like to reiterate support for XISF format: I'm the lucky owner of a ASI 6200MC Pro now, and each FITS file alone uses 122MB vs 59MB in XISF. I often have more than 120 subs (192 today) so it's starting to use a lot. I know it wouldn't change the fact that APP would still require GBs of free space for its own processing, but it would definitely help to make sure I have enough free space for processing.

Thanks again!

Hi Cédric @melix and @wvreeven,

"I would like to suggest an improvement to the mosaic processing process. I'm at the moment processing a mosaic of 4 panels, which were all taken during a single night via NINA. Usually when I do mosaics, I noticed that it doesn't quite work with APP if you try to register and integrate all subs from the 4 panels at once: you need to process each panel individually, then do a registration of the resulting panel masters, which works pretty well.

However, I wondered if using "sessions" would actually change the process in APP and do exactly that. I was wrong 🙂 So I assigned each panel to a session, and I started integration. My laptop has been burning for literally more than 15 hours straight, now,  on a 6 core i9-8950HK with 32GB of RAM and it's still processing... To give you an idea, integration requires 710GB of free space and I'm using an SSD 🙂 I didn't want to stop the process because I didn't expect it to take that long, so I'm waiting for the result but I guess it's not going to work well."

Yes, the reason is simply that in the current mosaic algorithms, each frame will need to find connections to all the other frames... and if you try to make a mosaic from the single subs, it will take forever, a 3x3 mosaic, becomes a 300x300 mosaic ... . I am already developing new algorithms that will not have this issue and thus will be smarter 😉 For now, you really need to first integrate the individual panels and make the mosaic from those panels.

Once the improved algortihms are there, the workflow as it is now, will still be preferred though, because making a mosaic using the stacked mosaic panels is way more robust and efficient,.... because the mosaic panels are of much higher quality than the individual frames/subs 😉 So I would always first make the individual panels 😉 and then create the mosaic.

1. add a "multi-panel" integration mode, where you assign lights to panels, and each panel is processed independently, then registered as a whole. I guess because not everyone would shot every panel in a single night you'd also need to assume that the optics differ, but this shouldn't really be an issue since it's a different process, not a technical difference

So this option will not be needed, because of new mosaic algorithms coming...

2. add an option to "pause" the process instead of cancel: as I explained it takes a LOT of hours to process, and sometimes I need to take video calls for example, which are quite CPU intensive. I made the mistake to cancel registration, which made me lost several hours

Great suggestion, I have added it to my ToDo list.

I'd like to reiterate support for XISF format: I'm the lucky owner of a ASI 6200MC Pro now, and each FITS file alone uses 122MB vs 59MB in XISF. I often have more than 120 subs (192 today) so it's starting to use a lot. I know it wouldn't change the fact that APP would still require GBs of free space for its own processing, but it would definitely help to make sure I have enough free space for processing.

XISF support is on our Todo List for sure, but it is quite low on the list because other things are simply more pressing at the moment for developing and expanding Astro Pixel Processor I think. The space concern will be solved by another thing on my ToDo list. Implementing Lossless compression in FITS files, which can be done quickly. So I can say for now that XISF support can and will be implemented at some stage.

Mabula


ReplyQuote
(@mabula-admin)
Quasar Admin
Joined: 5 years ago
Posts: 3122
 
Posted by: @melix

So I'm happy to tell you that processing each panel independently was much faster (~4hours). You can see the result here: https://www.astrobin.com/8qpcdq/

 

2021 09 01 heart soul nebula resized

Excellent result @melix 🙂 ! Thanks for sharing 🙂

 


ReplyQuote
(@melix)
Molecular Cloud Customer
Joined: 2 years ago
Posts: 7
Topic starter  

Thanks a lot for your answer, @mabula-admin !


ReplyQuote
Share: