Problems with creat...
 
Share:
Notifications
Clear all

Problems with creating a mosaic 🙁  

Page 3 / 3
  RSS

(@vincent-mod)
Universe Admin
Joined: 4 years ago
Posts: 3961
October 27, 2021 16:12  

Hi Michal,

Thanks for the explanation and huge tests. 😉 So, did you try to make panels first? When I do a mosaic I always start with;

- Scale start of 5, stop to 10

- Advanced normalization

- 2000 stars to analyze.

- Same camera/optics disabled and DDC on.

With a very big mosaic I always make panels first.


ReplyQuote
(@mostaszewski)
Brown Dwarf Customer
Joined: 4 months ago
Posts: 7
October 27, 2021 16:42  

Hi Vincent, 

Thank you for so fast answer. 
Panels are ready and calibrated now everything what I am tying to do, it is on 32bits fits. But I did not tried to stitch for example a 47(282/6=47) panels into bigger panel and then make them into one final panorama, its my last strategy to try(6x47->6->1). Do you recommend to do it already, not try to stitch every 282 at once(282->1)?

So your idea is to lower the stars number and combinations needed?
Should I left it on quadrilaterals recognition?


ReplyQuote
(@vincent-mod)
Universe Admin
Joined: 4 years ago
Posts: 3961
October 27, 2021 16:59  

I recommend not to do all at once no. Calibrated projective is a good idea, it may speed it up a bit. I always try to set a mosaic up like I mentioned and not change anything else. If that doesn't work, only then I'll try other strategies. Let's try do this step by step, if you try this first then post your progress and we can see what the next best action might be. I would maybe even try to make one half of the mosaic from panels (not all subs again) and then the other half...


ReplyQuote
(@mostaszewski)
Brown Dwarf Customer
Joined: 4 months ago
Posts: 7
October 27, 2021 17:01  

Ok, thank you for your ideas. I will try to make it in few steps and I will be back with results 😊


ReplyQuote
(@mostaszewski)
Brown Dwarf Customer
Joined: 4 months ago
Posts: 7
November 11, 2021 18:55  

Hi @Vincent
A few updates and new questions. 

First of all, after 2 weeks of really intensive I found working registration settings for all 282 panels at once. Right at the moment I am waiting for first final integration of my panorama. But I found a few follow questions listed by numbers and letters:

1A. I asked you here and in other mentioned thread about possible of stitching so big panorama(a few gigapixels panorama in resolution) and you said that it is possible, but I also know about the file size limit 715mpx for RGB image noticed in Quick Reference Guide. At the final step on Integration I got an notification that APP has to downscale an image to maximum 715mpx. What is a little troublesome because in this resolution I will lose a lot of details at this focal view or I have to go back and make 5 or 6 panels each for these 715mpx and later stitch them together in different panorama program but the then problem could be a normalization of panels. Also stitching in different program than APP could be problematic (I mostly think about not perfect connections), so if I already managed to register all panels at once It will be it will be very nice if I can do it at once in APP 🙂
1B. Does this limit is based on technical aspects of FITS file or Integration engine of APP which caused that limit can't be enabled?
1C. If it's not any technological limit is it possible to get an installation file of APP version with enabled limit. I really care about it an in Quick Reference Guide, Mabula noticed that it shouldn't be a thing in coming updates of APP.

During whole registration processes I did a lot of tests with different input sets, different parameters and I learned a lot about all the algorithms behind the registration engine. I also made these tests on different computers with different CPUs. At the end I noticed that registration algorithm canceled in most attempts at step RMS 11 and displayed error "MultipleViewCameraCalibrationSimpleDistortionCorrectionNotSameCameraIncomplete"...Solution contains uncountable numbers

2A. I also noticed that I can successfully end the registration in not every attempt (but I think its mostly caused that RANSAC is non deterministic and I had so big data set at input that my parameters sometimes were "countable" and sometimes they didn't depends on random set which was chosen by RANSAC - is it correct way of think? 

Is the RANSAC depends on number of available CPU threads? Because when I analyzed usage of CPU during the registration APP working on as many threads as I gave available to APP by the slider, but speed of achieving final result of registration in general don't change. I could even say that APP gives me final registration faster when I gave only 4-8 CPU threads available to the process. When I set for example 24 threads, whole process slow down and mostly fails. 

2B. If there is no optimized RANSAC multithread algorithm, so what calculated all these available threads of CPU?

I will be very thankful for answers. Mostly about my question of file size limit. It's very important for me and whole my project. 
When I will end it I will also share all my thoughts about taking panoramas like this because I think that I came to a few very important conclusions during this project. 

Regards, 
Michał


ReplyQuote
(@vincent-mod)
Universe Admin
Joined: 4 years ago
Posts: 3961
November 11, 2021 19:02  

Great questions and you sure are testing the limits of any system. 😉 I'll ask Mabula to chime in..


ReplyQuote
(@mostaszewski)
Brown Dwarf Customer
Joined: 4 months ago
Posts: 7
November 16, 2021 14:50  

Hi, 
Do you know something new about my questions?
I am mostly interested is the FITS file format does have this size limit?


ReplyQuote
(@vincent-mod)
Universe Admin
Joined: 4 years ago
Posts: 3961
November 18, 2021 13:54  

Theoretically it shouldn't. I'll ping Mabula again to see if he can answer, sorry about the delay.


ReplyQuote
(@wvreeven)
Quasar Admin
Joined: 3 years ago
Posts: 1226
November 18, 2021 18:52  

@mostaszewski FITS files get used by professional astronomers for LOTS of data. So the only limit to FITS file size is set by the operating system. Not sure if there is a limit to the size that APP can handle, again apart from the limits set by the operating system and the memory of the computer processing the data.


ReplyQuote
(@mabula-admin)
Universe Admin
Joined: 5 years ago
Posts: 2747
November 18, 2021 22:50  
Posted by: @mostaszewski

Hi @Vincent
A few updates and new questions. 

First of all, after 2 weeks of really intensive I found working registration settings for all 282 panels at once. Right at the moment I am waiting for first final integration of my panorama. But I found a few follow questions listed by numbers and letters:

1A. I asked you here and in other mentioned thread about possible of stitching so big panorama(a few gigapixels panorama in resolution) and you said that it is possible, but I also know about the file size limit 715mpx for RGB image noticed in Quick Reference Guide. At the final step on Integration I got an notification that APP has to downscale an image to maximum 715mpx. What is a little troublesome because in this resolution I will lose a lot of details at this focal view or I have to go back and make 5 or 6 panels each for these 715mpx and later stitch them together in different panorama program but the then problem could be a normalization of panels. Also stitching in different program than APP could be problematic (I mostly think about not perfect connections), so if I already managed to register all panels at once It will be it will be very nice if I can do it at once in APP 🙂
1B. Does this limit is based on technical aspects of FITS file or Integration engine of APP which caused that limit can't be enabled?
1C. If it's not any technological limit is it possible to get an installation file of APP version with enabled limit. I really care about it an in Quick Reference Guide, Mabula noticed that it shouldn't be a thing in coming updates of APP.

During whole registration processes I did a lot of tests with different input sets, different parameters and I learned a lot about all the algorithms behind the registration engine. I also made these tests on different computers with different CPUs. At the end I noticed that registration algorithm canceled in most attempts at step RMS 11 and displayed error "MultipleViewCameraCalibrationSimpleDistortionCorrectionNotSameCameraIncomplete"...Solution contains uncountable numbers

2A. I also noticed that I can successfully end the registration in not every attempt (but I think its mostly caused that RANSAC is non deterministic and I had so big data set at input that my parameters sometimes were "countable" and sometimes they didn't depends on random set which was chosen by RANSAC - is it correct way of think? 

Is the RANSAC depends on number of available CPU threads? Because when I analyzed usage of CPU during the registration APP working on as many threads as I gave available to APP by the slider, but speed of achieving final result of registration in general don't change. I could even say that APP gives me final registration faster when I gave only 4-8 CPU threads available to the process. When I set for example 24 threads, whole process slow down and mostly fails. 

2B. If there is no optimized RANSAC multithread algorithm, so what calculated all these available threads of CPU?

I will be very thankful for answers. Mostly about my question of file size limit. It's very important for me and whole my project. 
When I will end it I will also share all my thoughts about taking panoramas like this because I think that I came to a few very important conclusions during this project. 

Regards, 
Michał

Hi @mostaszewski,

Congratulations on your work to create such a large mosaic of the sky!

Let me address your questions now...

1A: The pixelsize limit is there and is caused by the image size limit in Java that we currently use. We can't have more pixels than 2^(32-1) in an image. So for a monochrom image, you can have 2.1 Gigapixel, but for an RGB this is 2.1/3= 715MegaPixels. It is definitely on our ToDo list to make much larger images, but for now it is not yet possible. This is the reason that the mosaic needs to be downscaled.

1B: answered in 1A

1C: answered in 1A

2A: The incountable solution occurs when not all frames were successfully registered in the mosaic, a bug then actually occurs which is on the list to fix. Ransac needs to use a random number generator, so yes, that causes it to sometimes get a slightly different result.

Is the RANSAC depends on number of available CPU threads? Because when I analyzed usage of CPU during the registration APP working on as many threads as I gave available to APP by the slider, but speed of achieving final result of registration in general don't change. I could even say that APP gives me final registration faster when I gave only 4-8 CPU threads available to the process. When I set for example 24 threads, whole process slow down and mostly fails. 

Okay, that is a very interesting observation of which I am not aware myself. If it is like you say, then maybe there is a concurrency issue. I will investigate as soon as possible.

2B: At the moment, the current registration engine can perform multiple Ransac calculation for different images at the same time. But those Ransac calculations per image are single threaded. I just had a good look at our code, and I think I can easily make it multi-threaded which should really speed things up immediately. I have made a big note of this to implement quickly 😉

Thank you very much Michał for sharing your questions, experience and thoughts about this.

Mabula


ReplyQuote
(@mabula-admin)
Universe Admin
Joined: 5 years ago
Posts: 2747
November 28, 2021 22:23  

Hi Michał @mostaszewski, @wvreeven, @vincent-mod,

Thank you very much for bringing the strange behaviour to my attention regarding the use of many cores and then slow down of the registration of your big mosaic...

I am very happy to say that I have located the problem that caused the slowdown and i have fixed it. Initial tests on small mosaics give a very big speed increase now! and you will see that the cpu usage stays much higher while performing the mosaic calculation 🙂

The performance increase will be included in the next version for sure.

Mabula

 

 


ReplyQuote
Page 3 / 3
Share: