15th Feb 2024: Astro Pixel Processor 2.0.0-beta29 released - macOS native File Chooser, macOS CMD-Q fixed, read-only Fits on network fixed and other bug fixes
7th December 2023: added payment option Alipay to purchase Astro Pixel Processor from China, Hong Kong, Macau, Taiwan, Korea, Japan and other countries where Alipay is used.
Memory usage seems fine in 1.063 until the process gets to the point in integration just past all files being written to the file mapper. Once it attempts to begin the actual integration, memory usage hits the cap (62GB) and produces an error in APP. I am integrating a rather large data set of around 1,400 images. In version 1.062, I was able to successfully integrate nearly 3,000 images without memory issues. Not sure what, if anything has changed between the two versions of APP.
IMPROVED, MEMORY USAGE, some internal memory parameters have been adjusted to improve processing throughput and reduced memory consumption. APP will now again release memory in between processing steps.
I am about to release APP 1.064, but I will first have a thorough look at the difference between the settings between the 2 versions to make sure it behaves better again.
3000 images, wow, that might be record ! The biggest integration that I run was about 700 frames.
Do you see a clear difference between integrating 1000 or 3000 images of the same dataset?
Is it data collected over several years or are you shooting simply many short exposures ? If so, what is your main argument to use many short exposures?
I'm trying out an integration with 1.064 currently to see how it goes.
The ~3000 images were part of a 3-panel mosaic taken with an ASI183MM camera on a 100mm Esprit Apo scope. It wasn't so much a matter of necessity as it was just seeing if a very large number of short exposures would work well. I chose an easy target in M31 (integrated image attached).
I know this is an older topic since we now have 1.066, but I just read it so I would like to add my reasons for having well over 1000 frames in a typical project.
With my skies and my faster scopes, my L frames can be as short as 10 seconds to get to my desired mean ADU (around 1300), so typically I can have 500 or more L frames in a project. If I take LRGB plus Ha, it's easy to get 1000 to 1500 frames to get to the area of 10 to 15 hours of total integration time, which seems to be the total time that shows noticeable improvements over shorter total exposures.
In addition to my light frames, I can easily acquire hundreds of flat frames in a multi-session project. This is because I use ACP Expert, which tries to shoot when the object is close to the meridian, and as a result my 10 to 15 hours of total light frames are typically taken over 3 months or more of calendar time, probably around 10 different sessions. So I have multiple filter flats from 10 or more different sessions, 25 flats per filter per session!
I am using the ASI1600MM cool so frames are 32MB. I am using a computer with 32GB of RAM and with APP 1.065 (haven't run 1.066 yet), the memory use bars are red during the processing.
I'm trying out an integration with 1.064 currently to see how it goes.
The ~3000 images were part of a 3-panel mosaic taken with an ASI183MM camera on a 100mm Esprit Apo scope. It wasn't so much a matter of necessity as it was just seeing if a very large number of short exposures would work well. I chose an easy target in M31 (integrated image attached).
In the last APP updates, I have changed the memory settings several times. I might be able to tweak these internal controls still a bit better, but user feedback on this is vital. Can you comment on how APP 1.068 is working memory wise when integrating so many frames?
I know this is an older topic since we now have 1.066, but I just read it so I would like to add my reasons for having well over 1000 frames in a typical project.
With my skies and my faster scopes, my L frames can be as short as 10 seconds to get to my desired mean ADU (around 1300), so typically I can have 500 or more L frames in a project. If I take LRGB plus Ha, it's easy to get 1000 to 1500 frames to get to the area of 10 to 15 hours of total integration time, which seems to be the total time that shows noticeable improvements over shorter total exposures.
In addition to my light frames, I can easily acquire hundreds of flat frames in a multi-session project. This is because I use ACP Expert, which tries to shoot when the object is close to the meridian, and as a result my 10 to 15 hours of total light frames are typically taken over 3 months or more of calendar time, probably around 10 different sessions. So I have multiple filter flats from 10 or more different sessions, 25 flats per filter per session!
I am using the ASI1600MM cool so frames are 32MB. I am using a computer with 32GB of RAM and with APP 1.065 (haven't run 1.066 yet), the memory use bars are red during the processing.
There is always a comprise to be made with memory controls. If I set controls in such a manner, that memory is released quickly, it can have a negative effect on performance. On the other hand, if it is to slow wit releasing memory, the amount of memory that APP uses can actually have a negative impact on performance as well. I think that the memory controls in APP 1.068 are fine for most situations, but please let me know how it works on such large datasets. User feedback is essential.
I can always provide you with a modified test version that might be better suited for such large datasets, if you are interested? Then based on your feedback and from others, I might be able to provide a special configuration of APP for large datasets.
I can always provide you with a modified test version that might be better suited for such large datasets, if you are interested? Then based on your feedback and from others, I might be able to provide a special configuration of APP for large datasets.
Kind regards,
Mabula
Same problem on my side - memory error always when try to process more then 700 lights.... can You please provide me this version for testing ?
Total memory will always be a hard limit in some way or another, Mabula ofcourse tries to optimize usage as much as possible but there's only so much that can be done with these amounts of data. Turning off LNC might help, but I would go for sub integrations of 100 and combine those later on.
Yes and no... you need a minimum amount of physical memory to manage a paged or swapped memory manager. But I have 32 GB of RAM on my computer, that should be enough to create a file-based paging or swapping algorithm that may be slow, but won't hang or fail. Said with due respect to the mods and Mabula for his hard work!!
I agree that using a smaller amount of subs works, but part of the beauty of APP is its ability to do so much with one setup and one "go do it" button push, and that's why I think this is important to pursue. I have been successful with 1.068 with 450 light frames, although I had a problem with local normalization causing a hang. I don't want to derail this thread, just fyi. I am trying to find a small reproducible case to submit.
