Does APP works with SWAP memory?

I do know APP manages all memory by itself, but if that means it automatically uses swap I’m not sure about. I’ll ask Mabula for specifics. 

Yesterday I tried to stack mosaic and APP during stacking process told me that my system is out of memory. Then I saw popup windows with information that my mosaic must be down-scaled with some factor 0.[n] :/ and it start some task to process something that consumed my CPU with 100% usage indefinitely. After waiting over 1 hour, nothing happened, no progress at all, no information in the APP console window what it's doing.

I recognized that my system SWAP mem was not used at all only RAM (16 GB) was consumed. That was a reason of my earlier question about using SWAP by APP.

If APP is not using SWAP memory at all on Linux, then it will be a huge disadvantage in use. Hope it use somehow or it's possible to turn it on?

I don't know if the windows version also has this problem?

Ok, so APP is not using a swap file on the system, also not on Windows. The issue would be that making this mosaic then would take ages. Unfortunately it would mean that if you want a mosaic this big, you either would need more memory in the system or scale it down so it does fit. The subsequent 100% cpu and kind of freezing thing as it seems, is maybe not good, I would need to investigate that. But best way forward would be to scale it down.

A possible workaround on the APP side could also be to create the mosaic in parts and save those parts top disk. But this requires a rather big reprogramming of that module, so won't be available soon.

@vincent-mod

On the one hand, scaling the mosaic down is a solution, but in a wider point of view it doesn't make sense at all in my opinion. If you have to scale down picture, so why do mosaics at all?
The essence of the mosaic is to increase the resolution and detail of the photographed object to see more f.e.

APP support for SWAP system memory would help a lot or other functionality that would allow you to process a large image.
I don't care that SWAP memory will slow down the whole processing data and will extend time to process. For me, the computer could count even a week if needed.

Well, for now I will use DSS :/ for mosaics.

BTW, if you need to test any new versions of APP or functionalities on Linux, I can help. Linux on my PC is a basic system.

Hi @pekdar. Swap memory in Linux is not used like you're describing. Swap is managed by the kernel. It isn't an extra piece of RAM available on disk (like a Windows RAM Boost using a USB drive - but that's an awful idea IMHO). With Java you set the amount of memory allocated to the Java process (and APP is a Java program). If you set 8Gb to APP (in the CFG button at the top of the screen) then Java will set the max memory to be 8Gb and then Java manages this (it has its own memory management). However, to Linux, this is an 8Gb process. If your Linux system is running low on RAM (and Linux does a very good job here) then it will try and page out unused but allocated RAM from other processes - and it stores them in Swap. The intention for Linux is to have RAM that is "hot" and touched often, readily available. If you to a 'vmstat' or a 'free' command in a terminal you can see the amount of Swap being allocated. Personally, I use the more advance terminal monitoring called 'glances' (yum install or apt install glances) to monitor my Linux box.

But Swap is the domain of Linux, not the domain of APP (or technically in this instance, Java). And Swap is never a quick way to add more RAM to Linux, it is used to Swap out unused RAM and put it back when required.

You can set the "swappiness" of a Linux system (how sensitive the Linux system is to wanting to page out memory) by looking in /etc/sysctl.conf (as root):

My system has 32Gb RAM and I allocate 20Gb to APP. It leaves 12Gb for the system to do what it wants. That "system" - out of the box on Ubuntu 18.04 - sets swappiness to be at 60. Basically it likes to use Swap and I found it unnecessary. So I reduced to 1 (use it only when really needed).

vm.swappiness=1

Then do (as root or sudo):

sysctl -p

To reread and set the swappiness. In your case you want it to be more sensitive so you can view the value (as the defaults are not usually present in /etc/sysctl.conf so if you don't have that value it is set to the default):

cat /proc/sys/vm/swappiness

However, it sounds like you need more RAM or you're referencing some "scratch disk" functionality and not actual RAM?

Hope this helps!

Kev

@itarchitectkev

You just totally miss my point according swap in use with APP and what was behind this. Look on DSS how it use "swap" files for his needs. I think then you'll understand my point.

however, I already bought more RAM and now it's OK 😉

I googled "DeepSkyStacker swap" and I couldn't find what you mean. If it is "temporary files used during the processing of  DSS" then you set that with the Working Directory in APP. Ensure you have enough space here as temporary and processed files live here.

If not, you then go on to say you want to use Swap file when RAM is unavailable. Even DSS doesn't do this. If RAM is exhausted, RAM is exhausted. If the OS can't find dirty ram pages (unused ram) to swap out so the current process can utilise the memory - it's an OOM error. The OS (even Windows) decides this, not APP and not DSS.

As this is a Linux thread, in Linux a process cannot choose whether it can be swapped out to disk or not. The VMM system handles that. And in the event you're talking about temporary files used for calculations - check the files produced in the Working Directory which will be the equivalent.

But I explained that Java manages its own memory heap and APP adheres to this and that behind the scenes the OS then sees something quite opaque but the pages of RAM can still be swap in and out. To you, APP is "using swap". And it will use this if the OS decides it needs/wants to swap out parts of RAM to disk. I'm not familiar with DSS innards. What DSS is doing? I couldn't google "deepskystacker swap configuration" as it turned up blank. As it runs on the OS and not in a "VM" (JVM in the case of APP) or "Container" is that it, I assume, it is trying to utilise its own memory management system to keep its own memory usage down. Of course you can unload a file, and clear memory - but it needs to be loaded again at some point - but in your working area are temporary files produced that workaround some of these things - is that the swapping you want to see?

My point you missed is that you need to either configure Java's garbage collection/heap to do this or the OS. Not APP. I don't see it making sense in this context. Given DSS runs on Windows it's a different ball-game in how memory management works - although roughly the same principles apply: dirty pages of ram are swapped out. How you desire this to work or a program tries to keep memory utilisation down when it isn't shrouded in a VM like Java that already has routines to manage memory, I would assume is the wild-west.

Good luck, and clear skies!

Kev