APP 1.081 on ubuntu crashes randomly

Posted by: @shlomi

Hi,

I installed version 1.081 on my ubuntu box today, attempted to integrate a small session I did last night. APP is randomly getting killed while working. It happened a few times, the last time I started it from the console to try some logs. If there are more logs somewhere, please let me know where I can find them.

 

/opt/AstroPixelProcessor> ./AstroPixelProcessor 
Jun 27, 2020 12:31:34 PM it.geosolutions.imageio.gdalframework.GDALUtilities loadGDAL
INFO: Failed to load the GDAL native libs from "gdalalljni". Falling back to "gdaljni".
java.lang.UnsatisfiedLinkError: no gdalalljni in java.library.path
Jun 27, 2020 12:31:34 PM it.geosolutions.imageio.gdalframework.GDALUtilities loadGDAL
WARNING: Failed to load the GDAL native libs. This is not a problem unless you need to use the GDAL plugins: they won't be enabled.
java.lang.UnsatisfiedLinkError: no gdaljni in java.library.path
Jun 27, 2020 12:31:34 PM it.geosolutions.imageio.plugins.turbojpeg.TurboJpegUtilities loadTurboJpeg
WARNING: Failed to load the TurboJpeg native libs. This is not a problem, but the TurboJpeg encoder won't be available: java.lang.UnsatisfiedLinkError: no turbojpeg in java.library.path
Jun 27, 2020 12:31:34 PM it.geosolutions.util.KakaduUtilities loadKakadu
WARNING: Failed to load the Kakadu native libs. This is not a problem unless you need to use the Kakadu plugin: it won't be enabled. java.lang.UnsatisfiedLinkError: no kdu_jni in java.library.path




[1] 1411480 killed ./AstroPixelProcessor

I was away from the computer each time this happened, so we can rule out accidental killing by the user. As it is, I cannot use the software.. 🙁

 

Please help!

Thanks,

Shlomi

Dear Shlomi @shlomi,

If APP would crash with a fatal bug, it would have shown on the console/terminal with a fatal crash report. So to me, this does not seem like a fatal crash in APP.

The output 1411480 killed ./AstroProcessor must be coming from another process on your Ubuntu installation. I have never seen this happening myself I must admit.

I think you need to find out which process is responsible and why. APP would never kill itself off course, so the cause is in your Ubuntu installation somehow I think.

Did you experience this with other APP versions on the same Ubuntu installation as well? Or is this now only occuring with 1.081 ? Or did you change/upgrade your Ubuntu installation recently?

Mabula

Hi, 

This only happens with 1.081. Regarding the crash - its unlikely that any other process kills just this APP process. It is likely however, that some fault is not caught and therefor does not print anything, especially if faults lie on the java boundary layer (jni, opengl, etc..). Are there any other logs I could look into? 

It is extremely unlikely that some other process is killing this process. What I can try to do, is make a brand new user to run this process - other processes (unless root), will not be able to kill it.

Just started this process again, trying to record the screen. Maybe the APP built-in console did show something that was not shown from the bash console (does not show anything really..). 

If there's a way to redirect the built-in APP console to file it would be really useful in such cases.. 

Will update here when I get more info.

Ok, so my first attempt ended up being successful. I tried again, from scratch,  and sure enough, it happened again. Please see

, around 5:30 is where things begin to misbehave. It seems like things stop to happen, the log stops moving until it all suddenly vanishes. 

The last log which gets stuck for a while is "re-instatiating multi-core analytical memory blocks":

Hope this is helpful!

Hi @shlomi,

Thank you for the video, that really helps 😉

The video actually shows us exactly the problem that you have and it confirms that the issue is not in APP, but it is your Ubuntu installation configuration.

Have a good look at what APP is showing you for RAM consumption, your video starts with values of about:

RAM APP 40/14336  OS 12677/15936

This means that APP can use 14GB of Ram out of 16GB of RAM in your system, right?

With these values, APP is only using 40 MegaBytes of RAM and the OS (Ubuntu) is already using 12667 MegaBytes = 12,5 GigaByte of RAM !

Simple math tells us know that unless the OS frees some memory, APP will only have 3,3 GB of RAM to play with at this moment... right?

Now have a very good look at what APP is reporting for the OS RAM usage when things start to break down at 6 minutes into your video:

RAM APP 5064/14336  OS 15807/15936

 At this point while analysing the created MasterFlat APP is killed ! and it is killed by the Ubuntu Watchdog to keep the Operating System stable. Your Operating System is going Out Of Memory at this point and Ubuntu purposely kills the process that is about to trigger the Out Of Memory problem... this then tells me that either Ubuntu is not allowed to use the Linux Swap partition or there isn't even a Linux Swap partition installed in your Ubuntu installation? Please check if you have a Linux Swap partition and if so, how large is it?

If there is a Linux swap partition and if it is big enough, Ubuntu would have no need to kill APP, because it can prevent Out of Memory on Operating System level by swapping the processes that are consuming the memory....

Furthermore, your video also shows us, that other processes on your installation already are consuming 14GB of RAM, when you start APP. If you would stop some of those processes then APP is most likely not killed as well when working 😉

So, for me, the root cause of your issue is either

• a missing Linux Swap partition
• a too small Linux Swap partition
• a Linux swap partition that is not being used

And what you can also try is the following, if you don't want to stop the other processes that are already consuming 14GBs of RAM, simply limit APP's RAM usage, by configuring APP to use only 4GB of RAM for example with the CFG button. Then APP will not grow beyond 4GB of RAM and can probably still process your data without going Out Of Memory on the application level 😉

Kind regards,

Mabula

https://www.astropixelprocessor.com/community/postid/10201/

Hello Mabula, maybe there is a connection. 🤔 

Hi @mabula,

Wonderful analysis, thank you so much! completely forgot about the dreaded linux OOM killer (aka watchdog) :facepalm:

You are correct, and it can be verified by the system logs:

/opt/AstroPixelProcessor > dmesg | grep -i "Killed process"
[407907.510428] Out of memory: Killed process 1414667 (QtWebEngineProc) total-vm:1336044kB, anon-rss:14864kB, file-rss:0kB, shmem-rss:32kB, UID:1000 pgtables:872kB oom_score_adj:300
[407919.709709] Out of memory: Killed process 1414668 (QtWebEngineProc) total-vm:1319296kB, anon-rss:3708kB, file-rss:0kB, shmem-rss:12kB, UID:1000 pgtables:756kB oom_score_adj:300
[407922.609468] Out of memory: Killed process 1414700 (QtWebEngineProc) total-vm:1319296kB, anon-rss:3712kB, file-rss:0kB, shmem-rss:12kB, UID:1000 pgtables:752kB oom_score_adj:300
[407923.180589] Out of memory: Killed process 1422254 (AstroPixelProce) total-vm:21149784kB, anon-rss:4374000kB, file-rss:0kB, shmem-rss:2160kB, UID:1000 pgtables:9492kB oom_score_adj:0

These we can see my killed APP PID from the video above..So ungraceful!

Glad the culprit is found, thank you!!

Shlomi