[Solved] Ridiculous amount of disk space requested on beta39

How are you doing the registration? And are you using the new beta?

How many stars are you using?

@digitaliz-se All settings are at default AFAIK. The RAM warning also appears when I try to save normalised frames for further investigation. Each normalised saved frame is 7.4GBytes starting from a FITS of < 35MBytes !!

Ouch. Tried to uninstall and reinstall? Try another version as well to see if the problem persists.

@digitaliz-se beta32 is OK on this dataset. I thought you ought to know in case beta39 is to be the base for 2.0.0 release.

Posted by: @jonathankimmitt

↑

Ridiculous amount of disk space requested for 727 light frames normal registration mode (also complains about running out of RAM on 32GB Linux machine with 128GB of swap space)

It seems that there is not enough free space on this harddisk partition for APP to perform it's task.

Please choose what to do now:

1. I will try to clear space on my harddrive now before trying to continue.
2. Let APP try to test the actual available space.

Sometimes, an Operating System is not transparant about the actual free space, especially Macos.

So please try this option, if you think that APP is not detecting the free space correctly.

1. just ignore this warning and try to continue...
2. cancel this Integration task.

Solutions could be to choose

• another Work Directory on another harddisk or
• reduce the integration scale factor before starting a new integration task.

free space left in GBs: 893.3

harddisk space required for this task in GBs: 5353.8

@jonathankimmitt,

I see that the integration task needs:

harddisk space required for this task in GBs: 5353.8

so that is 5 TeraByte 

and per file like you indicate, 7,4 GigaByte is needed. That is very big indeed.

This can only happen when processing either a gigantic mosaic consisting of many mosaic panels or you are trying to drizzle 1000s of images with a scale factor bigger than 1.0, for instance 2.0 or 3.0 ?

I have calculated and have explained this in the manual that is nearly finished. Please have a look at this screenshot from the soon to be released manual:

If you integrate 2000 images of 50MegaPixels each with Drizzle 3x, then the task will need 5,4 TeraByte for instance.

So the question here is:

• where you creating a very big mosaic?
• Or are you using drizzle with 1000s of images, or perhaps,
• you set the scale factor in 6) integrate even higher than 3.0x?

Somehow, this must explain it? Let me know if this clarifies it or not? I am slightly suspecting that you are using a very big scale factor here without realizing it?

Mabula

i’m not doing any of those things, it is just a bug in the latest beta, a previous beta works fine on the same dataset.

Hi @jonathankimmitt, okay, so you are seeing this enormous data usage reported consistenly with beta39? All seems to be normal when I run it.

Can you check in the integration console what is reported once it starts with loading the images, like:

17:36:46 -
17:36:46 - 6) INTEGRATE: integrate light frames: loading frames of integration ...
17:36:46 - 6) INTEGRATE: integrate light frames: loading 1st frame
17:36:53 - 6) INTEGRATE: integrate light frames: got frame details, setting up integration task...
17:36:53 - 6) INTEGRATE: integrate light frames: got size of 1 frame: 2355 MegaBytes
17:36:53 - 6) INTEGRATE: integrate light frames: got datasize (# of pixels * # of channels) of 1 frame: 589 Mega Pixels
17:36:53 - 6) INTEGRATE: integrate light frames: using read buffer per thread of 32 KiloBytes
17:36:53 - 6) INTEGRATE: integrate light frames: using total read buffer of 512 KiloBytes
17:36:53 - 6) INTEGRATE: integrate light frames: integration buffer consumes 54 MegaBytes of RAM memory

That might give a clue about what is wrong?

I just ran a 3x3 mosaic that produces a mosaic with 17544x11731 pixels for RGB=3x = 600 Mega Pixels. The integration task with MBB needs 31.1 GB and that is consistent with calculating what would be needed. 

9 panels of 600 megaPixels

Each pixel is 32bits, so each pixel is 4 bytes.

600 megapixels = 2400 MegaBytes

2400 MegaBytes * 9 frames = 21,6 GigaByte

MBB was enabled that also need 0.5 of the data -> 32 GigaBytes needed.

So this seems 100% consistent with what is reported and what is calculated and also consistent with what was actually used which I tracked as well.

Maybe some setting in your APP configuration is different like the scale factor that I mentioned?

Mabula

I know you are sceptical about this being a bug. Have a look at these debug outputs that I generated, you can see something is seriously wrong. I am not using mosaic or non-default settings in Beta-39:

(base) jonathan@jonathan-nuc10i7fnk:/mnt/fits-backup/Pictures/IC_434/APP$ ls -l ../Light/session2|grep 1037
-rw-r--r-- 1 jonathan jonathan 37558080 Dec 14 01:36 IC_434_Light_1037.fits
(base) jonathan@jonathan-nuc10i7fnk:/mnt/fits-backup/Pictures/IC_434/APP$ ls -l ../Light/session2|grep 1087
-rw-r--r-- 1 jonathan jonathan 37558080 Dec 14 01:57 IC_434_Light_1087.fits
(base) jonathan@jonathan-nuc10i7fnk:/mnt/fits-backup/Pictures/IC_434/APP$ ls -l ../Light/session2|grep 1093
-rw-r--r-- 1 jonathan jonathan 37558080 Dec 14 01:59 IC_434_Light_1093.fits
(base) jonathan@jonathan-nuc10i7fnk:/mnt/fits-backup/Pictures/IC_434/APP$ ls -l ../Light/session2|grep 665
-rw-r--r-- 1 jonathan jonathan 37558080 Dec 13 22:41 IC_434_Light_665.fits
(base) jonathan@jonathan-nuc10i7fnk:/mnt/fits-backup/Pictures/IC_434/APP$ ls -l ../Light/session2|grep 832
-rw-r--r-- 1 jonathan jonathan 37558080 Dec 13 23:55 IC_434_Light_832.fits
(base) jonathan@jonathan-nuc10i7fnk:/mnt/fits-backup/Pictures/IC_434/APP$ ls -l ../Light/session2|grep 853
-rw-r--r-- 1 jonathan jonathan 37558080 Dec 14 00:04 IC_434_Light_853.fits
(base) jonathan@jonathan-nuc10i7fnk:/mnt/fits-backup/Pictures/IC_434/APP$ ls -l normalised/
total 46331640
-rw-r--r-- 1 jonathan jonathan 7907258880 Dec 16 17:04 IC_434_Light_1037-reg-norm.fits
-rw-r--r-- 1 jonathan jonathan 7907258880 Dec 16 17:05 IC_434_Light_1087-reg-norm.fits
-rw-r--r-- 1 jonathan jonathan 7907258880 Dec 16 17:02 IC_434_Light_1093-reg-norm.fits
-rw-r--r-- 1 jonathan jonathan 7907258880 Dec 16 17:04 IC_434_Light_665-reg-norm.fits
-rw-r--r-- 1 jonathan jonathan 7907258880 Dec 16 17:05 IC_434_Light_832-reg-norm.fits
-rw-r--r-- 1 jonathan jonathan 7907258880 Dec 16 17:03 IC_434_Light_853-reg-norm.fits

what is happening is something seriously wrong with registration, generating huge normalised files, a 37MB FITS should not expand 7.9GB after registration and normalisation. This is a bug in your latest beta and I do not want it to persist in the 2.0 release.

I reran the problematical integration and that bug has gone away (meanwhile I updated my system to the latest patch status), perhaps that had something to do with it. You can close this issue now.

Hi Jonathan @jonathankimmitt,

Thank you very much for all the feedback. From this:

I know you are sceptical about this being a bug. Have a look at these debug outputs that I generated, you can see something is seriously wrong. I am not using mosaic or non-default settings in Beta-39:

(base) jonathan@jonathan-nuc10i7fnk:/mnt/fits-backup/Pictures/IC_434/APP$ ls -l ../Light/session2|grep 1037
-rw-r--r-- 1 jonathan jonathan 37558080 Dec 14 01:36 IC_434_Light_1037.fits
(base) jonathan@jonathan-nuc10i7fnk:/mnt/fits-backup/Pictures/IC_434/APP$ ls -l ../Light/session2|grep 1087
-rw-r--r-- 1 jonathan jonathan 37558080 Dec 14 01:57 IC_434_Light_1087.fits
(base) jonathan@jonathan-nuc10i7fnk:/mnt/fits-backup/Pictures/IC_434/APP$ ls -l ../Light/session2|grep 1093
-rw-r--r-- 1 jonathan jonathan 37558080 Dec 14 01:59 IC_434_Light_1093.fits
(base) jonathan@jonathan-nuc10i7fnk:/mnt/fits-backup/Pictures/IC_434/APP$ ls -l ../Light/session2|grep 665
-rw-r--r-- 1 jonathan jonathan 37558080 Dec 13 22:41 IC_434_Light_665.fits
(base) jonathan@jonathan-nuc10i7fnk:/mnt/fits-backup/Pictures/IC_434/APP$ ls -l ../Light/session2|grep 832
-rw-r--r-- 1 jonathan jonathan 37558080 Dec 13 23:55 IC_434_Light_832.fits
(base) jonathan@jonathan-nuc10i7fnk:/mnt/fits-backup/Pictures/IC_434/APP$ ls -l ../Light/session2|grep 853
-rw-r--r-- 1 jonathan jonathan 37558080 Dec 14 00:04 IC_434_Light_853.fits
(base) jonathan@jonathan-nuc10i7fnk:/mnt/fits-backup/Pictures/IC_434/APP$ ls -l normalised/
total 46331640
-rw-r--r-- 1 jonathan jonathan 7907258880 Dec 16 17:04 IC_434_Light_1037-reg-norm.fits
-rw-r--r-- 1 jonathan jonathan 7907258880 Dec 16 17:05 IC_434_Light_1087-reg-norm.fits
-rw-r--r-- 1 jonathan jonathan 7907258880 Dec 16 17:02 IC_434_Light_1093-reg-norm.fits
-rw-r--r-- 1 jonathan jonathan 7907258880 Dec 16 17:04 IC_434_Light_665-reg-norm.fits
-rw-r--r-- 1 jonathan jonathan 7907258880 Dec 16 17:05 IC_434_Light_832-reg-norm.fits
-rw-r--r-- 1 jonathan jonathan 7907258880 Dec 16 17:03 IC_434_Light_853-reg-norm.fits

what is happening is something seriously wrong with registration, generating huge normalised files, a 37MB FITS should not expand 7.9GB after registration and normalisation. This is a bug in your latest beta and I do not want it to persist in the 2.0 release.

I can only conclude that the registration was way off for some reason. If registration fails, or if you register a big mosaic, in some cases, the result can blow up especially with pre-beta39 versions and that seems to be the only explanation.

I am glad to read that a later test performed normally. If the problem happens again though, please let me know and share the data and the registration settings with me so I can test and solve it. Thanks 😉

Mabula