15th Feb 2024: Astro Pixel Processor 2.0.0-beta29 released - macOS native File Chooser, macOS CMD-Q fixed, read-only Fits on network fixed and other bug fixes
7th December 2023: added payment option Alipay to purchase Astro Pixel Processor from China, Hong Kong, Macau, Taiwan, Korea, Japan and other countries where Alipay is used.
[Solved] Incorrect Available Storage in macOS -- Potential Bug
I experienced an issue with APP that makes it completely unusable on my Mac. APP refuses to integrate images because apparently I am running low on disk space, even though I have more than enough disk space remaining. Notice in the screenshot that I have close to 200GB space on my only hard drive (my working directory was in the same drive with at least 200GB space). I have the latest APP version 1.0174.1 & macOS Mojave 10.14.1, but I noticed this bug in several older releases of APP as well, and even on older macOS releases. The only way to get around this issue is to plug in an external flash drive with 32GB or more and setting the working/output directory to that flash drive, which makes the integration process painfully slow.
This topic was modified 5 years ago 3 times by rathijit
This topic was modified 5 years ago by Mabula-Admin
I've seen this before and it happens when you start APP from a location that is limited in space. For instance when you launch APP from within the installer image, you need to launch APP from the Applications folder (or that's where it's supposed to be).
As I had mentioned in my message, I made sure I was using the right working directory that had enough storage. I am 100% confident that the drive I was saving to (which happens to be my only drive) has about 200GB of available storage. This is more likely an issue in the way hard drive storage is calculated on a Mac, because I do not face the same issue on my PC.
@rathijit please read @vincent-mod ‘s comment again. He is not talking about the working directory, he is talking about the way APP gets started. Can you please confirm that you have installed APP in the /Applications directory and that you start it from there?
Sorry, yes, I can confirm that it is running from the /Applications directory in my Mac. It was installed and not just executed from a dmg image. It works fine usually... it only fails when I try to integrate 500+ subs.
Ah ok, so normally it does work, but with huge task it doesn't. That might be an actual bug then, I'll notify Mabula (@mabula-admin), thanks for mentioning it!
I will try to solve this issue, it has been mentioned before by other users and I am aware of it.
The issue is caused by MacOS showing free space, where in fact part of that space at that particular moment is reserved for the OS for certain purposes. So an external application will think that the space is not free at all. I need to figure out how I can detect that "reserved space" and compensate for it...
Okay, after more investigation, I am quite sure that these HDD space issues on MacOS are caused by the so-called purgeable free space that has been introduced by Apple since the release of MacOS Sierra.
MacOS is reserving space for iTunes files (music/movies) and Time Machine backups, iCloud files that are regularly accessed etc... this data is consuming HDD space and can be very significant ! but it is stated as purgeable if you check the hdd space with disc utility or About this Mac-> storage . So what is happening is that the space actually is not there since it is consumed, but it will be freed/purged once HDD space is dropping and space need to be freed...
Now Apple has not provided any commandline tool/command to be able to correct the free space on the drive with the purgeable amount as I am aware of now. You can only get this information using the Graphical User Interface tools in MacOS unfortunately...
There are 2 things we can do now so MacOS users with Sierra and later can still complete big integration tasks without being upset by big purgeable amounts in their MacOS environment.
I can disable the hdd space warning limitation as an option for MacOS users... but once you really consume all data, off course you will not get warned and the task will still fail... 😪
The user is made responsible and aware of MacOS purgeable space and should remove this before starting a big integration... this off course is not very user friendly and the user needs to get quite some instructions to get rid of the purgeable space (removing old Time Machine slots for instance are also in the purgeable space)
As it is right now, I am not happy with either option. I am also not very happy with Apple in this regard that they would implement this purgeable feauture without an easy way to get rid of this or correct for this with a simple command/query.
I will investigate further, but I suspect this issue comes down to the MacOS user needing to have a better understanding of their own MacOS environment so they can prepare their system if they need to do a big integration.
The following links discuss how you can rid of this purgeable space, even how to prevent it from becoming very big to start with. Personally, I don't use Time Machine, don't use iTunes, and I don't use iCloud and as a consequence my purgeable space is always very small so I never run into this issue. I am not saying that you shouldn't use these Apple MacOS features, but I am saying that these features on MacOS Sierra and later, in combination with the purgeable space feature, are actually causing this issue, so you can be aware of this for now, until we get a better solution 😉
As a work around for this I did the following to force the space to be cleared.
In the terminal I used the mkfile command to create a file large enought to completely consume the remaining free space on my drive.
"mkfile -n 60g ~/Desktop/BIGFILE"
Replace 60g with whatever file size you would like.
Once that command was finished I rebooted. I am not sure if this next step is necessary, but I didn't want deleting theses files to just create more purgable space so I used RM in the terminal.
"sudo RM ~/Desktop/BIGFile"
By doing this I was able to clear enough space for what I was wanting to do. I have also not noticed any unwanted side affects of this process and I have now done this multiple times for various reasons.
So create a very big temporary file on the drive that you are going to use for the APP work directory. This will force MacOS to completely remove the purgeable space and thus make it completely free for APP 😉
Kind regards,
Mabula
This post was modified 5 years ago by Mabula-Admin
I experienced an issue with APP that makes it completely unusable on my Mac. APP refuses to integrate images because apparently I am running low on disk space, even though I have more than enough disk space remaining. Notice in the screenshot that I have close to 200GB space on my only hard drive (my working directory was in the same drive with at least 200GB space). I have the latest APP version 1.0174.1 & macOS Mojave 10.14.1, but I noticed this bug in several older releases of APP as well, and even on older macOS releases. The only way to get around this issue is to plug in an external flash drive with 32GB or more and setting the working/output directory to that flash drive, which makes the integration process painfully slow.
The screenshot of Rathijit actually is indicative of MacOS not being transparant on the Operating System level about HDD space... only the GUI MacOS tools show this, I can not get this information from the command line which would be needed to solve this properly I am afraid. In Rathijit's case here, maybe another MacOS feature is consuming a lot of space allthough About my Mac -> Storage is not showing this... Maybe this is caused by Time Machine? Rathijit, do you use any of the following MacOS features:
Yes, I use all 3. If it is an issue with unreported purgeable space, then I would prefer to be given an option to "Continue anyway" with a warning message for macOS users (maybe even as an advanced option under APP settings). This way, novice users need not worry about this option, and advanced users who understand this issue can get around it if they know that they have enough storage space.
By the way, I had the same suspicion and wrote a very large file (larger than the required space), and that got around the issue. Maybe APP can attempt to write a large file on macOS?
Yes, I am adjusting it now, so that you will still get warned, but that you can still elect to continue 😉 So the integration will not be cancelled unless you prefer to cancel it.
I will try the option of writing a large file on MacOS to see if that is a viable option to get a more accurate indication of free space.
Okay, I have implemented a solution now that will be part of the next release 😉 The solution has been tested on all platforms (Windows, MacOS & Linux).
When APP thinks that there is not enough space for an integration task, APP will no longer simply cancel the task, but it will give the user 4 choices 😉
Before continuing with the integration, you can free HDD space yourself and then continue.
Let APP test the actual HDD space requirement by creating a temporary file of 105% of the space needed for the integration. This goes very fast and it will actually test if the integration would be possible... this is the solution for APP users that are plagued by the purgeable free space on MacOS.
just ignore the warning and let APP continue...
cancel the integration task
I think that this will prove a good solution for now. It should definitely enable any user to continue with the integration although APP would think it would not be possible initially 😉
This is the pop-up that will show if the problem occurs, (please disregard the actual numbers for freespace in the window, it's a popup triggered by testing this)
I too have this issue. But also another one where the application runs out of memory (RAM) during the integration. The major issue I have with encountering both these bugs is that they don't appear until integration has started. I'm trying to process 1,140 APSC sized OSC light frames. I have to do 60s subs to keep under my light pollution ceiling ADU value, so I have lots of subs.
Well, during processing, I don't get to the integration dialog errors until my computer (2017 iMac Pro, 32GB ram, 8 core Xeon) has processed the images for 5 hours to register, calibrate, and normalize. So after spending 5 hours doing this process, I got the above error about not enough disk space, despite having 600GB of free space. So then, I cleared more space, rebooted, and tried again for another 5 hours, only to have it fail with the same error. So then I moved my files to my external RAID array where I had 3 TB free space, and I got all the way back another 5 hours to the message above about not enough ram despite allocating 24GB of ram to the application. I can allocate up to 30GB, but have to wait another 5 hours to see if it works. So all this processing time wastes around 20 hours now just to get an error message, and have to start completely over again because no progress is saved during processing.
What are some things that can be done to address these issues? Can progress be saved after each step of the process (calibrate, analyst stars, register, normalize)? Can a check be run prior to starting a long integration job to know if memory or hard drive space will not be enough? Am I not using the application properly? I'm integrating 3 sessions with around 400 images each.
I too have this issue. But also another one where the application runs out of memory (RAM) during the integration. The major issue I have with encountering both these bugs is that they don't appear until integration has started. I'm trying to process 1,140 APSC sized OSC light frames. I have to do 60s subs to keep under my light pollution ceiling ADU value, so I have lots of subs.
Well, during processing, I don't get to the integration dialog errors until my computer (2017 iMac Pro, 32GB ram, 8 core Xeon) has processed the images for 5 hours to register, calibrate, and normalize. So after spending 5 hours doing this process, I got the above error about not enough disk space, despite having 600GB of free space. So then, I cleared more space, rebooted, and tried again for another 5 hours, only to have it fail with the same error. So then I moved my files to my external RAID array where I had 3 TB free space, and I got all the way back another 5 hours to the message above about not enough ram despite allocating 24GB of ram to the application. I can allocate up to 30GB, but have to wait another 5 hours to see if it works. So all this processing time wastes around 20 hours now just to get an error message, and have to start completely over again because no progress is saved during processing.
What are some things that can be done to address these issues? Can progress be saved after each step of the process (calibrate, analyst stars, register, normalize)? Can a check be run prior to starting a long integration job to know if memory or hard drive space will not be enough? Am I not using the application properly? I'm integrating 3 sessions with around 400 images each.
@mabula-admin Btw, I have also run into the RAM issue before on large jobs, but my system is not as beefy as @lead_weight 's system. So I have wondered the same thing: Can progress be saved? I sometimes register & normalize after each night's data collection, so it would be nice to not have to re-register, re-normalize the whole data set each time and only do so for the new images (I understand the best reference frame may not be selected if that is the case, but perhaps there is a way around this?). Even if I add one subframe, the whole calibration, registration, normalization process starts from scratch, wasting a lot of time.
@lead_weight To get around my RAM limit issue, I have stacked each color channel separately and combined them later. Hope that works for you as well.
I’m stacking 3 sets of color data, not RGB separately. I could do one stack at a time, but I want it to remove the 60 worst frames from all stacks at once, not each stack individually, since most of the poor images could be in a single stack.
I was finally able to complete the process stacking 1080 APSC sized OSC subs. I started around 11pm last night, and it finished by 9am this morning. 10 hours to process it. In order to complete it, I had to move the subs to my external drive where I had 3TB of free space, and I set RAM use to 30GB, up from 24. This seemed to get through all the hiccups.
I'm really looking forward to GPU processing to see how much difference it makes for a set of images like this. The CPU processing "feels" quite slow with all this data. But I still think Astro Pixel Processor is the best system for handling all this data. It would have been a nightmare working with all the flats, bias, darks, and lights from three sessions, quality grading all of the subs, and trying to keep sane.
