15th Feb 2024: Astro Pixel Processor 2.0.0-beta29 released - macOS native File Chooser, macOS CMD-Q fixed, read-only Fits on network fixed and other bug fixes
7th December 2023: added payment option Alipay to purchase Astro Pixel Processor from China, Hong Kong, Macau, Taiwan, Korea, Japan and other countries where Alipay is used.
[Solved] Integration Write Fails - File Type Unsupported
I am having an issue I have not seen before. I suspect it could be data and am trying more testing, but wanted to see if anyone has experienced something similar.
I am processing just under 500 raw images over 12 sessions. Each session is using common Master Bias and BPM, but it’s own master dark and flat. I create the masters in a separate session and then pull the master files in when I run steps 2-6. It goes through all the steps and starts writing the file when it gets to 100% it posts an error, and no file is saved in the working directory
”this file type is unsupported or locked by another process”
I am using MacOS Mojave on a Mac Pro. the machine has been rebooted and is running only APP. I previously processed the same data minus the last 2 sessions and it worked properly, which is why I suspect data. I have tried processing 3 times with the same result. I’m going to rerun without the last 2 sessions to see if it will work properly again. At approx 9~10 hours to process it’s a little frustrating
Do you mean, that the whole integration of step 6) finishes? And it errors on actually saving the final integration?
What does the console panel exactly show when it occurs?
The mentioned error message:
”this file type is unsupported or locked by another process”
is related to file loading and not to saving, which makes this a bit odd... the output of the console panel with all details will really help to determine what is happening.
That is correct, the whole integration (Step 6) completes. The error is on saving the final integration. I was confused by the behavior as well.
I had repeated the attempt 3 times with the same result. I then removed the last days data and tried; it worked as expected. So then I went back ind integrated only the last day of data, and that was successful.
Right now I have another integration attempt using all the data (Same as failed previously) to see if it fails again. I will post the result once it completes. If I get the failure I will grab the console output. (I did not think to capture that the previous fails...).
More information. I repeated with all the sessions and it failed again. I apologize for the crudeness, but I took the photos below to get the information quickly.
Sessions being processed: (When I added Oct 5 it failed, Only Oct 5 processes fine).
Mm, this is a very very long filename, I don't think that should be an issue but it's something that jumps out, I never saw that. Is it possible to try to shorten that?
I hadn't thought about file name length. I've just been letting APP use whatever it defaults to. Since there are so many separate sessions it is creating a long one.
Is there a way to pre-select a filename before integration? By the time I get the error the only option is OK, and no files are in the working directory.
I have no idea if this would be the problem, I kind of doubt it even, but otherwise I wouldn't know what would cause it. Maybe @mabula-admin has an idea?
I see this issue as well (both on Windows and Mac OS X). I am convinced it's because of the long file names. I have more than 10 sessions on some filters and once all of that is added into the filename, I believe we might be hitting the limit (PATH_MAX on Mac for e.g.). Is there any way to customize the filename? A lot of the information that's in the file name, is possibly also in the fits headers (or can be dumped into the fits header instead) and the file name made shorter. It's a bit unwieldy right now.
Just to add some more data points, My OIII integration is the one that's failing. SII and Ha work fine.
OIII file name is 124 characters. Path Name is 148 characters
SII file name is 117 characters. Path Name is 141 characters
Ha file name is 118 characters. Path Name is 142 characters
So those additional 6 characters in the OIII file/path name is some how causing the program to fail saving it. This is on Mac OS X, where the file name limit is supposed to be 255 chars and the path name soft limit is supposed to 1024 chars.
I can't really do that because APP is failing to write the file to disk. So there is nothing there to change the file name of. A few more things I tried:
- Manually ran "touch" on the file name that APP would end up with for the OIII integration (created a 0 byte file with the name). When integration was complete for OIII APP did ask me if I want to overwrite the file. Saying yes, still resulted in APP failing to write the fail claiming unsupported or locked image file. This tells me it's not a file system access issue
- Removed one sessions from the integration stack for OIII, this essentially shortened the file name by 2 chars (Full filename 122 chars). This time APP succeeded in writing the file. So did removing two or three sessions (shortening the file name to 120 and 118 chars)
So it looks like there is some limit in the software where if the filename is >= 123/124 chars (not sure which is the limit), it can't save the file. This can't be a Java limit either because it has no such limits.
Same issue I'm having. The error occurs before the option comes up to change the filename. When the error is dismissed the work is gone (No files in working folder). It only happens when I have many sessions, causing the long file name.
Just for kicks, I tried reducing the number of chars in the file name by changing the filter tag instead of removing some sessions. This way I brought the number of chars in the file name down to 121 chars, and......it still failed! even though earlier it had saved the file fine with 122 chars!
That got me thinking.....I don't think it's the number of chars in the filename. Looking at my stacks, OIII which is failing has the largest number of sessions, 10 to be exact. If I reduce this to 9 everything is fine.
So it looks like there is a bug/limit in APP where if the number of sessions is >= 10 it can't save the stack!! Please look into it?
Also while you are at it, a few requests:
- Save information in the FITS header through custom tags. Putting them in the file name seems inefficient and the info is lost if someone changes the filename
- For sessions add the capability to use capture date as the session (instead of choosing "session1" etc. or some other custom thing). Most FITS, if not all will have the capture date. This way they can be grouped fairly easily
I'm having the same problem. I'm using the latest version. I tried 8 times so far with no luck. Can you guys please look into this. I don't have the best computer and it's taking 8 hours to do a 200 file stack from 11 sessions and I can't really do anything else on the computer while stacking.
I don't know if it's gonna work but tonight I'm planning to try it again but instead of naming the sessions from 6 to 11, session 6 and so on, I will only write s6 to s11. Maybe this is happening because the name of the finak stack is too long, and maybe, with s6.. s11 instead of session 6... session 11 it will manage to save the file.
Do you mean, that the whole integration of step 6) finishes? And it errors on actually saving the final integration?
What does the console panel exactly show when it occurs?
The mentioned error message:
”this file type is unsupported or locked by another process”
is related to file loading and not to saving, which makes this a bit odd... the output of the console panel with all details will really help to determine what is happening.
I am working to fix this issue, please accept my apologies, it is taking some time.
I will be storing all metadata of the integrations in the FITS header, so there will no longer be need to keep all details in the integration file name, which will prevent it from becoming so long. That should be the best solution.
I will make sure it's fixed in the next release and I will release a beta with a fix as soon as possible, I think it will be available in a couple of days 😉
Thank you but are you sure the reason is the name being too long? I shortened the name considerably when I replaced the session witha simple s and I still got the error. Maybe it's because it can't stack that many sessions.
I can confirm that this is not related to the name itself (although saving this data in metadata as opposed to file name is the way to go). Please look at my detailed experiments from before (same thread).
The issue is related to the number of sessions itself. There seems to be a hardcoded limit somewhere of 9. If you have >= 10 sessions the integration will fail (consistent across platforms as well. Have tried Mac and Windows).
I have solved this issue. Indeed, the error was not related to the length of the file name as suspected. The problem was that a Fits meta header tag become longer than 8 characters and this is not allowed.
It's the SESSIO-x tag, with more than10 sessions, that tag could become SESSIO-11 which is 9 characters long, this is not allowed by FITS conventions and that caused the saving to fail. The tag will now become SESS-11 😉
So in APP 1.078 you will be able to get past the limitation of combing 10 sessions:
In the screenshot you will also see new header tags, all integration settings are now there 😉
So I will now reduce the integration filenames completely and will not put the integration settings in the name anymore. So the integration file names will be simple and short from now on as well, because all relevant information will be in the FITS header.
I have solved this issue. Indeed, the error was not related to the length of the file name as suspected. The problem was that a Fits meta header tag become longer than 8 characters and this is not allowed.
It's the SESSIO-x tag, with more than10 sessions, that tag could become SESSIO-11 which is 9 characters long, this is not allowed by FITS conventions and that caused the saving to fail. The tag will now become SESS-11 😉
So in APP 1.078 you will be able to get past the limitation of combing 10 sessions:
In the screenshot you will also see new header tags, all integration settings are now there 😉
So I will now reduce the integration filenames completely and will not put the integration settings in the name anymore. So the integration file names will be simple and short from now on as well, because all relevant information will be in the FITS header.
Kind regards,
Mabula
That sounds great, thank you. I have a few more targets with more than 10 sessions and I was waiting for this fix. 👍👍
Thank you Mabula! With this we should be able to go up to 99 sessions. I doubt anyone is going beyond that. If they are you could just change it to “S-“ 🙂
