Please come all to AstroFest in London to ask us (Mabula & Vincent) questions and to see live demos of APP!
2023-01-19: APP 2.0.0-beta13 has been released !
!!! Big performance increase due to optimizations in integration !!!
and upgraded development platform to GraalVM 22.3 based on openJDK19
We are very close now to releasing APP 2.0.0 stable with a complete printable manual...
[Solved] Integration Write Fails - File Type Unsupported
I am having an issue I have not seen before. I suspect it could be data and am trying more testing, but wanted to see if anyone has experienced something similar.
I am processing just under 500 raw images over 12 sessions. Each session is using common Master Bias and BPM, but it’s own master dark and flat. I create the masters in a separate session and then pull the master files in when I run steps 2-6. It goes through all the steps and starts writing the file when it gets to 100% it posts an error, and no file is saved in the working directory
”this file type is unsupported or locked by another process”
I am using MacOS Mojave on a Mac Pro. the machine has been rebooted and is running only APP. I previously processed the same data minus the last 2 sessions and it worked properly, which is why I suspect data. I have tried processing 3 times with the same result. I’m going to rerun without the last 2 sessions to see if it will work properly again. At approx 9~10 hours to process it’s a little frustrating
Any thoughts / Suggestions?
Do you mean, that the whole integration of step 6) finishes? And it errors on actually saving the final integration?
What does the console panel exactly show when it occurs?
The mentioned error message:
”this file type is unsupported or locked by another process”
is related to file loading and not to saving, which makes this a bit odd... the output of the console panel with all details will really help to determine what is happening.
That is correct, the whole integration (Step 6) completes. The error is on saving the final integration. I was confused by the behavior as well.
I had repeated the attempt 3 times with the same result. I then removed the last days data and tried; it worked as expected. So then I went back ind integrated only the last day of data, and that was successful.
Right now I have another integration attempt using all the data (Same as failed previously) to see if it fails again. I will post the result once it completes. If I get the failure I will grab the console output. (I did not think to capture that the previous fails...).
It really is an amazing processing tool.
More information. I repeated with all the sessions and it failed again. I apologize for the crudeness, but I took the photos below to get the information quickly.
Sessions being processed: (When I added Oct 5 it failed, Only Oct 5 processes fine).
Error after it completes integration:
Mm, this is a very very long filename, I don't think that should be an issue but it's something that jumps out, I never saw that. Is it possible to try to shorten that?
I hadn't thought about file name length. I've just been letting APP use whatever it defaults to. Since there are so many separate sessions it is creating a long one.
Is there a way to pre-select a filename before integration? By the time I get the error the only option is OK, and no files are in the working directory.
I have no idea if this would be the problem, I kind of doubt it even, but otherwise I wouldn't know what would cause it. Maybe @mabula-admin has an idea?
I see this issue as well (both on Windows and Mac OS X). I am convinced it's because of the long file names. I have more than 10 sessions on some filters and once all of that is added into the filename, I believe we might be hitting the limit (PATH_MAX on Mac for e.g.). Is there any way to customize the filename? A lot of the information that's in the file name, is possibly also in the fits headers (or can be dumped into the fits header instead) and the file name made shorter. It's a bit unwieldy right now.
I've asked Mabula to have a look at this issue, hopefully he can answer shortly. Thanks for the report!
Just to add some more data points, My OIII integration is the one that's failing. SII and Ha work fine.
OIII file name is 124 characters. Path Name is 148 characters
SII file name is 117 characters. Path Name is 141 characters
Ha file name is 118 characters. Path Name is 142 characters
So those additional 6 characters in the OIII file/path name is some how causing the program to fail saving it. This is on Mac OS X, where the file name limit is supposed to be 255 chars and the path name soft limit is supposed to 1024 chars.
Does it work when you manually shorten the name in the finder?
I can't really do that because APP is failing to write the file to disk. So there is nothing there to change the file name of. A few more things I tried:
- Manually ran "touch" on the file name that APP would end up with for the OIII integration (created a 0 byte file with the name). When integration was complete for OIII APP did ask me if I want to overwrite the file. Saying yes, still resulted in APP failing to write the fail claiming unsupported or locked image file. This tells me it's not a file system access issue
- Removed one sessions from the integration stack for OIII, this essentially shortened the file name by 2 chars (Full filename 122 chars). This time APP succeeded in writing the file. So did removing two or three sessions (shortening the file name to 120 and 118 chars)
So it looks like there is some limit in the software where if the filename is >= 123/124 chars (not sure which is the limit), it can't save the file. This can't be a Java limit either because it has no such limits.
Same issue I'm having. The error occurs before the option comes up to change the filename. When the error is dismissed the work is gone (No files in working folder). It only happens when I have many sessions, causing the long file name.
Wanted to add some more data points.
Just for kicks, I tried reducing the number of chars in the file name by changing the filter tag instead of removing some sessions. This way I brought the number of chars in the file name down to 121 chars, and......it still failed! even though earlier it had saved the file fine with 122 chars!
That got me thinking.....I don't think it's the number of chars in the filename. Looking at my stacks, OIII which is failing has the largest number of sessions, 10 to be exact. If I reduce this to 9 everything is fine.
So it looks like there is a bug/limit in APP where if the number of sessions is >= 10 it can't save the stack!! Please look into it?
Also while you are at it, a few requests:
- Save information in the FITS header through custom tags. Putting them in the file name seems inefficient and the info is lost if someone changes the filename
- For sessions add the capability to use capture date as the session (instead of choosing "session1" etc. or some other custom thing). Most FITS, if not all will have the capture date. This way they can be grouped fairly easily
Thanks so much for the deep dive into the issue Manoj! I will pass this on to Mabula right now.
Any update on this? I am unable to process my captures right now because of this limitation of sessions to 9 sessions.
Not yet sorry, I haven't been able to contact Mabula for a few days now.. he was/is abroad so it might be he's travelling. Will let you know.
I'm having the same problem. I'm using the latest version. I tried 8 times so far with no luck. Can you guys please look into this. I don't have the best computer and it's taking 8 hours to do a 200 file stack from 11 sessions and I can't really do anything else on the computer while stacking.
I don't know if it's gonna work but tonight I'm planning to try it again but instead of naming the sessions from 6 to 11, session 6 and so on, I will only write s6 to s11. Maybe this is happening because the name of the finak stack is too long, and maybe, with s6.. s11 instead of session 6... session 11 it will manage to save the file.
I changed the session names to sX instead of the sessionX to see if it's because the name is too long and unfortunately it didn't work