2019 September: Astro Pixel Processor and iTelescope.net celebrate a new Partnership!
Integration Write Fails - File Type Unsupported
I am having an issue I have not seen before. I suspect it could be data and am trying more testing, but wanted to see if anyone has experienced something similar.
I am processing just under 500 raw images over 12 sessions. Each session is using common Master Bias and BPM, but it’s own master dark and flat. I create the masters in a separate session and then pull the master files in when I run steps 2-6. It goes through all the steps and starts writing the file when it gets to 100% it posts an error, and no file is saved in the working directory
”this file type is unsupported or locked by another process”
I am using MacOS Mojave on a Mac Pro. the machine has been rebooted and is running only APP. I previously processed the same data minus the last 2 sessions and it worked properly, which is why I suspect data. I have tried processing 3 times with the same result. I’m going to rerun without the last 2 sessions to see if it will work properly again. At approx 9~10 hours to process it’s a little frustrating
Any thoughts / Suggestions?
Do you mean, that the whole integration of step 6) finishes? And it errors on actually saving the final integration?
What does the console panel exactly show when it occurs?
The mentioned error message:
”this file type is unsupported or locked by another process”
is related to file loading and not to saving, which makes this a bit odd... the output of the console panel with all details will really help to determine what is happening.
Main developer of Astro Pixel Processor and owner of Aries Productions
That is correct, the whole integration (Step 6) completes. The error is on saving the final integration. I was confused by the behavior as well.
I had repeated the attempt 3 times with the same result. I then removed the last days data and tried; it worked as expected. So then I went back ind integrated only the last day of data, and that was successful.
Right now I have another integration attempt using all the data (Same as failed previously) to see if it fails again. I will post the result once it completes. If I get the failure I will grab the console output. (I did not think to capture that the previous fails...).
It really is an amazing processing tool.
More information. I repeated with all the sessions and it failed again. I apologize for the crudeness, but I took the photos below to get the information quickly.
Sessions being processed: (When I added Oct 5 it failed, Only Oct 5 processes fine).
Error after it completes integration:
I hadn't thought about file name length. I've just been letting APP use whatever it defaults to. Since there are so many separate sessions it is creating a long one.
Is there a way to pre-select a filename before integration? By the time I get the error the only option is OK, and no files are in the working directory.
I have no idea if this would be the problem, I kind of doubt it even, but otherwise I wouldn't know what would cause it. Maybe @mabula-admin has an idea?
I see this issue as well (both on Windows and Mac OS X). I am convinced it's because of the long file names. I have more than 10 sessions on some filters and once all of that is added into the filename, I believe we might be hitting the limit (PATH_MAX on Mac for e.g.). Is there any way to customize the filename? A lot of the information that's in the file name, is possibly also in the fits headers (or can be dumped into the fits header instead) and the file name made shorter. It's a bit unwieldy right now.
Just to add some more data points, My OIII integration is the one that's failing. SII and Ha work fine.
OIII file name is 124 characters. Path Name is 148 characters
SII file name is 117 characters. Path Name is 141 characters
Ha file name is 118 characters. Path Name is 142 characters
So those additional 6 characters in the OIII file/path name is some how causing the program to fail saving it. This is on Mac OS X, where the file name limit is supposed to be 255 chars and the path name soft limit is supposed to 1024 chars.
I can't really do that because APP is failing to write the file to disk. So there is nothing there to change the file name of. A few more things I tried:
- Manually ran "touch" on the file name that APP would end up with for the OIII integration (created a 0 byte file with the name). When integration was complete for OIII APP did ask me if I want to overwrite the file. Saying yes, still resulted in APP failing to write the fail claiming unsupported or locked image file. This tells me it's not a file system access issue
- Removed one sessions from the integration stack for OIII, this essentially shortened the file name by 2 chars (Full filename 122 chars). This time APP succeeded in writing the file. So did removing two or three sessions (shortening the file name to 120 and 118 chars)
So it looks like there is some limit in the software where if the filename is >= 123/124 chars (not sure which is the limit), it can't save the file. This can't be a Java limit either because it has no such limits.
Wanted to add some more data points.
Just for kicks, I tried reducing the number of chars in the file name by changing the filter tag instead of removing some sessions. This way I brought the number of chars in the file name down to 121 chars, and......it still failed! even though earlier it had saved the file fine with 122 chars!
That got me thinking.....I don't think it's the number of chars in the filename. Looking at my stacks, OIII which is failing has the largest number of sessions, 10 to be exact. If I reduce this to 9 everything is fine.
So it looks like there is a bug/limit in APP where if the number of sessions is >= 10 it can't save the stack!! Please look into it?
Also while you are at it, a few requests:
- Save information in the FITS header through custom tags. Putting them in the file name seems inefficient and the info is lost if someone changes the filename
- For sessions add the capability to use capture date as the session (instead of choosing "session1" etc. or some other custom thing). Most FITS, if not all will have the capture date. This way they can be grouped fairly easily