Integration Write F...
 
Share:

APP 1.075 has been released !

2019 September: Astro Pixel Processor and iTelescope.net celebrate a new Partnership!

Integration Write Fails - File Type Unsupported  

  RSS

(@rdricks)
Brown Dwarf Customer
Joined: 11 months ago
Posts: 7
October 14, 2019 06:10  

I am having an issue I have not seen before.  I suspect it could be data and am trying more testing, but wanted to see if anyone has experienced something similar.

I am processing just under 500 raw images over 12 sessions.  Each session is using common Master Bias and BPM, but it’s own master dark and flat. I create the masters in a separate session and then pull the master files in when I run steps 2-6.   It goes through all the steps and starts writing the file  when it gets to 100% it posts an error, and no file is saved in the working directory  

”this file type is unsupported or locked by another process”

I am using MacOS Mojave on a Mac Pro.  the machine has been rebooted and is running only APP. I previously processed the same data minus the last 2 sessions and it worked properly, which is why I suspect data.  I have tried processing 3 times with the same result.  I’m going to rerun without the last 2 sessions to see if it will work properly again. At approx 9~10 hours to process it’s a little frustrating  

Any thoughts / Suggestions?

 


ReplyQuote
(@mabula-admin)
Quasar Admin
Joined: 3 years ago
Posts: 2160
October 15, 2019 21:09  

Dear @rdricks,

Do you mean, that the whole integration of step 6) finishes? And it errors on actually saving the final integration?

What does the console panel exactly show when it occurs?

The mentioned error message:

”this file type is unsupported or locked by another process”

is related to file loading and not to saving, which makes this a bit odd... the output of the console panel with all details will really help to determine what is happening.

Kind regards,

Mabula

 

Main developer of Astro Pixel Processor and owner of Aries Productions


ReplyQuote
(@rdricks)
Brown Dwarf Customer
Joined: 11 months ago
Posts: 7
October 18, 2019 21:58  

That is correct, the whole integration (Step 6) completes.  The error is on saving the final integration.  I was confused by the behavior as well.

I had repeated the attempt 3 times with the same result.  I then removed the last days data and tried; it worked as expected.  So then I went back ind integrated only the last day of data, and that was successful.

Right now I have another integration attempt using all the data (Same as failed previously) to see if it fails again.  I will post the result once it completes.  If I get the failure I will grab the console output.  (I did not think to capture that the previous fails...).

It really is an amazing processing tool.


ReplyQuote
(@rdricks)
Brown Dwarf Customer
Joined: 11 months ago
Posts: 7
October 19, 2019 04:56  

More information.  I repeated with all the sessions and it failed again.  I apologize for the crudeness, but I took the photos below to get the information quickly.

Sessions being processed: (When I added Oct 5 it failed, Only Oct 5 processes fine).

APP Sessions

Error after it completes integration:

APP Error Message

Console (Image):

APP Console


ReplyQuote
(@vincent-mod)
Galaxy Moderator
Joined: 2 years ago
Posts: 928
October 19, 2019 11:52  

Mm, this is a very very long filename, I don't think that should be an issue but it's something that jumps out, I never saw that. Is it possible to try to shorten that?


ReplyQuote
(@rdricks)
Brown Dwarf Customer
Joined: 11 months ago
Posts: 7
October 19, 2019 17:31  

I hadn't thought about file name length.  I've just been letting APP use whatever it defaults to.  Since there are so many separate sessions it is creating a long one.  

Is there a way to pre-select a filename before integration?  By the time I get the error the only option is OK, and no files are in the working directory.


ReplyQuote
(@vincent-mod)
Galaxy Moderator
Joined: 2 years ago
Posts: 928
October 19, 2019 21:29  

I have no idea if this would be the problem, I kind of doubt it even, but otherwise I wouldn't know what would cause it. Maybe @mabula-admin has an idea?


ReplyQuote
(@manojkoushik)
Brown Dwarf Customer
Joined: 2 months ago
Posts: 8
November 6, 2019 09:57  

I see this issue as well (both on Windows and Mac OS X). I am convinced it's because of the long file names. I have more than 10 sessions on some filters and once all of that is added into the filename, I believe we might be hitting the limit (PATH_MAX on Mac for e.g.). Is there any way to customize the filename? A lot of the information that's in the file name, is possibly also in the fits headers (or can be dumped into the fits header instead) and the file name made shorter. It's a bit unwieldy right now. 


(@vincent-mod)
Galaxy Moderator
Joined: 2 years ago
Posts: 928
November 6, 2019 18:56  

I've asked Mabula to have a look at this issue, hopefully he can answer shortly. Thanks for the report!


ReplyQuote
(@manojkoushik)
Brown Dwarf Customer
Joined: 2 months ago
Posts: 8
November 6, 2019 19:23  

Just to add some more data points, My OIII integration is the one that's failing. SII and Ha work fine. 

 

OIII file name is 124 characters. Path Name is 148 characters

SII file name is 117 characters. Path Name is 141 characters

Ha file name is 118 characters. Path Name is 142 characters

 

So those additional 6 characters in the OIII file/path name is some how causing the program to fail saving it. This is on Mac OS X, where the file name limit is supposed to be 255 chars and the path name soft limit is supposed to 1024 chars. 

 


(@vincent-mod)
Galaxy Moderator
Joined: 2 years ago
Posts: 928
November 6, 2019 21:02  

Does it work when you manually shorten the name in the finder?


ReplyQuote
(@manojkoushik)
Brown Dwarf Customer
Joined: 2 months ago
Posts: 8
November 6, 2019 22:05  

I can't really do that because APP is failing to write the file to disk. So there is nothing there to change the file name of. A few more things I tried:

- Manually ran "touch" on the file name that APP would end up with for the OIII integration (created a 0 byte file with the name). When integration was complete for OIII APP did ask me if I want to overwrite the file. Saying yes, still resulted in APP failing to write the fail claiming unsupported or locked image file. This tells me it's not a file system access issue

- Removed one sessions from the integration stack for OIII, this essentially shortened the file name by 2 chars (Full filename 122 chars). This time APP succeeded in writing the file. So did removing two or three sessions (shortening the file name to 120 and 118 chars)

So it looks like there is some limit in the software where if the filename is >= 123/124 chars (not sure which is the limit), it can't save the file. This can't be a Java limit either because it has no such limits. 

 


(@rdricks)
Brown Dwarf Customer
Joined: 11 months ago
Posts: 7
November 7, 2019 00:11  

Same issue I'm having.  The error occurs before the option comes up to change the filename.  When the error is dismissed the work is gone (No files in working folder).  It only happens when I have many sessions, causing the long file name.


ReplyQuote
(@manojkoushik)
Brown Dwarf Customer
Joined: 2 months ago
Posts: 8
November 7, 2019 05:37  

Wanted to add some more data points.

Just for kicks, I tried reducing the number of chars in the file name by changing the filter tag instead of removing some sessions. This way I brought the number of chars in the file name down to 121 chars, and......it still failed! even though earlier it had saved the file fine with 122 chars!

That got me thinking.....I don't think it's the number of chars in the filename. Looking at my stacks, OIII which is failing has the largest number of sessions, 10 to be exact. If I reduce this to 9 everything is fine. 

 

So it looks like there is a bug/limit in APP where if the number of sessions is >= 10 it can't save the stack!! Please look into it?

 

Also while you are at it, a few requests:

- Save information in the FITS header through custom tags. Putting them in the file name seems inefficient and the info is lost if someone changes the filename

- For sessions add the capability to use capture date as the session (instead of choosing "session1" etc. or some other custom thing). Most FITS, if not all will have the capture date. This way they can be grouped fairly easily

 

 


(@vincent-mod)
Galaxy Moderator
Joined: 2 years ago
Posts: 928
November 7, 2019 12:08  

Thanks so much for the deep dive into the issue Manoj! I will pass this on to Mabula right now.


ReplyQuote
Share: