[Solved] Error reading images from networked share

Hi Craig @null,

Please try with the latest APP 2.0.0-beta24 😉 It has many updates, bugfixes and also updated FITS support. Let me know if that solves it 😉 You can find the latest release in the download section:

https://www.astropixelprocessor.com/downloads/

@mabula-admin Oops, I didn't realize I wasn't on the latest beta. In any case, I'm seeing the same issue with beta 24 as well.

Many thanks for the incredibly fast response!

HI Craig, okay thank you, then it must be related to the network mapping somehow, because others are using the network extensively with FITS data and report no such issues.

I will try to see if i can reproduce the issue and fix it. I have opened an issue on our ToDo list.

Do you have the netwerk folder \astro\DSO\... mapped to a static folder in Windows, from your information it does not seem the case. I would think that If you would, the problem will be gone. So please try that as well.

Mabula

Hi, Mabula,

I experience the same issue if I map the networked share (e.g., Z:\Astro\...), or when accessing via `Network`, which results in SMB/CIFT syntax (\\server.local\Astro\...)

It's interesting to hear that others have success with a similar setup. I'll note that other programs (e.g. PixInsight) can read files in question when accessed either way, so there must be something deeply strange going on. Let me know if I can provide any more information/debug logs/whatnot.

Thanks again,

Craig

Hi, Mabula,

The issue has something to do with file permissions. The files I was testing with were readable, but not writable, by remote users. If I change the files to be writeable as well as readable, they load successfully. Just being readable isn't enough, it seems.

I wasn't able to reproduce the same issue using a file on a local filesystem. As such, I'm not sure if this is a server/smb/cifs issue, or something else. I'll keep experimenting.

Regards,

Craig

Hi Craig @null,

Okay, that is usefull information, I guess it has to do then with our file reader which for FITS works differently than for JPEG or TIFF or RAWS. Did you try other image formats?

HI Mabula @mabula-admin,

JPEG and TIFF don't exhibit the same problem. It does appear to be FITS-specific.

I don't know what FITS reader you're using, but I tested with a simple program that uses CCFITS/CFITSIO under the covers, and couldn't reproduce the issue.

Dear Craig @null,

We use nom.tam.Fits managed and created by Nasa. We use random access fits IO, so maybe it requires Read/write under the covers in their library. I will check it out 😉 thanks !

Mabula

Hi Craig @null,

I have done some testing on my network, I can read FITS files without issues over my network using the SMB protocol. Also when the files have only read access rights.

And there are no issues if I map the network drive in Windows nor when I access the network share directly.

So it does not seem an issue related to APP's fits IO. In addition, many users use their network to access files these days with APP without issues. Some even put the working directory on a network share i know.

I think the issue is related to how you have setup your network or how you are trying to access it :

If i access the network share with a mapping to the R drive, the file name becomes: R:\[folder]\Light_M42_300.0s_Bin1_2600MC_gain100_20230321-215925_0001.fit

If I access the network directly it is: \\[network device]\[network folder]\[folder]\Light_M42_300.0s_Bin1_2600MC_gain100_20230321-215925_0001.fit

and it all works.

So your path to the file on the network: file: \\server.local\astro\DSO\M31\2023-09-26\Light\frame001.fit can't be properly resolved by APP somehow. Is server.local the network device name on your network? How is SMB setup on that device?

My network SMB is setup with a Synology NAS and all works without issues I think. The protocol is SMB2/SMB3. I have tested with both windows 10 and 11 and macosx Sonoma with the latest updates and APP 2.0.0-beta25.

When I access the network with windows 11 with all updates, windows 11 forces me to access it with a mapping even. I can not access it directly at the moment. Network discovery on windows 11 is not showing the network device allthough everything is enabled in the network settings on windows 11 and on the network device which is odd at the moment. I am probably missing something here with windows 11 settings.

Mabula

Hi, Mabula @mabula-admin,

Many thanks again for looking at this issue, even when I have a workaround. Much appreciated!

I've done some more testing, and the results are (still) confusing.

I have tested using three different servers hosting SMB shares: One running the latest release of TrueNAS Scale, one running Windows 11, and one running the latest release of MacOS.

In all three cases, APP issues an error when I attempt to load a FITS file if the file is not writable. In the Win11 server case, the file must be shared with "full permissions" in order for me to load them, which is separate from the on-disk permissions associated with the file itself.

The error occurs if I access the shares via the Network 'folder' (e.g., \\myserver\astro\x.fits) or via a mapped drive (e.g., Z:\astro\).

I've tested using APP on two different Windows 11 hosts, with the same results. Accessing the read-only files via APP on a mac works fine. Accessing the read-only files using other fits-reading programs (PixInsight, ASIFitsView, etc.) on Windows also works fine.

I've tested using both FITS files that I've generated, and those downloaded from elsewhere, with the same results.

Note that things indeed work fine everywhere if the fits files are writable.

The issue seems limited to APP on Windows loading non-writable files from an SMB share.

So, no concrete answers, but some more data. I'll keep testing; let me know if there's any experiments you'd like me to try. It's curious that you're not able to reproduce the issue; I'll give that some more thought.

Craig

Hi Mabula @mabula-admin,

After some more experimentation, another clue.

Looking at nom-tam-fits' randomInit() implementation in src/main/java/nom/tam/fits/Fits.java, one sees that if the canWrite() method of the given File returns true, the file is opened in "rw" mode (instead of just "r" mode).

    // TODO make private
    @Deprecated
    protected void randomInit(File file) throws FitsException {

        String permissions = "r";
        if (!file.exists() || !file.canRead()) {
            throw new FitsException("Non-existent or unreadable file");
        }
        if (file.canWrite()) {
            permissions += "w";
        }
        try {
            dataStr = new FitsFile(file, permissions);
            ((FitsFile) dataStr).seek(0);
        } catch (IOException e) {
            throw new FitsException("Unable to open file " + file.getPath(), e);
        }
    }

Of course I'm not sure if this randomInit() method is called by APP, but in case it is: It seems that the canWrite() method returns true (at least in my testing) for read-only files on SMB shares, though it works as expected on local files.

If that is indeed the code path APP takes, nom-tam-fits then attempts to open the file in "rw" mode, which I expect would fail because the file is not actually writable.

Doing a bit of googling, I see others noting canWrite() falsely returning true on read-only files, and others suggesting that isWritable() is more trustworthy than canWrite().

Of course, none of this explains why you've been unable to reproduce the issue on your end.

I should add that although File::canWrite() incorrectly returns true on read-only smb-hosted files when on windows, in my testing File::canWrite() correctly returns false on read-only smb-hosted files when on MacOS.

Given that the above matches the behavior of the APP issue I'm seeing, I think this is a solid candidate for the cause of the issue. I'll see if I can reproduce the behavior using a simple program that leverages nom-tam-fits. It may take me a bit, as I'm a java neophyte (more of a C++/CUDA/python type 🙂

Hi Mabula @mabula-admin,

I was able to reproduce read-only issue with a standalone test program, and root cause does seem to be the nom-tam-fits issue outlined above.

I have submitted a pull request to the nom-tam-fits project that addresses the issue. See: https://github.com/nom-tam-fits/nom-tam-fits/pull/496

I tried to drop the updated nom-tam-fits jar file into the APP lib folder to see if it fixes the issue in APP proper, but hit an error. I'm not sure if its due to version differences, or something else.

With any luck, the nom-tam-fits folks will eventually accept the change, and you can pick it up the next time you pull in the library. For now, there's nothing to do but wait.

Thanks again!

Craig

Hi Mabula @mabula-admin,

One more note: I backported my nom-tam-fits change to the version used by APP (1.18.0), and replaced APP's libs/nom-tam-fits-1.18.0.java with the patched version. Doing so did indeed fix the APP issue I've been seeing.

So this is definitely a nom-tam-fits issue, which hopefully will be fixed in their next release.

Cheers,

Craig

Hi Mabula @mabula-admin,

The fix has been merged into nom-tam-fits/master, so it should appear in the next release. Phew!

Many thanks again,

Craig

Hi Craig @null,

That is awesome, thanks for reporting this to nom-tam-fits. I am glad that it was fixed so quickly. I will try to incorporate this fix in APP as soon as possible !

I guess my testing failed to set the network files to read-only somehow and I missed the issue. I will check again to make sure !

Thanks,

Mabula