[Solved] Astro Pixel Processor crashes when loading Fuji raw files

Hi Sven @tricx,

Thank you very much for posting your issue.

I have about 10 test images of the X-T4 camera and they all work fine on my MacBookPro with M4 max chip on Sequoia 15.2, I just tested it. All also okay on a mac mini M1 and an intel based mac pro 2017.

Are you sure that the Fuji raws are not corrupt somehow? Do they load in other applications?

Is the losslees compressed format new for the camera perhaps ? Was a firmware update needed for that format maybe? If so, maybe our raw support with 3rd party LIbRaw needs an update to support the files.

Please upload your raws here so I can test and fix as soon as possible.

https://upload.astropixelprocessor.com/

username: uploadData

password: uploadTestData

please make a folder with your name and issue like : Sven-Fuji-Raw-issue

and upload the raws there and let me know once uploaded. I will take a look as soon as possbile then.

Mabula

I processed some Fuji Raws from an XT-3 in December without any issues. I wasn't aware that they were compressed though - just the standard RAF file you get when you choose the saving option of 'RAW+jPEG'. That was with Mac OS Sequoia running on an M1 studio.

Ah, now I see in the 'RAW recording menu' that you can choose a lossless compressed version. I have never used that - but maybe APP is not decompressing them?

Hi Sven @tricx @jdwood,

I am following up to check if loading your FUJI raws can be fixed.

Our raw file loading depends on a 3rd party, LibRaw, I know that there is an update so that might fix the issue. It will help us greatly if you can upload some of your Raws to our server so we can test it and fix it.

Jon, can you perhaps make a couple of fuji raws with the lossless compression for us to test as well? Thanks !

> https://upload.astropixelprocessor.com/ <

username: uploadData

password: uploadTestData

please make a folder with your name and issue like : Sven-Fuji-Raw-issue

and upload the raws there and let me know once uploaded. I will take a look as soon as possbile then.

Thanks,

Mabula

Hi Mabula @mabula-admin,

I have just taken 3 random sky shots (no tracker so they will be blurred 😀 ), one saved 'uncompressed', one 'lossless compressed' and the other 'compressed' from Fuji XT5.

Have uploaded these to your server.  

For me they all open OK in APP beta 32.  I didn't try any processing but did not observe a crash on loading.  M1 Mac Studio running Sequoia 15.3.

Jon

Well, I don't know how to write it, but I can't reproduce the crashes.

Directly after installation and the first attempt to load some Fuji raws, the app crashed.
So, I decided to try on another system with exactly the same result.
That's why I decided to report here about my experience.

Driven by Mabula's offer to check with my files, I wanted to check how many files are required to reproduce the issue - in order to upload them. Now, I'm happy the report that the issue is gone.
My raws could be opened on both of my systems.

It looks like the issue happens only directly after a fresh installation and might not be related with Fuji raw files at all.
So I tried to uninstall and re-install APP, but I couldn't reprooduce the crash. Maybe I missed to remove some config files?

However, I apologize for any inconvenience,
Sven

Good that it worked out OK Sven. I find that the Fuji's are great for astro work... 😀 

Posted by: @jdwood

↑

Hi Mabula @mabula-admin,

I have just taken 3 random sky shots (no tracker so they will be blurred 😀 ), one saved 'uncompressed', one 'lossless compressed' and the other 'compressed' from Fuji XT5.

Have uploaded these to your server.  

For me they all open OK in APP beta 32.  I didn't try any processing but did not observe a crash on loading.  M1 Mac Studio running Sequoia 15.3.

Jon

Thanks Jon @jdwood,

Indeed, all good here as well with testing these RAFs. Thanks for the upload and helping out !

Mabula

Posted by: @tricx

↑

Well, I don't know how to write it, but I can't reproduce the crashes.

Directly after installation and the first attempt to load some Fuji raws, the app crashed.
So, I decided to try on another system with exactly the same result.
That's why I decided to report here about my experience.

Driven by Mabula's offer to check with my files, I wanted to check how many files are required to reproduce the issue - in order to upload them. Now, I'm happy the report that the issue is gone.
My raws could be opened on both of my systems.

It looks like the issue happens only directly after a fresh installation and might not be related with Fuji raw files at all.
So I tried to uninstall and re-install APP, but I couldn't reprooduce the crash. Maybe I missed to remove some config files?

However, I apologize for any inconvenience,
Sven

Dear Sven @tricx,

Thank you very much for your feedback. I am very happy to hear that all is well now somehow. I must say that once I freshly installed APP on macOS 15.2, it did crash on some other operation as well and only once... There might be a bug in macOS 15 here actually, because after that initial crash alll has been fine here... after using APP for many times on a new macBook Pro.

Mabula

Hi Mabula,
yesterday the sky was clear at night, so I went out for another astro-photography session.
Since I'm a newbie in this field, I started with M42.
However, processing in APP results in an Java-Exception (ArrayIndexOutOfBoundsException) when APP processes the flats.
It is reproducible on both of my machines (MacBook Pro M4, and MacBook Pro Intel), I use for my tests.
I attached a screenshot of the error message.
I have logs and could share also my Fuji Raw files.
What is the quota on the file server? The full set of files would be around 4GB.
I can try to reproduce it with a limited set of files, but it takes time for trial and error. Please advise.

Thank you advance,
Sven

Hi Mabula.

I use this thread, to expand on the specialty of Fuji RAW files (RAF).
For newer cameras (definitely the APS-C cameras X-H2S, X-H2, X-T5, and also medium format GFX 100S, GFX 100S II) there are 3 RAF formats:
- uncompressed ... existing "since ever", file size c. 55MB for X-H2S, ca. 200MB for the GFX 100S.
- lossless compressed ... reducing file size by c. 40%
- lossy compressed ... reducing a further c. 25%

For photography, the lossy compressed format is nearly perfect: smallest file size (not so much re. storage, more so for fast shooting e.g. for birds in flight) and according to all sources, no visible loss in sharpness, colour, etc.

----

Long story short: I yesterday had my first astro session with tracker, produced some 200 lights, 50 darks and a bunch of flats, bias, dark flats  ... did not change the file settings ... all in lossy compressed.

Now the questions are:

1. Does APP support lossy compressed Fuji RAF? I am a bit pessimistic, because a quick googling said that "libraw" on Mac does only support uncompressed and lossless compressed Fuji RAF. 🙁
2. Does it make any sense to demoniac in (e.g.) Adobe Camera RAW, export to 16 bit TIFF ... and load all the frames into APP? I know, RAW is much, much better ... but clear sky is a rare event here in Aachen, I would like to see at least something.
3. If TIFF for the light frames is possible ... does it make any sense to use TIFF for Darks, Flats etc.?

I would be happy, if one of the questions can be answered with yes.

Thank you and best regards.

Michael 

Hello Michael,

as a work arround you could try to load your images into Siril as a sequence. Select Fits as output format and press the convert button. Then all your files will be converted to the fits format which you can further process with APP. 

Hope Siril will be able to read and convert this format as well.  

Best Regards 

Ralf

Posted by: @tricx

↑

Hi Mabula,
yesterday the sky was clear at night, so I went out for another astro-photography session.
Since I'm a newbie in this field, I started with M42.
However, processing in APP results in an Java-Exception (ArrayIndexOutOfBoundsException) when APP processes the flats.
It is reproducible on both of my machines (MacBook Pro M4, and MacBook Pro Intel), I use for my tests.
I attached a screenshot of the error message.
I have logs and could share also my Fuji Raw files.
What is the quota on the file server? The full set of files would be around 4GB.
I can try to reproduce it with a limited set of files, but it takes time for trial and error. Please advise.

Thank you advance,
Sven

 

 

Dear Sven @tricx,

Thank you very much for sharing this error.

Can you please upload the lights and calibration frames that produce this error?

https://upload.astropixelprocessor.com/

username: uploadData

password: uploadTestData

Please make a folder with your name and issue like: Sven-Fuji-Crash

and let me know once uploaded, I will test and fix this issue as soon as possible. We have no Qouta on the server, only a cap on the upload speed 😉 so 4GB is no issue.

Thanks,

Mabula

Dear Sven @tricx,

Thank you very much for sharing this error.

Can you please upload the lights and calibration frames that produce this error?

https://upload.astropixelprocessor.com/

username: uploadData

password: uploadTestData

Please make a folder with your name and issue like: Sven-Fuji-Crash

and let me know once uploaded, I will test and fix this issue as soon as possible. We have no Qouta on the server, only a cap on the upload speed 😉 so 4GB is no issue.

Thanks,

Mabula

 

Hi Mabula,

I've uploaded my files to "IndeoxOutOfBounds-Sven".
Additionally you will find there some screenshots on what I did in APP to reproduce the issue.

Thank you in advance,
Sven

Posted by: @tricx

↑

Dear Sven @tricx,

Thank you very much for sharing this error.

Can you please upload the lights and calibration frames that produce this error?

https://upload.astropixelprocessor.com/

username: uploadData

password: uploadTestData

Please make a folder with your name and issue like: Sven-Fuji-Crash

and let me know once uploaded, I will test and fix this issue as soon as possible. We have no Qouta on the server, only a cap on the upload speed 😉 so 4GB is no issue.

Thanks,

Mabula

 

Hi Mabula,

I've uploaded my files to "IndeoxOutOfBounds-Sven".
Additionally you will find there some screenshots on what I did in APP to reproduce the issue.

Thank you in advance,
Sven

 

Hi Sven,

Thanks for the upload, I have found and fixed the issue 🙂 APP 2.0.0-beta34 will have the fix, I will try to release it in the coming days 😉

https://www.astropixelprocessor.com/community/release-information/astro-pixel-processor-2-0-0-beta34-release-notes-work-in-progress/#post-32365

• Fixed Fuji X-Trans RAF ArrayIndexOutOfBoundsException in ImageNormalizer

  As mentioned here  https://www.astropixelprocessor.com/community/main-forum/astro-pixel-processor-crashes-when-loading-fuji-raw-files/#post-32320 when trying to normalize light or flat frames from Fuji X-Trans RAF files, APP could throw an ArrayIndexOutOfBoundsException in the ImageNormalizer module. This bug has been fixed in calibration, normalization and integration.

Thank you very much for reporting the issue.

Hi Mabula.

I can happily confirm that the above error is fixed.
APP is processing Fuji (X-H2S) RAF, both lossy compressed and lossless compressed. (Uncompressed I did not try, but have no doubts.)

Thank you 👍

It's so great to see, how APP is using on my MacBook Pro M3max all 16 CPU cores, all fully loaded.

Super, Mabula.

Hi Michael @michaelacg,

Great, thank you very much for your feedback! You are most welcome.

Sven @tricx,

All okay with 2.0.0-beta34 for your Fuji X-Trans files as well now?

Mabula

Posted by: @mabula-admin

↑

Sven @tricx,

All okay with 2.0.0-beta34 for your Fuji X-Trans files as well now?

Mabula

Hi Mabula,

yes the bug seems to be fixed.
Thank you for promptly addressing the issue!

Now I can start testing APP 🙂

BR,
Sven

Awesome Sven @tricx,

Thank you very much for your feedback, if you need more testing time on your Trial license, let me know and I will extend it for you 😉

Mabula

Posted by: @mabula-admin

↑

Awesome Sven @tricx,

Thank you very much for your feedback, if you need more testing time on your Trial license, let me know and I will extend it for you 😉

Mabula

That would be great, since the current trial license expires already tomorrow. 🙁

Hi Sven @tricx,

I have extended your Trial license with another 30-days, you should have received a confirmation of this by email as well. There is no need to re-enter the license, once your start/restart APP, the new expiry date will be found and used 😉

Mabula