Invalid FITS Header...
 
Share:
Notifications
Clear all

2023-03-15: APP 2.0.0-beta14 has been released !

IMPROVED FRAME LIST, sorting on column header click and you can move the columns now which will be preserved between restarts.

 

We are very close now to  releasing APP 2.0.0 stable with a complete printable manual...

 

Astro Pixel Processor Windows 64-bit

Astro Pixel Processor macOS Intel 64-bit

Astro Pixel Processor macOS Apple M Silicon 64-bit

Astro Pixel Processor Linux DEB 64-bit

Astro Pixel Processor Linux RPM 64-bit

Invalid FITS Header on ZWO ASI 178MC

26 Posts
6 Users
1 Likes
1,528 Views
(@spectre_68)
Brown Dwarf
Joined: 2 years ago
Posts: 7
Topic starter  
HI!
my name is Riccardo, and I am a new (trial) user of APP.
I am trying to calibrate images created with a ZWO, but every time I try to preview the FITS file I get this error:
"This fits file seems to be incomplete and / or corrupt Error message: IO error: java.io.IOException: Invalid FITS Header "

With other programs I can load and align files.
Can you help me?



   
ReplyQuote
(@vincent-mod)
Universe Admin
Joined: 6 years ago
Posts: 5368
 

That sounds like the file itself does have a problem somehow. In fact the FITS header is the issue, so while other programs may load it, they must be ignoring that while APP is using it to judge what image it is. What program did you use for shooting and saving the images?


   
ReplyQuote
(@spectre_68)
Brown Dwarf
Joined: 2 years ago
Posts: 7
Topic starter  

Hi!

Voyager on Zwo Asi last driver.


   
ReplyQuote
(@spectre_68)
Brown Dwarf
Joined: 2 years ago
Posts: 7
Topic starter  

this is my FITS file created with Voyager.


   
ReplyQuote
(@vincent-mod)
Universe Admin
Joined: 6 years ago
Posts: 5368
 

Mm, I'm not familiar with Voyager, do you still have the raw data before using Voyager? If so, please try to load those, I suspect Voyager isn't writing the Fits header properly. I'll check the fits as well here.


   
ReplyQuote
(@vincent-mod)
Universe Admin
Joined: 6 years ago
Posts: 5368
 

Having said that, I also can load it in another package, so I'll forward this one to Mabula to see what the issue is if at all. Thanks for sharing!


   
ReplyQuote
(@spectre_68)
Brown Dwarf
Joined: 2 years ago
Posts: 7
Topic starter  

Hi!

Ufortunately I have no raw data, voyager writes the file directly in the updated format. I am waiting for any developments.

Riccardo.


   
ReplyQuote
(@vincent-mod)
Universe Admin
Joined: 6 years ago
Posts: 5368
 

Mm, that's something you can't change in preferences? I would advice to always keep the original data when possible. There should be no need to process the raw data in between as APP is able to do this together with the raw calibration data.


   
ReplyQuote
(@spectre_68)
Brown Dwarf
Joined: 2 years ago
Posts: 7
Topic starter  

HI!

my opinion APP expects to find a value in the header that is not written by vayager in the FITS file. But all the other programs read without problem the FITS files generated by this program APP seems to be the only one to have this type of problem (pixinshight, maximDL, DeepStaker, FITS Viewer read without problems)


   
ReplyQuote
(@vincent-mod)
Universe Admin
Joined: 6 years ago
Posts: 5368
 

Yes this could very well be, I've forwarded the file to Mabula so I'm waiting for his analysis. The original raw data however will very likely work anyway.


   
ReplyQuote
(@spectre_68)
Brown Dwarf
Joined: 2 years ago
Posts: 7
Topic starter  
Hi, any news on the problem? If the trial license expires I will not be able to test this software.
Best Regards, Spectre

   
ReplyQuote
(@vincent-mod)
Universe Admin
Joined: 6 years ago
Posts: 5368
 

Not yet no, Mabula is working hard on the next version, when he doesn't respond in a few days I'll notify him again (please ask here again as well then, so we're sure we can get you the help needed).


   
ReplyQuote
(@mabula-admin)
Universe Admin
Joined: 6 years ago
Posts: 3450
 

Hi Riccardo @spectre_68,

I am investigating your voyager fits file right now. It does seem that the FITS is not created properly according to FITS NASA specification. We use a NASA library to READ/WRITE FITS files, so a properly created FITS would not give this error I think... but let me investigate a bit more for now. I will report back later 😉

Mabula


   
ReplyQuote
(@mabula-admin)
Universe Admin
Joined: 6 years ago
Posts: 3450
 

Hi Riccardo @spectre_68,

After more investigation, it is becoming more clear now that this FITS file has a serious problem in terms of the FITS specification. I can only conclude at the moment that something went wrong in Voyager when the file was written to your harddisk. Do the other Fits files from voyager work properly?

Mabula


   
ReplyQuote
(@spectre_68)
Brown Dwarf
Joined: 2 years ago
Posts: 7
Topic starter  

@mabula-admin

Hi!

only FITS generated with the ASI give this problem.

   
ReplyQuote
(@mpemme)
Molecular Cloud
Joined: 3 years ago
Posts: 3
 

Hi,

my name is Paolo,

same problem encountered with images from a Starlight Xpress Trius-SX26C Color - ATEO2A Insight Observatory -

, see FIT attached. (change type file from .txt to .zip and dezip the file)

This post was modified 2 years ago by Paolo Madurini

   
ReplyQuote
(@wvreeven)
Quasar
Joined: 5 years ago
Posts: 2102
 

@mpemme Was this file created by Voyager as well?


   
ReplyQuote
(@mpemme)
Molecular Cloud
Joined: 3 years ago
Posts: 3
 

Hi,

I don't know, but other files always downloaded from insight observatory's Starbase platform - from mono sensors like FLI Proline 16803 or SBIG STL11000 - I work very very well with A.P.P.


   
ReplyQuote
(@mpemme)
Molecular Cloud
Joined: 3 years ago
Posts: 3
 

Hi,

problem solved, I opened and saved all files with M DL. Now it would seem that A P P works them.


   
ReplyQuote
(@mabula-admin)
Universe Admin
Joined: 6 years ago
Posts: 3450
 

Hi all @mpemme, @wvreeven, @spectre_68, @vincent-mod

Together with Leonardo Orazi, developer of Voyager, we have found the problem. The FITS writer code that Voyager uses has a bug in terms of FITS specification.

The FITS headers are not closed properly according to FITS specification.

https://fits.gsfc.nasa.gov/fits_primer.html

"Every HDU consists of an ASCII formatted `Header Unit' followed by an optional `Data Unit'. Each header or data unit is a multiple of 2880 bytes long. If necessary, the header or data unit is padded out to the required length with ASCII blanks or NULLs depending on the type of unit.

The header is padded with additional blank records if necessary so that it is a multiple of 2880 bytes (equivalent to 36 80-byte keywords) long. Note that the header unit may only contain ASCII text characters ranging from hexadecimal 20 to 7E"

The problematic voyager files have the HDU bytes filled up with 00 which is not allowed in the FITS specification. Those 00 should be 20 in hexadecimals... which is 32 in decimals... which is the ASCII blank character.

I understand from Leonardo that the FITS writer code in Voyager is a port of the same NASA library that we use and so this port was not done correctly then... Apparently, even the ASCOM driver of ASI has this bug it seems then.

I will make sure that APP will catch this bug and deals with it so the files can still be read in APP's next release 😉 allthough the problem is caused by the capture software or driver.

Cheers,

Mabula


   
ReplyQuote
(@the_bluester)
Main Sequence Star
Joined: 4 years ago
Posts: 26
 

I have actually just run in to this problem with a few .fit files shot using Voyager and a QSI6162 camera. I tried using the current beta but that still threw an invalid fits header error. Is the workaround that was based on the ASI cameras fully implemented in the current beta 1? Can I send a file via wetransfer or similar? The output of the 6162 is 30.9 meg so I can't attach a sample file to examine here.

Incidentally, I have always used an ASI camera with Voyager (First an ASI294MC Pro and now an ASI2600MC Pro) without ever seeing this error, it must be specific to particular cameras or drivers.


   
ReplyQuote
(@the_bluester)
Main Sequence Star
Joined: 4 years ago
Posts: 26
 

I just realised that I think I have misread release notes for 1.83 beta 2 as applying to beta 1. If that is the case I will wait with bated breath for the beta 2 release, hopefully in the coming days.


   
ReplyQuote
(@wvreeven)
Quasar
Joined: 5 years ago
Posts: 2102
 

@the_bluester Yes there will be better handling of some types of corrupt FITS files in 1.083 beta 2. If you want to know if your files can be opened with that version then please upload a few files to

https://upload.astropixelprocessor.com/

using upload5 for both username and password. Please create a directory called the_bluester_corrupt_fits and place the files there. As soon as the files are there I will ask Mabula to have a look at them.


   
ReplyQuote
(@the_bluester)
Main Sequence Star
Joined: 4 years ago
Posts: 26
 

Thanks, I have uploaded two, even one would probably do. My friend is considering a move to Voyager so it would be good to know up front that 1.083 will open files produced by it from his camera.


   
ReplyQuote
(@mabula-admin)
Universe Admin
Joined: 6 years ago
Posts: 3450
 

Hi @the_bluester & @wvreeven,

Yes, APP 1.083-beta2 has the fix for this problem:

WORK-AROUND IN FITS READER, FOR APPLICATIONS THAT USE A FAULTY FITS WRITER: some users reported problems with some FITS files where APP could not read these, but other applications could. The topic is : https://www.astropixelprocessor.com/community/camerasupport/invalid-fits-header-on-zwo-asi-178mc/ Some of the problem FITS files come from the Voyager capture software developed by Leonardo Orazi. Together with Leonardo, we have found the root cause of the problem. It turned out that the FITS writer code that is used in Voyager is a badly ported version of the nasa java fits library that we use in Astro Pixel Processor. The ported version is not following the FITS specification for how the FITS Header bytes should be filled with byte values once all metadata has been stored. So the root cause is not in APP, the root cause of the problem is in the capture software or the ascom driver that such a capture package uses. Leonardo has fixed this issue in Voyager and in APP, I have implemented a work-around so that this particular FITS issue is detected and the files are still read properly.

So the error actually is in older Voyager versions and other applications that use a faulty fits writer. Leonardo Orazi from Voyager has solved this issue form his end as well 😉 And APP 1.083 will be able to read these problem files as well.

To be sure, I have just checked your uploaded files in APP 1.083-beta2 and yes they work fine now 😉

PROBLEM FITS WORKS NOW

Mabula


   
ReplyQuote
(@the_bluester)
Main Sequence Star
Joined: 4 years ago
Posts: 26
 

Thank you for checking. I will have to follow up with Leo, the version I was using is pretty up to date, it might be a separate problem if the fix has been implemented already. It might be a while before I can test again, we were supposed to have another go tonight but we both live in Victoria (Australia) and are under a COVID lockdown at the moment so that is not going to happen.

Regardless of that, it will be no problem in the future given 1.083 will open the files anyway, looking forward to the beta 2 release.


   
ReplyQuote
Share: