JWST data processing

Hi! I have to say I never looked at it, I'll download the data and will see if that is possible at all.

Hi Vincent, have you been able to look at the data yet? Or download it? Because the download link will expire in the next few days.

With best regards.

I did, sorry I forgot to reply. 😉 So I don't think these files can actually be used properly, it seems like they have been processed already with very black background values etc. Are these the actual raw files?

These are calibrated and stacked master files in linear state. This is how they are provided by NASA.

Right, so I think the calibration is where it gets very different from our workflow, are there more examples I can have a look at?

Hello Vincent, under the following link ( https://mast.stsci.edu/portal/Mashup/Clients/Mast/Portal.html), you can go directly to the portal to download more images. There you will also find images of the Hubble Telescope and other projects.
Some of the images I uploaded have an enormous dynamic range of 64 bit. What I found out during my research.

With best regards.

Right, my guess is that might also cause an issue perhaps. It's completely new to me, so bear with me. 🙂

Hi @minusman,

Perhaps you can upload some of the files? Then we can test and analyse and hopefully make it possible to process them well with APP ?

https://upload.astropixelprocessor.com/

username: uploadData

password: uploadTestData

Please make folder like minusman-JWST-data

and let me know once uploaded, I have opened an issue on our todo list to check what's happening 😉

Mabula

Hello, I have uploaded a few sample fits files from the JWST. These are parts of the CERRS survey.
The folder has the suggested name.
I am curious if the data can be processed.

With best regards.

Dear @minusman,

Thanks for the upload, I had to finish work on another issue which is now solved, I will look at this soon now 😉

Mabula

Hi @minusman,

I am looking at the data, first of all these are not exactly image files but image archives in FITS format. The first image is 64bits and not normalized as we would expect, which I can fix, then the file hlsp_ceers_jwst_nircam_nircam1_f115w_v0.5_i2d.fits loads like this in APP, this is the first 64bits image in the archive

This is not suitable I think for data processing on it. I I will try if I can have APP to show the other images, which are 32bits...

From the header you can see that there are many images in there:

FITS HDUs: 13
HDU1 - SIMPLE = T / conforms to FITS standard
HDU1 - BITPIX = 8 / array data type
HDU1 - NAXIS = 0 / number of array dimensions
HDU1 - EXTEND = T
HDU1 - DATE = '2022-10-06T19:19:36.002' / UTC date file created
HDU1 - ORIGIN = 'STSCI ' / Organization responsible for creating file
HDU1 - TIMESYS = 'UTC ' / principal time system for time

...

A normal FITS image only has 1 HDU, this one has 13 ! The second HDU contains the first 64 image, then the 3rd will contain a 32 bits image

...

HDU2 - END
HDU3 - XTENSION= 'IMAGE ' / Image extension
HDU3 - BITPIX = -32 / array data type
HDU3 - NAXIS = 2 / number of array dimensions
HDU3 - NAXIS1 = 10600
HDU3 - NAXIS2 = 4800
HDU3 - PCOUNT = 0 / number of parameters
HDU3 - GCOUNT = 1 / number of groups
HDU3 - EXTNAME = 'SCI ' / extension name
HDU3 - MJD-BEG = 59751.7567899221 / [d] exposure start time in MJD
HDU3 - MJD-AVG = 59751.80492383742 / [d] exposure mid-point in MJD
HDU3 - MJD-END = 59751.85247748843 / [d] exposure end time in MJD
HDU3 - TDB-BEG = 59751.75638807951 / [d] TDB time of exposure start in MJD
HDU3 - TDB-MID = 59751.8045247492 / [d] TDB time of exposure mid-point in MJD
HDU3 - TDB-END = 59751.85208112191 / [d] TDB time of exposure end in MJD
HDU3 - XPOSURE = 49475.03999999999 / [s] Effective exposure time
HDU3 - TELAPSE = 49475.03999999999 / [s] Total elapsed exposure time

...

If you can find fits images to process and not fits archives like this, things will be much easier for sure. This would need a lot of labour to have APP interpret all image data in such an archive and add those separate images to the frame list.

Mabula

Hallo Mabula, das HUD 2 sollte das richtige Bild sein. Bei den anderen im Container enthaltenen Dateien handelt es sich um Kalibrierungsdaten aus der NASA-Verarbeitungspipeline.

( https://iopscience.iop.org/article/10.3847/2041-8213/acbb08)

Diese sind immer im Container verpackt. Dies ist auch bei Daten anderer Teleskope wie Hubble der Fall. Die einzige Möglichkeit, den Fitscontainer zu öffnen, besteht darin, die Endung „.gz“ zu entfernen und ihn dann in pixinsight zu öffnen. Leider hat es in der APP nicht funktioniert.

Das erste Problem besteht darin, dass das Histogramm in den normalisierten 0-1-Raum eingeschnitten werden muss. Die JWST-Bilder haben alle negative Werte bis 1.

Das zweite Problem sind die Grauwerte rund um die eigentlichen Bildinformationen. Diese verwirren den Vorschaufilter und werden daher falsch gestreckt. Ich weiß nicht, warum die NASA das tut.

Ich habe kürzlich einen Vortrag im TIAC gesehen, in dem gezeigt wurde, wie die sogenannten hübschen Bilder entstehen. Das Verfahren besteht darin, die Bilder in Fits Liberator zu strecken und sie dann in Photoshop zu kombinieren und weiter zu verarbeiten, um ein Farbbild zu erstellen. Ich hatte gehofft, dies alles in der APP zu erledigen, da alle Tools enthalten sind. Leider erwiesen sich die Bilddaten selbst als problematisch. Ich hoffe, dass das gelöst werden kann.

Nochmals vielen Dank für die Info, Mabula.

Hi @minusman, these files are old and when extracted, they even do not have a file extension. Simply extend them with .fits and APP will read and process them.

I would suggest to crop the data that you want to use and use the add/multiply Batch tool to correct the data ranges if needed. Studying the data range present in the files is easies with for isntance FITS liberator I think.

Mabula