RAW Files for Proce...
 
Share:
Notifications
Clear all

2022-05-29: APP 2.0.0-beta2 has been released !

Release notes

Download links per platform:

windows 2.0.0-beta2

macOS x86_64 2.0.0-beta2

macOS arm64 M1 2.0.0-beta2

Linux DEB 2.0.0-beta2

Linux RPM 2.0.0-beta2

[Sticky] RAW Files for Processing Practice

Page 1 / 3

(@1cm69)
Red Giant Customer
Joined: 5 years ago
Posts: 133
Topic starter  

A very nice member on AstroBin is sharing their RAW data files to anyone to use to practice processing with.

Here's the AstroBin forum post:  https://www.astrobin.com/forum/c/astrophotography/other/all-my-raw-files-are-or-will-be-public-online/

Here's the direct link to the files:  https://drive.google.com/drive/folders/0B9-mgz7OQ4fnYzJHLTdwZzRldEU

 

enjoy

 

Edit by Mabula: made this a sticky 😉


artem liked
ReplyQuote
(@mabula-admin)
Quasar Admin
Joined: 5 years ago
Posts: 3158
 

Hi @1cm69,

It's data of Wei-Hao Wang (@wei-hao_wang) . Should be great for practice 😉 

Have you processed some of it yet?

Cheers,

Mabula

 


ReplyQuote
(@wei-hao_wang)
Brown Dwarf Customer
Joined: 5 years ago
Posts: 11
 

It will be fantastic if someone uses my data to demonstrate APP's capability!

Personally I am waiting for the support of Pentax 645z's raw files in APP, so I can try it myself.  All my major recent works were done with 645z.


ReplyQuote
(@1cm69)
Red Giant Customer
Joined: 5 years ago
Posts: 133
Topic starter  

Not as yet


ReplyQuote
(@mabula-admin)
Quasar Admin
Joined: 5 years ago
Posts: 3158
 
Posted by: Wei-Hao Wang

It will be fantastic if someone uses my data to demonstrate APP's capability!

Personally I am waiting for the support of Pentax 645z's raw files in APP, so I can try it myself.  All my major recent works were done with 645z.

Thanks Wei-Hao @wei-hao_wang for sharing your data. I will certain have a look soon and will try to process some of the Canon and/or Nikon data 😉

I understand that I can also download a lot of Pentax data from you now for testing and implementing support for Pentax. That's great 😉

Mabula


ReplyQuote
(@1cm69)
Red Giant Customer
Joined: 5 years ago
Posts: 133
Topic starter  

I finally got around to trying some of the data.

This particular image M16 from the Canon 5D2 folder [2011.07.31]

I had issues attempting to use the included MasterFlat dated 2011.07.30, so I decided against using it.

This image is just the Lights & Darks, all proceesed in APP except the diffraction spikes that I added in PS.

 

M16

ReplyQuote
(@1cm69)
Red Giant Customer
Joined: 5 years ago
Posts: 133
Topic starter  

And another, NGC253 from the Nikon D800 folder [2016.11.04]

Issue with the associated LED Flats again, these left a green vignette on the image but luckily there was enough room to crop it out.

Hard to get too much detail here as the image is quite small in the frame & there were not many Light frame which didn't help with noise.

Once again, all done in APP except diffraction spikes.

 

combine RGB image mod cbg csc St SC St

ReplyQuote
(@wei-hao_wang)
Brown Dwarf Customer
Joined: 5 years ago
Posts: 11
 

Hi,

Your M16 is better than my old processing. It looks beautiful. There should be at least two nights of data for NGC253, separated by a few years and both with D800, including some H-alpha. If you can identify both sets of NGC253 images, you should be able to achieve higher S/N.  The second year of NGC253 images were taken under poor seeing, however.

I am surprised to hear about the flat issues for both images. I had no problem flattening them. The first set of NGC253 images were taken with D800 without a firmware hack to restore the true dark level and bias level. So there will be residual large-scale pattern in the image background. However, this can't be seen without extremely strong stretch (much much stronger then what this NGC253 image requires). So it shouldn't be the reason for your problems in flat-fielding.  Did you use the bias images in the calibration?  If you use the correct set of biases, flats, and darks, the stacked image should be very flat.  

For 5D2, I used to skip biases.  This works because DeepSkyStacker automatically subtracts the 1024 mean bias level in 5D2 files. (I believe DSS only does this on very early cameras.) So I was lucky.  After I realized this, I started to take biases all the time, so the images can be calibrated even if DSS no longer does such magic thing, or if the images are to be calibrated with other programs.  For the M16 images, you won't find bias files on the date when they were taken.  Please go to the bias library under the 5D2 folder. Those are biases I took quite many years later, but they should still work.

Cheers,

Wei-Hao


ReplyQuote
(@mabula-admin)
Quasar Admin
Joined: 5 years ago
Posts: 3158
 
Posted by: 1CM69

I finally got around to trying some of the data.

This particular image M16 from the Canon 5D2 folder [2011.07.31]

I had issues attempting to use the included MasterFlat dated 2011.07.30, so I decided against using it.

This image is just the Lights & Darks, all proceesed in APP except the diffraction spikes that I added in PS.

 

M16

Looks great indeed @1cm69 😉

What happened with the masterflat of @wei-hao_wang ?  I bet it's probably incompatible for dimensions, right?

This has to do with how DCRAW and APP handles CR2 images (and other DSLR Raws).

  • APP uses the whole sensor for calibration which, I think, is the most logical thing to do.
  • Applications that use DCRAW like DSS, clearly don't do this, Besides that, the provided image dimensions are DCRAW specific as well, not according to the camera's model specifiication even.

 

Wei Hao's masterflat can be corrected though (have done it) but that's a bit complicated. The batch modify tool is needed to add some extra pixels to the borders of the original masterflat, then it should work.

Great image 😉

Wei-Hao, is this your original version?

https://www.astrobin.com/full/78394/0/

 

Mabula

 


ReplyQuote
(@mabula-admin)
Quasar Admin
Joined: 5 years ago
Posts: 3158
 
Posted by: 1CM69

And another, NGC253 from the Nikon D800 folder [2016.11.04]

Issue with the associated LED Flats again, these left a green vignette on the image but luckily there was enough room to crop it out.

Hard to get too much detail here as the image is quite small in the frame & there were not many Light frame which didn't help with noise.

Once again, all done in APP except diffraction spikes.

 

combine RGB image mod cbg csc St SC St

Looks great as well @1cm69 😉

So the masterflat dimesions were compatible here? I will have to test this myself probably. If the masterflat isn't working well, it must have to do with the bias pedestal in both lights and flat calibration.

Mabula


ReplyQuote
(@mabula-admin)
Quasar Admin
Joined: 5 years ago
Posts: 3158
 
Posted by: Wei-Hao Wang

Hi,

Your M16 is better than my old processing. It looks beautiful. There should be at least two nights of data for NGC253, separated by a few years and both with D800, including some H-alpha. If you can identify both sets of NGC253 images, you should be able to achieve higher S/N.  The second year of NGC253 images were taken under poor seeing, however.

I am surprised to hear about the flat issues for both images. I had no problem flattening them. The first set of NGC253 images were taken with D800 without a firmware hack to restore the true dark level and bias level. So there will be residual large-scale pattern in the image background. However, this can't be seen without extremely strong stretch (much much stronger then what this NGC253 image requires). So it shouldn't be the reason for your problems in flat-fielding.  Did you use the bias images in the calibration?  If you use the correct set of biases, flats, and darks, the stacked image should be very flat.  

For 5D2, I used to skip biases.  This works because DeepSkyStacker automatically subtracts the 1024 mean bias level in 5D2 files. (I believe DSS only does this on very early cameras.) So I was lucky.  After I realized this, I started to take biases all the time, so the images can be calibrated even if DSS no longer does such magic thing, or if the images are to be calibrated with other programs.  For the M16 images, you won't find bias files on the date when they were taken.  Please go to the bias library under the 5D2 folder. Those are biases I took quite many years later, but they should still work.

Cheers,

Wei-Hao

Hi Wei-Hao @wei-hao_wang,

It's a very nice M16 indeed 😉

I am surprised to hear about the flat issues for both images. I had no problem flattening them. The first set of NGC253 images were taken with D800 without a firmware hack to restore the true dark level and bias level. So there will be residual large-scale pattern in the image background. However, this can't be seen without extremely strong stretch (much much stronger then what this NGC253 image requires). So it shouldn't be the reason for your problems in flat-fielding. Did you use the bias images in the calibration? If you use the correct set of biases, flats, and darks, the stacked image should be very flat.

Regarding the flat-field calibration issues. I think I need to have a go at your data as well.

I am currenty working on a big upgrade of APP's calibration engine. The current calibration rules in APP are too strict and quite a few users have trouble in getting good calibration. So I am making it more flexible and smarter. Currently, bias, dark, flat calibration requires matching of ISO/gain and exposure (for darks and darkflats). I will let those requirements go and I will also introduce dark scaling to APP.

For 5D2, I used to skip biases. This works because DeepSkyStacker automatically subtracts the 1024 mean bias level in 5D2 files. (I believe DSS only does this on very early cameras.) So I was lucky. After I realized this, I started to take biases all the time, so the images can be calibrated even if DSS no longer does such magic thing, or if the images are to be calibrated with other programs. For the M16 images, you won't find bias files on the date when they were taken. Please go to the bias library under the 5D2 folder. Those are biases I took quite many years later, but they should still work.

DSS uses dcraw, and dcraw subtracted that 1024 value. This is the internal black point of the camera. It's similat to an offset in CCD's.  It's in 14bits as well.

The internal black point is not the same for all Canon Camera's, some have a value of 2048 for instance. 

APP's processing of Canon CR2s never subtracts this internal black point, so both lights and flats need to have this offset subtracted by using either bias or darks. But you could also subtract a value of 1024 from the frames for instance (batch modify tool can add/subtract an offset) .

@1cm69, if you can find the bias, like Wei-Hao indicates, subtract those from the lights 😉

Cheers,

Mabula


ReplyQuote
(@1cm69)
Red Giant Customer
Joined: 5 years ago
Posts: 133
Topic starter  

I’ll have another look at this and possibly list the particular folder/files that I used for each image just to see if I am missing anything. 

The indexing system for the data took a bit of getting used too but I did think I understood it. 

In the M16 image it was that the MF size was incompatible & APP kept spitting out a warning message, so Flat never actually got used. 

For NGC253 there were no Flats for the Ha data, it stated on the log file that they were forgotten so I never had any to use for the Ha integration. 


ReplyQuote
(@1cm69)
Red Giant Customer
Joined: 5 years ago
Posts: 133
Topic starter  

More than likely, this is an error on my part with not processing the calibration files in either the correct order or along with the correct other calibration files. 

Heres a quick lowdown of what I do & the order I do it. 

1. Load Bias to create MB

2. Clear Load

3. Load Flats & MB (matching ISO) to create MF

4. Clear Load

5. Load Darks to create MD

6. Clear Load

7. Load MD & MF to create BPM

8. Clear Load

9. Load Lights, MF, MD & BPM

10. Calibrate & continue further steps. 

Am I incorrect in my procedure?

i know that the Darks in step 5 have to match the Lights in step 9 in terms of ISO, Exposure Length & Temperature. 


ReplyQuote
(@1cm69)
Red Giant Customer
Joined: 5 years ago
Posts: 133
Topic starter  

OK, I have just had another go at processing this data & I do not understand what is going wrong.

Every set of data I have tried to process in APP has shown major issues with the Flats.

I am following Mabula's @mabula-haverkamp-administrator steps here:  https://www.astropixelprocessor.com/a-dslr-data-calibration-workflow/ to the letter and nothing is working.

For this latest attempt I am using data from the folder - NIKON_D800 > 2016 > 20161106

and as specified in the LOG for this data I am using NEF files:

6111 - 6122 for BIAS ISO800 1/8000s

6281 - 6312 for FLAT ISO ISO800 1/20s

6323 - 6334 for Dark ISO800 180s

6262 - 6271 for LIGHT ISO800 180s

This is how I process:

  1. Load F & D to create BPM (this also creates MF & MD which I delete) then clear the load
  2. load F, B & BPM to create MF - then clear the load
  3. Load D to create MD - then clear load
  4. Load L, BPM, MF & MD and hit Calibrate

At this stage I took a screenshot of the Linear data & also the Calibrated data:

lin

 

cal

 

You can easily see the vignette in the Linear data but this should disappear in the Calibrated data but it doesn't, it just changes colour.

 

So what is going on??


ReplyQuote
(@wei-hao_wang)
Brown Dwarf Customer
Joined: 5 years ago
Posts: 11
 

It is very weird.  The attached screenshot is what I got with PixInsight using exactly the same set of files, with standard calibration and stacking workflow without any customized procedure.

There must be something seriously wrong.

Screen Shot 2018 03 30 at 1.55.24 PM

BTW, I was never able to complete this mosaic. Only very few panels were collected. So you are going to be disappointed if you wish to compose a mosaic using this set of images. I even did not try to process any of the images until now. Now seeing this, I think I probably should try to complete it.


ReplyQuote
(@1cm69)
Red Giant Customer
Joined: 5 years ago
Posts: 133
Topic starter  
Posted by: Wei-Hao Wang

It is very weird.  The attached screenshot is what I got with PixInsight using exactly the same set of files, with standard calibration and stacking workflow without any customized procedure.

There must be something seriously wrong.

Screen Shot 2018 03 30 at 1.55.24 PM

BTW, I was never able to complete this mosaic. Only very few panels were collected. So you are going to be disappointed if you wish to compose a mosaic using this set of images. I even did not try to process any of the images until now. Now seeing this, I think I probably should try to complete it.

So this is PixInsight output using the exact files I listed that I had used in APP?

 

It would be interesting to see your output after running the exact same files through APP, no post processing, just finish at integration.

 

I did try this data a second time but this time I just loaded all the files at once in to APP & ran straight through to integration, it produced exactly the same result as my initial run.

Either I am missing something or there is a major FLATs issue with APP.

 

Odd thing is that I am sure that I have used data with Flats in the past when testing APP before buying & all was OK, now nothing seems to work.


ReplyQuote
(@1cm69)
Red Giant Customer
Joined: 5 years ago
Posts: 133
Topic starter  

Right, I have just had a third attempt at this data & this time I left all the Master files to AVERAGE without setting any below 20 files to MEDIAN in the CALIBRATE tab.

I thought that maybe there was something here causing an issue but no, I still got the exact result as shown in my first post above.


ReplyQuote
(@wei-hao_wang)
Brown Dwarf Customer
Joined: 5 years ago
Posts: 11
 

Right.  The image I showed used exactly the same files described by your file numbers.  I even didn't look at my own log to figure out whether they are correct.  I simply use the file numbers you described.  I use PI's BPP for calibration and then straight integration, all with basic setting and nothing else.  The displayed image was stretched using the automatic screen transfer function to enhance the brightness and contrast.  No additional post-processing was applied.

The situation here is quite weird.


ReplyQuote
(@1cm69)
Red Giant Customer
Joined: 5 years ago
Posts: 133
Topic starter  
Posted by: Wei-Hao Wang

Right.  The image I showed used exactly the same files described by your file numbers.  I even didn't look at my own log to figure out whether they are correct.  I simply use the file numbers you described.  I use PI's BPP for calibration and then straight integration, all with basic setting and nothing else.  The displayed image was stretched using the automatic screen transfer function to enhance the brightness and contrast.  No additional post-processing was applied.

The situation here is quite weird.

I am not a PI user so have no understanding of the processes involved when using it but could you do me a favour and process the same files in APP so we could compare results.

You only need to go as far as CALIBRATE

Click on any one of the LIGHTs in APP's file list to load the image into the Image Viewer making sure the 'linear (l)' option is chosen, then take a screen shot.

Then just change the Image Viewer dropdown to 'l-calibrated' and when the calibrated image loads in the viewer, take another screen shot.

Can you then post up both screen shots.

Thanks..,

Kirk


ReplyQuote
(@wei-hao_wang)
Brown Dwarf Customer
Joined: 5 years ago
Posts: 11
 

Ah, unfortunately I ended my role in the APP beta test team because I don’t have much time to test it. So I no longer have a working APP installed in my computer. Someone else here might help.


ReplyQuote
Page 1 / 3
Share: