Help needed, still can't get Flats to work correctly  

  RSS

(@skestergmail-com)
White Dwarf Customer
Joined: 5 months ago
Posts: 17
November 12, 2018 15:46  

I posted about my issue a few weeks ago and thought I had figured it out.  But based on my latest processing attempts clearly I have not.  My issue is when I include the flat files I took at the same time I captured my lights they are over correcting the image.  Dust motes go from being dark spots to bright spots, and the vignetting in the corners also goes from being dark to bright.  I have gone through the complete calibration workflow multiple times and the results are always the same.  I tried creating the master flat first, calibrated with the darks, bias and bad pixel map.  Then I tried all the files at the same time.  No matter what I do the final image looks like the master flat is being applied either to strongly, or more than once.

Below is an integration that shows the image results when the flats are included:

https://ibb.co/kBvGCq

I have uploaded all data used to create the image in a tar.gz format.  Included are the lights, darks, flats, master bias and bad pixel map.  If I can't get to the bottom of this issue I will be forced to use something other than APP to process my images.  Thanks for any help.

https://www.dropbox.com/s/i548873sijs1140/M33_data.tar.gz?dl=0

This topic was modified 3 months ago 3 times by Scott

ReplyQuote
(@skestergmail-com)
White Dwarf Customer
Joined: 5 months ago
Posts: 17
November 13, 2018 05:43  

Update, I tried stacking my data with DSS, and got the same result as the APP image, washed out corners and the dust motes are bright spots.  I also took some test flats today via SGP saved as fits files to eliminate that as a possible cause of the issue.  Again I got the same results, with the flats 'over correcting' the lights.  Clearly the issue is not related to APP, its something with my data or the flats themselves.

My flats exposure was set to place the image histogram at around 50%, which is what I have read is correct.  I would appreciate any ideas on what is causing my issues.


ReplyQuote
(@vincent-mod)
Black Hole Moderator
Joined: 2 years ago
Posts: 264
November 20, 2018 23:48  

Sorry for the delay Scott, had my parents come over to New Zealand so was out of the usual forum activity. 😉 Yes, thanks for analyzing it a bit further yourself. If other programs show the same, it must be the data indeed (I would have expected that already as there doesn't seem to be a huge problem or bug for the flat workflow). 50% of the histogram should be correct yes, given you look at a linear histogram. So in SGP for example (which always works on 16-bit no matter what camera you use) you aim for an ADU value of about 30.000. Is that what you do?


ReplyQuote
(@skestergmail-com)
White Dwarf Customer
Joined: 5 months ago
Posts: 17
November 25, 2018 23:52  

Vincent, thanks for the reply.  I believe the entire issue was due to Sequence Generator Pro saving the lights as .fits files (default setting), while my flats were stored as Nikon Raw .NEF format.  I found out over at the CN forum that I could set SGP to save all images as .NEF instead of .fits.  I imaged M15 and saved my lights, darks and flats as .NEF.  I had no problem in APP with the flats and this new dataset, so problem solved going forward.

I do have another question you might be able to answer.  When I take my flats and look at the back of camera histogram all 3 color channels are similar at around 50%, and the image is a mid gray color.  However, when I read those files into APP, the red is always at a much lower value (25%), blue in the middle and green at close to 75%.  The images render in APP as green/blue, instead of gray as expected.  I assume the difference comes down to APP displaying linear data, while the camera applies some type of gamma correction that raises the red levels?  Do you know what is going on and is this common? 

Thanks,

Scott

 


(@vincent-mod)
Black Hole Moderator
Joined: 2 years ago
Posts: 264
December 19, 2018 02:53  

I'm repeating myself way too often lately, but sorry for the delay again.

I think your assumption might be correct yes, the camera is displaying the data in a non-linear (I believe logarithmic) way and differences between the channels might not pop up then.

This post was modified 2 months ago 2 times by Vincent Groenewold - Moderator

ReplyQuote
(@astrogee)
White Dwarf Customer
Joined: 7 months ago
Posts: 20
December 19, 2018 23:38  
Posted by: Scott

Vincent, thanks for the reply.  I believe the entire issue was due to Sequence Generator Pro saving the lights as .fits files (default setting), while my flats were stored as Nikon Raw .NEF format.  I found out over at the CN forum that I could set SGP to save all images as .NEF instead of .fits.  I imaged M15 and saved my lights, darks and flats as .NEF.  I had no problem in APP with the flats and this new dataset, so problem solved going forward.

I do have another question you might be able to answer.  When I take my flats and look at the back of camera histogram all 3 color channels are similar at around 50%, and the image is a mid gray color.  However, when I read those files into APP, the red is always at a much lower value (25%), blue in the middle and green at close to 75%.  The images render in APP as green/blue, instead of gray as expected.  I assume the difference comes down to APP displaying linear data, while the camera applies some type of gamma correction that raises the red levels?  Do you know what is going on and is this common? 

Thanks,

Scott

 

You might try Rawtherapee to see how it looks relative to your camera and APP. Rawtherapee can show you the raw (linear) histogram and the non-linear histogram. I'd be interested to know what you find -cheers


(@skestergmail-com)
White Dwarf Customer
Joined: 5 months ago
Posts: 17
December 21, 2018 17:43  

astrogee,

I did as you suggested and used Rawtherapee to view the raw histogram for one of my flat files.  It does indeed match that displayed by APP, with the green channel showing a significantly higher level than blue, which was again higher than red.  When I view the standard (non-raw) histogram with Rawtherapee the curve matches that displayed by the camera with red/green/blue all displaying identical values and a perfect gray tone.  

So I think we can conclude that APP is doing exactly what it should and that the green/blue tone of my flats is purely down to the difference between a linear view of the data, and the gama corrected view displayed by the camera and most image viewers.


ReplyQuote
(@astrogee)
White Dwarf Customer
Joined: 7 months ago
Posts: 20
December 21, 2018 22:44  

Hi Scott. I’m having similar overcorrection with flats. Are you saying you fixed it by using raw for lights and flats?


ReplyQuote
(@skestergmail-com)
White Dwarf Customer
Joined: 5 months ago
Posts: 17
December 22, 2018 01:09  

astrogee, I determined the issue I was having was down to having stored my light frames as .fits files, while the flat frames were Nikon .NEF files.  I shot the flats directly with the camera, while the lights were taken via Sequence Generator Pro, which stored the lights in .fits format by default. 

I changed the setting in SGP to store all images in .NEF format and have had no issue with flats calibration since. I can only conclude that using two different file formats for lights and flats was the cause of the over correction I was having with flats calibration.


astrogee liked
ReplyQuote
(@vincent-mod)
Black Hole Moderator
Joined: 2 years ago
Posts: 264
December 22, 2018 02:47  

Shouldn't be the case I think, but it's always a good idea to have the raw data without intervening software conversion loaded into APP. SGP does have a very nice flats calibration wizard which I like a lot. Aim for at least a second exposure time per flat-sub and using a very even illuminated flat-field.


ReplyQuote
Share: