Help needed, still ...
 
Share:

APP 1.075 has been released !

2019 September: Astro Pixel Processor and iTelescope.net celebrate a new Partnership!

Help needed, still can't get Flats to work correctly  

  RSS

(@skestergmail-com)
White Dwarf Customer
Joined: 1 year ago
Posts: 20
November 12, 2018 15:46  

I posted about my issue a few weeks ago and thought I had figured it out.  But based on my latest processing attempts clearly I have not.  My issue is when I include the flat files I took at the same time I captured my lights they are over correcting the image.  Dust motes go from being dark spots to bright spots, and the vignetting in the corners also goes from being dark to bright.  I have gone through the complete calibration workflow multiple times and the results are always the same.  I tried creating the master flat first, calibrated with the darks, bias and bad pixel map.  Then I tried all the files at the same time.  No matter what I do the final image looks like the master flat is being applied either to strongly, or more than once.

Below is an integration that shows the image results when the flats are included:

https://ibb.co/kBvGCq

I have uploaded all data used to create the image in a tar.gz format.  Included are the lights, darks, flats, master bias and bad pixel map.  If I can't get to the bottom of this issue I will be forced to use something other than APP to process my images.  Thanks for any help.

https://www.dropbox.com/s/i548873sijs1140/M33_data.tar.gz?dl=0

This topic was modified 12 months ago 3 times by Scott

ReplyQuote
(@skestergmail-com)
White Dwarf Customer
Joined: 1 year ago
Posts: 20
November 13, 2018 05:43  

Update, I tried stacking my data with DSS, and got the same result as the APP image, washed out corners and the dust motes are bright spots.  I also took some test flats today via SGP saved as fits files to eliminate that as a possible cause of the issue.  Again I got the same results, with the flats 'over correcting' the lights.  Clearly the issue is not related to APP, its something with my data or the flats themselves.

My flats exposure was set to place the image histogram at around 50%, which is what I have read is correct.  I would appreciate any ideas on what is causing my issues.


ReplyQuote
(@vincent-mod)
Galaxy Moderator
Joined: 2 years ago
Posts: 928
November 20, 2018 23:48  

Sorry for the delay Scott, had my parents come over to New Zealand so was out of the usual forum activity. 😉 Yes, thanks for analyzing it a bit further yourself. If other programs show the same, it must be the data indeed (I would have expected that already as there doesn't seem to be a huge problem or bug for the flat workflow). 50% of the histogram should be correct yes, given you look at a linear histogram. So in SGP for example (which always works on 16-bit no matter what camera you use) you aim for an ADU value of about 30.000. Is that what you do?


ReplyQuote
(@skestergmail-com)
White Dwarf Customer
Joined: 1 year ago
Posts: 20
November 25, 2018 23:52  

Vincent, thanks for the reply.  I believe the entire issue was due to Sequence Generator Pro saving the lights as .fits files (default setting), while my flats were stored as Nikon Raw .NEF format.  I found out over at the CN forum that I could set SGP to save all images as .NEF instead of .fits.  I imaged M15 and saved my lights, darks and flats as .NEF.  I had no problem in APP with the flats and this new dataset, so problem solved going forward.

I do have another question you might be able to answer.  When I take my flats and look at the back of camera histogram all 3 color channels are similar at around 50%, and the image is a mid gray color.  However, when I read those files into APP, the red is always at a much lower value (25%), blue in the middle and green at close to 75%.  The images render in APP as green/blue, instead of gray as expected.  I assume the difference comes down to APP displaying linear data, while the camera applies some type of gamma correction that raises the red levels?  Do you know what is going on and is this common? 

Thanks,

Scott

 


(@vincent-mod)
Galaxy Moderator
Joined: 2 years ago
Posts: 928
December 19, 2018 02:53  

I'm repeating myself way too often lately, but sorry for the delay again.

I think your assumption might be correct yes, the camera is displaying the data in a non-linear (I believe logarithmic) way and differences between the channels might not pop up then.

This post was modified 11 months ago 2 times by Vincent Groenewold - Moderator

ReplyQuote
(@astrogee)
Red Giant Customer
Joined: 1 year ago
Posts: 52
December 19, 2018 23:38  
Posted by: Scott

Vincent, thanks for the reply.  I believe the entire issue was due to Sequence Generator Pro saving the lights as .fits files (default setting), while my flats were stored as Nikon Raw .NEF format.  I found out over at the CN forum that I could set SGP to save all images as .NEF instead of .fits.  I imaged M15 and saved my lights, darks and flats as .NEF.  I had no problem in APP with the flats and this new dataset, so problem solved going forward.

I do have another question you might be able to answer.  When I take my flats and look at the back of camera histogram all 3 color channels are similar at around 50%, and the image is a mid gray color.  However, when I read those files into APP, the red is always at a much lower value (25%), blue in the middle and green at close to 75%.  The images render in APP as green/blue, instead of gray as expected.  I assume the difference comes down to APP displaying linear data, while the camera applies some type of gamma correction that raises the red levels?  Do you know what is going on and is this common? 

Thanks,

Scott

 

You might try Rawtherapee to see how it looks relative to your camera and APP. Rawtherapee can show you the raw (linear) histogram and the non-linear histogram. I'd be interested to know what you find -cheers


(@skestergmail-com)
White Dwarf Customer
Joined: 1 year ago
Posts: 20
December 21, 2018 17:43  

astrogee,

I did as you suggested and used Rawtherapee to view the raw histogram for one of my flat files.  It does indeed match that displayed by APP, with the green channel showing a significantly higher level than blue, which was again higher than red.  When I view the standard (non-raw) histogram with Rawtherapee the curve matches that displayed by the camera with red/green/blue all displaying identical values and a perfect gray tone.  

So I think we can conclude that APP is doing exactly what it should and that the green/blue tone of my flats is purely down to the difference between a linear view of the data, and the gama corrected view displayed by the camera and most image viewers.


ReplyQuote
(@astrogee)
Red Giant Customer
Joined: 1 year ago
Posts: 52
December 21, 2018 22:44  

Hi Scott. I’m having similar overcorrection with flats. Are you saying you fixed it by using raw for lights and flats?


ReplyQuote
(@skestergmail-com)
White Dwarf Customer
Joined: 1 year ago
Posts: 20
December 22, 2018 01:09  

astrogee, I determined the issue I was having was down to having stored my light frames as .fits files, while the flat frames were Nikon .NEF files.  I shot the flats directly with the camera, while the lights were taken via Sequence Generator Pro, which stored the lights in .fits format by default. 

I changed the setting in SGP to store all images in .NEF format and have had no issue with flats calibration since. I can only conclude that using two different file formats for lights and flats was the cause of the over correction I was having with flats calibration.


astrogee liked
ReplyQuote
(@vincent-mod)
Galaxy Moderator
Joined: 2 years ago
Posts: 928
December 22, 2018 02:47  

Shouldn't be the case I think, but it's always a good idea to have the raw data without intervening software conversion loaded into APP. SGP does have a very nice flats calibration wizard which I like a lot. Aim for at least a second exposure time per flat-sub and using a very even illuminated flat-field.


ReplyQuote
(@tomzdk)
Brown Dwarf Customer
Joined: 2 years ago
Posts: 9
August 31, 2019 14:49  

Hi Guys....SAME issue here with overexposed Flats (APP version 1.074.1)

Last night i was doing a small test with: 10 Lights - 20 flats - 20 Darks - 120 Bias frames.  Done with my Canon 6D (astromod) and a 24mm lens - Same night, same ISO and same temperature.....(RAW files in a row)

I made masters i 2 different ways:

1): Loaded all the BIAS - and created af MasterBIAS. Then loaded all the Flats, Darks and MB - and made all the the calibration frames including BPM. 

2):  Loaded ALL Bias, Flats, Darks and Created Master Calibration files (MF - MD - MB - BPM)

Now i Ran the process with calibrating my lights (both Calibration setups, with Novariation in results)

1): With MasterFlat:

With flats

2): Without MasterFlat:

Without Flats

Here is a 30% stretched BW converted MasterFlat:

Master flat BW

 

Advice Needed..... Thank you in advance

Thomas

This post was modified 2 months ago by TomzDK

ReplyQuote
(@vincent-mod)
Galaxy Moderator
Joined: 2 years ago
Posts: 928
August 31, 2019 15:13  

The flat isn't right, it's better to leave it out then (also doesn't look overexposed, but taken on a very uneven light source). As you can see without it looks better and using the Light Pollution Correction tool (tab 9) can help you a lot then.

This post was modified 2 months ago by Vincent Groenewold - Moderator

ReplyQuote
(@b4silio)
White Dwarf Customer
Joined: 1 year ago
Posts: 10
August 31, 2019 15:47  

Your flat looks wonky. Did you use the white t-shirt method or did you just point at the sky? If the t-shirt isn't flattened out then you might have those blotches. The main reason to use flats is to get rid of vignetting (if you don't have too much dust on your scope or sensor), so you can go back and retake new flats, making sure they are, indeed, flat.

But as Vincent said (and from what I can see in your non flat image) you can just do a remove light pollution pass.


ReplyQuote
(@vincent-mod)
Galaxy Moderator
Joined: 2 years ago
Posts: 928
August 31, 2019 16:32  

I actually didn't have a telescope at hand in NZ, so I also resorted to using my wide lens and DSLR. I wanted good flats ofcourse and it was very difficult to get those right even with shirt and moving the lens around to blur any unevenness. Eventually I succeeded, this way: 🙂

IMG 0747

With ofcourse that white piece of shirt tightly wrapped on the front of the lens. So light bouncing of of an as even object as possible. I got similar flats like Tom when I did it on the sky with clouds.


ReplyQuote
(@tomzdk)
Brown Dwarf Customer
Joined: 2 years ago
Posts: 9
August 31, 2019 17:26  

Hi Guys...... Back again.

Made some NEW flats from another Flat-panel........... *BINGO ! (Something most have been way of last night with my little portable Flat-panel.... but could be anything, actually)

New Flat:

New Master Flat image BW

 

After NEW flats was added the process done once again:

NEW Flats example

 

(please add your own Homer Simpson outburst) Thanks Again

 

Greetings from Denmark

 Thomas


ReplyQuote
(@vincent-mod)
Galaxy Moderator
Joined: 2 years ago
Posts: 928
August 31, 2019 17:29  

Doh! 😉 But sorry to say, they still aren't ok. As you can see in the corrected photo, there are dust particles visible which should have been removed if the flat is correct. Did you have the exact same focus as when you took the lights?

I also took the liberty of removing most of the light pollution.. 😉

Screenshot 2019 08 31 at 16.34.47

This post was modified 2 months ago by Vincent Groenewold - Moderator

ReplyQuote
(@tomzdk)
Brown Dwarf Customer
Joined: 2 years ago
Posts: 9
August 31, 2019 21:09  

@vincent-mod

Thanks..... But I noticed these Black spots and cleaned my sensor earlier today - before making New flats. Seemes like this turned out well 😉.

Time to proceed and have a look on My portabel flatbox. 

Thank you again 👍

 

Thomas


(@b4silio)
White Dwarf Customer
Joined: 1 year ago
Posts: 10
August 31, 2019 21:48  

Turns out your new flats are TOO clean this time! 😀

Looking a lot better and as Vincent did LPR should take care of most. For the dust spots you can always fix that in photoshop. 


ReplyQuote
(@vincent-mod)
Galaxy Moderator
Joined: 2 years ago
Posts: 928
September 1, 2019 00:15  

haha well glad it's sorted!


ReplyQuote
Share: