Help needed, still can't get Flats to work correctly

Update, I tried stacking my data with DSS, and got the same result as the APP image, washed out corners and the dust motes are bright spots.  I also took some test flats today via SGP saved as fits files to eliminate that as a possible cause of the issue.  Again I got the same results, with the flats 'over correcting' the lights.  Clearly the issue is not related to APP, its something with my data or the flats themselves.

My flats exposure was set to place the image histogram at around 50%, which is what I have read is correct.  I would appreciate any ideas on what is causing my issues.

Sorry for the delay Scott, had my parents come over to New Zealand so was out of the usual forum activity. 😉 Yes, thanks for analyzing it a bit further yourself. If other programs show the same, it must be the data indeed (I would have expected that already as there doesn't seem to be a huge problem or bug for the flat workflow). 50% of the histogram should be correct yes, given you look at a linear histogram. So in SGP for example (which always works on 16-bit no matter what camera you use) you aim for an ADU value of about 30.000. Is that what you do?

Vincent, thanks for the reply.  I believe the entire issue was due to Sequence Generator Pro saving the lights as .fits files (default setting), while my flats were stored as Nikon Raw .NEF format.  I found out over at the CN forum that I could set SGP to save all images as .NEF instead of .fits.  I imaged M15 and saved my lights, darks and flats as .NEF.  I had no problem in APP with the flats and this new dataset, so problem solved going forward.

I do have another question you might be able to answer.  When I take my flats and look at the back of camera histogram all 3 color channels are similar at around 50%, and the image is a mid gray color.  However, when I read those files into APP, the red is always at a much lower value (25%), blue in the middle and green at close to 75%.  The images render in APP as green/blue, instead of gray as expected.  I assume the difference comes down to APP displaying linear data, while the camera applies some type of gamma correction that raises the red levels?  Do you know what is going on and is this common? 

Thanks,

Scott

I'm repeating myself way too often lately, but sorry for the delay again.

I think your assumption might be correct yes, the camera is displaying the data in a non-linear (I believe logarithmic) way and differences between the channels might not pop up then.

Posted by: Scott

Vincent, thanks for the reply.  I believe the entire issue was due to Sequence Generator Pro saving the lights as .fits files (default setting), while my flats were stored as Nikon Raw .NEF format.  I found out over at the CN forum that I could set SGP to save all images as .NEF instead of .fits.  I imaged M15 and saved my lights, darks and flats as .NEF.  I had no problem in APP with the flats and this new dataset, so problem solved going forward.

I do have another question you might be able to answer.  When I take my flats and look at the back of camera histogram all 3 color channels are similar at around 50%, and the image is a mid gray color.  However, when I read those files into APP, the red is always at a much lower value (25%), blue in the middle and green at close to 75%.  The images render in APP as green/blue, instead of gray as expected.  I assume the difference comes down to APP displaying linear data, while the camera applies some type of gamma correction that raises the red levels?  Do you know what is going on and is this common? 

Thanks,

Scott

 

You might try Rawtherapee to see how it looks relative to your camera and APP. Rawtherapee can show you the raw (linear) histogram and the non-linear histogram. I'd be interested to know what you find -cheers

astrogee,

I did as you suggested and used Rawtherapee to view the raw histogram for one of my flat files.  It does indeed match that displayed by APP, with the green channel showing a significantly higher level than blue, which was again higher than red.  When I view the standard (non-raw) histogram with Rawtherapee the curve matches that displayed by the camera with red/green/blue all displaying identical values and a perfect gray tone.  

So I think we can conclude that APP is doing exactly what it should and that the green/blue tone of my flats is purely down to the difference between a linear view of the data, and the gama corrected view displayed by the camera and most image viewers.

Hi Scott. I’m having similar overcorrection with flats. Are you saying you fixed it by using raw for lights and flats?

astrogee, I determined the issue I was having was down to having stored my light frames as .fits files, while the flat frames were Nikon .NEF files.  I shot the flats directly with the camera, while the lights were taken via Sequence Generator Pro, which stored the lights in .fits format by default. 

I changed the setting in SGP to store all images in .NEF format and have had no issue with flats calibration since. I can only conclude that using two different file formats for lights and flats was the cause of the over correction I was having with flats calibration.

Shouldn't be the case I think, but it's always a good idea to have the raw data without intervening software conversion loaded into APP. SGP does have a very nice flats calibration wizard which I like a lot. Aim for at least a second exposure time per flat-sub and using a very even illuminated flat-field.

Hi Guys....SAME issue here with overexposed Flats (APP version 1.074.1)

Last night i was doing a small test with: 10 Lights - 20 flats - 20 Darks - 120 Bias frames.  Done with my Canon 6D (astromod) and a 24mm lens - Same night, same ISO and same temperature.....(RAW files in a row)

I made masters i 2 different ways:

1): Loaded all the BIAS - and created af MasterBIAS. Then loaded all the Flats, Darks and MB - and made all the the calibration frames including BPM. 

2):  Loaded ALL Bias, Flats, Darks and Created Master Calibration files (MF - MD - MB - BPM)

Now i Ran the process with calibrating my lights (both Calibration setups, with Novariation in results)

1): With MasterFlat:

2): Without MasterFlat:

Here is a 30% stretched BW converted MasterFlat:

Advice Needed..... Thank you in advance

Thomas

The flat isn't right, it's better to leave it out then (also doesn't look overexposed, but taken on a very uneven light source). As you can see without it looks better and using the Light Pollution Correction tool (tab 9) can help you a lot then.

Your flat looks wonky. Did you use the white t-shirt method or did you just point at the sky? If the t-shirt isn't flattened out then you might have those blotches. The main reason to use flats is to get rid of vignetting (if you don't have too much dust on your scope or sensor), so you can go back and retake new flats, making sure they are, indeed, flat.

But as Vincent said (and from what I can see in your non flat image) you can just do a remove light pollution pass.

I actually didn't have a telescope at hand in NZ, so I also resorted to using my wide lens and DSLR. I wanted good flats ofcourse and it was very difficult to get those right even with shirt and moving the lens around to blur any unevenness. Eventually I succeeded, this way: 🙂

With ofcourse that white piece of shirt tightly wrapped on the front of the lens. So light bouncing of of an as even object as possible. I got similar flats like Tom when I did it on the sky with clouds.

Hi Guys...... Back again.

Made some NEW flats from another Flat-panel........... *BINGO ! (Something most have been way of last night with my little portable Flat-panel.... but could be anything, actually)

New Flat:

After NEW flats was added the process done once again:

(please add your own Homer Simpson outburst) Thanks Again

Greetings from Denmark

 Thomas

Doh! 😉 But sorry to say, they still aren't ok. As you can see in the corrected photo, there are dust particles visible which should have been removed if the flat is correct. Did you have the exact same focus as when you took the lights?

I also took the liberty of removing most of the light pollution.. 😉

@vincent-mod

Thanks..... But I noticed these Black spots and cleaned my sensor earlier today - before making New flats. Seemes like this turned out well 😉.

Time to proceed and have a look on My portabel flatbox. 

Thank you again 👍

Thomas

Turns out your new flats are TOO clean this time! 😀

Looking a lot better and as Vincent did LPR should take care of most. For the dust spots you can always fix that in photoshop. 

haha well glad it's sorted!