The case of overcorrecting flats, need help

I'll have a look, thanks for sharing the data.

@vincent-mod

Thank you. Just to save some time -- flats in the pack are of pretty short exposures. I will be able to upload a new session data hopefully within a couple of days (cloudy right now). Like I said, the problem persists no matter how those frames are taken, but just to eliminate these obvious causes, I'll try to retake them asap.

The immediate cause of the issue seem to be that the illumination profiles of light frames and flats do not match. But I do not know of any tools (graphs?) that could help to demonstrate it clearly and control for it. I've had people look at the data through PI tools and there's nothing that is obviously wrong with it. I was hoping darks and biases might be the culprit, but they seem ok.

My setup is pretty simple -- canon 60d, L-pro clip-in and 135mm f3.5 lens.

@igor_cheb I downloaded the data as well. One thing I noticed is that the bias were shot at ISO 400 and all other files at ISO 500. Is there a reason for that?

@wvreeven

@vincent-mod

Thank you for taking the time. No reason, just a mix up. I've reuploded the zip with correct biases. Link is the same. Apologies.

@igor_cheb I downloaded the zip again, and again the bias frames are ISO 400. Could you perhaps use our upload server? That way you don't have to upload a zip file, which would considerably reduce download times for us. Use upload for both username and password on

https://upload.astropixelprocessor.com/

and create a directory called igor_cheb_overcorrected_flat

@wvreeven

Have reuploaded to google drive (updated link in the original post) and am uploading to the APP server, igor_cheb_overcorrected_flat folder. Will take about an hour, speed is not great for some reason.

@igor_cheb Mabula is hosting the upload server himself. He has limited the upload speed to 1 Mb/sec so that's why it takes a while. Thanks a lot for making the data available. Please let us know when the upload has finished and we will have take look.

@wvreeven

upload done.

@igor_cheb Many thanks! I downloaded the data and in this data set the bias have ISO 500 as well. I am looking into this now.

@igor_cheb I can reproduce your result but don't know what the reason is for the over-correction. I have asked Mabula to take a look. He will do so as soon as he has time.

Thank you , hope it's solvable

Hi Igor @igor_cheb and Wouter @wvreeven,

I have checked the data and I also do not see any clear reason why the flats overcorrect.

Overcorrection will occur when:

- the vignetting profile in the flats is stronger than in the lights..

- or the sensor offset was different between between flats and the bias, or lights and the bias.

With this consumer camera Canon 60D, the sensor offset is fixed, so the problem must be found in how the lights and the flats were shot... something must be different which has caused the problem.

Igor, did you make changes to the optical train between having shot the lights and the flats?

Were both types shot with exactly the same optical train?

Was there no focus creep of the lens between creating the lights and the flats, so are you sure the focus did not change?

Shot with same aperture of the lens?

Were the flats taken at night or during the day? Using a flat panel or?

Mabula

@mabula-admin

@wvreeven 

Sorry for the delay and thank you for taking the time. The answer to all questions regarding light train is no -- no focus creep (very stiff grease in the lens), same gear used, same aperture. 

As for the methods used -- that particular set of flats was taken inside with a phone screen as light source and several sheets of paper as a diffuser. Since then I've done a few things.

I've replaced both the lens and the camera (modified 550d with Samyang 135 this time).

I've tried taking flats with various light sources (phone screen, white wall reflecting light, dim sky) and various diffusers (t-shirt, layered sheets of paper, macbook plasticky folding material, specialised diffusing material used for studio lights equipment). Checked for light leaks repeatedly. Inspected stretched flat frames with various lenses. Tried to rotate LP filter and take flat frames in various positions.

Have no substantial clues to report, the problem remains irrespective of flats acquisition approach. My conclusion so far is that the vignetting is not overcorrection, but a light pollution that happen to come at this peculiar angle making it seem like over correction by flats. This is bortle 9, so LP is to be expected.

The only thing that still seems suspicious to me is how master flats look after APP calibration -- very saturated. But perhaps a stretched flat always looks like that:

Couple of pictures I took in short sessions recently.

Navi star region in Cassiopeia. Very short exposures with almost no tracking, did not use LP filter. Flats captured inside with specialised diffuser material and dim sky light reflecting off a white wall. Gradients are very similar to the pic in the original post. Again, no method to differentiate between LP and overcorrection. In this case, like in the original post, I was positioned on a small football field, nearest street light was about 100 meters away.

Wider shot of Cygnus and Lyra, 35mm lens, LP filter used. Flats taken on site, phone screen as light source and sheets of paper as diffuser. Gradient here is of different nature. It matches my expectations of where the LP should be coming from. This is bortle 9 as well, but I was positioned further away from LP source -- about a kilometre away from a nearest street.

So question is -- is there any way LP could be differentiated from overcorrection purely from data? Could LP somehow mess-up the process of calibration?

I guess the only way for me to test this motion (LP not overcorrection) now is to try to get to a darker site and take all frames there.

No LP doesn't mess up flat calibration. Flats are just for removing dust in the image train and the natural vignetting of the system itself. It will thus correct the overall illumination of the frame to make it more equal. LP however is part of the real signal, so this will still be present. Your second picture there shows a good corrected image with LP, which can be removed with the LP tool. It's also a nice gradient, which makes the LP tool work very well usually.

Phone screens as flat sources I wouldn't use. You need a nice broadband signal, that isn't suddenly putting out or cutting out specific regions in the histogram. Phones and such can do that. So a source like the sun or a true flat panel, is ideal. When I didn't have my flat panel with me when I was in NZ, I took some flats during a cloudy day, put down a white bed sheet on a tabel, and a white sheet onto my lens and took a picture of the table at a spot that made the result very evenly lit. It was the best I could do, but it worked. So you need something like that, same focus, exact same focal length (if you use a zoom lens) as used when taking the data. Good even illumination, linear histogram at least to 50-70% (if you use the histogram displayed on the DSLR, it's not linear, so then you need to go to the right quite a bit) etc. It can be a tricky thing.

@vincent-mod

Posted by: @vincent-mod

Your second picture there shows a good corrected image with LP

Sorry, did you mean second (Navi) or third (Cygnus and Lyra)? If you actually mean second, how do you know it's LP and not flats overcorrection?

Sorry, yes third. 🙂 Usually over-correcting flats have an issue with the way they were taken. They then don't match the illumination profile of your regular subs. This can be due to multiple things, amongst which a non-ideal illumination source is one (or like Mabula mentioned, a different offset between sub and flat).

Just to finalise this thread, the original issue turned out to be LP disguising as flats calibration failure. I've managed to get out to bortle 4-5 with the same gear and here's the result of 40 mins stack. Still a little gradient coming from where the nearby town is, but nothing close to the original circular pattern. Very unusual case. Hopefully it is now closed.  

I too am now plagued with Flats Over-Correction.  I have moved from a SharpStar 15028HNT to a SharpStar 20032PNT (Slightly slower but larger aperture), and I have tried flats at 29500ADU and 22500ADU and there's overcorrecting in all of my filters.

I have taken the flats, and associated dark flats for each filter, and created the master flats, the master dark flats are ok, there's no source from those that could in any way shape or form affect the flats, but nevertheless I am getting overcorrection in my light frames.

It is the same camera, EFW, filters that I used on my 15028HNT and I would always shoot flats at 22500ADU.  I have even lowered the light level of my EL panel so that my flat expsoures are not "Too short", but the issue still persists, here is an example:

Red - Master Flat:

Red Frame - Linear:

Red Frame - Calibrated

Red Frames Stacked

Blue - Master Flat

Blue Frame - Linear

Blue Frame Calibrated

Blue Frames Stacked

All Light Frames, Flat Frames, Dark Flats were taken at the same Gain/Offset, same temperature, and exposure time of flats matches dark flats, I just cannot get to the source of the over correction yet

Simon

Mm, might be that you have an issue with background levels then, like an issue with the darks or bias or darkflats. How does a dark and bias look like?

@vincent-mod 

Darkflats look fine to me, I have just ordered a larger flat panel one of the Pegasus Astro ones so that I can set light levels per filter as well, as I have 3 sheets of ND Filter in my Gerd Neumaan one which is no good for NB flats as the Ha exposure would be >1600S :-O

Here's the Master Dark Flat for Red:

And Blue:

Regards

Simon

Not something very clear to see no. If you can't find out I can always have a look at the data of course.

@vincent-mod Let's wait for my new flatpanel to arrive and see if the issue goes away, it could be that the illumination circle on my GN Panel isn't enough, it's supposed to be for up to 8" but the illumination doesn't fill the whole apperute, hence why I have gone for a Flatpanel 250

@vincent-mod

So far there does not appear to be much different in flats between the GN Panel and the Pegasus Astro one, here they are side by side (GN Panel on left, Pegasus Astro on right)

However, I have not got all the required frames yet to produce the master flats to compare those.  But the Pegasus Astro panel makes lights so much easier as SGPro adjusts the intensity of the panel per filter so no more ND film 😀

Not completely there, but definitely much better

Left was original GN Flat panel and 29.5K ADU, right is new Pegasus Flat Panel again 29.5k ADU

Nice, so would you say that solves it?

@stastro just let me throw in that I had a similar problem way back when I changed from ccd to cmos camera. On my ccd, I only exposed flats to about 20-25k of the 65k adu saturation, to get get good calibration. when I moved to asi071mc cmos, I got same correction problem until I exposed to around 45k mean adu, with max adu up to 60k.  That may seem theoretically wrong,  but it has worked for me ever since. May be worth just upping the exposure in a test to see. 🤷‍♂️

That's not wrong at all Andy, as long as you don't clip any data on either side, it's better to get the histogram a bit more to the right, just to make absolutely sure the darkest parts are also well illuminated.

@vincent-mod 

That's really interesting as the general rule of thumb has always been 1/3 to 1/2 way up the histogram.  I might give that a try and see what happens, I made changes to the imaging train last night, and took some new flats as well as lights, so waiting for the calibration to complete

Well, yes, that has been a "rule" people simply have told each other for years. Doesn't mean it's necessarily correct. 🙂 I think it has been said because it works for a lot of people and ensures you're never clipping. But with the high dynamic range of new sensors and some sensors in particular, they behave less well in the low illuminated area's (not very linear). It doesn't matter if you're at 50%, it matters you're not clipping and to make sure everything is well illuminated, shifting to the right is a good thing to do, but you have to test and check you're not clipping.