19 June 2021: Our upload server https://upload.astropixelprocessor.com/ has been migrated successfully to our new office with higher upload and download speeds (nearly 10MByte/sec up/down ) ! We now have 1 general upload user called: upload with password: upload. The users upload1 - upload5 have been disabled.
31 May 2021: APP 1.083-beta2 has been released ! APP 1.083 stable will follow soon afterwards. It includes a completely new Star Reducer Tool, New File Saver Module, Improved Comet registration and much more, check the release notes here!
The case of overcorrecting flats, need help
I've had this problem for a few weeks, every new session gives the same result. Would be great If someone could take a look at the data and tell why those flats don't work.
All the subs are in the zipped folder here:
I'll have a look, thanks for sharing the data.
Thank you. Just to save some time -- flats in the pack are of pretty short exposures. I will be able to upload a new session data hopefully within a couple of days (cloudy right now). Like I said, the problem persists no matter how those frames are taken, but just to eliminate these obvious causes, I'll try to retake them asap.
The immediate cause of the issue seem to be that the illumination profiles of light frames and flats do not match. But I do not know of any tools (graphs?) that could help to demonstrate it clearly and control for it. I've had people look at the data through PI tools and there's nothing that is obviously wrong with it. I was hoping darks and biases might be the culprit, but they seem ok.
My setup is pretty simple -- canon 60d, L-pro clip-in and 135mm f3.5 lens.
@igor_cheb I downloaded the data as well. One thing I noticed is that the bias were shot at ISO 400 and all other files at ISO 500. Is there a reason for that?
@igor_cheb I downloaded the zip again, and again the bias frames are ISO 400. Could you perhaps use our upload server? That way you don't have to upload a zip file, which would considerably reduce download times for us. Use upload for both username and password on
and create a directory called igor_cheb_overcorrected_flat
Have reuploaded to google drive (updated link in the original post) and am uploading to the APP server, igor_cheb_overcorrected_flat folder. Will take about an hour, speed is not great for some reason.
@igor_cheb Mabula is hosting the upload server himself. He has limited the upload speed to 1 Mb/sec so that's why it takes a while. Thanks a lot for making the data available. Please let us know when the upload has finished and we will have take look.
@igor_cheb Many thanks! I downloaded the data and in this data set the bias have ISO 500 as well. I am looking into this now.
@igor_cheb I can reproduce your result but don't know what the reason is for the over-correction. I have asked Mabula to take a look. He will do so as soon as he has time.
Thank you , hope it's solvable
I have checked the data and I also do not see any clear reason why the flats overcorrect.
Overcorrection will occur when:
- the vignetting profile in the flats is stronger than in the lights..
- or the sensor offset was different between between flats and the bias, or lights and the bias.
With this consumer camera Canon 60D, the sensor offset is fixed, so the problem must be found in how the lights and the flats were shot... something must be different which has caused the problem.
Igor, did you make changes to the optical train between having shot the lights and the flats?
Were both types shot with exactly the same optical train?
Was there no focus creep of the lens between creating the lights and the flats, so are you sure the focus did not change?
Shot with same aperture of the lens?
Were the flats taken at night or during the day? Using a flat panel or?
Sorry for the delay and thank you for taking the time. The answer to all questions regarding light train is no -- no focus creep (very stiff grease in the lens), same gear used, same aperture.
As for the methods used -- that particular set of flats was taken inside with a phone screen as light source and several sheets of paper as a diffuser. Since then I've done a few things.
I've replaced both the lens and the camera (modified 550d with Samyang 135 this time).
I've tried taking flats with various light sources (phone screen, white wall reflecting light, dim sky) and various diffusers (t-shirt, layered sheets of paper, macbook plasticky folding material, specialised diffusing material used for studio lights equipment). Checked for light leaks repeatedly. Inspected stretched flat frames with various lenses. Tried to rotate LP filter and take flat frames in various positions.
Have no substantial clues to report, the problem remains irrespective of flats acquisition approach. My conclusion so far is that the vignetting is not overcorrection, but a light pollution that happen to come at this peculiar angle making it seem like over correction by flats. This is bortle 9, so LP is to be expected.
The only thing that still seems suspicious to me is how master flats look after APP calibration -- very saturated. But perhaps a stretched flat always looks like that:
Couple of pictures I took in short sessions recently.
Navi star region in Cassiopeia. Very short exposures with almost no tracking, did not use LP filter. Flats captured inside with specialised diffuser material and dim sky light reflecting off a white wall. Gradients are very similar to the pic in the original post. Again, no method to differentiate between LP and overcorrection. In this case, like in the original post, I was positioned on a small football field, nearest street light was about 100 meters away.
Wider shot of Cygnus and Lyra, 35mm lens, LP filter used. Flats taken on site, phone screen as light source and sheets of paper as diffuser. Gradient here is of different nature. It matches my expectations of where the LP should be coming from. This is bortle 9 as well, but I was positioned further away from LP source -- about a kilometre away from a nearest street.
So question is -- is there any way LP could be differentiated from overcorrection purely from data? Could LP somehow mess-up the process of calibration?
I guess the only way for me to test this motion (LP not overcorrection) now is to try to get to a darker site and take all frames there.
No LP doesn't mess up flat calibration. Flats are just for removing dust in the image train and the natural vignetting of the system itself. It will thus correct the overall illumination of the frame to make it more equal. LP however is part of the real signal, so this will still be present. Your second picture there shows a good corrected image with LP, which can be removed with the LP tool. It's also a nice gradient, which makes the LP tool work very well usually.
Phone screens as flat sources I wouldn't use. You need a nice broadband signal, that isn't suddenly putting out or cutting out specific regions in the histogram. Phones and such can do that. So a source like the sun or a true flat panel, is ideal. When I didn't have my flat panel with me when I was in NZ, I took some flats during a cloudy day, put down a white bed sheet on a tabel, and a white sheet onto my lens and took a picture of the table at a spot that made the result very evenly lit. It was the best I could do, but it worked. So you need something like that, same focus, exact same focal length (if you use a zoom lens) as used when taking the data. Good even illumination, linear histogram at least to 50-70% (if you use the histogram displayed on the DSLR, it's not linear, so then you need to go to the right quite a bit) etc. It can be a tricky thing.
Your second picture there shows a good corrected image with LP
Sorry, did you mean second (Navi) or third (Cygnus and Lyra)? If you actually mean second, how do you know it's LP and not flats overcorrection?
Sorry, yes third. 🙂 Usually over-correcting flats have an issue with the way they were taken. They then don't match the illumination profile of your regular subs. This can be due to multiple things, amongst which a non-ideal illumination source is one (or like Mabula mentioned, a different offset between sub and flat).
Just to finalise this thread, the original issue turned out to be LP disguising as flats calibration failure. I've managed to get out to bortle 4-5 with the same gear and here's the result of 40 mins stack. Still a little gradient coming from where the nearby town is, but nothing close to the original circular pattern. Very unusual case. Hopefully it is now closed.