The case of overcor...
 
Share:
Notifications
Clear all

2022-05-29: APP 2.0.0-beta2 has been released !

Release notes

Download links per platform:

windows 2.0.0-beta2

macOS x86_64 2.0.0-beta2

macOS arm64 M1 2.0.0-beta2

Linux DEB 2.0.0-beta2

Linux RPM 2.0.0-beta2

The case of overcorrecting flats, need help

Page 1 / 3

(@igor_cheb)
Brown Dwarf Customer
Joined: 12 months ago
Posts: 12
Topic starter  
image

I've had this problem for a few weeks, every new session gives the same result. Would be great If someone could take a look at the data and tell why those flats don't work.
All the subs are in the zipped folder here:

https://drive.google.com/file/d/11RAZl4t4YkaEtAVIBOeFvMMcEFzcle6C/view?usp=sharing

This topic was modified 11 months ago 2 times by Igor Chebuniaev

ReplyQuote
(@vincent-mod)
Quasar Admin
Joined: 5 years ago
Posts: 4830
 

I'll have a look, thanks for sharing the data.


ReplyQuote
(@igor_cheb)
Brown Dwarf Customer
Joined: 12 months ago
Posts: 12
Topic starter  

@vincent-mod

Thank you. Just to save some time -- flats in the pack are of pretty short exposures. I will be able to upload a new session data hopefully within a couple of days (cloudy right now). Like I said, the problem persists no matter how those frames are taken, but just to eliminate these obvious causes, I'll try to retake them asap.

The immediate cause of the issue seem to be that the illumination profiles of light frames and flats do not match. But I do not know of any tools (graphs?) that could help to demonstrate it clearly and control for it. I've had people look at the data through PI tools and there's nothing that is obviously wrong with it. I was hoping darks and biases might be the culprit, but they seem ok.

My setup is pretty simple -- canon 60d, L-pro clip-in and 135mm f3.5 lens.

This post was modified 11 months ago 2 times by Igor Chebuniaev

ReplyQuote
(@wvreeven)
Galaxy Admin
Joined: 4 years ago
Posts: 1723
 

@igor_cheb I downloaded the data as well. One thing I noticed is that the bias were shot at ISO 400 and all other files at ISO 500. Is there a reason for that?


ReplyQuote
(@igor_cheb)
Brown Dwarf Customer
Joined: 12 months ago
Posts: 12
Topic starter  

@wvreeven

@vincent-mod

Thank you for taking the time. No reason, just a mix up. I've reuploded the zip with correct biases. Link is the same. Apologies.

This post was modified 11 months ago 2 times by Igor Chebuniaev

ReplyQuote
(@wvreeven)
Galaxy Admin
Joined: 4 years ago
Posts: 1723
 

@igor_cheb I downloaded the zip again, and again the bias frames are ISO 400. Could you perhaps use our upload server? That way you don't have to upload a zip file, which would considerably reduce download times for us. Use upload for both username and password on

https://upload.astropixelprocessor.com/

and create a directory called igor_cheb_overcorrected_flat


ReplyQuote
(@igor_cheb)
Brown Dwarf Customer
Joined: 12 months ago
Posts: 12
Topic starter  

@wvreeven

Have reuploaded to google drive (updated link in the original post) and am uploading to the APP server, igor_cheb_overcorrected_flat folder. Will take about an hour, speed is not great for some reason.


ReplyQuote
(@wvreeven)
Galaxy Admin
Joined: 4 years ago
Posts: 1723
 

@igor_cheb Mabula is hosting the upload server himself. He has limited the upload speed to 1 Mb/sec so that's why it takes a while. Thanks a lot for making the data available. Please let us know when the upload has finished and we will have take look.


ReplyQuote
(@igor_cheb)
Brown Dwarf Customer
Joined: 12 months ago
Posts: 12
Topic starter  

@wvreeven

upload done.


ReplyQuote
(@wvreeven)
Galaxy Admin
Joined: 4 years ago
Posts: 1723
 

@igor_cheb Many thanks! I downloaded the data and in this data set the bias have ISO 500 as well. I am looking into this now.


ReplyQuote
(@wvreeven)
Galaxy Admin
Joined: 4 years ago
Posts: 1723
 

@igor_cheb I can reproduce your result but don't know what the reason is for the over-correction. I have asked Mabula to take a look. He will do so as soon as he has time.


ReplyQuote
(@igor_cheb)
Brown Dwarf Customer
Joined: 12 months ago
Posts: 12
Topic starter  

Thank you , hope it's solvable


ReplyQuote
(@mabula-admin)
Quasar Admin
Joined: 5 years ago
Posts: 3158
 

Hi Igor @igor_cheb and Wouter @wvreeven,

I have checked the data and I also do not see any clear reason why the flats overcorrect.

Overcorrection will occur when:

- the vignetting profile in the flats is stronger than in the lights..

- or the sensor offset was different between between flats and the bias, or lights and the bias.

With this consumer camera Canon 60D, the sensor offset is fixed, so the problem must be found in how the lights and the flats were shot... something must be different which has caused the problem.

Igor, did you make changes to the optical train between having shot the lights and the flats?

Were both types shot with exactly the same optical train?

Was there no focus creep of the lens between creating the lights and the flats, so are you sure the focus did not change?

Shot with same aperture of the lens?

Were the flats taken at night or during the day? Using a flat panel or?

Mabula


ReplyQuote
(@igor_cheb)
Brown Dwarf Customer
Joined: 12 months ago
Posts: 12
Topic starter  

@mabula-admin

@wvreeven 

 

Sorry for the delay and thank you for taking the time. The answer to all questions regarding light train is no -- no focus creep (very stiff grease in the lens), same gear used, same aperture. 

As for the methods used -- that particular set of flats was taken inside with a phone screen as light source and several sheets of paper as a diffuser. Since then I've done a few things.

I've replaced both the lens and the camera (modified 550d with Samyang 135 this time).

I've tried taking flats with various light sources (phone screen, white wall reflecting light, dim sky) and various diffusers (t-shirt, layered sheets of paper, macbook plasticky folding material, specialised diffusing material used for studio lights equipment). Checked for light leaks repeatedly. Inspected stretched flat frames with various lenses. Tried to rotate LP filter and take flat frames in various positions.

Have no substantial clues to report, the problem remains irrespective of flats acquisition approach. My conclusion so far is that the vignetting is not overcorrection, but a light pollution that happen to come at this peculiar angle making it seem like over correction by flats. This is bortle 9, so LP is to be expected.

The only thing that still seems suspicious to me is how master flats look after APP calibration -- very saturated. But perhaps a stretched flat always looks like that:

image

Couple of pictures I took in short sessions recently.

image

Navi star region in Cassiopeia. Very short exposures with almost no tracking, did not use LP filter. Flats captured inside with specialised diffuser material and dim sky light reflecting off a white wall. Gradients are very similar to the pic in the original post. Again, no method to differentiate between LP and overcorrection. In this case, like in the original post, I was positioned on a small football field, nearest street light was about 100 meters away.

image

Wider shot of Cygnus and Lyra, 35mm lens, LP filter used. Flats taken on site, phone screen as light source and sheets of paper as diffuser. Gradient here is of different nature. It matches my expectations of where the LP should be coming from. This is bortle 9 as well, but I was positioned further away from LP source -- about a kilometre away from a nearest street.

So question is -- is there any way LP could be differentiated from overcorrection purely from data? Could LP somehow mess-up the process of calibration?

 

I guess the only way for me to test this motion (LP not overcorrection) now is to try to get to a darker site and take all frames there.

This post was modified 11 months ago 3 times by Igor Chebuniaev

ReplyQuote
(@vincent-mod)
Quasar Admin
Joined: 5 years ago
Posts: 4830
 

No LP doesn't mess up flat calibration. Flats are just for removing dust in the image train and the natural vignetting of the system itself. It will thus correct the overall illumination of the frame to make it more equal. LP however is part of the real signal, so this will still be present. Your second picture there shows a good corrected image with LP, which can be removed with the LP tool. It's also a nice gradient, which makes the LP tool work very well usually.

Phone screens as flat sources I wouldn't use. You need a nice broadband signal, that isn't suddenly putting out or cutting out specific regions in the histogram. Phones and such can do that. So a source like the sun or a true flat panel, is ideal. When I didn't have my flat panel with me when I was in NZ, I took some flats during a cloudy day, put down a white bed sheet on a tabel, and a white sheet onto my lens and took a picture of the table at a spot that made the result very evenly lit. It was the best I could do, but it worked. So you need something like that, same focus, exact same focal length (if you use a zoom lens) as used when taking the data. Good even illumination, linear histogram at least to 50-70% (if you use the histogram displayed on the DSLR, it's not linear, so then you need to go to the right quite a bit) etc. It can be a tricky thing.


ReplyQuote
(@igor_cheb)
Brown Dwarf Customer
Joined: 12 months ago
Posts: 12
Topic starter  

@vincent-mod

Posted by: @vincent-mod

Your second picture there shows a good corrected image with LP

Sorry, did you mean second (Navi) or third (Cygnus and Lyra)? If you actually mean second, how do you know it's LP and not flats overcorrection?


ReplyQuote
(@vincent-mod)
Quasar Admin
Joined: 5 years ago
Posts: 4830
 

Sorry, yes third. 🙂 Usually over-correcting flats have an issue with the way they were taken. They then don't match the illumination profile of your regular subs. This can be due to multiple things, amongst which a non-ideal illumination source is one (or like Mabula mentioned, a different offset between sub and flat).


ReplyQuote
(@igor_cheb)
Brown Dwarf Customer
Joined: 12 months ago
Posts: 12
Topic starter  

Just to finalise this thread, the original issue turned out to be LP disguising as flats calibration failure. I've managed to get out to bortle 4-5 with the same gear and here's the result of 40 mins stack. Still a little gradient coming from where the nearby town is, but nothing close to the original circular pattern. Very unusual case. Hopefully it is now closed.  

for forum

 

This post was modified 10 months ago by Igor Chebuniaev

ReplyQuote
(@stastro)
Main Sequence Star Customer
Joined: 3 years ago
Posts: 80
 

I too am now plagued with Flats Over-Correction.  I have moved from a SharpStar 15028HNT to a SharpStar 20032PNT (Slightly slower but larger aperture), and I have tried flats at 29500ADU and 22500ADU and there's overcorrecting in all of my filters.

I have taken the flats, and associated dark flats for each filter, and created the master flats, the master dark flats are ok, there's no source from those that could in any way shape or form affect the flats, but nevertheless I am getting overcorrection in my light frames.

It is the same camera, EFW, filters that I used on my 15028HNT and I would always shoot flats at 22500ADU.  I have even lowered the light level of my EL panel so that my flat expsoures are not "Too short", but the issue still persists, here is an example:

Red - Master Flat:

image

Red Frame - Linear:

image

Red Frame - Calibrated

image

Red Frames Stacked

image

Blue - Master Flat

image

Blue Frame - Linear

image

Blue Frame Calibrated

image

Blue Frames Stacked

image

 

All Light Frames, Flat Frames, Dark Flats were taken at the same Gain/Offset, same temperature, and exposure time of flats matches dark flats, I just cannot get to the source of the over correction yet

Simon

 


ReplyQuote
(@vincent-mod)
Quasar Admin
Joined: 5 years ago
Posts: 4830
 

Mm, might be that you have an issue with background levels then, like an issue with the darks or bias or darkflats. How does a dark and bias look like?


ReplyQuote
Page 1 / 3
Share: