Problem with Master Flats versus Single Flat

@stastro Simon, what version of APP are you using? Can you check and verify that "32 bit masters" is enabled at the bottom of tab 2?

Posted by: @wvreeven

@stastro Simon, what version of APP are you using? Can you check and verify that "32 bit masters" is enabled at the bottom of tab 2?

Morning Wouter

I am using 2.0.0-Beta1 and yes 32 bit masters is checked

Regards

Simon

@stastro Good morning Simon! Could you upload, say, 30 flats and dark flats using

@wvreeven 

I'll do that as soon as I am finished presenting, I'll create two subfolders, one for flats, one for darks, and I will also upload my current master flat and master dark flat too for the same filter

Hi Wouter

Files all uploaded now

Regards

Simon

@stastro Thanks for the upload Simon. I loaded the 30 flats and 30 dark flats and processed them in APP by going to tab 2 and clicking "create Masters & assign to Lights" without changing anything else. The resulting MasterFlat looks like this:

which looks fine to me. Is this not what you expected and, if it isn't, what did you expect?

@wvreeven Look carefully at the flat that was created, it contains gradients that are not gradual, and you can also see boundary lines in the pattern of a tree stump grain, is this normal?

So I wonder why a master flat is having such a negative impact on the image stack, despite the fact a single flat file seems to work pretty good

Posted by: @stastro

Look carefully at the flat that was created, it contains gradients that are not gradual, and you can also see boundary lines in the pattern of a tree stump grain, is this normal?

It maybe just a presentation issue. Have you actually zoomed in on the MF to see if the boundary lines remain?

I can investigate further if you want. Can you upload Lights and Darks (and if necessary Bias) please?

@stastro When I zoom in, I see this:

As you can see, the lines disappear and the gradient becomes smooth.

@stastro Zoomed in even further:

The lines that are visible now are due to the screenshot I took on my MacBook. They weren't visible on my screen.

Hi,

I may be missing something by looking at your screen-grabs but I think the streaking effect you are seeing is likely a result of limitations of screen display technology than of APP MF production.  

Basically you are displaying a 32-bit resolution MF on a screen that is not capable of display to the same level of granularity. To deal with this the source 32 bit values must be rounded to a lower-range  grayscale value for display purposes (aka posterization).  Because the shading of your MF is so gradual the step points at which the 32 bit values have been rounded to a lower level (?256 steps) are just starting to become visible.  The effect is probably also more pronounced than it actually is as a result of a screen stretch.

If one zooms in sufficiently all sorts of pixel level imperfections start to become visible.

I conjecture that your single frame MF is less 'smooth' so the step-points are less apparent but none the less must still be there. 

I have to say that I cannot see the effect in your final images.

Regards

@stastro I completely agree with @mestutters, who has explained the display effect much better than me in my previous comments. The effect is called posterization. 

Thank you for chiming in @mestutters!

I think the crux of my problem is the severe vignetting on the SharpStar 20032PNT with the ASI6200MM Pro, which is strange because the scope is sold to be excellent for full frame, I had a 15028HNT before this that did not produce vignetting so bad.

This is from the 20032PNT, Luminance with STF applied in PixInsight

And here is a Master Flat from the same camera, same filterwheel but using the 15028HNT, again STF Applied

Now the 15028HNT Had an OAG in there as well, but that should not make much difference.

Now the interesting part is, according to the light cone for a 44mm diagonal sensor, at the 55mm distance from the sensor, the light cone would be 61.1875mm, which is a larger diameter than the M54 imaging train (distance from sensor / focal ratio) + sensor diagonal, (55/3.2)+44, however the bit that throws me out there is on the 15028HNT I was using M48 imaging train, and 55mm from the sensor would have had a light cone of (55/2.8)+44 = 63.64286mm, which is significantly higher than the 20032PNT requirement but I had so much less severe vignetting with that scope using the same camera and filters. 

So the investigation continues!

@stastro Interesting analysis. I hope you manage to get to the bottom of this. I am facing a similar challenge using an ASI6200MM on a RedCat 71.

Out of curiosity, what filters do you use? And what light source to take flats? I have used filters before that had an IR leak combined with an iPad for taking flats and that introduced all kinds of weird artifacts in the integrations. 

Posted by: @wvreeven

@stastro Interesting analysis. I hope you manage to get to the bottom of this. I am facing a similar challenge using an ASI6200MM on a RedCat 71.

Out of curiosity, what filters do you use? And what light source to take flats? I have used filters before that had an IR leak combined with an iPad for taking flats and that introduced all kinds of weird artifacts in the integrations. 

Hi Wouter

For the flat light source I am using the Pegasus Astro Flat Panel, as it has full control over the brightness, so for example, I can have it on 1 for Lum, 40 for RGB and 100% for NB filters.  I used to use a GerdNeumaan FlatPanel, but I wanted to automate flat creation as much as possible.

My filters are Baader CMOS Optimised 50.4 unmounted, and the Narrowband ones are the Ultra Fast NB filters from Baader also, they are sat in my ZWO EFW, so the filters are around 18mm from the sensor, so the filters are bigger than the light cone at 18mm distance.

I have just discovered that the backfocus distance from the M68 to the sensor is 61mm, as the M68 to M54 is 6mm in thickness, so at 61mm, the light cone width would be 63.0625mm which is lower than the M68, so I might go from M54 to M68 to see if that helps.  I have reached out to a couple of people with regards to them using full frame sensors on the 20032pnt to find out what their imaging train is, and also to get a view on their flats, just to see if what I am getting is Normal and if my flats are flawed in some way shape or form.

Regards

Simon

As a test I am also going to try and increase the exposure time of my flats, so for RGB lower the flat panel intensity by 50% which should in theory double the exposure time to see if that helps, not sure how I am going to get luminance longer as the flat panel is already at the lowest value of 1 for flats at 22500ADU, I could also opt for higher ADU since it makes no difference whatsoever to flats whether the ADU is 22500, 29500 or 45000, that should get around the problem with Luminance

I'm getting there, here's the current processed image with new flats

Far less of an issue with the above, so what did I do?  Well after doing soem research, CMOS cameras do not produce good flats with short exposures, so I opted to push the exposures up, rather than change the intensity of the flat panel, I just upped the ADU to 49.5K from 22.5K (22.5K is what I always shot Flats at on the 15028HNT), comparing flats between the ADUs is starkly noticable (both have had Auto STF applied), 22.5K on left, 49.5K on right:

So whilst vignetting is still an issue, it is starting to be corrected successfully, now the difference in exposure times are like this:

22.5K ADU

L - 1.31S
R - 2.1S
G - 2.08S
B - 2.15S

49.5K ADU

L - 4.32S
R - 5.87S
G - 5.94S
B - 6.0S

I am not even close to clipping the histogram on the right either

So I am starting to get there, but maybe you should try some of these with your Redcat setup Wouter

Regards

Simon

Posted by: @stastro

maybe you should try some of these with your Redcat setup Wouter

Thanks for the tip! I generally take flats of between 1 and 5 seconds and make sure that the peak is as far to the right as possible without clipping. This works for Ha but not for OIII and I don't know why. Light leak or internal reflections (possibly in IR) are likely candidates.

Anyway, I am glad that you're managing to solve your flat issues.

I have not done my NB filters with the higher ADU yet, I dread to think what the exposure time is going to be for Ha and SII, they are going to be very long indeed

Posted by: @wvreeven

Posted by: @stastro

maybe you should try some of these with your Redcat setup Wouter

Thanks for the tip! I generally take flats of between 1 and 5 seconds and make sure that the peak is as far to the right as possible without clipping. This works for Ha but not for OIII and I don't know why. Light leak or internal reflections (possibly in IR) are likely candidates.

Anyway, I am glad that you're managing to solve your flat issues.

Hi Wouter

So the exposure time of the flats clearly does make a difference, I re-did my 22500ADU flats for RGB with a longer exposure time by reducing the intensity of the flat panel, and 22500ADU actually works now.  ADU22500 seems to correct the image better in the corner than the higher ADU Flats too, each one below just has an STF applied

ADU22500

ADU32500

ADU49500

ADU55000

So like my Flats from the 15028HNT were all taken at 22500, it seems that I was not going mad, just my exposure time of the flats was not long enough, incidentally I looked back at my flats from my 15028HNT and they were all around the exposure time I now have my 22500ADU flats.

Fully post processing the images, you can see a difference that the different ADU flats have as an effect on the finalised images, each one was processed in exactly the same way with exactly the same settings:

ADU22500

ADU32500

ADU49500

ADU55000

So I think ADU22500 is the winner, the image stack is only 85x60S exposures with RGB (No Luminance) at Gain 100

Note to self: Never go with short exposure flats....EVER! 😀

Simon

It would appear that Sky-Flats are working better at correcting the vignetting, so I have a question about Dark Flats.  Because the sky-flats have a varying amount of exposure time due to target ADU and a gradually darkening sky, how do I tackle this with Dark Flats, my thoughts are to take dark flats at the longest flat frame exposure time for each filter and apply that master dark flat to all the flat files.  Thoughts?

You should be able to use darkflats that are in the range of your exposure times (say your range is from 2-4 seconds, I would take darkflats at 3 for instance). Darkflats basically are like bias frames, maybe with a few extra hot pixels (if at all). Personally I like flat panels a bit better, specifically due to not having any variations or the chance of some cloud changing illumination.

@vincent-mod Thanks

Yeah, I think until I am able to use the M68 connector of the scope (Waiting for an adapter), sky-flats seem to be the only route to actually solving the correction of vignetting.  Once I am using the M68 connector on the scope, the vignetting should be greatly reduced.

How can I check to see that a Master Dark Flat was used in the calibration of the Flats?  I do not see the MDF being applied in the flats list, and during the processing of the flats, it does not mention MDF at all

They do need to have the same settings (like gain, offset, etc), just like a bias, is that the case? If so, the list should show the tags for MDF in the flats when calibrating.

@vincent-mod 

I have created my master dark flats, they contain the same exposure, gain and offset as the flats, but you can see here they do not appear to be being assigned to my flats:

Nor do I see any mention of the MDF in the console screen:

I double checked and it isn't actually mentioned in this case, when there is no suitable bias or darkflat then APP will warn for that.