flat frame not being applied

@mabula-admin  Uploaded. 

@mabula-admin  "In the case where you just divide a light with a flat, that sensor offset/pedestal is not removed leading to a bad result logically. Thus that Rule Of Thumb should be: Divide a bias/dark subtracted light with a bias/darflat subtracted flat "

Thanks for your support!!  FYI, I was referring to standard light calibration workflow found online which seems to match what PI was doing in some of my tests using subtraction and division.  When I went the pixelmath direction using an inverse method for testing purposes, I had already corrected for darkflat and dark subtraction prior to my test.    Pixelmath can be funny sometimes with subtraction, not sure why.   Sometimes I have to find a work around.   

So let me ask you this,  because flats mean ADU values from user to user differs, do you scale the flat ADU to a curtain mean value in APP or leave it as is?     Example,  Re the data I sent you, if you scale the flat mean to 10% ADU and the darkflat accordingly, if I use this formula the calibration seems to work. (($T*~(MF*~MDF))*~MD).  Please dont be to critical with me over this test and math used, just testing here 😀   I have not tested this on other data yet, but was wondering if there could be a calculated correlation here to help with flat removal better.  My original Flat ADU was set to 33% and some people go higher, some people go as low as the light frame adu value, so I wanted to test  with different mean ADU values to see what had the best result, so I tried reducing it to a level of 10% and then used pixelmath which gave me a better result that what I was getting in APP and I tested APP also at different flat adu values.   That's why I wasn't sure if there was a bug.    but for what ever reason, I could not get a good calibrated light frame in APP with this data.   I have 4 nights of Ha imaging, all at the same exposure time and flat method for each night, and all have different results.  On another thread I posted an issue I was having with normalization too as when I normalize all data and save, some data doesn't correct as it should.   Anyhow, let me know what you find.   Thanks!   

Hi Bill @blanshan91,

I see that you uploaded calibration masters, but if you want me to get to the bottom of this, I need the single calibration frames, not the masters.Could you upload 5 of each type (dark,bias, etc)?

I will have a look at the data and masters now, to see if I can already see something odd.

EDIT: I see that the masters have been altered by Pixinsight, so I can not use those masters to make any reasonable explanation here I am afraid. I have no clue what happened to the data in Pixinsight.

EDIT2: Assuming Pixinsight did nothing to change the data, I see that for the darks and darkflats different sensor offsets where used. Are you aware of this? Maybe the sensor offsets are the problem in your calibration workflow.

Mabula

@mabula-admin   I deleted the flat, darkflat and dark frames when I created the masters.  All masters were made using default APP settings.   Re the Pixinsight editing, yes, I had opened them in PI and saved them back as 16bit, nothing else was changed.  I had to do this as I wanted to use them in SharpCap for live stacking viewing.   For some reason, SharpCap would not open them right out of APP.  My Lights and Darks were done at Gain 150, Offset 50.   It could be my flats were done at Offset 30 by default, didnt think it would matter as a flat.  I will have to look into that though.   Note:  I don't see the offset info in the header, where do you see this?

Hi Well its safe to say it not IR i got a 4w IR torch so very bright and have no change on the camera on all filters even the Lum one when i shine it down the OTA at the dead of night. I also moved it around the area of the moonlite where i know there may be a very small light leak and again no change on the camera but this is before the filters. What i should have done is then used a normal white light torch around the moonlite so i will do that another night.

I will also try taking some flats at night but even on my day time flats and sky flats then it comes to the darkflat i don't see any light leak so i am sure that's not the issue. 

Starting to think its a internal reflection with in the reducer but its a good one a optec lepus x0.8

Mike

@moviecells OK, one possible cause eliminated. Did you make sure to create 32 bit masters in tab 2?

@wvreeven

I think 32 bit masters are the default in this version anyway yes it is ticked.

Mike

@moviecells OK thanks for the confirmation. There are several sub-threads in this thread so I am getting lost for who is using which APP versions and settings. Could you maybe upload your data set to

https://upload.astropixelprocessor.com/

using upload both for username and password? Please create a directory called moviecells_flat_issue and put the files in there. Let me know when the upload is done and I'll have a look at the data.

Posted by: @blanshan91

@mabula-admin   I deleted the flat, darkflat and dark frames when I created the masters.  All masters were made using default APP settings.   Re the Pixinsight editing, yes, I had opened them in PI and saved them back as 16bit, nothing else was changed.  I had to do this as I wanted to use them in SharpCap for live stacking viewing.   For some reason, SharpCap would not open them right out of APP.  My Lights and Darks were done at Gain 150, Offset 50.   It could be my flats were done at Offset 30 by default, didnt think it would matter as a flat.  I will have to look into that though.   Note:  I don't see the offset info in the header, where do you see this?

Hi Bill @blanshan91,

Okay, that is unfortunate.

Mixing calibration masters between different packages can create all sort of problems and unexpected behaviour, so if you want live stacking in SharpCap, please create masters in SharpCap 😉

If APP made 32bits masters, and you reduced them to 16bit in Pixinsight, again, this could create problems and I can not comment on what Pixinsight does internally as you can understand. APP can convert the files to 16bits as well, with Batch Modify Fits tool in 1.083-beta2 😉

If the flats had a different offset (30) then that will be the problem for sure. It matters a lot actually. If the flats were shot with offset 30 and the darkflats with offset 50 then that is the issue. No doubt. The sensor offset is vital for correct calibration results.

The lights needs bias/darks with the same offset as the lights.

The flats need bias/darkflats with the same offset as the flats.

So it is perfectly possible to use different offsets for lights and flats, but you need to be aware of matching the calibration frames as well 😉

The offset was not to be found in the header, but it is easily verified just by looking at the median/average values of the masters in the metadata. Or simply look at the histogram and check where the peak of the histogram is... the peak will be at the sensor offset or slightly higher with darks due to the added dark current signal.

Mabula.

@wvreeven

Hi files have been uploaded if you could take a look it would be great

Mike

@moviecells Thanks Mike. I'll have a look right away.

@moviecells Mike, when I load the lights, flats and dark flats in APP 1.083-beta2 I get the following warning:

The reason for this warning is that you have created dark flats for each flat exposure time and then try to integrate those per channel to create master dark flats for each of L, R, G and B. Apparently some of the exposure times do not agree which is why the popup is shown.

The correct way of doing is is to shoot one set of dark flats of for instance one second and then also shoot a large set of bias frames of 0.01 second. Then create a master bias and use that together with the dark flats to create a master dark flat. Those two together can then be used to properly calibrate the flats and create the master flats for each channel. Can you give that a try?

Posted by: @mabula-admin

Posted by: @blanshan91

@mabula-admin   I deleted the flat, darkflat and dark frames when I created the masters.  All masters were made using default APP settings.   Re the Pixinsight editing, yes, I had opened them in PI and saved them back as 16bit, nothing else was changed.  I had to do this as I wanted to use them in SharpCap for live stacking viewing.   For some reason, SharpCap would not open them right out of APP.  My Lights and Darks were done at Gain 150, Offset 50.   It could be my flats were done at Offset 30 by default, didnt think it would matter as a flat.  I will have to look into that though.   Note:  I don't see the offset info in the header, where do you see this?

Hi Bill @blanshan91,

Okay, that is unfortunate.

Mixing calibration masters between different packages can create all sort of problems and unexpected behaviour, so if you want live stacking in SharpCap, please create masters in SharpCap 😉

If APP made 32bits masters, and you reduced them to 16bit in Pixinsight, again, this could create problems and I can not comment on what Pixinsight does internally as you can understand. APP can convert the files to 16bits as well, with Batch Modify Fits tool in 1.083-beta2 😉

If the flats had a different offset (30) then that will be the problem for sure. It matters a lot actually. If the flats were shot with offset 30 and the darkflats with offset 50 then that is the issue. No doubt. The sensor offset is vital for correct calibration results.

The lights needs bias/darks with the same offset as the lights.

The flats need bias/darkflats with the same offset as the flats.

So it is perfectly possible to use different offsets for lights and flats, but you need to be aware of matching the calibration frames as well 😉

The offset was not to be found in the header, but it is easily verified just by looking at the median/average values of the masters in the metadata. Are simply look at the histogram and check where the peak of the histogram is... the peak will be at the sensor offset or slightly higher with darks due to the added dark current signal.

Mabula.

Thanks for the reply!   FYI, I only tried to run in sharpcap after I noticed the flat removal wasn't working right for my data in APP, which was why I had to convert it to 16bit as sharpcap wasnt reading 32bit fits for flats.    Thanks for telling me about the 16 bit conversion options in APP, could be good to use in future.  Also, flats and flat darks are done at the same time in NINA so I dont think they have different offsets, unless NINA did something weird.  I will defiantly keep this in mind when using different camera offsets. 

Regarding FITS headers, you think you could bring over the original fits information from the lights and then you add your APP header info for the calibrated, normalized and final integration images so all info is available?   Obviously the integrated image header info will be slightly different but seems like more of the original header info could be added to the calibrated and normalized images.  Thoughts?       Also, will your next beta release have the ability to save APP settings or the ability to open with previously used settings so you don't have to reenter everything again? 

@wvreeven

I am using a darkflat and flats not bias its a cmos camera so bias files are not recommended for this type of camera.

I get no error my end when i load the darkflats, flats and lights you are sure you loaded them that way I have no bias frames. If you check you should see my flats and darkflats do match exposure, offset and gain there should be no errors. I used NINA flat/darkflat wizard its very good for this and mean there can be no mismatch to give an error.

Mike

@wvreeven

Can you check again i can't see why you are having a error my flats do match my darkflats. Also the statement about masterdark not matching the exposure time of the light it can't match. The exposure time of the flat must match the exposure of the darkflat which it does. I don't see any errors here this is what i do

Load all my light frames.

Load all my flat frames

Load all my Darkflat frames

Load my master dark

Then go to tab integrate and just run integrate

I don't see any error just not very good final integrated images.

Mike

@moviecells Mike, bias frames may not be recommended (I shoot with a CMOS camera as well and are familiar with the recommendation) but it is essential for dark scaling. If the exposure times of darks and lights or flats do not correspond then APP needs a master bias to scale the darks.

By the way, for modern CMOS cameras flat exposure times of under a second are not recommended either due to the fact that these cameras have a rolling shutter. Too short exposure times may introduce unevenly flats that can introduce unwanted gradients and other issues. You may want to consider dimming your light source so you can use longer exposure times for the L flats.

I did indeed notice that the dark flat exposure times match the exposure times of the flats so there should not be an issue. And, when I load all files (so all lights, flats, dark flats and the master dark) then I indeed do not get that warning. However, I do indeed also get the uncorrected integration results. I'll check with Mabula.

@wvreeven

asking Mabula would be great not sure whats going on.

so how did you load the files and get the error ?

so are we saying i should be getting an error to indicate a problem and i am not ? so my lights are not corrected and i need bias files to correct this ?

mike

Posted by: @moviecells

so how did you load the files and get the error ?

I only loaded the lights, flats and dark flats. Not the master dark. I asked Mabula why that gives an error.

Posted by: @moviecells

so are we saying i should be getting an error to indicate a problem and i am not ? so my lights are not corrected and i need bias files to correct this ?

That is what i thought at first but this is not the case. The exposure times of the flats and dark flats correspond per channel so a bias is not necessary. You could, however, consider taking bias frames so you only need one set of dark flats of, say, 1 second which you then can apply to all the flats instead of shooting dedicated dark flats for each channel. You could do the same for the lights by the way. Just shoot darks of 60 seconds and then apply the master bias to scale them to the actual exposure time of the lights. That's what I do all the time.

In any case, it looks like you have discovered a bug in APP. When I only load the lights, flats and dark flats for L and the master dark and then integrate those I get a nearly perfectly corrected image:

The same for, respectively, Blue, Green and Red:

The final artifacts you probably can eliminate by making sure that the peak of the histogram of the flats is much further to the right. For some reason people think that the peak should be in the middle. And apparently this is what NINA tries to aim for as well. However, when the peak is much further to the right (without saturating any pixels) then a larger dynamic range of the sensor is used leading to much better correction.

I suspect that the bug is related to the fact that you have dark flats with different exposure times and it look like those do not get applied correctly to the flats. But I may be wrong and this is for Mabula to investigate. Thanks very much for your patience and for answering all of our questions!

@moviecells By the way, if you shoot, say, 67 bias frames of 0.01 seconds and load those with one set of dark flats and then also all flats and lights and the master dark, then you will see that you'll get the same result in one integration run. This is how I process all my data.

@wvreeven

So there is a bug there then ?

So to be sure how did you get the good final integrated images they look great want to try here.

As you can see from my results they have not being good

Mike

@wvreeven

So are you say you run the process one at a time for L then R then G then B and that worked but when running LRGB all in one process it did not work. I am correct i think thats what you indicated

Problem doing that would be it would not register the frames so they are stacked correctly

So if this is the problem a fix from Mabula would be great and a temp workaround if you can give me one, have load of nights of images i would love to process.

Mike

Posted by: @moviecells

So to be sure how did you get the good final integrated images they look great want to try here.

Load only the lights, flats and dark flats of L together with the master dark and integrate. Then repeat that for B, G and R.

@wvreeven

You put

some reason people think that the peak should be in the middle. And apparently this is what NINA tries to aim for as well. However, when the peak is much further to the right (without saturating any pixels) then a larger dynamic range of the sensor is used leading to much better correction.

So for the flats do you mean shift the ADU up so the peak is higher.

Mike

Posted by: @moviecells

Problem doing that would be it would not register the frames so they are stacked correctly

Once you have the four integration results then you can load those four as lights. APP will indicate that they already were processed and will ask if you are sure. Yes, you are sure. Then go to tab 4 and click the Register button. Once done, click the Save Registered Frames button and then you can channel combine them.

Posted by: @moviecells

So for the flats do you mean shift the ADU up so the peak is higher.

Exactly.

@wvreeven

Thanks for the work around for now but there will be a fix correct this is lot more work doing it this way i normal just load all the file and start it going on multiple night images it can take a good few hours to process. 

Having just paid for a full version and upgraded from yearly version only a few weeks ago feel a little let down.

If this is not working for me how are other people not having a problem, is there another way to do all this so i can load all the files and get good results.

Mike

@wvreeven

Hi it did not work i loading just Lum light, Lum flats, Lum darkflats and master dark and integrated but got this below still major problems. I got no error but this was the result not sure how your integrated looked just fine

if i switch the view options i can't see any difference from linear,image and calibrated they all look the same its like there is no correction applied. i saved a copy of the log if you need it

Posted by: @wvreeven

@moviecells Mike, when I load the lights, flats and dark flats in APP 1.083-beta2 I get the following warning:

The reason for this warning is that you have created dark flats for each flat exposure time and then try to integrate those per channel to create master dark flats for each of L, R, G and B. Apparently some of the exposure times do not agree which is why the popup is shown.

The correct way of doing is is to shoot one set of dark flats of for instance one second and then also shoot a large set of bias frames of 0.01 second. Then create a master bias and use that together with the dark flats to create a master dark flat. Those two together can then be used to properly calibrate the flats and create the master flats for each channel. Can you give that a try?

Hi @moviecells & @wvreeven,

The warning in this case is caused by the missing MasterDark for the light frame calibration. If there is no MasterBias or no MasterDark, Flat-Field calibration can never work because the sensor offset will not be removed from the light frames.

Mabula

@mabula and Wouter

I am getting more and more confused here.

I supplied files for flats and matching darkflats, light frames and a master dark. So how can it be missing. Wouter did you load the master dark when you had this error can you contact madula

More important is i don't see any errors my end and i still don't get calibrated images.

Wouter seems to think there is a bug when i load all my lights and process them all in one go so asked me to process them one at a time but i can't get it to work.

Could you talk to wouter not sure what going on and try and figure out the bug and also why i can't reproduce the method of doing one colour at a time as i tried L but get same uncalibrated image.

having just spent money out for a owners version all i want to do is process my images and get a good result

Mike

Hi @moviecells & @wvreeven,

I have checked the data, I integrated Lum,R,G,B in one go and get the same results as you, which are not good, especially the L-channel shows severe ringing.

If I process only the L data from scratch I get the exact same result as processing L,R,G,B from scratch, so in that sense I don't see a bug here in how APP calibrates the data by applying the different masters to the light frames.

The Luminance integration from the L,R,G,B processing, notice the wider field of fiew of the integration because it is registered for the whole data set:

The Luminance integration from only processing the Lum data, looks exactly the same with the same auto stretch (30% BG, 2sigma):

The L-calibrated image viewer per light shows the same problem. Michael, double click on 1 of the light frames in the frame list, set the image viewer to Linear and you see the uncalibrated raw data of the light frame:

Then set the image viewer to L-calibrated with the drop-down box above the image viewer and you will see the calibrated light frame where the masters that are assigned to the light frame are used. The masters that are assigned are shown in the frame column below the image viewer:

Since this is only 1 light frame, it is harder to see the ringing appear when compared to an integration of 5 of these light frames. But by further manually stretching, you see it clearly. I manually increased the stretch by lowering the ST slider and increased the Black Point by increasing the B-slider (Make sure the slider is set to 3-4 zoom level, so you can do this easily 😉 ):

So at first glance, i don't see any indication of a problem in APP's calibration module. I suspect the problem to be either:

• that the MasterDark is only 16bits, we can check easily if this is the problem if Michael creates a new 32bits MasterDark using the the same temperature, sensor gain + offset  as was used in creating this 16bits MasterDark. We do know that for these newer CMOS camera's 32bits masters are needed 😉 so there is a good change that this is our problem.
• The flats are simply not shot correctly, this can be due to light leakage or the method that was used to create the flats.

Mabula