flat frame not bein...
 
Share:
Notifications
Clear all

19 June 2021: Our upload server https://upload.astropixelprocessor.com/ has been migrated successfully to our new office with higher upload and download speeds (nearly 10MByte/sec up/down ) ! We now have 1 general upload user called: upload with password: upload. The users upload1 - upload5 have been disabled.

31 May 2021: APP 1.083-beta2 has been released ! APP 1.083 stable will follow soon afterwards. It includes a completely new Star Reducer Tool, New File Saver Module, Improved Comet registration and much more, check the release notes here!

DOWNLOADS are available HERE!

 

flat frame not being applied  

Page 4 / 7
  RSS

 WB91
(@blanshan91)
Main Sequence Star Customer
Joined: 1 year ago
Posts: 31
July 22, 2021 00:03  

@mabula-admin  Uploaded. 


ReplyQuote
 WB91
(@blanshan91)
Main Sequence Star Customer
Joined: 1 year ago
Posts: 31
July 22, 2021 00:34  

@mabula-admin  "In the case where you just divide a light with a flat, that sensor offset/pedestal is not removed leading to a bad result logically. Thus that Rule Of Thumb should be: Divide a bias/dark subtracted light with a bias/darflat subtracted flat "

Thanks for your support!!  FYI, I was referring to standard light calibration workflow found online which seems to match what PI was doing in some of my tests using subtraction and division.  When I went the pixelmath direction using an inverse method for testing purposes, I had already corrected for darkflat and dark subtraction prior to my test.    Pixelmath can be funny sometimes with subtraction, not sure why.   Sometimes I have to find a work around.   

 

So let me ask you this,  because flats mean ADU values from user to user differs, do you scale the flat ADU to a curtain mean value in APP or leave it as is?     Example,  Re the data I sent you, if you scale the flat mean to 10% ADU and the darkflat accordingly, if I use this formula the calibration seems to work. (($T*~(MF*~MDF))*~MD).  Please dont be to critical with me over this test and math used, just testing here 😀   I have not tested this on other data yet, but was wondering if there could be a calculated correlation here to help with flat removal better.  My original Flat ADU was set to 33% and some people go higher, some people go as low as the light frame adu value, so I wanted to test  with different mean ADU values to see what had the best result, so I tried reducing it to a level of 10% and then used pixelmath which gave me a better result that what I was getting in APP and I tested APP also at different flat adu values.   That's why I wasn't sure if there was a bug.    but for what ever reason, I could not get a good calibrated light frame in APP with this data.   I have 4 nights of Ha imaging, all at the same exposure time and flat method for each night, and all have different results.  On another thread I posted an issue I was having with normalization too as when I normalize all data and save, some data doesn't correct as it should.   Anyhow, let me know what you find.   Thanks!   


ReplyQuote
(@mabula-admin)
Universe Admin
Joined: 4 years ago
Posts: 2692
July 23, 2021 22:23  

Hi Bill @blanshan91,

I see that you uploaded calibration masters, but if you want me to get to the bottom of this, I need the single calibration frames, not the masters.Could you upload 5 of each type (dark,bias, etc)?

I will have a look at the data and masters now, to see if I can already see something odd.

EDIT: I see that the masters have been altered by Pixinsight, so I can not use those masters to make any reasonable explanation here I am afraid. I have no clue what happened to the data in Pixinsight.

EDIT2: Assuming Pixinsight did nothing to change the data, I see that for the darks and darkflats different sensor offsets where used. Are you aware of this? Maybe the sensor offsets are the problem in your calibration workflow.

Mabula

This post was modified 2 months ago 2 times by Mabula-Admin

ReplyQuote
 WB91
(@blanshan91)
Main Sequence Star Customer
Joined: 1 year ago
Posts: 31
July 23, 2021 22:54  

@mabula-admin   I deleted the flat, darkflat and dark frames when I created the masters.  All masters were made using default APP settings.   Re the Pixinsight editing, yes, I had opened them in PI and saved them back as 16bit, nothing else was changed.  I had to do this as I wanted to use them in SharpCap for live stacking viewing.   For some reason, SharpCap would not open them right out of APP.  My Lights and Darks were done at Gain 150, Offset 50.   It could be my flats were done at Offset 30 by default, didnt think it would matter as a flat.  I will have to look into that though.   Note:  I don't see the offset info in the header, where do you see this?


ReplyQuote
(@moviecells)
Red Giant Customer
Joined: 1 year ago
Posts: 87
July 25, 2021 16:35  

Hi Well its safe to say it not IR i got a 4w IR torch so very bright and have no change on the camera on all filters even the Lum one when i shine it down the OTA at the dead of night. I also moved it around the area of the moonlite where i know there may be a very small light leak and again no change on the camera but this is before the filters. What i should have done is then used a normal white light torch around the moonlite so i will do that another night.

 

I will also try taking some flats at night but even on my day time flats and sky flats then it comes to the darkflat i don't see any light leak so i am sure that's not the issue. 

 

Starting to think its a internal reflection with in the reducer but its a good one a optec lepus x0.8

 

Mike

This post was modified 2 months ago by Michael Purver

ReplyQuote
(@wvreeven)
Quasar Admin
Joined: 3 years ago
Posts: 1110
July 25, 2021 16:51  

@moviecells OK, one possible cause eliminated. Did you make sure to create 32 bit masters in tab 2?


ReplyQuote
(@moviecells)
Red Giant Customer
Joined: 1 year ago
Posts: 87
July 25, 2021 18:47  

@wvreeven

I think 32 bit masters are the default in this version anyway yes it is ticked.

Mike


ReplyQuote
(@wvreeven)
Quasar Admin
Joined: 3 years ago
Posts: 1110
July 25, 2021 18:53  

@moviecells OK thanks for the confirmation. There are several sub-threads in this thread so I am getting lost for who is using which APP versions and settings. Could you maybe upload your data set to

https://upload.astropixelprocessor.com/

using upload both for username and password? Please create a directory called moviecells_flat_issue and put the files in there. Let me know when the upload is done and I'll have a look at the data.


ReplyQuote
(@mabula-admin)
Universe Admin
Joined: 4 years ago
Posts: 2692
July 27, 2021 22:34  
Posted by: @blanshan91

@mabula-admin   I deleted the flat, darkflat and dark frames when I created the masters.  All masters were made using default APP settings.   Re the Pixinsight editing, yes, I had opened them in PI and saved them back as 16bit, nothing else was changed.  I had to do this as I wanted to use them in SharpCap for live stacking viewing.   For some reason, SharpCap would not open them right out of APP.  My Lights and Darks were done at Gain 150, Offset 50.   It could be my flats were done at Offset 30 by default, didnt think it would matter as a flat.  I will have to look into that though.   Note:  I don't see the offset info in the header, where do you see this?

Hi Bill @blanshan91,

Okay, that is unfortunate.

Mixing calibration masters between different packages can create all sort of problems and unexpected behaviour, so if you want live stacking in SharpCap, please create masters in SharpCap 😉

If APP made 32bits masters, and you reduced them to 16bit in Pixinsight, again, this could create problems and I can not comment on what Pixinsight does internally as you can understand. APP can convert the files to 16bits as well, with Batch Modify Fits tool in 1.083-beta2 😉

If the flats had a different offset (30) then that will be the problem for sure. It matters a lot actually. If the flats were shot with offset 30 and the darkflats with offset 50 then that is the issue. No doubt. The sensor offset is vital for correct calibration results.

The lights needs bias/darks with the same offset as the lights.

The flats need bias/darkflats with the same offset as the flats.

So it is perfectly possible to use different offsets for lights and flats, but you need to be aware of matching the calibration frames as well 😉

The offset was not to be found in the header, but it is easily verified just by looking at the median/average values of the masters in the metadata. Or simply look at the histogram and check where the peak of the histogram is... the peak will be at the sensor offset or slightly higher with darks due to the added dark current signal.

Mabula.

This post was modified 2 months ago by Mabula-Admin

ReplyQuote
(@moviecells)
Red Giant Customer
Joined: 1 year ago
Posts: 87
July 27, 2021 23:19  

@wvreeven

 

Hi files have been uploaded if you could take a look it would be great

 

Mike


ReplyQuote
(@wvreeven)
Quasar Admin
Joined: 3 years ago
Posts: 1110
July 27, 2021 23:23  

@moviecells Thanks Mike. I'll have a look right away.


ReplyQuote
(@wvreeven)
Quasar Admin
Joined: 3 years ago
Posts: 1110
July 27, 2021 23:56  

@moviecells Mike, when I load the lights, flats and dark flats in APP 1.083-beta2 I get the following warning:

Screenshot 2021 07 27 at 17.52.01

The reason for this warning is that you have created dark flats for each flat exposure time and then try to integrate those per channel to create master dark flats for each of L, R, G and B. Apparently some of the exposure times do not agree which is why the popup is shown.

The correct way of doing is is to shoot one set of dark flats of for instance one second and then also shoot a large set of bias frames of 0.01 second. Then create a master bias and use that together with the dark flats to create a master dark flat. Those two together can then be used to properly calibrate the flats and create the master flats for each channel. Can you give that a try?


ReplyQuote
 WB91
(@blanshan91)
Main Sequence Star Customer
Joined: 1 year ago
Posts: 31
July 28, 2021 17:15  
Posted by: @mabula-admin
Posted by: @blanshan91

@mabula-admin   I deleted the flat, darkflat and dark frames when I created the masters.  All masters were made using default APP settings.   Re the Pixinsight editing, yes, I had opened them in PI and saved them back as 16bit, nothing else was changed.  I had to do this as I wanted to use them in SharpCap for live stacking viewing.   For some reason, SharpCap would not open them right out of APP.  My Lights and Darks were done at Gain 150, Offset 50.   It could be my flats were done at Offset 30 by default, didnt think it would matter as a flat.  I will have to look into that though.   Note:  I don't see the offset info in the header, where do you see this?

Hi Bill @blanshan91,

Okay, that is unfortunate.

Mixing calibration masters between different packages can create all sort of problems and unexpected behaviour, so if you want live stacking in SharpCap, please create masters in SharpCap 😉

If APP made 32bits masters, and you reduced them to 16bit in Pixinsight, again, this could create problems and I can not comment on what Pixinsight does internally as you can understand. APP can convert the files to 16bits as well, with Batch Modify Fits tool in 1.083-beta2 😉

If the flats had a different offset (30) then that will be the problem for sure. It matters a lot actually. If the flats were shot with offset 30 and the darkflats with offset 50 then that is the issue. No doubt. The sensor offset is vital for correct calibration results.

The lights needs bias/darks with the same offset as the lights.

The flats need bias/darkflats with the same offset as the flats.

So it is perfectly possible to use different offsets for lights and flats, but you need to be aware of matching the calibration frames as well 😉

The offset was not to be found in the header, but it is easily verified just by looking at the median/average values of the masters in the metadata. Are simply look at the histogram and check where the peak of the histogram is... the peak will be at the sensor offset or slightly higher with darks due to the added dark current signal.

Mabula.

Thanks for the reply!   FYI, I only tried to run in sharpcap after I noticed the flat removal wasn't working right for my data in APP, which was why I had to convert it to 16bit as sharpcap wasnt reading 32bit fits for flats.    Thanks for telling me about the 16 bit conversion options in APP, could be good to use in future.  Also, flats and flat darks are done at the same time in NINA so I dont think they have different offsets, unless NINA did something weird.  I will defiantly keep this in mind when using different camera offsets. 

 

Regarding FITS headers, you think you could bring over the original fits information from the lights and then you add your APP header info for the calibrated, normalized and final integration images so all info is available?   Obviously the integrated image header info will be slightly different but seems like more of the original header info could be added to the calibrated and normalized images.  Thoughts?       Also, will your next beta release have the ability to save APP settings or the ability to open with previously used settings so you don't have to reenter everything again? 


ReplyQuote
(@moviecells)
Red Giant Customer
Joined: 1 year ago
Posts: 87
July 28, 2021 20:16  

@wvreeven

 

I am using a darkflat and flats not bias its a cmos camera so bias files are not recommended for this type of camera.

I get no error my end when i load the darkflats, flats and lights you are sure you loaded them that way I have no bias frames. If you check you should see my flats and darkflats do match exposure, offset and gain there should be no errors. I used NINA flat/darkflat wizard its very good for this and mean there can be no mismatch to give an error.

 

Mike


ReplyQuote
(@moviecells)
Red Giant Customer
Joined: 1 year ago
Posts: 87
July 28, 2021 20:28  

@wvreeven

 

Can you check again i can't see why you are having a error my flats do match my darkflats. Also the statement about masterdark not matching the exposure time of the light it can't match. The exposure time of the flat must match the exposure of the darkflat which it does. I don't see any errors here this is what i do

Load all my light frames.

Load all my flat frames

Load all my Darkflat frames

Load my master dark

Then go to tab integrate and just run integrate

I don't see any error just not very good final integrated images.

 

Mike


ReplyQuote
(@wvreeven)
Quasar Admin
Joined: 3 years ago
Posts: 1110
July 28, 2021 22:31  

@moviecells Mike, bias frames may not be recommended (I shoot with a CMOS camera as well and are familiar with the recommendation) but it is essential for dark scaling. If the exposure times of darks and lights or flats do not correspond then APP needs a master bias to scale the darks.

By the way, for modern CMOS cameras flat exposure times of under a second are not recommended either due to the fact that these cameras have a rolling shutter. Too short exposure times may introduce unevenly flats that can introduce unwanted gradients and other issues. You may want to consider dimming your light source so you can use longer exposure times for the L flats.

I did indeed notice that the dark flat exposure times match the exposure times of the flats so there should not be an issue. And, when I load all files (so all lights, flats, dark flats and the master dark) then I indeed do not get that warning. However, I do indeed also get the uncorrected integration results. I'll check with Mabula.


ReplyQuote
(@moviecells)
Red Giant Customer
Joined: 1 year ago
Posts: 87
July 28, 2021 22:57  

@wvreeven

asking Mabula would be great not sure whats going on.

so how did you load the files and get the error ?

so are we saying i should be getting an error to indicate a problem and i am not ? so my lights are not corrected and i need bias files to correct this ?

mike

 


ReplyQuote
(@wvreeven)
Quasar Admin
Joined: 3 years ago
Posts: 1110
July 28, 2021 23:12  
Posted by: @moviecells

so how did you load the files and get the error ?

I only loaded the lights, flats and dark flats. Not the master dark. I asked Mabula why that gives an error.

Posted by: @moviecells

so are we saying i should be getting an error to indicate a problem and i am not ? so my lights are not corrected and i need bias files to correct this ?

That is what i thought at first but this is not the case. The exposure times of the flats and dark flats correspond per channel so a bias is not necessary. You could, however, consider taking bias frames so you only need one set of dark flats of, say, 1 second which you then can apply to all the flats instead of shooting dedicated dark flats for each channel. You could do the same for the lights by the way. Just shoot darks of 60 seconds and then apply the master bias to scale them to the actual exposure time of the lights. That's what I do all the time.

In any case, it looks like you have discovered a bug in APP. When I only load the lights, flats and dark flats for L and the master dark and then integrate those I get a nearly perfectly corrected image:

Screenshot 2021 07 28 at 17.06.48

The same for, respectively, Blue, Green and Red:

Screenshot 2021 07 28 at 17.07.03
Screenshot 2021 07 28 at 17.07.19
Screenshot 2021 07 28 at 17.07.40

The final artifacts you probably can eliminate by making sure that the peak of the histogram of the flats is much further to the right. For some reason people think that the peak should be in the middle. And apparently this is what NINA tries to aim for as well. However, when the peak is much further to the right (without saturating any pixels) then a larger dynamic range of the sensor is used leading to much better correction.

I suspect that the bug is related to the fact that you have dark flats with different exposure times and it look like those do not get applied correctly to the flats. But I may be wrong and this is for Mabula to investigate. Thanks very much for your patience and for answering all of our questions!


ReplyQuote
(@wvreeven)
Quasar Admin
Joined: 3 years ago
Posts: 1110
July 28, 2021 23:23  

@moviecells By the way, if you shoot, say, 67 bias frames of 0.01 seconds and load those with one set of dark flats and then also all flats and lights and the master dark, then you will see that you'll get the same result in one integration run. This is how I process all my data.


ReplyQuote
(@moviecells)
Red Giant Customer
Joined: 1 year ago
Posts: 87
July 28, 2021 23:34  

@wvreeven

 

So there is a bug there then ?

So to be sure how did you get the good final integrated images they look great want to try here.

As you can see from my results they have not being good

 

Mike

 


ReplyQuote
Page 4 / 7
Share: