15th Feb 2024: Astro Pixel Processor 2.0.0-beta29 released - macOS native File Chooser, macOS CMD-Q fixed, read-only Fits on network fixed and other bug fixes
7th December 2023: added payment option Alipay to purchase Astro Pixel Processor from China, Hong Kong, Macau, Taiwan, Korea, Japan and other countries where Alipay is used.
The first thing you should do in data calibration is to try to create a very good Bad Pixel Map of your sensor.
A good Bad Pixel Map will correct all bad pixels of your sensor.
Bad pixels are pixels that don't behave linear.
We have two types of bad pixels, cold and hot:
Cold Pixels - these pixels are either defect are respond really bad to incoming photons (and/or the dark current of your sensor). These will be detected using flat frames.
Hot Pixels - these pixels respond very strongly on incoming photons (and/or to the dark current of your sensor). These will be detected using dark frames.
For the detection of the cold pixels, only a couple of flats are needed to extract them robustly and add them to the bad pixel map.
For the detection of the hot pixels, you need to supply at least 20 darks and more is always better in this case.
The Bad Pixel Map is a Master Calibration frame that you only need to create once per year or a couple of years. ( I have been using one successfully for more than 3 years now for a Nikon D5100 BCF mod, firmware hacked with the nikonhacker true dark current patch).
I am using 5 darks and 5 flats in this example from his Nikon D750.
Load the 5 darks and 5 flats and go to 2) CALIBRATE, scroll down to Bad Pixel Map, see screenshot:
Then select "create bad pixel map" and set 2 parameters for hot and cold detection in your flats and darks.
The cold pixel % is a percentage of the median value of the integrated master flat that will be constructed from your flats.
"COLD" when ADU value of pixel < (less than) cold pixel % * (median value of the integrated master flat)
In most cases, the 50% works perfectly. Only if you have very strong vignetting, you should decrease this to 20-30%. Dead columns should be detected as well.
The hot pixels kappa detemines how much a pixel's ADU (Analog Digital Unit) value must be above the
localMedian + (hot pixels kappa* noise)
value to be marked as "HOT", so
"HOT" when ADU value of pixel > (greater than) localMedian + (hot pixels kappa* noise)
with noise the gaussian MRS (Multi Resolution Scale) noise estimate of the integrated master darks created from the darks.
Usually a value of 2-3 kappa works perfectly to remove almost all hot pixels or hot columns if you have these on your sensor.
In this example I use
cold pixels 50% and hot pixels kappa 2.5
Now click on calibrate to create the Bad Pixel Map.
APP will integrate your darks and flats and will perform analysis on the integrations to subtract the hot and cold pixels of your camera's sensor. When it's finished no Master Dark or Master Flat is being saved to your disc, only a Bad Pixel Map. The reason being: when you create a Bad Pixel Map, you shouldn't use any outlier rejection filter at all while integrating the flats and darks to be able to better detect these spurious pixels. On the other hand, when you create a Master Dark or Master Flat, an outlier rejection filter can be used to create a better Master Dark or Master Flat.
For your convenience, when you create the Bad Pixel Map, APP will ignore the ourlier rejection filter options that are selected.
Once APP is finished with creating the Bad Pixel Map, it will be saved to your current Working Directory:
The values in the bad pixel map can take on 3 values:
0 - cold pixel
127 - linear pixel (so correct)
255- hot pixel
It's best seen without the DPP stretch on, so click DPP to off in the preview filter to the right of the image viewer if not done so already.
Furthermore, you can find statistics on the amount of hot and cold pixels in the Fits Header.
In this case, we have 0 percent cold pixels which is normal for the newer DSLR camera's. And about 2% of hot pixels, normal values are between 0.1% - 3%. Try to find the kappa value that corrects all your bad pixels. You can verify this easliy with the l-calibrated image viewer mode.
An example of Bad Pixel correction using a Bad Pixel Map:
a Canon CR2 light frame of 600s exposure length, ISO200, Canon 450D (courtesy of Scott Rosen) corrected with a Bad Pixel Map created out of 100 darks of 600 seconds exposure length.
A crop of the sensor is shown, showing plenty of bad pixels which we want to remove.
Top: uncalibrated Bottom: calibrated
Left: bilinear debayering and no Linear Background Neutralization
Middle: bilinear debayering and Linear Background Neutralization
Right: Adaptive Airy Disc debayering and and Linear Background Neutralization (notice the dramatic increase in sharpness and better star shapes and change of colors 😉 due to AAD debayer).
Once you have created the Bad Pixel Map, load it ( using any of the light, flat, dark, bias buttons in 1) LOAD ) whenever you need to calibrate lights or flats.
The Bad Pixel Map will correct your hot and cold pixels without introducing any kind of noise. The pixels are corrected using interpolation from the surrounding "good" pixels.
Let me know if this is clear, I added a visual example using a Canon CR2 light frame the main post showing how efficient it can work.
The first file looks black in the image viewer, whereas the 2nd one looks like a Bad Pixel Map - I can see column defects, bad pixels, etc. Can I presume that it's safe to just delete the 1st file and ONLY use the 2nd file for loading future BPMs?
The 2 Bad Pixel Maps are created because the files have different camera/instrument names in their headers. Probably the second will work then if you see the artefacts which you want to calibrate away.
Having different camera names in the headers while they are actually from the same camera can occur due to having used different capture software or even different versions of capture software.
To get the best Bad Pixel Map, all data should be used to detect the bad pixels, so probably we need to find a better way to use both camera names and assign it to one Bad Pixel Map.
When you load the Bad Pixel Map, it will only be used for calibration when the instrument/camera names of the Bad Pixel Map and light frames match.
You can manually adjust however at the moment using the batch modify tool. You can create new FITS files with an adjusted instrument name in the FITS header. You need to adjust the files so they match with the instrument fits tag in your light frames.
I think you're suspicion about using different capture software is correct. I captured my darks a couple of months ago using Images Plus, whereas I captured my flats last night using SGP. So, that explains it. I think the BPM looks pretty good, but I'll plan on capturing some new darks with SGP to ensure matching names for a more certain BPM.
BPM calibration is totally unrelated to flat-field calibration and is applied as the final calibration step if you have loaded both a masterflat and a Bad pixel Map (and masterbias' or masterdarks).
What do you mean by:
After using BPM look like flat doesnt`t work on final stack.
Are you trying to apply a masterflat on the integration file?
Have you checked the calibration of your frames with the l-calibrated image viewer mode ?
So to be clear, if I build a BPM then I should use this in place of Darks, is that correct? Is there any advantage to using longer exposures for the Darks from which the BPM is generated - that is will a BPM from a 1200s exposure look teh same as one from a 600s exposure?
No not really, the Bad Pixel Map can be used with or without a Master Dark.
I would only advise to not use a MasterDark if you don't have significant fixed patterns in the dark current of your sensor, or if you can dither the fixed patterns away, usually, you would need strong dithering (more than 20 pixels) then.
The Bad Pixel Map should be used always for both lights and flat frames, if you have one. It won't hurt SNR and will help you to get rid of cold/defect/hot pixels before the actual integration starts.
For instance, I used to calibrate data of my Nikon D5100 BCF mod in the following way:
the flats were calibrated with a masterflatdark (or is it masterdarkflat) and my BPM.
The lights were then calibrated with a masterdark, the masterflat and the BPM.
So, depending on your camera's sensor, you might want to use darks as well, together with the Bad Pixel Map. Whatever you do, just always use the BPM 😉
will a BPM from a 1200s exposure look teh same as one from a 600s exposure?
The difference will probably be little. If you use darks of 10 seconds versus 300 seconds, then the difference will be much bigger. So, yes, with increasing exposure time the quality of the BPM will increase but 300seconds per dark should be more than enough. More influence will have the amount of darks that you use. The more darks that you load the better the statistics will be to isolate the hot pixels. So I would choose to do 20x 300s instead of 10x600s.
No not really, the Bad Pixel Map can be used with or without a Master Dark.
OK - so how do I incorporate a BPM into the calibration then? All I can load are lights, flats, darks and biases. I was loading the BPM as a master dark.
No not really, the Bad Pixel Map can be used with or without a Master Dark.
OK - so how do I incorporate a BPM into the calibration then? All I can load are lights, flats, darks and biases. I was loading the BPM as a master dark.
The BPM can be loaded (just as the other master calibration frames) using any load button. APP will recognize it's a BPM (or masterbias/masterflat/masterdark) from the fits header metadata.
In the bottom frame list panel. the frame column will indicate that it is a Bad Pixel Map.
No not really, the Bad Pixel Map can be used with or without a Master Dark.
OK - so how do I incorporate a BPM into the calibration then? All I can load are lights, flats, darks and biases. I was loading the BPM as a master dark.
The BPM can be loaded (just as the other master calibration frames) using any load button. APP will recognize it's a BPM (or masterbias/masterflat/masterdark) from the fits header metadata.
In the bottom frame list panel. the frame column will indicate that it is a Bad Pixel Map.
I see what I did. I changed the name of the BPM to something a bit more memorable (to me). When I loaded this in, APP just took it to be another 'light' (or whatever button I used). So I assume I am not permitted to change the name of the BPM that APP created?
Hi Steve, as long as the fits header isn't changed it really shouldn't affect APP from being able to recognize it as a BPM.
If a FITS file is loaded, using any button, the FITS header is examined and if the following similar header line is found:
CALFRAME= 'bad pixel map' / bad pixel map for instrument CANON EOS 1200D
Then it should always be assigned as a BPM by APP.
I have just double checked, the same BPM is loaded three times, after having made 2 copies and changed the names, one with -kopie.fits and the other just randomly named 2.fits. All three are loaded as a BPM for the same instrument/camera as you can see in the screenshot below:
So just changing the filename shouldn't be a problem, if the original BPM was converted to another FITS, then most likely the fits header line is gone and it won't be recognized by APP as a BPM. Could this be the case? If you changed it to a TIF, then it won't work either.
I have created it in such a way that the filename has no influence, the information needed is in the metadata. Maybe you can double check the metadata of the BPM file you refer to ?
Mabula - some food for thought. I think there have been other people that didn't realize you can load the BPM by choosing it as one of the "darks" in 1) Load. Perhaps it would be helpful to rename the "Darks" button to "Darks + Bad Pixel Map"?
Hi Steve, as long as the fits header isn't changed it really shouldn't affect APP from being able to recognize it as a BPM.
If a FITS file is loaded, using any button, the FITS header is examined and if the following similar header line is found:
CALFRAME= 'bad pixel map' / bad pixel map for instrument CANON EOS 1200D
Then it should always be assigned as a BPM by APP.
I have just double checked, the same BPM is loaded three times, after having made 2 copies and changed the names, one with -kopie.fits and the other just randomly named 2.fits. All three are loaded as a BPM for the same instrument/camera as you can see in the screenshot below:
So just changing the filename shouldn't be a problem, if the original BPM was converted to another FITS, then most likely the fits header line is gone and it won't be recognized by APP as a BPM. Could this be the case? If you changed it to a TIF, then it won't work either.
I have created it in such a way that the filename has no influence, the information needed is in the metadata. Maybe you can double check the metadata of the BPM file you refer to ?
Cheers,
Mabula
OK - I just checked it and you are correct. (Same goes for Bias, Darks and Flats.) I must have done something wrong last time.
Mabula - some food for thought. I think there have been other people that didn't realize you can load the BPM by choosing it as one of the "darks" in 1) Load. Perhaps it would be helpful to rename the "Darks" button to "Darks + Bad Pixel Map"?
I think it doesn't matter which 'button' you use to load the BPM, APP always recognizes it as a BPM and uses it as such.
I guess what I'm saying is that you have buttons for each of the types of calibration frames, but none of the buttons say BPM. As such, it's unclear how you would load a BPM. So, it doesn't necessarily need to be Darks + BPM - even a separate BPM button would be useful. I just think that something that indicated you can click it and load a BPM would help new users to know that's how you load a BPM.
Mabula - some food for thought. I think there have been other people that didn't realize you can load the BPM by choosing it as one of the "darks" in 1) Load. Perhaps it would be helpful to rename the "Darks" button to "Darks + Bad Pixel Map"?
Hi Scott,
The tooltips for the load buttons for light, flat, dark, and bias currently explain this behaviour:
Would this be clear enough ?
In version 1.045 I have corrected a lot of tooltips and added a lot more documentation in APP using tooltips 😉
Hi Steve, as long as the fits header isn't changed it really shouldn't affect APP from being able to recognize it as a BPM.
If a FITS file is loaded, using any button, the FITS header is examined and if the following similar header line is found:
CALFRAME= 'bad pixel map' / bad pixel map for instrument CANON EOS 1200D
Then it should always be assigned as a BPM by APP.
I have just double checked, the same BPM is loaded three times, after having made 2 copies and changed the names, one with -kopie.fits and the other just randomly named 2.fits. All three are loaded as a BPM for the same instrument/camera as you can see in the screenshot below:
So just changing the filename shouldn't be a problem, if the original BPM was converted to another FITS, then most likely the fits header line is gone and it won't be recognized by APP as a BPM. Could this be the case? If you changed it to a TIF, then it won't work either.
I have created it in such a way that the filename has no influence, the information needed is in the metadata. Maybe you can double check the metadata of the BPM file you refer to ?
Cheers,
Mabula
OK - I just checked it and you are correct. (Same goes for Bias, Darks and Flats.) I must have done something wrong last time.
Okay, confirmation is always good, thanks for your feedback Steve 😉
I guess what I'm saying is that you have buttons for each of the types of calibration frames, but none of the buttons say BPM. As such, it's unclear how you would load a BPM. So, it doesn't necessarily need to be Darks + BPM - even a separate BPM button would be useful. I just think that something that indicated you can click it and load a BPM would help new users to know that's how you load a BPM.
Thanks, yes you may have a valid point here, I'll see if I can add a button for loading of master calibration frames, together with the all/non togglebutton and clean button. That would make it definitely more user friendly. I agree 😉
I have added it to one of my direct priorities, because it's fast to implement.
The Bad Pixel Map can be created from one set of darks and flats.
darks, make sure that you have at least 20 darks (more is better) and that their exposure time is several minutes. The longer the dark current can build up the better the defects can be detected. So for a camera that you normally cool, you could even leave cooling disabled to get good darks for the BPM creation.
flats, 5-10 flats is fine.
The Bad Pixel Map has no dependency on exposure time and temperture, so it can be used on all your data once you have a good one. You can use a Bad Pixel Map for more than 1 year in msot cases. It depends on the sensor's quality and age. Some sensor's will deteriorate quicker than others.
If you have a sensor that has bad or defect columns, enable the bad column detection.
Let me know if this covers all your questions about a Bad Pixel Map.
Thank you for answering my question. The question was constructed very badly. What I meant to ask was:
Do you recommend that we redo our "old" BPM with 1.061 and if there is a new or different workflow with 1.061? I presume it would be better to redo BPM? For the sensor ageing sake as well as the improved methodology used?
Apologies if the next two questions are really stupid. Over reading, a lack of understanding and poor memory creates loads of confusion:
1. What about gain for the darks and the flats used in BPM?
2. How do I know if I have defect columns in my camera?
Only recreate a new BPM if you think the current one doesn't correct all bad pixels/columns. I ahev been using a single BPM for more than 3 years on Nikon DSLRs. For older and lower quality CCD camera's I think you will need to update the BPM more often though, because they can deteriorate rather clearly.
Workflow for applying the BPM in calibration hasn't changed. APP does a better job of correction the bad pixels though, but will do that equally on older Bad Pixel Maps.
gain for darks and flats in creating the BPM does not matter significantly. But simply use gain values that ar emore or less the same as you would normally use for the darks and flats. Same applies for the exposure time. For the darks, I do recommend to use darks of several minutes instead of several seconds. The idea is to properly sample deviations of pixels in the dark current as it builds up over time 😉
You can visually see it in your light frames, this is an example with hot and cold columns:
does anyone know why this causes a problem? I just createtd a BPM which will not be included in further processing as seen here. what am I blindly overseeing? (APP 1.061, never had this before)
