Share:

[Sticky] Creating a Bad Pixel Map  

Page 1 / 3
  RSS

(@mabula-admin)
Quasar Admin
Joined: 2 years ago
Posts: 2127
June 5, 2017 19:08  

The first thing you should do in data calibration is to try to create a very good Bad Pixel Map of your sensor.

A good Bad Pixel Map will correct all bad pixels of your sensor.

Bad pixels are pixels that don't behave linear.

We have two types of bad pixels, cold and hot:

  • Cold Pixels - these pixels are either defect are respond really bad to incoming photons (and/or the dark current of your sensor). These will be detected using flat frames.
  • Hot Pixels - these pixels respond very strongly on incoming photons (and/or to the dark current of your sensor). These will be detected using dark frames.

For the detection of the cold pixels, only a couple of flats are needed to extract them robustly and add them to the bad pixel map.

For the detection of the hot pixels, you need to supply at least 20 darks and more is always better in this case.

The Bad Pixel Map is a Master Calibration frame that you only need to create once per year or a couple of years. ( I have been using one successfully for more than 3 years now for a Nikon D5100 BCF mod, firmware hacked with the nikonhacker true dark current patch).

How to create the Bad Pixel Map in APP:

Data is courtesy of Michel.

I am using 5 darks and 5 flats in this example from his Nikon D750.

Load the 5 darks and 5 flats and go to 2) CALIBRATE, scroll down to Bad Pixel Map, see screenshot:

app BPM1

Then select "create bad pixel map"  and set 2 parameters for hot and cold detection in your flats and darks.

 

The cold pixel % is a percentage of the median value of the integrated master flat that will be constructed from your flats.

"COLD" when ADU value of pixel  < (less than)  cold pixel % * (median value of the integrated master flat)

In most cases, the 50% works perfectly. Only if you have very strong vignetting, you should decrease this to 20-30%. Dead columns should be detected as well.

 

The hot pixels kappa detemines how much a pixel's ADU (Analog Digital Unit) value must be above the

localMedian + (hot pixels kappa* noise)

value to be marked as "HOT", so

"HOT" when ADU value of pixel  > (greater than) localMedian + (hot pixels kappa* noise)

with noise the gaussian MRS (Multi Resolution Scale) noise estimate of the integrated master darks created from the darks.

Usually a value of 2-3 kappa works perfectly to remove almost all hot pixels or hot columns if you have these on your sensor.

 

In this example I use

cold pixels 50% and hot pixels kappa 2.5

Now click on calibrate to create the Bad Pixel Map.

APP will integrate your darks and flats and will perform analysis on the integrations to subtract the hot and cold pixels of your camera's sensor. When it's finished no Master Dark or Master Flat is being saved to your disc, only a Bad Pixel Map. The reason being: when you create a Bad Pixel Map, you shouldn't use any outlier rejection filter at all while integrating the flats and darks to be able to better detect these spurious pixels. On the other hand, when you create a Master Dark or Master Flat, an outlier rejection filter can be used to create a better Master Dark or Master Flat.

For your convenience, when you create the Bad Pixel Map, APP will ignore the ourlier rejection filter options that are selected.

Once APP is finished with creating the Bad Pixel Map, it will be saved to your current Working Directory:

app BPM2
app BPM3

The values in the bad pixel map can take on 3 values:

  • 0 - cold pixel
  • 127 - linear pixel (so correct)
  • 255- hot pixel

It's best seen without the DPP stretch on, so click DPP to off in the preview filter to the right of the image viewer if not done so already.

Furthermore, you can find statistics on the amount of hot and cold pixels in the Fits Header.

In this case, we have 0 percent cold pixels which is normal for the newer DSLR camera's. And about 2% of hot pixels, normal values are between 0.1% - 3%. Try to find the kappa value that corrects all your bad pixels. You can verify this easliy with the l-calibrated image viewer mode.

An example of Bad Pixel correction using a Bad Pixel Map:

a Canon CR2 light frame of 600s exposure length, ISO200, Canon 450D (courtesy of Scott Rosen) corrected with a Bad Pixel Map created out of 100 darks of 600 seconds exposure length.

A crop of the sensor is shown, showing plenty of bad pixels which we want to remove.

ALL2

Top: uncalibrated    Bottom: calibrated

Left: bilinear debayering and no Linear Background Neutralization

Middle: bilinear debayering and Linear Background Neutralization

Right: Adaptive Airy Disc debayering and and Linear Background Neutralization (notice the dramatic increase in sharpness and better star shapes and change of colors 😉 due to AAD debayer).

Mabula

Main developer of Astro Pixel Processor and owner of Aries Productions


ReplyQuote
 Sara
(@swag72)
Red Giant Customer
Joined: 2 years ago
Posts: 67
June 5, 2017 19:38  

Brilliant - BPM done - Very simple!! Thanks!!!


ReplyQuote
(@mabula-admin)
Quasar Admin
Joined: 2 years ago
Posts: 2127
June 5, 2017 20:07  
Posted by: Sara

Brilliant - BPM done - Very simple!! Thanks!!!

Thank you Sara 😉 !

Main developer of Astro Pixel Processor and owner of Aries Productions


ReplyQuote
(@michel82)
Main Sequence Star Customer
Joined: 2 years ago
Posts: 49
June 6, 2017 21:33  
The Bad Pixel Map is a Master Calibration frame that you only need to create once per year or a couple of years.

Mabula,

how do i implement this master calibration file? or is the purpose of this file only to know the bad pixels for the inputs i need to make in APP?

Michel


ReplyQuote
(@mabula-admin)
Quasar Admin
Joined: 2 years ago
Posts: 2127
June 6, 2017 21:51  

Hi Michel,

Once you have created the Bad Pixel Map, load it ( using any of the light, flat, dark, bias buttons in 1) LOAD ) whenever you need to calibrate lights or flats.

The Bad Pixel Map will correct your hot and cold pixels without introducing any kind of noise. The pixels are corrected using interpolation from the surrounding "good" pixels.

Let me know if this is clear, I added a visual example using a Canon CR2 light frame the main post showing how efficient it can work.

Mabula

Main developer of Astro Pixel Processor and owner of Aries Productions


ReplyQuote
(@scott_rosen)
Red Giant Customer
Joined: 2 years ago
Posts: 59
June 16, 2017 01:38  

When I create a Bad Pixel Map, I end up with 2 files:

BPM-QSI690wsHW12.12.00FW07.01.01S_N00604166-3388x2712.fits

BPM-QSICCDCamera-3388x2712.fits

The first file looks black in the image viewer, whereas the 2nd one looks like a Bad Pixel Map - I can see column defects, bad pixels, etc. Can I presume that it's safe to just delete the 1st file and ONLY use the 2nd file for loading future BPMs?

Scott


ReplyQuote
(@mabula-admin)
Quasar Admin
Joined: 2 years ago
Posts: 2127
June 16, 2017 02:24  

Hi Scott,

The 2 Bad Pixel Maps are created because the files have different camera/instrument names in their headers. Probably the second will work then if you see the artefacts which you want to calibrate away.

Having different camera names in the headers while they are actually from the same camera can occur due to having used different capture software or even different versions of capture software.

To get the best Bad Pixel Map, all data should be used to detect the bad pixels, so probably we need to find a better way to use both camera names and assign it to one Bad Pixel Map.

When you load the Bad Pixel Map, it will only be used for calibration when the instrument/camera names of the Bad Pixel Map and light frames match.

You can manually adjust however at the moment using the batch modify tool. You can create new FITS files with an adjusted instrument name in the FITS header. You need to adjust the files so they match with the instrument fits tag in your light frames.

Mabula

 

Main developer of Astro Pixel Processor and owner of Aries Productions


ReplyQuote
(@scott_rosen)
Red Giant Customer
Joined: 2 years ago
Posts: 59
June 16, 2017 06:52  

Hi Mabula -

I think you're suspicion about using different capture software is correct.  I captured my darks a couple of months ago using Images Plus, whereas I captured my flats last night using SGP.  So, that explains it.  I think the BPM looks pretty good, but I'll plan on capturing some new darks with SGP to ensure matching names for a more certain BPM.

Thanks for your help!


ReplyQuote
(@_spirit_)
White Dwarf Customer
Joined: 2 years ago
Posts: 10
September 5, 2017 02:48  

Hi.

Don`t undestand. If BPM is loaded what about flats? After using BPM look like flat doesnt`t work on final stack.


ReplyQuote
(@mabula-admin)
Quasar Admin
Joined: 2 years ago
Posts: 2127
September 5, 2017 12:58  

Hi Roman,

BPM calibration is totally unrelated to flat-field calibration and is applied as the final calibration step if you have loaded both a masterflat and a Bad pixel Map (and masterbias' or masterdarks).

What do you mean by:

After using BPM look like flat doesnt`t work on final stack.

Are you trying to apply a masterflat on the integration file?

Have you checked the calibration of your frames with the l-calibrated image viewer mode ?

Cheers,

Mabula

Main developer of Astro Pixel Processor and owner of Aries Productions


ReplyQuote
(@gnomus)
Main Sequence Star Customer
Joined: 2 years ago
Posts: 46
September 5, 2017 18:27  

Hi Mabula

So to be clear, if I build a BPM then I should use this in place of Darks, is that correct?  Is there any advantage to using longer exposures for the Darks from which the BPM is generated - that is will a BPM from a 1200s exposure look teh same as one from a 600s exposure?  

Steve


ReplyQuote
(@mabula-admin)
Quasar Admin
Joined: 2 years ago
Posts: 2127
September 5, 2017 18:43  

Hi Steve,

No not really, the Bad Pixel Map can be used with or without a Master Dark.

I would only advise to not use a MasterDark if you don't have significant fixed patterns in the dark current of your sensor, or if you can dither the fixed patterns away, usually, you would need strong dithering (more than 20 pixels) then.

The Bad Pixel Map should be used always for both lights and flat frames, if you have one. It won't hurt SNR and will help you to get rid of cold/defect/hot pixels before the actual integration starts.

For instance, I used to calibrate data of my Nikon D5100 BCF mod in the following way:

the flats were calibrated with a masterflatdark (or is it masterdarkflat) and my BPM.

The lights were then calibrated with a masterdark, the masterflat and the BPM.

So, depending on your camera's sensor, you might want to use darks as well, together with the Bad Pixel Map. Whatever you do, just always use the BPM 😉

will a BPM from a 1200s exposure look teh same as one from a 600s exposure? 

The difference will probably be little. If you use darks of 10 seconds versus 300 seconds, then the difference will be much bigger. So, yes, with increasing exposure time the quality of the BPM will increase but 300seconds per dark should be more than enough. More influence will have the amount of darks that you use. The more darks that you load the better the statistics will be to isolate the hot pixels. So I would choose to do 20x 300s instead of 10x600s.

Kind regards,

Mabula

 

Main developer of Astro Pixel Processor and owner of Aries Productions


ReplyQuote
(@gnomus)
Main Sequence Star Customer
Joined: 2 years ago
Posts: 46
September 5, 2017 19:02  
Posted by: Mabula Haverkamp

Hi Steve,

No not really, the Bad Pixel Map can be used with or without a Master Dark.

OK - so how do I incorporate a BPM into the calibration then?  All I can load are lights, flats, darks and biases.  I was loading the BPM as a master dark.


ReplyQuote
(@mabula-admin)
Quasar Admin
Joined: 2 years ago
Posts: 2127
September 5, 2017 20:19  
Posted by: Steve
Posted by: Mabula Haverkamp

Hi Steve,

No not really, the Bad Pixel Map can be used with or without a Master Dark.

OK - so how do I incorporate a BPM into the calibration then?  All I can load are lights, flats, darks and biases.  I was loading the BPM as a master dark.

The BPM can be loaded (just as the other master calibration frames) using any load button. APP will recognize it's a BPM (or masterbias/masterflat/masterdark) from the fits header metadata.

In the bottom frame list panel. the frame column will indicate that it is a Bad Pixel Map.

Main developer of Astro Pixel Processor and owner of Aries Productions


ReplyQuote
(@gnomus)
Main Sequence Star Customer
Joined: 2 years ago
Posts: 46
September 5, 2017 21:43  
Posted by: Mabula Haverkamp
Posted by: Steve
Posted by: Mabula Haverkamp

Hi Steve,

No not really, the Bad Pixel Map can be used with or without a Master Dark.

OK - so how do I incorporate a BPM into the calibration then?  All I can load are lights, flats, darks and biases.  I was loading the BPM as a master dark.

The BPM can be loaded (just as the other master calibration frames) using any load button. APP will recognize it's a BPM (or masterbias/masterflat/masterdark) from the fits header metadata.

In the bottom frame list panel. the frame column will indicate that it is a Bad Pixel Map.

I see what I did.  I changed the name of the BPM to something a bit more memorable (to me).  When I loaded this in, APP just took it to be another 'light' (or whatever button I used).  So I assume I am not permitted to change the name of the BPM that APP created?  


ReplyQuote
(@mabula-admin)
Quasar Admin
Joined: 2 years ago
Posts: 2127
September 5, 2017 23:16  

Hi Steve, as long as the fits header isn't changed it really shouldn't affect APP from being able to recognize it as a BPM.

If a FITS file is loaded, using any button, the FITS header is examined and if the following similar header line is found:

CALFRAME= 'bad pixel map'      / bad pixel map for instrument CANON EOS 1200D

Then it should always be assigned as a BPM by APP.

I have just double checked, the same BPM is loaded three times, after having made 2 copies and changed the names, one with -kopie.fits and the other just randomly named 2.fits. All three are loaded as a BPM for the same instrument/camera as you can see in the screenshot below:

APP header BPM

So just changing the filename shouldn't be a problem, if the original BPM was converted to another FITS, then most likely the fits header line is gone and it won't be recognized by APP as a BPM. Could this be the case? If you changed it to a TIF, then it won't work either.

I have created it in such a way that the filename has no influence, the information needed is in the metadata. Maybe you can double check the metadata of the BPM file you refer to ?

Cheers,

Mabula

 

Main developer of Astro Pixel Processor and owner of Aries Productions


ReplyQuote
(@scott_rosen)
Red Giant Customer
Joined: 2 years ago
Posts: 59
September 6, 2017 06:28  

Mabula - some food for thought.  I think there have been other people that didn't realize you can load the BPM by choosing it as one of the "darks" in 1) Load.  Perhaps it would be helpful to rename the "Darks" button to "Darks + Bad Pixel Map"?


ReplyQuote
(@gnomus)
Main Sequence Star Customer
Joined: 2 years ago
Posts: 46
September 6, 2017 11:14  
Posted by: Mabula Haverkamp

Hi Steve, as long as the fits header isn't changed it really shouldn't affect APP from being able to recognize it as a BPM.

If a FITS file is loaded, using any button, the FITS header is examined and if the following similar header line is found:

CALFRAME= 'bad pixel map'      / bad pixel map for instrument CANON EOS 1200D

Then it should always be assigned as a BPM by APP.

I have just double checked, the same BPM is loaded three times, after having made 2 copies and changed the names, one with -kopie.fits and the other just randomly named 2.fits. All three are loaded as a BPM for the same instrument/camera as you can see in the screenshot below:

APP header BPM

So just changing the filename shouldn't be a problem, if the original BPM was converted to another FITS, then most likely the fits header line is gone and it won't be recognized by APP as a BPM. Could this be the case? If you changed it to a TIF, then it won't work either.

I have created it in such a way that the filename has no influence, the information needed is in the metadata. Maybe you can double check the metadata of the BPM file you refer to ?

Cheers,

Mabula

 

OK - I just checked it and you are correct.  (Same goes for Bias, Darks and Flats.)  I must have done something wrong last time.  


ReplyQuote
(@gnomus)
Main Sequence Star Customer
Joined: 2 years ago
Posts: 46
September 6, 2017 11:15  
Posted by: Scott Rosen

Mabula - some food for thought.  I think there have been other people that didn't realize you can load the BPM by choosing it as one of the "darks" in 1) Load.  Perhaps it would be helpful to rename the "Darks" button to "Darks + Bad Pixel Map"?

I think it doesn't matter which 'button' you use to load the BPM, APP always recognizes it as a BPM and uses it as such. 


ReplyQuote
(@scott_rosen)
Red Giant Customer
Joined: 2 years ago
Posts: 59
September 6, 2017 18:19  

Hi Mabula -

I guess what I'm saying is that you have buttons for each of the types of calibration frames, but none of the buttons say BPM.  As such, it's unclear how you would load a BPM.  So, it doesn't necessarily need to be Darks + BPM - even a separate BPM button would be useful.  I just think that something that indicated you can click it and load a BPM would help new users to know that's how you load a BPM.

Hope this helps clarify,

Scott -  http://www.astronomersdoitinthedark.com/


ReplyQuote
Page 1 / 3
Share: