Bad Pixel Map

Hi Hans,

No, this will have no significant effect on a Bad Pixel Map, assuming the Bad Pixel Map is created out of darks of considerably exposure length ( more than 60 seconds at least) and plenty of darks (at least 20).

The principle of the BPM is that it has information on the non-linear pixels of your sensor. With higher temperatures they (the hot pixels) are more apparent, so if you make darks for a bad pixel map you can just make them on room temperatures inside (with some coffee 😉 ). Once you have created a good BPM you can use it for months (or even years) to come. 

The temperature of darks is important for the dark current/signal & noise calibration, but not important for the creation of the BPM. If the BPM works, it just works and will keep working 😉

Cheers,

Mabula

Ha Mabula,

I just wondered about this. Thanks for your clear answer.

Making them inside looks like a good option.

Regards,

Hans

Ha Mabula,

As usual one question leads to an other.

I understand that with flats the orientation of your camara behind your telescope is important.

Is this not the case with the darks of the BPM?

Thanks,

Hans

Hi Hans,

Yes for the creation of flat frames, it's always recommended to shoot the flat frames with the exact same optical train as your light frames.

For the darks of the BPM however, this has zero influence. You can even make the darks for the BPM without the same telescope. For instance, with a DSLR, shoot them indoors with a regular objective. The only requirements are

• make sure no light enters the camera, so it's a real dark
• use an exposure time of at least 60 seconds per dark (more will be better, I would use 5 minutes per dark)
• shoot enough darks for good statistics on location of the pixels that behave non-linear. So shoot at least 20, in this case, more is always better, it will improve statistics.

If you are using a DSLR, you can use any ISO value, but I would recommend to use the same ISO value as your light frames 😉

Let me know if you have more questions.

Kind regards,

Mabula

Ha Mabula,

Thanks a lot for your answer this helps me a lot further.

A very last question.

If you use also flats for your BPM, what about the orientation of your DSLR connected to your telescope.

You can never get this exactly the same.

In the case by using flats for cold pixels do you have to make a new BPM every time?

Regards,

Hans 

Hi Hans,

Regarding the flats for the BPM, in this case, the orientation of the optics isn't important. APP uses the flats to detect if there are cold and/or defect pixels. Any set of flats will do for this purpose 😉 And just like with darks fot the hot pixels, you only need to make the BPM once. If it works, it just works and keeps working.

Kind regards,

Mabula

Ha Mabula,

Thanks !! Totaly clear.

Kind regards,

Hans 

In Sara Wager's BPM video it is said that a hot pixel part of around 3% should be expected. When I created a BPM for my Nikon D750 I got 12,1%, which seems like a lot. I tried with Kappa 3 and then 2, but had exactly the same result for both. Darks are 4 minutes exposure, 20 pcs.

Is this normal for a DSLR or is there something I might be doing wrong? 

Hi Heno,

Normally with a hot pixel kappa of 3 the amount of hot pixels will be somewhere between 2 and 4 %.

12,1% is way too much indeed, try increasing the kappa value until you have about 3%.

A possible explanation for this is probably the way Nikon camera's work. I have a Nikon D610 and D5100 so I have some Nikon camera experience.

My D5100 is firmware hacked using th etrue dark current patch from Nikon Hacker. That is a very big difference when compared to the unhacked camera.

The Nikon camera's have a noise suppresion algortihm that removes/destroys the ideal gaussian shaped noise distribution of the dark current. Furthermore, most Nikon camera's subtract the bias value internally, which doesn't help either. So for an unhacked Nikon camera, you might need to use a much higher kappa to get good results for your Bad Pixel Map. 

In addition to this, the Nikon D810a in this respect doesn't suffer from these problems like most of the other camera's, so Nikon surely improved big time with this camera for astrophotography.

And, the D750 is a great camera for sure, also for astrophotography, my D610 perfoms excellent as well ;-). It's good to realize however that the dark and bias signals are disturbed unfortunately by internal Nikon algortihms before storing the Raw data.

I highly recommend using dithering in your image capture setup in this case, are you able to dither between frames that you capture?

Kind regards,

Mabula

If you think something is fishy here, you're welcome to send me some of your darks so I can have a look myself.

Thanks Mabula for your complementary answer.

I will try to create a new BPM with a higher Kappa. If I don't succeed reducing the % bad pixels I will revert with some of the dark frames for you to inspect. 

Helge 

Hi Mabula.

I tried with Kappa = 6 and 10, but it had no effect whatsoever.

Wanted to attached a dark and a BPM, but could not, file types not allowed. Neither a ZIP file was allowed.

How do I send you these files?

Helge.

Hi Helge,

Are you using NEF frames? If not, then try with NEF frames first.

Otherwise, send the NEFs to me using wetransfer.com, use as e-mail address support@astropixelprocessor.com .

(ZIP, NEF, FITS frames aren't allowed to be attached to posts currently. I could enable is, but the attachments are mainly needed to add JPG, PNG images to a post. Attachments are also limited to 15MB in size, which is fine for JPG and PNG images.)

Kind regards,

Mabula

Yes. I'm using NEF.

Files sent as instructed.

Regards

Helge.

Hi Helge,

Thank you, I'll have a look right now and will let you know 😉

Mabula

Hi Helge,

I suspect something went wrong when you created the Bad Pixel Map, because the darks look fine and I can make a good Bad Pixel Map without problems.

• I loaded the 2 Nef Dark frames: using the dark load button:

• I zoomed in on one of the dark frames, to see if there are clear Hot pixels on the sensor. They are shown clearly:

• In 2) I enabled create Bad Pixel Map and kept the hot pixel kappa at 3, this produced a normal looking Bad Pixel Map, hot pixel count is very low, (this is due to Nikon's hot pixel removal internally before saving the raw data). The distinct pattern is very typical for the (Sony?) CMOS sensors used in your camera, my D610 and D5100 show the same patterns:

• The hot pixel count in the fits header shows only 0.184 percent of hot pixels. this is 1/5th of 1 percent. That is really low 😉

Can you try to reproduce this with just these 2 frames? And then try with 10 or 20 dark frames? Let me know what the results are 😉

I know from these Nikon camera's that creating a very good Bad Pixel Map, can be a bit tricky. This is mainly due to Nikon delivering Raws that aren't real raws anymore, due to the internal hot pixel removal algorithm. This algorithm doesn't behave the same on the hot pixels in all darks and therefore statistics on the hot pixels in these darks have increased uncertainty.

Kind regards,

Mabula

Hi Mabula and thank you so much for your help and efforts.

I made a BPM of two frames and that worked really well, as for you. Being an optimist I added 10 more darks, but the BP count went through the roof. So I tried with 6 darks, same result. So I tried with 3 darks and had the same result as for two frames, 0,138%. I then tried with four darks and got 4,35%. Anything more than that sends the % skyrocketing.

Something strange is going on here and you might want to investigate. I'll be happy to send you more darks if you want.

One suggestion: I would have liked to see the number of darks used (and flats if included) in the details of the PBM.  Now I have to remember which BPM was created from which (number of) files.

Best regards

Helge

Hi Helge,

Okay, yes, do send me more darks then so I can have a look at what's happening. I suspect it really has to do with the mentioned Nikon algorithms, but I do recall being able to create pretty good BPMs  for my Nikon D610.

Ok, all add the number of darks and flats info to the BPM. I have added this to my priorities list.

Kind regards,

Mabula

Hi Mabula

The dark files have now been sent.

Clear skies

Helge

Hi Helge,

My apologies for the late response, I am currently working on your problem.

Something is fishy indeed.

I'll report back later with my findings.

Mabula

Hi Mabula.

No reason for apologies. I'm sure you are a busy man.
I'm just glad that it isn't me that's fishy. 🙂
Take your time. I have my BPM made from 3 darks. That will do fine for now I'm sure.

Helge.

Hi Helge,

thanks 😉

The problem is quite clear, I'll do a test with my Nikon D610 which is quite similar.. should know more later today...

Mabula

Hi Helge,

I have found the problem.

I was using a noise calculation approach that was a problem with such a low amount of noise. A have changed the noise calculation to MSR gaussian noise and now it perfectly works with 6 darks.

So this will be fixed in the next APP version.

I do think that the BPM that you currently have based on only 2 darks should work pretty good.

But please check this BPM, this one is based on your 6 darks that you sent me with a hot pixel kappa of 2, nearly 1% hot pixels found now:

Thank you for sharing this problem 😉

Mabula

Posted by: Heno

 

One suggestion: I would have liked to see the number of darks used (and flats if included) in the details of the PBM.  Now I have to remember which BPM was created from which (number of) files.

Best regards

Helge

Hi Helge,

I have added this the FITS metadata 😉

FITS HDUs: 1
HDU1 - SIMPLE  =                    T / Java FITS: Sun Oct 22 20:09:49 CEST 2017      
HDU1 - BITPIX  =                    8 / bits per data value                           
HDU1 - NAXIS   =                    2 / number of axes                                
HDU1 - NAXIS1  =                 6032 / size of the n'th axis                         
HDU1 - NAXIS2  =                 4032 / size of the n'th axis                         
HDU1 - EXTEND  =                    T / Extensions are permitted                      
HDU1 - DATE    = '2017-10-22T18:09:49' / creation date of bad pixel map               
HDU1 - SOFTWARE= 'Astro Pixel Processor by Aries Productions' / software              
HDU1 - VERSION = '1.054   '           / Astro Pixel Processor version                 
HDU1 - CALFRAME= 'bad pixel map'      / bad pixel map for instrument NIKON D750       
HDU1 - INSTRUME= 'NIKON D750'         / instrument name                               
HDU1 - CFAIMAGE= 'no      '           / Color Filter Array pattern                    
HDU1 - NPIX    =             24321024 / raw number of pixels                          
HDU1 - HOTKAPPA= '2,00    '           / kappa value used for hot pixel determination  
HDU1 - COLDFRAC= '0,50    '           / percentage used for cold pixel determination  
HDU1 - NBADPIX =               201827 / number of bad pixels                          
HDU1 - PBADPIX = '0,830   '           / percentage of bad pixels                      
HDU1 - NHOTPIX =               201827 / number of hot pixels                          
HDU1 - PHOTPIX = '0,830   '           / percentage of hot pixels                      
HDU1 - NCOLDPIX=                    0 / number of cold pixels                         
HDU1 - PCOLDPIX= '0,000   '           / percentage of cold pixels                     
HDU1 - NLINPIX =             24119197 / number of linear pixels                       
HDU1 - PLINPIX = '99,170  '           / percentage of linear pixels                   
HDU1 - NUMDARKS=                    6 / # of dark frames used in creation of the BPM  
HDU1 - NUMFLATS=                    0 / # of flat frames used in creation of the BPM  
HDU1 - END                                                                            

This is the BPM:

Kind regards,

Mabula

Hi Mabula

Thanks for the BPM and thanks for solving the issue so quickly.
It is really cool to see ones proposal being adopted, even a small one like this. 😉

Clear skies

Helge

You're most welcome Helge 😉