[Sticky] Creating a Bad Pixel Map

Hi Stefan @elgol,

The BPM is created from the initial raw darks and/or flats, right? And I think this is DSLR data?

The lights that you are trying to calibrate are already cropped from their raw borders most likely and you will need to calibrate the raw data straight from the camera.

For instance, a Canon CR2 (or Nikon NEF) file has raw image dimensions of 7380x4928. You actually only see 7360x4912 since the raw borders of the sensor are cropped away. This always happens with DSLR data.

So if you save a CR2 to a fits, you lose the raw border data. And if you then try to calibrate the fits data with a BPM created from CR2's or NEFs, you will get this problem.

So if you make calirbation masters using DLSR raw frames, you will need to apply them on raw CR2/NEF frames, not on FITS frames.

Let me know if this clarifies your problem.

Kind regards,

Mabula

What happened with the "details" checkbox? I am making a new BPM and want to see the FITS header info, but i can not find an option to read the %hot pixels etc. (I now use Pixinsight to read the FITS header)

Hi Kees,

Yes, so that's a bit hidden in the new releases, probably needs to be changed as more seem to miss it. You can read the FITS header by clicking on the tiny arrows.

Thanks Vincent, "a bit hidden" is an understatement, let's call it a feature that is now an Easter Egg....   And, why? There is still all this room where the checkbox was, the tutorials refer to the checkbox etc, so why?

Hi, I´m writing from Mexico, I recently bough my first DSLR camera, is an used Nikon D5100 and I´ve noticed after using it on a trip that it has some hot pixels,is an horizontal straigth red line on the left of the camera sensor and it is only visible on some exposures. I´m new on this camera things so I hope that someone hera can tell me if I can solve this problem whit the program and how to do it, I have the factory version firmware. 

I can´t find the way to paste images in the comment, so is you can taell me how, I can post the photos here. thanks.

Hi @luis-manuel-samaniego-alvaez,

It is not allowed to post attachments on your first post on the forum, but now it should be possible 😉 Below the window where you type your message, there is a button Attach Files. You will need to use that to upload images with your posts.

is an used Nikon D5100 and I´ve noticed after using it on a trip that it has some hot pixels,is an horizontal straigth red line on the left of the camera sensor and it is only visible on some exposures. I´m new on this camera things so I hope that someone hera can tell me if I can solve this problem whit the program and how to do it, I have the factory version firmware.

Okay, do you know how to mark Dark frames ? If so, create a couple and try to create a Bad Pixel Map using those darks and please share how it looks together with one of the dark frames.

Kind regards,

Mabula

Hello Mabula, I am just trying to get a feel for APP and have chosen to create a BPM following your instructions. I have loaded 5 dark and 5 Flats.

I then scrolled down to create Bad pixel Map and selected the box. I am obviously doing something wrong since I do not get the Calibrate option.

This is my first attempt to create Flat frames so could this be the problem with quality. I use a cannon 550 D.

ThanksPeter

New to APP having just downloaded the trial version.  It all looks quite straightforward so I think I must be in 'dim' mode as I c an't persuade it to create a BPM or master calibration files!  For the BPM have loaded some 300 sec darks and some flats - lum only. After loading I press calibrate and under the BPM sub menu have hot pixels kappa set to 2 and cold pixels at 50%.  I make sure 'BPM' is checked and then click on calibrate again but nothing happens!  It must be something very obvious but it's beginning to drive me mad that I can't spot it!!

Pam

Hi Pam, no worries we should be able to get you going. Did you have a look at this short video from Sara Wager yet? https://www.astropixelprocessor.com/how-to-create-a-bad-pixel-map-bpm-by-sara-wager/

It might help you. If not, ask away.

Yes I've seen that .  The interface is slightly different to the one I have just downloaded.  I assume the video refers to an older version - I wonder if this is the problem.  Unless I've missed something very obvious - eminently possible knowing me - have followed it carefully.  Will have another look though. Have also tried following Sara's video on creating  master calibration files and can't persuade this to work either.  Am going to have an other look tomorrow and post again if I'm still stuck.  Thanks for a quick response.

It is very interesting though, it should be very straight forward and just work. You load in your longest darks, your flats, check the "create BPM" and press "create Masters". You keep the darks and flats selected in the list I assume?

I do wonder what you mean by pressing "Calibrate" now that I think of it. It should be the button in the above screenshot.

I think I've just been following Sara's video by rote instead of using my brain and working out that I should be pressing 'create masters' instead of pressing 'calibrate'  at the top  - will try that and engage my brain this time,  am sure that will work as you say it's quite straightforward!

Thanks for the response

Pam

Have now been able to create a BPM and master calibration files - what a relief!  Although APP creates a BPM just using darks and flats it obviously won't calibrate the flats without a master bias.  Does this affect the quality of the BPM?  Sorry for being a bit dim I can be a bit slow to understand some fundamentals though normally get there in the end!

Pam

Oh no problem Pam, we're here to help and it's always better to ask a question then to spend many hours on a simple misunderstanding. 🙂

The BPM is made with darks and flats and it won't create masters when "create bpm" is selected if I'm not mistaken (been a while). It's only creating a masterdark and masterflat for the purpose of the BPM and this is done without any outlier rejections. A proper masterdark and masterflat might benefit from a slight correction so this is why it's not created during this bpm process. The BPM will only consist of hot- and cold pixels which can be found in the darks and flats.

That makes sense.  

Thanks Vincent.

Pam

Should FlatDarks be used when making a BPM?

I think it does use them to calibrate the darks and/or flats during the process so I would include them yes (not 100% sure sorry, it's been a while since I created my BPM). Either that or bias frames.

Sorry guys to come back on the BPM creation. We have on the forum videos and explanations showing different processes and settings.

Sara and Mabula use darks and flats. Sara is setting: hot pixels kappa 2, cold pixels 25%. Mabula is setting: kappa 2.5 - 50% hot pixels.  Bad columns settings didn't exist at the time of the video and there is therefore no indication on these. 

Then nitesky here in the forum uses darks and darkflats (no flats), plus hot pixels kappa 2.4, cold pixels 25%. He does not check the bad columns settings. He mentions his hot pixels kappa is determined by the percentage of bad pixels he wants to limit to around 3%.

I have tried with flats or darkflats and I don't see (visually) big changes and numbers of bad pixels seem similar. What appears to be the most discriminating is the detection of bad columns. If not selected, I can reach the 3% bad pixels with bad pixels kappa set at 6 (seems high for me, is it?) and 50% cold pixels (attached file1).  Now when in addition, I check the bad columns detection, and leaving the bad pixels kappa at 6, the bad pixels % surges at 13% (attached file2)!! I then need to set the bad column kappa at 6 as well to come back to 4% bad pixels (attached file3). I would have to increase more the bad pixels kappa to lower the bad pixels to 3%.     

So, 3 questions:

1) Should we use flats? and/or darkflats in creating the BPM?

2) What is the rational for choosing the kappa (1-3% of bad pixels ?) and the cold pixels % (same?).

3) Should we check the Bad Column? (I understand from Mabula's comment that we should do this if we have bad columns... but I don't know if I have some). And what is the rational for choosing the kappa? Should we tweak these columns kappa independantly of the bad pixels kappa or together?

Sorry for this long post and thanks in advance for clarifying! 

Philippe

file2.fits

file1.fits

file3.fits

Also here, @mabula-admin will probably give an in-depth answer. But to answer quickly;

1. Yes, but they are only used for cold-pixel detection, which modern sensors rarely have anymore. So that won't affect the BPM much. Dark-flats are essentially bias subs, but with a slightly longer exposure. Might be some camera's need this. The longer exposure probably doesn't have many hot pixels, just the bias signal mainly which is definitely what you need.

2. Here you set what you think, based on results usually, is the threshold for APP to look for hot pixels in the darks and cold pixels in the flats (based on a percentage of the median signal in the flats). If you see that the BPM is correcting pixels that are not hot-pixels, you might want to change this setting. It'll vary per sensor, standard settings tend to be ok though.

3. Same here, the kappa will depend on your sensor and data, a lower value will detect (or if set too low, maybe detect columns that aren't actual bad columns) more columns, higher less. It's going to be an experiment for your data to see what works and what not. Again, usually the default is fine. If you're not sure you have bad columns, you probably don't have them. 🙂 I don't think it'll harm when you select it.

Hi Vincent,

Thanks! 

1) as you say, we have to set the kappa level to the point where no good pixels are wrongly removed and (reversely) no bad pixels stay in the final image. I am not sure though of being able to say how to check this (this is a question from a relatively newbie and seasonned users probably can notice that easily!). As you mentionned, if I have not noticed them I probably don't have them!

2) In my BPM (whichever version above), the amp glow from the ASI294MC is identified as bad pixels. That is probably why I have to raise the kappa level much higher than the recommanded 2 to 3  because the pixels forming the ampglow are many and with these values it announces 11-15% of bad pixels! Shouldn't the Amp Glow be corrected through the darks? Should I then keep the kappa level lower (2 to 3) even if it announces a high level of bad pixels (we are now back to point 1 I believe)?

Thanks!

Philippe

Hi Philippe,

1) Well, it's just a matter of looking at your data at 100% zoom and looking around. If the calibration worked well, there should be no or extremely little left of hot-pixels (bright red or bright blue). If you don't see them, it's fine. You should ideally see a nice, random noise pattern that remains even after all the calibration, no weird regular things line stuck pixels or entire columns.

2) Same here, just use the BPM with the bad pixel columns and without and test both. I have no experience with amp glow in this case so I'm not sure if the BPM can better be used without, but as far as I heard it should work. It's basically also bad pixels as it's fixed noise added to your subs.

Try it first without changing the kappa, they should be fine in most cases.

Perfect! Thanks Vincent!

Sorry if I overread it, but I understand the BPM as following:

I´ll take today some darks (30 - 50) and flats (30 - 50) with my modded Canon 1200Da and my Samyang 135mm. In the next step I´ll create a BPM with your wonderful software and in the future I do not have to take any more darks/flats?

With further integrations I´ll only proceed with those steps:

1.Creating the MasterBias

2.Load lights, add Masterbias and BPM

3.chill?

Is this restricted to some camera lenses or telescopes?

Thank you very much for your answere 🙂

Greets

Christian

I have just tried this, 5 Dark and 5 Flat but I always get this message.
What am I doing wrong or is it "normal" and does not need to be taken into account?
I have the latest APP version 1.079
Thanks and compliments great software

Posted by: @uwe1475

I have just tried this, 5 Dark and 5 Flat but I always get this message.
What am I doing wrong or is it "normal" and does not need to be taken into account?
I have the latest APP version 1.079
Thanks and compliments great software

 

As the warning indicates, you need a bias or dark flat to properly calibrate flats.

@gru89

Sorry, I missed your question. If your camera has no amp-glow, you can use the BPM in place of the darks. Flats however are always needed as they also correct dust and any changes in light across the frame, the flats used in a BPM are just there for detecting any cold pixels (if you have them at all).

@vincent-mod

Hello Vincent,
Thank you for your answer.
Yes, but I just want to create a BMP and not calibrate flats.
In Mabula's instructions, he only used darks and flats.

Do the darks still have their bias signal or did you precalibrate those?

I started APP then loaded 5 Dark and 5 Flat and ticked in 2) Calibrate Bad Pixel Map and then started the calibration as explained by Mabula.

I have now done a BPM with everything, Dark, Flats, DarksFlats and Bias and hope that this is correct so far.

It should be. Best practice is to load a dark with as high of an exposure as possible. You can check what the BPM is doing by loading in a light, zooming (100%) to a field with hot pixels, and switching on "l-calibrated" on top.