Please come all to AstroFest in London to ask us (Mabula & Vincent) questions and to see live demos of APP!
2023-01-19: APP 2.0.0-beta13 has been released !
!!! Big performance increase due to optimizations in integration !!!
and upgraded development platform to GraalVM 22.3 based on openJDK19
We are very close now to releasing APP 2.0.0 stable with a complete printable manual...
Bad Pixel Map
Is it better to make for every photo session a badpixel map because of the temperature differences of the darks used for the BPM?
No, this will have no significant effect on a Bad Pixel Map, assuming the Bad Pixel Map is created out of darks of considerably exposure length ( more than 60 seconds at least) and plenty of darks (at least 20).
The principle of the BPM is that it has information on the non-linear pixels of your sensor. With higher temperatures they (the hot pixels) are more apparent, so if you make darks for a bad pixel map you can just make them on room temperatures inside (with some coffee 😉 ). Once you have created a good BPM you can use it for months (or even years) to come.
The temperature of darks is important for the dark current/signal & noise calibration, but not important for the creation of the BPM. If the BPM works, it just works and will keep working 😉
I just wondered about this. Thanks for your clear answer.
Making them inside looks like a good option.
As usual one question leads to an other.
I understand that with flats the orientation of your camara behind your telescope is important.
Is this not the case with the darks of the BPM?
Yes for the creation of flat frames, it's always recommended to shoot the flat frames with the exact same optical train as your light frames.
For the darks of the BPM however, this has zero influence. You can even make the darks for the BPM without the same telescope. For instance, with a DSLR, shoot them indoors with a regular objective. The only requirements are
- make sure no light enters the camera, so it's a real dark
- use an exposure time of at least 60 seconds per dark (more will be better, I would use 5 minutes per dark)
- shoot enough darks for good statistics on location of the pixels that behave non-linear. So shoot at least 20, in this case, more is always better, it will improve statistics.
If you are using a DSLR, you can use any ISO value, but I would recommend to use the same ISO value as your light frames 😉
Let me know if you have more questions.
Thanks a lot for your answer this helps me a lot further.
A very last question.
If you use also flats for your BPM, what about the orientation of your DSLR connected to your telescope.
You can never get this exactly the same.
In the case by using flats for cold pixels do you have to make a new BPM every time?
Regarding the flats for the BPM, in this case, the orientation of the optics isn't important. APP uses the flats to detect if there are cold and/or defect pixels. Any set of flats will do for this purpose 😉 And just like with darks fot the hot pixels, you only need to make the BPM once. If it works, it just works and keeps working.
Thanks !! Totaly clear.
In Sara Wager's BPM video it is said that a hot pixel part of around 3% should be expected. When I created a BPM for my Nikon D750 I got 12,1%, which seems like a lot. I tried with Kappa 3 and then 2, but had exactly the same result for both. Darks are 4 minutes exposure, 20 pcs.
Is this normal for a DSLR or is there something I might be doing wrong?
Normally with a hot pixel kappa of 3 the amount of hot pixels will be somewhere between 2 and 4 %.
12,1% is way too much indeed, try increasing the kappa value until you have about 3%.
A possible explanation for this is probably the way Nikon camera's work. I have a Nikon D610 and D5100 so I have some Nikon camera experience.
My D5100 is firmware hacked using th etrue dark current patch from Nikon Hacker. That is a very big difference when compared to the unhacked camera.
The Nikon camera's have a noise suppresion algortihm that removes/destroys the ideal gaussian shaped noise distribution of the dark current. Furthermore, most Nikon camera's subtract the bias value internally, which doesn't help either. So for an unhacked Nikon camera, you might need to use a much higher kappa to get good results for your Bad Pixel Map.
In addition to this, the Nikon D810a in this respect doesn't suffer from these problems like most of the other camera's, so Nikon surely improved big time with this camera for astrophotography.
And, the D750 is a great camera for sure, also for astrophotography, my D610 perfoms excellent as well ;-). It's good to realize however that the dark and bias signals are disturbed unfortunately by internal Nikon algortihms before storing the Raw data.
I highly recommend using dithering in your image capture setup in this case, are you able to dither between frames that you capture?
If you think something is fishy here, you're welcome to send me some of your darks so I can have a look myself.
Thanks Mabula for your complementary answer.
I will try to create a new BPM with a higher Kappa. If I don't succeed reducing the % bad pixels I will revert with some of the dark frames for you to inspect.
I tried with Kappa = 6 and 10, but it had no effect whatsoever.
Wanted to attached a dark and a BPM, but could not, file types not allowed. Neither a ZIP file was allowed.
How do I send you these files?
Are you using NEF frames? If not, then try with NEF frames first.
Otherwise, send the NEFs to me using wetransfer.com, use as e-mail address firstname.lastname@example.org .
(ZIP, NEF, FITS frames aren't allowed to be attached to posts currently. I could enable is, but the attachments are mainly needed to add JPG, PNG images to a post. Attachments are also limited to 15MB in size, which is fine for JPG and PNG images.)
Yes. I'm using NEF.
Files sent as instructed.
Thank you, I'll have a look right now and will let you know 😉
I suspect something went wrong when you created the Bad Pixel Map, because the darks look fine and I can make a good Bad Pixel Map without problems.
- I loaded the 2 Nef Dark frames: using the dark load button:
- I zoomed in on one of the dark frames, to see if there are clear Hot pixels on the sensor. They are shown clearly:
- In 2) I enabled create Bad Pixel Map and kept the hot pixel kappa at 3, this produced a normal looking Bad Pixel Map, hot pixel count is very low, (this is due to Nikon's hot pixel removal internally before saving the raw data). The distinct pattern is very typical for the (Sony?) CMOS sensors used in your camera, my D610 and D5100 show the same patterns:
- The hot pixel count in the fits header shows only 0.184 percent of hot pixels. this is 1/5th of 1 percent. That is really low 😉
Can you try to reproduce this with just these 2 frames? And then try with 10 or 20 dark frames? Let me know what the results are 😉
I know from these Nikon camera's that creating a very good Bad Pixel Map, can be a bit tricky. This is mainly due to Nikon delivering Raws that aren't real raws anymore, due to the internal hot pixel removal algorithm. This algorithm doesn't behave the same on the hot pixels in all darks and therefore statistics on the hot pixels in these darks have increased uncertainty.
Hi Mabula and thank you so much for your help and efforts.
I made a BPM of two frames and that worked really well, as for you. Being an optimist I added 10 more darks, but the BP count went through the roof. So I tried with 6 darks, same result. So I tried with 3 darks and had the same result as for two frames, 0,138%. I then tried with four darks and got 4,35%. Anything more than that sends the % skyrocketing.
Something strange is going on here and you might want to investigate. I'll be happy to send you more darks if you want.
One suggestion: I would have liked to see the number of darks used (and flats if included) in the details of the PBM. Now I have to remember which BPM was created from which (number of) files.
Okay, yes, do send me more darks then so I can have a look at what's happening. I suspect it really has to do with the mentioned Nikon algorithms, but I do recall being able to create pretty good BPMs for my Nikon D610.
Ok, all add the number of darks and flats info to the BPM. I have added this to my priorities list.
The dark files have now been sent.
My apologies for the late response, I am currently working on your problem.
Something is fishy indeed.
I'll report back later with my findings.