No dark subs anymore?

I forgot to mention the use of dithering for dslr! 😉

Hi Guido,

1) yes, create 1 good Bad Pixel Map, use this for the coming year of even longer 😉 in all your data calibration

2) the use of darks frames for dark calibration could still be required depending on the sensor characteristics.

• If your sensor has clear fixed pattern noise in the dark current (horizontal and/or vertical stripes that aren't visible in the bias frames) you would still need to use darks, or otherwise, dither very aggresively (20-50 pixel).
• if your sensor has strong amp glow, then darks are really needed. Dithering can't really solve that.

You can check the fixed pattern noise of the dark current and the amp glow, by making a masterbias of let's say 40 bias frames and a masterdark of 40 darks (use a couple of minutes exposure time). Then stretch them both equally and look for differences. Here is an example of a Canon EOS 550D (courtesy of Ed Defesche). The first image is the masterbias, the second one is the masterdark. You can clearly see horizontal stripes in the masterdark that aren't visible in the masterbias. In this case, you need to use dark calibration or dither very aggresively, because this is fixed pattern noise in the dark current.

3) usually, you would need to create the masterbias also only once per year give or take, it shouldn't be that temperature sensitive. Couple of years back, I created masterbias frames of 250 bias frames (both iso 200, iso 400)) and I am still able to use those single masterbias frames to good effect 😉 (for both Nikon D5100 with dark current hack/patch and a Nikon D610)

So unless you have strong dark current patterns and amp glow, then with strong dithering you would only need to calibrate with bias, flat and BPM and that would indeed give very good results for DSLR astrophotography.

As a side note, dithering will always help a lot anyway, so try to incorporate that into your capture method 😉

Hi Mabula,

Can i use darks from different dates (same exposures) to make the master dark?

Hi Guido,

Yes, you can do this, but I would recommend to create them at more or less the same temperature for the best results 😉

If the temperature difference is too big between the nights (more than 5 degrees Celsius difference) then results will probably be less than optimal.

I intend to implement scaling of darks soon to solve this problem 😉

Cheers,

Mabula

Thanks Mabula!

I did the test...and strong dithering will be necessary...keep you posted! 😉

Cheers

Excellent Guido 😉

Hi Mabula,

I made it yesterday without using Darks. Only +/- 25 lights, bias, flats and BMP 50 minutes. I had to stretch it a bit to see the colours better, using Gimp aswell. Do you think this is the way for me? Supose i'll have 4 hours (i go on a holliday South of France ) 😉 will the quality (details) be significantly better?The dithering was +/- 10, maybe a bit more? Or do i need darks?

Cheers

Guido

Cheers

Hi Guido,

I don't think I see any problem in the calibration resulting from uncalibrated dark current signals like horizontal patterns or amp glow. It looks good to me.

Do you think this is the way for me?

So, yes I do think so 😉

Dithering of +/- 10 pixels is enough then I think, but a little bit more can't hurt as well.

Supose i'll have 4 hours (i go on a holliday South of France ) will the quality (details) be significantly better?

It certainly will 😉 ! 4 hours versus 1 hour normally means 2x less noise roughly, and darker skies are always better !

If you combine more hours of data, you'll be able to stretch the data more. Possible calibration problems will then show more easily off course.  But this looks pretty good. To be on the safe side, I would suggest to maybe dither a bit more than 10 pixels so removal of any small/narrow dark current signals will be very efficient while integrating.

Kind regards,

Mabula

Hi Mabula, one more question on that matter. If you have quite a bit of Amp glow and you use darks, do you recommend to also use the bpm or is this redundant?

Best Frank 

Hi Mabula, not sure if this hasn't popped up so I just quickly bump this question 🙂 Thanks in advance for your help!

Posted by: Frank

Hi Mabula, one more question on that matter. If you have quite a bit of Amp glow and you use darks, do you recommend to also use the bpm or is this redundant?

Best Frank 

Hi Frank,

Good question, yes, amp glow should be corrected with darks to get the best results.

And yes, always use a BPM 😉 darks or no darks.

A BPM will simply correct the hot and cold pixels and will never harm your data, it won't inject noise. You can and want to apply the BPM in any calibration workflow.

Mabula

Fabulous, thanks Mabula!!!

Hi Mabula,

I've had two problems with the 1.055 version.

The first is that when creating a working folder, the available HD space disappears and a relevant warning appears, requiring a restart.

The more difficult issue is that the master dark is much brighter than the individual dark frames, leading to the calibrated lights being almost blacked out. trying again without using darks is the only way to get even a reasonable result. A nuisance when processing 300 lights.

Neil

Hi again Mabula,

Things are a little weird now. Upon trying yet again, the master dark looks OK, but the Bad Pixel Map looks bright. Regardless as to what I think the cause is, the following is the result at the calibration step.

Neil

Hi Neil,

Thank you for sharing your issues:

The first is that when creating a working folder, the available HD space disappears and a relevant warning appears, requiring a restart.

Do you mean that the HDD space disappears if you manually remove the current work directory so it no longer exists? Or does it fail when you create a new folder? Maybe you can show a screenshot of the moment when it fails, so I have a better understanding where the problem might be?

The more difficult issue is that the master dark is much brighter than the individual dark frames, leading to the calibrated lights being almost blacked out. trying again without using darks is the only way to get even a reasonable result. A nuisance when processing 300 lights.

The master dark will never be much brighter than the individual dark frames, I suspect it seems this way due to the auto DDP function. Since the masterdark will have less nois, the autostretch will be stronger giving the impression that the masterdark is brighter, where in fact it isn't. You can test this, load a single dark frame into the image viewer, then disable the auto DDP function ( auto selectbox on the right next to the DDP selectbox) and load your masterdark. Then it should have teh same brightness, and you should notice that the masterdark is less noisy.

"leading to the calibrated lights being almost blacked out."

So the cause of this is not the masterdark's brightness but most likely  a wrong calibration workflow. Probably you are subtracting the bias pedestal tiwce from your lights by using both a masterbias and a masterdark that didn't have the masterbias subtracted, but this is just a guess. It will help me if you can tell me what calibration files you used, which gain/iso, exposure and also the gain/iso, exposure of your lights. Then we should be able to solve this.

The Bad Pixel Map on the other hand, will always look bright in APP. The linear pixels will have ADU values of 127, cold pixels of 0 and hot pixels will have 255 ADU values. This is a simple bitmap file that stores the location and behaviour of the pixels on your sensor.

Before starting processing at step 3), always try to check if calibration is working like expected, using the l-calirbated image viewer mode when you have your light frames and master calibration files loaded, if the image doesn't look good with the l-calibrated image viewer mode, then you will need to first fix the calibration before proceeding.

Kind regards,

Mabula

Hi Mabula,

The first is that when creating a working folder, the available HD space disappears and a relevant warning appears, requiring a restart.

This issue has not reoccurred, so I can't send a screen shot. It seemed to happen several times when I opened a new working folder. A Windows 7 issue?

The more difficult issue is that the master dark is much brighter than the individual dark frames, leading to the calibrated lights being almost blacked out. trying again without using darks is the only way to get even a reasonable result. A nuisance when processing 300 lights.

After sending my previous post, realised the Bad Pixel Map would have been stretched .

I have been following the same workflow since first installing APP. On experimenting with smaller data sets, I found that if the Darks were left out, it worked reasonably well except for noisy calibrated Lights, and if I left out the Bias frames instead, they produced was very reasonable calibrated Lights. I may have to revisit my workflow, though what I have used so far had greatly improved results in previous work from DSS. It's just this set of Lights that have given me issues. I even re-acquired the Darks and Bias frames as near to the temperatures of the Lights as possible to try to eliminate any mistakes with no change in results.

It will help me if you can tell me what calibration files you used, which gain/iso, exposure and also the gain/iso, exposure of your lights. Then we should be able to solve this.

Lights - Canon raw CR2 files - 30 seconds at iso800 x 300

Darks - Lens cap replaced on telescope - 30 seconds at iso800 x 30

Bias - immediately after the Darks -1/400th sec at iso800 x 30

Flats -  inside house with white board target lit by leds - iso800 at 1/6th second to obtain mid "grey" x 30

As I said above, this has always worked before. Coincidentally, this happened after upgrading to 1.055.

Probably you are subtracting the bias pedestal tiwce from your lights by using both a masterbias and a masterdark that didn't have the masterbias subtracted, but this is just a guess.

I agree, based on my limited understanding of the process in theory. I may just have to do a lot more limited set trials to reduce the processing times and try some of your ideas.

Neil

Hi Neil,

Thank you for all the information.

The first is that when creating a working folder, the available HD space disappears and a relevant warning appears, requiring a restart.

This issue has not reoccurred, so I can't send a screen shot. It seemed to happen several times when I opened a new working folder. A Windows 7 issue?

I will do some testing, to see if I can find a problem here. I don't think it's a typical Windows 7 issue, since I have been using APP on windows 7 without this problem occuring.

The more difficult issue is that the master dark is much brighter than the individual dark frames, leading to the calibrated lights being almost blacked out. trying again without using darks is the only way to get even a reasonable result. A nuisance when processing 300 lights.

After sending my previous post, realised the Bad Pixel Map would have been stretched .

No, the Bad Pixel Map will automatically be shown unstretched, at least in the lastest APP version, in older versions it would be stretched. So the BPM should like gray, with white pixels indicating hot pixels and black pixels indicating cold/defect pixels.

I have been following the same workflow since first installing APP. On experimenting with smaller data sets, I found that if the Darks were left out, it worked reasonably well except for noisy calibrated Lights, and if I left out the Bias frames instead, they produced was very reasonable calibrated Lights. I may have to revisit my workflow, though what I have used so far had greatly improved results in previous work from DSS. It's just this set of Lights that have given me issues. I even re-acquired the Darks and Bias frames as near to the temperatures of the Lights as possible to try to eliminate any mistakes with no change in results.

It will help me if you can tell me what calibration files you used, which gain/iso, exposure and also the gain/iso, exposure of your lights. Then we should be able to solve this.

Lights - Canon raw CR2 files - 30 seconds at iso800 x 300

Darks - Lens cap replaced on telescope - 30 seconds at iso800 x 30

Bias - immediately after the Darks -1/400th sec at iso800 x 30

Flats -  inside house with white board target lit by leds - iso800 at 1/6th second to obtain mid "grey" x 30

As I said above, this has always worked before. Coincidentally, this happened after upgrading to 1.055.

Okay, so all data is shot with ISo 800.

I would recommend the following workflow:

FIrst create a BPM.

1)Load the darks and the flats and create a BPM with default settings. (only need to create a BPM once, then you can use it for all other data)

Then the regular calibration master frames:

1) make a masterbias of the bias frames.

2) clear the frame list

3) load all flats and your masterbias and create a masterflat. This will subtract the masterbias from your flats.

4) clear the frame list

5) load your darks and create a masterdark. (since you have darks, don't use the masterbias in this case)

6) Load you lights, the masterdark, the masterflat and the BPM and calibrate your lights, (so don't load the masterbias). Verify the calirbation with the l-calibrated inage viewer mode.

That should work properly. Let me know if it does.

Kind regards,

Mabula

Interesting so you suggest not subtracting the master bias from dem the darks ? I always considered doing that and then using the darks and the bias for the lights. Actually worked ok with me

Best Frank 

Hi Frank,

Yes, that can work as well, but since the bias is contained in the darks, it's not needed to do so.

And I can confirm that not subtracting the masterbias form the darks is for now the recommended method. This has to do with clipping of the data in the calibration engine. This is a problem that will be fixed most likely in 1.057. Then it won't matter which route you take 😉

Mabula

Wow. Are you saying to abandoned bias and use only darks? I just got around to dithering so I don't have to take darks. But there was not the best result without using Tarts so I use dark from another session and it worked better I'm confused.

Hi Greg,

No not exactly.

If you are going to use dark frame subtraction, the bias subtraction is not really necessary since it is already contained in the darks.

(But... in future versions, I'll implement dark scaling, then you will need to supply both bias and dark frames to be able to use dark scaling)

Dark frame subtraction is needed in certain cases

1. your sensor has clear fixed noise patterns in the dark current, like horizontal banding
2. your sensor has clear amp glow.

1) can sometimes be fixed without darks, but you'll probably need very strong dithering, i.e. more than 40 pixels per dither to get rid of the fixed noise patterns

2) can't be fixed with dithering

If you don't have these 2 problems with your sensor, maybe the calibration is better without using darks at all. It really depends on the sensor's characteristics, so there is not a general "best"calibration path.

Kind regards,

Mabula

Thanks Mabula,

I have a Canon t3i on a refractor, so yes have fpn.  I'll try a larger dither, and take some darks. Hopefully i can take minimal darks, say 100 lights, 10 darks, 30 flats and a master bias.

I went back and processed some old subs with APP. These are my first successes with APP, however they are also the first images I've taken. They are from the summer of 2016 and are under an hour integration, but turned out fantastic compared to the DSS-PS route. What helped alot was doing a background and star calibration post processing( all in APP). How large of a jpg can i attach here and Ill post them. These are from full data sets. Until these, my results were always as if there were no flats. Always came out extreme gradients. This set, although not very good, came out just right. APP seems sensitive to good flats, more so tha DSS.  I now see the potential here and like it! 

M33 ALWAYS turned out blueish using my old methods. If I ignore the red pixel streak here, this is the best rendition of m33 i have goten with respect to color balance and background. 

Equip: Canon t3i, ES127APO.

Hi Greg,

Very nice images indeed 😉

Thank you for your feedback !

You can attach images of 30MB size on the forum, that should be plenty 😉

Cheers

Mabula 

Thanks Mabula,

Also, these are unguided subs so I found the final integration worked best when the weight was put on star shape 😉

Here is m13 processed:

I am now trying a recent shoot of m42 with the old flat and dark sets....temperature be *#!!$*#

And redone again with light pollution tool.

Hi Greg,

Indeed excellent, if you have exposures with strongly varying star shapes, which will happen with unguided exposures, the star shape weights integrate setting is a very good option to get the tightest and roundest stars 😉

Mabula

Hi Mabula,

Your suggested workflow made a huge difference. I can see the logic of your process, I just don't know why the method I used before didn't work this time. Maybe I'll go back and try reprocessing all my previous work. It may be even better than the great results I had before.

Thank you for an amazing program and your help.

Neil

Hi Neil,

Excellent and thank you 😉 you're most welcome !

The calibration engine currently takes some explaining, and isn't "smart". That probably explains why it previously didn't work. But I intend to upgrade the calibration engine in 1.057 big time 😉

Tomorrow, I'll probably release 1.056 with lots of other improvements...]

Mabula

Hi Mabula,

The zero hard drive error occurred again. It happened after opening a new work directory.

Neil