black screen after integration

Hi there,

I too am experiencing this very same issue. My darks are the culprit, but I've done them the exact same way as I always have. Offset (20) and gain (90) match my light frames because that's what I always set it to.

This definitely seems to be a bug. I've seen multiple forum posts, including one from today, about this issue.

I agree with Brian. I have not changed anything in my process (same darks, same settings as always), yet the image is coming black once I updated to newer version of APP.

Please fix this bug!

Hi All, @dickvantatenhove, @jan-monsuur, @nobbybuddy, @morrienz, @vincent-mod, @tchylek, @absorbingphotons

Let's solve this issue then!

I will need to see all data sets to understand what is happening. I have so many data myself and test data of other APP users  and I have never seen this problem appear myself, so at the moment i can only assume something fishy is happening in either

• APP or
• in the data itself (sensor gain/offset problems)
• or in the capture software that is being used...

Anyway, without seeing the data it will be hard to solve this and give you a good explanation.

Please upload your data if you have not done so already here:

https://upload.astropixelprocessor.com/

username and password : upload

Please let us know when uploaded and we will test a.s.a.p.

Mabula

Hi All, @dickvantatenhove, @jan-monsuur, @nobbybuddy, @morrienz, @vincent-mod, @tchylek, @absorbingphotons

In addition, if the issue is in fact caused by a problem in your calibration data, it will be easily verified with the l-calibrated image viewer mode. Have you all checked how your calibrated data looks? Have you seen any odd results in star analysis or zero background values perhaps after 5) Normalization?

Are you all creating 32bits masters? Or 16bits?

Mabula

Posted by: @mabula-admin

Hi All, @dickvantatenhove, @jan-monsuur, @nobbybuddy, @morrienz, @vincent-mod, @tchylek, @absorbingphotons

In addition, if the issue is in fact caused by a problem in your calibration data, it will be easily verified with the l-calibrated image viewer mode. Have you all checked how your calibrated data looks? Have you seen any odd results in star analysis or zero background values perhaps after 5) Normalization?

Are you all creating 32bits masters? Or 16bits?

Mabula

Hi Mabula, thank you for taking the time to look into this issue. I am uploading my light and dark frames to upload.www.astropixelprocessor.com as we speak. In the meantime, here are screenshots of the following:

1. My first light frame in the "image" viewer mode:

2. My first light frame in the "l-calibrated" viewer mode:

3. My final stacked image in the "image" viewer mode (blank/black):

4. My final stacked image in the "l-calibrated" viewer mode (blank/black):

@mabula-admin Hi Mabula, you can now find all of my light frames and dark frames I'm using to get this issue under the folder absorbingphotons_blankimageissue. Thank you.

Posted by: @absorbingphotons

@mabula-admin Hi Mabula, you can now find all of my light frames and dark frames I'm using to get this issue under the folder absorbingphotons_blankimageissue. Thank you.

Thanks Brian @absorbingphotons,

Will have a look a.s.a.p. One more small question: in which APP version do you experience this? Have you tried both APP 1.082 and a.1.083-beta2 ?

Hi Brian @absorbingphotons,

Thank you very much for uploading your data so we can get to the bottom of this.

I started with checking your dark frames: they are not correct, I am afraid, and this fully explains your problem. So in your case, it definitely is a data capture issue with your capture software or otherwise your optical train must be leaking light somehow when you create the darks. As a result, data normalisation fails for your data and noise statistics can't be calculated leading to zero noise values and unlimited SNR... and then a black integration... (which I can prevent now and build in a warning for you to tell you that something is not right with your data 😉 )

All 5 darks that you uploaded are inconsistent. If I stretch them all with the same stretch settings they look very different and this is not what you expect. You expect all darks to have the same dark current average value. I auto stretched the first dark with the strongest DDP preset on the right, after which I disabled the auto setting in the preview filter on the right, which keeps the stretch exactly the same for the following darks:

All 5 are different where they should like identical. This in itself is a huge data issue caused by a problem in data capture by your software or light leakage in your optical train. Furthermore, the darks should have a grey color, but they are blue, purple... so something is not right in how these darks were created.

APP would integrate these 5 darks into a MasterDark and then the average dark current value of the MasterDark is much higher than the background level of your lights... so if you subtract the MasterDark from these lights, all data is lost in the sky background, it is clipped to zero and then normalization fails and you will end up with a black integration.

After Normalization we end up with the following info in the frame list panel:

The zero noise values should never occur, and this creates the black integration... it is the result of the data in especially the blue channel clipping to zero by the dark subtraction.

I will think about the best way to warn the user with data that has this problem, so you wil be notified that there is in fact a serious data problem.

Mabula

@mabula-admin Hi Mabula,

Thank you for pointing this out! I believe I know the reason this happened. I took my dark frames while my camera was still attached to my telescope with the dust cap on the front of the scope, which I usually do. But this time, it was already morning when I did that. Usually, when I do this and it worked in the past, it was still nighttime when I took the darks, so it never presented a problem. I guess there was some light leakage into the telescope this time from the pre-dawn light, otherwise my dark frames would have appeared all grey. In my FITS Viewer program, the auto-stretch appeared completely normal for my darks, but I guess it had not stretched them enough because I too see they are brighter than normal.

Long story short: may this be a word of warning to anyone taking dark frames to make sure you disconnect your camera from your telescope/imaging train and cap it so that this doesn't happen to anyone else. 😆 

Thanks for getting to the bottom of this Mabula, I appreciate it! A warning in APP would be great so others don't run into this same issue.

@mabula-admin Thanks Mabula. It would be great to have an error message/warning as early as possible in the APP phases that you can detect such a subtraction problem with darks that will cause a black integration, to save wasting time getting right through to the end and find a black integration. I reported the problem here twice at different times.

In the second occurrence of the issue for me, this year, I know for sure that my darks were bad. I found later that by mistake I had taken them with a completely wrong gain, and redoing them at the correct gain fixed the issue completely.

In the first occurrence though, they were older darks that had always worked in APP previously, but suddenly stooped working and gave me the black integration. However, whatever the underlying cause, retaking those older darks did fix the problem, so maybe that first time it happened to me (I think back in 2019 or 2020) was something to do with a change in the way APP handled things in a new version, but I can't be sure of that at all, and I wouldn't now know what APP versions I was using back then.

Cheers, Chris M

Posted by: @absorbingphotons

@mabula-admin Hi Mabula,

Thank you for pointing this out! I believe I know the reason this happened. I took my dark frames while my camera was still attached to my telescope with the dust cap on the front of the scope, which I usually do. But this time, it was already morning when I did that. Usually, when I do this and it worked in the past, it was still nighttime when I took the darks, so it never presented a problem. I guess there was some light leakage into the telescope this time from the pre-dawn light, otherwise my dark frames would have appeared all grey. In my FITS Viewer program, the auto-stretch appeared completely normal for my darks, but I guess it had not stretched them enough because I too see they are brighter than normal.

Long story short: may this be a word of warning to anyone taking dark frames to make sure you disconnect your camera from your telescope/imaging train and cap it so that this doesn't happen to anyone else. 😆 

Thanks for getting to the bottom of this Mabula, I appreciate it! A warning in APP would be great so others don't run into this same issue.

I have also had light leaks ruin darks that I tried to take in daylight, with the lens cap on my APM 140/970 refractor and my ZWO ASi294MC Pro camera, but I was able to immediately see the light leak in the stretched darks so it didn't go on to cause me later problems, and I just retook them later that night after dark. The light leak appeared to be coming in via the camera body somewhere, not via the OTA etc. I tested that by taking darks in daylight again with the camera removed from the OTA and the camera aperture blocked with a thick tight fitting black cap, and I still got the exact same light leaks showing on the sensor. 

Chris M

Posted by: @absorbingphotons

@mabula-admin Hi Mabula,

Thank you for pointing this out! I believe I know the reason this happened. I took my dark frames while my camera was still attached to my telescope with the dust cap on the front of the scope, which I usually do. But this time, it was already morning when I did that. Usually, when I do this and it worked in the past, it was still nighttime when I took the darks, so it never presented a problem. I guess there was some light leakage into the telescope this time from the pre-dawn light, otherwise my dark frames would have appeared all grey. In my FITS Viewer program, the auto-stretch appeared completely normal for my darks, but I guess it had not stretched them enough because I too see they are brighter than normal.

Long story short: may this be a word of warning to anyone taking dark frames to make sure you disconnect your camera from your telescope/imaging train and cap it so that this doesn't happen to anyone else. 😆 

Thanks for getting to the bottom of this Mabula, I appreciate it! A warning in APP would be great so others don't run into this same issue.

Hi Brian @absorbingphotons,

Glad that we have found a clear explanation for your problem 😉 and that there also is good explanation from your side that confirms that indeed a data capture problem was the source of all problems.

I am implementing a warning system for this right now and will make sure it is in the 1.083 stable release 😉

Posted by: @morrienz

I have also had light leaks ruin darks that I tried to take in daylight, with the lens cap on my APM 140/970 refractor and my ZWO ASi294MC Pro camera, but I was able to immediately see the light leak in the stretched darks so it didn't go on to cause me later problems, and I just retook them later that night after dark. The light leak appeared to be coming in via the camera body somewhere, not via the OTA etc. I tested that by taking darks in daylight again with the camera removed from the OTA and the camera aperture blocked with a thick tight fitting black cap, and I still got the exact same light leaks showing on the sensor. 

Chris M

Hi Chris @morrienz,

Thanks for sharing your experience in these matters 😉 Indeed, I suffered myself a lot a couple of years ago when I had light leakage in my optical train. I simply could not make good flats and darks in daylight because light was entering the system through a very expensive Takahashi adapter.... 🙄 I could only solve it by modifying my optical train with another adapter. Sometimes it can be very hard to find the cause...

Anyway, I will make a warning system that detects such a light leakage in darks and bias frames when they are subtracted from the lights. If too much pixels get clipped to zero, the system will start to warn 😉 and tell the user that he needs to check

• sensor gain and offset levels for lights and darks/bias
• and otherwise light leakage

Posted by: @mabula-admin

Hi All, @dickvantatenhove, @jan-monsuur, @nobbybuddy, @morrienz, @vincent-mod, @tchylek, @absorbingphotons

Let's solve this issue then!

I will need to see all data sets to understand what is happening. I have so many data myself and test data of other APP users  and I have never seen this problem appear myself, so at the moment i can only assume something fishy is happening in either

• APP or
• in the data itself (sensor gain/offset problems)
• or in the capture software that is being used...

Anyway, without seeing the data it will be hard to solve this and give you a good explanation.

Please upload your data if you have not done so already here:

https://upload.astropixelprocessor.com/

username and password : upload

Please let us know when uploaded and we will test a.s.a.p.

Mabula

Hi All, @dickvantatenhove, @jan-monsuur, @nobbybuddy, @morrienz, @vincent-mod, @tchylek,

Do I need to check your data as well to see if we can solve the issue completely and in all cases?

Mabula

Posted by: @mabula-admin

Posted by: @mabula-admin

Hi All, @dickvantatenhove, @jan-monsuur, @nobbybuddy, @morrienz, @vincent-mod, @tchylek, @absorbingphotons

Let's solve this issue then!

I will need to see all data sets to understand what is happening. I have so many data myself and test data of other APP users  and I have never seen this problem appear myself, so at the moment i can only assume something fishy is happening in either

• APP or
• in the data itself (sensor gain/offset problems)
• or in the capture software that is being used...

Anyway, without seeing the data it will be hard to solve this and give you a good explanation.

Please upload your data if you have not done so already here:

https://upload.astropixelprocessor.com/

username and password : upload

Please let us know when uploaded and we will test a.s.a.p.

Mabula

Hi All, @dickvantatenhove, @jan-monsuur, @nobbybuddy, @morrienz, @vincent-mod, @tchylek,

Do I need to check your data as well to see if we can solve the issue completely and in all cases?

Mabula

Hi Mabula. It was quite a while ago this happened to me, twice, and I don't think I kept (or would now know which darks they were even if I did keep them) the two sets of darks that caused the problem for me, so unfortunately I'm not able to send in any useful data now. I wish I'd kept and knew which were the first ones that caused the problem as they were darks that I am fairly sure had previously been working fine in APP but then just out of the blue one day started giving that black final integration problem. At the time this discussion wasn't as well developed and didn't have as many reports of the issue and I just reported it here, took new darks, which was the advice to me then from Vincent, and those worked, so we left it at that.

The warning will be very useful.

Cheers and Thanks, Chris M

Posted by: @morrienz

Hi Mabula. It was quite a while ago this happened to me, twice, and I don't think I kept (or would now know which darks they were even if I did keep them) the two sets of darks that caused the problem for me, so unfortunately I'm not able to send in any useful data now. I wish I'd kept and knew which were the first ones that caused the problem as they were darks that I am fairly sure had previously been working fine in APP but then just out of the blue one day started giving that black final integration problem. At the time this discussion wasn't as well developed and didn't have as many reports of the issue and I just reported it here, took new darks, which was the advice to me then from Vincent, and those worked, so we left it at that.

The warning will be very useful.

Cheers and Thanks, Chris M

Thanks Chris @morrienz,

Probably your capture software has made an adjustment/correction for the sensor offset. So the metadata was reporting it as the same, but the data was actually shot with a different offset... this happens now and then. So my advise is always:

• don't shoot calibration data and lights with different software packages
• don't shoot calibration data and light with different versions of the same software package

I know many make dark libraries and that this can be usefull. But please be aware that if there is an software update of your capture software, things might be different with regard to the sensor gain and offset which can upset your routine... I have seen this problem many years with different capture packages unfortunately.

Mabula

Hi All, @dickvantatenhove, @jan-monsuur, @nobbybuddy, @morrienz, @vincent-mod, @tchylek, @absorbingphotons,

For APP 1.083 stable, which will soon be there, I have implemented the warning system now and I have completed eliminated the cause that was creating black integrations. This will no longer happen.

A black integration would occur if you would have

• multi-band/color data
• background neutralization enabled in 5)NORMALIZE
• an adaptive pedestal was needed because of a data problem where the background values of the reference frame without the pedestal would be negative...

The warning system is implemented in both the calibration engine as well as the normalization engine...

2) Calibrate, warning system shown when too many pixels clip to zero despite using the adaptive pedestal, indicating a serious data problem:

5) Normalize, warning system kicks in when background values of the reference frame are negative without the needed pedestal, causing Background Neutralization to be disabled because it can not work properly with data that has negative background values corrected for the artificial pedestal.

So this will always warn the user when a data problem is detected and the resulting data can still be processed without getting a black integration ;-), but the user is now aware that something is not right with the data and by fixing the issue, he/she will get better results for sure!

The problems that were causing black integrations were always one of the following 3:

1. data is underexposed
2. bias/MasterBias/dark/MasterDark frame(s) are not compatible with lights due to having a different sensor offset/pedestal
3. a light leakage in the optical train so the bias/dark frames are not good

Thank you all for reporting the issue.

Mabula

Thanks Mabula for solving our darkframe problems!. Looking forward to the new version! 

Dick

Thanks very much for putting this new warning in Mabula.

Chris M