SNR - NAN and Quality Infinite

Hi Jan,

Yes, an upload of say 10 lights and your master-calibration frames would be very appreciated. We can then have a look if something obvious is going wrong here. When you say you always did the same, is this also true for the camera used etc? New sensors may behave differently which may cause issues if not handled properly for specifically that sensor. What gain and offset are you using to take the calibration frames and lights?

Go to https://upload.astropixelprocessor.com and for the username and password, use: upload

Create a directory named “janwalschap-blackIntegration” and upload in there. Thank you!

Hi Vincent

upload of say 10 lights and your master-calibration frames

I Uploaded all frames (already partly fine tuned after analyse stars step on starshape and quality)

Subdir Darks (251 files) + subdir MD and BPM (2 files) which are created from the darks

(note I applied these darks also to another panel of lights with succes)

I added 1 subdir with the integration file which is logically black, given the NAN-0 and quality = infinite parameters.

When you say you always did the same, is this also true for the camera used etc? New sensors may behave differently which may cause issues if not handled properly for specifically that sensor.

=> Guilty, indeed. It's the same brand and model, fixed setup (Vaonis - Stellina) but from someone else.

So, it's the same sensor but different serialnumber if you want ... 

Equipment setup:

Brand Vaonis /Model Stellina /Type Apo Refractor ED-Doublet
Opening 80mm/ Focal 400 mm / Speed F5 /
FoV 1° x 0.7° °
Brand Sony CMOS / Type IMX178 / Exmor Starvis / resolution 6.44 Mpixel
size 3096 x 2080 pixels/ size pixel 2.4 µm/ Pixeldepth 14 bit
Filters CLS (CLS cutoff 450nm-530nm + under 650nm)

What gain and offset are you using to take the calibration frames and lights? For all (We cannot change = fixed) :

GAIN = 200 / Sensor gain
OFFSET = 200 / Camera brightness parameter

All exposures = 10sec

Ah Stellina! That is good to know, that system does have some quirks with properly reporting things sometimes. I'm currently moving, so will get back to your data asap, likely within a day or 2 if you don't mind. Thanks for uploading.

Hi Jan @janwalschap,

I have checked your darks and they are simply not darks, all 483 darks are lights and look like this:

So it not strange that your data processing is not working. You are subtracting a MasterLight from the lights. Please have a good look at your darks 😉

Mabula

@mabula-admin, Hi Mabula, If that would be the case, it could have been, it would be a huge mistake of course. But I just checked and the darks are in the subdir Darks (251), there are (867) lights. Furthermore I placed a subdir with only 1 MD and 1BPM, the result of the 251 darks. I just added an additional an excel file with the dump of the results (listing) according to all parameters after normalisation, so you can read them out very fast.

Concerning Stellina, -I have a lot of other equipment and I'm no newbie, I helped many others with several practical issues, as I posed also here on the site with some python scripts- I had never had any big issues (yet) 😉 

Looking forward for any updates, I wait till you're settled of course.

Dear Jan, @janwalschap

My bad... i made a mistake, I loaded the lights as darks i see now... Sorry 🙄 

Now i took a look at the darks in the darks folder and they don't look okay either... I have no clue what the Stellina is doing here, but a dark of a color camera should be gray and not colored like this...

There are White Balance multipliers in the Fits header indicating that white balance is applied to the darks? That should never happen. White Balance is only to be applied on lights after data calibration for a proper workflow. I would suggest you contact Stellina support about this to understand what the Stellina is doing and why darks would have white balance applied?

Mabula

@mabula-admin, I understand, but just some feedback: These darks are not mine, but I developed panel 4 with the same set without issues. This suggests that the darks must be ok, despite the presence of non-zero white balance parameters?

Moreover, I just checked my own darks to be sure and I Uploaded just 15 of them in a separate folder: "somofmyown_Stellina_darks"

They also include the same white balance 77/64 settings in the fits header. ....

I presume this is a default calibration added on ALL Stellina devices as statistical median for the Sony Exmor 178X sensor,

implemented by Vaonis. These darks are shot in the same way as lights, but only with a covered telescope of course, so the lights contain the same white balance parameters (just checked)

Last full year of lot of APP processing did not produce this effect (NAN-0 Q- infinite) once ...

Also, the results from last year were very realistic to what one should expect ...

What could be the effect of the white balance?

I presume the error must have been produced by something else?

Till Analyse stars, everything goes well, but after normalisation is get this flaw.

Thx for digging into this data.

@mabula-admin, I got feedback from Vaonis. The settings of white balance are correct for the lights. this is a general calibration for the CLS which is standard in the Stellinia astrograph. BUT as mentioned the darks should register the ampglow, bad pixels, hot pixels .... only. Calibration with flat, there it is correct to use the White balance again OR we should use zero at lights and flats. That's all I can give you right now.

So to be correct, we should only apply the BPM and not he darks since they would have an impact on the colour balance

Is my reasoning correct?

Posted by: @janwalschap

@mabula-admin, I understand, but just some feedback: These darks are not mine, but I developed panel 4 with the same set without issues. This suggests that the darks must be ok, despite the presence of non-zero white balance parameters?

Moreover, I just checked my own darks to be sure and I Uploaded just 15 of them in a separate folder: "somofmyown_Stellina_darks"

They also include the same white balance 77/64 settings in the fits header. ....

I presume this is a default calibration added on ALL Stellina devices as statistical median for the Sony Exmor 178X sensor,

implemented by Vaonis. These darks are shot in the same way as lights, but only with a covered telescope of course, so the lights contain the same white balance parameters (just checked)

Last full year of lot of APP processing did not produce this effect (NAN-0 Q- infinite) once ...

Also, the results from last year were very realistic to what one should expect ...

What could be the effect of the white balance?

I presume the error must have been produced by something else?

Till Analyse stars, everything goes well, but after normalisation is get this flaw.

Thx for digging into this data.

Hi Jan @janwalschap,

Did you perhaps get an update on the Stellina after which these problems started?

Your own darks are not correct either, they are not gray. White Balance should never be applied in the data calibration fase. It should be applied after complete data calibration on the pixels as detected by the camera's sensor.

Maybe you can ask Stellina the following: Should APP correct both the lights and the darks for the (assumed now) applied white balance to be able to calibrate them properly after which it should again be applied to the calibrated data?

Now looking at your own darks and the other darks that you received from someone else... They are not compatible clearly. The median values of the masterdarks that you borrowed are quite a bit higher than yours allthough the fits header states the same sensor offset of 200 is used. That is indicative of a problem or a change in how the Stellina works I would think versus your own Stellina version ?

But now, If i calibrate your lights with your own darks, the problem is actually...  not there 🙂 , allthough I can't understand what the Stellina White Balance is doing in the darks.

So perhaps the biggest question now is, why where you using someone else's darks? And not the darks of your own Stellina's sensor?

Mabula

@mabula-admin, Hi Mabula, As I explained, we must take the darks, while shooting our object and cover the scope. This periode is for capturing darks. Now knowing that the CLS filter is always there, Vaonis sets default calibration of WB_B = 77 / White balance (blue) WB_R = 64 / White balance (red) in ALL their Vaonis Stellina devices to correct their lights. This means that also our darks contain that specific WB, which is as you mentioned NOT correct. (Fact is that I Used and captured them for the last year as is and used it in my workflow without any problems). BUT this might be the reason that I found (subjectively) that I had a little bit shortage on the red.

Concerning the gain of 200, this is a setting of the Sony Exmor X178 embedded in Stellina, and default value. As you mentioned, one might have alle the same models but every chip is different. (Also important, we both used Stellina V.27  at the moment of comparison in your upload directories). I'm helping others and guiding Stellina users in APP. Many have come over yet ;-). Therefor I'm now making a mosaic and I'm documenting step by step their data. I did this for several other from this person in this particular case. For all datasets, no issue. But now for Panel 3 I have lights + darks that gave a problem! (For Panel 1, 2, 3B and 4 no issues !). Panel 3B was made as parallel when I mentioned that we have issues with Panel 3. Now I have also same issues with Panel C (center). I want to find the reason of this issue. 

so, I'm NOT using or mixing darks: I use the other persons darks AND lights. I uploaded in a separate directory only a sample of 10 darks of mine, so you could compare 2 different Stellina sets. In fact, Stellina is a wonderful thing in that way that it produces standardized output, with the disadvantage that the tend to use strange conventions sometimes, I agree 😉

What confuses me is that I needed the MD + BMP created out of the darks set created from these darks for Panel 3 also for Panel 4, and there it worked like a charm .... (Understand who can). (We create libraries for darks or darksets for a certain temperature range) >=> So, we could conclude that the problem is not in the darks, but test of you and I prove otherwise. It must be a combination of that BMP & MD with that specific Panel 3. Today I even combined in 2 sessions Panel 3 and Panel 3B (both same area, different moment, same setup) and got also the NAN. Therefore, I presumed the errors must be in Panel 3 FITS?

Another thing: I tried to integrate the complete set but on my other APP installation (other PC) => same NAN issue. BUT if I try to process the Panel 3 fits for integration and use only the BPM, it works and I get an integration, bad pixels are gone, but I get an integrating with massive amp glow on top and bottom of my sensor (which is in our case logical). Our sensor is not cooled, and electronics are all around the CMOS hence heating up the sensor. So, we definitely need the darks.

So I'm a bit surprised with your result 😉 But it's obvious we can't use my darks for the lights of another scope, even if it is the same brand/model/Cmos ....

Do you see where the problem might be?

I want to dig in and find the core of this problem.

1 question in between concerning this issue: IF we take flats and the CLS is in between (can't be removed), it is in this case legitimate to count in this same white balance I presume?

Thx for your Help!

@mabula-admin, Hi Mabula, I got feedback from Vaonis. As told, we must take the darks, while shooting our object and cover the scope. This periode is for capturing darks. Now knowing that the CLS filter is always there, Vaonis sets default calibration of WB_B = 77 / White balance (blue) WB_R = 64 / White balance (red) in ALL their Vaonis Stellina devices to correct their lights. This means that also our darks contain that specific WB. This is correct, because if we didn't do this, we would subtract wrong data. The WB is a calibration for the sensor+CLS, while in darks, we don't have light (theoretically) so we might make the mistake to think that we may eliminate the WB. 

In the mean time, I was able to get new darks from the other person and they work! Combining both darks to get better SNR => fail, same NAN/intfinite error. Is there a way how I can find the faulty darks?

Last but not least, by upgrading from 1.082 => 1.083 beta 2, I have a hudge whitecast on my individual panels. I can not find any settings that are different at first sight. Do you have a clue? If not I will be forced to downgrade to 1.082, reprocess them and find out if this artefact is from my darks or from different settings.

Dear Jan @janwalschap,

Okay, I understand why Stellina is doing this and that as long WB is fixed for the device it will work with data calibration. (Still it would be much better if WB was in the header but only applied to lights after data calibration. Anyway, I can not fix that and will leave it to Stellina.)

In the mean time, I was able to get new darks from the other person and they work! Combining both darks to get better SNR => fail, same NAN/intfinite error. Is there a way how I can find the faulty darks?

Well, you simply should never mix up darks of similar individual sensors, that is not right and I would never recommend that. The darks do not only have gaussian noise, they have other signals like amp-glow, and fixed pattern noise (bad pixels) and these signals will be different for different individual sensors. Yes, the SNR will be better if you combine darks of different sensors, but the capture and correction of the dark current of the sensor, which is the purpose of dark frame subtraction is just not happening correctly in this way. So you simply should not do this.

Last but not least, by upgrading from 1.082 => 1.083 beta 2, I have a hudge whitecast on my individual panels. I can not find any settings that are different at first sight. Do you have a clue? If not I will be forced to downgrade to 1.082, reprocess them and find out if this artefact is from my darks or from different settings.

Please check what happens if you only use darks on lights which were shot with the same Stellina, if you have the problem still then, please show a screenshot or share the data with us so we can have a look 😉

Mabula

@mabula-admin, Hi Mabula, of course! I expressed myself probably not fully: when I say that I combined Darks, it were the darks set (a) @ temp.range X of telescope 1 + recently new received darks (b) @ temp range X of telescope 1. NEVER mix darks from different telescopes nor different temperature set. (bad pixels are different, amp glow diffferent etc => wrong correction!)

When I use set b => ok, set a => NAN/infinite error. When I use set a+b => NAN/infinite.

So my question was how can I discover/detect the bad darks (on what parameters). If so I want to attain an as high as possible number of Dark frames to obtain  a max SNR.

Indeed the WB  is the same for ALL frames, including darks and thus can be regarded as ZERO or at least the same baseline, hence resulting in a correct substruction of darks from the lights. (This is what I understood from your explanation ad seems logical. In the past (1.082) I got always correct results ...). Putting them to zero would be logical but the implement the default impact of the (permanent) CLS filter. Addressing this issue is indeed only possible by Vaonis. They won't do that of course since they use that adapted raw info to autostack their jpegs.

Concerning the second issue, I will reprocess and check as requested and come back as soon as possible with the results and findings.