Mosaic normalization issue (NAN or normalization failed)

Mmm, I think it probably is a data issue. What happens if you don't use any calibration frames on a panel (just test a panel that fails)?

Thank you @vincent-mod for your answer.

I've made a few other tests.

I've taken the same 9 panels.

- With individual lights not calibrated, it works (auto ref image is n°6)

- With individual lights calibrated, it works (auto ref image is n°6)

- With integrated lights (5x3min) calibrated, it doesn't work (auto ref image is n°4)

- With integrated lights (5x3min) calibrated, it works (manual ref image is n°6, that previously failed to normalize)

So, it seems that it's not a calibration issue.

The normalization of integrated panels seems to fail depending on the reference image chosen.

My integration settings here are : median, winsorized 6-3 with local normalization rejection checked, bayerdrizzle.

Here are the screenshots of the trials with integrated panels.

So maybe it's an integration issue, but I don't understand why and where it the fault...

One more thing, when I checked neutralize background in the histogram adjustment, the picture becomes white. Maybe there is a relation here, or not.

Mmm, the all white result might signal an issue in the background maybe. Is your offset set correctly for instance? Oh and one other possibility, are you using your calibration file twice? Like on the single frames and then again with the integrated files? If so, APP will substract twice and that is also an issue.

I don't know what the offset setting is as I use an ASIAIR PRO (it's a factory setting for the offset if I'm right).

And no, I haven't use the calibration twice. 

Maybe something with the masterbias, but it doesn't explain why some panels are correctly normalized one time, and the other time not...

Are they anywhere temp files that APP is using on a Mac OS ?

I mean, I was running two versions of APP on my Mac, maybe there is a conflict somewhere.

Running 2 APP versions and using the same working directory I never tested, but I can imagine that might not be the best thing to do indeed.

If things are not improving, I can have a look at the data if you want.

I made some other tests.

I've tried by creating an other masterbias with different files, same issue.

The question is why does normalization works with calibrated single frames but not with 5 stacked frames.

When I unchecked neutralize background at the 5 stage, the normalization worked for every stacked files, but the back ground of the mosaic doesn't seem to be uniform (it's darker on some frames).

@Vincent-mod, what data do you need to make further investigations ?

David

Those are good questions indeed. So, let me ask for the data to investigate. 🙂

Go to https://upload.astropixelprocessor.com and for the username and password, use: upload

Create a directory named “dav78-mosaicBackgroundIssue” and upload in there. Just upload like 10 lights per panel and your master calibration files. Thanks!

I'm sending the files right now : master calibration files, and 9 panels (5 frames each, I haven't more), the 9 mentioned above.

Thanx 😉

David

Excellent, I'll download them today and will have a look later.

So 1 thing I notice about the data is that the stars look a bit weird, square, meaning (I think) you're undersampling quite a bit. Having issues with beta 2 on analyzing them (which is a known problem, though I only now run into that). So, let me download 1.082 first as well. 🙂 I'm doing the first tests without bias, simply because the exposure time is 0.01s and I know that some sensors don't like that (they don't behave linear in that time). I'd do a bias with an exposure of something like 0.5s.

And another thing, the master-calibration files were made in 16-bit, which was the norm till 1.082 indeed, but we noticed that in some cases this caused an issue. Nowadays we create them in 32-bit (in 1.082 you have to switch that on, it's standard in a version above). Maybe you can do that as well, remake the masters in 32-bit, skip the bias and test.

Ok, so using the above setup (1.082, no use of the bias, 1600 stars to analyze, advanced normalization mode, mosaic registration, LNC3 1 it, 10% MBB) I get the following result;

Switching on background normalization on the right-hand panel doesn't influence it (tiny bit but not in a degrading way at all).

Stretching the mosaic to the max, adding saturation to the max and making some Light Pollution Corrections;

(Above is the LPC box placement, I lowered the stretch again in this screenshot)

And the end-result;

So my guess is that the bias was causing an issue. My advice is to either only use dark-flats, darks, flats and a BPM (Bad Pixel Map) or when a bias is used, take a higher exposure length per sub of say 0.5s.

Hi @vincent-mod, thank you for your feedback.

I've always used masters in 16-bit with this camera and never saw any issue (single panels or mosaics) and the masterbias was made with the shortest exposure time (has it was recommended, but things may have changed with newest cameras).

My other masterbias is 0,1 sec, and the issue is the same.

I will try with masterbias at 0,5 sec and 32-bit masters. For dark flats, how does it improve the calibration ? The masterbias is there for reading noise, right ? As I never make my flats with the same exposure time, it will be a huge work to make a master dark flat library ^^

For the picture made above, have used single frames or stacked frames ?

For the normalization of the mosaic, the background normalization box was checked ?

And yes, I'm under sampling, but not that much, my camera is ASI533 with 3,76 microns pixels, and the lens is a Samyang 135mm f2. Maybe not enough frame for an efficient bayerdrizzle...

For the above I used the single frames, more because it's just a few per panel. The only difference was that I excluded the bias. All settings were standard, I didn't change anything apart from setting it up for a mosaic.

For drizzling to work, you indeed need more data, the more the better really. But it would work very nicely I think.

@vincent-mod Ok, thanx.

Single frames mosaic works perfectly with full calibration (flats, darks and bias).

So it works too without masterbias considering your picture.

Maybe it would be usefull to try to reproduce the issue with stacked frames and full calibration to analyse what APP is saying (if you have the time of course).

What does NAN means ?

It tried today with stacked frames calibrated with masterflat and masterdark (without masterbias). It worked (but I don't know how it can affect the resulting picture).

The question remain the same : why does normalization fails with stacked calibrated frames and works with single calibrated frames (with same masters)? And why, in my case, the success of the normalization can depend on the reference image chosen?

Considering our results, the masterbias seams to be part of the issue.

In the mean time I will try : 32-bit masters, a new masterbias with 0,5 sec exposures, and masterdarkflat.

David 

Sure, I can test it with the panels as well, please allow for a bit of time as there is more data for me to analyze. 🙂 I think there isn't a huge issue here, maybe first try out the new bias or even don't use a bias but only darks and a bad pixel map, that may work better for your sensor anyway.

 NaN means "Not a Number", so it couldn't produce a result for some reason.

I'm back with some more tests.

@vincent-mod, as you advised it, I have created a masterdarkflat instead of a masterbias, and 32-bit masters (all done with 1-083beta2)

Unfortunately the issue is still there : for some stacked calibrated frames the normalization failed, but sometimes it succeeds, depending on the reference image chosen. And even when the normalization succeeds, the result isn't great. 

So the masterbias wasn't the cause, but maybe the masterflat. I don't know.

One more thing about star detection with the beta2. Star detection fails most of the time, but in my case, AAP algorithm was chosen (by default). But, as my pictures where taken with duo band filter, I've chosen the HaOiii color algorithm, and star detection worked! That's strange because the algorithm shouldn't influence the star detection, right ?

I don't think it should matter, but this can be something that has to do with the issue we have in beta2, so I'm not sure on that one. For the other problem I'm not sure how to deal with that, so I'll ask Mabula to also have a closer look. Does the automatic reference image work? So when you don't change it? Also, I never actually asked why you want to manually pick a different frame, it may be that APP already selects the optimum for that and then it does work?

Hi Dav @dav78,

I am looking at your data. I think the issue is in your bias and darks. They don't seem to be fully compatible with your lights... Did you create them with the same software and the same software version as your lights?

Can you upload some of the single bias, darks and flats that you used to create the masters?

Reason for me to suspect this is that allthough you have exposed for 180seconds with I think broadband illumination of the sensor, many pixels are clipped to the black point after bias/dark subtraction. This should not occur on data like this. The adaptive pedestal kicks in to prevent the black clipping, but still the sky background even with the pedestal is very low...

Then depending on the reference frame for the mosaic, some of the panels will be problematic because the data normalization causes too many pixels to become 0... ending in failed normalization on NaN infinite SNR results, because noise can't be calculated properly.

Mabula

Hi @dav78,

With all your calibration data, so bias, darks and flats i get no problems to create the mosaic in APP 1.082 with automatic reference by APP. Reference was panel 10 for the mosaic.

With Panel 18, like in your screenshots where it goes wrong, I also don't get any problems with 5) regular normalization mode

With 5) advanced normalization mode I get the problem however:

So this means the following, advanced normalization only uses the background and dispersion in the regions where the mosaic panels overlap with each other, instead of calculating this for the whole individual mosaic panel.

Clearly, with these background values in de overlap areas, when the frames are normalized in the normalization engine, the noise and SNR calculations fail on some of them, because now too many pixels get clipped the black point... which is indicative of a data calibration issue still.

I guess we really need to see the individual bias, dark and flats to get any explanation for the issue. Can you upload them?

Mabula