Mosaic normalizatio...
 
Share:
Notifications
Clear all

Mosaic normalization issue (NAN or normalization failed)  

Page 1 / 2
  RSS

(@dav78)
Red Giant Customer
Joined: 3 years ago
Posts: 67
October 12, 2021 21:22  

Hi,

 

I'm working on a mosaic (35 tiles for now) and at normalization step, for some tiles normalization failed, and for others it's NAN - 0 (I don't know what it means).

All the tiles are processed with same masters, and are taken with same settings.

I've made a screen shot of the integration result and tiles informations.

 

APP1

 

My settings are basically the same I usually use for processing mosaic (2500-3000 stars, dynamic distortion correction, advanced mode, LNC MBB...).

I tried with less tiles (9), but some tiles failed to normalize too. If I try again, the failed tiles may change. And if I change the reference image, and choose one that was previously failed at normalization step, all files are normalize (with those 9 tiles at least).

APP2

 

For more than two years using APP, I never saw this issue.

What have changed since is my camera (ASI533), and my computer (iMac 27 with Big Sur OS, before I was with MacBook Air and Mojave OS).

APP 1.082 and 1.083beta, same results (worth with 1.083beta2 as sometime star analysis fails with most of the images).

 

Any idea of what's happening here and how to solve it ?

 

Thanx !

David

 

 


ReplyQuote
(@vincent-mod)
Universe Admin
Joined: 4 years ago
Posts: 3961
October 13, 2021 09:30  

Mmm, I think it probably is a data issue. What happens if you don't use any calibration frames on a panel (just test a panel that fails)?


ReplyQuote
(@dav78)
Red Giant Customer
Joined: 3 years ago
Posts: 67
October 13, 2021 11:44  

Thank you @vincent-mod for your answer.

 

I've made a few other tests.

I've taken the same 9 panels.

- With individual lights not calibrated, it works (auto ref image is n°6)

- With individual lights calibrated, it works (auto ref image is n°6)

- With integrated lights (5x3min) calibrated, it doesn't work (auto ref image is n°4)

- With integrated lights (5x3min) calibrated, it works (manual ref image is n°6, that previously failed to normalize)

 

So, it seems that it's not a calibration issue.

The normalization of integrated panels seems to fail depending on the reference image chosen.

My integration settings here are : median, winsorized 6-3 with local normalization rejection checked, bayerdrizzle.

 

Here are the screenshots of the trials with integrated panels.

So maybe it's an integration issue, but I don't understand why and where it the fault...

One more thing, when I checked neutralize background in the histogram adjustment, the picture becomes white. Maybe there is a relation here, or not.

 

Capture d’écran 2021 10 13 à 11.17
Capture d’écran 2021 10 13 à 11.02

ReplyQuote
(@vincent-mod)
Universe Admin
Joined: 4 years ago
Posts: 3961
October 13, 2021 15:23  

Mmm, the all white result might signal an issue in the background maybe. Is your offset set correctly for instance? Oh and one other possibility, are you using your calibration file twice? Like on the single frames and then again with the integrated files? If so, APP will substract twice and that is also an issue.


ReplyQuote
(@dav78)
Red Giant Customer
Joined: 3 years ago
Posts: 67
October 14, 2021 21:52  

I don't know what the offset setting is as I use an ASIAIR PRO (it's a factory setting for the offset if I'm right).

And no, I haven't use the calibration twice. 

Maybe something with the masterbias, but it doesn't explain why some panels are correctly normalized one time, and the other time not...

 

Are they anywhere temp files that APP is using on a Mac OS ?

I mean, I was running two versions of APP on my Mac, maybe there is a conflict somewhere.

 


ReplyQuote
(@vincent-mod)
Universe Admin
Joined: 4 years ago
Posts: 3961
October 15, 2021 14:40  

Running 2 APP versions and using the same working directory I never tested, but I can imagine that might not be the best thing to do indeed.

If things are not improving, I can have a look at the data if you want.


ReplyQuote
(@dav78)
Red Giant Customer
Joined: 3 years ago
Posts: 67
October 20, 2021 17:49  

I made some other tests.

I've tried by creating an other masterbias with different files, same issue.

The question is why does normalization works with calibrated single frames but not with 5 stacked frames.

When I unchecked neutralize background at the 5 stage, the normalization worked for every stacked files, but the back ground of the mosaic doesn't seem to be uniform (it's darker on some frames).

@Vincent-mod, what data do you need to make further investigations ?

David

 


ReplyQuote
(@vincent-mod)
Universe Admin
Joined: 4 years ago
Posts: 3961
October 20, 2021 18:34  

Those are good questions indeed. So, let me ask for the data to investigate. 🙂

Go to https://upload.astropixelprocessor.com and for the username and password, use: upload

Create a directory named “dav78-mosaicBackgroundIssue” and upload in there. Just upload like 10 lights per panel and your master calibration files. Thanks!


ReplyQuote
(@dav78)
Red Giant Customer
Joined: 3 years ago
Posts: 67
October 20, 2021 21:02  

I'm sending the files right now : master calibration files, and 9 panels (5 frames each, I haven't more), the 9 mentioned above.

Thanx 😉

 

David


ReplyQuote
(@vincent-mod)
Universe Admin
Joined: 4 years ago
Posts: 3961
October 21, 2021 11:18  

Excellent, I'll download them today and will have a look later.


ReplyQuote
(@vincent-mod)
Universe Admin
Joined: 4 years ago
Posts: 3961
October 21, 2021 11:51  

So 1 thing I notice about the data is that the stars look a bit weird, square, meaning (I think) you're undersampling quite a bit. Having issues with beta 2 on analyzing them (which is a known problem, though I only now run into that). So, let me download 1.082 first as well. 🙂 I'm doing the first tests without bias, simply because the exposure time is 0.01s and I know that some sensors don't like that (they don't behave linear in that time). I'd do a bias with an exposure of something like 0.5s.

And another thing, the master-calibration files were made in 16-bit, which was the norm till 1.082 indeed, but we noticed that in some cases this caused an issue. Nowadays we create them in 32-bit (in 1.082 you have to switch that on, it's standard in a version above). Maybe you can do that as well, remake the masters in 32-bit, skip the bias and test.

 


ReplyQuote
(@vincent-mod)
Universe Admin
Joined: 4 years ago
Posts: 3961
October 21, 2021 13:21  

Ok, so using the above setup (1.082, no use of the bias, 1600 stars to analyze, advanced normalization mode, mosaic registration, LNC3 1 it, 10% MBB) I get the following result;

MosaicResult

Switching on background normalization on the right-hand panel doesn't influence it (tiny bit but not in a degrading way at all).

Stretching the mosaic to the max, adding saturation to the max and making some Light Pollution Corrections;

MosaicLPC

(Above is the LPC box placement, I lowered the stretch again in this screenshot)

And the end-result;

MosaicEndResult

So my guess is that the bias was causing an issue. My advice is to either only use dark-flats, darks, flats and a BPM (Bad Pixel Map) or when a bias is used, take a higher exposure length per sub of say 0.5s.


ReplyQuote
(@dav78)
Red Giant Customer
Joined: 3 years ago
Posts: 67
October 23, 2021 13:46  

Hi @vincent-mod, thank you for your feedback.

I've always used masters in 16-bit with this camera and never saw any issue (single panels or mosaics) and the masterbias was made with the shortest exposure time (has it was recommended, but things may have changed with newest cameras).

My other masterbias is 0,1 sec, and the issue is the same.

I will try with masterbias at 0,5 sec and 32-bit masters. For dark flats, how does it improve the calibration ? The masterbias is there for reading noise, right ? As I never make my flats with the same exposure time, it will be a huge work to make a master dark flat library ^^

For the picture made above, have used single frames or stacked frames ?

For the normalization of the mosaic, the background normalization box was checked ?

 

And yes, I'm under sampling, but not that much, my camera is ASI533 with 3,76 microns pixels, and the lens is a Samyang 135mm f2. Maybe not enough frame for an efficient bayerdrizzle...

 

 

 


ReplyQuote
(@vincent-mod)
Universe Admin
Joined: 4 years ago
Posts: 3961
October 24, 2021 13:55  

For the above I used the single frames, more because it's just a few per panel. The only difference was that I excluded the bias. All settings were standard, I didn't change anything apart from setting it up for a mosaic.

For drizzling to work, you indeed need more data, the more the better really. But it would work very nicely I think.


ReplyQuote
(@dav78)
Red Giant Customer
Joined: 3 years ago
Posts: 67
October 24, 2021 23:52  

@vincent-mod Ok, thanx.

Single frames mosaic works perfectly with full calibration (flats, darks and bias).

So it works too without masterbias considering your picture.

Maybe it would be usefull to try to reproduce the issue with stacked frames and full calibration to analyse what APP is saying (if you have the time of course).

What does NAN means ?

 

It tried today with stacked frames calibrated with masterflat and masterdark (without masterbias). It worked (but I don't know how it can affect the resulting picture).

 

The question remain the same : why does normalization fails with stacked calibrated frames and works with single calibrated frames (with same masters)? And why, in my case, the success of the normalization can depend on the reference image chosen?

Considering our results, the masterbias seams to be part of the issue.

 

In the mean time I will try : 32-bit masters, a new masterbias with 0,5 sec exposures, and masterdarkflat.

 

David 

 

 

 

 

 


ReplyQuote
(@vincent-mod)
Universe Admin
Joined: 4 years ago
Posts: 3961
October 26, 2021 13:22  

Sure, I can test it with the panels as well, please allow for a bit of time as there is more data for me to analyze. 🙂 I think there isn't a huge issue here, maybe first try out the new bias or even don't use a bias but only darks and a bad pixel map, that may work better for your sensor anyway.

 NaN means "Not a Number", so it couldn't produce a result for some reason.


ReplyQuote
(@dav78)
Red Giant Customer
Joined: 3 years ago
Posts: 67
November 2, 2021 00:36  

I'm back with some more tests.

@vincent-mod, as you advised it, I have created a masterdarkflat instead of a masterbias, and 32-bit masters (all done with 1-083beta2)

Unfortunately the issue is still there : for some stacked calibrated frames the normalization failed, but sometimes it succeeds, depending on the reference image chosen. And even when the normalization succeeds, the result isn't great. 

So the masterbias wasn't the cause, but maybe the masterflat. I don't know.

 

One more thing about star detection with the beta2. Star detection fails most of the time, but in my case, AAP algorithm was chosen (by default). But, as my pictures where taken with duo band filter, I've chosen the HaOiii color algorithm, and star detection worked! That's strange because the algorithm shouldn't influence the star detection, right ?

 

Capture d’écran 2021 11 01 à 22.11
Capture d’écran 2021 11 01 à 21.30

 

 


ReplyQuote
(@vincent-mod)
Universe Admin
Joined: 4 years ago
Posts: 3961
November 2, 2021 13:59  

I don't think it should matter, but this can be something that has to do with the issue we have in beta2, so I'm not sure on that one. For the other problem I'm not sure how to deal with that, so I'll ask Mabula to also have a closer look. Does the automatic reference image work? So when you don't change it? Also, I never actually asked why you want to manually pick a different frame, it may be that APP already selects the optimum for that and then it does work?


Dav78 liked
ReplyQuote
(@mabula-admin)
Universe Admin
Joined: 5 years ago
Posts: 2747
November 2, 2021 15:14  

Hi Dav @dav78,

I am looking at your data. I think the issue is in your bias and darks. They don't seem to be fully compatible with your lights... Did you create them with the same software and the same software version as your lights?

Can you upload some of the single bias, darks and flats that you used to create the masters?

Reason for me to suspect this is that allthough you have exposed for 180seconds with I think broadband illumination of the sensor, many pixels are clipped to the black point after bias/dark subtraction. This should not occur on data like this. The adaptive pedestal kicks in to prevent the black clipping, but still the sky background even with the pedestal is very low...

Then depending on the reference frame for the mosaic, some of the panels will be problematic because the data normalization causes too many pixels to become 0... ending in failed normalization on NaN infinite SNR results, because noise can't be calculated properly.

Mabula

 


ReplyQuote
(@mabula-admin)
Universe Admin
Joined: 5 years ago
Posts: 2747
November 2, 2021 15:49  

Hi @dav78,

With all your calibration data, so bias, darks and flats i get no problems to create the mosaic in APP 1.082 with automatic reference by APP. Reference was panel 10 for the mosaic.

With Panel 18, like in your screenshots where it goes wrong, I also don't get any problems with 5) regular normalization mode

Mosaic ref panel18 all good normal normalization

With 5) advanced normalization mode I get the problem however:

AdvancedNormalization triggers issue

So this means the following, advanced normalization only uses the background and dispersion in the regions where the mosaic panels overlap with each other, instead of calculating this for the whole individual mosaic panel.

Clearly, with these background values in de overlap areas, when the frames are normalized in the normalization engine, the noise and SNR calculations fail on some of them, because now too many pixels get clipped the black point... which is indicative of a data calibration issue still.

I guess we really need to see the individual bias, dark and flats to get any explanation for the issue. Can you upload them?

Mabula

 


Dav78 liked
ReplyQuote
Page 1 / 2
Share: