Problem with stacking data from multi-scope system.

Hi @mxcoppell,

First of all, for data like this, you do not need the mosaic registration mode 😉 regular should work just fine.

Which APP version were you using?

It must be a problem from either normalization, MBB, LNC I think. The background neutralization has no influence because that only applies to multi-channel/RGB data.

Can you try the following: disable MBB and LNC and in 5) normalize, set the normalization method to none. Does that give a normal result?

I guess the error in stacking will be caused by a problem in 1 or multiple frames after 5) Normalize somehow. You can check the normalization of each frame with the l-c-r-normalised image viewer mode after 5) normalize is completed, did you check this on the frames?

Please let me know if I need to take a look at the data to solve this 😉

Mabula

Thanks a lot, Mabula!

I am using "version 1.083-beta1". Looking forward to the upcoming release!

Let me test your suggestion first - turning off normalization, LNC, and MBB.  But I didn't find the "l-c-r-normalised image viewer mode". Do you mean just double-click saved normalized frame to load it into the viewer? 

Mabula, I definitely would like your help to look at the data. But the data set is pretty huge (from ASI6200mm) - it's about 8GB. Let me find a place to upload it and share with you. 

Another off-the-topic question - The "Align Channels" function in 2) Calibrate, is it only useful for OSC one-shot camera data? Can APP help the same issue with LRGB data?

Thanks!

-Min

Hello Mabula,

Just want to give an update on this topic. Following your suggestion, I tried to do each step manually, and the result surprised me. With normalization, LNC and MBB turned on, the problem is gone. I repeated the test on two data sets and can't reproduce the issue anymore.

There must be something wrong with my settings when trying to run it as "one-click" in the "Integrate" tab. I will post updates if I find out more.

Thanks again!

-Min

Dear Mabula & Vincent,

In my recent projects, the issues I reported in this thread still present. Could you help to take a look at a data set I put on Google Drive? 

If so, what would be the email address that I could send the Google Drive link? 

Thanks!

-Min

Dear Mabula & Vincent,

Instead of the trouble of uploading a huge data set. I made 3 screen recordings to show the issue that I am trying to solve.

All 3 screen recordings show loading up the lights and calibration masters from two different scopes of our SBS system. With different configurations and results:

The first one, I was attempting to do a "Normal Registration" with the best light frame from the TOA-130. Got into all sorts of registration issue and I did the "Star Analysis" again. APP automatically selected the best light frame from the FSQ-106 as the reference. The end result is good - but the data from TOA-130 got binned 2x. That's not what I want, I need to keep the data details from the TOA-130.

First pass:

The second one, "Mosaic" registration mode is used with the best light frame from the TOA-130 (I need to interrupt the process after star analysis is done to re-select the reference frame from TOA-130). The result is what I reported in my previous posts. The data from FSQ-106 was "drizzled" 2X and TOA-130 data kept the original resolution.

Second pass:

The third one, the "Normal" registration mode is used. I got into all kinds of registration warnings. Following the on-screen help and bumped the stars in Star Analysis step to 2000 to make the registration work. I made sure the best TOA-130 frame was selected as the reference. The result is pretty much the same as the second scenario.

Third pass:

Of course, the FOV of the data from TOA-130 occupies only the center portion of the FSQ-106 FOV. When I select the 106 light frame, everything is fine. The problem only happens when the 130 light frame is used as the reference.

Is the reference frame used for both registration and normalization?

Mabula and Vincent, let me know if you want to try out the data set. I could send you an email with a Google Drive link. 

-Min

Sorry for the missing of an update on this one! We've informed Mabula, but I'll also have a look.

Maybe it's not necessary to upload the entire dataset, does it also go wrong when you take a subset of say 10 lights and the master calibration frames?

Hi @mxcoppell, @vincent-mod , @wreeven

I will have a good look tomorrow at your videos and will report back 😉

But, to be clear, you are dealing with data with clearly different iamge scales, right?

So increasing star count and/or scale stop in registration needs to be done to have it work correctly. The reason is that if you use the TOA-130 as reference, your are seeing more stars in a narrower field of view, right... To register the FSQ106 data, the defaults in 4) register are clearly insufficient. The FSQ106 data is not resolving stars probaly as good as the TOA-130, and therefor the scale stop must be increased I think.

Now, I have been working for a long time already on a new registration module where exactly these problems will be addressed to make it work 😉 automatically. That new module will be released going forward.

For testing purposes, it will be very usefull for me to have such a complete dataset of yours. I can really use it well to make the registration module more smart 😉

Can you upload one of these problematic sets here:

https://upload.astropixelprocessor.com/

username and password: upload4

Is the reference frame used for both registration and normalization?

Yes it is.

Thanks a lot,

Mabula

Thanks a lot, Mabula and Vincent.

Mabula, you are right. The image scales are 0.77 and 1.46 for TOA-130 and FSQ-106 respectively.

The data set has been uploaded to: /AstroPixelProcessorUpload/mxcoppell-dataset-toa130-fsq106/App-Question-MultiScope-Integration

Yes, for the registration part indeed there are some parameters that need to be changed from the default values to make the registration work.

Our main challenge is the normalization part. If the smaller FOV TOA-130 frames are selected as the reference frame, the normalization result is not correct. Using bigger FOV FSQ-106 frames as the reference frame will solve the normalization issue but also leads to TOA-130 frames get binned 2X. This is not what I wanted.

Or maybe different reference frames could be used for registration and normalization?

-Min

Dear Mabula and Vincent,

Just wondering if you got chance to try out the data set we uploaded? Any updates?

Thanks!

-Min

I'm not sure Min, sorry about that. I'll notify Mabula again.

edit: he mentioned he'll get back to you tomorrow, he has some data connection issues due to moving over. Should be resolved soon.

Ok, I'll download the data as well. Sorry for the delay, seems Mabula is quite busy at the moment. I see it's probably the whole dataset at 6 GB. I hope the download doesn't stall, but a smaller set with the issue would've been a bit easier. 🙂

edit: Got it to download, very nice folder structure, thanks! I'll see if I can find what's up, might be till tomorrow before I can get a possible answer.

Thanks, Vincent!

Yeah, those are IMX455 subs. In our standard, that's indeed a "very small" data set. 🙂

The biggest concern to us is the APP normalization outcome when the smaller FOV sub is selected as the reference. Looking forward to your findings! 

-Min

Ok, just a couple of things I noticed;

Data of the FSQ-106:

- You've used an old APP version for the masters (1.077), I would advice to reprocess with the latest version just to be sure (probably not an issue).
- The master-bias is missing data in the background, are you taking the bias frames at 0 seconds of exposure? I'd advice to use bias frames with exposures of 0.5 seconds. This is not a true bias perse, but it does work fine and prevents issues that modern CMOS chips have.
- The master-dark also seems to have this, which is a bit stranger. When I zoom into a master-dark I seem to see stars, can that be correct? I stretched the masterdark to 30% to clearly visualize this. That is an issue as well, you can't use calibration data like this properly.

Questions;

- What is the offset used in the creation of the bias and darks?
- How exactly are you taking these as it seems light can come through?
- Could you try a small integration test without any calibration data, just to see if that might solve some of your issues? (just use regular settings, no mosaic setting)

I'll update this post during my analysis..

Thanks, Vincent!

• Yes, some of the masters were created last year with 1.077.
• Thanks for the advice on the bias. We will recreate the bias masters with 0.5 seconds. 
• We use offset 50 for all lights/darks/biases/flats (for both Gain 0 and Gain 100)
• For the "stars" in the master dark, I am not sure that's possible. The scope was covered when the darks and biases were taken. It might be hot pixels? 
• We've done the test for the normalization issue that we are facing. There is no difference between calibrated and uncalibrated processes. One example, as showing in the first video, with calibration frames, if we chose the bigger FOV light frame from FSQ-106 as the reference frame, the result is just fine. The primary problem happens only when choosing the smaller FOV light frame from TOA-130 as the reference frame. 

-Min

You're welcome!

- Might be interesting to see if a higher offset might be better, but 50 seems like a good one. I'm just wondering about the missing data in the lower end of the calibration data (the dark vertical columns in the histogram).

- This is the masterdark on the FSQ106:

Which I think are stars, they cover multiple pixels as can be seen in higher magnification:

The white dots of 1 pixel around them are hot pixels.

- I'll try an integration as well with reference frames and such. I think I'll do this first without calibration data, stay tuned, will continue tomorrow.

Interesting, we might need to redo the darks. Thanks, Vincent!

Yes and when you solved that, redo the bias as well as I think they might have a tiny bit of light in them as well if that is taken in a similar manner.

Dear Mabula and Vincent,

Just want to touch base with you guys to see if there are any updates about the normalization issue. Any tests were done with the dataset we uploaded? 

Thanks!

Yes, we did right, 2 posts above I analyzed the issues in the darks and likely the bias frames. These might already explain some of the issues you have, did you retake the darks and bias to check?

Thanks, Vincent.

But, the calibration frame is not the root cause of the issue we reported for the normalization (when choosing TOA-130 FOV as the reference). 

We've done tests with/without dark and got the same result. 

Have you tried to do the following 2 integration tests with Normalization, LNC, and MBB enabled? 

• Use the best scored TOA-130 light frame as the reference
• Use the best scored FSQ-106 light frame as the reference

Thanks!

Ok, I will analyze the data further, I was hoping it was the calibration data. 🙂 Sorry for the time it takes, this can happen due to the busy schedule sometimes. Please allow for 1-2 days.

Going to let APP process the data now..

Ok, got the first data processed with the reference set to a 530mm focal length frame. I used 1.083-beta 2 of APP on Linux. These are the settings I used:

- In tab 0; select Ha for the algorithm
- In tab 1; I loaded the data of both scopes in 2 sessions, I didn't use calibration data
- In tab 4; I used dynamic distortion correction and switched "same camera and optics off", registration mode normal
- In tab 5; I switched to advanced mode as this is required for this type of data
- In tab 6; MBB and LNC were switched on with their default values

This came out:

Then, some light pollution correction:

Amazing data btw!

Other way around, same settings as mentioned above, now with a 1000mm focal length sub as reference. This made APP complain a bit with registration, so I increased the number of stars to 1000 and scale stop was set to 10.

This came out:

Corrected:

Thanks, Vincent! It's about the same result here. 

"Corrected" - How is this done?

I looked carefully into it. Still can see the uneven normalization between the two FOVs. Especially for the case with 1000mm FOV as the reference. Even a large value of MBB is applied, my guess is that there is not enough information to normalized the bigger FOV frames from the 530mm. 

The purpose we use the 1000mm FOV as the reference is trying to retain the details from the larger scope - at the same time, normalization should be done with the 530mm FOV reference. Could this be done?  

@mxcoppell Corrected was done using Light Pollution Correction in the tools menu.

Biggest difference with your workflow I think, was that I didn't use mosaic mode (this isn't a mosaic). You followed my settings I guess? If so, I'm not sure what you're seeing as the data looks really good to me. There is of course more data now for the center part of the image so slight differences compared to the outer part might be visible. Normalization is, as far as I know, done on the whole dataset.

Yes, Vincent. In my latest test, I am using exactly the same setup as yours (no more mosaic). The concern is still about the normalization - if the normalization is done on the whole dataset, including the 1000mm and 530mm FOV, the reference frame of 1000mm should not be used as the normalization reference right? It doesn't have the data to cover the 530mm FOV. 

Could that be the reason the brightness of the 1000mm FOV area is much brighter than the outer 530mm FOV area, and lost much of the contrast in the integration result. 

My feeling is that this difference is likely due to the fact there is more data in the middle. But I'll pass this question on to Mabula as I'm not sure this is the correct feeling.