[Sticky] Tricky Mosaic workflow, alignment problem

Anyone want to have a go at stacking these four panel files into a mosaic and report how you get on? I keep getting distortions in the final mosaic following Mabula’s tutorial exactly and any number of other variations on that workflow. 

Chris

Works just fine for me.

What I did:

• Fresh start of APP 1.079
• Load the fits files, no multi-channel, no multi-session
• Set registration mode in 4) Resister to "mosaic"
• Click "integrate" in 6) Integrate, and use the proposed modifications when asked to change from the default settings.

I think the problem with the registration, is the under sampling of the data. When you zoom in, the stars look very blocky. I would suggest to integrate the panels with another pixel interpolation method (e.g. Cubic B-Spline). Or to take much more pictures per panel to use Drizzle.
Did you calibrate the images with Masterflat? The panels still have vignetting.

Excellent stuff Ralph and Minusman!

Posted by: @ralph

Works just fine for me.

What I did:

• Fresh start of APP 1.079
• Load the fits files, no multi-channel, no multi-session
• Set registration mode in 4) Resister to "mosaic"
• Click "integrate" in 6) Integrate, and use the proposed modifications when asked to change from the default settings.

Ralph, please do me one more favour and post a zoomed in section from the top centre join. I have been able to get close using your suggested technique but the stars at the top centre join are doubled. I know this because I have the R and B mosaic to compare against.

thanks for responding 

Chris

Hm, with more careful inspection there are some alignment mismatches in the right hand scarf.

And with 1400 stars instead of 500 I get worse registration RMS and can reproduce your warped space artefacts in the upper left hand quadrant.

I increased the number of stars detected to 5000 (actually detected stars about 1500 per panel) and played around a bit with the settings. With mosaic registration, scale start 4 and scale stop 6, and dynamic optical distortion on I get decent registration results. For the integration task I set 2nd degree LNC with 10 iterations (yes I know, probably overkill).

There is still some small misalignment visible in the center of the image where the 4 frames overlap, but it's only visible when I search for it. The rest of the alignment looks OK.

It is in my experience unusual to run into these problems, I think you happen to have a bit of bad luck with the way your data turned out for this channel. Advice for future integrations: increase the overlap between the images, or acquire more or deeper subs in order to identify enough stars in the overlap regions.

Alternatively, using the original exposures, select a handful of them per panel and detect the stars and do the alignment of all 4 panels. Have the "same camera and optics" option on, and save the distortion model. Then, use the distortion model for the registration of all the frames for each panel. I think this will reduce the artificial warping you seem to have introduced in your stacked panels.

Comparing the image in the previous answer to the one here, I see a lot of (visibly acceptable) distortion in the centres of the individual panels. This is of course possible for the registration routine since there are no stars in other panels to constrain the positions of the stars there or constrain the deformation. But it is at a first glance unexpected behaviour for the registration outcome.

@vincent-mod do you happen to have anything to add?

@minusman Thanks for responding. I did use a master Dark, Flat and Bias for the panels. 
I’ll try your suggested pixel interpolation method and post my results. 

Chris

I have no experience yet with APP and multi-channel processing, so hopefully somebody else can jump in here.

Since stars should end up at exactly the same place irrespective of their colour, I would do the registration of all colours at the same time in one go. This way, it should be ensured that all stars in all colours are properly aligned, and that you don't run into the problem of one colour behaving in an odd way, as is the case here.

@ralph

I’m incredibly relieved to hear someone else has run into the issue. I used 2x2 binning 300sec subs and one hours worth of data in all three colour channels as there isn’t much colour in 95% of this IFN mosaic. Strangely, despite having precisely the same overlaps only the green channel is doing this. I reshot 11 subs Friday night and they stacked fine but had the same distortion when combined into a mosaic. 

thanks for the workflow break down. I’m learning all the time on this one. 

Kind regards

Chris

@ralph

As you may have seen from my linked thread on this forum ‘Pig and Fish images’ (under General) I have worked for days on this issue. I initially tried putting all the subs in together as you suggested but the integration ran overnight and then gave me the images you can see in my previous post. Mabula helped by restating the process and successfully integrated the Lum mosaic. His technique has not yet worked for me in the case of green. I’m frustrated as it’s not logical and looks like I’m at fault as all the other mosaics have worked and new data made no difference, so I cannot finish of this project until I crack green. 

Thanks for the continued help.

Chris

Ok, so distortion issues in this case it seems as well as undersampling. You will have a lot better results when you dither and use drizzle integration.

So, maybe I missed it, but did you select to use distortion correction? And what % is your overlap? I will give it a go later as well, so thanks for your data and thanks @ralph for your valuable input!

Vincent, 

Panel overlap was set at 10% for all subs in the entire HaLRGB mosaic. I have tried with and without DDC in all possible combinations with Same Camera Same optics (both following and ignoring APP warnings and Mabula's workflow guidance) including not having either selected.

My personal closest result (much to my surprise) to something I could put together with the other mosaics and get 95% star alignment apart from in an area I could crop out (but would impact M81 framing) was achieved this morning with the following workflow:

3) 3000 stars

4) start scale 1, stop scale 15  DDC and SC&O enabled (I know that's not recommended), mode: Mosaic

5) Advanced and Neutralise

6) Automatic, Full, 2nd Degree LNC, 3x Iterations, MBB 45%

Very non-standard but as I told Mabula I have tried everything else.

Thanks for offering to give this your time.

Chris

Posted by: @minusman

I think the problem with the registration, is the under sampling of the data. When you zoom in, the stars look very blocky. I would suggest to integrate the panels with another pixel interpolation method (e.g. Cubic B-Spline). Or to take much more pictures per panel to use Drizzle.
Did you calibrate the images with Masterflat? The panels still have vignetting.

 

 

 

I have just tried the Cubic B-Spline setting as you kindly suggested with Mabula's recommended settings other than that. You'll see that I got the same 'plug hole' distortion in the middle of the bottom left hand panel as you did plus a tail on the upper left. Generally the stars did fatten up and became much less blocky. I wonder how that would integrate with the Ha, Lum, Red and Blue (green halos perhaps?).

That's not vignetting by the way, its the consistent LP I have from this site.

Master Flat:

Chris

Thanks for your patience Chris, it might be the data itself, but hopefully I can get something out of it. If not, it'll be very interesting for Mabula as he always wants to optimize things to make tricky situations work as well. If possible.

The exposure time of the masterflat is extremely short.
Mabula has always recommended exposure times of at least 1s or more. Because some sensors do not behave linear at short exposure times. This can lead to an overcorrection in the lights. I have experienced this myself.

@minusman  That's an interesting point but are you or Mabula referring to CCD's or CMOS sensors or both?  I have used the following system for nearly three years and have never had issues before with distortion or with vignette or dust mote repair.  I guess the exposure is shorter because I'm using 2x2 binning. 

I used the Flats tool in APT. It dictates the exposure against the light source, in this case a light panel and the full well, which I set at 20K as suggested by ATIK for the 460EX mono.  So, if I understand you correctly I should be using a dimmer panel as the other parameters are automatic or driven by the manufacturer.

I'll see if I can find any threads on Flats timing on here.

regards

Chris

These longer exposures are indeed advised especially of the new CMOS sensors which seem to have an issue with this. Dark flats are also a fine alternative.

@vincent-mod                   So over 1 sec is also recommended for the ATIK 460EX CCD?  Given that my current exposure for 2x2 bin is 0.01 that's a x100 variance on my current Flats illumination. It's a dim flats panel already. Surely that's not reasonable?

That's a CCD and I don't think they had that issue. If you do have trouble though, it still won't hurt to go to a higher exposure.

When you've dimmed a lot already, you can always add an extra layer to the panel or something like that.

OK, I tried a different approach. I took the liberty of downloading the files you uploaded to the APP server and did my own analysis.

Very simply put, I dumped all the exposures per panel on one big heap (of course taking care to use the proper bias, dark and flats for calibration) and did the integration using the default settings of APP. Since all exposures overlap to large degree this worked without any issue.

Then I took the 4 panels (with all LRGB data dumped into a "masterluminosity" integration) and did the mosaic out of this. No cropping of edges, not worrying too much about visible edges yet. The registration RMS of this mosaic ended up at 0.09 pixels! That's awesome. In total 243 common stars were needed. Visual inspection confirms the very good registration RMS value, I could find no registration errors.

So now there is a master alignment reference that can be used to align all other individual exposures in normal registration mode. Before integrating, deselect the master alignment reference in the list of lights, so only the LRGB original frames are used in the final integration. I haven't done this step yet but will set things up to do it now.

Okay, that's really out there. Just so I understand you:

1. Stake the whole of each filter data set in turn in one process using default APP settings with associated Flats, Darks, and Bias to make each of the filter Mosaic's. No LP fixing or stretch adjustment.

2. Then, with the four filter mosaics you combined them in normal mode into a single 'SuperLum' mosaic. Still in rough.

3. Then you went back and registered each filter panel, again in one go, into a filter mosaic but registered with the 'SuperLum' file. The SL file was then deselected/removed for that filter's integration to make that filter's mosaic.

4. We now have four mosaics, all registered to a common 'SuperLum' file which can be tidied up for LP and then be finally registered together, ready for colour processing etc as required. 

Have I got this right. I'm wondering if the four mosaics, once tidied up need a final registration together or will they already be good to layer and work directly in Photoshop?

Chris

Almost right. There is only one mosaic in the whole process.

I did the following procedure:

1. Generate the masterflats, masterdarks, masterbias.
2. Select all data for what you identified as Panel 1 in your directory naming. This was the luminosity data (from two different dates), the red data, green data, and blue data. Using multi-channel and multi-session processing I assigned the correct masters to the lights. I used 2 different dates as 2 different sessions.
3. Analyse stars, 500 was enough; register with the default settings; normalise with the default settings; integrate with "integrate all" for both multi-channel and multi-session options. This generated one SuperLum frame for Panel 1.
4. Repeat step 3 for the 3 other panels. We now have in total 4 SuperLum panels, covering your entire intended field.
5. Remove multi-channel, multi-session form the settings. Load the 4 SuperLum panels.
6. Analyse stars, default settings with 500 stars.
7. Register in mosaic mode with dynamic distortion correction, not the same camera and optics, scale start 5, scale stop 10 (I don't think the settings are that critical as long as the mosaic mode is selected). This worked like a charm with beautiful 0.09 pixel RMS alignment.
8. Integrate with 2nd degree LNC and 10 iterations (not that critical). This generates what I call the Alignment Reference, it's the picture in my previous answer, its size is about 5500 x 4500 pixels in my case.
9. Re-activate multi-channel and multi-session. This will clean all your input files in the list.
10. Load masters, and assign to the correct channels and sessions.
11. Load all lights, yes, that's 221 of them. Make sure to assign the proper channels and sessions.
12. Now the important step: load the Alignment Reference. I just assigned it to its own channel and session 1, but that shouldn't matter I think.
13. Analyse stars, I selected 5000 stars here. With 500 stars quite some frames failed during registration due to too few star pairs in common.
14. Register, make sure to use normal registration mode otherwise it's going to take forever. Also make sure that the Alignment Reference is indeed selected as reference frame for the alignment. Scale start of 1, scale stop of 10, dynamic distortion correction on. This is going to take quite a bit of time, half an hour in my case. One registration had an RMS of over 100 pixels and had clear distortion, so I removed that one. There were a few others that had an RMS of well over 1 pixel, but the majority of the images had a very decent RMS value.
15. Normalise, I set the advanced mode here.
16. Uncheck the Alignment Reference.
17. Integrate with your favourite settings.
18. Combine RGB.

At the time of writing step 15 is still running, but I don't expect anything strange to happen from there on.

@ralph  

What can I say but a huge thank you. It looks like a very comprehensive guide. I guess we should both get some sleep now and re-attack this in the morning.

kind regards

Chris

https://www.astropixelprocessor.com/community/tutorials-workflows/astronomical-data-calibration-priciples-must-read/#post-2516

Here is the link to the forum post about data calibration. The reason why I noticed this is that the gradient in the panels has a strange shape. Top and bottom right, dark corners.

Very nice set of data, looking at the combined result!

I've done just a little bit of light pollution removal on the final image. It is something that requires a lot of manual labour, but something that also pays off. Take some time to finish it properly. Tips: enable saturation in the tickbox on the right, select 30% BG, 2 sigma, 0,0% base, set SAT to 0.50, SAT.TH to 0.00. This will make everything in your data horribly visible 😉 and allow you to attack the unwanted gradients in your image. 

You're way ahead of me Ralph, thanks so much! Excellent result I think.

Posted by: @vincent-mod

You're way ahead of me Ralph, thanks so much! Excellent result I think.

Yes, a massive thank you to you all for helping me unpick this problem. I’ll be running through all the superb workflow advice today and hopefully will have first hand sight of my IFN mosaic soon. 

Clear skies and good health to you all.

Chris

Lately there have been some great posts, I'm making a sticky of this one as well.

So I started on the big dataset as well as the 4 panels didn't work for me either, likely because indeed some settings should've been tweaked during the making of the panels. Apologies if some of my findings are already mentioned above, I just went on first without looking at the other nice result to see if I can make it work. There are other workflows to do this as well, this is just mine.

My findings and some recommendations for the Luminance;

- Per panel;
  Star amount: 500
  Scales: Start 1, Stop 5
  No dynamic distortion correction, same camera and optics
  1st degree LNC, 3 iterations, 5% MBB
- I decided to leave the flats out as they didn't correct the luminance correctly (without them, the overall background illumination looked better, so the flats need work).
- I processed each panel fully, correcting with the light pollution tool (judging where the nubilosity is by overstretching the preview) then saved that result.
- Seems some data was taken with a different capture program? Panel 2, Luminance from 04-21 has a slightly different resolution (by 1 pixel) and different description of the instrument used. I'd advise to keep the capturing exactly the same, as different capture programs might influence the data differently (preferably they don't influence it at all, but most important is to keep this the same). APP can handle all kinds of resolutions just fine, but calibration files need to fit exactly with the light frames.

RGB;

- Same settings as Luminance
- Didn't have the flats for these
- I loaded the R, G and B per panel (assigning them manually to the correct channels) and made a combined RGB result per panel.
- I did the removal of light pollution and color calibration.

Mosaic;

- Restarted APP
- Loaded the Luminance panels as Luminance
- Star amount: 2000, Scales: start 4, stop 12, dynamic correction, not the same camera/optics
- Advanced mode in normalize, MBB to 10%, 1st degree LNC, 3 iterations

That gave this, beautiful, result. This is without any extra correction and very much stretched but the Luminance lends itself for that I think:

And then I had to give up for today as the RGB panels didn't want to register properly with each other. Was the overlap on the RGB panels 10% as well? Apparently it can work as Ralph shows, but it might need some extra steps like he did with all the frames. Shouldn't be necessary though.. I also get low (100-300) star counts on the RGB panels, so that might be a bit of a data issue. Will continue tomorrow.

I think I'll reprocess the RGB to see if I can make that better.