7th December 2023: added payment option Alipay to purchase Astro Pixel Processor from China, Hong Kong, Macau, Taiwan, Korea, Japan and other countries where Alipay is used.
Lets start off by writing that i am a relative beginner with APP, but have created a working mosaic last year with the trial version. That time i was able to "brute force" it by feeding all the subs at once and having APP crunch the numbers over night. It was also with flawed data so did not pay much attention to the end result, which did have some seams i was able to hide later in processing somewhat. This time i have almost 1000 subs, so that will obviously not work. Tried it, but wouldnt go anywhere for hours which i have read in other posts here will take an eternity and is not the way to go.
I have gotten a result that is acceptable in terms of visible seams, but 2 out of the 4 panels are very green, while the 2 others are good. Example below:
What setting/step am i missing for the process? I am 99% sure it is user error somehow, but not sure how to improve. This is by far the best result so far so kind of stumped.
What i did with this one:
Integrated the panels one by one first, removed light pollution from the tools tab on each of the panels, then mosaic registered, normalized and integrated into the mosaic in the picture above. I used MBB at 15%, as that is roughly what the overlap is. No LNC, it makes the result worse. The data itself is decent from a dark bortle 3-4 sky. Data was captured with an OSC camera, but i calibrated and split the RGGB channels into mono before feeding them to APP, so all the work APP is doing is with already calibrated mono subs.
Previous attempts were without removing the light pollution from the individual panels and these had not so great seams visible so i am going in the right direction at least.
I have no idea why you are getting this result, but when I have gotten unexpected OSC multi-session results, e.g. strong shift to blue, the calibrate star color tool often will bring the integration back to a much more reasonable color profile
Figured out a "hack" to salvage this particular mosaic integration: SCNR green to get rid of the greens and then photometrically colour calibrate in Siril. Also, another SCNR after that.
This works OK, but its not a solution for the issue and more of a band aid on it. If anyone has a clue why the green thing happened, please do comment.
I think that in this case it's better for me to have a look at the data, could you share like 10 frames of each panel and your calibration files? If you want to;
I did some more testing and found out that the choice of reference frame has an affect on how the mosaic turns out. Choosing one of the panels that are mostly background turns the mostly signal panels purple. With the reference frame set to either of the mostly signal panels i get the original issue, green panels 1 and 4 (the mostly background ones). For now i am testing with just normal OSC subs that have not been split to mono for simplicity.
Not sure how uploading the raw data and calibration frames will help, they are calibrated well with matching flats, darkflats and darks with no light leaks, offset issues, or temperature issues in the latter 2. Flats were taken directly after the imaging run and seem to be well matched (at least for a newtonian). Could i upload maybe 10 calibrated frames of each panel instead?
This was integrated with normalization disabled. This way the integration will work with any of the 4 panels as a reference frame. So looks like this is is some kind of normalization issue?
Another update, this time i got it to work, although with a bit of ass-backwards tree climbing. Normalization will not work with the 4 panels the way they are composed for this target. Tried a whole bunch of things.
What got an acceptable result across the image was:
Crop the 4 panels, LP removal of the panels, integrate with no normalization and no LNC.
This creates a flawed mosaic around the edges, but the core and surroundings are solid. Then i LP removed that image until it was as good as it gets, but still no good edges. Then i cropped that flawed mosaic so that only the core and a little bit of surroundings remained and used that panel as a reference frame to integrate again. Then integrated the 4 original panels to the newly created core panel without normalization (still green with norm) but with LNC 1 set to 3 iterations. This creates an acceptable mosaic with no seams and only the 2 bland corners with remaining gradient/LP/something extra in background values. This image was salvageable with background removal (in Siril) to create a good mosaic. Crude autostretch below:
With the raw data, I can go through the entire process myself so that may allow me to see any issue in the data, if present. Sometimes you can miss something even though all looks fine and by asking for a subset of the raw data (this can be 10 lights, 10 darks, etc.) I can get to the issue faster.
I am uploading 10 raw lights per panel for a total of 40 lights, all the darks, darkflats, and flats i used to calibrate them.
In case its necessary to know details about the setup:
Camera is a RisingCam one with an IMX571 OSC sensor, the lights have only 20 electrons of median background signal, but this is plenty enough to swamp read noise at gain 100. The cooler is not amazing so there is some fluctuation to either side of -10, but at this temperature it takes over 32 minutes for a single electron of thermal signal to arise (on average). Thermal signal doubles every 6c, so within 1c at these temperatures the difference is negligible, perhaps unmeasurable from single subs.
Scope is an 8 inch f/4.4 newtonian with paracorr to f/5. Dont remember what the fits headers say about the scope and focal length, but the focal length is 1025mm in case you need this info for some processing reason.
I would be very interested in hearing a possible solution for the normalization issue between corners 1/4 and 2/3. I stacked it so many times i most likely missed something elementary that should have/should not have been ticked.
Looking at the data now, couple of things I'm noticing:
- The flats seem to be a bit underexposed, you want to aim for about 50-70% of the linear histogram. Usually people aim for an exposure of about 1-2 seconds, same exposure then for the darkflats.
- The header of all files indicate a sensor offset of 768! This is way too high, usually this is around 40 (depends on the sensor).
- There seem to be a few files that have a different camera type (I see both ATR3CMOS26000KPA and ATR3CMOS26000KPA_U). If these are the same, my advice would be to change these filenames as APP now thinks they are different.
I'm now continuing with the panels. Thanks for making such a nice directory structure.
My flat panel is a little bit janky, a very cheap tracing panel from amazon that is a little bit too blue so the levels are all over the place. Calibrating with these flats works well for me in Siril however, so have not looked to improve them all that much.
768 offset is default in all the ToupTek made cameras (includes my risingcam) because at higher gain values there can be 0 value pixels otherwise and i sometimes use gain 251 if skies are dark and its windy (and i want to keep exposure time at the absolute minimum). At gain 100, the gain used here, the min value is detached from the left side by at least 300 so yes it is a little bit too high. Its not important though since its a 16 bit camera and losing a couple hundred ADUs from 65k to lose from is hardly noticeable. The instrument name thing is a mystery, sometimes its noted as ATR3CMOS26000KPA and sometimes its ATR3CMOS26000KPA_USB2 (the name cuts off) even though the same USB hub is used all of the time. I think the hub might be working as USB2 when i power the mini-pc with USB-C 100w and USB3 when powered with AC power. Siril, which i use for calibration, does not care about the instrument name so also haven't looked into that too much.
A gain is indeed something you can change and with a 16-bit camera you can choose to loose some of the dynamic range by increasing it. The offset however shouldn't be that high, you may optimize it by looking at when the histogram is starting to clip at a certain gain, but you'll then always land in the range of 50 (give or take). Of course this is your choice, but with stars being so bright and easily overexposing on the right, it's a pitty not to fully use a 16-bit's dynamic range. I've never heard of a sensor needing to be that high, so I can be mistaken of course. 😉
Ok, so I took all data and indeed LNC is not very useful for the mosaic. That can happen, LNC is not a requirement and depends on the data. Without I get a very normal result actually. I cranked up saturation to the max and lowered the saturation threshold (to really see the gradients), then I selected the max stretch preset of 30% and start performing light pollution removal. This resulted in the following:
Followed by star color calibration, the graphs look nice so the data you recorded have a nice broadband spectrum.
First, the offset argument. Your advice of 50 offset is not only wrong, but it is actively sabotaging anyone's attempt at astrophotography with this particular model of camera. For ZWO 2600MC (or also others, not sure) the offset value follows a 1:10 arbitrary offset value to image ADU value ratio meaning a 50 offset will result in a 500ADU bias frame. For the ToupTek made cameras a 50 offset would mean 50 ADU bias frame which is irredeemably clipped on the left side, leading to guaranteed calibration issues as pixevalues of 0 are impossible to calibrate properly. If you adviced for the offset to be 500 ADU, it would be ok for gain 100 with my camera, but clipped for anything more.
Why i mentioned gain is that increasing it will also increase the distance between the lowest pixelvalue and the highest pixelvalue due to noise distribution being 1 read noises worth (averaged) on either side of the set offset. Using a gain of 251, which has an e-/ADU ratio of 0.1 instead of the 0.25 of gain 100 will distribute pixels 2.5x wider away from the set offset value than with gain 100. See where the issue is? 768 is not that far off from the left side, and with gain 251 an offset of 500 would be too low and not all frames would always have a min value higher than 0.
But the main issue is that you did not in fact succeed with the mosaicing and was left with the same issue i was. I think we have different meanings for the word "very normal image" since yours is the same circus palette i too got initially, and the point of this thread. You can clearly see from your final image here that each of the panels have their own colour palette and they are not even remotely close to being well equalized with an obvious gradient within M31 itself... Attempting to star calibrate an image that was composed of unequal panels will result in an unequal image, no way around it other than having the mosaicing itself work as intended to create a normalized image.
Well, first off "actively sabotaging" is of course not what I'm trying to do here, I also mention that you can find the offset yourself and may depend on the gain as well. I simply don't have the experience with the most recent sensors yet and that's also what I mention. We're all learning right. 🙂
The color gradients all over the place are greatly exaggerated due to me over saturating the data (like maximum settings), the LP correction can then even these out. You started with the issue of having very clear green patches and that is definitely not what I'm seeing. I do see a difference indeed, but this was just a quick trial to try to help. If I have more time I can check to see if that can be tracked down.
This was integrated with normalization disabled. This way the integration will work with any of the 4 panels as a reference frame. So looks like this is is some kind of normalization issue?
Indeed @onikkinen, in fact I think this is a known problem and it is caused by having Backgroun Neutralization enabled in 5) Normalize.
I would suggest the following workflow:
Do not process the mosaic panels after having them stacked, I read that you already processed them with the Remove Light Pollution Tool. It might seem counter-intuitive, but chances are big that you have made things worse and not better by doing that. In my own experience, it is much better to do Remove Light Pollution only on the final mosaic 😉
normalize using regular or advancced normalization and background neutralization disabled. Since you have the core of M31 on the mosaic overlap areas you have created a difficult normalization in general I must point out.
process the resulting mosaic with Remove Light Pollution to remove remaining gradients.
Please let us know if this creates a better result.
Tried again, this time made sure that background neutralization was disabled throughout the process, including stacking the panels. Previously i may have had it on, not so sure now as i have stacked it a couple dozen times so have lost track of what resulted in which result. Results in most of the same, although i cant get it to be so obviously flawed with greens and purples anymore (must have been the background neutralization), but there still is a colour gradient across the disc of M31 itself. Granted, not a huge colour gradient, but with the full dataset and a heavy stretch it will be distracting.
The synthetic center panel method i described a few posts ago remains the best method of making this work, and i believe the best course of action would be to just shoot an actual center panel so that the normalization algorithms have a better chance of making the mosaic work. That i will test whenever the clouds go away and i get a similar quality moonless night as with this dataset.
Just to reiterate the workflow that i found to produce the best result here:
Stack panels as non-mosaic ones normally with no background neutralization, Run the LP removal tool on them (if not, colour issues with the panels). Mosaic register and integrate them with normalization disabled (if enabled, crisscross levels and seams around the image again). This creates an almost good mosaic where the core is good, but the rest of the image not so much. Crop the newly created mosaic so that just the core and a bit of all the 4 panels remain, use this as a reference frame for mosaic registration to stack the 4 LP removed panels again. Still no normalization, but this time LNC 1st degree with 3 iterations. This creates a mosaic with no seams, but bright corner LNC artifacts that can be easily removed with a background removal tool such as APPs LP removal, or the background removal tool in Siril.
I believe normalization just isn't given a chance to work with the way i have composed this shot (out of necessity, only way to make M31 fit on an APS-C chip at 1025mm fl and 4 panels), still happy to be proven wrong if there is a simpler workflow to follow.
Took some work, and i would prefer there to be a less roundabout way to get it done, but the end result is hardly recognizable as a mosaic which means in the end APP prevailed and succeeded in mosaicing the difficult data. Below is what i produced from the full dataset and the synthetic center panel - circus method of mosaicing. I do plan on expanding the integration by from 4 to 8 or 12 hours, so similar troubles are surely ahead.
Thank you very much for your feedback and the final result. I am glad that you get a much better version now 🙂
Regarding normalization, yes, by centering a High Dynamic Range object like M31 on the mosaic overlap area of this 4 panel mosaic, you have created the worst case scenario to get things right with correcting illumination across the mosaic.
I am surprised to read how complicate a workflow you took to get the result. Did you try my recommendation to NOT apply Light Pollution/gradient correction on the mosaic panels before combining them with both LNC and MBB enabled? Did you create the mosaic panels with MBB enabled to remove the stacking artefacts?
I want to drastically improve this in APP and your data would be great for testing and improving things here. One question for now: do you have uploaded all data for this result now? Or was the upload a smaller dataset?
Ok, i stacked them again, this time with MBB at 5% for the individual panels. That seemed to do the trick and the mosaic came out perfect with no merry-go-round workflow for the center panel. Heavy stretch and extreme saturation below, did not pay attention to other processing but this is just a diagnostic run and should be evident that M31 is well equalized:
For this shot, and the one i already processed fully i binned the subs x3 (with ASTAP batch bin) after calibration and debayering. Not binned or binned x2 will just take forever to stack and will not result in any extra detail as the FWHM values are not that great for the data.
For the individual panels:
Normalization: advanced, multiply scale, BMWMV, neutralize background not selected.
MBB at 5%, no lnc, integerated to reference to have the minimum edge artifacts just from dithers and 30 arcsec platesolve tolerance.
This time did not need to do the LP removal for the panels for the best result, and in fact if i do the result comes out worse.
For the mosaic:
No normalization, MBB 15% LNC 1st degree 3 iterations. The end result is a bit funny in the background values for the panels, but M31 is good all around.
Then crop edges, LP removal tool and the end result is as good/better than what i got from the previous circus attempt. Turns out i was the clown in the circus after all!
Was it just the MBB for the individual panels that did the trick? I honestly dont know, i have stacked it so many times i have no idea what i tried and did not try, i guess i did not try MBB for the individual panels at any point.
As for the data, i have not uploaded all of the frames. I have 58-60 workable subs per panel (at 60s per sub), i could upload them if you want to run them through for whatever reason? I have an unmetered connection so no big deal, will just take a while to upload.
Thank you very much for sharing your result and feedback.
Glad to see that MBB on the panels did their work so the mosaic combination immediately becomes easier, so yes, that is a very important step here.
I will now work on the normalization engine using your data to fix this issue properly. If I think I need more data for testing, I will surely let you know.
The solution will probably be that I remove the Background Neutralization option from APP's normalization engine and replace it with actual and direct Background calibration using samples chosen by the user from the reference frame, just like in the Calibrate Background Tool.
