Multiple Images In Stacked Output

Could you explain what exactly we're looking at? Like, what did you do to get here, what type of data etc? Thanks!

I've been getting the same thing...

for months my stacks came out perfect and fine. I haven't changed anything in my settings, darks, flats, images, etc.. haven't touched anything in the APP process for the star analyses, registration etc. I have left everything the same as it always has been. And that has always given good stacks.

 And for some reason, now they're not stacking properly anymore. started about a month ago, sometimes it works.. but often it has problems, and for some reason it keeps doing it. Even on stacks that I have done previously without any issues. 

- I'm on the latest APP version.

- I've removed and reinstalled APP

- Again; haven't changed anything in my process. Haven't touched any of the sliders/options/ticks/boxes etc in the registration, star analysis/normalising/etc. I've pretty much left them all the same since day one. since it has worked just fine for me.

- No issues with the light frames; I've removed all the ones with bad quality scores from the integration (lights to stack slider, or just remove them all together) and don't attempt to stack those in. and all remaining ones have a star density in the 500's and good quality scores. Which should be more than enough. I haven't changed anything in the capturing of those frames. Same camera, same file names, same everything.. nothing has changed in that part of the process. (ZWO ASI183MM Cooled). So it's not like I'm trying to stack bad data. (in my screenshots it's about 40x 600s data).. in the screenshot I've used it with masters, but I've also made attempts with 50 darks, 30 flats, 100 bias. etc. no joy.

- Darks/flats/bias, it does it when I use the individual frames.. does it if I use masters. Even used new masters, and added some new darks, as I thought maybe something changed in my darks/camera.. but still does it.

Yet despite all of that, it doesn't do a proper stack, even on data that has stacked before without any issues... It's like it is stacking frames next to each other instead of on top of each other. Resulting in a "ghost like" image with a few of the nebula next to each other.

 This is a "stack" of the tadpole nebula. With data that I've already stacked before without any problems. yet somehow.. today it doesn't want to stack. And I don't know why.

Hopefully someone can tell me why. because it's getting very annoying that I let my computer do all the work for a stack that I can't then use. I obviously want to be able to reliably stack my data, when there is nothing wrong with the data.

I don't know if it's an issue with registration or normalising or anything like it.

And I have no idea why, for months, things have been fine. And now for some reason it doesn't want to stack anymore. When even there hasn't been any updates to the program that would all of a sudden explain this. (unless it automatically downloaded an update? but I don't think that APP does that?!)

It is also not limited to only this data/camera setup. As I've had the same issue when stacking images from a Canon DSLR. So I'm hesitant to blame my data.

How does APP load in its data for the stacking and processing? one thought (and I haven't tested this at all... not until I know how APP takes in its data).... could it be that APP has issues using data from external harddrives? that the harddrive on which the frames are stowed is getting slower/too slow?

As, over the months. More data/frames have been put on the harddrive and APP isn't as good at reading the data?? maybe?? now..... I don't think this would be the case.. it might make the process slower from a harddrive instead of an SSD and it shouldn't cause issues... But.. it's a thought....?... but that needs to be explained by someone from APP, to explain how APP is reading data and if that might be possible.

@stevendevet

I’m having the same problem.  As a new user everything worked swimmingly, then I started getting triple images on the integrated output.  I am able to load 30 images and get a good result, but beyond that, garbage.

More attempts made to try and figure out what the hell this issue is and to try to narrow it down.... please bare with me.. it's a long testing process; but hopefully. By being as detailed as possible. we can narrow it down or someone at APP can give information.

- Downloaded the 1.083 Beta to try a stack in there (to see if it might be an issue with the 1.082 or settings or something else).

But again, I have the exact same issue. My Oiii data stacks fine. My Sii data stacks fine. But my HA data is still being stacked in this weird way. 

The data for Oiii/Sii/HA is all the same. Same camera, same rig, same setup, same filenames, same cooled temperatures, etc, etc, etc, etc, etc. The only difference is that it's more HA frames then it is Sii and Oiii frames. 

HA 43 frames / Oiii 25 frames / Sii 10 frames. But, that shouldn't be an issue.

Now; you might say; "it could be an issue since I'm doing all 3 data stacks at the same time". but.. since I've also had this issue when stacking only 1 set of data. And had this issue when stacking full colour Canon DSRL data; I doubt that that is the issue.

- So, maybe narrowed it down to my HA stack (?)

Next; Made an attempt to stack myHA data without any darks/flats/bias or anything. In an attempt to narrow down wether or not the issue lies anywhere in my calibration frames.

HA stack - NO calibration frames = good stack

This is my HA data after registration/normalising WITH all my darks and BIAS:

This is my HA data after registration/normalising WIHTOUT all my darks and BIAS:

Now...... I don't see a difference.  Numbers/quality scores/signal to noise/background-dispersion/etc/etc. All seem very similar in both. So.. no idea.. but it's not like the calibration frames are messing up something in that data.

- Next; An HA stack with FLATS+BIAS

I have no idea if my previous good stack without calibration frames was just luck, or an issue with the frames. So, next step.. isolating what frames are/might be the issue..

HA stack + FLAT + BIAS = a good stack

just with a massive amp glow in the frame; as well... there are no darks...

so, this might limit my stacking issue to my DARK frames?

I have to stress. I also have had this issue with APP also when attempting to stack data from a Canon DSLR with and without calibration frames.. So, while currently the culprit at this time during my test seem to be my darks. I've had this issue also on different data and a different camera setups. So I doubt that this is limited to my DARKS.  (also, since others are reporting the same issue)

- Next; HA with different portions of my DARKS.

I've made new darks recently, as I was using 20x600s darks. And wanted to get closer to about 40-50 frames. So; 30 new darks were made last week. the first 20 were made on the 11th of feb. the other 30 on the 9th of March. Only a month in between, so.. there shouldn't be a big issue there and I see no reason why there might be issues.

Especially since I've had this stacking issue while using the original 20 darks. and have the stacking issue when using the full 50.

. But, for the sake of the test; I've loaded in my 20 darks, re-ran the calibration/registration/etc and go for a stack. (I've checked every dark frame individually to see if there are issues with some. But they all look fine):

HA stack + FLAT + BIAS + 20 DARKFRAMES = good stack

HA stack + FLAT + BIAS + 30 (new) DARKFRAMES = good stack

HA stack + FLAT + BIAS + 50 DARKFRAMES = good stack

this.. does.. not.. make.. sense... As my first initial stack in 1.083 used all darks, all bias frames, all flats. And it didn't stack properly. but in the current attempts, it stacked fine??????. when NOTHING has changed in the settings and data. so there is no reason for this...

And again.. I've had this issue when stacking data from another camera/setup. with and without darks.. So, I very much doubt that this is related to my dark frames and that any good stacks in this test, are more "luck" than finding the actual issue behind this.

fine.. one more attempt/test

- Completely reload everything;

All HA frames, ALL darks, BIAS, FLATS loaded in. And attempt another stack with everything

Second HA stack + FLAT + BIAS + 50 DARKFRAMES = BAD stack ...... 

what.... the..... hell..... I just stacked all this data just fine?!! there is nothing wrong with the data.. why doesn't it work this time?

How am I supposed to be using a program that can't reliably stack my data anymore?!?

With all of that in mind. and with more people reporting the issue; I would guess, that the issue is somewhere in the software. But, I don't know where. I also don't know why for months I had no issues at all with stacking. But for about 1 month now, I've had mixed results with stacking.

Anyway, hope this data/tests/text helps in any way to find what's wrong. And someone can give some guidance.

@stevendevet Thanks for the thorough analysis. This is a strange situation. Could you upload all files to

https://upload.astropixelprocessor.com/

using upload5 both for username and password? Please create a folder called stevendevet_multiple_images and put all files in there. Then we can have a look and try to help you find out what's going on.

Thanks! Wouter

@wvreeven

Thanks for the reply Wouter, this has now been done.

Uploaded all the Ha files (48 think) This does include 5 bad frames. I let APP analyse them, and them kick them out. I didn't go into my files to filter those out. And in my tests it was done with the good 43 frames.

Uploaded the masters of my Bias and Flat for HA, only went for masters. As I doubt there are any problems there.

Uploaded the 50 darks.

I have had better results with these files when stacking them normally. But as soon as touch anything in the drizzle settings, it doesn't work for me. But the Oiii and Sii data do drizzle fine.

But, this issue isn't limited to drizzling. I've also had the issues with other stacks of different targets without even attempting the drizzle (also a stack from a different camera). And the stacks failed. 

@stevendevet Steven, I have done several integrations with your data on my MacBook using APP 1.083.

1. I loaded all lights (just the FITS, not the jpg) and went straight to tab 6 and hit integrate. The results was good thought understandably lots of amp glow.
2. In tab 6 I modified mode from interpolation to drizzle and scale from 1.0 and 2.0. This I did straight after the first integration finished. The result was good though still lots of amp glow.
3. I restarted APP. Then I loaded all lights (again just the FITS), the darks, the master flat and the master bias. Then, again, I went straight to tab 6 and hit integrate. The result looked good and the amp glow was gone.
4. Then while still in tab 6 I again modified mode from interpolation to drizzle and scale from 1.0 and 2.0. This I again did straight after the previous integration finished. The result was good again. No double images at all.

Here is a screenshot of the result of the first integration:

Here is a screenshot of the second integration. Notice the difference in the number of pixels for both axis X and Y:

Here a screenshot of the third integration. Notice how the amp glow is gone and the number of pixels is similar to that of the first integration:

And a screenshot of the fourth integration. Again no amp glow and the number of pixels similar to the second integration:

So the question is: what did you do different?

@wvreeven

Hello Wouter, thanks for the reply and the tests that you have done. 

I've just done the same test as you; (your test 3 and 4 at least)

- fresh start on 1.083 (also MacOS)

- Loaded in the Lights (fits), darks, master bias+flat.

- Went to step (6)

- Press integrate.

the result:

what I'm noticing already are the sides of the image. There are a lot of lines on the edges, and they are lightly noticeable in the image. No idea why they are there. I didn't do anything different. But we're already getting different results... and not the same quality of integration 

Anyway... to continue the test; Next;

- staying in step (6)

- set to drizzle - set to scale "2"

- Press integrate

the result: 

So....... not great.... and a difference that I can't explain, as we took the same steps.

The next thing I wanted to try, just to rule it out as an issue, is the use of an external hard drive.

I keep all my data and files on an external harddrive. Maybe APP has issues loading in significant amounts of data from external sources?, or the harddrive isn't quick enough?.. or .. whatever.. I just wanted to rule it out.. the more things we rule out.. the closer we might get to a solution

So, as a next test; I've copied all the frames (Light/dark/master flat/master bias) to my internal hard drive. And I ran the test again;.

- fresh start on 1.083 (also MacOS)

- Loaded in the Lights (fits), darks, master bias+flat.

- Went to step (6)

- Press integrate.

the result:

OH?! interesting.... no more lines on the side, so this already seems better. Might be onto something here? (possibly "luck"... but.. it does seem better).. Nothing was changed in the settings or steps at all.

So..... Next;

- staying in step (6)

- set to drizzle - set to scale "2"

- Press integrate

the result: 

Perfect result. Can't see any lines, edges look good, no issues at all.

So, it might be related to external harddrives/external sources. (mine is a WD elements hard drive.. not an SSD) Don't know if the others that have this issue in this forum are also loading in their data from external sources or not. (from an external harddrive, SD card, USB stick, etc)

I believe (if I recall correctly) I also had issues when integrating the CANON data from an SD card directly.. so.. could be the same issue..

From here on out, I'll be making sure that I'll be doing any integration directly from my internal harddrive (Or possibly upgrade my external harddrive to an SSD). And see if this issue won't happen anymore.

Thanks again Wouter for running the tests.

It might be too soon to tell at this time; but perhaps the possible issue of external drives can be relayed to the APP team. It's probably more likely that the fault would lie with a slower harddrive or external sources, but maybe the issue would lie within the software and programs of APP, where the software might have issues with integrating project from external drives. Might be worth double checking.

@stevendevet Steven, I actually wanted to ask about using an external HD and forgot. Yes, that certainly is the issue. APP doesn't handle external HDs well, unless they are SSDs. I myself use a 1 Tb SSD and that works very well. Glad to see you worked this out yourself and sorry for forgetting to mention that.

@rsmunromarsweb-cm I have the same problem.  I have asked support about it and they suggested a Registration issue, but I don't think that's the case.

@wvreeven Hi Wouter, I have yet to remedy the problem that caused me to start this post.  I've had success using an external hard drive, until now.  I've tried it on two different computers with the same, bad result.  I'll try moving the files to my internal hard drive but it seems impractical for astro photographers to have to move their files to an internal hard drive for every integration. I even got so frustrated with this problem that I downloaded a trial of PixInsight.  I loaded the same data on it and got a great result.  I'd rather use APP because it's a bit easier to use but the lack of training or end to end tutorials for beginners is making me think I should just spend the time learning PixInsight.  Is there such a beginning to end training tutorial for beginner users of APP?  I know that softwares have their issues but it seems there are many people having this issue.

@vincent-mod Hi Vincent, the images I uploaded were a single light and a stacked result of M42, the Orion Nebula.  It seems there is a problem in APP as I've tried this same data, and different data on two different machines and gotten the same problem.

We did ask Mabula to maybe check if someone is using an external drive or not as this can cause issues. At this moment APP will use the external device then to process on and requires a load of data processing expecting a normal and fast flow, so at the moment this is a requirement.

@wvreeven I gather my 256 GB USB drive might be the problem.  When I split the data into smaller groups and then combine the groups later, my results are good.

@rsmunromarsweb-cm Yes that probably is the cause if you see multiple images and load a lot of files at once. It may indeed be worth integrating smaller groups and then combining as opposed to copying all data from the USB drive to your hard disk, integrating them and moving them back.

@wvreeven having the same issue as described above. Four duplicate images in the single final integrated frame.  I am using an external HD, but like you, it's a Samsung 870 1TB SSD connected via eSATA and its read/write is 560/530 MB/s.  It does run through a hub, Stone Pro, rather directly to my Mac Mini.  But I use the same hub during all my software development and is connect to my Mac via a 10-Gbps

USB-C, so I can't imagine it's creating any lag.

Per your last comment, and never having done small batches only to integrate them later, do you follow the same process and then add all the small batches together as if they were individual lights?  Do you still have to use flats, darks, bias, or BPM when integrating the batches together?

@slaughters85j Hi John, I usually process everything straight from the SSD though I use a 2019 MacBook Pro with 64 Gb of memory so that may be the reason why it works for me. Have you tried copying all data to the internal HD of the MacMini to see if that solves the issue?

No, but I'll give it a try.  My Mac mini is the 2018 model and has 32GB of RAM and I've allocated up to 10 GB to APP.  Def not a performance issue, at least I'd hope not.

@slaughters85j Why not give is more memory? 24 Gb or even 28? And make sure (at least for now) to not do anything with the Mac while processing in APP and also make sure that the screensaver doesn't kick in.

@wvreeven I would if necessary, but as is with just 10, APP is only utilizing less than 25%.

@wvreeven nonetheless, moving everything to the local environment did the trick.  Definitely wasn't a performance thing, but is something with the HD.  What's off is that I used a thumb stick last week and it worked fine.  Both the thumb drive and the SSD are in FAT32, for hardware compatibility reasons with my ASI Air Pro.

Also, I switched the drizzle setting back to normal interpolation after canceling the initial run (on my local machine) after the normalization .DAT file grew to nearly 150GB for 158 16-bit 30s lights.

@slaughters85j Glad to read that you managed to make progress on this. I have reported all issues related to using external drives to Mabula.