Stacking Technique for Multiple Night Capture

Dom

Generally you are better off loading all your frames into the final integration. This will give the best overall result. Maybe you could start it going and go watch a movie.

But it is certainly feasible to integrate each night's files separately and then load these integrations as lights (no calibration) and stack them together to produce the final file.

It is helpful to understand the difference between median and average stacking in this context. For each night you may have a hundred or so lights. These will be averaged in stacking, which reduces noise and also magically (or at least arithmetically 😉) creates extra bit depth, which may reveal extra detail. When the final files are stacked together these will be stacked using median stacking (assuming there are less than 20 files). This may reduce noise a little, but will not create any additional bit depth. That probably wont matter much, but is theoretically not quite as good as stacking all the files at once. But I don't want to overstate the difference, which may not be visible.

From a normalisation perspective it is again better to normalise all the lights at once, but again the difference may well be negligible.

If you are drizzling then that changes things a bit, as it is important to maximise the number of dithers in each stack. I don't know how much control over dithers the Dwarf 3 gives you, but I would want at least 40-60 dithers per stack (and I'd rather have a hundred). That might affect how many nights you put into each stack.

JC

Thanks John. Of course I had to have a go at the session FITS then stack the integrated method. There seems to be a little less noise in the all FITS method. On the median / average - I used average on the FITS and stacked FITS - but does APP override to median if there are less than 15?

I'm fairly sure APP doesn't override your choice if you change 'automatic' to 'average' with small number of frames. I just didn't want to provide too much information at once - I do that sometimes.

JC

I've found that for multiple nights you need to at least create calibrated frames of all your lights and then the calibrated frames can be used together for stacking. The reason why is that the lights may have different conditions on different nights - including filters. This may not be such an issue with automated scope like a dwarf but definitely true for regular AP setups especially if using  DSLRs because the operating temperature can vary as well. I recently did a multi-night multi-band combined image of the lobster claw nebula. Having a cooled camera makes the dark frames stable but flats can always change and of course the band filter will change the lights as well. So I save calibrated lights for each filter (narrow band or light pollution) for future stacking and combining.

Yes I use calibration frames - the D3 does its own Darks as necessary and I use those - and I make my own Flats and Bias. On the first run of a multi session I save the Masters then use those on subsequent sessions.... Saves a lot of time.

@astrogee

I initially disagreed with your advice on creating new calibration frames for each night (except where a dslr is used or the setup is changed each night). I routinely reuse my masterflats and masterdarks for months at a time.

But I have just discovered that the Dwarf 3 uses an uncooled sensor. I know that there are some who say that modern sensors don't need darks, but I'm not one of them - especially for an uncooled sensor. I now believe that your advice on creating calibrated files for each night is good advice, at least for the Dwarf 3.

Of course this is different to the question of whether files can be stacked each night, but its still useful advice.

JC

@domhenry Hi Dom

I read in your start node that your 9 hours integration takes 2 hours stacking on an M4 Mac. 
if I calculate right, these are 540 frames - a lot, but 2 hours on a M4 seems quite long. But only 6 MPixel per fram, if I‘m right.

I stack sometimes some 700 frames APS-C, so 26 MP, on a M3 max and even with high LNC it‘s only an hour or so. But I admit, I have 64 GB RAM.

• Do you use LNC with high degree and many iterations? This would explain it. (With „high“ I mean 4th degree and >3 iterations). Maybe you try, whether 2nd with 3 iterations gives also smooth local gradients. 
• Is your working directory on the Mac‘s internal SSD, or on a probably much slower external drive? This would make a huge difference. Try to use the internal SSD - for these 540 images you might need a few 100 GB free space. 
• if you have only little RAM, or only allocate little to APP, this is also a reason. Allocate all but some 6 GB to APP. 

Best regards.

Michael 

Posted by: @connor231

↑

I initially disagreed with your advice on creating new calibration frames for each night (except where a dslr is used or the setup is changed each night). I routinely reuse my masterflats and masterdarks for months at a time.

 

By default I create flats every night I go out. Mainly because it's easy to do and I have to re-setup each time at a dark site. In this case, one can never know if the optical train is the same each night so it's safer just to take the flats and apply them. Once the lights are calibrated, they can be use at any time in the future for any stacking and mixing. Just need to know what filter was used.

(So I save my data like so: session date/filter/calibrated lights/ and in the session folder I have a text file (keyword file) where the files name is use to tag the session; for example: dso-sh2-157-lobster-claw-nebula-asi294-ed102cf-0.8x-le-lancaster-20250831-3stars.keyword. This way I can do searches on the tag file. Inside the file I have a legend for abbreviations used in the tag, like "le" (Optolong L-Enhance). I'm mentioning this because you will have to have a good filing system as soon as you start combining nights.)

@astrogee

I agree with everything you said. I'm lucky enough (lazy enough 😉) that I can leave my main setup in my back yard for months at a time. If I was breaking it down every night I would pretty much do the same as you.

JC

@michaelacg I have the LNC at 4 interations, 24GB on the M4 Mini, external fast 4TB SSD. I wouldn't mind the run time for the stacking if it were 1 off - but I like to see how I'm going after each session so its repeated getting bigger each time.. I did a compare with the full stack and the stacking of the integrated sessions - and I think after post processing (in PI) then there is very little difference. I also tried stacking the scope stacked images - also a reasonable outcome and very quick. Just a small JPG

Hi Dom,

By 'LNC at 4 iterations' do you mean with 'degree 1 LNC'....or do you mean 'LNC 4th degree'? As Michael notes a high degree of LNC would be overkill and will greatly increase your processing time.  I'm only using a humble M1 Mac but can stack many hundreds of 250MB calibrated fits (from a 6200MM) in a few hours.... My default for a first look is LNC 1st degree, 3 iterations'

Mabula's advice for normal (non mosaic) images is to 'start with 1st degree and 1 iteration' and only increase if you see any undesirable gradients https://www.astropixelprocessor.com/community/postid/908/

The external SSD will also be slowing down processing a lot. It would be a useful test transferring some to your internal drive and seeing what the actual speed hit is. My guess is that it would be large. It is a pain but I always transfer files onto my internal drive for processing then dump them off into an external SSD for archival purposes when I have finished with a project.

cheers, Jon

Hi @jdwood @domhenry

My experience is that it’s only necessary to assign APP‘s working folder to the internal SSD. The subs can remaining the external SSD, they are only read once. 
As the internal SSD is 4x - 8x faster than the external, the working folder with all the intermediate files makes the difference. 
Best regards.

Michael 

thanks Michael @michaelacg 

That's useful to know. Probably will not help with my workflow where I just use the external SSD as a transport medium to get the data from the scope (PC) to my Mac but I can see that some folk might wish to operate that way.

cheers, Jon