UPDATE FROM MABULA
I have had a very rough 2 months unfortunately health wise. I was struck 3x in a row with bacterial infections. The second infection occurred after a routine hospital checkup and I became very sick. I had to rest and take a lot of antibiotics. Once I was recovering and restarting work 1 month ago, I again became sick. The infection was not yet gone, so even more antibiotics and rest was needed. Needless to say, it took a lot of my energy and I needed a lot of rest.
Finally, the infection is really gone and my energy is coming back step-by-step now and I have started work again. I am terribly sorry to have kept you waiting for support. I will address all outstanding questions and e-mails step-by step and will be on the forum daily from today.
APP 2.0.0-beta47 will come soon as well, a lot of work was already completed before I became very ill, so the release is also nearly ready for you. Beta47 will be much faster actually. Many workflows will be more than 2x faster, mosaics can even be 10x faster than before because registration really received a major boost... all compared to beta46. Before I release it, I will make sure that everything is working properly and then I will release it.
I'm having real problems with both disc space and impossibly long processing times. My laptop isn't the latest, but it's not ancient either. 500GB SSD, 8GB RAM, i7-5600U CPU @ 2.60GHz. I've allocated 7GB RAM although only 4GB gets used typically. The default CPU usage already causes CPU overheating, so I cant increase that.
Admittedly I have a lot of subs - 1100 on the project - but I left it overnight and it still didn't complete and then ran out of disc space. I think it needed 80GB! I cleared out 100GB of space but it virtually just stopped. And this is with darks only, no flats!
Right now, I'm doing LUM only, and hoping to do the same with RG and B, and then register the stacks together. But even with just 550 LUM files, I'm only 42% done after 2.5 hours.
I've used APP in the past, typically with fewer files I guess, and found it excellent. I have now retired and able to spend more time getting subs - but unfortunately APP seems virtually unusable.
I'm using the default settings BTW. Maybe skipping BG neutralization would help?
Any suggestions please? (Apart from reverting to CCD and fewer subs or new laptop.)
Thanks, Tom
@tommohawk Hi Tom. What version of APP are you using?
Normally with very large data sets like yours it helps to split up the data set in batches of, say, 100 files. When done, the integration results per filter can be loaded as lights (no need to load calibration files) and then integrated together to get the final result. The final results per filter can then be loaded as lights (again no calibration files) and registered against each other for further processing.
HTH, Wouter
Hi Wouter - thanks for that.
Trouble is I have 1100 subs so thats 11 runs minimum. It just seems incredibly slow. By comparison, I've just done stacks in DSS and SIRIL and even with 64 flats and dark flats included it only took about 60 mins for 550 LUMs.
You might say that APP gives better more precise results - but TBH if anything the SIRIL result looks better (smaller stars) though to be fair that stack had flats applied.
Still trying to find a way to solve this. One thing I realise is that I use large numbers of calibration files for Darks, Flats and Dark Flats - 64 of each. I don't yet have masters so its likely calibration is slowing things down. (BTW I don't have significant dust bunnies, so I tend to use library flats/dark flats)
So I am now doing a run with only a couple of lights per channel, purely to get masters for darks, flats, dark flats, and will then try and use these masters for a run including the full set of lights.
One question. I think I read somewhere that when using masters, I should use MD, and MF only, no MDF. Is that correct? If so I cant help wonder what would happen if you did include the MDF (in error) I would think that is what most people would do, and it might throw the calibration if the MF "includes" the MDF.
@tommohawk If you want to create the master calibration files then you only need to load the raw calibration data. No lights needed. Then go to tab 2 and click the calibrate button. APP will automatically save the masters for you.
Indeed, only MD and MF are needed. MDF is used to calibrate the raw flats to be able to create the MF. If you load the MDF together with the MD and MF then APP will ignore the MDF.
Thanks Wouter. So - some progress, but raises a few new queries.
First - by first creating MD MF and MDFs from sets of 64 cal files using just a few lights, and then running with these masters I have successfully processed 572 lights in 90 minutes I believe. So that is at least acceptable - BUT still way slower than alternative software!
But new queries arise.
1. I keep getting calibration warning messages re the flats BUT the flats and dark flats definitely match. I checked the headers carefully and exposure times and gain all correct. Interestingly, these flats were done with NINA wizard, and the exp times are given to many decimal places eg 0.818168748807657 secs. (But still match the DFs exactly) Maybe that number of decimal places is an issue.
2. If the MF is all that's required, ie no MDF, why is there an option in the masters section to upload a MDF. That seems wrong?
In fact I loaded the MDF (having not yet seen your reply) and calibration is very poor - lots of vignetting. Not sure why this should be - the MDF in itself should not cause this I'd have thought.
3. Next step would be to repeat, but without the MDF as per your suggestion (but still begs the question why is the MDF option there?)
However, I know from experience if I simply remove the MDF and do integration again I will get lots of issues. Somehow APP remembers values from the first run.
But to remove all the lights and cal files and reload (or close APP and start again which is what I've had to do in the past), is a right pain. So how do I rerun, eg with a minor change like in this case no MDF (or maybe with no background neutralisation or whatever) without having to start afresh.
Lots of questions.....
07/04/2022 11:14 am. PS I briefly posted that I had made an error in using unbinned flats. But looking again at the masters I can see I did in fact use correct flats. So I deleted that post. I'll do some ongoing tests on this. If you could maybe deal with queries 2 and 3 please. Thanks.
Hi @tommohawk,
1) Which version are you using? Please make sure it is the latest, one, the initial 1.083 release had a bug in the calibration engine causing problems like you indicate. The number of decimals definitely is not the issue.
2+3) The MDF - MasterDarkFlat is very important if you still need to make a MasterFlat. You need either the MDF-MasterDarkFlat or a MB-MasterBias to make a functioning MasterFlat. The MDF and/or MB are subtracted from the flats, then flats get normalized and then integrated into a MasterFlat. Once the MasterFlat is created, yes, the MDF is no longer needed so you don't need to load it then, if you would it would simply get ignored like @wvreeven indicated 😉 So a button to load a MDF is rather essential and definitely not wrong. A MDF can be used from a library to be used on many projects, where MFs need to be created per session if you want optimal results.
But to remove all the lights and cal files and reload (or close APP and start again which is what I've had to do in the past), is a right pain. So how do I rerun, eg with a minor change like in this case no MDF (or maybe with no background neutralisation or whatever) without having to start afresh.
Well, if you have the MasterFlat and MasterDark/MasterBias (and BPM) loaded, then all you need to do is, clear all lights and calibration subs. Load the new batch of lights and it definitely should work if the MasterDark/MasterBias and MasterFlat do match the new set of lights properly. Background neutralization off will do nothing in terms of speed or RAM usage. A function like LNC and MBB in integration will have a big impact on speed and RAM and HDD usage.
Please let me know if this helps 😉
Mabula
Hi Mabula. Yes it helps!
Version - Wouter asked that and I overlooked it, sorry. I have version 1.083.4.
I realise now that of course you may want to reuse the MDF, and probably the MD too, but use separate new flats (although of course the exposure time of the new flats would have to match that of the MDF.) So that makes sense.
I'm pretty sure I had a problem where I cleared everything, and did a re-run, but got odd results. Maybe I just got muddled - a few late nights and maybe not thinking straight.
Tried the star reduction tool and that seems nice. Might be good to allow saving of settings here and also a return to default button?
One of the great things about APP is that you can load everything, and just let it run - registration and stacking of all channels is done.
But even with masters as outlined above it took 7 hours to process - so if something could be done to speed it up it would be great!
Regards
Tom
Might be good to allow saving of settings here
Saving settings will be part of the next release.
But even with masters as outlined above it took 7 hours to process - so if something could be done to speed it up it would be great!
Mabula is working on that as well so stay tuned!
Great stuff, thanks Wouter.
