What do you get when saving normalized lights?

Hi Jay ( @jay ),

Thank you for your question and welcome to the forum 😉

The possibility to save your frames after registration or normalization is there for several reasons.

1. Yes, you can save them for continuation later, in that case you can set the registration mode to no registration and normalization mode to none.
2. Or it you want to import the images into another application. Some users register their images on the stars and normalize the data in APP before importing the data into another application for comet registration. (a feature that will be added to APP as well).
3. In some workflows for multi-channel data, you can register and normalize the channels and save them before importing them into the RGB combine tool to make any sort of compositie.

The registered and normalized frames indeed have everthing applied. The calibration, registration parameters and normalization parameters.

But normally, you don't need to save the data after normalization, just integrate with settings in 6). After integration, just alter some settings in 6) and you can directly integrate using those settings.

Let me know if this answers your question 😉 or if you have any other questions.

Kind regards,

Mabula

OK that makes sense, thank you.

Hi Dr Haverkamp,

I want to first thank you for a very nice tool to process my astro photos.  I tried the trial version and followed some of your tutorials, and after getting much better results than with other software, I decided to purchase.

I did run into one issue though, and it's kinda related to this topic, so I figured to post here.  Incidentally, I have no idea on how to start a new topic, so any hints to do that would be greatly appreciated.

My issue is this.  I was integrating 480 lights the other day, and the process went well all through normalization, then failed with out of memory during integration.  So I saved the normalized files, then shut down and restarted AP with more memory.  Then I tried to load the normalized lights as lights, and after selecting them and pressing ok, the program just hangs with the file dialog box still open.  Hours later and it's still stuck.

So my first question is this.  Is this the correct flow when continuing an interrupted process?  Saving the normalized frames then reloading them back as lights?  Will the program recognized them as normalized?  I noticed you mentioned above to set registration to none and normalization to none.  

Second question, is there a limit to how many frames I can select at once?  When I initially loaded the lights they were in different folders, so I never loaded more than 100 at a time.

Thank you in advance.

Daniel

Posted by: dciobota

Hi Dr Haverkamp,

I want to first thank you for a very nice tool to process my astro photos.  I tried the trial version and followed some of your tutorials, and after getting much better results than with other software, I decided to purchase.

I did run into one issue though, and it's kinda related to this topic, so I figured to post here.  Incidentally, I have no idea on how to start a new topic, so any hints to do that would be greatly appreciated.

My issue is this.  I was integrating 480 lights the other day, and the process went well all through normalization, then failed with out of memory during integration.  So I saved the normalized files, then shut down and restarted AP with more memory.  Then I tried to load the normalized lights as lights, and after selecting them and pressing ok, the program just hangs with the file dialog box still open.  Hours later and it's still stuck.

So my first question is this.  Is this the correct flow when continuing an interrupted process?  Saving the normalized frames then reloading them back as lights?  Will the program recognized them as normalized?  I noticed you mentioned above to set registration to none and normalization to none.  

At the moment, continuing with normalized and registered frames is not optimal, since you will need to do star analysis again which shouldn't be needed. I will work to make sure that in the future you will be able to simply load the normalized frames and continue directly with integratione 😉 Currently, saving normalized and registered frames is needed if you want to bring the data to another application for some reason or another.

If you process a next set of data, APP should be able to continue with integration directly off course with the increased available memory.

"Hours later and it's still stuck."

Were you saving them perhaps as TIFF files, loading TIFFs is not very efficient at the moment. Please save the data as FITS, that will be much faster. If it were FITS files, then the behaviour is abnormal. Does it happen each time that you try to load the normalized data? If so, please send some files to me, so I can have a look at what is exactly happening.

Second question, is there a limit to how many frames I can select at once?  When I initially loaded the lights they were in different folders, so I never loaded more than 100 at a time.

No, not really, I haven't implemented limits. When images are loaded, all metadata is checked, so if you load 100s of frames it can take some time. I know some users are loading more than 1000 frames at once so that should be possible.

Thank you in advance.

Daniel

Kind regards,

Mabula

Thank you for the quick reply, much appreciated.

Can you explain what you mean by this:

"If you process a next set of data, APP should be able to continue with integration directly off course with the increased available memory."

So I restarted APP with increased memory, and I set my workspace to exactly the same folder I had before it crashed.  How do I proceed directly with integration?  Do I reload all the lights/darks/flats and then just click on integrate?  Will it not go through the other steps?  I'm trying to understand how APP "remembers" what state I left my project last time.  Sometimes I don't have a chance to finish my work for days or weeks, and I would like to be able to save some state of my work so I can continue where I left off.

As to the normalized files, they were saved as fits, just like the original lights.  And it was literally 8 hours or more (I left the dialog box open overnight to see if it finished reading the normalized files) and it was still stuck with the Open button pressed and nothing else.  I had to use Task Manager to shut it off.

Thanks again for the help.

Daniel

So is there an answer?  Or do I have to do everything from the beginning?  I see so many posts about people adding more frames to something that has already been integrated, but I have no idea how that works.  Can someone explain to me or at least point to a link that describes how APP preserves the information, so if I quit the program at some step I can bring it up and continue where I left off?  This is something not at all clear to me.

Thanks in advance.

Daniel

Yes, you can simply take a fully integrated result and add more frames to it, if it can detect enough stars those new frames will be registered to the fully integrated result and a new result pops out. It's great for combining data from various sources. In your case I'm actually not sure, but on a small test I just did it indeed goes through the registration process again. Maybe you could try a small, but still rather big subset and see if it goes faster as all are already registered? I think the option to save them is just for processing purposes in another application, when desired.

Thanks Vincent.  I decided to rerun everything by selecting integrate, and it looks like it did rerun through all the steps, but some of them like registration did seem faster, so maybe it did detect the fact the frames were already registered.  It's doing integrate now, so I'm crossing fingers it won't run out of memory.

Interesting, did you update APP before this rerun? I'm crossing my fingers as well!

I just checked, and my version is 1.065.  I just noticed that there are newer versions, maybe I should update.

But, I am very happy to say, it successfully finished.  It did take almost 15 hours in total to run, but it was worth it.  🙂

Now I did make some mistakes (like forgetting to use a rejection algorithm) and my processing skills are not the best, but the end result is one of the best images of M31 I have been able to get, ever.  And this is from a Bortle 7-8 zone, with no filters.  The light pollution tools is light years ahead of anything else I have ever tried (like a certain app whose initials resemble the famous ratio of a circle diameter to it's perimeter).  This alone has been worth the price.

Now, to practice some more.  

It might be wise to update indeed, although not sure if that has anything to do with what you were running into. Glad it worked! Do post your result here when you're finished, love to see it! As for post-processing, you can always ask away here as well.

Thanks Vincent.  Well, here is the finished result:

https://photos.app.goo.gl/zyTANLHL2QVfaiGXA

And this was the original after integration and some cropping.  I just applied the ddp settings on the right, like stretch and background neutralization before I saved, so you can see the LP I had to deal with.  Plus mismatched flats in some cases (my fault):

https://photos.app.goo.gl/rsJCfZfqrFipjKHV7

When I tried to process this last year with DSS and PI I pretty much gave up.  So I gave APP the toughest test with this, not expecting to do much better.  Boy was I surprised.  I do admit I practiced on two other images of mine first in removing LP according to the video tutorial Dr. Haverkamp posted on the site.  Very helpful and intuitive.

I will go ahead and update as well.

I think that's looking very nice, you're getting the blue out nicely for instance which is always pretty difficult. I couldn't do that back when I did my M31 (which wasn't with APP yet).

That's a beautiful image Vincent, much deeper than mine.  I can't wait until I get to try M31 again next year, this time from my favorite dark site.  Hope the weather cooperates.

This wasn't from a dark side and maybe because of that the blue isn't as prominent, I couldn't get that to pop out nicely. But overall I like it indeed, one of my best so far. 😉

Posted by: dciobota

Thank you for the quick reply, much appreciated.

Can you explain what you mean by this:

"If you process a next set of data, APP should be able to continue with integration directly off course with the increased available memory."

So I restarted APP with increased memory, and I set my workspace to exactly the same folder I had before it crashed.  How do I proceed directly with integration?  Do I reload all the lights/darks/flats and then just click on integrate?  Will it not go through the other steps?  I'm trying to understand how APP "remembers" what state I left my project last time.  Sometimes I don't have a chance to finish my work for days or weeks, and I would like to be able to save some state of my work so I can continue where I left off.

As to the normalized files, they were saved as fits, just like the original lights.  And it was literally 8 hours or more (I left the dialog box open overnight to see if it finished reading the normalized files) and it was still stuck with the Open button pressed and nothing else.  I had to use Task Manager to shut it off.

Thanks again for the help.

 

Daniel

 

Hi @dciobota,

Apologies for my late response on your questions. I think I created some confusion in my previous answer. Let me try to clarify here:

"So I restarted APP with increased memory, and I set my workspace to exactly the same folder I had before it crashed.  How do I proceed directly with integration?  Do I reload all the lights/darks/flats and then just click on integrate?  Will it not go through the other steps?  I'm trying to understand how APP "remembers" what state I left my project last time.  Sometimes I don't have a chance to finish my work for days or weeks, and I would like to be able to save some state of my work so I can continue where I left off."

Once you run the process with the increased amount of RAM it should be able to continue and finish. Once APP runs out of memory, I am afraid nothing is stored, so you need to run the whole process again with an increased amount of RAM. Sorry for the misconfusion that I created ;-(

The actual ability to load/save a project will be implemented in due time. It's on my development list 😉

Great M31 ! Looks great to me 😉

Mabula

Posted by: dciobota

Thanks Vincent.  Well, here is the finished result:

https://photos.app.goo.gl/zyTANLHL2QVfaiGXA

And this was the original after integration and some cropping.  I just applied the ddp settings on the right, like stretch and background neutralization before I saved, so you can see the LP I had to deal with.  Plus mismatched flats in some cases (my fault):

https://photos.app.goo.gl/rsJCfZfqrFipjKHV7

When I tried to process this last year with DSS and PI I pretty much gave up.  So I gave APP the toughest test with this, not expecting to do much better.  Boy was I surprised.  I do admit I practiced on two other images of mine first in removing LP according to the video tutorial Dr. Haverkamp posted on the site.  Very helpful and intuitive.

I will go ahead and update as well.

 

Excellent @dciobota

Very nice Andromeda 😉 !

Mabula

Thank you very much! 🙂