Save list and settings

Oh and maybe one more thing would be cool (not sure if it belongs in RFC): a log next to the stacked image with the settings and the contents of the list view would be awesome

Best Frank

Thank you Frank for this request,

A log of the integration and a file with the file list details would be no problem to create. I will also more and more information of the integration in the FITS header, so I am planning to add all relative integration, normalization, registration, star analysis settings in the FITS header.

Will work on this 😉

Mabula

Awesome, thanks! With the log it also would be easier to share the exact settings here in case of a question. Anyways I am still amazed with the quality of the results of AstroPixelProcessor, even though I am still learning about all the possible settings! Great job Mabula 

Posted by: Frank

Awesome, thanks! With the log it also would be easier to share the exact settings here in case of a question. Anyways I am still amazed with the quality of the results of AstroPixelProcessor, even though I am still learning about all the possible settings! Great job Mabula 

Yes indeed, I agree, it's a good request so I will be happy to implement this 😉

Thank you Frank for the very nice compliment, highly appreciated..

I am very new to APP but it is now my GOTO calibration and integration tool - thank you for this as I have literally struggled with other processing software!  I echo Frank's remarks.

I was about to ask a similar question when I stumbled upon this thread, now some 8 months since the last post.  It would really be nice to able to save my settings in all modules from 2) Calibration to 6) Integration as my Default.  I may well be missing something, so please let me know if you have already implemented this feature.

Many thanks!

@cartgate,

Thank you for your nice comments and welcome to the forum 😉 

The ability to save the settings is something that will be implemented in a future version. Currently I have some more pressing priorities to implement, but it will certainly come.

Kind regards,

Mabula

Posted by: Mabula Haverkamp - Admin

 

The ability to save the settings is something that will be implemented in a future version. Currently I have some more pressing priorities to implement, but it will certainly come.

 

Many thanks!  I look forward to this implementation in the near future.

Best regards, Kwan

Same for me, i am working on a 72 panel Mosaic and that needs many sessions to fine tune all the settings for the best result. That takes many days and i now have to enter all the settings from a list by hand every session. There must be a better way......

I'd like to add my voice to the request to save file list and settings.  

Thank you

Hi

I second this feature request!

Thanks

I would also love the ability to save a project, its a bit difficult to work without it at the moment.

Other than that far the software looks amazing!

Woah.  Do I really have to add all 365 lights, 260 flats, 252 darkflats and 60 darks split over three filters and six nights of imaging again BY HAND if I need to restart the program?  That's insane!  Surely this should be the highest priority possible to implement, no?

The reason I need to restart is to change the configuration -- by default APP only uses 2Gb of ram, and it turns out that this isn't enough to integrate my files.  My machine has 16Gb but it configured itself to only use 2Gb.  So I've gone through hours of configuration and processing to get to stage 6, integration, only to find out that the default amount of memory is too low and I need to restart.  And because I have to restart I have to start everything from scratch, including painstakingly adding all of the files to the correct filters and sessions...

This makes the software a non-starter for any non-trivial project. I know I'm only on a trial of the software, but that means that I'm a future income stream.  PLEASE fix this.

It's definitely problematic when you have a big set of data with all these variables in it, we agree. Saving settings is high on the list, just like other much requested features are ofcourse. This is the problem sometimes with 1 developer, it takes time and many people have their own issue which need top priority. 😉 But next releases have a lot of those requests in them, when the saving of options is there I don't know myself, I just know it's on his top 5 shortlist.

I would btw make the master darks, bias and flats first and save those. Having too many frames in a large project will make the system use too much memory. It's best to split some up when it's getting really big.

I would btw make the master darks, bias and flats first and save those. Having too many frames in a large project will make the system use too much memory. It's best to split some up when it's getting really big.

Indeed, but making the masters is just the "easy" part; the issue is with the multi-sessions / multi-channels combined.

Just an idea - why not assuming that a session corresponds to a date/time range say from 6PM to 8AM for instance, and let the system automatically create the sessions based on those ranges?

my 2 cents.

That's an interesting idea, I'll see what Mabula thinks of that. Thanks!

Indeed, the real issue is that the sessions and channels are lost.  I have a bunch of masters now from my previous work but no knowledge as to which master was for which session or which channel (might be inside the header information, haven't bothered to check) and more importantly APP has know knowledge of my filter channels or nighttime sessions anyway.  So I basically have to start everything from scratch.

IF you don't like the idea of assuming a session corresponds to a date/time range, might I suggest the possibility of a "known directory structure"?  E.g. you specify something like \session\channel\filetype, and people put their data into that directory structure and point APP at it.  My data might then be at:

c:\astro\night1\ha\light
c:\astro\night1\ha\flat
c:\astro\night1\ha\darkflat
c:\astro\night1\ha\dark
c:\astro\night1\sii\light
c:\astro\night1\sii\flat
c:\astro\night1\sii\darkflat
c:\astro\night1\sii\dark
c:\astro\night1\oiii\light
c:\astro\night1\oiii\flat
c:\astro\night1\oiii\darkflat
c:\astro\night1\sii\dark
c:\astro\night2\ha\light
c:\astro\night2\ha\flat
...

It would be trivial for me to move my data into this structure (or whatever structure you prefer), and then I point APP at c:\astro and it will always know about the sessions and channels when it starts up (and perhaps could even auto-load masters from the directory and already understand them etc.

It seems like this would be a relatively trivial piece of work which would really enhance quality of life for users who want to use APP for non-trivial amounts of data...

-simon

Posted by: @szymon

Indeed, the real issue is that the sessions and channels are lost.  I have a bunch of masters now from my previous work but no knowledge as to which master was for which session or which channel (might be inside the header information, haven't bothered to check) and more importantly APP has know knowledge of my filter channels or nighttime sessions anyway.  So I basically have to start everything from scratch.

I don't know what capture software you use, but SGP does embed the date/time and filter used at capture time in the FITS header, with the DATE-LOC, DATE-OBS and FILTER keywords. APP does read the FILTER keyword to automatically assign the subframe to the right filter group.

So by reading the date/time info from the FITS header, the notion of session could be created.

Only issue I see with this approach are FLATS, that must be assigned to the right light session(s), ie the lights sessions you want to apply those flats to. But this is difficult to automate as every operator has his own strategy for flats.

Again, these are just ideas to improve automation, not sure we'll ever have a definite answer to this question.

Posted by: @rodolgo

Posted by: @szymon

Indeed, the real issue is that the sessions and channels are lost.  I have a bunch of masters now from my previous work but no knowledge as to which master was for which session or which channel (might be inside the header information, haven't bothered to check) and more importantly APP has know knowledge of my filter channels or nighttime sessions anyway.  So I basically have to start everything from scratch.

I don't know what capture software you use, but SGP does embed the date/time and filter used at capture time in the FITS header, with the DATE-LOC, DATE-OBS and FILTER keywords. APP does read the FILTER keyword to automatically assign the subframe to the right filter group.

So by reading the date/time info from the FITS header, the notion of session could be created.

Only issue I see with this approach are FLATS, that must be assigned to the right light session(s), ie the lights sessions you want to apply those flats to. But this is difficult to automate as every operator has his own strategy for flats.

Again, these are just ideas to improve automation, not sure we'll ever have a definite answer to this question.

Actually my capture software (N.I.N.A. -- a superb piece of free software) does embed filter information into the header (either FITS or XISF) along with date/time and a whole bunch of other information.  However other people may not be so lucky, which is why I proposed a folder structure.  I'm not saying your approach is better or worse, I just suggested an alternative.  I'll be very happy with either of these solutions, or even, frankly, a simple "save file list as text file" like old DSS uses.  Anything, just to make sure I don't have to keep reselecting everything. 🙂

(As an aside -- my local astro society is working on a collaborative project of M33.  I think we have something in the region of a quarter terabyte of data so far, in literally thousands of frames.  I was going to have a go at integrating it using APP, but I think I'll leave it to the guys who own PI, at least until I can save a simple file list in APP!).

N.I.N.A -- http://nighttime-imaging.eu

Where do we stand on this topic ?

This is an old request, was it done and hidden somewhere ?  I have looked all over for it, sounds simple to me but then again what do I know ?

Thanks

Claude

Posted by: @croy1962

Where do we stand on this topic ?

This is an old request, was it done and hidden somewhere ?  I have looked all over for it, sounds simple to me but then again what do I know ?

 

Thanks

Claude

It was February 2018 that I raised this matter.....over to Mabula and Vincent.

Thanks and all the best, Kwan

@Mabula & @Vincent

I have 2 questions which are more or less add-on/related to this section:

Situation 1:

- When APP Crashes (seldom, but it happens)

- When I want to continue processing or retest integration with different parameters

=> What is the best workflow to restart? Because today you must input all data(folders) again

What I do is saving the normalized frames (which takes a lot of space). Restart some days later and

reload the normalized frames as lights and cross autodetect masters and integration (although I think that in this stage it doesn't matter) .... an advantage is that all processed "normalized lights" are in one directory bin?

=> But this should be in 1.083. which is coming up soon?

Situation 2:

(I read something higher up in this mail which I think is a wrong workflow)

Suppose you work on a project and want to integrate different layouts, sizes, from different devices ...

=> Issue because when you treat them as different sessions, they ALL use 1 BPM!

So, in fact, I presume you must integrate every source as a whole and store it as an integration

When all that work is done, restart APP, reload now only all integrations and start like threating them as lights

I guess higher quality may be obtained if we could mark an option that every session has to be treated in that session (ie, has its own BPM and correction files).

Now corrections files (bias, dark, flat, darkflat) are treated as if they are originated from the same device, only at different temperatures, environment, or filters ....

Is my assumption correct?

What is the best practice?

Maybe Situation 1 and 2 are stuff to be placed under tutorial?

And the key question to all: when will a "configuration list of what was uploaded" be available in cases like a crash or restart

Hi Jan,

1) APP doesn't have settings you can save yet unfortunately, that will come in a later version. The next beta version will have the possibility to skip steps, when you already did them indeed.

2) A BPM is totally independent of a session, it's basically a noise readout of hot-pixels of your sensor and non-destructive. So it's best to make one using a long dark, it should have as many bad pixels as you'd encounter then and will correct those. If you use shorter exposures with less bad pixels, the BPM won't influence that in a bad way. So you can use it for that sensor for all sessions and for a pretty long time.

Other calibration data is session dependent. In the sense that flats of course are pretty unique per session (unless you have a fixed setup), darks and bias data needs to have the same ISO/Gain as the data. If you have a library of those with the correct values, you can use them for a while as well.

@vincent-mod, thx, so

1) if we saved calibrated, or registered or normalized frames in the future, we should be able to depart from that?
2) agree, but question was: "from different devices ..." and each device will have a specific BPM
I saw that every session uses the same BPM, further calibration data (darks, flats etc are indeed session independent and yes I've a library set of those :-))