APP's file sort win...
 
Share:
Notifications
Clear all

Mar 28 2026 APP 2.0.0-beta40 will be released in 7 days.

It did take a long time to have the work finished on this and it  will have a major performance boost of 30-50% over 2.0.0-beta39 from calibration to integration. We extensively optimized many critical parts of APP. All has been tested to guarantee correct optimizations. Drizzle and image resampling is much faster for instance, those modules have been completely rewritten. Much less memory usage. LNC 2.0 will be released which works much better and faster than LNC in it's current state. And more, all will be added to the release notes in the coming weeks...

Update on the 2.0.0 release & the full manual

We are getting close to the 2.0.0 stable release and the full manual. The manual will soon become available on the website and also in PDF format. Both versions will be identical and once released, will start to follow the APP release cycle and thus will stay up-to-date to the latest APP version.

Once 2.0.0 is released, the price for APP will increase. Owner's license holders will not need to pay an upgrade fee to use 2.0.0, neither do Renter's license holders.

 

APP's file sort window bogs way down past 1500 subs

7 Posts
3 Users
0 Reactions
1,414 Views
(@readyjetty)
Neutron Star
Joined: 4 years ago
Posts: 79
Topic starter  

I occasionally do some very long multi-night integrations and I was hoping to integrate together as, say, 5 sessions rather than try to integrate each and then integrate those together, because often I don't think the results are as good.  But I found that once I get much above 1500-2000 subs, the program becomes very slow and the UI slows way down as it spends all it's time rebuilding the frame list over and over.  It's so slow, that when you select a file, it'll often select the wrong one, or selections take 10-15 seconds to actually register.   I had a 5 night, 5000 sub job and after registration the program UI just became totally unresponsive unusable on an 8-core 64GB system with 3TB of free SSD space.  It appears the program has something that grows as an N^2 performance issue with respect the number of subs with respect to the UI.

I know APP perhaps wasn't designed with >1500 subs in mind, but with lucky imaging on deep sky targets and more people using shorter exposures on Alt/Az systems, this may become more common.  Perhaps it would be of value if the UI could handle more subs effectively.

 

Also, I noticed it's neither CPU nor SSD disk limited, so it seems there is something very non-linear in the system with respect to the number of subs.

image
image

This topic was modified 2 years ago 3 times by Steven Miller

   
ReplyQuote
(@Anonymous 174)
Joined: 9 years ago
Posts: 5702
 

That many subs is not advised indeed, I think it'll become very limited in either memory usage or in overall complexity. What usually works way better in these cases is to combine smaller amounts and then stack those together. Say 500 subs at a time and then having 10 stacks to integrate with each other.



   
ReplyQuote
(@mabula-admin)
Universe Admin
Joined: 9 years ago
Posts: 5056
 

@readyjetty Hi Steven,

Thank you very much for reporting this. I was not aware, because my largest data set is maybe 300 images. Reason is that if you start to shoot more than 100 images on any subject, things will be much easier if you lower sensor gain and increase exposure time if your setup allows it. Results will be better and you will have faster results and less disk usage/storage as well. I know that shooting short exposures is becoming popular but it still can't beat higher quality subs to start with since the camera sensors sill have read noise above 0 i think. The less read noise in the whole stack the better.

But let's go back to your problem, this is probably an issue with the frame list that can be solved quickly I think. It must be related to how APP reacts to sorting commands indeed.

Would you be willing to upload 1000 or maybe even 2500 subs so I can test this behaviour and implement needed optimizations here?

If so, please upload here:

https://upload.astropixelprocessor.com/

username: uploadData

password: uploadTestData

Please make a folder with your name and issue like: StevenMiller-slowFrameList

and let me know once uploaded.

Thanks,

Mabula



   
ReplyQuote
(@readyjetty)
Neutron Star
Joined: 4 years ago
Posts: 79
Topic starter  

I had backed up my files but I'm reloading a big job to replicate the issue and then I'll either load them up, or find a way to simulate it by simply replicating a smaller set of files, which may make it easier for you to test.  I'll update you when I have a repeatable test.



   
ReplyQuote
(@mabula-admin)
Universe Admin
Joined: 9 years ago
Posts: 5056
 

Hi Steven Miller @readyjetty,

Please do ! Such a big project would really help testing this and optimizing the user interface further 😉

If needed, I will shoot such a big project myself when wheather allows over here, it has been clouds and rain here for many weeks...just terrible.

Mabula



   
ReplyQuote
(@readyjetty)
Neutron Star
Joined: 4 years ago
Posts: 79
Topic starter  

Boy, this is embarrassing.  After running into the problem a few times, now I try to recreate it with up to 5000 lights even, I can't get the performance issue I had.

Yes, it can take up to 30-45 seconds to resort and re-display the file list panel after deselecting a few lights and sorting on another criteria and it would be nice if that was faster, but I'm not seeing the mode where it just slows way down and takes several minutes or gets way behind on mouse clicks which causes the wrong files to be selected.

So my apologies, but I'll have to withdraw my report until I can find a repeatable case.

I don't know the cause of the different behavior, but possibly one element is when I register and it fails on a few files (because they are bad lights or something) and then I start eliminating those and others like them... but again, I'll continue to search for a simple repeatable case.

Oh, also, I found you can create thousands of "lights" just by making copies of your lights directory until you have enough and load up these "identical" lights.  I think this is roughly equivalent.



   
ReplyQuote
(@readyjetty)
Neutron Star
Joined: 4 years ago
Posts: 79
Topic starter  

Well, I was able to reproduce it, but it's not a trivial case, but here is the pattern:

1) Start with a ton of lights, if you really want to see it load up 4000, with perhaps 100 bias, 25 flats.  You don't need 4000 unique lights, I just copied 2000 into another directory and loaded them up too, so they were essentially duplicates.  I imagine you can do this with 8x500 if you have to.

2) I found the problem tended to happen when registration failed because of a few bad subs (usually clouds or the scope was bumped), but I'm not sure you need this, but just to be safe, put in a few lights where registration will fail (perhaps from another target, same camera/bias)

3) After registration fails, my usual task is to sort on various quality aspects and deselect the worst subs with bad attributes:  I sort on number of stars and click on a few down the list until I find ones that look good and deselect the bad ones further up the list.  Then I do that on star shape: sort on shape, click down on list, deselect those worse than the first decent one.   Then on quality, and FWHM and pretty soon I when I double on a file to preview it, things just stop responding or gets behind on clicks and files start selecting and unselecting in mysterious ways.

However, I found if I let the system sit for 1-2 minutes, it eventually responds.

I'm on an 8-Core Ryzen 9 5900HX with 64GB of RAM and a 2TB nVME SSD.

I know this is low priority as it's currently a very niche case, but if you do get a chance to look at performance of managing the file list with a large number of lights, maybe you'll get lucky and may find it's something pretty basic to fix.


This post was modified 2 years ago 2 times by Steven Miller

   
ReplyQuote
Share: