Share:
Notifications
Clear all

Please note our new Downloads page here

2023-01-19: APP 2.0.0-beta13 has been released !

!!! Big performance increase due to optimizations in integration !!!

and upgraded development platform to GraalVM 22.3 based on openJDK19

 

We are very close now to  releasing APP 2.0.0 stable with a complete printable manual...

 

Astro Pixel Processor Windows 64-bit

Astro Pixel Processor macOS Intel 64-bit

Astro Pixel Processor macOS Apple M Silicon 64-bit

Astro Pixel Processor Linux DEB 64-bit

Astro Pixel Processor Linux RPM 64-bit

Can APP use external diskdrives instead the systems HD?


(@pietervorstenbosch)
Hydrogen Atom Customer
Joined: 3 years ago
Posts: 1
Topic starter  

I bought a relative small HD in my Mac because of working in the Cloud.

I've trouble in making enough free memory on my Mac for APP because I don't want to disturb my other work on my computer to much. 

There are nowadays very fast external SSD. Is it possible in the near future to work with APP on an external SSD?

Pieter Vorstenbosch


ReplyQuote
Topic Tags
(@vincent-mod)
Universe Admin
Joined: 6 years ago
Posts: 5218
 

You can already, but it will depend on how well this works. The external drive should have the proper filesystem at least. Memory though is still needed for APP, doing extensive processing requires quite a bit of memory. I wouldn't advice to do a lot of other work during processing.


ReplyQuote
(@wvreeven)
Galaxy Customer
Joined: 4 years ago
Posts: 2090
 

I have a MacBook with more than enough disk space. Still, I use an SSD on my Linux system and the images are written directly to that. Then I disconnect the SSD and connect it to my MackBook and process the images directly on that. Works like a charm. I am not sure if APP uses swap space on the SSD or still on the "local' hard disk though. I guess either Vincent @vincent-mod or Mabula @mabula-admin would be able to tell us.

FWIW I never got fuse working properly on my MackBook, so I formatted the SSD with HFS and disabled journaling on it. Then Linux can mount it and write to it as well.

 

HTH, Wouter


ReplyQuote
(@lvigano)
White Dwarf Customer
Joined: 5 years ago
Posts: 17
 

Ciao,

as far as I noticed it write a *.dat file (not remember if the extension is really *dat or whatewer) in the working directory. 

Me too I'm used to load images from a NAS but set the working directory locally on my pc.

@vincent-mod : do you think a RAM disk set as working dir will speed up the process a little bit ?

Best,

Luciano


ReplyQuote
(@vincent-mod)
Universe Admin
Joined: 6 years ago
Posts: 5218
 

@lvigano I never used a RAM disk to be honest. If that is stable and seen as a regular disk by the system it might work. But I can imagine it being less reliable maybe, would be nice to test and also to check the speed versus SSD.


ReplyQuote
(@lvigano)
White Dwarf Customer
Joined: 5 years ago
Posts: 17
 

ciao @vincent-mod : I'm quite used to it in work environment (and also with PI at home) and it's really stable ... of course you have to move files to a physical HDD before shutting down the system 🙂

As soon I have time i'll power up a VM on AWS or google with 96 gb of memory and test the i\o of the tmpfs.

best,

Luciano


ReplyQuote
(@itarchitectkev)
Main Sequence Star Customer
Joined: 5 years ago
Posts: 94
 

I would avoid RAM disk. A RAM disk is very fast, temporary storage utilising the RAM, taking it away from being used by the CPU. APP is a Java application. Java likes RAM. You are better giving APP more RAM than using a RAM disk for your Working Directory. It doesn't scale well either. I've iterated through many disks and recently upgraded to a 1Tb NVMe due to recent integrations going over 512Gb! That's not tenable for a RAM disk.

Adding a USB 3 disk is a good approach if opening up a PC/Mac isn't feasible. Adding NVMe is better if you can internally. I'd avoid USB2.0 based SSDs, although obviously would still work - just not utilising the full speed of SSD, choked by USB 2's bandwidth.

Overall though, whilst disk speed is noticeable when you go from spinning disks to solid state, the biggest bottleneck becomes the CPU - processing images is CPU intensive, where RAM for the processing is your friend here.


ReplyQuote
(@vincent-mod)
Universe Admin
Joined: 6 years ago
Posts: 5218
 

Ah yes, I looked over the scaling part, that is a good point indeed.


ReplyQuote
(@lvigano)
White Dwarf Customer
Joined: 5 years ago
Posts: 17
 

Ciao,

finally I managed to create a VM on AWS with these specifications:

- m5.8xlarge with 128G RAM and 32 core Intel(R) Xeon(R) Platinum 8175M CPU @ 2.50GHz plus SDD storage space;

- it's a basic EC2 device, so is not using 2nd or 3rd Xeon generation

- Debian 10 with KDE Plasma and XRDP (linux because it's cheaper and doesn't have fragmentation problems)

- it cost about 1,5$/hour

Basically the output of the test are the following (w\r a 5G file using linux "fio"):

SSD

TEST: Laying out IO file (1 file / 5120MiB)
Jobs: 1 (f=1): [m(1)][13.1%][r=4572KiB/s,w=4656KiB/s][r=1143,w=1164 IOPS][eta 00m:53s]
Jobs: 1 (f=1): [m(1)][23.0%][r=4164KiB/s,w=3864KiB/s][r=1041,w=966 IOPS][eta 00m:47s]
Jobs: 1 (f=1): [m(1)][32.8%][r=4040KiB/s,w=3980KiB/s][r=1010,w=995 IOPS][eta 00m:41s]
Jobs: 1 (f=1): [m(1)][42.6%][r=3672KiB/s,w=3836KiB/s][r=918,w=959 IOPS][eta 00m:35s]
Jobs: 1 (f=1): [m(1)][52.5%][r=3632KiB/s,w=3520KiB/s][r=908,w=880 IOPS][eta 00m:29s]
Jobs: 1 (f=1): [m(1)][62.3%][r=3804KiB/s,w=3724KiB/s][r=951,w=931 IOPS][eta 00m:23s]
Jobs: 1 (f=1): [m(1)][72.1%][r=4036KiB/s,w=3396KiB/s][r=1009,w=849 IOPS][eta 00m:17s]
Jobs: 1 (f=1): [m(1)][82.0%][r=3612KiB/s,w=3716KiB/s][r=903,w=929 IOPS][eta 00m:11s]
Jobs: 1 (f=1): [m(1)][91.8%][r=3576KiB/s,w=3576KiB/s][r=894,w=894 IOPS][eta 00m:05s]
Jobs: 1 (f=0): [f(1)][100.0%][r=3512KiB/s,w=3604KiB/s][r=878,w=901 IOPS][eta 00m:00s]

RAM

TEST: Laying out IO file (1 file / 5120MiB)
Jobs: 1 (f=1): [m(1)][80.0%][r=568MiB/s,w=567MiB/s][r=145k,w=145k IOPS][eta 00m:02s]
Jobs: 1 (f=1): [m(1)][100.0%][r=568MiB/s,w=568MiB/s][r=145k,w=145k IOPS][eta 00m:00s]

The tmpfs was about 40G, so still plenty of RAM to be used by the system.

@itarchitectkev and @vincent-mod : I agree with you about the scalability issue when creating a tmpfs but please consider that a NVMe is not a really cheap option if you don't have a MB withe NVMe support. Moreover, as per the SDD disk, you still have problems with the maximum write cycle you can perform (for example Samsung EVO 970 is guarantee for 5 years or 150TBW)

In my humble opinion, I think that there is not a single solution which fit for every case :-); in my case - with a maximum of 20G required for integration - I think that the tmpfs will absolve it's job in a very good way and, let me say, a VM in AWS it's really the win choice!

Moreover it will be very interesting to see the results of the same fio command on a NVMe HHD ... but I don't have it 🙂

 

best,

Luciano


ReplyQuote
(@vincent-mod)
Universe Admin
Joined: 6 years ago
Posts: 5218
 

True, there is no single solution so it's very nice to see your efforts here, thanks for sharing!


ReplyQuote
(@dem0niac)
Hydrogen Atom Customer
Joined: 2 years ago
Posts: 3
 

Good day, guys!

Please, give me a solution for my problem. I have a pretty little SSD on my MacBook Air - just 128 Gb. And I have a TimeCapsule external drive with 2Tb which I use for most of my work. It is connected via WiFi, but APP doesn't see it when I try to choose it as a working directory, thus the number of photos to work with is limited within 70-90.

How can I use this external drive by WiFi connection?

Thank you very much!


ReplyQuote
(@vincent-mod)
Universe Admin
Joined: 6 years ago
Posts: 5218
 

I would not advice to use wifi for the working directory, that speed is very slow compared to an SSD locally and APP might take forever on a wireless connection. Maybe an external SSD is a better idea?


Stephen T liked
ReplyQuote
(@dem0niac)
Hydrogen Atom Customer
Joined: 2 years ago
Posts: 3
 

@vincent-mod it seems to be so 🙂

Thank you, I'll look at external SSD's assortment!


ReplyQuote
(@vincent-mod)
Universe Admin
Joined: 6 years ago
Posts: 5218
 

If you have USB-C or something like USB3 that would be preferred as well, look for a fast connection to that SSD as well.


Stephen T liked
ReplyQuote
(@dem0niac)
Hydrogen Atom Customer
Joined: 2 years ago
Posts: 3
 

@vincent-mod

Yep, got it! Many thanks)


ReplyQuote
Share: