Read/Write Error

@garycphoto Hi Gary, can you make sure that you are using 1.083-2 for MacOS?

Thanks, Wouter

@wvreeven thank you for the reply. I’m using that version. I ran it again from another hard drive and it worked. Now I’m only getting the critical warning ⚠️ I’m using all the data I used before and this has only started happening. I’m pretty sure my lights and calibration frames are good as it ran already once. Don’t know why this is happening now. 

@garycphoto Gary, APP is very I/O intensive and will have issues with data stored on external HDDs. External SSDs do work well.

Regarding the warning, this is new since version 1.083. The popup explains exactly what is going on. If you want we can have a look at your data to help you find out what's going on. 

@wvreeven thank you. I’ve never had issues up until now. I just think the disk permissions changed when I unplugged the hard drive. That park works fine off my backup.

That would be great. Can you send me the upload link please. Also, do you want all the data or just a few of each? 

Uploading now under garycphoto. Thank you!

@garycphoto Thanks for uploading the data Gary. I have had a look at found several issues:

• The bias frames all have very high temperatures, starting at 25º C and ending at 8ºC. All other data have been shot at -10ºC so make sure to shoot bias at -10ºC as well.
• The flats are not very good. See the screenshot below. The blue channel is too low and the peak of the histogram doesn't even come loose from the left. Green, on the other hand, is very nice. Red is not bad but has a very strange tail to the left. See the screenshot below.
• When I leave out bias and flats and integrate with only darks and lights, I still get the popup but not when I integrate light only. This seems to indicate that the lights are underexposed.

Are the lights taken with a filter like L-eNhace or L-eXtreme? If yes, you need to expose longer. For L-eNhance probably at least 180 sec and for L-eXtreme at least 300 sec. Those filters have narrow to very narrow passbands and therefore require long exposure times. The warning was introduced in APP 1.083 which may explain why you didn't see this before. But I am fairly sure that when you process older data with 1.083, you'll get the same popup.

Regarding the flats, what light source do you use? It doesn't seem to be compatible with your camera or camera/filter combination. Having said that, ASIAir fixes the white balance of ZWO cameras and I am not sure if that can be adjusted. It may help to try and set the white balance of the B channel to a higher value. Remember that if you do that, you will need to take new calibration frames and lights all with the same white balance, temperature, gain and offset.

Here is the screenshot of a flat:

@wvreeven Wouter! Thank you so much for the in-depth response. I use an L-Extreme and have never shot longer than 180s with no problems. I don’t really want to shoot 300s exposures due to the nature of my setup. 

I will reshoot my bias and flats. I just find it unusual that I could run my first session with no problems and then when I ran the second session this popup began.

What if I downgrade to an older version for now? This popup has only become an issue in the last week and it’s a bit frustrating tbh. 

Posted by: @garycphoto

I just find it unusual that I could run my first session with no problems and then when I ran the second session this popup began.

Can you elaborate on the sessions? What was the first session and what the second? What data were used for both sessions?

Posted by: @garycphoto

What if I downgrade to an older version for now? This popup has only become an issue in the last week and it’s a bit frustrating tbh. 

The popup is merely a warning so you can ignore it. I wouldn't recommend downgrading to an older version if only because of the improvements in the workflow and the underlying algorithms of APP.

@wvreeven for my first session it was the first set of subs that I took with the calibration frames. Then for the second session I added another 50 or 60 lights and the same calibration frames. Then when I saw that that wasn’t working, I tried to run the first session again and it wouldn’t  run.

I tried to run the first session again and it would run.

As for the pop-up, it tells me that the background neutralization has been turned off and then when I turn it on the image doesn’t look like it would after a successful run. 

@garycphoto Are that first and second session among the files that you uploaded? If yes, which lights belong to the 1st and which to the 2nd session? If no, can you upload those as well?

@wvreeven sure I’ll upload the second session in 20 mins. They’re the same settings as the first session. 

@garycphoto Great! And that's the session that works with these calibration files?

@wvreeven No only the first session worked with the cali files. When I added the second session to the first is when it went wrong and the popup appeared. They're uploading now.

Hi Gary @garycphoto,

Wouter asked me to have a look as well to see what is going on with your data.

The 1st and 3rd warnings from your initial post are definitely caused by not mounting the harddrive properly in both Read and Write mode.

The 2nd warning is about Background Neutralization being turned off because your calibrated lights are severely clipping to the Black point / 0-value corrected for the artificial adaptive pedestal that is kicking in on your data to be able to still process it and give you a more or less decent result. In this case, background neutralization is turned off by APP because it can not work with data that has real negative background values.

Now, by default, the adaptive pedestal is turned on. If you would turn it off in 2) Calibrate, you would immediately see that you have a serious data problem here. This is a calibrated light with the pedestal turned off:

You clearly see that many ! pixels (500.000+) are clipping to zero and that the sky background is seriously too low when the calibration masters are applied.

The question now is : why does this happen?

The critical warning from APP indicates 3 causes:

1. under exposure
2. incompatible sensor offset for lights versus calibration frames
3. light leak in optical train

I created all masters and had a good look at how the calibration masters. Clearly, the masterdark shows that you have a light leak in your optical train so APP spotted the problem correctly.

This is the MasterDark:

The masterdark is showing a vignetting profile which can only occur if light is entering the optical train... which should not happen when shooting darks.

Looking at the median value of the MasterBias and the MasterDark, we can conclude the same:

MasterBias (info in Fits header)

HDU1 - MED-R = ' 2.9356E-02' / median of channel-R
HDU1 - MED-G1 = ' 2.9349E-02' / median of channel-G1
HDU1 - MED-G2 = ' 2.9346E-02' / median of channel-G2
HDU1 - MED-B = ' 2.9355E-02' / median of channel-B

MasterDark

HDU1 - MED-R = ' 3.3631E-02' / median of channel-R
HDU1 - MED-G1 = ' 3.3503E-02' / median of channel-G1
HDU1 - MED-G2 = ' 3.3497E-02' / median of channel-G2
HDU1 - MED-B = ' 3.3460E-02' / median of channel-B

You would expect the median value of the masterdark to be only slightly higher (1-5%) then the median value of the masterbias. In this case, it is much higher indicating something is not right.

Please let us know if all is clear now 😉 If you solve the light leak and shoot new darks all will be fine !

Mabula

@mabula-admin thank you so much for this. I’ll try that for sure. As for the read/write issue. It seems to be happening again. Any ideas? 

Posted by: @garycphoto

@mabula-admin thank you so much for this. I’ll try that for sure. As for the read/write issue. It seems to be happening again. Any ideas?

Hi Gary @garycphoto,

I can only recommend that you make sure that mounting the drive is done properly by macOS and that you use a file system on that drive that is 100% supported by macOS. Then the problem should never happen for sure.

Mabula

@mabula-admin 

Hi Mabula

How are you? Hope all is well. I recently installed APP 2.0 Beta and I am getting the read/write error again. I am running Mac Ventura and all folders are set to read/write. Any help at all would be greatly appreciated.

Thank you!

Gary.

This must be a security issue on that location, did you allow APP in the security preferences?

@vincent-mod Thanks for the reply! Yes, I'm sure all preferences allow APP.

You mentioned in another post your files were on the desktop, could this be the issue? What if you put them into a folder that you know for sure should work?

@vincent-mod it’s a strange one Vincent but there has to be some kind of work around. Nothing is working off my external hd either. Happened last year on the older version too on OS Catalina. I’m at a loss. I’ve tried everything.

One difference I noticed with version 2.0 is that the volumes window doesn't pop up upon opening anymore. Is that normal?

Yes, the settings in the tabs are now saved automatically. Including the place of the working directory, maybe that is the issue? You can change the directory by clicking on the directory shown on the top left of APP.

Hi Gary @garycphoto,

In addition to Vincent's suggestion, I think the read/write error will occur if you have the Working Directory of APP on a location that requires extra privileges over your user account on macOS.

So what happens if you use another location for the Working Directory? If your working directory is on an external drive, you need to make sure that your macOS user account has full access to this drive.

Mabula

@mabula-admin fixed. I was loading lights straight in from the pop up box and not setting the directory. Thank you both!

Hi Gary @garycphoto,

So no Working Directory was set? Then APP should ask you to set a Working Directory first always. Did it not do this? In what part or tool of APP did this happen then? Maybe we need to have a closer look there then.

Mabula

@mabula-admin correct. No directory was set and APP didn’t ask me to set one like in version 1.083. So I did as suggested and selected the directory from the volumes tab on the top left of the programme. Works fine now but maybe reinstate the directory prompt in the next update. 
The data run would always fail at step 6(integration).

Hi Gary, @garycphoto

Please accept my apologies for my late reply, I am actually on holiday right now, I will return home on August 14.

We have added the problem to our issue list, so we will make sure that this can't happen again going forward 😉

Mabula