Problem with satellite trails

What version are you using at the moment?

1.083.4

Ok, that shouldn't be a problem. The standard settings (set when you start APP) already are enough for most data. If you have enough frames (like 30) and proper calibration files, that should work very well, if you have trails from geo-stationary satellites, it may be trickier. Is that the case? Could you post a few screenshots of some lights?

I also noticed this with 1083.4.

Earlier versions of APP did such a good job on satellites that I usually ignored them, but the other night I had two that were visible in my final integration using 1083.4.

I had 42xred, 42xgreen and 42xblue. There were two red frames with bad satellite trails and some fainter trails in other frames. The two worst were easily visible in the final integration. I used standard settings for all tabs.

In the end I inspected every frame (takes a while - I wish there was a blink tool) and deleted the two worst. Fixed the problem, but I was surprised it was an issue at all. Maybe I've just never had trails as bad as this. It was a longer focal length than I usually use.

By the way - congrats on APP. Its a terrific program. Mabula has done an amazing job.

JC

@vincent-mod

Here are two screenshots of M51.

First one with the sat trail.

Second one without the trail.

Camera was ZWO ASI 2600 MC Pro.

Telescope was Meade 14-inch ACF.

Exposure time 300s each.

I first extracted the RGB channels, because I took H Alpha frames on another telescope parallel with that one.

If I don't use the frame with the sat trail, the finale result is nice as always with APP.

Maybe, it has something to do with the 3.5m focal length?

Thanks for looking into my case.

Regards,

 Ulrich

Really shouldn't be an issue. I think I need the data to be able to check out what's going on, if you want you can share it to our server;

@vincent-mod 

Dear Vincent!

I am just uploading the files.

You will find a folder aurora74-outlier-calibration

In that folder I uploaded the original (unseparated) RGB files from ASI2600MC plus masterbias, masterdark and BPM.

In the subfolder HAlpha you will find the HAlpha files from a ASI1600MM plus masterbias, masterdark and BPM.

I hope you can use the files for your evaluation of the problem.

Thanks.

Regards,

 Ulrich

Hi Ulrich, sorry it's been a bit busy. I'll have a look today!

So, I think your masterdark has some stars in it. Is that possible? It's best not to use calibration frames which have any kind of pollution in them.

The satellite though is the biggest issue, I think it wasn't rejected well due to how big it appears with your FOV. To be able to reject it better, higher dithering steps may help. I managed to get it somewhat better with setting integration to median, switching to adaptive rejection and a kappa low of 2.2, kappa high of 4.0. Not completely gone, but reduced a great amount.

Hi,

I also had seen this effect, stacking 400+ files while just one strong plane trail remained clearlyvisible (though dimmed). Outlier rejection should have handled this. The only way out was manual frame deletion (Which is a bit boring with 100s of files)

I have 2  questions here:

1)Once APP fidentified an Outlier Pixel, i.e plane/sat trail, what does it do about it?

a) frame gets discarded

b) Pixel gets discarded

c) Pixel value gets replaced, e.g by median of that very pixel (DSS has that option which works pretty well for loose Kappa andmultiple iterations)

2) To avoid tight kappa values, one would require more iterations of the outlier rejection process. With which control can this be achieved?

thanks,

Jochen

Vincent

I don't understand your suggestion -

    "To be able to reject it better, higher dithering steps may help."

I dither fairly aggressively now, but as it is extremely unlikely that a satellite trail will be in exactly the same position in different frames surely dithering is irrelevant to this issue. Have I misunderstood something here ?

Your other comment about the size of the satellite trail in the FOV sounds right to me. I'm starting to think that I may need to manage satellite trails better when shooting at longer focal lengths, either by deleting more frames or by playing with the rejection settings.

JC

Yes, then dithering won't have an effect, it does depend on the type of satellites though, geo-stationary I could imagine benefiting from it.

In this case it seems the size of the trail and mainly the amount of signal it produced, was to blame. The outlier rejections aren't very aggressive as APP does want to preserve as much of faint signals as possible. So that's why you would need to experiment a bit when there indeed is an issue in the end. A better way to quickly screen for very big trails and such would indeed be nice.

Do you have any answers to my above questions, please?

thanks,

Jochen

Posted by: @jochen-scharmann

I have 2  questions here:

1)Once APP fidentified an Outlier Pixel, i.e plane/sat trail, what does it do about it?

a) frame gets discarded

b) Pixel gets discarded

c) Pixel value gets replaced, e.g by median of that very pixel (DSS has that option which works pretty well for loose Kappa andmultiple iterations)

I have to say I'm not an expert on the actual algorithm used here, but the frame definitely isn't discarded. Mabula always mentions not to just throw away frames with satellite trails as you should be able to deal with that and the rest of the image can still contribute a lot to the signal. It does, normally, work very well with the standard settings and enough data though, so when you do run into the fact it wasn't fully rejected, you have to tweak the kappa parameters. If that is needed to be set to very tight parameters, then it's better to remove the offending frame.

2) To avoid tight kappa values, one would require more iterations of the outlier rejection process. With which control can this be achieved?

This is done internally by APP itself, you can choose different algorithms as well. We don't have an iteration setting to be set.

@vincent-mod understood.. but with 100's of files, any plane trial should be recognized as outlier and ironed out, no matter what kappa, but obviously it doesnt work. as Aurora74 mentiond, that started with 1.083, so maybe this breaks up after the first iteration. I'd rather use the conservative APP kappa values with 1 iteration more than to turn the kappa srcrew, so if APP does that automatically it should recognize the trail remainder or just provide manual control.

Well, it may definitely be improved upon, but we haven't had many issues regarding this so it doesn't seem to be a wide-spread issue, but more towards specific data sets. I'll ask Mabula to give a more in depth reply on the algorithm used here, he's sick at the moment, so please allow for a few days.

Hi all @aurora74 @connor231 @jochen-scharmann,

I think I need to see this  for myself, i never have outlier rejection issues like you describe myself, not with 20-30 frames and not with 100+ frames.

Could it be that you use the automatic integration setting in the mosaic integration mode? If so, that explains why no rejection occurs. In mosaic mode, you don't want that to be enabled and thus it's disabled in mosaic mode.

When the integration starts, the generale console and the process console will show you in details what outlier rejection settings are being used if any. And the settings are stored in the FITS header of the integration, so can you check what happens/happened?

1)Once APP fidentified an Outlier Pixel, i.e plane/sat trail, what does it do about it?

a) frame gets discarded

b) Pixel gets discarded

c) Pixel value gets replaced, e.g by median of that very pixel (DSS has that option which works pretty well for loose Kappa andmultiple iterations)

 

2) To avoid tight kappa values, one would require more iterations of the outlier rejection process. With which control can this be achieved?

1) A whole frame is never discarded, that would be very bad and a huge waste of good data... Now depending on the algorithm.... for each pixel stack, a median and dispersion/scale is calculated. With winsor rejection for example, the pixels that lie outside of the bounds (central value +- kappa*dispersion) are replaced with the bounds, then winsored median and dispersion/scale is calculated. That is the essence of Winsor rejection..

2) Iteration control has been made redundant with smart implementation that APP now uses. It will continue iterating always until all pixels outside of the +- kappa*dispersion bounds are removed. So once no pixels are removed after an iteration, it will stop for that pixel stack and the median/average(/max) of the survived pixels becomes that integrated pixel.

Mabula

Hello Mabula, I've also recently had problems with the rejection of satellite tracks in a dataset. The data set now includes 460 frames and if I now integrate everything in automatic mode, there are always remnants of satellite tracks. Even with aggressive rejection settings, trace resters still remained. I found out more about what could be wrong. As far as I understand it, my data set is due to the strongly changing recording conditions. If images are now brightened by moonlight, you have very noisy frames. And this noise creates a high dispersion of the kappa central value, so the rest of the satellite track can lie within the central value. Which are then reintroduced into the pixel table after rejection and thus become visible again in the final integration result. I tried again to reprocess the data set, but this time I didn't use the best frame quality as a reference, but the picture with the lowest background brightness. Now all traces could be rejected. I have to investigate it further with other datasets to see if it works there too. I can also upload the dataset once. maybe helps to better understand the problem. I think the thing with the satellite tracks will increase in the near future (Starlink etc.)

With kind regards Henry

Thanks @Mabula for the detailed explanation!

"With winsor rejection for example, the pixels that lie outside of the bounds (central value +- kappa*dispersion) are replaced with the bounds, then winsored median and dispersion/scale is calculated. That is the essence of Winsor rejection.."

This might explain the reults, as I do not see weak satelite trails but just weak remnants of previously strong plane trails. as they are replaced by the bounds instead of a median, these might still survive. 

Usually for my 200+ frames stack I select average / adaptive rejection. I do not do mosaics, so I might be slightly displaced in that thread

Thanks again,

Jochen

Dear Mabula!

Thanks for looking i that case.
No, I did not use mosaic integration.
But maybe it is some oversampling effect.
Perhaps the sat trails are visible on too many pixels and that's why the rejection does not work.
I should use binning mode at the ASI camera next time and check, if the sat trails get erased properly than.
The combination of 3500mm and a ASI2600 might be the problem. I think Vincent already mentioned that point.
But if you find another solution, it would also be great.
Regards,

 Ulrich

I just came to to the forum looking for fixes for this -- typically 1-2 out of 20 of my subs have satellite trails and if I don't exclude those frames from stacking, the trails are still faintly present in the final result (with typically ~200 stacked frames).  Has anything changed for this in 2.0.0?  I'm observing from Bortle 7 skies with highly variable background sky brightness, Henry may be on to something as far as the cause.

Posted by: @eebclee

I just came to to the forum looking for fixes for this -- typically 1-2 out of 20 of my subs have satellite trails and if I don't exclude those frames from stacking, the trails are still faintly present in the final result (with typically ~200 stacked frames).  Has anything changed for this in 2.0.0?  I'm observing from Bortle 7 skies with highly variable background sky brightness, Henry may be on to something as far as the cause.

Hi Brian @eebclee,

Thank you for your question.

Do I understand correctly then that you would integrate +- 200 frames and that about 10% has satellite trails? Then all trails should simply be gone...

COuld it be that you set the registration mode to mosaic? In that case, the automatic integration willl not apply outlier rejection.

Have you tried different outlier rejection settings yourself by disabling the automatic integration settings?

Mabula

@mabula-admin I'm using the defaults for the most part, registration mode is set to "normal" and integration is "automatic."

Hi Brian, 

You may want to decrease the high kappa value to  slightly below 2.5 (but not below 2.0). That should hopefully remove the trails. I can confirm using the default kappa setting there is usually a slight remnant of the trail. More problematic for me are plane trails as those are much bright and a huge airport is just 50 miles away.

Clear Skies,

Jochen

Posted by: @eebclee

@mabula-admin I'm using the defaults for the most part, registration mode is set to "normal" and integration is "automatic."

Hi Brian @eebclee,

Like Jochen indicates, you probably then need to set the outlier rejection settings yourself with a lower high kappa. Disable automatic integration, set it to average and then adjust the outlier rejection settings 😉

Mabula

Hello admins,

I am also having great problems with satellite trail removal.

I am using 1.083.4

I have lots of data of comet 2022 E3 ZTF which has satellite trails. I've tried integration of 3 images up to 100 images. I have tried registration made 'Comet' and 'Normal', and all available rejection options - Average, median, maximum. And have experimented with a wide range of hi and low kappa values, including below 2.

All of these setting variables give exactly the same result, and do not remove the satellite trails (apart from median integration, which gives a fainter, but VERY visible trail).

It seems that many people have experienced the same issue.

I have spent hours trying to get trail rejection to work, but nothing seems to work. 

Please can you help?

The only option I have is to manually delete frames with trails (time consuming and should be unecessary), or go back to using DSS.

Thanks for any assistance

Kevin

Strange it has this much of an issue with your data. Could you upload maybe 30 frames and your master calibration files for us to check?

Hello,

I am also having some problems with satellite trail removal.

I am using 2.0.0 beta26. I took 396 180s sub (with 95 darks, 50 bias, flat, darkflats).

The trails are removed but only partially. I've tried many different high and low kappa values but impossible to completely get rid of them. (average, quality, adaptive rejection).

There you have the rejection map of the same area as the screenshot up here and the full version.

As we can see, all the satellite trails are gone but there is this high density band (all sub have some trails there in a different way).

Are there some special settings for this kind of situation ? I usually have no problem whatsoever, but also never encountered this before

With the high kappa value, I went as low as 2.2 but found that the best setting was around 4.0 (where the trail are almost gone, see first screenshot).

The other part of the image looks awesome otherwise 😉

Thank you for the assistance

Fred

Hi Fred @fredmt,

Thank you very much for sharing this. Yes, I can clearly see the issue and you have so many satellite trails in that dataset...

I think it should really if you can share the complete dataset with me, so I can test and further improve the outlier rejection in Astro Pixel Processor.

Can you upload the dataset here ?

https://upload.astropixelprocessor.com/

username: uploadData

password: uploadTestData

Please make a folder with your name and issue like : Fred-OutlierRejection-issue

and upload the data there, thanks !

Normally, the outlier rejection will be better with lower high kappa values for satellite trails. Perhaps in this case, median integration instead of average will help, have you tried that? And did you enable Local Normalization rejection?

Mabula