Diagonal pattern in images

@dtc1999 Hi Damian,

I am tempted to say that this is walking noise but I am not entirely sure. Walking noise usually isn't as broad as the lines in your image. A few questions for you:

• Good that you're dithering! How much do you dither? In order for walking noise to be effectively eliminated, you need to dither at least 15 or 20 pixels in the imaging camera.
• Is the pattern in your images always in the same direction? If yes, what happens to the pattern if you rotate the camera over 90º?
• Do you have a fixed set up or do you set it up every time you go imaging? In both cases, are you sure that the polar alignment is OK? How do you do polar alignement?
• Does the pattern show in the raw lights as well or only after integrating? You may need to stretch a raw light to the extreme in order to be able to see it. If only after integrating, do you see it as well if you integrate without calibration frames?
• What do you mean with "unity gain"? ASI2600 cameras work best at gain 0 or gain 100.

Depending on your answers I may have more questions 🙂

Thanks, Wouter

@Wouter-Moderator Thank you for the reply Wouter.

I too thought this was walking noise when I first noticed the issue, but many people in another forum did not believe that it was walking noise, as walking noise will give a much thinner "rain-like" appearance. I am not sure how many pixels I am dithering over the imaging camera. I have entered all the correct info in PHD2 for my guide scope and camera, other than that all settings are default.

I have not yet tried to rotate the camera to see if the pattern follows the change or not. However I am setup to automatically perform meridian flips. I know one occurred in my example image. The pattern stayed the same on both sides of the flip. My thinking this could eliminate an outside stray light source as the cause, correct?

I keep my setup fully built, but I do wheel it in and out of the garage for my sessions. I perform a new polar alignment every time I bring it out. I polar align manually by looking through the pole scope. I am very through when I do my alignment and have been doing it this way for over 5 years. I am very confident that my polar alignments are good.

It is very hard to tell when I look at individual light frames that are uncalibrated. I would be more than happy to upload them along with my calibration masters if you or anyone else wanted to take a look at them.

I do image my 2600s in gain 100. The example image is 100. I have also imaged at gain 0 and got the same results.

Could you show us your flat? In linear and stretched form? Linear you get by loading the flat and then switching to "no stretch" in the stretch presets on the right of APP.

Hello @Vincent-Moderator 

Here is an unstretched masterflat file:

This is the masterflat in the default stretch of 15%BG 3sigma 2,5%base:

These files were uploaded in jpg, let me know if you want to see the fits

Thanks, I was wondering if it had to do with the flats maybe. The diagonal lines are very subtle and I think bigger than anything related to walking noise. What happens when you make an integration without any calibration data?

Here is a stack of 20 of the subs with just a crop and LP removal. Stretched to the max 30%

Ah, there it's super clear indeed, so it's definitely in the data itself. I am wondering if bigger dithering steps will help, normally people tend to do dithering with just a few pixels, but it can be very good to dither with 10-15 pixels even. I would try that first. It does look a little like walking noise here so that may work.

What is strange is that with only the subs, the pattern is vertical. With calibration files, the pattern is diagonal and looks much wider. 

Here is one sub that is calibrated, LP removal, and stretched 30%. Although faint, it looks if the diagonal pattern is still there. Would walking noise appear in one sub?

@dtc1999 I find it hard to see in the single sub, but I think it's right that walking noise you see only after integration. I think this is where we should ask for your data to have a closer look. 🙂 I suspect multiple noise patterns in your data.

Thanks Vincent. I have uploaded under the folder name you suggested and included my master calibration files and 24 subs.. 12 subs are before the meridian flip, and 12 after 

Thanks for uploading! Please allow for a day or 2 as we're always a bit busy in general. 😉

Ok having a first look at the data;

What I notice is that the signal is very faint with a single sub, I'm wondering if that may cause a bit of an issue as it's more difficult to get out of the noise floor. Another thing is that the unstretched histogram is quite far from the left, is your offset very high by any chance?

My subs are 300 seconds, gain 100, offset 25

How far is it showing you that they are off the left of the histogram? It's showing me that they are roughly 5-10% from the left.

actually I was looking at the FIT viewer histogram. The histogram in APP is showing between 20-30% to the left

I know you guys are busy, and I appreciate you looking at my data. I noticed something after looking over my imaging software. Between late last year and early this year, which is roughly around the time I noticed the issue happening, I switched from SharpCap to NINA for my imaging software. Around the same time period, I also switched from an OAG back to a regular guide scope. I felt the guide scope was working better than the OAG with my Redcat51. 

What I have noticed was with SharpCap, my dither pixels was set to 10. NINA still had the default setting of 5 pixels. I can't say for certain that this was the cause of the issue. Even though when I changed my guiding system, I did update PHD2 with the new focal length for guiding, the amount of actual pixels I was dithering did change since NINA used a smaller number by default. 

The more I look at my data, while it doesn't resemble walking noise when there is no guiding at all, it could be faint remaining artifacts from not enough dithering. Possibly even Fixed Pattern Noise. For troubleshooting, I have changed the dither pixel value in NINA to 20. Starting on Sunday, I have a stretch of clear skies for 3 or 4 days. I will get a chance to test this changed setting. I am also going to shoot a variety of exposure lengths: 120, 180, 240, and 300 seconds to compare the results. In addition, I will try NINA's Flat Wizard for the first time, as all other calibration frames are made with NINA, but I was still using Sharpcap to create flats as I felt Sharpcap has a much better detailed histogram. Using NINA's Flat Wizard, I will not need to visually inspect a histogram.

I will report back with the results.

Great! Yes, I think the effect isn't very big, but indeed is there. My guess is that this might get sorted with a small change in data collection, like the dither increase. Very curious if that solves it!

After a few nights of clear skies this week, I had a chance to change some dither settings. The first night I set my dither pixels to 20. I timed the dithers from start to finish and they were completing in about 30 seconds. After processing the data, I noticed the pattern was improved a lot, but I was still able to see faint remains of the pattern in aggressive stretches. 

Last night, I increased the dither pixels to 40. I also increased the dither settle timeout to 60 seconds to be safe. Again I timed the dithers and they were now completing in 45-50 seconds. I just finished processing the data, trying to get the pattern to appear in the image. Here are two results from last night with a 15% and 30% stretch. There is 6 hours of 300 second subs, gain 100, and using an IR Cut filter. 

The data looks good as far as I can see. Dithering 40 pixels with my setup is actually moving the imaging camera 83 pixels. I am sure that trial and error will show the minimum number needed to dither will fall somewhere between 20 and 40. At first, that seemed very high, but realizing my APS-C imaging camera has a pixel size of 3.76 and the Redcat51 is only a 250mm focal length, a larger number of pixels needed to dither does start to make sense. What do you guys think? 

Going through all the data I collected while troubleshooting my issue, including the sessions that had dithering set to 5 pixels, I had 36 hours of mono data. I stayed on M101 so it was easier to compare results after making changes. 

Today I decided to go ahead and stack it all to give APP ver2.0 a workout... 433 subs at 5 minutes each.. 6 sessions total. I always keep my OTA built, so I only needed 2 sets of flats and I was able to stack everything in 2 sessions The subs with the diagonal patterns seemed to average out pretty well and completely disappeared when I selected 10% stretch for this image.

I must say that Ver2.0 handled the data quite well, although I cant give a total completion time because I got a hard drive space warning in the middle of the integration and had to delete a few files to continue. A good suggestion for an upcoming version would be for a sound alert when you get a warning. Just something to get your attention like the Gong sound..

Tomorrow I will start working on my color data.. 

Awesome! I still don't know for sure what the pattern would be you're seeing, but because it gets way less when dithering more aggressively, it must be a data issue. I would maybe try to dither that much every 3 frames, just to see if that is enough already and to save you some time. The wide FOV combined with your sensor might definitely make it harder to remove this kind of pattern, apparently the pattern is quite big and thus needs this large of a dither step.

@vincent-mod I have been reviewing all my data, as I keep every session organized with notes of the date, sky conditions, equipment and software used, etc. As I stated earlier, the only imaging changes that I thought were made during the time of the issue starting was the switch from OAG to guide scope and from Sharpcap to NINA.. After reviewing more slightly older data, I found that this is not the case. I actually made those changes a few months prior to the issue starting. I have normal looking data with the new equipment and imaging software. Investigating further, I found something that I overlooked this whole time..

The earliest I found that the issue happened was in November 2021. After looking over everything in my laptop, the only thing that I could find that coincides with the issue is that during the same month, my laptop did the free update from Windows 10 to Windows 11.. I can not say this is the answer to the issue as I thought all my software would still be compatible. My camera drivers are current and I am using the current non-beta version of NINA.

The only change after the Win11 update was I did also upgrade my ZWO 071MC to a 2600MC to image along side my 2600MM on my dual imaging setup. The 071 creates subs about 30MB and the 2600MC creates 50MB subs. I know that's a slight increase in data transfer even with the MM and MC imaging at the same time on 2 separate USB3 cables and ports, it is well within the data transfer limits of USB3.. Also I would see the issue when I was only imaging with one camera during testing, so I can not see it being a data transfer issue...

This laptop is a 2018 model (bought in 2017) Dell Inspiron 5579 with 16GB RAM, 4 cores with 8 processors, BIOS, and windows update always kept up-to-date. It has always ran great and never once crashed during an imaging session. I can't see how going up to Windows 11 could cause an issue like this. I am past the point of no return for going back to Win10 as Microsoft only gives you 10 days to change your mind after the free upgrade.. Only option now would be a clean install, and I don't know if I want to do that when its not even certain that is causing the issue.

Well, you're organized for sure! Very nice. I highly doubt Windows 11 would be the reason for the pattern you're seeing, to me it's more likely that it's something to do with data transfer/hardware, maybe even the new camera. Is it possible to tackle these variables one at a time? Like trying to record data locally, then transfer that to your computer without the need of USB etc? Then after that, check to see if it's only that camera and not another.. Check if you have common ground everywhere, so no power interference can get on the data cables.

When it's difficult to pinpoint, you have to do 1 change at a time, otherwise it can get very frustrating (which it already is I'm sure).