5

A simple change took almost a week to resolve

 3 years ago
source link: http://rachelbythebay.com/w/2013/05/04/ticket/
Go to the source link to view the article. You can view the picture content, updated content and better typesetting reading experience. If the link is broken, please click the button below to view the snapshot at that time.

A simple change took almost a week to resolve

Quite some time after I escaped from the web hosting business, I got to see a raw copy of a ticket. It showed me exactly how things were being handled with the current set of techs they had recently hired, and wow, was it bad. If you've ever had to deal with tech support and had to go around and around trying to get something done, this will seem very familiar to you.

Here's a challenge to anyone of the sysadmin persuasion: read through this without skipping ahead and see when you realize what's actually being requested here. Then see how many people totally miss it.

OK? Here we go.

Ticket: adding a new mail server ip to our dns records
Friday 7:25p Customer: A few months ago we went through this big deal to get our emails to stop being label as spam when we sent them. One of the things we neded to fix was to set our IP in our dns records as being allowed to send mail for our domain (something like that). Anyway, we're installing [mail sender program] on our [hosting company] server but we're having problems getting it working. We already have it working on our development server so we want to make sure that when we send email from that IP we won't be flagged as spam. So I'm guessing you just need to add that IP to our [hosting company] DNS to specify that it is authorized to send mail for our domains. Our development server IP is [DEV-IP]. Let me know if I'm making sense or not. Thanks
Friday 7:25p System: [Pointless auto-ack message by ticketing system]
Friday 8:12p Tech1: Hello, Do you mean you want to allow [DEV-IP] to relay mail through your server or do you want the domain added so mail can be delivered. If you have any questions or additional comments regarding this support ticket, do not hesitate to upate this ticket, or contact us 24 hours a day. We are always available to support you. [Snipped 7 lines of pointless signature]
Friday 8:16p Customer: We will be sending mail directly from that IP and not relaying it. We just want that IP added to the DNS so it doesn't get flagged as spam when we say we're sending mail from [DOMAIN].
Friday 9:48p Internal comment by Tech1: Have you guys ever done something like this?
Friday 9:48p Ticket forwarded to DNS admins by Tech1
Friday 10:17p Internal comment by Tech2: Two of the "spam" things we look at are rDNS and SPF. The domain has both. Aside from that or what extra things they might have done, you might want to ask what domain they were working on last time and compare the two.

[Snip signature]

Sunday 9:40a Tech3: Hi [customer], Mail sent from your server should not be flagged as spam. I've checked your setup, and your mail server name resolves to the server IP, and the PTR for the server IP resolves to the mail server name. Additionally a TXT record is setup in your zone file (a SPF record) which allows relaying:

[Tech3 pastes in run of 'host (ip-of-server-at-hosting-company)' and gets back a PTR record]

[Tech3 pastes in run of 'telnet (ip-of-server-at-hosting-company) 25' and gets back a sendmail banner before sending 'quit']

[Tech3 pastes in a run of 'dig (customer-domain) TXT' and gets back a SPF record of 'v=spf1 mx ~all']

If you have any other problems or questions, please feel free to give us a call or update this ticket through your customer portal!

[Snip signature]

Monday 2:34p Customer: Yes, we had that setup properly a few months ago.. What about the new IP othough? Are we ok to start sending mail from that new IP: [DEV-IP] ?
Monday 9:55p Tech4: Hello, I am not sure we are understanding you on this one. My best guess is you are talking about your SPF record (Sender Permitted From). For the domain [DOMAIN]. it basically says the only server allowed to send mail as that domain is the MX server, which is your [hosting company] server.

If this is what you mean, please include any and all domains that will need this modification, and we can do them all at once.

If you have any questions or additional comments regarding this support ticket, do not hesitate to update this ticket, or contact us 24 hours a day. We are always available to assist you.

[snip sig proclaiming "Unix Technical Support Level II"]

Tue 1:42p Customer: Yes, please add the IP [DEV-IP] to the MX record then for the [DOMAIN] and [DOMAIN2]
Tue 4:20p Tech5: Good afternoon [customer],

The MX record for both [DOMAIN] and [DOMAIN2] are both pointed to server1.[DOMAIN2]. Changing the IP address for the server1 hostname will accomplish what you want. However, if you are using this hostname for other functionality this will break that functionality. Please confirm that changing the A record for server1 is acceptable for us to continue in this regard.

If you have any other concerns or questions, please let us know. We are here 24/7/365, toll free at [NUMBER] or you can update and create tickets from the [PORTAL URL]. Thank you for choosing [hosting company]!

[Snip sig]

Tue 5:27p Customer: No that's now that I want. Please read the ticket for an explanation of what I'm looking for. We only wnat to make sure that when we send mail from the IP [DEV-IP] and set the from address to something like info@[DOMAIN] that the proper SPF records are set such that we aren't flagged as spam. I don't want to break any configuration that's already setup. For instance we'll still be sending our normal mail through our [hosting company] server and I don't want to break any of that. I just want to be able to send our bulk mails from the IP I specified and have them not flagged as spam because they aren't authorized to send mail using [DOMAIN] or [DOMAIN2] in the from address.
Tue 10:00p Tech6: Hello [customer],

I've added 'ipv4:[DEV-IP]' to the existing SPF records for [DOMAIN] and [DOMAIN2]. This will go into effect in our DNS servers at the beginning of the next hour. DNS servers that have cached the SPF record may continue to do so for up to 24 hours after the change goes into effect on our DNS servers.

[Snip sig]

Wed 6:24p Customer: thanks. that's exactly what i was looking for.

Did you see that "Tech5" basically wanted to change the A record for the actual hostname of the customer's server at the hosting company?

Did you notice that 6 different techs were involved, and that it started on Friday at 7:25 PM but the customer wasn't satisfied until Wednesday at 6:24 PM?

Go back and look at what happens around the time of those internal comments on Friday night. The ticket sits there with no response to the customer's Friday/8:16 PM comment until Sunday at 9:40 AM!

This ticket should have taken at most 10 minutes to do. That's 5 minutes to flip through their ticket history to see which domains they touched last time, 3 minutes to twiddle the SPF records in the DNS tool, and 2 minutes to send an update to the customer.

Absolute insanity.


report

Recommend

About Joyk


Aggregate valuable and interesting links.
Joyk means Joy of geeK