Server Offline - Reboot in progress

There were some mail server issues today (dropped connections, webmail inaccessible) and subsequent attempts to restart/repair services were also ineffective. An attempt to restart the server was made, and is currently in progress. The Tier II (and higher) techs are currently working to restore services.

UPDATE: 4:00pm ET:

Four hours later, and still "in progress". Very sorry. I am assured every time that the resolution will come soon, and I'll be notified once the server is back online.

FINAL UPDATE: 4:40pm ET:

The server is back online, and should be functioning properly. Please contact me if you are noticing any further connection issues.

The official response from the server techs was this: "...it seems there was an issue with the IP address associated with your account and routing. Our system administration team routed the IP correctly and this issue isn't happening anymore, we apologize for any inconvenience this issue may have caused."

Sadly, this was caused by their reconfiguration of *another* issue they were attempting to correct.

My apologies for the down time.

Slow or Offline Websites/Email ** UPDATED **

Hi folks

You may have noticed that your website or email have been fairly frequently going down the past few weeks. We have been investigating and attempting to correct as soon as we become aware of it (which has been sometimes nearly instantly). The hosting company has also been very quick to respond, and has assured me every time that the culprit is not a BWD-hosted account. (All BWD-hosted accounts are on a shared hosting server, meaning there are other "accounts" on the same machine. That's where the problem was originating.)

The culprit appears to be some malicious (and over-active) scripts on another account on the server. They've been deleting/removing them as soon as they discover them, but they have not successfully removed all of them until (they are fairly certain) today. Here's the message from the hosting company.

There have been so many script problems on your server recently. I see that we disabled one script on the 13th and two scripts yesterday. I was seeing a higher than normal load on your server right now and I contacted one of our team of experts that monitor your server. They found yet another script to disable. The load average is coming down and things appear to be getting back to normal. He is not seeing any other problems at this time.

I'm very sorry that you are having to deal with other users' script problems. Our team of monitors are viewing our shared servers around the clock, making sure that all services are running and users are "playing nice". Please let us know if there is anything else we can do for you.

Sorry for the inconvenience. I know that some of you have had trouble sending email (hitting the mail server at one of the times it was overloaded/down). Hopefully the problem has been (finally) completely corrected.

Thanks,
Greg Campbell
BWD-Graphics.com

UPDATE: Issue Re-Opened

The issues are apparently not completely resolved. I am working directly with level II tech support with the hosting company to resolve this as quickly as possible. Again, the issue is not with any BWD-hosted account, but with a handful of other accounts hosted on the server. When the source is discovered it is immediately disabled, but the original source has not yet been discovered as new scripts are constantly surfacing and bogging down the server, at least hourly or more! Will update again when a final resolution has been reached.

UPDATE Jun 23 10:55am ET: The server was down AGAIN today... this issue has obviously not been corrected. Working on it again with the hosting company. Very sorry for the inconvenience.

High Server Load - Resolved

The server that hosts all BWD-hosted accounts was experiencing unusually high loads over the past few days. This was causing at least slowness, and sometimes websites and email were intermittently down. After investigation, the cause seems to be an account owned by another reseller on the same server. The account has been brought under control now, and you should not have any further downtime for your websites or email.

Please contact us if you are still having trouble connecting to the server for either.

Thanks,
Greg Campbell
BWD-Graphics

Servers Down

It would appear that as of about 1pm ET the BWD server is down. I am checking into it, but am unable to reach tech support at the moment. Past experience says that this will be remedied in 5-10 minutes, but I'll keep on top of it and update this post if I learn of any more delay than that.

Thanks,
Greg Campbell
BWD

UPDATE:
A couple bad signs are that the tech support line is "busy" (or, when I did get through a message said they are not accepting calls at the moment) and most of their own access points are down as well. So it's the hosting company, or a large chunk of it, not just the BWD server.

UPDATE:
And, right on cue, at 1:06pm ... the server seems to be operational once again. Sorry for any inconvenience.

BWD Hosting Down: 9:30am ET 03 May 2010

The BWD servers are currently completely down. Web and email is inaccessible until service is restored. Will continue to monitor the situation and provide updates here.

UPDATE: 10:05am ... just after I posted the initial notice, the service has been restored. Please contact me if you are having any continued connection issues.

Email Problems (Premium Hosting Accounts)

If you are a BWD Premium Hosting account holder, you may be experiencing some issues with your email today. (My account is repeatedly having trouble connecting to the server.) Here's the official update from the hosting company:

Update: Incoming Email

Update: Network engineers have identified the cause of the issue and have taken steps to resolve the issue. Mail should be back to normal within 30 minutes at the latest.

We are aware of an issue that may cause email sent to mailboxes hosted on our platform to be returned to the sender with a bounceback error indicating the retry limit has been exceeded. Our network engineers are currently looking into the issue. We apologize for any inconvenience caused by the bounceback errors. More detail will be posted on this issue as it becomes available.

- 11/23/09 at 14:12 ET

Email: Outgoing Servers Now Working

It appears the outgoing mail servers are now functioning properly.  You should be able to send and receive email on your BWD-hosted accounts.  (11:38am)

Email: Further Details

One thing to add, I am able to receive mail, but not send on the bwd-graphics.com mail servers.  My Mail app lets me pick a different outgoing server, so I am able to send mail via my ISP's mail server.  If you're able to make that modification, you should be able to use your BWD-hosted email account.

I still do not have FTP access to any accounts.

More to come...