The Wholesale Forums

[Anthony]

Dear Community,

A lot of you may have already been emailed about this, some of you won't have, therefore you are advised to read this thread entirely.

Dear Community Member,

You are receiving this email because you are a registered member of The Wholesale Forums.

I am emailing you personally to provide you with an update on why The Wholesale Forums has been inaccessible since Monday 12th (Afternoon). Since our downtime occurred, this is the first opportunity I have had to get an email out, for reasons which will become apparent as you read on.

Firstly, extensive details on what exactly has gone on over the past five days can be found appended at the bottom of this email. I will summarise and cut to the chase for those of you who do not wish to read all of this email.

Q. Why has the forum been offline? What has caused its downtime?
A. We have suffered as a result of yet another incompetant server provider. Our dedicated server became inaccessible on Monday afternoon and through a number of different incidents, all of which at the hands of our previous provider, it has resulted in us having to move to another provider and get another server setup. *For more info, read the appended part of this email.*

Q. What are the impacts of the downtime on the forum?
A. Unfortunately, the forum has been quite badly affected. By no means have we suffered an absolute disaster, we have lost circa 4 months worth of posts/threads and members accounts. On the outset this of course is devastating, and nobody is more upset about this than me, but we have to remain positive and recover what we can. We are working hard to recover the lost data and at this stage I am not ruling out the fact that we may be able to retrieve it, but at this time, I cannot comment any further. *For more info, read the appended part of this email.*

Q. When will the forum be back up?
A. I'm sure this is a question on a lot of people's minds. We moved servers today and we were able to restore the forum. In terms of when when we will be back up, all we are waiting for right now is for the nameservers to pick up the change of servers and for the domain to point to the current server. Those familiar with this process know that this process can take around 6-24 hours to process. Since this change was made at around 11am GMT this morning (Friday), it shouldn't be too long at all before some of you, if not a lot of you, start getting access to the forums once again. I request that you remain patient and be assured that we will be back, in force, very shortly!

Q. Where do we go from here?
A. Within the next few days, access will be restored for the community. That has been our number one priority, getting the forums back up and running and to restore what service we could, as quickly as possible. As you can imagine, the situation with the old server provider is unacceptable and this something that we have passed into more appropriate hands to pursue.

Q. What should we do now?
A. I am recommending that you attempt to access the forum via its URL (not a bookmark) at various points over the next day or so where you should get access. There are various threads already on the forum detailing past events and certain notices that you are strongly advised to look at. Apart from that, it is really a case of helping restore what we may have potentially lost, so get yourself on there and get some discussions flowing.

Important: Any emails that were sent to our domain addresses (ie; @thewholesaleforums.co.uk) after the point of downtime (12th May) will not have been received. Therefore if you haven't received a reply, it isn't because we are ignoring you, it is because it simply hasn't come through due to the server going offline. This may be the case for a few days while the new server settles in. Any immediate or urgent emails can be sent by directly replying to this email or sending them to: thewholesaleforums@gmail.com for which we are using as our temporary email account.

Thank you to some of you who have offered your support and concerns over the past few days, it has been a difficult time, as you can imagine!

We look forward to seeing as many of you as possible over the course of the next few days once full service resumes!

Thank you for your understanding and co-operation.

Kind Regards,

Anthony Trollope
Forum Admin & Owner
The Wholesale Forums

History of Events

Monday 12th May

On Monday afternoon, we received notification that our server had gone offline. Having rung the server company within five minutes of receiving that notification, I was told that there was a networking problem being experienced by the server company and that it would soon be resolved. I requested to stay on the line while the matter was resolved and twenty minutes into the phone call I was told that it was going to take longer and I would be called back.

Two hours later, the server was still offline and I had received no communication from the server provider. I called again and was told they had no further information to provide at this time. At the time I queried whether this was an isolated incident or whether others were experiencing similar problems. I was told that it was isolated, it was only our server that was affected. (Odd for a networking issue I thought at the time).

It was soon 6pm in the evening and we were still down and I hadn't received a call back. When I called again, to my amazement, there was no response. This is a company that claims to offer 24 hour support. I continued phone calls, emails & support tickets in the hope of reaching somebody quicker.

At 21.50pm I received a text message stating that the server would be up and running within the next 45 minutes. I waited on tenterhooks praying that normal service would be resumed as promised. An hour later, the server was still not up and their support line was still not being answered!

Tuesday 13th May

I was surprised to see that the server was still offline the next morning and that I hadn't received any communication in response to my support requests.

A further 23 phone calls and 5 emails were sent during the day. A response was finally received late afternoon via a support ticket. The support technician stated that the server hard drive had suffered severe corruption and that was the reason for the server being offline. I responded immediately asking firstly how it had taken them this length of time to respond to my support requests and why I was originally told that it was a network problem. I also made a point of mentioning that if the primary hard drive had failed, why the second hard drive hadn't taken over in its place. When the server was originally specified we opted for a second hard drive to run in RAID 1 with the first hard drive, meaning that all data would be mirrored onto a second hard drive to avoid loss of data and disruption to service.

Three hours later I received a response from their support department. They claimed that they couldn't account for the fact the RAID hadn't taken over, but that the data had been backed up and that they were making arrangements to retrieve the data within the hour. They also claimed that BT were experiencing problems in the local area, that was the reason why they couldn't answer their phones.

At this stage, the server had been offline for over 24 hours, we were uninformed as to why the service was down and whether our data was secure. There was also no indication as to when the server would be up and running again.

The day drew to a close and no further response was received from the server provider despite maximum attempt to contact them.

In set panic!

Wednesday 14th May

Yet again I was to find no new emails and that the server was still offline. By this point I was beyond concerned, I was frantically trying to contact the via any means. By this point, according to my call log I had made 73 phone calls to get in contact with the company.

In my experience, downtime like this had never been heard of. I have experienced outages of 24 hours before, but nothing like 3 days! I made the decision that if by Wednesday PM a response wasn't received or that the server wasn't operational, we would have to proceed with getting another server setup with a new company. Although I knew at the time that without an up-to-date version of our server data, we would be in a real mess, I simply couldn't rely on this company any further and had to try and move forward with what data we had.

Under normal circumstances, I would have moved servers after the first 12 hours of downtime but I was reliant on the server company having the data that hadn't been locally backed up. (I made the assumption, which to date has proved poor judgement, that two hard drives running in RAID1, a NAS backup drive and a 14 day retention period would cover us for data loss).

The day drew to a close with yet no correspondence from the server company. I went ahead as planned and made arrangements to get another server sorted for the following day.

Thursday 15th May

An early start Thursday and still no communication. I was losing all hope and patience that we would receive a response from the server company.

As planned a new server was specified and purchased with a new company. I was fortunate enough to get the server built within 12 hours due to an appreciation of our situation. I was informed that the server would be up and online by Friday morning, a turnaround of less than 24 hours.

The afternoon gave me a chance to continue my assault on the old server providers, further phone calls were placed, emails sent and so forth. You guessed it, they went unanswered, every single one.

I spent the evening recovering what data I could through few methods that I knew of. I found a way of viewing indexed content via Google, saved what I could on web pages and prepared for getting the new server setup the next day. TWF is/was not the only portfolio to be affected by this, three other sites were also affected, TWD being one of them.

Friday 16th May

As promised by the new server provider, the new server was ready to use and online by 11am this morning. I was able to recover a local backup of the forum database, but the bad news is, this database is 4 months old, dating back to 24-01-2008. Fortunately I was also able to recover the forum software and some of its associated files and hooked everything up by mid afternoon.

I will leave this thread open for now for any further queries or problems that any of you may have. I will do my best to answer them as promptly and as detailed as I can at this point.

Thank you.

[BrandDirect]

Glad your back and I dont know if this is where you want me to post this , so feel free to move.
I never knew it was down until a good few days ago but I did find out the reasons over at the ukbusiness forums and I feel for you with all the hassle this has caused. It wont be long before this is ancient history and things are back to running to normal.
Quick question if I may?. I am sure through a previous post you stated that the server you used was American based. I am intrigued after reading the hassle trying to contact them, Have you opted for another US host or has it made you look elsewhere maybe closer to home?. Just curious thats all.

Chris

[PT]

Its good to see everything is running smoothly again.

Just one question though Anthony, do you think it will be possible to recover the data from before?

[Weedog]

Originally Posted by BrandDirect

I am sure through a previous post you stated that the server you used was American based. I am intrigued after reading the hassle trying to contact them, Have you opted for another US host or has it made you look elsewhere maybe closer to home?. Just curious thats all.

Chris

The server was in the UK

[Anthony]

The simple reason why I was unable to email people out was because we couldn't access the server. Which also meant I couldn't access the forum software, where member accounts are held. I got an email out as soon as I could.

The server was in the UK and we will name & shame the company concerned but I have been advised against this at this stage. Regarding data recovery, I can't comment on that either at this stage, it is a variable at this time.

Thanks.

[Ems1]

glad things are back to normal!!!

[cbeebies]

I was on a trip out of the town for 10 days and didn't have lots of chance to access the internet. However, I received a phone call from a guy in UK and asked me if I know why the TWF was down. "Maybe the earthquake, you know, on May.12 there's 8.0 level earthquake occured in China Sichuan, and more than 32,476 people died so far." I told him,"you should call Anthony for the reason."

It is good to see the forum back to normal.

Only have gone through the hardships, will we become more strong!

[nocturnal]

Originally Posted by cbeebies

I was on a trip out of the town for 10 days and didn't have lots of chance to access the internet. However, I received a phone call from a guy in UK and asked me if I know why the TWF was down. "Maybe the earthquake, you know, on May.12 there's 8.0 level earthquake occured in China Sichuan, and more than 32,476 people died so far." I told him,"you should call Anthony for the reason."

It is good to see the forum back to normal.

Only have gone through the hardships, will we become more strong!

I'm sure I speak for many in saying our thoughts are with the people of China at this time.

Anthony good to have the forums back good luck with the data retrieval.

[Anthony]

Brief Downtime - 5AM-9PM - Tuesday 20th May

We had a small period of downtime this morning (our American friends won't have noticed). I had a RAM upgrade scheduled for 5am GMT this morning, this was successful but upon reboot a few services failed to start. It wasn't until this morning that I could alert the support department, thus it required a reboot of all services.

The server should be quicker than ever before as we now have twice as much ram as we did with the last server. We went from 1GB to 2GB. (Please let me know if you notice a performance change over the coming days).

That should be all of the teething issues out of the way with the new server and we can all return to normal service. (Fingers crossed eh )

Thanks.

[cbeebies]

Now, I have to wait more than 4 minutes for the page to show up. The new server's response time is extremly slow today.

I don't have much patient. (doh)