Hello friends and site admins,
We're back online and things seem to be running well!
1st, let me apologize of course for the downtime. There are many reasons for it that seemed out of my control, but no excuses or passing the buck...we (mostly I), need to do better. As a service provider, it is our team's/my job to figure out everything and make sure the sites run smoothly. I can confidently say that there has been no shortage of effort on my part, but it seems like 1 problem happened after the next, and I could not solve them as fast as I wanted to. Fortunately, I finally feel like everything is back...I hope faster than it was before. Some days I feel like this site takes 2 steps forward and 1 step back...and sometimes I feel like the equation is flipped.
Unfortunately, some sites have moved to other services, but not that many. Looking at the features alone, the sites that moved I think are not gaining anything (and with many providers they are compromising functionality), so the positive is that our service remains very competitive...the negative is that reliability is something that we need to solve. While I can understand how some sites need reliability greater than we have been able to provide in the past, I am motivated and will work hard to make those sites regret their decision to move.
No hard feelings, I just want everyone to know that I have been and continue to be committed to improving things here. When people vote with their feet, I am hurt, but mostly fired up to redeem the reputation of myself and this service.
I REALLY can't wait until I can start posting feature updates, it seems like for the longest time I have been in reactive mode, responding to issues and posting fixes to the infrastructure and spam issues that have been popping up regularly. Only time will tell if I am able to post the feature updates that I have been working on for a LONG time, but it is my plan to post major feature updates sooner rather than later.
In summary, this is what happened:
1. Our old datacenter had a dispute with our contract and unplugged 12 servers and held them hostage
2. The unethical and possibly illegal aspect was the fact that they never informed us of the dispute in writing and did not respond or give us an option to pay (until 4 days after they turned the servers off)
3. I was at a friends wedding when the servers went offline at 4:30pm on Friday
4. I was leaving the next day for the World Cup (to meet my brother and a friend in S. Africa)
5. When I arrived in Cape Town, I immediately went to an internet cafe and probably spent 8-12 hours everyday coordinating with people in Los Angeles to move the servers to a new datacenter
6. I double paid for the month of July, plus had to pay several thousand dollars in unexplained fees to the old datacenter just to be able to take our own servers away from their custody
7. The move was understandably very rocky since I was halfway around the world
8. After fixing some issues with the firewall, dns and general application issues that emerged, we were back online but getting hammered by spammers
9. Lastly, it took me 3+ additional days around the clock to come up with a larger fix for the spam bombardment...this brings us to the "New Anti-spam ip verification"
About the ip verification process:
http://pss.invisionplus.net/?mforum=pss&showtopic=13671 (feedback welcome)
I believe everyone should have seen the new changes. For now, every user must complete a captcha form either daily, or sooner if the ip address gets flagged as a potential spammer. This solution thus far has brought the server load down by a factor of 95%, making things fast and seemingly stable. Time will tell of course how good this new change performs, but so far so good.
Questions:
1. Can everyone now access all of the sites?
2. Are the sites usable and fast enough?
3. Is anyone experiencing anything weird?
I know there are some pending admin requests...I plan to get to all of them by the middle of next week, hopefully sooner (like Monday), but it depends on how the servers perform.
Thanks for those people who are patient and understanding. Like I said, it is of great personal importance to me...including a major source of personal self worth...to be able to restore this service and take it to greatness, so have faith and I will work very hard to prove the critics wrong about Invision Plus.
Thanks again to the staff for remaining helpful, patient and supportive!
Regards,
Scott (TeamIP)
http://pss.invisionplus.net/?mforum=pss&showuser=53486doug
Author
