There's a
Sarah Brightman song with the lyrics:
There's an ocean between us.
You know where to find me.
You reach out and touch me.
I feel you in my own heart.
More than a lifetime
Still goes on forever.
But it helps to remember
You're only an ocean away.
Back in SF, we rented a 600sf condo because it was a minute away from the office and 5 minutes away from the
data center. At $1800/mo, it by no means was cheap but compared to larger places in the area at $3000-$4000/mo, it was what we had to make do with. In the beginning, our servers and software was held together with duct tape so being close was imperative. Not only did we need to keep our production servers up for our customers to access but our office machines were just as important for us to process data during the night and over the weekend. Because we didn't have the money for the expensive enterprise hardware and software, we had to compensate with extreme dilligence and I routinely made midnight runs to troubleshoot computers.
As we've grown though, we've dramatically improved our infrastructure. Multiple servers, multiple data centers, real-time multi-master replication, local load balancing, global load balancing, remote power cycling, super-redundant RAID, tested admin tools, better release process and so on. If we didn't periodically make stupid brainfarts, hitting the 99.9999% uptime benchmark would be within our reach.
Once in a while though, we still have problems with our office servers which for cost reasons are not at the same infrastructure level as our production machines. Last weekend, it was a power outage that lasted longer than our UPSes could handle and one important server was stuck at a 3ware BIOS warning screen. Ordinarily, I would be driving towards the office ASAP to check on the servers in case we had pressing data processing jobs still in the queue. But being an ocean away, all I can do is pull up some Sarah Brightman, chill out and wait for the IT staff to come in on Monday.
To be honest, it's a change I need to adjust to. Even being business owner, work is still just work -- it's not life-or-death. Considering how the poor track record of our competition, the panic is mostly in our own minds -- we're competing against memories of ourselves in startup mode. And yes, the IT staff came in Monday, got the computers up within an hour and no real issues arose from the one day downtime.
(Filed in philosophy, technology)
Downside of being an ocean away
Posted by Mossy
April 19, 2010 10:53 AM
Back in SF, we rented a 600sf condo because it was a minute away from the office and 5 minutes away from the data center. At $1800/mo, it by no means was cheap but compared to larger places in the area at $3000-$4000/mo, it was what we had to make do with. In the beginning, our servers and software was held together with duct tape so being close was imperative. Not only did we need to keep our production servers up for our customers to access but our office machines were just as important for us to process data during the night and over the weekend. Because we didn't have the money for the expensive enterprise hardware and software, we had to compensate with extreme dilligence and I routinely made midnight runs to troubleshoot computers.
As we've grown though, we've dramatically improved our infrastructure. Multiple servers, multiple data centers, real-time multi-master replication, local load balancing, global load balancing, remote power cycling, super-redundant RAID, tested admin tools, better release process and so on. If we didn't periodically make stupid brainfarts, hitting the 99.9999% uptime benchmark would be within our reach.
Once in a while though, we still have problems with our office servers which for cost reasons are not at the same infrastructure level as our production machines. Last weekend, it was a power outage that lasted longer than our UPSes could handle and one important server was stuck at a 3ware BIOS warning screen. Ordinarily, I would be driving towards the office ASAP to check on the servers in case we had pressing data processing jobs still in the queue. But being an ocean away, all I can do is pull up some Sarah Brightman, chill out and wait for the IT staff to come in on Monday.
To be honest, it's a change I need to adjust to. Even being business owner, work is still just work -- it's not life-or-death. Considering how the poor track record of our competition, the panic is mostly in our own minds -- we're competing against memories of ourselves in startup mode. And yes, the IT staff came in Monday, got the computers up within an hour and no real issues arose from the one day downtime.
(Filed in philosophy, technology)
Leave a comment