but we emailed gandi and they fixed it so freakin fast omg. thanks gandi!
asheesh emailed kuri-san in japan to ask him to reboot the machine.
parker and asheesh plan to move everything to the gandi VM jan 1st-3rd.
Everything came back this morning, after the computer in Japan was re-set and Asheesh flipped a switch to re-spawn all our services.
Hi dear readers,
The physical computer in Japan on which freeculture.org is hosted seems to be unavailable. We can’t even ping it, which means we don’t know what’s wrong or how long it will take to fix it.
For that reason, we’re taking this chance to move websites to a new server. You can chat with us about the move using IRC, on irc.freenode.net #freeculture .
We hope to have things working and stable soon (hours to days) and will keep the blog informed of major updates.
Update 2010-06-25 23:32 pm (US/Eastern): The latest backups we have: Some databases and all files: 2010-02-04. Remaining databases: 2009-05-02. You can imagine I’m rather disappointed with myself. The filesystem where the data are stored has RAID-5, so there’s a good chance that there’s no catastrophic data loss. It’s hard to say right now. My plan for now is to see if we can get the machine rebooted within a day or so, and once that happens see how the disks are. If websites aren’t back online by the end of the weekend, then I’ll start restoring from backups. As for email, I will be setting up a secondary MX that will receive email, so at least they won’t bounce.
Update 2010-06-26 01:18 (US/Eastern): We have a secondary MX, so email should not bounce. I’ll queue it up in Minnesota.
Kuri in Japan graciously rebooted rose, and everything is coming back to life.
Once the DNS changes propagate across the world, everyone should see good old glorious freeculture.org.
We’re working on moving from rose to somewhere more reliable.
Update 8:41 PM US/Eastern: conference.freeculture.org is online-ish, and we’ve requested a reboot for the rest of the services. Hopefully it will happen Monday morning Japan time, about 24 hours from now.
Dear freeculture.org fans,
freeculture.org is down. The server that hosts the freeculture.org virtual machine is unresponsive, probably due to high load.
I’m looking into the problem. Because the server is so unresponsive, I don’t have an estimate yet for when things will be back online.
My top priority is getting a conference site back up. Then I’m going to investigate our backups. It seems that they weren’t running nightly, but I know they had been running before so at worst we should have old backups we can use.
To chat with us interactively about how things are going, join #freeculture on irc.freenode.net.
Update, 3:25 PM US/Eastern: We’re moving conference.freeculture.org over to a box in Minnesota. I looked in my email logs and discovered the heavy load began at 9 AM US/Eastern today (Sat Jan 23).