I've been trying to steal time to do some long-promised work on dsource.org recently. It was precipitated by stonecobra donating a server for our dedicated use. It's a monster, with dual CPU's, 8GB memory and seriously fast mirrored disks. This will get us out of the virtual host situation that, while an improvement over the duck tape and toothpicks computer and crappy home DSL, still leaves us wanting for more memory.
Along those lines, I would like to be able to not only weather a full site crawl by one of the search engines, but to thrive during it. This means some heavy caching using Tango. stonecobra has offered to help, so that's sweet. We will probably make the interface look a whole lot like memcached so the Django portion of the site is relatively seamless, and then we have to crawl through Trac code for some caching love.
It seems like getting this server has triggered a series of dependencies, but I realize they are of my own doing. For instance, I refuse to install php / mod_php / phpBB on this server... That means that the following needs to be in place:
1. Django registration: done
2. Profile management: almost done (could do it in Django or reuse TracForums)
3. Replace forums, and port all old forums, topics, posts, watch data, etc.: almost done
Pragma has resurfaced with a bit of prodding from me, and picked up TracForums development again. We're both working evenings trying to get it polished, and fit into the new dsource software. Once we do that, we convert all of the existing phpBB forum data into the new db tables, and deploy. /me drools...
Here's a teaser:
Kirk McDonald has been doing a fine job of maintaining the D lexer in Pygments, which is used to highlight syntax in dsource's wiki and code-browser. And he gets some insane turn-around times by the Pygments SVN committers, unless he's one already.
In any case, a mere few hours after Walter had announced the new Traits feature, bringing a bit of compile-time reflection to D, Kirk had the highlighter updated. http://www.dsource.org has been updated with this new version, so start using traits in your code.
I'm planning on getting a plumbing overhaul completed before the D Conference at the end of August. The particulars involve using Django to serve everything on the site that isn't an individual project. This will include the Home, Site, and even Project List pages.
In the past, I've used Trac to do this, and it resulted in some substantial modifications to the Trac codebase. As they try to move to 0.11 and Genshi, I would like to get as much of this custom code factored out. I believe it is my mods that expose the regexp bug in Python's _sre.c file, and causes the dsource server to hang every now and then. So the switch to Django should be fairly positive.
In working with Django again, I remembered what a joy their template language is to use. It doesn't incur the overhead of an XML parser, it supports template inheritance, and you can't call Python code or have side-effects at all. These are good things, imho. In line with this article, and StringTemplate & SGTE.
I also plan on getting rid of phpBB (finally) and moving to Eric Anderton's TracForums plugin in each project. This means that I'll need a new Authn/Authz subsystem and I think I'm going to use Django's. In fact, I'll be using Django 'user' model for auth in the dsource.org Trac plugins, and I might as well use dsource's 'projects' model in those plugins as well. I'm pretty sure it will piss me off when I have to use Genshi instead of Django to do some things in Trac, but whaddayagonnado?
My target is to get something onto the server by the end of July and then get some enhancements to the Projects List page by the conference. We'll see how much the newborn lets me get accomplished ;)
There has been a long-standing issue with the dsource server beginning to eat all the resources. I happen to believe it's in my modifications to Trac that allow dsource to host multiple projects and such, and will get on to fixing that real soon now. Probably upgrade to Trac 0.11 and Genshi while I'm at it.
Maide and I used gdb to track it down to a probable bug in _sre.c, the Python regexp code. It seems that an endless loop happens here, basically hosing the Apache / mod_python process (or if compiled with threads, the thread).
Still, even though we may have found a bug in Python (and we're looking to upgrade to Python 2.5 because the changelog says some work has been done on this front), it's not good to have introduced the code to Trac that exposed this.
So, until I can get some time away from \${dayjob} the issue remains. What to do? How about come up with a brutal hack that works, but is embarrassing. This script basically parses 'uptime' and if the short and medium term usage items are over thresholds, we stop Apache, wait for it to die, and then restart it. OMFG:
#!/usr/bin/env python
import commands
import os, sys
from time import localtime, sleep, strftime
DEV = False
MAX_ATTEMPTS = 24
LOGFILE = "/var/log/restarts.log"
def send_oh_shit_mail():
SENDMAIL = "/usr/sbin/sendmail" # sendmail location
p = os.popen("%s -t" % SENDMAIL, "w")
p.write("To: admin@dsource.org\n")
p.write("Subject: dsource screwed!\n")
p.write("\n") # blank line separating headers from body
p.write(":(\n")
sts = p.close()
if sts != 0:
print "Sendmail exit status", sts
def stop_apache():
result = commands.getstatusoutput("/etc/init.d/apache2 stop")
print result
def wait_for_apache_to_die():
cmd = "ps -ef | grep apache | grep -v grep | wc -l"
count = 2
attempts = 0
while count > 1 and attempts < MAX_ATTEMPTS:
attempts += 1
result = commands.getstatusoutput(cmd)
count = int(result[1])
print "%s - apache instances: %s" \
% (strftime("%a, %d %b %Y %H:%M:%S", localtime()), count)
sleep(5) # seconds
if count > 1:
send_oh_shit_mail()
def start_apache():
result = commands.getstatusoutput("/etc/init.d/apache2 start")
print result
def write_to_log(msg):
f = open(LOGFILE, 'a')
f.write(msg)
def get_nums():
uptime = commands.getstatusoutput("uptime")[1]
nums = uptime[uptime.find("load average: ")+14:].split(", ")
return [float(num) for num in nums]
def main():
try:
short, medium, long = get_nums()
print short, medium, long
if short > 3 or DEV:
if medium > 2 or DEV:
dt = strftime("%a, %d %b %Y %H:%M:%S", localtime())
write_to_log("restarting Apache: %s %s %s - %s\n" \
% (short, medium, long, dt))
stop_apache()
wait_for_apache_to_die()
start_apache()
except Exception, e:
sys.stderr.write("error: %s\n" % str(e))
sys.exit(1)
if __name__ == "__main__":
main()
RSS Feed