I'm planning on getting a plumbing overhaul completed before the D Conference at the end of August.  The particulars involve using Django to serve everything on the site that isn't an individual project.  This will include the Home, Site, and even Project List pages.

In the past, I've used Trac to do this, and it resulted in some substantial modifications to the Trac codebase.  As they try to move to 0.11 and Genshi, I would like to get as much of this custom code factored out.  I believe it is my mods that expose the regexp bug in Python's _sre.c file, and causes the dsource server to hang every now and then.  So the switch to Django should be fairly positive.

In working with Django again, I remembered what a joy their template language is to use.  It doesn't incur the overhead of an XML parser, it supports template inheritance, and you can't call Python code or have side-effects at all.  These are good things, imho.  In line with this article, and StringTemplate & SGTE.

I also plan on getting rid of phpBB (finally) and moving to Eric Anderton's TracForums plugin in each project.  This means that I'll need a new Authn/Authz subsystem and I think I'm going to use Django's.  In fact, I'll be using Django 'user' model for auth in the dsource.org Trac plugins, and I might as well use dsource's 'projects' model in those plugins as well.  I'm pretty sure it will piss me off when I have to use Genshi instead of Django to do some things in Trac, but whaddayagonnado? 

My target is to get something onto the server by the end of July and then get some enhancements to the Projects List page by the conference.  We'll see how much the newborn lets me get accomplished ;)

Claire Alexis joined us on Tuesday, June 26th at 3:45pm.  She was 7 lbs. 1 oz.  Mom and Claire are doing great!

A lot of people are asking me why I'm programming in Erlang these days.  And yes, I'm talking about the ones who are in the tech world, not my mother and grandmother.  Those two are wondering why I don't just use English.

I have read a lot of blog posts and articles recently that led me down this path.  Reddit had been, and may still be, hot on Erlang posts.  But the teaser article on Pragmatic Programmers for Joe Armstrong's book is the one that I think covers the most bases.

The same inquisitive people ask me if Erlang is used in any PROD apps.  Well, does the mobile phone grid for England count?  That's where a 2 million line Erlang program is achieving nine 9's reliability.  For those of you scoring at home, that's 32ms of downtime per year.  Pfft.  hacks...

Making a program just about 32 times faster on a 32-core machine is quite appealing.  Down in the trenches, though...  If Erlang, Yaws (web server), and Mnesia (in-memory and on-disk distributed database) are used to write a Facebook app and the server doesn't melt under all the load, then that's pretty bad-ass.  Just in case, I'll try to put the app in Amazon's EC2 so I can drop another Erlang node into place quickly.  We'll see...

Ooof.  When you run again after a (way too) long layoff, your legs are jello.  It felt good to get the blood flowing again, though.  It's a good 'hurt' and I'm off to do it again today...

There has been a long-standing issue with the dsource server beginning to eat all the resources.  I happen to believe it's in my modifications to Trac that allow dsource to host multiple projects and such, and will get on to fixing that real soon now.  Probably upgrade to Trac 0.11 and Genshi while I'm at it.

Maide and I used gdb to track it down to a probable bug in _sre.c, the Python regexp code.  It seems that an endless loop happens here, basically hosing the Apache / mod_python process (or if compiled with threads, the thread).

Still, even though we may have found a bug in Python (and we're looking to upgrade to Python 2.5 because the changelog says some work has been done on this front), it's not good to have introduced the code to Trac that exposed this.

So, until I can get some time away from \${dayjob} the issue remains.  What to do?  How about come up with a brutal hack that works, but is embarrassing.  This script basically parses 'uptime' and if the short and medium term usage items are over thresholds, we stop Apache, wait for it to die, and then restart it.  OMFG:

#!/usr/bin/env python

import commands
import os, sys
from time import localtime, sleep, strftime

DEV = False
MAX_ATTEMPTS = 24
LOGFILE = "/var/log/restarts.log"

def send_oh_shit_mail():
  SENDMAIL = "/usr/sbin/sendmail" # sendmail location
  p = os.popen("%s -t" % SENDMAIL, "w")
  p.write("To: admin@dsource.org\n")
  p.write("Subject: dsource screwed!\n")
  p.write("\n") # blank line separating headers from body
  p.write(":(\n")
  sts = p.close()
  if sts != 0:
      print "Sendmail exit status", sts

def stop_apache():
    result = commands.getstatusoutput("/etc/init.d/apache2 stop")
    print result


def wait_for_apache_to_die():
    cmd = "ps -ef | grep apache | grep -v grep | wc -l"
    count = 2
    attempts = 0
    while count > 1 and attempts < MAX_ATTEMPTS:
        attempts += 1
        result = commands.getstatusoutput(cmd)
        count = int(result[1])
        print "%s - apache instances: %s" \
            % (strftime("%a, %d %b %Y %H:%M:%S", localtime()), count)
        sleep(5) # seconds

    if count > 1:
        send_oh_shit_mail()

def start_apache():
    result = commands.getstatusoutput("/etc/init.d/apache2 start")
    print result

def write_to_log(msg):
    f = open(LOGFILE, 'a')
    f.write(msg)

def get_nums():
    uptime = commands.getstatusoutput("uptime")[1]
    nums = uptime[uptime.find("load average: ")+14:].split(", ")
    return [float(num) for num in nums]


def main():
    try:
        short, medium, long = get_nums()
        print short, medium, long

        if short > 3 or DEV:
            if medium > 2 or DEV:
                dt = strftime("%a, %d %b %Y %H:%M:%S", localtime())
                write_to_log("restarting Apache: %s %s %s - %s\n" \
                    % (short, medium, long, dt))
                stop_apache()
                wait_for_apache_to_die()
                start_apache()

    except Exception, e:
        sys.stderr.write("error: %s\n" % str(e))
        sys.exit(1)

if __name__ == "__main__":
    main()

Um, hi.

So it's mid-2007 and a self-proclaimed geek like me is just getting a blog?  What's up with that?  What the hell have I been doing?  Well, there have been a few things...

At Mirus, and at Sankaty before being acquired, we are building a SaaS Business Intelligence company.  I'm not sure when you stop calling yourself a startup.  I guess you stop when you're making your own money.  We've been doing that (only took 8 years), but maybe you drop the startup tag when you're making enough to pay everyone closer to market value than the sweatshop wages you have been paying.  In any case, the experience has been phenomenal, from technology to corporate structure and finance, I have a wee bit to offer the world now, and hope to share intermittently in this blog.

My sidebars have been investigating new programming languages.  I came from the VB world, and was elated to 'get out.'  We web-enabled the product above with Java, and that was okay for the most part.  I briefly studied K&R to learn C, but then started to hit some languages that I really enjoyed.

 * the D Programming Language - I liked it enough to start the language's version of sourceforge.  See http://www.dsource.org.  I will also be speaking at the first D Conference.  Gotta think of something interesting to say...

 * Lisp - quite the eye-opener.  We attempted to rewrite Mirus' BI query engine in Lisp, but my developer's minds weren't as twisted as mine.  They felt more productive in Java, and I got the usual pressure from the higher-ups that it's easier to bring people off the street with Java skills than Lisp.  Pfft.

 * Python - great language, and I've totally dissected the Trac codebase while learning it.  Dsource uses Trac and a lot of custom plugins to operate all the projects it hosts.  Django is pretty bad-ass as a web framework as well, and I hope to contribute some code (enhanced db backend) to that project real soon.

 * Erlang -wow, this language has just been sitting there, waiting patiently with all its goodness, for someone to come along and do amazing things with it.  That someone will hopefully be me ;)  Concurrent, fault tolerant, distributed, functional...  /me drools.  So far, there's one web server, Yaws, that can kick the crap out of Apache.  I'm wondering if Erlang is as well suited as it appears to be for a cometd server.  That'd help out some people with traffic between their servers and the browser, eh?  All while saving them hardware moolah because of the stupid amount of processes Erlang can run.  Maybe if I do this for someone, they get to pay the developers more?  Maybe if I use it myself, I get to break-even faster?

Finally, and most importantly, the family has been growing over the past few years.  Nora is almost three, and she gets a baby sister in about two weeks.  gulp...

Wow, so I'm new to blogging, and this could have been divided up into 18 different posts.  That's coming...  I just had to braindump to get started.

Cheers