I've been trying to steal time to do some long-promised work on dsource.org recently.  It was precipitated by stonecobra donating a server for our dedicated use.  It's a monster, with dual CPU's, 8GB memory and seriously fast mirrored disks.  This will get us out of the virtual host situation that, while an improvement over the duck tape and toothpicks computer and crappy home DSL, still leaves us wanting for more memory.

Along those lines, I would like to be able to not only weather a full site crawl by one of the search engines, but to thrive during it.  This means some heavy caching using Tango.  stonecobra has offered to help, so that's sweet.  We will probably make the interface look a whole lot like memcached so the Django portion of the site is relatively seamless, and then we have to crawl through Trac code for some caching love.

It seems like getting this server has triggered a series of dependencies, but I realize they are of my own doing.  For instance, I refuse to install php / mod_php / phpBB on this server...  That means that the following needs to be in place:

1.  Django registration:  done
2.  Profile management:  almost done (could do it in Django or reuse TracForums)
3.  Replace forums, and port all old forums, topics, posts, watch data, etc.: almost done

Pragma has resurfaced with a bit of prodding from me, and picked up TracForums development again.  We're both working evenings trying to get it polished, and fit into the new dsource software.  Once we do that, we convert all of the existing phpBB forum data into the new db tables, and deploy.  /me drools...

Here's a teaser:

 


Comments

JimPanic

Wed, 26 Mar 2008 13:11:21

Awesome stuff. Keep up the good work and a big Thank You! for the hard work you've already put into dsource.

 

Brad Roberts

Wed, 26 Mar 2008 20:32:19

You really ought to start looking at horizontal scaling rather than vertical scaling. Eventually you'll out grow the new bigger box and might be able to do one more upgrade step, but the costs of big bad hardware is higher than the cost of two smaller boxes in most cases.

Where's the new box hosted? Does it have redundant network routes? Good power redundancy? Quick availability of replacement hardware for when it dies? These are the hidden but important benefits of using a big hosting provider.

Later,
Brad

 

Brad Anderson

Wed, 02 Apr 2008 19:54:36

Hi braddr,

I agree with you on horizontal scaling, rather than vertical. Certain steps are going to be taken along that front, including using Tango Cache that can span many boxes. This will hopefully give a needed boost to some Trac internals. With the cache layer, it will hopefully matter less which box the request goes to, after solving the session issue, that is. Kris & stonecobra are believers in authenticating each time, as well as the entire server portion of the site just serving up data (json maybe?), but I'm thinking that new hotness is for a re-write in the future some time... ;)

Cheers,
BA

 



Leave a Reply

Name (required)
Email (not published)
Website