Why does OSQA spend so much time rendering templates?

I tracked down some of the slowness in my templates, but I don't think it's the same as yours. Still, this information might be useful to someone else.

Firebug reported about 0.7 seconds to display the main forum page. That seems very long. I copied SocketServer.py into my top-level directory so I could edit it and override the system implementation. My change was an

 import cProfile

and in serve_forever:

    counter = 0
    while self.__serving:
        # XXX: Consider using another file descriptor or
        # connecting to the socket to wake this up instead of
        # polling. Polling reduces our responsiveness to a
        # shutdown request and wastes cpu at all other times.
        r, w, e = select.select([self], [], [], poll_interval)
        if r:
            cProfile.runctx("self._handle_request_noblock()",
                            {}, dict(self=self),
                            "request.%d.stats"%counter);
            counter += 1
            #self._handle_request_noblock()

Basically, each request gets profiled, with the results in its own file.

I used pyprof2calltree to turn that into something kcachegrind can use

pyprof2calltree -i request.48.stats
kcachegrind request.48.stats.log

and clear as day, posix.listdir and stat overwhelm the response time.

I moved OSQA from the network filesystem to /tmp and Firebug reports 0.25s for the main page. With no obvious hotspots that I can see in the profile report.

Templates are very complex stuff, the text have to be processed/compiled into some machine representation and then rendered, django 1.2 introduces template caching, but that won't affect much, the big bottleneck is the same as in most web applications. Database calls.

Most of the data we pass into templates is not the complete data, but rather queryset objects, which are representations of lazy loaded lists. Only when you iterate through them for rendering, the query is performed. Also, stuff like {{ question.author }}, the attribute author is not there yet when you request it in the template, but is lazy initialized when needed.

And if you're wondering if memcached is being used, since you use django toolbar do the experience. Turn off cache, render the main page, see how much queries were executed. Turn it back on, render the main page again at least twice (the first so that things get cached). See the difference?