How can I import 300K+ questions?

Well after digging in mysql logs, I have found that using the following steps, you can import your users, tags and questions by generating an appropriate SQL script.

Users

Since there are a manageable amount of users for my site, I have used the previously mentioned AskAction(user=user).save(data=qdata) python script.

Tags

Before adding questions I chose to add all of the tags belonging to all questions first. Doing tags is the easiest part. After getting your tags into a list or better a set, just use the following python script and do not worry about use counts of the tags yet.

for tag in allTags:    
  fTags.write("INSERT INTO forum\\\\_tag(name, created\\\\_by\\\\_id, used\\\\_count) VALUES ('%s', 1, 1);\\\\n" % tag)

Note: created\\_by\\_id = 1 is for my user. You can get the used ID from auth_user table with the query: Within mysql console:

select id from auth_user where username = 'admin';

Questions

For the question part I will give you the order of inserts to tables and give my sample SQL script but it may not work for your input.

1. First you have to add your question to the table forum_node. We will later update some fileds in this table.
2. Second table you want to use is forum_noderevision. You have to have the latest revision info in this table.
3. After inserting into forum_noderevision, we update active_revision_id of forum_node table.
4. Then we insert into the forum_action table to notify the system that a question has been asked.
5. Besides my questions' own tags, I added their subject as a tag and modified the subject a little. Here I insert into forum\\_tag table my quesion's subject.
6. All the imported tags are correlated with the forum\\_node with forum\\_node\\_tags table.

For each question I wrote the following to an .sql file:

START TRANSACTION;

INSERT INTO forum_node (title, tagnames, author_id, body, node_type, added_at, score, state_string, extra_count, marked) VALUES( '%s', '%s', %s, '%s', 'question', '%s', 0, '', 0, 0);

SET @forumNodeId = LAST_INSERT_ID();

INSERT INTO forum_noderevision(title, tagnames, author_id, body, node_id, summary, revision, revised_at) VALUES('%s', '%s', %s, '%s', @forumNodeId, '', 1, now());

SET @activeRevId = LAST_INSERT_ID();

UPDATE forum_node SET active_revision_id = @activeRevId WHERE id = @forumNodeId;

INSERT INTO forum_action (user_id, ip, node_id, action_type, action_date, canceled, canceled_ip) VALUES(%s, '', @forumNodeId, 'ask', now(), 0, '');

INSERT INTO forum_tag(name, created_by_id, used_count) VALUES ('%s', 1, 1);

-- iterate all tags of this question:
INSERT INTO forum_node_tags(node_id, tag_id) SELECT @forumNodeId, id FROM forum_tag WHERE name = '%s' LIMIT 1;

COMMIT;

After running the question script, you have to update forum\\_tag.used\\_count it will take a looong time because the way mySQL handles COUNT(*) and the way that query is.

UPDATE forum_tag SET used_count = (SELECT COUNT(*) FROM forum_node_tags WHERE tag_id = forum_tag.id);

If you notice anything odd please leave a comment.