How can I import data from another question and answer script

Question

I have http://zinf.org:8082/answers up and running working great!!!. Now we would like to be able to import data slowly from our old qna software to osqa. In php that would be a snap for me to write... but um this python stuff is putting me at a disadvantage to be sure. Anyway if there are any docs on how to import from any other system into osqa I could use that to hack together some sort of a routine to insert questions and answers until the migration is complete. I just don't want to import the data and find out later some field in some other table I thought was unrelated was actually critical etc. So seeing any working code to add a question / answer would be fantastic.

Thanks in advance, Doug

Answer 1

4

Well, this could be a subject for an entire book :) I can surely give you some hints, but I need some more info. Here are some random questions:

How is the data from the old site available? Is it on an xml file, on another database, text files, etc.
Do you really want to move everything question by question, or would it be better to do it in a single shot?
What about users tags and stuff? Is that data available in the other script? How close do you think the schema relates?

But in a short version, you should use the action classes (in the core code) to insert the content, to ensure that everything will be populated correctly.

EDIT (Tips start here :)

So first of all I'm guessing you will be importing users. You can start by this and import a few.

The best way to run a python script while having access to your osqa site resources is through a management command. So in your osqa installation folder, under forum/management/commands, there is a script named sample_command.py, copy that script to the same folder with some name, for example, import_user.py, then you'll be able to run it with:

python manage.py import_user

So, then edit the script, and start by importing the necessary classes into it:

from forum.models import User
from forum.actions import UserJoinsAction
import datetime

Then inside the handle_noargs method, you need to to something like this:

#load a user data from somewhere, you can start by hardcoding a user just to test
username = 'someuser'
email = 'someuser@somedomain.tld'
#and if you want to preserve membership dates
join_date = datetime.datetime(<year>, <month>, <day>,[optionaly: <hours>, <minutes>, <seconds>)

#then create the user
user = User(username=username, email=email, date_joined=join_date)
user.email_isvalid = True
user.set_unusable_password()
user.save()

#it's important that you then instruct your users on how they can login then

#now make sure the user is correctly created issuing the proper action

UserJoinsAction(user=user).save()

Now run the script and check if the user is correctly imported. If that works for you, then everything else is much simpler, and I can give you some more hints once you make it work :)

So basically, you need a way to load the data, if you have the patience to do so, just hardcode it in the python code, otherwise you can use xml or json, or some other format you're confort with or that your current script can produce.

Loading questions and asnswers

Pretty much the same thing, you'll need to create an AskAction and AnswerAction, for example, asking a question:

#retrieve the "asker" from the database
user = User.objects.get(id=<someid>)

prepare question data
qdata = dict(
   title = "How can I do stuff?",
   body = "I have stuff to do, just don't know how.",
   tagnames = "tag1 othertag supertag",
)
save the question, everything will be handled internally,
like creating the tags if they don't exist, etc
AskAction(user=user).save(data=qdata)

This should save your question, don't forget to import all the necessary stuff, for example:

from forum.models import 
from forum.actions import

To add answers is pretty much the same thing, except you don't use a tite or tagnames, and have to point which question it refers to, for example:

#retrieve the "answerer" from the database
user = User.objects.get(id=<someid>)
prepare question data
adata = dict(
   body = "I have stuff to do, just don't know how.",
   question = Question.objects.get(id=<questionid>),
)
save the question, everything will be handled internally,
like creating the tags if they don't exist, etc
AnswerAction(user=user).save(data=adata)

Those error you were seeing were probably osqa trying to send emails, so while you're in this proccess, you can edit /forum/startup.py and comment the line:

import forum.subscriptions

And that's pretty much it for questions and answers.

link

answered 20 Jul '10, 19:41

Hernani Cerq... ♦♦
16.8k●6●59●75
accept rate: 52%

edited 24 Jul '10, 13:12

Thanks for the tips. Yes I need to go question by question because making a huge change all at once can be detrimental to your traffic. We want to migrate gradually and see the impacts on traffic as the urls transition, etc. I could have the best treasure in the world but if people can't find it, it does them no good. So really I am looking for an example of some type of import of questions done before. An example import would be great. .. or even a way to add a question. Yes I know to use the classes, but an example really helps. I can put the data in whatever format is needed.

(20 Jul '10, 22:19) doug99

See the edited answer.

(21 Jul '10, 06:34) Hernani Cerq... ♦♦

Wow that code works! So awesome. It does however give this error (even though it works):

Exception in thread Thread-1 (most likely raised during interpreter shutdown)

(21 Jul '10, 10:23) doug99

Well i have been trying make code to add a question by myself but I am failing miserably. I dont know what a post is vs an answer, etc. The example code in sximporter.py is seems so interwoven that I can't pick out what I need or not. More sample code please....

(21 Jul '10, 15:39) doug99

I am still stuck adding a question. I can't get it to work. The code to add a user works great though. Please help!

(22 Jul '10, 10:13) doug99

Sorry, I've been a bit busy, I edited the answer, please check it.

(24 Jul '10, 13:13) Hernani Cerq... ♦♦

First I want to thank you for taking the time to write the code above! I have tried the code for adding an answer as many ways as I can. I always get stuck with this error:

print "before"
AskAction(user=ruser).save(data=qdata)
print "after"

PRINTS

before
Unknown command: 'import_q2'

So it dies somewhere in the askaction, and since I don't yet understand python very will I don't understand what the heck its talking about unknown command import_q2 when it clearly exists and is running it. I have to say this language, after 25 years programming and umpteen languages, is the most unintuitive language I have seen. (sorry for the rant).

(26 Jul '10, 21:46) doug99

Well, don't blame the language to much :) So can you make a pastebin or something of the entire code?

(26 Jul '10, 22:10) Hernani Cerq... ♦♦

from django.core.management.base import NoArgsCommand from forum.models import Comment

from forum.models import *
from forum.actions import *
import datetime

#I GET THE ERROR WITH OR WITHOUT THE FOLLOWING LINES COMMENTED OUT
#class Command(NoArgsCommand):
#    def handle_noargs(self, **options):
#        objs = Comment.objects.all()
#        print objs

#retrieve the "asker" from the database
ruser = User.objects.get(id=18)
print ruser.username
#prepare question data
qdata = dict(
   title = "How can I do stuff?",
   body = "I have stuff to do, just don't know how.",
   tagnames = "tag1 othertag supertag",
)

#save the question, everything will be handled internally,
#like creating the tags if they don't exist, etc
print "before"
AskAction(user=ruser).save(data=qdata)
print "after"

(26 Jul '10, 22:24) doug99

First of all you can drop that comments part :) That's just a sample. Also, are you indenting the code correctly?

(27 Jul '10, 06:41) Hernani Cerq... ♦♦

I believe so. As per the code above the only lines indented are the ones where qdata variables are being set. Does this code work for you? What svn version are you running? (I am on 550). Also I should mention again that it prints "before" and not "after" when this code is run, so its getting lost somewhere in the askaction. The only other place I see askaction used is in writers.py and the code looks very similar. I have been wondering of its some kind of path error or something because I have to run this code in a funny way because I had to install a second copy of python to get version 2.6. Thats why I was wondering if it runs on your installation. If so then I know its related to my install. Perhaps path settings, etc. Thanks, Doug

(27 Jul '10, 08:01) doug99

So the correct indentation would be that all ruser and qdatc code, etc, would have the exact same indentation as for example, the objs = Comment... sample code. So in the end it should be something like this:

from forum.models import *
from forum.actions import *
import datetime

class Command(NoArgsCommand):
    def handle_noargs(self, **options):

#retrieve the "asker" from the database
        ruser = User.objects.get(id=18)
        print ruser.username
        #prepare question data
        qdata = dict(
            title = "How can I do stuff?",
            body = "I have stuff to do, just don't know how.",
            tagnames = "tag1 othertag supertag",
        )

#save the question, everything will be handled internally,
        #like creating the tags if they don't exist, etc
        print "before"
        AskAction(user=ruser).save(data=qdata)
        print "after"

(27 Jul '10, 08:49) Hernani Cerq... ♦♦

Python is picky about this stuff, so you need to be carefull. And yes, for me it works perfectly, the error you got is very strange indeed :)

(27 Jul '10, 08:50) Hernani Cerq... ♦♦

Then it must be install related. Now I can focus my efforts on paths etc. I have to run it with /opt/ActivePython-2.6/bin/python /opt/OSQA/manage.py import_q2 because I need to run python from my python 2.6 install.

(27 Jul '10, 09:48) doug99

Importing a user works great, but importing a question failed.

I got error using the following code to import a question.

u = User.objects.get(id=1)
question = dict(
  title = "This is the question title",
  body = "This is the question body",
  tagnames = "question",
)
AskAction(user=u).save(data=question)

... File "/home/sam/osqa/forum/actions/node.py", line 22, in process_data question = Question(author=self.user, self.create_revision_data(True, data)) File "/home/sam/osqa/forum/actions/node.py", line 8, in create_revision_data revision_data = dict(summary=data.get('summary', (initial and _('Initial revision') or '')), body=data['text']) KeyError: 'text'

(05 Aug '10, 00:50) bbpan

@Hernani, I used the exact code you provided, the error always occurrs at AskAction(user=ruser).save(data=qdata)

(05 Aug '10, 01:54) bbpan

I got it figured out. Instead of 'body', it should be 'text'. data['text'] is referrenced in NodeEditAction.create_revision_data().

question = dict( title = "This is the question title", text = "This is the question body", tagnames = "question", )

(05 Aug '10, 04:04) bbpan

Ah, my bad then, I wrote the code without testing it.

(05 Aug '10, 09:26) Hernani Cerq... ♦♦

showing 5 of 18 show 13 more comments

How can I import data from another question and answer script

prepare question data

save the question, everything will be handled internally,

like creating the tags if they don't exist, etc

prepare question data

save the question, everything will be handled internally,

like creating the tags if they don't exist, etc

Follow this question

Learn more about OSQA

Related questions