Edited by author 04-27-2006 09:07 AM
That makes a lot of sense as wikipedia seems to be updated just about every few minutes.

I finally got my code requesting and parsing pages from ifsdb so now I have to learn to parse and clean up wiki entries (which are untidy code layouts IMHO) so that I can first:

a. Dump the table that I want in the format that I want
b. Use my code to update the notes field with my now working code.

I also wrote a MySql query passer (yes, passer, not PARSER, as it does no checking) so that I can dump data from tables fast from MySQL server and then straight to disk in the format that I want. Easier than using the interactive terminal mode!! And if it weren't for the over 1 meg MySql dll, my code would be around 25k compiled. I love my compiler!!!


I am having fun and learning a lot. Thank you Al and Rafe for your kind help and your patience.

This query will give the author fields and the note in a single query:

SELECT t1.*,t2.note_note FROM authors AS t1, notes AS t2
WHERE t2.note_id=t1.note_id AND t1.author_canonical='Robert A. Heinlein';

It should be noted ( :) ) that the notes field is somewhat depricated - it's not displayed in the ISFDB anymore as we rely more on Wikipedia for that data. So the notes that you find probably haven't been updated in a couple of years.

Sorry to be a nuisance again but could any of you show me how to do a nested query to use a third form normal search? I can do single form but don't know how to nest queries for third form.

Select * from authors where author='jon grisham'

I.e. How do I get the note data from the note table for 'John grisham' into that query as part of the returned data since the note field in aithors is a link to a third normal form?

Yes, I will be able to learn from the wise masters:):)

Edited by author 04-18-2006 08:38 AM
The database will be static, a reference only. I won't actually be undoing the linksin the original, just creating a table for myself from the database that will include data in the fields that now only includes links. Will be doing lots of studying!!

Before you undo the links, it's probably best to understand why they are there - there are very good reasons for using links rather than the actual data. It may of course be best for your purposes to do without them, but it's a good idea to know why you're doing it.

If your database is static it may well be OK, but if you are updating it then it is *much* easier to keep it consistent if you use the links.

I don't know how much you already know - if you're not familiar with the theory searching on "third normal form" will find you a lot of explanations of this.

THank you both for your help and advice. As I lack the experience and expertise to take this further, i'll stick to the way I am doing it now. At least it works:):)

Eventually, I want to convert the offline databse to sqlite again and merge various tables so they are full of data instead of links. First, I have a lot of learning to do!!!

Still, my little library book manager is beginning to look less useless than it was (ROFL)

Actually, if you're parsing the generated HTML pages, you'd be better off using the XML-based REST API... which isn't fully constructed or public yet. But if you have a specific API in mind, let me know and I'll whip one up that you can test out. (I currently only have two apps - one that returns publications using ISBNs, and one that submits data changes).

As Rafe pointed at, title.cgi uses the title_id as its argument, as that is 1} unique, and 2] immutable (title names *do* change over time as they are edited). If you want to obtain a list of records by title name, it would roughly be:

def SQLloadTitle(titlename):
        query = "select * from titles where title_title='%s';" % db.escape(titlename)
        db.query(query)
        result = db.store_result()
        title = result.fetch_row()
        results = []
        while title:
                results.append(title[0][0])
                title = result.fetch_row()
        return results

Which would return a list of all title_id's by that name. This would not be adequate, as authors sometimes name their collections after a particular short story (for example "Bears Discover Fire), it would also include reviews of the title in question (for example Greg Egan's "Distress").

As far as I cal tell it isn't possible to do it that way.

You would need to have an existing program on the server which would take the title and show that page, and there doesn't seem to be one.

I can't be sure of this though as I'm not clear how recent the code copy I downloaded is, and I never actually got it running. Plus I don't yet speak Python - I'm writing a similar setup in PHP and looked at the ISFDB for general ideas.

However I looked through the files I have, and none of them seem to do what you want.
title.py calls:
SQLloadTitle(title_id)
Whick does select * from titles where title_id = title_id
So it won't work with the text string. ea.py does use the text for the author name.

Unless there's an extra program somewhere or I've missed it on the quick look through I did, that functionality just isn't there. If you can't go via the author page, the only way I can think to do it is to cache the lookup yourself - hold a table with the titles and ids, and write something to accept the title, lookup the id, and call the original title program with the id.

Of course you will then run into disambiguation problems when there's more than one instance of a title.

Actually yes, that's exactly what I'd like to do since I wouldn't know the title id before I do a search for an author and parse the returned page.

Can it be done somehow like the below to directly fetch a title?(Far less fetching and parsing than the way I am doing it now and I know that's not the cgi directive I am using as an example)

http://www.isfdb.org/cgi-bin/title.cgi?the_brethren

(Am also trying to figure out ways of parsing the wikipedia entry to just get the comments for the book but haven't got that far yet (grin))

Could you clarify what you're trying to do please?

It looks like you want to retrieve the ISFDB's page for a particular title in the same way as you have done for the name?

Using a GET rather than a POST, you could use the links as displayed on the John Grisham page you show as an example.

So on that page the first Novel "A Time to Kill (1989)" has a link to http://www.isfdb.org/cgi-bin/title.cgi?183131
and going to that link displays the page for that title.

What extra functionality do you need over that - are you trying to search by the text name rather than the ID number?

Rafe, thanks for your help in the past. I finally decided not to attach the offline database to my code as it was too big for some of my users to handle. Now I am querying and parsing the pages from my code.

Now am trying to figure out what the POST method is for a HTTP query for titles.

If I do:

http://www.isfdb.org/cgi-bin/ea.cgi?john_grisham

I can get the author's page up, no problem there.

But what is the cgi string to retreive a title? Does anyone know please? Thanks all.

fangbeast@gmail.com