Steve suggested that I crosspost some thoughts about the differences between the the conversational model (ThreadNeedle) and the topic-based model being discussed in RidiculouslyEasyGroupForming Networks.

They are both good, they aren't the same, and they have different strengths and weaknesses.

Threadneedle is better for aggregating a human conversation, whose topic meanders under a named thread.

Topics are great for aggregate blogs that assemble posts about city events, or hiking trails or other specific subject.

A topic-focused blog won't get you a human conversation (that would be ai-complete). A human conversation won't get you a subject-organized index (not without editing after the fact).

The original post and some more thoughts on the topic are here.

Edited by author 01-12-2003 04:26 PM
Reasonable concern; a tool that takes a savvy person an hour to install probably won't get critical mass adoption yet.

The LazyWeb seems to be working pretty well, though.

* Nice plug-ins get integrated into MovableType, Radio, and other tools.
* Hosted services help those who want less complicated software they don't have to install.
* Bookmarklets provide one-click access to blog features.

If ThreadNeedle works, even as a demo for the cogniscenti, the LazyWeb will move the project forward.

"With enough eyes, all features are shallow" -- Clay Shirky

Adina, I agree that something that approximates real-world social networks, in that a disrupter can be ignored by the group who adhere to their conventions, is useful.

But that's not the problem I see here. My concern is that someone who'd ordinarily be welcomed into the conversation can break the thread by not adopting the technology. Even people responding to her who use the technology can't repair the thread -- every link is important. So in a situation like this, I think it's essential that the technology be extremely easy to adopt for the non-technical person.

Shelley, does the blogthread engine capture only trackback entries or can it also be configured to pick up comments?

Could a comments engine be modified to support thread discovery also?

- Adina

Steve, I think this may be a feature, not a bug. If a post in a thread is pathological (trolls, flames), it stops propogating. Threads die when participants stop participating.

Individual-driven moderation is a core difference between blogconversation and usenet.

This may have an opposite result from usenet anarchy. Usenet's openness makes conversations vulnerable to its most aggressive and crazy members. Blognet's decentralized control may make conversations more controlled, but more closed, because it's easy to ostracize pathological posts, or posts expressing strong differences of opinion.

More like village, for better and worse.

I haven't yet read all the way through the posts on Shelley's proposed architecture to understand exactly how this would work.

> If people don't want the functionality enough, then they
> don't need to use it.

The thing is, I think we (blog authors) are assuming that it's really the *readers* of the blogs that want the blogthread function. And these readers are foiled if any of the bloggers in a chain doesn't use it.

> As to the question about how will people see this stuff? There's
> two specialized RDF-to-XHTML pretty printers (one for PostCon
> and one for ThreadNeedle) that someone calls passing the
> appropriate rdf file as part of the URL. These pretty printer
> pages (PHP to start, then JSP, eventually CGI) display the Post
> Content information or the ThreadNeedle thread. How this page is
> called is up to person -- they can add a link within a template

It sounds like blog readers will usually need to click a link to separate pretty-printed XHTML page to see if there are any blogthread links. Yikes.
I'm monopolizing as usual here. Any comments from the other seven subscribers?

Steve, this is a modular app -- anyone can write a mighty bot if they want, pull stupid bot, put new bot in.

If people want PostCon and/or ThreadNeedle functionality, they're going to need to read the docs in how to use it, and install it and any necessary libraries. That could take up to an hour. If people don't want the functionality enough, then they don't need to use it. I'm not being defensive here -- just very matter of fact. I'm learning that sometimes you can't please the people no matter what you do, so don't waste calories you don't need to.

Making the bot smarter won't make the install easier. You still have to install the apps, read the docs, make some tweaks.

Now with the SOAP interface and open API, no reason an app can't incorporate the use of basic PostCon/ThreadNeedle functionality; apps such as MT or Radio or whatever; then the users won't have to do anything other than installing the outer app. But I can't tell Ben and Mena or Ev or Dave that they have to incorporate this.

(It wouldn't take much for anyone familiar with the blogger api and Radio's stuff to be able to wrap this. Still, someone will have to download it, install it, and read the apps for how to make it work, wrapped or not.)
As to the question about how will people see this stuff? There's two specialized RDF-to-XHTML pretty printers (one for PostCon and one for ThreadNeedle) that someone calls passing the appropriate rdf file as part of the URL. These pretty printer pages (PHP to start, then JSP, eventually CGI) display the Post Content information or the ThreadNeedle thread. How this page is called is up to person -- they can add a link within a template (the rdf file will have consistent naming based on web page with addition of pstcn or thrdndl qualifiers -- 000345.htm, 000345pstcn.rdf,
000345thrdndl.rdf using MT scheme.)

No reason why you all can't modify RSS files to add this in, except that it's outside the scope of RSS. Still have to have bots to traverse the URLS, but it might be easier to get people to modify their RSS templates. If no one uses the ThreadNeedle functionality of PostCon, I'll probably just pull it someday. But it's the PostCon app and file that's of interest to me -- ThreadNeedle is just another consumer of the data.

Still, I'm pretty sure I'll get users of the app. It's not going to be that difficult to install -- no more difficult than putting in a MT plugin.
As for your comment about a bot looking for the markup in the file, some confusion here, which was my doing. I'm not modifying the HTML of the source of the weblog posting or the static web pages. I'm building a separate RDF file. I'll be using MT tags to build this file, possibly, (might be using the MT API directly), but not embedding anything other than links (link to the associated RDF file, and a link to the pretty printer to see the PostCon data if they person wishes). There will be no RDF or markup contained in the XHTML or HTML page, per RDF group recommendations, and per XHTML validation specifications. If you scrape the page, you'll have to use a traditional HTML scrapper, find the post or relevant material and pull the links.
>
>
< replied-to message removed by QT >

Thanks for the further explanation, Shelley.

I have to say I'm concerned about the fact that this requires a non-techie to do an hour's work, and that if not all thread participants do it, the thread breaks. I hope I'm proven wrong, but it would be contrary to my experience with social apps: if a new technology must be used unanimously to be effective, it's very hard to get the necessary momentum.

For this reason, I'd like to at least allow for a backup approach: a "mighty bot" searches the blog content itself for the special markup (the one you'll use to generate the PostCon RDF) and builds the thread. So the smarter the bot can be, the less the average user needs to do technology-tweaking (not everyone gets the same thrill from it that we do :-).

One more question having to do with the special markup: how will readers of blogs know there's a blogthread?

Feel free to defer answers until your book is ready (and good luck with that).

Good questions, Steve.

How's about I finish the app (because I really do have a book deadline next week) and then I promise to post to this thread, and to you directly with references to the actual app as soon as it's done. It's so much better to work from real than vapor, and I've promised this functionality for some time now.

Short answers for now:

You can use one app, two apps, all apps (or should I say components?) -- up to you. There is no dependency except the existence of the PostCon RDF file (and the ThreadNeedle RDF file if a person wants to incorporate TN into their pages). Very similar to PHP XML at Source Forge (which I'm using).
If you're an MT user and you want to generate PostCon RDF files, you'll need to drop a couple of files into your MT installation, and you'll need to add a couple of lines to your individual posting archive file. The issue I'm working with now is how to call the functionality to generate the page -- MT plugins tend to be somewhat read only.

If you want to add ThreadNeedle support, you'll need to drop in a couple of more files, add a couple of more lines to your template, and manually edit a config page. You'll also need to create a CSS style sheet and template for the pretty printers (default ones will be given). This can be done through MT for MT users. The HTTP referer app will need a little additional config. The bot is also included as part of this installation. However, you can access the referer app and bot sourcedirectly if you want (only likely a techie will do this.)

All total: about an hour for a non-techie, much less for a techie. One time setup.

If you want to use PostCon's web-based application for manual PostCon RDF generation, you'll need to download a zipped file and unzip it, and you'll need to password protect your app directory (I'm not building security into first release). Let's say about the same amount of time or less to setup a MT blog, and about that skill level. If you want to incorporate an info link into your web pages to point to the RDF-to-XHTML pretty printer, you'll need to do this manually, or add it as part of a template for a MT type app. The pretty printer has a default look and you'll need to customize if you want something else.

If a referenced URL points to a site that doesn't use PostCon RDF, the bot stops. This is no different than an aggregator not being able to aggregate a site that doesn't support RSS. The bot doesn't try and parse the HTML or anything like that looking for links. I'm not building mighty bot, I'll leave that for someone else.

There's room for HTML-to-PostConRDF scrapers and smarter bots that will search for links et al and XSLT transformers et al.

Shelley

>
>
< replied-to message removed by QT >

Edited by author 12-12-2002 11:46 AM
I'm a little lost among these small apps. Does "Use one piece, use them all" mean "if I use one, I'm using them all" or "you can use one, several, or all of them"?

What do I, the (usually non-technical, remember) blog author, have to set up and use to get blogthreads working for me?

Sorry to repeat: What happens if a link in a blogthread isn't using PostCon or hasn't run a necessary manual process to create the PostCon RDF?

Hi Steve

Yup, you got. And a bit more clarification how this whole thing pulls together (PostCon):

PostCon is really a set of small, lightweight apps that can be used independently. No large framework, no download bandwidth busting and difficult to install and configure app. Use one piece, use them all.

If anything, PostCon is the RDF file -- the apps are nothing more than processing sugar.

One piece is an MT extension that generates a PostCon RDF file for the post when the post is published. And a link to the page is put into the head section for the posting individual page archive. (this is a one time template change -- easy).

This MT extension (integrating into app is still a tiny challenge at this point) will pull the information for the PostCon RDF page from the posting data, including any hypertext links it 'references'. Once the PostCon RDF file is built the MT extension app is finished. At this time, no ThreadNeedle file, and only one level of conversation is recorded.

Now, another app will access this RDF file, access the referenced property and traverse the referenced URL to look for more referers (drill down). If it can't find a PostCon RDF file, it stops. If it does, it continues until it finally hits an end. If an additional level to the conversation thread is found, the bitty bot builds a ThreadNeedle RDF file, automatically, and plops in the info. One threadneedle file for one PostCon file.

Another app accesses a combined format Apache log file for referers and plops them into the referenced by property for the appropriate PostCon RDF file. That's all it does -- listen for referers and update.

Another app (same as the bittybot mentioned above, but given a diff RDF property target) accesses the referenced by url and drills up. Same limitations with drills down -- no PostCon RDF files, it stops. It adds the new info to the ThreadNeedle RDF file.

So far, each is a small independent app which can be easily replaced -- the key is the PostCon RDF file, and the concept of following a specific thread. Another key is that the bot won't try to spider. It will look for PostCon RDF, traverse vertically, and that's it.

Now, another app could be written to follow the ThreadNeedle RDF files, and do the spidering. Much of its job is done by this vertical processing. However, all I'm concerned about at this point is capturing one single thread.

I'm hoping that by keeping the processing simple and stupid, which makes it lightweight and incredibly easily tweaked, propagation of PostCon RDF will occur quickly. And there's nothing more that techies like to do then tweak, and I'm giving them perfect tweaking material.

(BTW, this is all going in SourceForge as open source.)

Continuing into PostCon outside of ThreadNeedle -- another application is my XHTML web page app that generates a PostCon RDF file from manual entries. I'm using this on my existing web sites pages (not the weblog). That's all it does -- maintain the PostCon RDF file. This includes updates if the web resource moves, or is removed or replaced.

The core functionality of this will be exposed as an API (PHP at first, then Java, and then Perl), and will be wrapped eventually in SOAP/XML-RPC. So one could incorporate this functionality into exist apps, such as weblogging tools, content management systems, and so on.

And I have another tiny app that takes this information and adds to a 404 custom error handler (this is already in use at my site) datastore (hashed file), adding a reference to the new location for moved content, and to the RDF file if a page is deleted or replaced. When a 404 occurs, my errorhandler redirects links for moved pages to the new location automatically. It also calls my rdf-to-xhtml pretty printer for a replaced page, resulting in a very detailed info page that provides information about why the page was pulled, and recommendations for replacements, and so on. This is the same pretty printer (again, a separate app page -- one single php page) that prints out the page information if a person clicks the 'info' link I referenced in my earlier message.

(I feel like I'm selling a Sledge-o-matic, but one other app I'm hoping to have done by end of Jan is using Mozilla and composer and MT's SOAP access to create a desktop WYSIWYG app for MT. And because it's Mozilla, it's ported to all the Mozilla environments. Info not to go beyond this list at this time, please.)

All separate small, open source, easily tweaked, prime time language applications -- the most extensive app is the PostCon manual RDF entry application and that's dirt easy to read. A developer or user could use any one piece alone, or in combination.

There is one config file that pulls this all together if someone wants to use all together if someone wants to use the whole gang of apps.

Nothing fancy. Simple. No brain science stuff. And based on the premise that PostCon RDF files will become ubiquitous, at least among the virtual neighborhoods that want to use this.

(Of course, once this all goes open source, better developers than I will increase functionality and add good stuff. I hope.)

Edited by author 12-12-2002 10:53 AM
Thanks for the update, Shelley. I'd like to understand this better from the user's perspective. This sounds like a combination of the approach of /m15 (the user adds a tag before a link to a blogthread post and a crawler picks it up and builds a blogthread database) and the RDF-based approach.

In this case, it's still fairly simple for the user, who adds specialized tags to a posting, which are processed automatically by your MT plugin (or somehow processed using "HTML forms" -- can you explain that more?), to create RDF on the blog author's site, which is picked up by a crawler.

So it sounds like it has the relative ease of the simple approach, combined with the standardization of RDF, at the expense of one-time installation of an MT plugin or periodic manual processing via "HTML forms".

One question on this: "Additionally, the bot follows the referenced by links as they are pulled in and, again, looks for associated RDF pages -- it would grab the referenced by for that page and follow it and so on." Why does this need to be done each time? Is it just for robustness? It seems that the referenced-by nodes would already have been spidered independently. [Correction: the *references* nodes would have been spidered independently. But that raises another question: if referenced-by links are built from HTTP referrer links, what if no readers have followed the link yet? It seems your central thread-builder could have the smarts to build the relationships.]

My biggest concern is that it sounds like usage has to be unanimous in order for this to work. What happens if a link in a blogthread isn't using PostCon or hasn't run a necessary manual process to create the PostCon RDF? If using PostCon is mandatory, it had better be dead-simple to install and use. Even then, some people won't use it.

One final note: once the PostCon RDF pages are generated, any app can use them for any purpose. Same with the associated ThreadNeedle thread RDF page. My hope is that we'll see as many interesting thread readers and aggregators as we do RSS thread readers and aggregators. The premise is identical, except that ThreadNeedle is multi-level, which RSS is not.

Hi Steve

I've basically incorporated Threadneedle into my PostCon (Post Content) application. You can see an example RDF file from same at http://weblog.burningbird.net/chap0606.rdf. First release of the app will be built partially in Perl and partially in PHP. Second version of the app will retain the Perl bits, but replace the PHP with Java.

Note references and is referenced by statements. This is a Threadneedle 'level'.

A file such as this is generated for each of my web resources, either manually through HTML forms, which generate the RDF; or automatically through something such as an MT plugin I'm working on. One file is generated for each individual posting archive page.

The reference statements are pulled from the content, or again added manually with forms. In Movable Type, these are marked by specialized tags that are processed when the posting is published. The referenced by statemetns are pulled from HTTP referers ( or through trackbacks as a possibility).

When the RDF page is generated (but post publishing to not hold), a small bot follows the references URLs and accesses the PostCon RDF page for that page, if one exists. If it does exist, the references are accessed for that page and the bot records the info and continues the traversal, recording all 'references' it finds. This is recorded into ThreadNeedle RDF file generated for the posting/page (if the level extends beyond one level). Additionally, the bot follows the referenced by links as they are pulled in and, again, looks for associated RDF pages -- it would grab the referenced by for that page and follow it and so on. This information is added to the ThreadNeedle RDF file.

My PostCon application has a RDF-to-XHTML pretty printer that takes a PostCon RDF page and prints it out. This is linked to automatically from each web page (click info link, this page opens). The generated page contains a link to a ThreadNeedle page if one has been created (this becomes a new property in the PostCon RDF page). The application also has an associated, but separate RDF-to-XHTML pretty printer for ThreadNeedle RDF content. The page will then contain the ancestors and descendants for any PostCon managed page.

The only thing left at this point is a bot that looks for these ThreadNeedle 'thread' RDF pages, and builds the entire conversation. Not complicated -- one can use weblogs.com or bl.ogs, or just plain wondering bots to look for these. However, a database will most likely be needed to pull in entire conversations.

This all presupposes that each referer and referenced by has one of the PostCon RDF pages for every posting (or web page). With tools to generate these, though, that isn't necessarily a problem. Would just take time, and acceptance.

Anyway, this is the approach I'm working on and one I hope to finally finish by end of this month. I'm using bits and pieces of the code in the RDF book, which I have to finish next week.

Anything new here?

I just noted this via Dave Winer's blog:

http://www.myelin.co.nz/cgi-bin/wcswiki.pl...slyEasyGroupForming

I added my own reference to this discussion there, with my own bias towards my own suggestions here. Feel free of course to add your own, and to modify the BlogThreads definition I put there (it's a wiki).