>I see this actually has the full information, which may be >the easiest way to work with it if people want the more >detailed info.

Yes... this allows some additional parsing for some fields as well as additional information (like the geocode tag - the county FIPS code in our case - usefull if you are doing GIS stuff).

>Are you aware of an XML format for describing standard, >i.e. >non-alert, weather forecasts?

Not really... just some very informal ones here and there... some discussed in this thread. Was surfing today looking (again) when I stumbled across this thread.

The Weather community has had some very weather specific formats in place for many years (like METAR) that work great if you are an expert in those formats. Several attempts within the community to jump start some xml standards have apparently not seen much progress. That's one reason why we focused on these already existing formats - it made this a low hanging fruit project to help show how xml and weather data can work together.

However, for me, the key is to keep it simple. An all encompassing xml schema for "weather" will be so complex (IMHO) as to be unusable. That's why, in my own mind, I tend to break it up into several areas; forecasts, alerts, current obs, hydro/water, climate, historical, etc.

Also, keep your eye on this: http://weather.gov/ndfd/

NWS forecasts are now being produced for 5x5km grids (not large "zones" that are somewhat based on counties) every three hours instead of every 12 hours. This digital data is mostly raw numbers from which you can build a worded forecast from using a program.

Cheers,

Bob


>Boris Mann
>http://www.bmannconsulting.com

On Nov 12, 2003, at 4:45 PM, QT - Bob Bunge wrote:

> Interesting thread.
>
> Here at NWS we are starting to experiment with RSS feeds:
>
> http://weather.gov/alerts/

Thanks for the information, Bob. This looks like some really useful stuff. It would be more useful to have the entire announcement included in the RSS feed, rather than just the "header" information. I'll note it in the feedback form.

> We are also producing data in another xml format, CAP (Common
> Alerting Protocol), which is more tailored towards the hazard
> messages we are working with.

I see this actually has the full information, which may be the easiest way to work with it if people want the more detailed info.

Are you aware of an XML format for describing standard, i.e. non-alert, weather forecasts?

--
Boris Mann
http://www.bmannconsulting.com

Awesome!

Bob, I must compliment you and the NWS on the work you've done. It's nice to see a data format that is descriptive and useful, finally.

Interesting thread.

Here at NWS we are starting to experiment with RSS feeds:

http://weather.gov/alerts/

We are also producing data in another xml format, CAP (Common Alerting Protocol), which is more tailored towards the hazard messages we are working with.

If you have comments, filling out the survey/feedback form is the best way to ensure your comment is in the official record and will be taken into account when it's time to decide if the rss feeds should become an official product or not.

Bob

Wow. I just had a totally awesome idea, and I think I can pull it off.

I had thought about the problem of decoding raw METAR data. I found WeatherEase Toolkit, which is written in Java, and up until about 15 minutes ago I thought it was the best way to decode METAR.

In one of my classes, we've been talking about grammars, and how they relate to parsing a program. It's all interesting, but we weren't really applying it much in the class.. it would be better off as a project for a class on compilers and interpreters.

Fast forward to tonight. I was randomly poking around the site for Chaperon, a parser which "converts structured text to XML." You define a grammar, and Chaperon will produce XML output based on that. I didn't think about it in the context of how it's used.

Suddenly I put 2 and 2 together, and whammo, I figured it out. I'll write a grammar that describes a METAR report, and let Chaperon parse it. The METAR format is well defined, the problem is the order that report elements come in. Hoefully Chaperon can fix this.

So what do you guys think.. shall I forge ahead into this new territory of grammars, or should I try to shoehorn something like the WeatherEase Toolkit into openWeather?

Tony

I hit 'reply' but not 'reply all', which might have messed it up. Strange. A blog or even a wiki would be good, but I guess I got a handle on the QT so it's all good for now.

On Wednesday, September 3, 2003, at 01:33 PM, QT - Tony Collen wrote:
> Dang, this quicktopic stuff is all messed up for me, so I'll
> resend the message I wrote but the list didn't seem to get:

Should just be able to hit "reply" and it just works.

> openWeather is me!

I almost suspected as much when I hit somewhere by mistake and it gave me a pretty Cocoon error page :p

> I still get tons and tons of weather rss searches, I just wish I
> had all the time to devote to it. I need a week off work and
> school or something.

Put me down in the same request for time off.

> I'm contemplating setting up a weather-rss mailing list in
> mailman, would you guys be interested in a real list by any
> chance?

For now, this Quicktopic seems to meet the needs -- especially with only three people. I sent an email to Chris Heathcote, inviting him to subscribe.

If there is to be a mailing list, there needs to be a common purpose -- like developing WeatherML or something.

And actually, a blog with multiple posting permission might serve just as well...

Dang, this quicktopic stuff is all messed up for me, so I'll resend the message I wrote but the list didn't seem to get:

Boris wrote:

> Since I'm in the process of aggregating, I should enter the
> other service that Chris pointed out OpenWeather.

openWeather is me!

> They serve up RSS as well as "raw" XML, but the XML is pretty
> useless -- for instance, the location (loc) field has the entire
> thing in one big string, instead of semantically tagging with
> city, state, etc. etc.
>
> The units as well are mixed in with data.

I know, I know, it was a super sloppy implementation that I did quickly due to time limitations. Last semester was hell on my body.

I've been busy doing some administrative things on the web server, like switching the main web server to port 80, along with reorganizing the other virtualhosts on this web server. All this is so I can use mod_proxy and point it at Cocoon.

Another thing I need to do is to rewrite my data downloader in Java, since Cocoon recently got a scheduler component, eliminating the need for a crontabbed python script.

Oh, and I might as well integrate that METAR decoder, too :)

I still get tons and tons of weather rss searches, I just wish I had all the time to devote to it. I need a week off work and school or something.

I'm contemplating setting up a weather-rss mailing list in mailman, would you guys be interested in a real list by any chance?

Tony

Since I'm in the process of aggregating, I should enter the other service that Chris pointed out OpenWeather.

They serve up RSS as well as "raw" XML, but the XML is pretty useless -- for instance, the location (loc) field has the entire thing in one big string, instead of semantically tagging with city, state, etc. etc.

The units as well are mixed in with data.

I'm still getting lots of referrer hits from search engines for people looking for RSS weather info.

Chris Heathcote at anti-mega has made a screen scraper (I think?) that uses worldweather.org as its source.

I just wanted to say sorry for being so quiet. I suggested this write before getting on the road for 2+ weeks. Great info so far. Hopefully I'll have some time to respond next week.

I've created a straw-man Current Weather DTD and a matching Sample Current Weather XML file.

Yet another opportunity to "test" QuickTopic's features. As well, I've been snooping around Google and found some interesting links:

Weather Info as RDF/XML -- takes interceptvector data and turns it into an RSS channel. What does this mean? Technically, we could create the XML format and just run it through this -- you can even download the XSLT transformations.

Here's another PHP parser

Another source for feeds from boygenius.com.

XML.org has a list of related mark-ups for weather. Ha. Except that weatherml.com is a site not found...it expires this October...

So, in summary, it looks like there is no open, XML-based, standard way of displaying weather. Let's go!

On Thursday, July 3, 2003, at 04:21 PM, QT - Tony Collen wrote:

> I think I mentioned it before, but we may have problems defining
> a weather XML description format which the "industry" will use..
> mostly because we're not meteorologists and we don't work for
> the NWS. It may be best to just invent a format which is
> descriptive and versatile, publish the DTD, and just put it out
> there.

Agreed. At the same time, it should be simple enough and yet
extensible/powerful enough for people to do whatever they want with the raw XML. I'm doing some snooping right now, and other than the "made up" XML format at http://weather.interceptvector.com, I haven't seen a common format.

> My main focus is RSS, however, and the "descriptive" XML
> format is just an intermediate step in my Cocoon transformation
> pipelines.

I have to disagree here -- the XML format is where all the info
actually lives. It will be useful to the largest number of people via an RSS feed, BUT, the early adopters and disruptors will want to do their own work with that data, and the raw XML feeds are going to be the easiest way to get it to them.

For you, working with Cocoon and all it's nifty XSLT/pipeline stuff, that's perfect. (Your illustration of pipelines in Cocoon was
excellent; I spent a bunch of time working with Cocoon in early 1999 but haven't really looked at it again -- it's still an excellent application/framework). For others, the XML will be important to build their own PHP/Perl/Python hooks.

> My goals in order of importance with openWeather are:
>
> - Provide a useful set of (free) RSS feeds for weather
> reports
> - Provide severe weather warnings/watches via RSS.
>
> The second one could possibly put services like
> http://www.weather.com/services/notify.html out of business. Or
> compete with them.

Good to get that out in the open. I am totally in sync with your first goal, except of course my ulterior goal is what uses the
consumers/users of those RSS feeds might do with it -- integrating into blog posting, gathering comments, etc. etc. Of course, once you've got an RSS feed, these other uses become trivial to implement.

I don't think we'll worry about competing with anyone yet :p

P.S. Just continue to reply to the sender and everything should
auto-magically appear in the forum.

sending this to the QT forum to see if it goes up.

On Thu, 3 Jul 2003, Boris Mann wrote:

> Yup. We are likely looking at defining a format for weather reports.
> The next step is to provide a couple of different things:
> - a "raw feed" of our new XML format, so people can do things we never
> expected
> - an XML-->HTML-->RSS feed with some options (append various parameters
> like "?icons=small, etc.)

This is all pretty trivial to customize the output in this way, especially with something like Cocoon. Except it looks more like:


Source Data (NWS Screenscrape, Decoded METAR, other Web Services) |
                  V
HTML<-- XSLT <-- XML --> XSLT --> Echo/RSS 0.9x/1.0/2.0
                  |
                  V
              Anything

Output here can be anything. Other XML (RSS/Echo), HTML, WAP, etc
I think I mentioned it before, but we may have problems defining a weather XML description format which the "industry" will use.. mostly because we're not meteorologists and we don't work for the NWS. It may be best to just invent a format which is descriptive and versatile, publish the DTD, and just put it out there. My main focus is RSS, however, and the "descriptive" XML format is just an intermediate step in my Cocoon transformation pipelines.

> > The permalink could link to a description of the feed, or the station,
> > or
> > lots of things. Or a historical report of that particular report. The
> > problem is there's lots of weather reports, so I think it would make
> > the
> > most sense to just have a permalink point to an HTML
> > description/representation of the feed.

> Right now, historical weather data is all for pay. There are lots of
> sources of weather data, but it is hard to get at data that is older
> than a week or so. So, this is the second part -- building some scripts
> that can handle creating blog-like entries out of weather information.

Yep, historical is quite the problem. There's so much of it. I used to work at a place where we had approximately 670GB of raw historical population data. I can only imagine what sort of raw historical weather data is out there in the NWS archives (or wherever it's kept).

My goals in order of importance with openWeather are:

 - Provide a useful set of (free) RSS feeds for weather reports
 - Provide severe weather warnings/watches via RSS.

The second one could possibly put services like
http://www.weather.com/services/notify.html out of business. Or compete with them.


Tony

--
Tony Collen
ICQ: 12410567
--
Cocoon: Internet Glue (A weblog about Apache Cocoon)
http://manero.org/weblog/
--

On 3 Jul 2003, QT - Boris Mann wrote:

oops, that last message didn't go to QT. testing 1234.

tony

>

--
Tony Collen
ICQ: 12410567
--
Cocoon: Internet Glue (A weblog about Apache Cocoon)
http://manero.org/weblog/
--
< replied-to message removed by QT >

On Thursday, July 3, 2003, at 01:24 PM, Tony Collen wrote:

> Well, now I think we're starting to talk about two different things.
>
> We're talking about:
>
> 1) A format for defining weather reports
> 2) A subscription format

Yup. We are likely looking at defining a format for weather reports. The next step is to provide a couple of different things:
- a "raw feed" of our new XML format, so people can do things we never expected
- an XML-->HTML-->RSS feed with some options (append various parameters like "?icons=small, etc.)

> The permalink could link to a description of the feed, or the station,
> or
> lots of things. Or a historical report of that particular report. The
> problem is there's lots of weather reports, so I think it would make
> the
> most sense to just have a permalink point to an HTML
> description/representation of the feed.

I think the permalink is part of the genius of weather feeds. Imagine a photo gallery that grabs info from our feed to display alongside a picture -- the weather at this place at this time was such-and-such.
Right now, historical weather data is all for pay. There are lots of sources of weather data, but it is hard to get at data that is older than a week or so. So, this is the second part -- building some scripts that can handle creating blog-like entries out of weather information.
--
Boris Mann
http://www.bmannconsulting.com