The Altair Community is migrating to a new platform to provide a better experience for you. In preparation for the migration, the Altair Community is on read-only mode from October 28 - November 6, 2024. Technical support via cases will continue to work as is. For any urgent requests from Students/Faculty members, please submit the form linked here

Accessing the US National Weather Service (NWS) Forecasting API

sgenzersgenzer Administrator, Moderator, Employee-RapidMiner, RapidMiner Certified Analyst, Community Manager, Member, University Professor, PM Moderator Posts: 2,959 Community Manager
edited February 2020 in Knowledge Base
So many of you know how much I love APIs, and as much as I love the Web Mining extension with Get Page and Enrich Data via Webservice, they have limitations. I have often dreamt really robust JSON parser in RapidMiner...well my friend @land at Old World Computing has created a beauty in his new Web Automation Extension - currently in beta as of this writing.

Fortunately for me, Sebastian gave me a copy of this beta to run through its paces and I thought, "Hey! I've been meaning to tackle that huge, nasty National Weather Service Forecast API for ages. This will be PERFECT!"

And it was. It's not pretty but it gets the job done. So here it goes...

If you're not familiar with the NWS API, it's fairly simple on the surface (pardon the pun). Just send latitude and longitude and you get a weather station. Then send a second query to that weather station and poof - you get forecast information in nice JSON. Like this:

Get the URL for the Weather Station covering Norwich, Vermont USA = 43.7022 latitude, 72.2978 longitude

URL: https://api.weather.gov/points/43.7022,72.2978

Response:

{
    "@context": [
        "https://raw.githubusercontent.com/geojson/geojson-ld/master/contexts/geojson-base.jsonld",
        {
            "wx": "https://api.weather.gov/ontology#",
            "s": "https://schema.org/",
            "geo": "http://www.opengis.net/ont/geosparql#",
            "unit": "http://codes.wmo.int/common/unit/",
            "@vocab": "https://api.weather.gov/ontology#",
            "geometry": {
                "@id": "s:GeoCoordinates",
                "@type": "geo:wktLiteral"
            },
            "city": "s:addressLocality",
            "state": "s:addressRegion",
            "distance": {
                "@id": "s:Distance",
                "@type": "s:QuantitativeValue"
            },
            "bearing": {
                "@type": "s:QuantitativeValue"
            },
            "value": {
                "@id": "s:value"
            },
            "unitCode": {
                "@id": "s:unitCode",
                "@type": "@id"
            },
            "forecastOffice": {
                "@type": "@id"
            },
            "forecastGridData": {
                "@type": "@id"
            },
            "publicZone": {
                "@type": "@id"
            },
            "county": {
                "@type": "@id"
            }
        }
    ],
    "id": "https://api.weather.gov/points/43.7022,-72.2978",
    "type": "Feature",
    "geometry": {
        "type": "Point",
        "coordinates": [
            -72.297799999999995,
            43.702199999999998
        ]
    },
    "properties": {
        "@id": "https://api.weather.gov/points/43.7022,-72.2978",
        "@type": "wx:Point",
        "cwa": "GYX",
        "forecastOffice": "https://api.weather.gov/offices/GYX",
        "gridX": 8,
        "gridY": 48,
        "forecast": "https://api.weather.gov/gridpoints/GYX/8,48/forecast",
        "forecastHourly": "https://api.weather.gov/gridpoints/GYX/8,48/forecast/hourly",
        "forecastGridData": "https://api.weather.gov/gridpoints/GYX/8,48",
        "observationStations": "https://api.weather.gov/gridpoints/GYX/8,48/stations",
        "relativeLocation": {
            "type": "Feature",
            "geometry": {
                "type": "Point",
                "coordinates": [
                    -72.305660000000003,
                    43.718015999999999
                ]
            },
            "properties": {
                "city": "Norwich",
                "state": "VT",
                "distance": {
                    "value": 1868.6929186969621,
                    "unitCode": "unit:m"
                },
                "bearing": {
                    "value": 160,
                    "unitCode": "unit:degrees_true"
                }
            }
        },
        "forecastZone": "https://api.weather.gov/zones/forecast/NHZ005",
        "county": "https://api.weather.gov/zones/county/NHC009",
        "fireWeatherZone": "https://api.weather.gov/zones/fire/NHZ005",
        "timeZone": "America/New_York",
        "radarStation": "KCXX"
    }
}

There's a lot of information there but all I really care about is the URL:

forecastGridData": "https://api.weather.gov/gridpoints/GYX/8,48
So now I put that into OWC's fancy JSON parser like this:

      
The OWC JSON parser uses a subprocess before the actual Parse JSON operator to specify how and where to parse. I chose "Process Object" because the root of this JSON was in this form:

{
   "foo" : "bar",
   "foo2" : "bar2",
...
}
So this is a list of objects inside one main object (objects are shown as {} in JSON). So if I only wanted to extract simple items like id and type, I would just put this inside of Process Object:





Which I do. But I also want that forecaseGridData URL which is inside the 'properties' object, so I do this instead:



And finally inside of Process Object (3) just another Extract Properties and Commit Row with forecastGridData. Easy!

Get the Weather Forecast from this NWS URL

So once you get this URL, you just send another Get Page and voilà - a MASSIVE JSON with all sorts of goodies. It's too long to post here so I'll just abbreviate it...

{
  "@context": [
    "https://raw.githubusercontent.com/geojson/geojson-ld/master/contexts/geojson-base.jsonld",
    {
      "wx": "https://api.weather.gov/ontology#",
      "s": "https://schema.org/",
      "geo": "http://www.opengis.net/ont/geosparql#",
      "unit": "http://codes.wmo.int/common/unit/",
      "@vocab": "https://api.weather.gov/ontology#",
      "geometry": {
        "@id": "s:GeoCoordinates",
        "@type": "geo:wktLiteral"
      },
      etc...
      "county": {
        "@type": "@id"
      }
    }
  ],
  "id": "https://api.weather.gov/gridpoints/GYX/8,48",
  "type": "Feature",
  "geometry": {
    "type": "Polygon",
    "coordinates": [
      [
        [
          120.5517605,
          90
        ],
        [
          120.5517605,
          90
        ],
        [
          120.5517605,
          90
        ],
        [
          120.5517605,
          90
        ]
      ]
    ]
  },
  "properties": {
    "@id": "https://api.weather.gov/gridpoints/GYX/8,48",
    "@type": "wx:Gridpoint",
    "updateTime": "2018-12-18T19:44:04+00:00",
    "validTimes": "2018-12-18T13:00:00+00:00/P8DT6H",
    "elevation": {
      "value": 160.93440000000001,
      "unitCode": "unit:m"
    },
    "forecastOffice": "https://api.weather.gov/offices/GYX",
    "gridId": "GYX",
    "gridX": "8",
    "gridY": "48",
    "temperature": {
      "sourceUnit": "F",
      "uom": "unit:degC",
      "values": [
        {
          "validTime": "2018-12-18T13:00:00+00:00/PT2H",
          "value": -6.6666666666666288
        }
        etc...
        {
          "validTime": "2018-12-26T17:00:00+00:00/PT1H",
          "value": -3.8888888888888573
        },
        {
          "validTime": "2018-12-26T18:00:00+00:00/PT1H",
          "value": -2.7777777777777715
        }
      ]
    },
    "dewpoint": {
      "sourceUnit": "F",
      "uom": "unit:degC",
      "values": [
        {
          "validTime": "2018-12-18T13:00:00+00:00/PT2H",
          "value": -12.222222222222229
        },
        etc...
        {
          "validTime": "2018-12-26T18:00:00+00:00/PT1H",
          "value": -9.4444444444444571
        }
      ]
    }
    A LOT GOES HERE....
    },
    "stability": {
      "values": [
        
      ]
    },
    "redFlagThreatIndex": {
      "values": [
        
      ]
    }
  }
}


Then you use the same JSON parsing tools (I also did some post-processing to convert units - yes I still think in inches and feet), and you get a five-day forecast in a nice RapidMiner ExampleSet:



Easy!

I will post the process on the Community Repository soon. Stay dry!

Scott
Tagged:

Comments

  • Telcontar120Telcontar120 RapidMiner Certified Analyst, RapidMiner Certified Expert, Member Posts: 1,635 Unicorn
    Excellent use case!  I would love to see that process to play around with it as well.
    I have also been testing the OWC JSON extension and it is great.
    Brian T.
    Lindon Ventures 
    Data Science Consulting from Certified RapidMiner Experts
  • sgenzersgenzer Administrator, Moderator, Employee-RapidMiner, RapidMiner Certified Analyst, Community Manager, Member, University Professor, PM Moderator Posts: 2,959 Community Manager
    I'm going to attach the process here...not sure when I can get it on the repo.

    Scott

Sign In or Register to comment.