Part 1: What’s Wrong With Weather Data?

“If two mock-suns appear, one to the south, the other to the north, and there is at the same time a halo, these indicate that it will shortly rain.” –Theophrastus

People have always tried to forecast the weather. Aristotle devoted four volumes to meteorology; his student Theophrastus summed up ancient forecasting. “The popular saying about flies is true,” he wrote, for example. “When they bite excessively, it is a sign of rain.”

It was really only in the 1920s, with the development of Numerical Weather Prediction (NWP), that modern forecasting began. The idea was to use a set of equations that try to predict the future state of the atmosphere given known initial conditions. The hope behind NWP began to be fulfilled in the 1950s, when computers came along that could solve these equations faster than it took the weather to actually occur. We got used to sitting at home and consuming weather from someone like this guy:

Behind the scenes, computing power and modeling improved, but for most people and businesses needing to make weather-related decisions, this is what was available: twice-a-day generalities about cold fronts and partly cloudy days, with temperatures dropping toward next evening.

Then the internet and the smartphone arrived, with websites and apps promising “hyper-localized” weather. Forecasts went from generalities, to region, then city, then even ZIP code scale.

Or so it appeared.

To understand why much of what presents itself as innovation can’t live up to the promises it makes, we need to peer beneath the colorful interfaces. How are today’s weather forecasts actually generated? Who is behind them?

Despite how simple it seems when you ask Siri for the weather, generating an understanding of current conditions and forecast weather is a massive undertaking, involving multiple multi-national organizations, billions of dollars of annual investment from dozens of countries, satellites, wind vanes, lasers, and some whiz-bang computing. Without delving too deeply, however, there are three major steps to getting the weather out to the world:

 

1. Observe – Figure out what the weather is currently doing.
2. Model – Take a guess at what it’s about to do.
3. Disseminate – Tell everyone about it.

Step 1 – Observe

Before we can do anything with weather, we need to figure out what is going on in the entirety of the atmosphere, the world’s oceans, and other major earth systems, right now. To do this, we need observations of all of the atmospheric variables like temperature, humidity, winds, barometric pressure, and so on. These observations are important: since the atmosphere and weather is a chaotic system, even small errors in understanding the initial conditions can lead to gross errors in the forecast.

To conduct weather observations, the global meteorological community has developed and deployed all sorts of equipment to sense the atmosphere. Some of these you’ve heard of, like rain gauges, thermometers, radars, and satellites. Some are more specialized: anemometers for wind speed, pyranometers for solar irradiance, radiosondes (weather sensing balloons), and so on.

These sensors are expensive to build, to calibrate, and to operate. As a small example, NOAA operates a set of environmental satellites; the two they’re launching in the next few years – the GOES-T and GOES-U – will cost something like $11B over their 15-year lifespan. The cost is limiting, so much so that good sensing equipment is deployed almost exclusively by wealthy, developed countries. The US has something like 4,000 weather observation stations. The Democratic Republic of Congo, with approximately ¼ the land and population, has five–and it’s questionable as to whether they’re calibrated and working.

As a result, there are accuracy-undermining gaps or “blind spots” in the picture of the atmosphere we use to feed into models for forecasting.

Step 2 – Model

All the collected observations (most gathered by public entities) feed complex computer models that try, first, to figure out current weather conditions, and then, on the basis of that, the future.

These models involve such complexity—and demand such computational power, when processing large stretches of geography and time, that they are run by only a handful of governmental and non-governmental agencies, and an even smaller number of private sector companies (including Tomorrow.io, of course). Once run, the outputs – generally global and local forecasts for several hours to several weeks out – are made available for public use.

Step 3 – Disseminate

This is where the private sector generally comes in. After the public sector has done the bulk of the work in observing and modeling, the private sector turns the outputs into business- and consumer-friendly weather reporting.

But here’s the open secret of the industry: since pretty much every company is relying on the same data, no matter how much they tweak and repackage and post-process it, they face similar limits in what they can provide to their customers.

Though they’re the public face of weather, the developer of your weather app, your favorite weather website, and even your favorite television weather person is just one link in the complex chain of forecasting work, and are only (typically small) part of the value generated by this whole process.

The best way to explain this is by borrowing an image published by the US National Weather Service. What the common person knows as a weather forecast is really only the tip of the iceberg. The thousands of person-hours of research and work, and the billions of dollars spent on generating observations and running complex models are woefully unappreciated by the lay observer.

The Problems Current Weather Forecasting Cannot Overcome

So, this all sounds like a pretty good system, doesn’t it? Well, not quite.

Today’s forecasting suffers from three key problems.

First, weather forecasts in most parts of the world are still quite bad. Because of the resources and costs required to run current systems, good governmental infrastructure – from sensing to modeling to data dissemination – exists almost exclusively in developed countries. Yet the stakes are, if anything, higher in the developing world, where farming and infrastructure are more exposed and less resilient. Take Sub-Saharan Africa, where the UN reports “nearly two disasters of significant proportions have been recorded every week since 2000, with around 12.5 million people per year affected.” The Central African Republic (CAR), smack dab in the middle of those recorded floods, storms, and wildfires, struggles even to keep a hydrometeorology service running. Currently, none of its 14 “weather observation stations” are in shape to report data. People seeking forecasts, from farmers to retailers, rely on mostly inaccurate and certain non-localized forecasts from models making poor guesses. And CAR isn’t alone – over 5 billion people globally have poor to no access to high-quality weather information.

 Second, even in places where the government is doing a great job, we still get generic answers for our specific questions. The needs of the Spanish wind-energy industry differ greatly from the needs of organizations concerned with flood risk in Malaysia, and both require much more specialized forecasts than general, regional weather trend predictions provide. The forecasts generated by major national/international modeling centers are meant to care for the public good — but they are not endowed to solve specific weather problems that may impact you, your groups, and your businesses. Though there are plenty of companies that try to serve individual verticals with weather and forecasting capabilities supposedly tailored to those industries’ needs, a lot of what they can do is repackage the same general weather information into a more palatable, use-case specific package.

The result is that enterprise and industry suffer from weather data inadequacy. They plan poorly, and profit less. Economies of whole countries and regions are less efficient.

This is even more pronounced when we consider the emerging needs of “new economy” companies — the ride-sharing on-demand drone-flying autonomous-vehicle producing ones. Their business models and operations are extremely weather-sensitive, and they’re data-hungry in a way that the basic weather forecast can’t satisfy. Greater weather insight would help power a large and growing sector of the economy.

Third, forecasting is tougher under conditions of climate change, with weather patterns becoming more variable and volatile. The limits of the current system are growing more critical, as such phenomena as rapid onset rain and severe storms become more common. And climate change exacerbates the first point above – those living in the poorest and least developed countries experience the most pain from lack of access to sufficiently specific forecasts.

The good news? This can all be changed, but it requires a significant thought leap from where we are today. In the next few posts, we’ll share how we at Tomorrow.io rethink weather from bottom to top.

Next: Virtual Sensors – Rethinking Observations

Weather observations are the currency of meteorology – without them, forecasting doesn’t work. Yet, this currency is in short supply, poorly distributed, and doesn’t often reach the places where it’s needed the most. Why? Hardware sensors have been only source for weather observations around the globe and they are prohibitively expensive to deploy and maintain.  

Tomorrow.io set out to change that in 2015 and is gearing up for a major revelation of the secrets of some of the world’s most unexpected weather sensors.

Stay tuned for the next post in our series.

I want to know more!