Getting the atmospheric model data, remotely¶

As Atmospheric Data Scientists, we would like to reach out to the latest outputs of the atmospheric models (GFS, NAM, HRRR, GEFS). However, we are not very keen on downloading all those data into our local machine. As a solution, this blog is going to engage with atmospheric model outputs remotely.

# Importing required libraries
import xarray as xr # dealing with atmospheric data
import numpy as np # dealing with arrays
import requests # making requests to webpages
from bs4 import BeautifulSoup # scraping webpage content

# Ignore warnings
import warnings
warnings.filterwarnings("ignore")

# The link of the server (nomads) we are going to utilize while obtaining our data
nomads_link = r'https://nomads.ncep.noaa.gov/dods/'

Possible atmospheric model datasets¶

Let’s see the atmospheric model lists on nomads server.

# Make a request to the nomads server link
page = requests.get(nomads_link)

# View the page content
soup = BeautifulSoup(page.content)

# Page content under <b> tag
soup.find_all('b')

[<b>1: akrtma/:</b>,
 <b>2: aqm/:</b>,
 <b>3: blend/:</b>,
 <b>4: cmcens/:</b>,
 <b>5: estofs/:</b>,
 <b>6: fens/:</b>,
 <b>7: fnl/:</b>,
 <b>8: gdas_0p25/:</b>,
 <b>9: gefs/:</b>,
 <b>10: gens/:</b>,
 <b>11: gens_bc/:</b>,
 <b>12: gens_ndgd/:</b>,
 <b>13: gfs_0p25/:</b>,
 <b>14: gfs_0p25_1hr/:</b>,
 <b>15: gfs_0p25_1hr_parafv3/:</b>,
 <b>16: gfs_0p25_parafv3/:</b>,
 <b>17: gfs_0p50/:</b>,
 <b>18: gfs_1p00/:</b>,
 <b>19: gurtma/:</b>,
 <b>20: hiresw/:</b>,
 <b>21: hirtma/:</b>,
 <b>22: hrrr/:</b>,
 <b>23: ice/:</b>,
 <b>24: naefs_bc/:</b>,
 <b>25: naefs_ndgd/:</b>,
 <b>26: nam/:</b>,
 <b>27: narre/:</b>,
 <b>28: ncom/:</b>,
 <b>29: prrtma/:</b>,
 <b>30: rap/:</b>,
 <b>31: rtma2p5/:</b>,
 <b>32: rtofs/:</b>,
 <b>33: sref/:</b>,
 <b>34: sref_bc/:</b>,
 <b>35: wave/:</b>]

As you see, some of the available atmospheric model dataset abbreviations (#35) on the nomads server are visible. We are going to make a list of these datasets.

# A function for making the list of model abbreviations
def get_data_abbreviations(soup):
    """
    Returns the model abbreviation list by searching 
    the page content under <b> tag
    
    Arguments:
        soup {BS object} -- BeautifulSoup object
        
    Returns:
        {list} -- model abbreviation list
    """
    
    # Search for <b> tag within the content and keep them
    b_tag = soup.find_all('b')
    
    # Get text of each content element
    texts = [b.get_text() for b in b_tag]
    
    # Texts are not in the format that we expect. Just do a little bit of trick
    abbrs = [text.split(' ')[1].split('/')[0] for text in texts]
    
    return abbrs

# Let's check some of the elements of the abbreviation list
abbr_list = get_data_abbreviations(soup)
print('Length of model data: ', len(abbr_list))
print('some of the elements: ', abbr_list[:5])

Length of model data:  35
some of the elements:  ['akrtma', 'aqm', 'blend', 'cmcens', 'estofs']

Available data dates¶

On the nomads server, atmospheric model data generally extend back up to 1 week. Nevertheless, let’s reveal the possible date range of the data we can study.

def get_possible_data_range(link, model_name):
    """
    Returns the list of the available data dates by searching 
    the page content under <b> tag
    
    Arguments:
        link {str} -- general link to nomads dod
        model_name {str} -- expected model data abbreviation
        
    Returns:
        {list} -- model available date list 'YYYYMMDD'
    """
    
    # Make a request to the model page
    page = requests.get(link + f'/{model_name}')
    
    # View the page content
    soup = BeautifulSoup(page.content)
    
    try:
        # Search for the <b> tag within the content and keep them
        b_tag = soup.find_all('b')

        # Get the text of each content element
        texts = [b.get_text() for b in b_tag]

        # Texts are not in the format we expect. Just do a little bit of trick
        dates_with_abbr = [text.split(' ')[1].split('/')[0] for text in texts]
        
        return dates_with_abbr
    
    # Exception
    except Exception as error:
        print(f'Something is uncommon in nomads server with this specific data: {model_name}' + repr(error))

# Let's see the date range of one of the model data
possible_dates = get_possible_data_range(nomads_link, abbr_list[0])
print('Length of date range: ', len(possible_dates))
print('some of the elements: ', possible_dates[:3])

Length of date range:  10
some of the elements:  ['akrtma20211002', 'akrtma20211003', 'akrtma20211004']

Seek for specific data links¶

Until now, we have viewed the available model data abbreviations and their available date ranges on the nomads server. Hereupon, we will look for the data links that we can fetch and study.

def get_specific_data_link(link, model_name, date_with_abbr):
    """
    Returns the list of available data dates by searching 
    the page content under <b> tag
    
    Arguments:
        link {str} -- general link to nomads dod
        model_name {str} -- expected model data abbreviation
        date_with_abbr {str} -- date with abbreviation in format 'abbrYYYYMMDD'
        
    Returns:
        {list} -- specific data links:                
    """
    
    # Make a request to the model page
    page = requests.get(link + f'/{model_name}' + f'/{date_with_abbr}')
    
    # View the page content
    soup = BeautifulSoup(page.content)
    
    try:
        # Search for <b> tag within the content and keep them
        b_tag = soup.find_all('b')

        # Get text of each content element
        texts = [b.get_text() for b in b_tag]

        # Texts are not in the format we expect. Just do a little bit of trick
        data_links = [text.split(' ')[1].split(':')[0].split('\n')[1] for text in texts]
        
        return data_links
        
    # Exception
    except Exception as error:
        print(f'Something is uncommon in nomads server with this specific data: {model_name}' + repr(error))

# Let's see the data links for:
#                             model_name: abbr_list[0] and
#                             date: possible_dates[0]

specific_data_links = get_specific_data_link(nomads_link, abbr_list[0], possible_dates[0])
print('Length of data links: ', len(specific_data_links))
print('some of the elements: ', specific_data_links[:3])

Length of data links:  72
some of the elements:  ['akrtma_anl_00z', 'akrtma_anl_01z', 'akrtma_anl_02z']

previous

Welcome

next

Basic operations on the time dimension of netCDF data using Xarray