NCEI Paleo Database API¶

About¶

The NCEI Paleo Web Service is searchable via its web interface:https://www.ncei.noaa.gov/access/paleo-search/

Here we are using its API: https://www.ncei.noaa.gov/access/paleo-search/api

Specifically, we are interested in Tree Ring Data

Tutorial 1: Using Python to access the API for tree ring `.rwl` data¶

Step 1: Activate/Create a Python environment¶

First, you need to import the necessary libraries in Python. The most common ones used for API calls are requests for making HTTP requests and json for handling JSON data.

We also want to plot the data using maplotlib, numpy, and pandas which are all core libraries for scientific Python.

Later, we will use fiona and Leaflet maps to visualize the site locations.

In [77]:

            
                Copied!
                
!mamba install -n dplpy -c conda-forge folium tabulate requests -y
!mamba install -n dplpy -c conda-forge folium tabulate requests -y

                  __    __    __    __
                 /  \  /  \  /  \  /  \
                /    \/    \/    \/    \
███████████████/  /██/  /██/  /██/  /████████████████████████
              /  / \   / \   / \   / \  \____
             /  /   \_/   \_/   \_/   \    o \__,
            / _/                       \_____/  `
            |/
        ███╗   ███╗ █████╗ ███╗   ███╗██████╗  █████╗
        ████╗ ████║██╔══██╗████╗ ████║██╔══██╗██╔══██╗
        ██╔████╔██║███████║██╔████╔██║██████╔╝███████║
        ██║╚██╔╝██║██╔══██║██║╚██╔╝██║██╔══██╗██╔══██║
        ██║ ╚═╝ ██║██║  ██║██║ ╚═╝ ██║██████╔╝██║  ██║
        ╚═╝     ╚═╝╚═╝  ╚═╝╚═╝     ╚═╝╚═════╝ ╚═╝  ╚═╝

        mamba (1.4.2) supported by @QuantStack

        GitHub:  https://github.com/mamba-org/mamba
        Twitter: https://twitter.com/QuantStack

█████████████████████████████████████████████████████████████


Looking for: ['folium', 'tabulate', 'requests']

[+] 0.0s
conda-forge/linux-64 ━━━━━━━╸━━━━━━━━━━━━━━━   0.0 B /  ??.?MB @  ??.?MB/s  0.0s
conda-forge/noarch   ━━━━━━━━━━━╸━━━━━━━━━━━   0.0 B /  ??.?MB @  ??.?MB/s  0.0s[+] 0.1s
conda-forge/linux-64 ━━━━━━━╸━━━━━━━━━━━━━━━   0.0 B /  ??.?MB @  ??.?MB/s  0.1s
conda-forge/noarch   ━━━━━━━━━━━╸━━━━━━━━━━━   0.0 B /  ??.?MB @  ??.?MB/s  0.1s[+] 0.2s
conda-forge/linux-64 ━━━━━━━━╸━━━━━━━━━━━━━━   0.0 B /  ??.?MB @  ??.?MB/s  0.2s
conda-forge/noarch   ━━━━━━━━━━━━━╸━━━━━━━━━   0.0 B /  ??.?MB @  ??.?MB/s  0.2s[+] 0.3s
conda-forge/linux-64 ━━━━━━━━━╸━━━━━━━━━━━━━   0.0 B /  ??.?MB @  ??.?MB/s  0.3s
conda-forge/noarch   ━━━━━━━━━━━━━━╸━━━━━━━━   0.0 B /  ??.?MB @  ??.?MB/s  0.3s[+] 0.4s
conda-forge/linux-64 ━━━━━━━━━╸━━━━━━━━━━━━━   0.0 B /  ??.?MB @  ??.?MB/s  0.4s
conda-forge/noarch   ━━━━━━━━━━━━━━╸━━━━━━━━   0.0 B /  ??.?MB @  ??.?MB/s  0.4s[+] 0.5s
conda-forge/linux-64 ━━━━━━━━━━━╸━━━━━━━━━━━ 834.8kB /  ??.?MB @   1.7MB/s  0.5s
conda-forge/noarch   ╸━━━━━━━━━━━━━━━╸━━━━━━ 959.1kB /  ??.?MB @   2.0MB/s  0.5s[+] 0.6s
conda-forge/linux-64 ━━━━━━━━╸━━━━━━━━━━━━━━   1.7MB /  ??.?MB @   2.9MB/s  0.6s
conda-forge/noarch   ━━╸━━━━━━━━━━━━━━━╸━━━━   2.0MB /  ??.?MB @   3.4MB/s  0.6s[+] 0.7s
conda-forge/linux-64 ━━━━━━━━━━╸━━━━━━━━━━━━   2.5MB /  ??.?MB @   3.7MB/s  0.7s
conda-forge/noarch   ━━━━╸━━━━━━━━━━━━━━━╸━━   2.9MB /  ??.?MB @   4.2MB/s  0.7s[+] 0.8s
conda-forge/linux-64 ━━━━━━━━━━━━╸━━━━━━━━━━   3.4MB /  ??.?MB @   4.3MB/s  0.8s
conda-forge/noarch   ━━━━━━╸━━━━━━━━━━━━━━━━   3.9MB /  ??.?MB @   4.9MB/s  0.8s[+] 0.9s
conda-forge/linux-64 ━━━━━━━━━━━━━━╸━━━━━━━━   4.0MB /  ??.?MB @   4.5MB/s  0.9s
conda-forge/noarch   ━━━━━━━━━╸━━━━━━━━━━━━━   4.8MB /  ??.?MB @   5.4MB/s  0.9s[+] 1.0s
conda-forge/linux-64 ╸━━━━━━━━━━━━━━━╸━━━━━━   4.8MB /  ??.?MB @   4.8MB/s  1.0s
conda-forge/noarch   ━━━━━━━━━━━╸━━━━━━━━━━━   5.7MB /  ??.?MB @   5.7MB/s  1.0s[+] 1.1s
conda-forge/linux-64 ━━━╸━━━━━━━━━━━━━━━╸━━━   5.7MB /  ??.?MB @   5.3MB/s  1.1s
conda-forge/noarch   ━━━━━━━╸━━━━━━━━━━━━━━━   6.6MB /  ??.?MB @   6.1MB/s  1.1s[+] 1.2s
conda-forge/linux-64 ━━━━━╸━━━━━━━━━━━━━━━╸━   6.5MB /  ??.?MB @   5.5MB/s  1.2s
conda-forge/noarch   ━━━━━━━━━╸━━━━━━━━━━━━━   7.6MB /  ??.?MB @   6.4MB/s  1.2s[+] 1.3s
conda-forge/linux-64 ━━━━━━━╸━━━━━━━━━━━━━━━   7.3MB /  ??.?MB @   5.7MB/s  1.3s
conda-forge/noarch   ━━━━━━━━━━━╸━━━━━━━━━━━   8.5MB /  ??.?MB @   6.5MB/s  1.3s[+] 1.4s
conda-forge/linux-64 ━━━━━━━━━╸━━━━━━━━━━━━━   8.3MB /  ??.?MB @   5.9MB/s  1.4s
conda-forge/noarch   ━━━━━━━━━━━━━━╸━━━━━━━━   9.5MB /  ??.?MB @   6.8MB/s  1.4s[+] 1.5s
conda-forge/linux-64 ━━━━━━━━━━━╸━━━━━━━━━━━   9.2MB /  ??.?MB @   6.1MB/s  1.5s
conda-forge/noarch   ╸━━━━━━━━━━━━━━━╸━━━━━━  10.4MB /  ??.?MB @   6.9MB/s  1.5s[+] 1.6s
conda-forge/linux-64 ━━━━━━━━╸━━━━━━━━━━━━━━  10.0MB /  ??.?MB @   6.3MB/s  1.6s
conda-forge/noarch   ━╸━━━━━━━━━━━━━━━╸━━━━━  10.9MB /  ??.?MB @   7.1MB/s  1.6s[+] 1.7s
conda-forge/linux-64 ━━━━━━━━━╸━━━━━━━━━━━━━  10.3MB /  ??.?MB @   6.3MB/s  1.7s
conda-forge/noarch   ━━━╸━━━━━━━━━━━━━━━╸━━━  11.9MB /  ??.?MB @   7.2MB/s  1.7s[+] 1.8s
conda-forge/linux-64 ━━━━━━━━━━━╸━━━━━━━━━━━  11.3MB /  ??.?MB @   6.5MB/s  1.8s
conda-forge/noarch   ━━━━━╸━━━━━━━━━━━━━━━╸━  12.7MB /  ??.?MB @   7.3MB/s  1.8s[+] 1.9s
conda-forge/linux-64 ━━━━━━━━━━━━━╸━━━━━━━━━  12.2MB /  ??.?MB @   6.6MB/s  1.9s
conda-forge/noarch   ━━━━━━━━╸━━━━━━━━━━━━━━  13.6MB /  ??.?MB @   7.4MB/s  1.9s[+] 2.0s
conda-forge/linux-64 ━━━━━━━━━━━━━━━╸━━━━━━━  12.7MB /  ??.?MB @   6.5MB/s  2.0s
conda-forge/noarch   ━━━━━━━━━━╸━━━━━━━━━━━━  14.4MB /  ??.?MB @   7.4MB/s  2.0s[+] 2.1s
conda-forge/linux-64 ╸━━━━━━━━━━━━━━━╸━━━━━  13.1MB @   6.5MB/s             2.1s
conda-forge/noarch   ━━━━━━━━━━━━━━━━━━━━━━  15.2MB @   7.4MB/s Finalizing  2.1sconda-forge/noarch                                 @   7.4MB/s  2.2s
[+] 2.2s
conda-forge/linux-64 ╸━━━━━━━━━━━━━━━╸━━━━━━  13.1MB /  ??.?MB @   6.5MB/s  2.2s[+] 2.3s
conda-forge/linux-64 ━━━╸━━━━━━━━━━━━━━━╸━━━  14.9MB /  ??.?MB @   6.6MB/s  2.3s[+] 2.4s
conda-forge/linux-64 ━━━━━╸━━━━━━━━━━━━━━━╸━  15.7MB /  ??.?MB @   6.7MB/s  2.4s[+] 2.5s
conda-forge/linux-64 ━━━━━━━╸━━━━━━━━━━━━━━━  16.5MB /  ??.?MB @   6.7MB/s  2.5s[+] 2.6s
conda-forge/linux-64 ━━━━━━━━━╸━━━━━━━━━━━━━  17.4MB /  ??.?MB @   6.8MB/s  2.6s[+] 2.7s
conda-forge/linux-64 ━━━━━━━━━━━╸━━━━━━━━━━━  18.4MB /  ??.?MB @   6.9MB/s  2.7s[+] 2.8s
conda-forge/linux-64 ━━━━━━━━╸━━━━━━━━━━━━━━  19.2MB /  ??.?MB @   7.0MB/s  2.8s[+] 2.9s
conda-forge/linux-64 ━━━━━━━━━━╸━━━━━━━━━━━━  20.2MB /  ??.?MB @   7.0MB/s  2.9s[+] 3.0s
conda-forge/linux-64 ━━━━━━━━━━━━╸━━━━━━━━━━  20.9MB /  ??.?MB @   7.0MB/s  3.0s[+] 3.1s
conda-forge/linux-64 ━━━━━━━━━━━━━━╸━━━━━━━━  21.5MB /  ??.?MB @   7.0MB/s  3.1s[+] 3.2s
conda-forge/linux-64 ╸━━━━━━━━━━━━━━━╸━━━━━━  22.3MB /  ??.?MB @   7.0MB/s  3.2s[+] 3.3s
conda-forge/linux-64 ━━╸━━━━━━━━━━━━━━━╸━━━━  22.7MB /  ??.?MB @   7.0MB/s  3.3s[+] 3.4s
conda-forge/linux-64 ━━━╸━━━━━━━━━━━━━━━╸━━━  23.2MB /  ??.?MB @   6.9MB/s  3.4s[+] 3.5s
conda-forge/linux-64 ━━━━━╸━━━━━━━━━━━━━━━╸━  24.7MB /  ??.?MB @   7.1MB/s  3.5s[+] 3.6s
conda-forge/linux-64 ━━━━━━━╸━━━━━━━━━━━━━━━  25.6MB /  ??.?MB @   7.2MB/s  3.6s[+] 3.7s
conda-forge/linux-64 ━━━━━━━━━╸━━━━━━━━━━━━━  26.5MB /  ??.?MB @   7.2MB/s  3.7s[+] 3.8s
conda-forge/linux-64 ━━━━━━━━━━━╸━━━━━━━━━━━  27.4MB /  ??.?MB @   7.3MB/s  3.8s[+] 3.9s
conda-forge/linux-64 ━━━━━━━━╸━━━━━━━━━━━━━━  28.3MB /  ??.?MB @   7.3MB/s  3.9s[+] 4.0s
conda-forge/linux-64 ━━━━━━━━━━╸━━━━━━━━━━━━  29.3MB /  ??.?MB @   7.4MB/s  4.0s[+] 4.1s
conda-forge/linux-64 ━━━━━━━━━━━━╸━━━━━━━━━━  30.3MB /  ??.?MB @   7.4MB/s  4.1s[+] 4.2s
conda-forge/linux-64 ━━━━━━━━━━━━━━╸━━━━━━━━  31.1MB /  ??.?MB @   7.5MB/s  4.2s[+] 4.3s
conda-forge/linux-64 ╸━━━━━━━━━━━━━━━╸━━━━━━  32.0MB /  ??.?MB @   7.5MB/s  4.3s[+] 4.4s
conda-forge/linux-64 ━━━╸━━━━━━━━━━━━━━━╸━━━  32.8MB /  ??.?MB @   7.5MB/s  4.4s[+] 4.5s
conda-forge/linux-64 ━━━━━╸━━━━━━━━━━━━━━━╸━  33.6MB /  ??.?MB @   7.5MB/s  4.5s[+] 4.6s
conda-forge/linux-64 ━━━━━━━╸━━━━━━━━━━━━━━━  34.4MB /  ??.?MB @   7.5MB/s  4.6s[+] 4.7s
conda-forge/linux-64 ━━━━━━━━━╸━━━━━━━━━━━━━  35.4MB /  ??.?MB @   7.6MB/s  4.7s[+] 4.8s
conda-forge/linux-64 ━━━━━━━━━━━╸━━━━━━━━━━━  36.3MB /  ??.?MB @   7.6MB/s  4.8s[+] 4.9s
conda-forge/linux-64 ━━━━━━━━━━━━━━━━━━━━━━  37.2MB @   7.6MB/s Downloaded  4.9s[+] 5.0s
conda-forge/linux-64 ━━━━━━━━━━━━━━━━━━━━━━  37.2MB @   7.6MB/s Finalizing  5.0s[+] 5.1s
conda-forge/linux-64 ━━━━━━━━━━━━━━━━━━━━━━  37.2MB @   7.6MB/s Finalizing  5.1s[+] 5.2s
conda-forge/linux-64 ━━━━━━━━━━━━━━━━━━━━━━  37.2MB @   7.6MB/s Finalizing  5.2s[+] 5.3s
conda-forge/linux-64 ━━━━━━━━━━━━━━━━━━━━━━  37.2MB @   7.6MB/s Finalizing  5.3s[+] 5.4s
conda-forge/linux-64 ━━━━━━━━━━━━━━━━━━━━━━  37.2MB @   7.6MB/s Finalizing  5.4s[+] 5.5s
conda-forge/linux-64 ━━━━━━━━━━━━━━━━━━━━━━  37.2MB @   7.6MB/s Finalizing  5.5s[+] 5.6s
conda-forge/linux-64 ━━━━━━━━━━━━━━━━━━━━━━  37.2MB @   7.6MB/s Finalizing  5.6s[+] 5.7s
conda-forge/linux-64 ━━━━━━━━━━━━━━━━━━━━━━  37.2MB @   7.6MB/s Finalizing  5.7s[+] 5.8s
conda-forge/linux-64 ━━━━━━━━━━━━━━━━━━━━━━  37.2MB @   7.6MB/s Finalizing  5.8s[+] 5.9s
conda-forge/linux-64 ━━━━━━━━━━━━━━━━━━━━━━  37.2MB @   7.6MB/s Finalizing  5.9s[+] 6.0s
conda-forge/linux-64                               @   7.6MB/s  6.0s

Pinned packages:
  - python 3.12.*


Transaction

  Prefix: /opt/conda/envs/dplpy

  All requested packages already installed

In [78]:

            
                Copied!
                
                    
                    
                
                

        
# import Python libraries
import matplotlib.pyplot as plt
import numpy as np
import json
import requests
import pandas as pd
# import Python libraries
import matplotlib.pyplot as plt
import numpy as np
import json
import requests
import pandas as pd

Step 2: Set the base url for the API as `api_base`¶

Define the base url to the NOAA Paleoclimate API, this URL is the starting point for all future requests.

In [79]:

            
                Copied!
                
# Call the NOAA API to get tree ring data 
api_base = "https://www.ncei.noaa.gov/access/paleo-search/study/search.json?"
# Call the NOAA API to get tree ring data 
api_base = "https://www.ncei.noaa.gov/access/paleo-search/study/search.json?"

Step 3: Search via API for tree ring (`dataTypeID=18`) studies of raw ring width.¶

Returns pages of 10 studies at a time. Set this limit to avoid overwhelming the API with large requests.

Results are returned in JSON format as a dict called "data".

Note, when searching via the user interface, the API call used to retrieve results is displayed immediately above the search results and can be used to generate a "req_str" relevant to the use case.

In [80]:

            
                Copied!
                
                    
                    
                
                

        
# build the request string for the API 
# data type is tree ring data (18), species is ponderosa pine (PIPO), limit to 5 records
# to limit the results to headersOnly, set headersOnly to true
req_params ="dataPublisher=NOAA&dataTypeId=18&species=PIPO&limit=5&headersOnly=true"
req_str = api_base + req_params

# make the request
response = requests.get(req_str)

# Convert the response to JSON
data = response.json()

# Pretty-print the JSON data to debug
print(json.dumps(data, indent=4))

# build the request string for the API 
# data type is tree ring data (18), species is ponderosa pine (PIPO), limit to 5 records
# to limit the results to headersOnly, set headersOnly to true
req_params ="dataPublisher=NOAA&dataTypeId=18&species=PIPO&limit=5&headersOnly=true"
req_str = api_base + req_params

# make the request
response = requests.get(req_str)

# Convert the response to JSON
data = response.json()

# Pretty-print the JSON data to debug
print(json.dumps(data, indent=4))

{
    "study": [
        {
            "xmlId": "64380",
            "NOAAStudyId": "26070",
            "studyName": "Azpeleta Taranc\u00f3n fire data from Mistletoe Canyon, south central New Mexico - IMPD USMC-001",
            "dataPublisher": "NOAA",
            "dataType": "FIRE HISTORY",
            "investigators": "Azpeleta Taranc\u00f3n, A.; Fule, P.Z.; S\u00e1nchez Meador, A.J.; Kim, Y.-S.; Padilla, T.",
            "onlineResourceLink": "https://www.ncei.noaa.gov/access/paleo-search/study/26070",
            "primaryInvestigator": "Azpeleta Taranc\u00f3n, A.",
            "reconstruction": "N",
            "contributionDate": "2019-02-04"
        },
        {
            "xmlId": "64375",
            "NOAAStudyId": "26071",
            "studyName": "Azpeleta Taranc\u00f3n fire data from Potato Patch NM, south central New Mexico - IMPD USPP-002",
            "dataPublisher": "NOAA",
            "dataType": "FIRE HISTORY",
            "investigators": "Azpeleta Taranc\u00f3n, A.; Fule, P.Z.; S\u00e1nchez Meador, A.J.; Kim, Y.-S.; Padilla, T.",
            "onlineResourceLink": "https://www.ncei.noaa.gov/access/paleo-search/study/26071",
            "primaryInvestigator": "Azpeleta Taranc\u00f3n, A.",
            "reconstruction": "N",
            "contributionDate": "2019-02-04"
        },
        {
            "xmlId": "64376",
            "NOAAStudyId": "26073",
            "studyName": "Azpeleta Taranc\u00f3n fire data from Snow Canyon, south central New Mexico - IMPD USSC-001",
            "dataPublisher": "NOAA",
            "dataType": "FIRE HISTORY",
            "investigators": "Azpeleta Taranc\u00f3n, A.; Fule, P.Z.; S\u00e1nchez Meador, A.J.; Kim, Y.-S.; Padilla, T.",
            "onlineResourceLink": "https://www.ncei.noaa.gov/access/paleo-search/study/26073",
            "primaryInvestigator": "Azpeleta Taranc\u00f3n, A.",
            "reconstruction": "N",
            "contributionDate": "2019-02-04"
        },
        {
            "xmlId": "64374",
            "NOAAStudyId": "26072",
            "studyName": "Azpeleta Taranc\u00f3n fire data from Solon, south central New Mexico - IMPD USS--001",
            "dataPublisher": "NOAA",
            "dataType": "FIRE HISTORY",
            "investigators": "Azpeleta Taranc\u00f3n, A.; Fule, P.Z.; S\u00e1nchez Meador, A.J.; Kim, Y.-S.; Padilla, T.",
            "onlineResourceLink": "https://www.ncei.noaa.gov/access/paleo-search/study/26072",
            "primaryInvestigator": "Azpeleta Taranc\u00f3n, A.",
            "reconstruction": "N",
            "contributionDate": "2019-02-04"
        },
        {
            "xmlId": "64381",
            "NOAAStudyId": "26074",
            "studyName": "Azpeleta Taranc\u00f3n fire data from Turkey Canyon, south central New Mexico - IMPD USTC-002",
            "dataPublisher": "NOAA",
            "dataType": "FIRE HISTORY",
            "investigators": "Azpeleta Taranc\u00f3n, A.; Fule, P.Z.; S\u00e1nchez Meador, A.J.; Kim, Y.-S.; Padilla, T.",
            "onlineResourceLink": "https://www.ncei.noaa.gov/access/paleo-search/study/26074",
            "primaryInvestigator": "Azpeleta Taranc\u00f3n, A.",
            "reconstruction": "N",
            "contributionDate": "2019-02-04"
        }
    ]
}

We want to view the data here, so we use pandas to create a DataFrame and then print it in a MarkDown format

In [81]:

            
                Copied!
                
# Convert the 'study' data to a pandas DataFrame
df = pd.DataFrame(data['study'])
# view the data as markdown

print(df.to_markdown())
# export the markdown dataframe as rendered text
with open('tree-ring-data.md', 'w') as f:
    f.write(df.to_markdown())

# view the markdown file in notebook as a new MarkDown cell
# Convert the 'study' data to a pandas DataFrame
df = pd.DataFrame(data['study'])
# view the data as markdown

print(df.to_markdown())
# export the markdown dataframe as rendered text
with open('tree-ring-data.md', 'w') as f:
    f.write(df.to_markdown())

# view the markdown file in notebook as a new MarkDown cell

|    |   xmlId |   NOAAStudyId | studyName                                                                                   | dataPublisher   | dataType     | investigators                                                                    | onlineResourceLink                                        | primaryInvestigator   | reconstruction   | contributionDate   |
|---:|--------:|--------------:|:--------------------------------------------------------------------------------------------|:----------------|:-------------|:---------------------------------------------------------------------------------|:----------------------------------------------------------|:----------------------|:-----------------|:-------------------|
|  0 |   64380 |         26070 | Azpeleta Tarancón fire data from Mistletoe Canyon, south central New Mexico - IMPD USMC-001 | NOAA            | FIRE HISTORY | Azpeleta Tarancón, A.; Fule, P.Z.; Sánchez Meador, A.J.; Kim, Y.-S.; Padilla, T. | https://www.ncei.noaa.gov/access/paleo-search/study/26070 | Azpeleta Tarancón, A. | N                | 2019-02-04         |
|  1 |   64375 |         26071 | Azpeleta Tarancón fire data from Potato Patch NM, south central New Mexico - IMPD USPP-002  | NOAA            | FIRE HISTORY | Azpeleta Tarancón, A.; Fule, P.Z.; Sánchez Meador, A.J.; Kim, Y.-S.; Padilla, T. | https://www.ncei.noaa.gov/access/paleo-search/study/26071 | Azpeleta Tarancón, A. | N                | 2019-02-04         |
|  2 |   64376 |         26073 | Azpeleta Tarancón fire data from Snow Canyon, south central New Mexico - IMPD USSC-001      | NOAA            | FIRE HISTORY | Azpeleta Tarancón, A.; Fule, P.Z.; Sánchez Meador, A.J.; Kim, Y.-S.; Padilla, T. | https://www.ncei.noaa.gov/access/paleo-search/study/26073 | Azpeleta Tarancón, A. | N                | 2019-02-04         |
|  3 |   64374 |         26072 | Azpeleta Tarancón fire data from Solon, south central New Mexico - IMPD USS--001            | NOAA            | FIRE HISTORY | Azpeleta Tarancón, A.; Fule, P.Z.; Sánchez Meador, A.J.; Kim, Y.-S.; Padilla, T. | https://www.ncei.noaa.gov/access/paleo-search/study/26072 | Azpeleta Tarancón, A. | N                | 2019-02-04         |
|  4 |   64381 |         26074 | Azpeleta Tarancón fire data from Turkey Canyon, south central New Mexico - IMPD USTC-002    | NOAA            | FIRE HISTORY | Azpeleta Tarancón, A.; Fule, P.Z.; Sánchez Meador, A.J.; Kim, Y.-S.; Padilla, T. | https://www.ncei.noaa.gov/access/paleo-search/study/26074 | Azpeleta Tarancón, A. | N                | 2019-02-04         |

Request the entire JSON response¶

Now that we have previewed the API and are satisfied with the response, we are going to re-run the code from above, but this time, we want the entire response, not just the headers.

Also, we are going to increase our limit request from limit=10 to limit=500 sites.

Note: we have removed &headersOnly=true from our req_params, the api_base request is unchanged.

In [82]:

            
                Copied!
                
                    
                    
                
                

        
# rebuild the request string for the API 
# data type is tree ring data (18), species is ponderosa pine (PIPO), limit to 500 records
# to limit the results to headersOnly, set headersOnly to true
req_params ="dataPublisher=NOAA&dataTypeId=18&species=PIPO&limit=500"
req_str = api_base + req_params

# make the request
response = requests.get(req_str)

# Convert the response to JSON
data = response.json()
# rebuild the request string for the API 
# data type is tree ring data (18), species is ponderosa pine (PIPO), limit to 500 records
# to limit the results to headersOnly, set headersOnly to true
req_params ="dataPublisher=NOAA&dataTypeId=18&species=PIPO&limit=500"
req_str = api_base + req_params

# make the request
response = requests.get(req_str)

# Convert the response to JSON
data = response.json()

Viewing the API output on a map¶

Step 4: install Python library `folium`¶

We need to install folium maps and its required dependencies into our Python environment. This can be done by using an environment.yml (see provided example), or by installing on the fly using conda, mamba, or pip

mamba install -c conda-forge folium

Step 5: Plot the locations of the API results on an interactive map¶

Create a base folium map, in our example we are zoomed in and centered over North America.

Next, we call the JSON data response from the API.

The JSON contains the geo geometric coordinates of the study area and its point location

In the map, we are tagging the doi, the Site Name and Site ID.

Optionally, if the file has a .rwl, a .pdf, or .png fire history figure associated with it, we provide a hyperlink.

On Mac OS X, press the command button ⌘ and click on the hyperlink to open in your browser.

If you click the link directly it may open in the Notebook widget. To go back, just re-run the notebook cell

In [83]:

            
                Copied!
                
                    
                    
                
                

        
# import packages for Leaflet
import folium

# Create a folium Map centered on the United States
m = folium.Map(location=[37.8, -96], zoom_start=4)

# Assuming that 'data' is your response data
for study in data['study']:
    for site in study['site']:
        lat = float(site['geo']['geometry']['coordinates'][0])
        lon = float(site['geo']['geometry']['coordinates'][1])
        # Create a popup with the site name
        popup_text = f"<a href='{study['doi']}'>url: {study['doi']}</a><br><a>Site Name: </a>{site['siteName']}<br><a>Site ID: </a>{site['NOAASiteId']}<br></a>"
        
        # Add links to any rwl files
        for file in site['paleoData'][0]['dataFile']:
            link = file['fileUrl']
            if link.endswith('.rwl'):
                # click to copy the URL to the clipboard
                popup_text += f"<br><a href='{link}'>view rwl file</a>"
                
        # Add links to any csv files
        for file in site['paleoData'][0]['dataFile']:
            link = file['fileUrl']
            if link.endswith('.csv'):
                popup_text += f"<br><a href='{link}'>Link to csv file</a>"
        
        # Add links to any PDF files
        for file in site['paleoData'][0]['dataFile']:
            link = file['fileUrl']
            if link.endswith('.pdf'):
                popup_text += f"<br><a href='{link}'>Link to PDF file</a>"
        
        # Add links to any figure files
        for file in site['paleoData'][0]['dataFile']:
            link = file['fileUrl']
            if link.endswith('.png'):
                popup_text += f"<br><a href='{link}'>Link to figure file</a>"
        
        popup = folium.Popup(popup_text, max_width=250)
        folium.Marker([lat, lon], popup=popup).add_to(m)

# Display the Map
m
# import packages for Leaflet
import folium

# Create a folium Map centered on the United States
m = folium.Map(location=[37.8, -96], zoom_start=4)

# Assuming that 'data' is your response data
for study in data['study']:
    for site in study['site']:
        lat = float(site['geo']['geometry']['coordinates'][0])
        lon = float(site['geo']['geometry']['coordinates'][1])
        # Create a popup with the site name
        popup_text = f"url: {study['doi']}
Site Name: {site['siteName']}
Site ID: {site['NOAASiteId']}
"
        
        # Add links to any rwl files
        for file in site['paleoData'][0]['dataFile']:
            link = file['fileUrl']
            if link.endswith('.rwl'):
                # click to copy the URL to the clipboard
                popup_text += f"
view rwl file"
                
        # Add links to any csv files
        for file in site['paleoData'][0]['dataFile']:
            link = file['fileUrl']
            if link.endswith('.csv'):
                popup_text += f"
Link to csv file"
        
        # Add links to any PDF files
        for file in site['paleoData'][0]['dataFile']:
            link = file['fileUrl']
            if link.endswith('.pdf'):
                popup_text += f"
Link to PDF file"
        
        # Add links to any figure files
        for file in site['paleoData'][0]['dataFile']:
            link = file['fileUrl']
            if link.endswith('.png'):
                popup_text += f"
Link to figure file"
        
        popup = folium.Popup(popup_text, max_width=250)
        folium.Marker([lat, lon], popup=popup).add_to(m)

# Display the Map
m

Out[83]:

Make this Notebook Trusted to load map: File -> Trust Notebook

Saving the Map for later use¶

Below is a script to save the map as .html to your computer

In [85]:

            
                Copied!
                
                    
                    
                
                

        
from datetime import datetime
save_map = input("Do you want to save the map.html locally? (y/n): ")
if save_map == 'y':
    # ask what name to save the map as
    map_name = input("What do you want to name the map? ")
    m.save(f'map_{map_name}.html')
    # if nothing entered
    if map_name == '':
        # Get the current date and time
        now = datetime.now()
        # Format the date and time as a string
        map_name = now.strftime("%Y%m%d_%H%M%S")
        # Use the timestamp in the filename
        m.save(f'map_{map_name}.html')
else:
    print('not saved')
from datetime import datetime
save_map = input("Do you want to save the map.html locally? (y/n): ")
if save_map == 'y':
    # ask what name to save the map as
    map_name = input("What do you want to name the map? ")
    m.save(f'map_{map_name}.html')
    # if nothing entered
    if map_name == '':
        # Get the current date and time
        now = datetime.now()
        # Format the date and time as a string
        map_name = now.strftime("%Y%m%d_%H%M%S")
        # Use the timestamp in the filename
        m.save(f'map_{map_name}.html')
else:
    print('not saved')

not saved

Step 7: Download the `.rwl` for a selected site using the `siteID`¶

In [86]:

            
                Copied!
                
                    
                    
                
                

        
# download a .rwl file directly from ITRDB website

import requests

def download_rwl_file(url, file_path):
    """
    Download a .rwl file from a given URL and save it locally.

    :param url: URL of the .rwl file
    :param file_path: Local path to save the file
    """
    try:
        response = requests.get(url)
        response.raise_for_status()  # Check if the request was successful

        with open(file_path, 'wb') as file:
            file.write(response.content)
        print(f"File downloaded successfully: {file_path}")
    except requests.RequestException as e:
        print(f"Error downloading the file: {e}")
# download a .rwl file directly from ITRDB website

import requests

def download_rwl_file(url, file_path):
    """
    Download a .rwl file from a given URL and save it locally.

    :param url: URL of the .rwl file
    :param file_path: Local path to save the file
    """
    try:
        response = requests.get(url)
        response.raise_for_status()  # Check if the request was successful

        with open(file_path, 'wb') as file:
            file.write(response.content)
        print(f"File downloaded successfully: {file_path}")
    except requests.RequestException as e:
        print(f"Error downloading the file: {e}")

In [87]:

            
                Copied!
                
# Example Usage
url = "https://www.ncei.noaa.gov/pub/data/paleo/treering/measurements/northamerica/usa/az547.rwl"
file_path = "az547.rwl"  # Replace with your desired path
download_rwl_file(url, file_path)
# Example Usage
url = "https://www.ncei.noaa.gov/pub/data/paleo/treering/measurements/northamerica/usa/az547.rwl"
file_path = "az547.rwl"  # Replace with your desired path
download_rwl_file(url, file_path)

File downloaded successfully: az547.rwl

In [88]:

            
                Copied!
                
import os
import dplpy as dpl

# set the working directory
print("Current Working Directory:", os.getcwd())
import os
import dplpy as dpl

# set the working directory
print("Current Working Directory:", os.getcwd())

Current Working Directory: /home/jovyan/data-store

In [89]:

            
                Copied!
                
help(dpl.readers)
help(dpl.readers)

Help on function readers in module readers:

readers(filename: str, skip_lines=0, header=False)
    Imports a common ring width data file

    Extended Summary
    ----------------
    This function imports common ring width data files (.csv, .rwls) as arrays

    Parameters
    ----------
    filename : str
               a data file (.CSV or .RWL) or a pandas dataframe imported from dpl.readers().

    Returns
    -------
    data : pandas dataframe

    Examples
    --------
    >>> import dplpy as dpl
    >>> data = dpl.readers("../tests/data/csv/file.csv")
    >>> data = dpl.readers("../tests/data/csv/file.rwl", header=True)

    References
    ----------
    .. [1] https:/opendendro.org/dplpy-man/#readers

In [90]:

            
                Copied!
                
data = dpl.readers("/home/jovyan/data-store/az547.rwl", header=True)
data = dpl.readers("/home/jovyan/data-store/az547.rwl", header=True)

Attempting to read input file: az547.rwl as .rwl format


SUCCESS!
File read as: .rwl file

Series names:
['WCP011', 'WCP012', 'WCP021', 'WCP022', 'WCP031', 'WCP032', 'WCP041', 'WCP042', 'WCP051', 'WCP052', 'WCP061', 'WCP062', 'WCP071', 'WCP072', 'WCP081', 'WCP082', 'WCP091', 'WCP092', 'WCP101', 'WCP111', 'WCP112']

In [91]:

            
                Copied!
                
dpl.summary(data)
dpl.summary(data)

Out[91]:

	WCP011	WCP012	WCP021	WCP022	WCP031	WCP032	WCP041	WCP042	WCP051	WCP052	...	WCP062	WCP071	WCP072	WCP081	WCP082	WCP091	WCP092	WCP101	WCP111	WCP112
count	367.000000	387.000000	574.000000	568.000000	270.000000	268.000000	377.000000	378.000000	378.000000	379.000000	...	460.000000	367.000000	368.000000	310.000000	308.000000	308.000000	305.000000	371.000000	366.000000	383.000000
mean	0.889591	1.119612	0.569948	0.518486	1.027111	1.139478	0.982414	0.948386	0.830873	0.875831	...	0.583478	0.946322	1.052201	1.117097	1.098019	1.198247	1.120295	0.842911	1.006284	1.016345
std	0.726893	0.782691	0.460331	0.313141	0.834130	0.826676	0.970882	0.694060	0.801188	0.618304	...	0.613571	0.687315	0.749256	0.564160	0.652438	0.748183	0.707423	0.870879	0.690556	0.665793
min	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	...	0.000000	0.020000	0.050000	0.130000	0.110000	0.000000	0.000000	0.000000	0.000000	0.000000
25%	0.375000	0.620000	0.282500	0.307500	0.420000	0.517500	0.380000	0.472500	0.320000	0.465000	...	0.170000	0.440000	0.430000	0.660000	0.557500	0.657500	0.640000	0.190000	0.530000	0.545000
50%	0.670000	0.930000	0.480000	0.470000	0.705000	0.830000	0.640000	0.795000	0.540000	0.730000	...	0.385000	0.690000	0.780000	0.980000	0.960000	0.955000	0.890000	0.360000	0.840000	0.870000
75%	1.160000	1.380000	0.750000	0.690000	1.455000	1.672500	1.320000	1.227500	1.080000	1.035000	...	0.762500	1.220000	1.542500	1.537500	1.592500	1.597500	1.380000	1.350000	1.267500	1.300000
max	3.450000	4.480000	4.870000	2.040000	4.150000	3.840000	5.760000	3.800000	3.760000	3.210000	...	3.300000	3.100000	3.250000	3.250000	2.910000	3.530000	3.740000	3.670000	3.780000	4.350000

8 rows × 21 columns

In [92]:

            
                Copied!
                
dpl.detrend(data["WCP011"])
dpl.detrend(data["WCP011"])

Out[92]:

1621    1.458716
1622    1.233723
1623    0.719140
1624    0.366121
1625    0.944886
          ...   
1983    1.431906
1984    0.832821
1985    1.838013
1986    1.168667
1987    1.274638
Name: WCP011, Length: 367, dtype: float64

In [93]:

            
                Copied!
                
dpl.autoreg(data["WCP011"], 20)
dpl.autoreg(data["WCP011"], 20)

Out[93]:

const         0.058818
WCP011.L1     0.370037
WCP011.L2     0.270136
WCP011.L3    -0.047990
WCP011.L4     0.016080
WCP011.L5    -0.009716
WCP011.L6     0.107719
WCP011.L7    -0.028582
WCP011.L8    -0.003267
WCP011.L9     0.065848
WCP011.L10   -0.072174
WCP011.L11   -0.035382
WCP011.L12    0.096421
WCP011.L13    0.031105
WCP011.L14   -0.046085
WCP011.L15    0.132907
WCP011.L16    0.034036
WCP011.L17   -0.135831
WCP011.L18    0.149418
dtype: float64

In [94]:

            
                Copied!
                
dpl.plot(data,type="spag")
dpl.plot(data,type="spag")

In [ ]:

            
                Copied!
                
dpl.xdate(data, prewhiten=True, corr="Spearman", slide_period=50,bin_floor=100, p_val=0.05, show_flags=True)
dpl.xdate(data, prewhiten=True, corr="Spearman", slide_period=50,bin_floor=100, p_val=0.05, show_flags=True)

Flags for WCP022
[B] Segment  High   -10    -9    -8    -7    -6    -5    -4    -3    -2    -1     0    +1    +2    +3    +4    +5    +6    +7    +8    +9   +10
   1525-1574   -6 -0.05  0.07 -0.14 -0.00  0.38  0.00 -0.29  0.05  0.16  0.13  0.27  0.05 -0.35 -0.01  0.01 -0.05  0.05 -0.14  0.02 -0.24  0.08

In [ ]:

            
                Copied!
                
dpl.xdate_plot(data)
dpl.xdate_plot(data)

In [ ]:

            
                Copied!
                
df_rwi = dpl.detrend(data, fit="spline", period=50, method="residual", plot=True)
df_rwi = dpl.detrend(data, fit="spline", period=50, method="residual", plot=True)

In [ ]:

Last update: 2024-01-09

NCEI Paleo Database API¶

About¶

Tutorial 1: Using Python to access the API for tree ring .rwl data¶

Step 1: Activate/Create a Python environment¶

Step 2: Set the base url for the API as api_base¶

Step 3: Search via API for tree ring (dataTypeID=18) studies of raw ring width.¶

Request the entire JSON response¶

Viewing the API output on a map¶

Step 4: install Python library folium¶

Step 5: Plot the locations of the API results on an interactive map¶

Saving the Map for later use¶

Step 7: Download the .rwl for a selected site using the siteID¶

Tutorial 1: Using Python to access the API for tree ring `.rwl` data¶

Step 2: Set the base url for the API as `api_base`¶

Step 3: Search via API for tree ring (`dataTypeID=18`) studies of raw ring width.¶

Step 4: install Python library `folium`¶

Step 7: Download the `.rwl` for a selected site using the `siteID`¶