Urban Climate Explorer

Explore and Emulate Urban Climate on AWS Cloud.

About

Introduction

This platform enables free and easy access to the urban climate simulations provided by National Center for Atmospheric Research and Amazon Web Services (AWS) via Cloud Computing. By providing the necessary information as the input (e.g., time, latitude, longitude, climate scenarios, etc.), users can explore and utilize urban climate data. Specifically, users can:

visualize/analyze urban climate of a particular city/cities under different climate change scenarios and different version model simulations (e.g., urban heat waves analysis)
train fast machine learning emulators of the urban climate (e.g., mapping from radiation to urban temperature) using a Automated Machine Learning tool (FLAML)
apply the machine learning emulators to users’ own data to create customized urban climate projections for their own needs

Relevant Publications

Zheng, Z., Zhao, L. & Oleson, K.W. Large model structural uncertainty in global projections of urban heat waves. Nat Commun 12, 3736 (2021). https://doi.org/10.1038/s41467-021-24113-9
Zhao, L., Oleson, K., Bou-Zeid, E. et al. Global multi-model projections of local urban climates. Nat. Clim. Chang. 11, 152–157 (2021). https://doi.org/10.1038/s41558-020-00958-8

Concept

_images/concept.png concept

Technical Workflow

_images/workflow.png workflow

Install and Run

Install

use conda to install the environment

$ git clone git@github.com:zzheng93/UrbanClimateExplorer.git
$ cd ./UrbanClimateExplorer/binder 
$ conda env create -f environment.yml
$ conda activate aws_urban

Run

Locally

$ cd ./UrbanClimateExplorer
$ git pull
$ cd ./docs/notebooks 
$ jupyter notebook

HPC (e.g., NCAR’s Casper clusters with a GPU)
- First, create a bash script (see below), and name it as aws_urban_env.sh, put it in the same folder with your UrbanClimateExplorer folder.
```
#!/bin/bash
source  /glade/work/zhonghua/miniconda3/bin/activate aws_urban
echo "ssh -N -L 8889:`hostname`:8889 $USER@`hostname`.ucar.edu"
jupyter notebook --no-browser --ip=`hostname` --port=8889
```
- Second, run the commands below
  
  Note: please use your own job code instead of “UIUC0021”. You can find more information about execcasper here
```
$ execcasper -A UIUC0021 -l gpu_type=v100 -l walltime=06:00:00 -l select=1:ncpus=18:mpiprocs=36:ngpus=1:mem=100GB
$ bash aws_urban_env.sh
```
- Thrid, launch a new terminal, copy and paste the command printed by the “echo” command, and log in. Then open your browser (e.g., Google Chrome), type https://localhost:8889.
  
  Note: Sometimes port 8889 may be used by others. In this case, please adjust your bash script accordingly, e.g., from 8889 to 8892:
```
#!/bin/bash
source  /glade/work/zhonghua/miniconda3/bin/activate aws_urban
echo "ssh -N -L 8889:`hostname`:8892 $USER@`hostname`.ucar.edu"
jupyter notebook --no-browser --ip=`hostname` --port=8892
```

Setup

Step 1: Follow Install and Run to launch a jupyter notebook locally or using HPC.
Step 2: Open config_cesm1.json or config_cesm2.json.

load_JSON
Step 3: If you are not familiar with CESM1 or CESM2, you can just edit city_loc and time_start and time_end.
- Please check here for more information about JSON file.
- Please check here for CESM1 variables and here for CESM2 variables
Step 4: Save the JSON file

save_JSON
Step 5: Open CESM1_demo.ipynb or CESM2_demo.ipynb, depends on which JSON you have edited. Now you are all set!

load_notebook

Example for CESM1

Reference:
- GitHub: https://github.com/ncar/cesm-lens-aws/
- Data/Variables Information: https://ncar.github.io/cesm-lens-aws/#data-catalog
- Reproduce CESM-LENS: http://gallery.pangeo.io/repos/NCAR/cesm-lens-aws/notebooks/kay-et-al-2015.v3.html

Step 0: load necessary packages and define parameters (no need to change)

[1]:

%%time
# Display output of plots directly in Notebook
%matplotlib inline
import matplotlib.pyplot as plt
import pandas as pd
import json
from flaml import AutoML
from sklearn.metrics import mean_squared_error, r2_score
import warnings
warnings.filterwarnings("ignore")
import util

with open("./config_cesm1.json",'r') as load_f:
#     param = json.loads(json.load(load_f))
    param = json.load(load_f)

    model = param["model"] # cesm1
    city_loc = param["city_loc"] # {"lat": 40.1164, "lon": -88.2434}
    l_component = param["l_component"]
    a_component = param["a_component"]
    experiment = param["experiment"]
    frequency = param["frequency"]
    cam_ls = param["cam_ls"]
    clm_ls = param["clm_ls"]
    time = slice(param["time_start"],param["time_end"])
    member_id = param["member_id"]
    estimator_list = param["estimator_list"]
    time_budget = param["time_budget"]
    features = param["features"]
    label = param["label"]
    clm_var_mask = param["label"][0]

# get a dataset
ds = util.get_data(model, city_loc, experiment, frequency, member_id, time, cam_ls, clm_ls)

# create a dataframe
ds['time'] = ds.indexes['time'].to_datetimeindex()
df = ds.to_dataframe().reset_index().dropna()

if "PRSN" in features:
    df["PRSN"] = df["PRECSC"] + df["PRECSL"]

# setup for automl
automl = AutoML()
automl_settings = {
    "time_budget": time_budget,  # in seconds
    "metric": 'r2',
    "task": 'regression',
    "estimator_list":estimator_list,
}

/glade/work/zhonghua/miniconda3/envs/aws_urban/lib/python3.8/site-packages/xgboost/compat.py:31: FutureWarning: pandas.Int64Index is deprecated and will be removed from pandas in a future version. Use pandas.Index with the appropriate dtype instead.
  from pandas import MultiIndex, Int64Index


--> The keys in the returned dictionary of datasets are constructed as follows:
        'component.experiment.frequency'

100.00% [2/2 00:00<00:00]

CPU times: user 47.5 s, sys: 24.2 s, total: 1min 11s
Wall time: 42.2 s

Step 1: data analysis

xarray.Dataset

[2]:

ds

pandas dataframe

[3]:

df.head()

[3]:

	member_id	time	TREFHT	TREFHTMX	FLNS	FSNS	PRECSL	PRECT	QBOT	UBOT	VBOT	lat	lon	TREFMXAV_U	PRSN
0	2	2081-01-02 12:00:00	267.530823	270.858276	67.127922	93.389015	3.047973e-09	3.047973e-09	0.001285	7.930864	0.497645	40.052357	271.25	291.566376	3.047973e-09
1	2	2081-01-03 12:00:00	267.687714	276.013184	75.707771	89.367584	8.636923e-10	8.637052e-10	0.001422	1.265992	2.597399	40.052357	271.25	271.861420	8.636923e-10
2	2	2081-01-04 12:00:00	275.672028	283.799103	53.098457	88.202408	1.331122e-09	2.900762e-09	0.001883	-1.618823	5.256704	40.052357	271.25	276.851349	1.331122e-09
3	2	2081-01-05 12:00:00	276.890686	284.918884	38.585438	43.366203	0.000000e+00	1.297357e-09	0.004626	-2.739517	5.846395	40.052357	271.25	284.498749	0.000000e+00
4	2	2081-01-06 12:00:00	287.462036	291.275452	7.098297	38.443039	1.156371e-14	6.182010e-08	0.009604	-0.635833	8.809414	40.052357	271.25	285.547607	1.156371e-14

data visualization

[4]:

ds["TREFMXAV_U"].plot()

[4]:

[<matplotlib.lines.Line2D at 0x2b4ed40910d0>]

Step 2: automated machine learning

train a model (emulator)

[5]:

%%time
# assume that we want to split the data into training data and testing data
# let's use first 95% for training, and the remaining for testing
idx = df.shape[0]
train = df.iloc[:int(0.95*idx),:]
test = df.iloc[int(0.95*idx):,:]
(X_train, y_train) = (train[features], train[label].values)
(X_test, y_test) = (test[features], test[label].values)

# train the model
automl.fit(X_train=X_train, y_train=y_train,
           **automl_settings, verbose=-1)
print(automl.model.estimator)

LGBMRegressor(colsample_bytree=0.6649148062238498,
              learning_rate=0.17402065726724145, max_bin=255,
              min_child_samples=3, n_estimators=93, num_leaves=15,
              reg_alpha=0.0009765625, reg_lambda=0.0067613624509965,
              verbose=-1)
CPU times: user 3min 4s, sys: 3.09 s, total: 3min 7s
Wall time: 15.5 s

apply and test the machine learning model

use automl.predict(X) to apply the model

[6]:

# training data
print("model performance using training data:")
y_pred = automl.predict(X_train)
print("root mean square error:",
      mean_squared_error(y_true=y_train, y_pred=y_pred, squared=False))
print("r2:", r2_score(y_true=y_train, y_pred=y_pred),"\n")
import pandas as pd
d_train = {"time":train["time"],"y_train":y_train.reshape(-1),"y_pred":y_pred.reshape(-1)}
df_train = pd.DataFrame(d_train).set_index("time")

# testing data
print("model performance using testing data:")
y_pred = automl.predict(X_test)
print("root mean square error:",
      mean_squared_error(y_true=y_test, y_pred=y_pred, squared=False))
print("r2:", r2_score(y_true=y_test, y_pred=y_pred))
d_test = {"time":test["time"],"y_test":y_test.reshape(-1),"y_pred":y_pred.reshape(-1)}
df_test = pd.DataFrame(d_test).set_index("time")

model performance using training data:
root mean square error: 1.891482463886013
r2: 0.9646972730399841

model performance using testing data:
root mean square error: 2.4148976103187043
r2: 0.9496869325210374

visualization

[7]:

fig, (ax1,ax2) = plt.subplots(1,2,figsize=(12,3))
fig.suptitle('emulator evaluation')
df_train["y_train"].plot(label="reference",c="k",ax=ax1)
df_train["y_pred"].plot(label="prediction",c="r",ax=ax1)
ax1.set_title("training data")
ax1.set_ylabel("urban daily maximum temperature, K")

df_test["y_test"].plot(label="reference",c="k",ax=ax2)
df_test["y_pred"].plot(label="prediction",c="r",ax=ax2)
ax2.set_title("testing data")
plt.legend()
plt.show()

Example for CESM2

NOTE: Compared to the CESM1 demo, here “Q” (QBOT), “U” (UBOT) and “V” (VBOT) are not included. When the bottom “lev” of “Q”, “U”, and “V” are merged, there is an issue.

Reference:
- GitHub: https://github.com/NCAR/cesm2-le-aws
- Data/Variables Information: https://ncar.github.io/cesm2-le-aws/model_documentation.html#data-catalog
- Reproduce CESM-LENS: https://github.com/NCAR/cesm2-le-aws/blob/main/notebooks/kay_et_al_lens2.ipynb

Step 0: load necessary packages and define parameters (no need to change)

[1]:

%%time
# Display output of plots directly in Notebook
%matplotlib inline
import matplotlib.pyplot as plt
import pandas as pd
import json
from flaml import AutoML
from sklearn.metrics import mean_squared_error, r2_score
import warnings
warnings.filterwarnings("ignore")
import util

with open("./config_cesm2.json",'r') as load_f:
#     param = json.loads(json.load(load_f))
    param = json.load(load_f)

    model = param["model"] # cesm2
    urban_type = param["urban_type"] # md
    city_loc = param["city_loc"] # {"lat": 40.1164, "lon": -88.2434}
    l_component = param["l_component"]
    a_component = param["a_component"]
    experiment = param["experiment"]
    frequency = param["frequency"]
    cam_ls = param["cam_ls"]
    clm_ls = param["clm_ls"]
    forcing_variant = param["forcing_variant"]
    time = slice(param["time_start"],param["time_end"])
    member_id = param["member_id"]
    estimator_list = param["estimator_list"]
    time_budget = param["time_budget"]
    features = param["features"]
    label = param["label"]
    clm_var_mask = param["label"][0]

# get a dataset
ds = util.get_data(model, city_loc, experiment, frequency, member_id, time, cam_ls, clm_ls,
                   forcing_variant=forcing_variant, urban_type=urban_type)

# create a dataframe
ds['time'] = ds.indexes['time'].to_datetimeindex()
df = ds.to_dataframe().reset_index().dropna()

if "PRSN" in features:
    df["PRSN"] = df["PRECSC"] + df["PRECSL"]
if "PRECT" in features:
    df["PRECT"] = df["PRECC"] + df["PRECL"]

# setup for automl
automl = AutoML()
automl_settings = {
    "time_budget": time_budget,  # in seconds
    "metric": 'r2',
    "task": 'regression',
    "estimator_list":estimator_list,
}

/glade/work/zhonghua/miniconda3/envs/aws_urban/lib/python3.8/site-packages/xgboost/compat.py:31: FutureWarning: pandas.Int64Index is deprecated and will be removed from pandas in a future version. Use pandas.Index with the appropriate dtype instead.
  from pandas import MultiIndex, Int64Index


--> The keys in the returned dictionary of datasets are constructed as follows:
        'component.experiment.frequency.forcing_variant'

100.00% [2/2 00:00<00:00]

different lat between CAM and CLM subgrid info, adjust subgrid info's lat
CPU times: user 55.3 s, sys: 32 s, total: 1min 27s
Wall time: 53.7 s

Step 1: data analysis

xarray.Dataset

[2]:

ds

pandas dataframe

[3]:

df.head()

[3]:

	member_id	time	TREFHT	TREFHTMX	FLNS	FSNS	PRECSL	PRECL	lat	lon	TREFMXAV	PRSN	PRECT
0	r1i1231p1f1	2081-01-02 12:00:00	275.049347	277.329407	59.856152	90.956940	1.516594e-10	4.766932e-10	40.052356	271.25	277.536102	1.516594e-10	4.766932e-10
1	r1i1231p1f1	2081-01-03 12:00:00	272.927429	274.842499	68.446648	74.588417	3.641785e-09	3.642020e-09	40.052356	271.25	275.493225	3.641785e-09	3.642020e-09
2	r1i1231p1f1	2081-01-04 12:00:00	273.798920	275.402435	24.263290	38.506813	1.956359e-09	2.209635e-09	40.052356	271.25	275.750916	1.956359e-09	2.209635e-09
3	r1i1231p1f1	2081-01-05 12:00:00	277.125549	286.210144	59.307308	86.190071	1.833845e-14	4.872912e-14	40.052356	271.25	286.460663	1.833845e-14	4.872912e-14
4	r1i1231p1f1	2081-01-06 12:00:00	280.623657	282.968323	38.607136	17.733625	3.573996e-24	2.074043e-10	40.052356	271.25	284.418915	3.573996e-24	2.074043e-10

data visualization

[4]:

ds["TREFMXAV"].plot()

[4]:

[<matplotlib.lines.Line2D at 0x2b60524db1c0>]

Step 2: automated machine learning

train a model (emulator)

[5]:

%%time
# assume that we want to split the data into training data and testing data
# let's use first 95% for training, and the remaining for testing
idx = df.shape[0]
train = df.iloc[:int(0.95*idx),:]
test = df.iloc[int(0.95*idx):,:]
(X_train, y_train) = (train[features], train[label].values)
(X_test, y_test) = (test[features], test[label].values)

# train the model
automl.fit(X_train=X_train, y_train=y_train,
           **automl_settings, verbose=-1)
print(automl.model.estimator)

LGBMRegressor(colsample_bytree=0.7463308378914483,
              learning_rate=0.1530612501227463, max_bin=1023,
              min_child_samples=2, n_estimators=60, num_leaves=49,
              reg_alpha=0.0009765625, reg_lambda=0.012698515198279536,
              verbose=-1)
CPU times: user 3min 4s, sys: 4.67 s, total: 3min 8s
Wall time: 15.5 s

apply and test the machine learning model

use automl.predict(X) to apply the model

[6]:

# training data
print("model performance using training data:")
y_pred = automl.predict(X_train)
print("root mean square error:",
      mean_squared_error(y_true=y_train, y_pred=y_pred, squared=False))
print("r2:", r2_score(y_true=y_train, y_pred=y_pred),"\n")
import pandas as pd
d_train = {"time":train["time"],"y_train":y_train.reshape(-1),"y_pred":y_pred.reshape(-1)}
df_train = pd.DataFrame(d_train).set_index("time")

# testing data
print("model performance using testing data:")
y_pred = automl.predict(X_test)
print("root mean square error:",
      mean_squared_error(y_true=y_test, y_pred=y_pred, squared=False))
print("r2:", r2_score(y_true=y_test, y_pred=y_pred))
d_test = {"time":test["time"],"y_test":y_test.reshape(-1),"y_pred":y_pred.reshape(-1)}
df_test = pd.DataFrame(d_test).set_index("time")

model performance using training data:
root mean square error: 1.0634138507297426
r2: 0.9928674646696605

model performance using testing data:
root mean square error: 1.6361120260552937
r2: 0.9852130533126713

visualization

[7]:

fig, (ax1,ax2) = plt.subplots(1,2,figsize=(12,3))
fig.suptitle('emulator evaluation')
df_train["y_train"].plot(label="reference",c="k",ax=ax1)
df_train["y_pred"].plot(label="prediction",c="r",ax=ax1)
ax1.set_title("training data")
ax1.set_ylabel("urban daily maximum temperature, K")

df_test["y_test"].plot(label="reference",c="k",ax=ax2)
df_test["y_pred"].plot(label="prediction",c="r",ax=ax2)
ax2.set_title("testing data")
plt.legend()
plt.show()

How to ask for help?

The GitHub issue tracker is the primary place for bug reports.

Acknowledgments

We thank AWS for providing AWS Cloud Credits for Research.

We would like to acknowledge high-performance computing support from Cheyenne (doi:10.5065/D6RX99HX) provided by NCAR’s Computational and Information Systems Laboratory, sponsored by the National Science Foundation.

Urban Climate Explorer

About

Introduction

Relevant Publications

Concept

Technical Workflow

Install and Run

Install

Run

Setup

Example for CESM1

Step 1: data analysis

Step 2: automated machine learning

Example for CESM2

Step 1: data analysis

Step 2: automated machine learning

How to create a JSON file?

How to create a mask for CESM1’s “urban areas”?

How to create a subgrid info file for CESM2’s CLM processing?

How to ask for help?

Acknowledgments