Skip to content
Snippets Groups Projects

Compare revisions

Changes are shown as if the source revision was being merged into the target revision. Learn more about comparing revisions.

Source

Select target project
No results found

Target

Select target project
  • loyolay/hackaton-entrega-grupo-3
  • hackathon-bella/plantillajupyterbook
  • copernicus-hackaton/plantillajupyterbook
3 results
Show changes
Commits on Source (19)
Hide whitespace changes
Inline Side-by-side
0_download_data.ipynb 0 → 100644
View file @ aa23b2c3
%% Cell type:markdown id: tags:
# Cargar bibliotecas
%% Cell type:code id: tags:
```
# CDS API
import cdsapi
```
%% Cell type:markdown id: tags:
# Directorios para almacenar datos
%% Cell type:code id: tags:
```
DATADIR = 'era5/'
LABELDIR = 'fire_danger/'
```
%% Cell type:markdown id: tags:
# Descargar datos ERA5 de Argentina desde 2002 hasta 2022
%% Cell type:code id: tags:
```
for year in range(2002,2023):
target_file = DATADIR + f'{year}.nc'
c = cdsapi.Client()
c.retrieve(
'reanalysis-era5-single-levels',
{
'product_type': 'reanalysis',
'format': 'netcdf',
'month': [
'01','02','12'
],
'day': [
'01', '02', '03',
'04', '05', '06',
'07', '08', '09',
'10', '11', '12',
'13', '14', '15',
'16', '17', '18',
'19', '20', '21',
'22', '23', '24',
'25', '26', '27',
'28', '29', '30',
'31',
],
'time': [
'15:00',
],
'variable': [
'10m_u_component_of_wind', '10m_v_component_of_wind', '2m_temperature',
'leaf_area_index_high_vegetation', 'leaf_area_index_low_vegetation', 'total_precipitation',
],
'year': [str(year)],
'area': [
-20, -79, -57, # North West South East
-43,
],
},
target_file)
```
%% Cell type:markdown id: tags:
# Descargar datos del índice de peligro de incendios
%% Cell type:code id: tags:
```
for year in range(2002,2023):
target_file = LABELDIR + f'{year}.nc'
c = cdsapi.Client()
c.retrieve(
'cems-fire-historical-v1',
{
'product_type': 'reanalysis',
'variable': 'fire_danger_index',
'dataset_type': 'consolidated_dataset',
'system_version': '4_1',
'year': '2002',
'month': [
'01', '02', '12',
],
'day': [
'01', '02', '03',
'04', '05', '06',
'07', '08', '09',
'10', '11', '12',
'13', '14', '15',
'16', '17', '18',
'19', '20', '21',
'22', '23', '24',
'25', '26', '27',
'28', '29', '30',
'31',
],
'area': [
-20, -79, -57,
-43,
],
'grid': '0.25/0.25',
'format': 'netcdf',
},
target_file)
```
%% Output
2023-07-28 12:15:49,638 INFO Welcome to the CDS
2023-07-28 12:15:49,639 INFO Sending request to https://cds.climate.copernicus.eu/api/v2/resources/cems-fire-historical-v1
2023-07-28 12:15:49,854 INFO Request is completed
2023-07-28 12:15:49,855 INFO Downloading https://download-0004-clone.copernicus-climate.eu/cache-compute-0004/cache/data0/adaptor.mars.external-1690558130.304354-9896-9-762d05cd-253d-467d-99ae-08ef92f88bbd.nc to fire_danger/2019.nc (7.4M)
2023-07-28 12:15:52,491 INFO Download rate 2.8M/s
2023-07-28 12:15:53,197 INFO Welcome to the CDS
2023-07-28 12:15:53,199 INFO Sending request to https://cds.climate.copernicus.eu/api/v2/resources/cems-fire-historical-v1
2023-07-28 12:15:53,439 INFO Request is completed
2023-07-28 12:15:53,440 INFO Downloading https://download-0004-clone.copernicus-climate.eu/cache-compute-0004/cache/data0/adaptor.mars.external-1690558130.304354-9896-9-762d05cd-253d-467d-99ae-08ef92f88bbd.nc to fire_danger/2020.nc (7.4M)
2023-07-28 12:15:54,590 INFO Download rate 6.5M/s
2023-07-28 12:15:54,976 INFO Welcome to the CDS
2023-07-28 12:15:54,977 INFO Sending request to https://cds.climate.copernicus.eu/api/v2/resources/cems-fire-historical-v1
2023-07-28 12:15:55,220 INFO Request is completed
2023-07-28 12:15:55,221 INFO Downloading https://download-0004-clone.copernicus-climate.eu/cache-compute-0004/cache/data0/adaptor.mars.external-1690558130.304354-9896-9-762d05cd-253d-467d-99ae-08ef92f88bbd.nc to fire_danger/2021.nc (7.4M)
2023-07-28 12:15:56,690 INFO Download rate 5.1M/s
2023-07-28 12:15:57,082 INFO Welcome to the CDS
2023-07-28 12:15:57,083 INFO Sending request to https://cds.climate.copernicus.eu/api/v2/resources/cems-fire-historical-v1
2023-07-28 12:15:57,301 INFO Request is completed
2023-07-28 12:15:57,302 INFO Downloading https://download-0004-clone.copernicus-climate.eu/cache-compute-0004/cache/data0/adaptor.mars.external-1690558130.304354-9896-9-762d05cd-253d-467d-99ae-08ef92f88bbd.nc to fire_danger/2022.nc (7.4M)
2023-07-28 12:15:58,812 INFO Download rate 4.9M/s
1_prepare_data.ipynb 0 → 100644
View file @ aa23b2c3
%% Cell type:markdown id: tags:
# Configurar directorios
%% Cell type:code id: tags:
``` python
DATADIR = 'era5/' # directory containing downloaded era5 data
FIREDATADIR = 'fire_danger/' # directory containing fire data
DESTDIR = 'processed_data/' # directory to save .npy files for each time step and variable
```
%% Cell type:markdown id: tags:
# Cargar bibliotecas
%% Cell type:code id: tags:
``` python
import numpy as np
import netCDF4 as nc
import os
from tqdm.notebook import tqdm
```
%% Cell type:markdown id: tags:
# Variables de configuración
%% Cell type:code id: tags:
``` python
vars = ['u10','v10','t2m','lai_hv','lai_lv','tp','fdimrk'] #considered variables (see 0_download_data.ipynb for long names)
months = [(1,31),(2,28),(12,31)] # months + days in month in dowloaded era5 .nc files
years = np.arange(2002,2023) # downloaded years
```
%% Cell type:markdown id: tags:
# Procesamiento de datos para crear archivos .npy
%% Cell type:code id: tags:
``` python
# Processing Data to create .npy files
for var in vars:
if not os.path.exists(DESTDIR + f"{var}"):
os.makedirs(DESTDIR + f"{var}")
for year in tqdm(years):
if var=='fdimrk':
root = nc.Dataset(FIREDATADIR + f"{year:d}.nc", 'r')
else:
root = nc.Dataset(DATADIR + f"{year:d}.nc", 'r')
v = root.variables[var][:,:-9,:-5] #crop to get to a size suitable for the considered Unet-like model, here 140x140
v = v.data
root.close()
if var in ['tp']: #change unit from m to mm for precipitation
v = 1000 * v
t = 0 # time step within v array that I am working on
for month, days in months:
for day in range(days):
np.save(DESTDIR + f"{var}/{year}_{month:02d}_{day+1:02d}.npy",v[t])
t += 1
```
%% Cell type:markdown id: tags:
# Guardar información de latitud/longitud
%% Cell type:code id: tags:
``` python
root = nc.Dataset(DATADIR + f"2002.nc", 'r') #constant in time -> take from any year
lat = root.variables['latitude'][:-9].data #crop to get to a size suitable for the considered Unet-like model
lon = root.variables['longitude'][:-5].data
np.save(DESTDIR + 'lat.npy', lat)
np.save(DESTDIR + 'lon.npy', lon)
```
2_norm_consts.ipynb 0 → 100644
View file @ aa23b2c3
%% Cell type:markdown id: tags:
# Configurar directorios
%% Cell type:code id: tags:
``` python
DATADIR = 'processed_data/'
```
%% Cell type:markdown id: tags:
# Cargar bibliotecas
%% Cell type:code id: tags:
``` python
import numpy as np
from tqdm.notebook import tqdm
import os, gc
```
%% Cell type:markdown id: tags:
# Variables de configuración
%% Cell type:code id: tags:
``` python
vars = ['u10','v10','t2m','lai_hv','lai_lv','tp']
target_vars = ['fdimrk']
months = [1,2,12]
val, test = [2015,2018], [2019,2022] #years to use for validation and testing, do not use these years to compute normalization constants
train = [x for x in np.arange(2002,2015) if x not in val+test]
```
%% Cell type:markdown id: tags:
# Constantes de normalización para variables de entrada:
%% Cell type:code id: tags:
``` python
for var in vars:
tmp = [] #save values from all time steps and locations into this list and then compute mean and std
files = sorted(os.listdir(DATADIR+var)) #SHOULD NOT CONTAIN ANY OTHER FILES THAN THOSE CREATED IN 1_prepareData.ipynb
for f in tqdm(files):
y,m,d=f.split('_')
if int(y) in train and int(m) in months:
t_array = np.load(DATADIR+var+'/'+f)
t_array[np.isnan(t_array)] = 0
tmp += list(t_array.flatten())
mean, std = np.mean(tmp), np.std(tmp)
print(f'Mean {mean}, std {std}')
np.save(f'norm_consts/input_{var}.npy',np.array([mean, std]))
del tmp
gc.collect()
```
%% Cell type:markdown id: tags:
# Constantes de normalización para variables objetivo:
%% Cell type:code id: tags:
``` python
for var in target_vars:
tmp = [] #save values from all time steps and locations into this list and then compute mean and std
files = sorted(os.listdir(DATADIR+var)) #SHOULD NOT CONTAIN ANY OTHER FILES THAN THOSE CREATED IN 1_prepareData.ipynb
for f in tqdm(files):
y,m,d=f.split('_')
if int(y) in train and int(m) in months:
t_array = np.load(DATADIR+var+'/'+f)
t_array[np.isnan(t_array)] = 0
tmp += list(t_array.flatten())
mean, std = np.mean(tmp), np.std(tmp)
print(f'Mean {mean}, std {std}')
np.save(f'norm_consts/target_{var}.npy',np.array([mean, std]))
del tmp
gc.collect()
```
README.md
View file @ aa23b2c3
# Plantilla Jupyter Book # BELLA Ideathon: Desafío de Innovación de Copérnico
Plantilla para la creación de documentación usando JupyterBook ## Descripción del problema
Los incendios forestales se han duplicado en todo el mundo en los últimos 20 años, y los incendios devoran el equivalente a 16 campos de fútbol por minuto, según un
estudio conjunto de la Universidad de Maryland en Estados Unidos, Global Forest Watch (GFW) y el World Resources Institute.
Ejemplo en [https://class.redclara.net/cyted/course2](https://class.redclara.net/cyted/course2) En Argentina, nos enfrentamos a uno de los problemas ambientales más devastadores. Según datos recabados por el Cuerpo Nacional de Bomberos
\ No newline at end of file Servicio de Gestión e INTA, los incendios forestales han arrasado hasta el momento más de 700.000 hectáreas en distintas provincias del país
en 2022.
La gente pierde sus hogares, tierras, trabajos y oxígeno. Incluso si el fuego ha terminado, los incendios forestales dejan enfermedades y epidemias en la vida silvestre, afectando la fotosíntesis, lo que es muy perjudicial para
nuestro planeta.
## Solución
Desarrollar una tecnología accesible a los ciudadanos, que utiliza inteligencia artificial (IA) para interpretar y procesar datos complejos y presentar la información que es comprensible para los humanos.
Este proyecto predice las posibilidades de que se inicie un incendio. Utiliza los siguientes parámetros: 0m u component of wind, 10m v component fo wind, 2m temperature, leaf area index high vegetation, leaf area index low vegetation y total precipitation de Datos por hora ERA5. Los datos de los índices de peligro de incendios se obtuvieron del Servicio de Gestión de Emergencias de Copernicus.
## Miembros del equipo
- David Akim
- Tomas Emanuel Schattmann
- Aldana Tedesco
## Requisitos de instalación
En la terminal ejecuta los siguientes comandos:
- `pip install cdsapi`
- `pip install numpy`
- `pip install netCDF4`
- `pip install torch`
- `pip install skorch`
- `pip install pandas`
## Orden para ejecutar cuadernos
- 0_download_data.ipynb
- 1_prepare_data.ipynb
- 2_norm_consts.ipynb
\ No newline at end of file
assets/fdimrk.jpg 0 → 100644
View file @ aa23b2c3
assets/fdimrk.jpg

59.1 KiB

assets/lai_hv.jpg 0 → 100644
View file @ aa23b2c3
assets/lai_hv.jpg

65.7 KiB

assets/t2m.jpg 0 → 100644
View file @ aa23b2c3
assets/t2m.jpg

50.2 KiB

notebooks.ipynb deleted 100644 → 0
View file @ 0298a327
%% Cell type:markdown id: tags:
# Content with notebooks
You can also create content with Jupyter Notebooks. This means that you can include
code blocks and their outputs in your book.
## Markdown + notebooks
As it is markdown, you can embed images, HTML, etc into your posts!
![](https://myst-parser.readthedocs.io/en/latest/_static/logo-wide.svg)
You can also $add_{math}$ and
$$
math^{blocks}
$$
or
$$
\begin{aligned}
\mbox{mean} la_{tex} \\ \\
math blocks
\end{aligned}
$$
But make sure you \$Escape \$your \$dollar signs \$you want to keep!
## MyST markdown
MyST markdown works in Jupyter Notebooks as well. For more information about MyST markdown, check
out [the MyST guide in Jupyter Book](https://jupyterbook.org/content/myst.html),
or see [the MyST markdown documentation](https://myst-parser.readthedocs.io/en/latest/).
## Code blocks and outputs
Jupyter Book will also embed your code blocks and output in your book.
For example, here's some sample Matplotlib code:
%% Cell type:code id: tags:
``` python
from matplotlib import rcParams, cycler
import matplotlib.pyplot as plt
import numpy as np
plt.ion()
```
%% Cell type:code id: tags:
``` python
# Fixing random state for reproducibility
np.random.seed(19680801)
N = 10
data = [np.logspace(0, 1, 100) + np.random.randn(100) + ii for ii in range(N)]
data = np.array(data).T
cmap = plt.cm.coolwarm
rcParams['axes.prop_cycle'] = cycler(color=cmap(np.linspace(0, 1, N)))
from matplotlib.lines import Line2D
custom_lines = [Line2D([0], [0], color=cmap(0.), lw=4),
Line2D([0], [0], color=cmap(.5), lw=4),
Line2D([0], [0], color=cmap(1.), lw=4)]
fig, ax = plt.subplots(figsize=(10, 5))
lines = ax.plot(data)
ax.legend(custom_lines, ['Cold', 'Medium', 'Hot']);
```
%% Cell type:markdown id: tags:
There is a lot more that you can do with outputs (such as including interactive outputs)
with your book. For more information about this, see [the Jupyter Book documentation](https://jupyterbook.org)
references.bib
View file @ aa23b2c3
--- ---
--- ---
@inproceedings{holdgraf_evidence_2014, @misc{prapas2021deep,
address = {Brisbane, Australia, Australia}, title={Deep Learning Methods for Daily Wildfire Danger Forecasting},
title = {Evidence for {Predictive} {Coding} in {Human} {Auditory} {Cortex}}, author={Ioannis Prapas and Spyros Kondylatos and Ioannis Papoutsis and Gustau Camps-Valls and Michele Ronco and Miguel-Ángel Fernández-Torres and Maria Piles Guillem and Nuno Carvalhais},
booktitle = {International {Conference} on {Cognitive} {Neuroscience}}, year={2021},
publisher = {Frontiers in Neuroscience}, eprint={2111.02736},
author = {Holdgraf, Christopher Ramsay and de Heer, Wendy and Pasley, Brian N. and Knight, Robert T.}, archivePrefix={arXiv},
year = {2014} primaryClass={cs.LG}
} }
@article{holdgraf_rapid_2016, @article{holdgraf_rapid_2016,
title = {Rapid tuning shifts in human auditory cortex enhance speech intelligibility}, title = {Causal deep learning models for studying the Earth system},
volume = {7}, volume = {16},
issn = {2041-1723}, issn = {2041-1723},
url = {http://www.nature.com/doifinder/10.1038/ncomms13654}, url = {https://gmd.copernicus.org/articles/16/2149/2023/},
doi = {10.1038/ncomms13654}, doi = {10.5194/gmd-16-2149-2023}
number = {May}, journal = {Geoscientific Model Development},
journal = {Nature Communications}, author = {Tesch, T. and Kollet, S. and Garcke, J.},
author = {Holdgraf, Christopher Ramsay and de Heer, Wendy and Pasley, Brian N. and Rieger, Jochem W. and Crone, Nathan and Lin, Jack J. and Knight, Robert T. and Theunissen, Frédéric E.}, year = {2023},
year = {2016}, pages = {2149--2166},
pages = {13654},
file = {Holdgraf et al. - 2016 - Rapid tuning shifts in human auditory cortex enhance speech intelligibility.pdf:C\:\\Users\\chold\\Zotero\\storage\\MDQP3JWE\\Holdgraf et al. - 2016 - Rapid tuning shifts in human auditory cortex enhance speech intelligibility.pdf:application/pdf} file = {Holdgraf et al. - 2016 - Rapid tuning shifts in human auditory cortex enhance speech intelligibility.pdf:C\:\\Users\\chold\\Zotero\\storage\\MDQP3JWE\\Holdgraf et al. - 2016 - Rapid tuning shifts in human auditory cortex enhance speech intelligibility.pdf:application/pdf}
} }
@inproceedings{holdgraf_portable_2017,
title = {Portable learning environments for hands-on computational instruction using container-and cloud-based technology to teach data science},
volume = {Part F1287},
isbn = {978-1-4503-5272-7},
doi = {10.1145/3093338.3093370},
abstract = {© 2017 ACM. There is an increasing interest in learning outside of the traditional classroom setting. This is especially true for topics covering computational tools and data science, as both are challenging to incorporate in the standard curriculum. These atypical learning environments offer new opportunities for teaching, particularly when it comes to combining conceptual knowledge with hands-on experience/expertise with methods and skills. Advances in cloud computing and containerized environments provide an attractive opportunity to improve the effciency and ease with which students can learn. This manuscript details recent advances towards using commonly-Available cloud computing services and advanced cyberinfrastructure support for improving the learning experience in bootcamp-style events. We cover the benets (and challenges) of using a server hosted remotely instead of relying on student laptops, discuss the technology that was used in order to make this possible, and give suggestions for how others could implement and improve upon this model for pedagogy and reproducibility.},
booktitle = {{ACM} {International} {Conference} {Proceeding} {Series}},
author = {Holdgraf, Christopher Ramsay and Culich, A. and Rokem, A. and Deniz, F. and Alegro, M. and Ushizima, D.},
year = {2017},
keywords = {Teaching, Bootcamps, Cloud computing, Data science, Docker, Pedagogy}
}
@article{holdgraf_encoding_2017,
title = {Encoding and decoding models in cognitive electrophysiology},
volume = {11},
issn = {16625137},
doi = {10.3389/fnsys.2017.00061},
abstract = {© 2017 Holdgraf, Rieger, Micheli, Martin, Knight and Theunissen. Cognitive neuroscience has seen rapid growth in the size and complexity of data recorded from the human brain as well as in the computational tools available to analyze this data. This data explosion has resulted in an increased use of multivariate, model-based methods for asking neuroscience questions, allowing scientists to investigate multiple hypotheses with a single dataset, to use complex, time-varying stimuli, and to study the human brain under more naturalistic conditions. These tools come in the form of “Encoding” models, in which stimulus features are used to model brain activity, and “Decoding” models, in which neural features are used to generated a stimulus output. Here we review the current state of encoding and decoding models in cognitive electrophysiology and provide a practical guide toward conducting experiments and analyses in this emerging field. Our examples focus on using linear models in the study of human language and audition. We show how to calculate auditory receptive fields from natural sounds as well as how to decode neural recordings to predict speech. The paper aims to be a useful tutorial to these approaches, and a practical introduction to using machine learning and applied statistics to build models of neural activity. The data analytic approaches we discuss may also be applied to other sensory modalities, motor systems, and cognitive systems, and we cover some examples in these areas. In addition, a collection of Jupyter notebooks is publicly available as a complement to the material covered in this paper, providing code examples and tutorials for predictive modeling in python. The aimis to provide a practical understanding of predictivemodeling of human brain data and to propose best-practices in conducting these analyses.},
journal = {Frontiers in Systems Neuroscience},
author = {Holdgraf, Christopher Ramsay and Rieger, J.W. and Micheli, C. and Martin, S. and Knight, R.T. and Theunissen, F.E.},
year = {2017},
keywords = {Decoding models, Encoding models, Electrocorticography (ECoG), Electrophysiology/evoked potentials, Machine learning applied to neuroscience, Natural stimuli, Predictive modeling, Tutorials}
}
@book{ruby,
title = {The Ruby Programming Language},
author = {Flanagan, David and Matsumoto, Yukihiro},
year = {2008},
publisher = {O'Reilly Media}
}