Agregar archivos iniciales

919d290a · Alexander Martínez Méndez · 3ad01915 · 3ad01915 · 3ad01915 · 3ad01915
Commit 919d290a authored 2 years ago by Alexander Martínez Méndez
--- a/.ipynb_checkpoints/amplitude-checkpoint.png
+++ b/.ipynb_checkpoints/amplitude-checkpoint.png
--- a/.ipynb_checkpoints/calibración-checkpoint.ipynb
+++ b/.ipynb_checkpoints/calibración-checkpoint.ipynb
--- a/Lightning/Lighting_2021_08_29_13_23.dat
+++ b/Lightning/Lighting_2021_08_29_13_23.dat
--- a/README.md
+++ b/README.md
@@ -6,5 +6,5 @@ Repositorio de análisis de datos del proyecto Racimo Tormenta.
 - Calibración del instrumento.
    - Datos: [https://dataverse.redclara.net/dataverse/storm](https://dataverse.redclara.net/dataverse/storm)
-    - Código: [https://gitmilab.redclara.net/mxrtinez1/analisis-de-datos](https://gitmilab.redclara.net/mxrtinez1/analisis-de-datos)
+    - Código: [https://gitmilab.redclara.net/halleyUIS/limonet/analisis-de-datos](https://gitmilab.redclara.net/halleyUIS/limonet/analisis-de-datos)
    - Artículo: [https://www.overleaf.com/read/yhvybynwktjv](https://www.overleaf.com/read/yhvybynwktjv)
\ No newline at end of file
--- a/Upload_dataverse_automatic.ipynb
+++ b/Upload_dataverse_automatic.ipynb
-{
- "cells": [
-  {
-   "cell_type": "markdown",
-   "id": "ff1fd295",
-   "metadata": {},
-   "source": [
-    "# Script for automatically uploading LiMoNet data to a Dataverse repository"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "id": "750b30d8",
-   "metadata": {},
-   "source": [
-    "This script uploads data collected by the LiMoNet (Lightning Monitoring Network) to a Dataverse repository. The code firstly load the python packages needed for the connection to dataverse, load metadata from a **.json** file and search data files in a folder. We define some functions: **create_dataset**, **modify_metadata**, **load_metadata** and **upload_data**. For more information, some references are listed along the script.\n",
-    "\n",
-    "Author: J. Peña-Rodríguez\n",
-    "2021"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "id": "5402e405",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "from dataverse import Connection\n",
-    "import numpy as np\n",
-    "import sys\n",
-    "import os\n",
-    "import dataverse\n",
-    "from lxml import etree\n",
-    "import json\n",
-    "import glob\n",
-    "import datetime\n",
-    "\n",
-    "%matplotlib inline\n",
-    "sys.getdefaultencoding()"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "id": "de9b2c88",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "def progressbar(it, prefix=\"\", size=60, file=sys.stdout):\n",
-    "    \n",
-    "    # Progress bar animation\n",
-    "    count = len(it)\n",
-    "    def show(j):\n",
-    "        x = int(size*j/count)\n",
-    "        file.write(\"%s[%s%s] %i/%i\\r\" % (prefix, \"#\"*x, \".\"*(size-x), j, count))\n",
-    "        file.flush()        \n",
-    "    show(0)\n",
-    "    for i, item in enumerate(it):\n",
-    "        yield item\n",
-    "        show(i+1)\n",
-    "    file.write(\"\\n\")\n",
-    "    file.flush()"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "id": "9e50bd8b",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "def create_dataset(dataset_name):\n",
-    "    \n",
-    "    # Metadata\n",
-    "    # https://docs.python.org/3/library/xml.etree.elementtree.html\n",
-    "    # https://www.tutorialspoint.com/python3/python_xml_processing.htm\n",
-    "    # https://lxml.de/2.0/parsing.html\n",
-    "    # https://github.com/IQSS/dataverse-client-python\n",
-    "\n",
-    "    description = 'This repository contains lightning data files recorded by LiMoNet at Bucaramanga, Colombia.'\n",
-    "    creator = 'Peña, Jesús'\n",
-    "    \n",
-    "    # Create dataset\n",
-    "\n",
-    "    dataset_id = dataverse.Dataverse.create_dataset(dataverse_id, dataset_name, description, creator)\n",
-    "    return dataset_id"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "id": "5c527720",
-   "metadata": {},
-   "source": [
-    "Los campos del archivo .json tienen palabras claves que se pueden encontrar aquí:\n",
-    "\n",
-    "https://guides.dataverse.org/en/4.18.1/_downloads/dataset-create-new-all-default-fields.json\n"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "id": "83d72718",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "def modify_metadata(dataset_name, date):\n",
-    "    \n",
-    "    # Modify the metadata file metadata_limonet.json\n",
-    "    # Modified metadata fields: title and dates\n",
-    "    # All the fields can be midified depending on your necessity\n",
-    "\n",
-    "    date = date\n",
-    "    title = dataset_name\n",
-    "\n",
-    "    with open(\"metadata_limonet.json\", 'r') as f:\n",
-    "        json_data = json.load(f)\n",
-    "        json_data['metadataBlocks']['citation']['fields'][3]['value'][0]['dsDescriptionDate']['value']= date\n",
-    "        json_data['metadataBlocks']['citation']['fields'][8]['value']= date\n",
-    "        json_data['metadataBlocks']['citation']['fields'][0]['value']= title\n",
-    "\n",
-    "    with open('metadata_limonet.json', 'w') as f:\n",
-    "        json.dump(json_data, f, indent = 2)"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "id": "98f30841",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "def load_metadata(dataset_id):\n",
-    "\n",
-    "    # Update the repository metadata\n",
-    "    \n",
-    "    metadata_file = open(\"metadata_limonet.json\",)\n",
-    "\n",
-    "    # Returns JSON object as a dictionary\n",
-    "    metadata = json.load(metadata_file)\n",
-    "    # Get metadata\n",
-    "    dataset_id.update_metadata(metadata)"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "id": "3e2ffe24",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "def upload_data(dataset_id, day):\n",
-    "    \n",
-    "    # Upload data\n",
-    "    # ej: dataset_id.upload_filepath('Lightning/Lighting_2021_04_01_18_52.dat')\n",
-    "\n",
-    "    files = sorted(glob.glob(\"Lightning/Lighting_\" + day + \"*.dat\")) # Sort datafiles\n",
-    "    M = len(files)\n",
-    "\n",
-    "    for i in progressbar(range(M), \"Uploading: \", 50):\n",
-    "\n",
-    "        dataset_id.upload_filepath(files[i])\n",
-    "\n",
-    "    print ('\\nData uploaded\\n')"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "id": "16a19fd8",
-   "metadata": {},
-   "source": [
-    "## Upload data"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "id": "151fd3dd-8ef1-4f5c-89a3-ec116949cf36",
-   "metadata": {},
-   "source": [
-    "Ir a https://dataverse.redclara.net/dataverseuser.xhtml?selectTab=apiTokenTab y copiar el **Token** de usuario"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "id": "283b362e",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "%env API_TOKEN=TOKENdeUSUARIO"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "id": "8d8bc25f",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "API_TOKEN = os.environ['API_TOKEN']\n",
-    "host = 'dataverse.redclara.net'                  # All clients >4.0 are supported\n",
-    "# Conexión a repositorio\n",
-    "connection = Connection(host, API_TOKEN)\n",
-    "# Selección de dataverse a user\n",
-    "dataverse_id = connection.get_dataverse('ticec')  # Dataverse id"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "id": "a437ef59",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "year = '2021'\n",
-    "month = '08'\n",
-    "\n",
-    "now = datetime.datetime.now()\n",
-    "upload_date = (\"%s-%s-%s\" % (now.year, now.month, now.day))\n",
-    "\n",
-    "for i in range(1,30):\n",
-    "    day = str(i).zfill(2)\n",
-    "    file_date = (\"%s_%s_%s\" % (year, month, day))\n",
-    "    file_name = (\"Lightning/Lighting_%s*.dat\" % file_date)\n",
-    "    \n",
-    "\n",
-    "    files = glob.glob(file_name)\n",
-    "    M = len(files)\n",
-    "\n",
-    "    if files != []:  # Check files existence\n",
-    "        \n",
-    "        dataset_name = (\"%s-LM_%s\" % (os.environ['USER'],file_date))\n",
-    "        print (\"Files: %s Dataset: %s Date: %s\" % (file_name, dataset_name, upload_date))\n",
-    "        \n",
-    "        dataset_id = create_dataset(dataset_name)\n",
-    "        modify_metadata(dataset_name, upload_date)\n",
-    "        load_metadata(dataset_id)\n",
-    "        upload_data(dataset_id, file_date)"
-   ]
-  }
- ],
- "metadata": {
-  "kernelspec": {
-   "display_name": "Python 3 (ipykernel)",
-   "language": "python",
-   "name": "python3"
-  },
-  "language_info": {
-   "codemirror_mode": {
-    "name": "ipython",
-    "version": 3
-   },
-   "file_extension": ".py",
-   "mimetype": "text/x-python",
-   "name": "python",
-   "nbconvert_exporter": "python",
-   "pygments_lexer": "ipython3",
-   "version": "3.8.10"
-  }
- },
- "nbformat": 4,
- "nbformat_minor": 5
-}
-%% Cell type:markdown id:ff1fd295 tags:
-# Script for automatically uploading LiMoNet data to a Dataverse repository
-%% Cell type:markdown id:750b30d8 tags:
-This script uploads data collected by the LiMoNet (Lightning Monitoring Network) to a Dataverse repository. The code firstly load the python packages needed for the connection to dataverse, load metadata from a **.json** file and search data files in a folder. We define some functions: **create_dataset**, **modify_metadata**, **load_metadata** and **upload_data**. For more information, some references are listed along the script.
-Author: J. Peña-Rodríguez
-2021
-%% Cell type:code id:5402e405 tags:
-``` python
-from dataverse import Connection
-import numpy as np
-import sys
-import os
-import dataverse
-from lxml import etree
-import json
-import glob
-import datetime
-%matplotlib inline
-sys.getdefaultencoding()
-```
-%% Cell type:code id:de9b2c88 tags:
-``` python
-def progressbar(it, prefix="", size=60, file=sys.stdout):
-    # Progress bar animation
-    count = len(it)
-    def show(j):
-        x = int(size*j/count)
-        file.write("%s[%s%s] %i/%i\r" % (prefix, "#"*x, "."*(size-x), j, count))
-        file.flush()
-    show(0)
-    for i, item in enumerate(it):
-        yield item
-        show(i+1)
-    file.write("\n")
-    file.flush()
-```
-%% Cell type:code id:9e50bd8b tags:
-``` python
-def create_dataset(dataset_name):
-    # Metadata
-    # https://docs.python.org/3/library/xml.etree.elementtree.html
-    # https://www.tutorialspoint.com/python3/python_xml_processing.htm
-    # https://lxml.de/2.0/parsing.html
-    # https://github.com/IQSS/dataverse-client-python
-    description = 'This repository contains lightning data files recorded by LiMoNet at Bucaramanga, Colombia.'
-    creator = 'Peña, Jesús'
-    # Create dataset
-    dataset_id = dataverse.Dataverse.create_dataset(dataverse_id, dataset_name, description, creator)
-    return dataset_id
-```
-%% Cell type:markdown id:5c527720 tags:
-Los campos del archivo .json tienen palabras claves que se pueden encontrar aquí:
-https://guides.dataverse.org/en/4.18.1/_downloads/dataset-create-new-all-default-fields.json
-%% Cell type:code id:83d72718 tags:
-``` python
-def modify_metadata(dataset_name, date):
-    # Modify the metadata file metadata_limonet.json
-    # Modified metadata fields: title and dates
-    # All the fields can be midified depending on your necessity
-    date = date
-    title = dataset_name
-    with open("metadata_limonet.json", 'r') as f:
-        json_data = json.load(f)
-        json_data['metadataBlocks']['citation']['fields'][3]['value'][0]['dsDescriptionDate']['value']= date
-        json_data['metadataBlocks']['citation']['fields'][8]['value']= date
-        json_data['metadataBlocks']['citation']['fields'][0]['value']= title
-    with open('metadata_limonet.json', 'w') as f:
-        json.dump(json_data, f, indent = 2)
-```
-%% Cell type:code id:98f30841 tags:
-``` python
-def load_metadata(dataset_id):
-    # Update the repository metadata
-    metadata_file = open("metadata_limonet.json",)
-    # Returns JSON object as a dictionary
-    metadata = json.load(metadata_file)
-    # Get metadata
-    dataset_id.update_metadata(metadata)
-```
-%% Cell type:code id:3e2ffe24 tags:
-``` python
-def upload_data(dataset_id, day):
-    # Upload data
-    # ej: dataset_id.upload_filepath('Lightning/Lighting_2021_04_01_18_52.dat')
-    files = sorted(glob.glob("Lightning/Lighting_" + day + "*.dat")) # Sort datafiles
-    M = len(files)
-    for i in progressbar(range(M), "Uploading: ", 50):
-        dataset_id.upload_filepath(files[i])
-    print ('\nData uploaded\n')
-```
-%% Cell type:markdown id:16a19fd8 tags:
-## Upload data
-%% Cell type:markdown id:151fd3dd-8ef1-4f5c-89a3-ec116949cf36 tags:
-Ir a https://dataverse.redclara.net/dataverseuser.xhtml?selectTab=apiTokenTab y copiar el **Token** de usuario
-%% Cell type:code id:283b362e tags:
-``` python
-%env API_TOKEN=TOKENdeUSUARIO
-```
-%% Cell type:code id:8d8bc25f tags:
-``` python
-API_TOKEN = os.environ['API_TOKEN']
-host = 'dataverse.redclara.net'                  # All clients >4.0 are supported
-# Conexión a repositorio
-connection = Connection(host, API_TOKEN)
-# Selección de dataverse a user
-dataverse_id = connection.get_dataverse('ticec')  # Dataverse id
-```
-%% Cell type:code id:a437ef59 tags:
-``` python
-year = '2021'
-month = '08'
-now = datetime.datetime.now()
-upload_date = ("%s-%s-%s" % (now.year, now.month, now.day))
-for i in range(1,30):
-    day = str(i).zfill(2)
-    file_date = ("%s_%s_%s" % (year, month, day))
-    file_name = ("Lightning/Lighting_%s*.dat" % file_date)
-    files = glob.glob(file_name)
-    M = len(files)
-    if files != []:  # Check files existence
-        dataset_name = ("%s-LM_%s" % (os.environ['USER'],file_date))
-        print ("Files: %s Dataset: %s Date: %s" % (file_name, dataset_name, upload_date))
-        dataset_id = create_dataset(dataset_name)
-        modify_metadata(dataset_name, upload_date)
-        load_metadata(dataset_id)
-        upload_data(dataset_id, file_date)
-```
--- a/amplitude.png
+++ b/amplitude.png