Master (#2104)

* Replace deprecated actions (#2016)

* Update build.yaml

* Update build.yaml

* Update manual-update-webinstaller.yml

* Update manual-update-webinstaller.yml

* Update manual-update-webinstaller.yml

* Update manual-update-webinstaller.yml

* Update build.yaml

* preparations for v15.0 (#2063)

* Migrated parameters, see https://github.com/jomjol/AI-on-the-edge-device/pull/2023

* remove no longer used "topic" parameter. This is a backport from https://github.com/jomjol/AI-on-the-edge-device/commit/b21e3c6c9d56555457e632bac9c5880e78c78eaf

* Fix wrong url-encoding, see https://github.com/jomjol/AI-on-the-edge-device/issues/2036 resp. https://github.com/jomjol/AI-on-the-edge-device/issues/2036

* Threashold -> Threshold

* updated changelog

---------

Co-authored-by: CaCO3 <caco@ruinelli.ch>

* DataLogActive is true by default

* updated changelog

* Bugfix #1933 (again :-))

* Update Influxdb

* re-add missing dropdownbox filling for Postprocessing Individual Parameters

* stop auto filling the release notes, it causes more confusion than it helps

* Update Changelog.md

---------

Co-authored-by: CaCO3 <caco@ruinelli.ch>
Co-authored-by: jomjol <30766535+jomjol@users.noreply.github.com>

.github/workflows/build.yaml

@@ -343,13 +343,13 @@ jobs:
 #        changelogPath: Changelog.md
 #        version: ${{ steps.get_version.outputs.version-without-v }}
             
-    # the release notes will be extracted from changelog 
-    - name: Extract release notes
-      id: extract-release-notes
-      if: startsWith(github.ref, 'refs/tags/') 
-      uses: ffurrer2/extract-release-notes@v1
-      with:
-          changelog_file: Changelog.md
+#    # the release notes will be extracted from changelog 
+#    - name: Extract release notes
+#      id: extract-release-notes
+#      if: startsWith(github.ref, 'refs/tags/') 
+#      uses: ffurrer2/extract-release-notes@v1
+#      with:
+#          changelog_file: Changelog.md
 
     # Releases should only be created on master by tagging the last commit.
     # all artifacts in firmware folder pushed to the release

Changelog.md

@@ -1,8 +1,22 @@
 ## [Unreleased]
 
-**Rolling**
+xxx
 
+### Update Procedure
+
+Update Procedure see [online documentation](https://jomjol.github.io/AI-on-the-edge-device-docs/Installation/#update-ota-over-the-air)
+
+:bangbang: Afterwards you should force-reload the Web Interface (usually Ctrl-F5 will do it).
+
+### Changes
+
+This release only migrates some parameters, see #2023 for details and a list of all parameter changes.
+The parameter migration happens automatically on the next startup. No user interaction is required.
+A backup of the config is stored on the SD-card as `config.bak`.
+
+Beside of the parameter change and the bugfix listed below, no changes are contained in this release!
 
+If you want to revert back to `v14` or earlier, you will have to revert the migration changes in `config.ini` manually!
 
 #### Added
 
@@ -14,18 +28,18 @@
 
 #### Changed
 
--   n.a.
+-   [#2023](https://github.com/jomjol/AI-on-the-edge-device/pull/2023) Migrated Parameters
+-   Removed old `Topic` parameter, it is not used anymore
 
 #### Fixed
-
--   
+-  n.a.
 
 #### Removed
 
 -   n.a.
 
 
-## [15.0.1] - 2023-02-23
+## [15.0.3] - 2023-02-28
 
 **Parameter Migration**
 
@@ -33,6 +47,8 @@
 
 Update Procedure see [online documentation](https://jomjol.github.io/AI-on-the-edge-device-docs/Installation/#update-ota-over-the-air)
 
+:bangbang: Afterwards you should force-reload the Web Interface (usually Ctrl-F5 will do it).
+
 ### Changes
 
 This release only migrates some parameters, see #2023 for details and a list of all parameter changes.
@@ -54,7 +70,13 @@ If you want to revert back to `v14` or earlier, you will have to revert the migr
 
 #### Fixed
 
--   [2036](https://github.com/jomjol/AI-on-the-edge-device/issues/2036) Fix wrong url-encoding
+-   [#2036](https://github.com/jomjol/AI-on-the-edge-device/issues/2036) Fix wrong url-encoding
+-   **NEW v15.0.2:**  [#1933](https://github.com/jomjol/AI-on-the-edge-device/issues/1933) Bugfix InfluxDB Timestamp
+-   **NEW v15.0.3:**  Re-added lost dropdownbox filling for Postprocessing Individual Parameters
+
+#### Removed
+
+-   n.a.
 
 
 ## [14.0.3] -2023-02-05
@@ -106,7 +128,6 @@ For a full list of changes see [Full list of changes](https://github.com/jomjol/
 -   [1530](https://github.com/jomjol/AI-on-the-edge-device/pull/1530) Homeassistant `Problem Sensor`
 -   [1518](https://github.com/jomjol/AI-on-the-edge-device/pull/1518) JSON Strings
 -   [1817](https://github.com/jomjol/AI-on-the-edge-device/pull/1817) DataGraph: datafiles sorted -> newest on top
--   **New 14.0.4:** Fix for InfluxDB timeshift problem [#1991](https://github.com/jomjol/AI-on-the-edge-device/issues/1991)
 
 #### Removed
 
@@ -862,7 +883,7 @@ External Illumination
 -   Initial Version
 
 
-[15.0.1]: https://github.com/jomjol/AI-on-the-edge-device/compare/v14.0.3...v15.0.1
+[15.0.3]: https://github.com/jomjol/AI-on-the-edge-device/compare/v14.0.3...v15.0.3
 [14.0.3]: https://github.com/jomjol/AI-on-the-edge-device/compare/v13.0.8...v14.0.3
 [13.0.8]: https://github.com/jomjol/AI-on-the-edge-device/compare/v12.0.1...v13.0.8
 [13.0.7]: https://github.com/jomjol/AI-on-the-edge-device/compare/v12.0.1...v13.0.7

code/components/jomjol_influxdb/interface_influxdb._c_pp_old

@@ -0,0 +1,214 @@
+#ifdef ENABLE_INFLUXDB
+#include "interface_influxdb.h"
+
+#include "esp_log.h"
+#include <time.h>
+#include "ClassLogFile.h"
+#include "esp_http_client.h"
+#include "../../include/defines.h"
+
+
+static const char *TAG = "INFLUXDB";
+
+std::string _influxDBURI;
+std::string _influxDBDatabase;
+std::string _influxDBMeasurement;
+std::string _influxDBUser;
+std::string _influxDBPassword;
+
+static esp_err_t http_event_handler(esp_http_client_event_t *evt)
+{
+    switch(evt->event_id)
+    {
+        case HTTP_EVENT_ERROR:
+            LogFile.WriteToFile(ESP_LOG_DEBUG, TAG, "HTTP Client Error encountered");
+            break;
+        case HTTP_EVENT_ON_CONNECTED:
+            LogFile.WriteToFile(ESP_LOG_DEBUG, TAG, "HTTP Client Error encountered");
+            ESP_LOGI(TAG, "HTTP Client Connected");
+            break;
+        case HTTP_EVENT_HEADERS_SENT:
+            LogFile.WriteToFile(ESP_LOG_DEBUG, TAG, "HTTP Client sent all request headers");
+            break;
+        case HTTP_EVENT_ON_HEADER:
+            LogFile.WriteToFile(ESP_LOG_DEBUG, TAG, "Header: key=" + std::string(evt->header_key) + ", value="  + std::string(evt->header_value));
+            break;
+        case HTTP_EVENT_ON_DATA:
+            LogFile.WriteToFile(ESP_LOG_DEBUG, TAG, "HTTP Client data recevied: len=" + std::to_string(evt->data_len));
+            break;
+        case HTTP_EVENT_ON_FINISH:
+            LogFile.WriteToFile(ESP_LOG_DEBUG, TAG, "HTTP Client finished");
+            break;
+         case HTTP_EVENT_DISCONNECTED:
+            LogFile.WriteToFile(ESP_LOG_DEBUG, TAG, "HTTP Client Disconnected");
+            break;
+    }
+    return ESP_OK;
+}
+
+void InfluxDBPublish(std::string _key, std::string _content, std::string _timestamp) {
+    char response_buffer[MAX_HTTP_OUTPUT_BUFFER] = {0};
+    esp_http_client_config_t http_config = {
+       .user_agent = "ESP32 Meter reader",
+       .method = HTTP_METHOD_POST,
+       .event_handler = http_event_handler,
+       .buffer_size = MAX_HTTP_OUTPUT_BUFFER,
+       .user_data = response_buffer
+    };
+
+    if (_influxDBUser.length() && _influxDBPassword.length()){
+       http_config.username = _influxDBUser.c_str();
+       http_config.password = _influxDBPassword.c_str();
+       http_config.auth_type = HTTP_AUTH_TYPE_BASIC;
+    }
+
+    LogFile.WriteToFile(ESP_LOG_DEBUG, TAG, "InfluxDBPublish - Key: " + _key + ", Content: " + _content + ", Timestamp: " + _timestamp);
+
+    char nowTimestamp[21];
+    std::string payload;
+
+    if (_timestamp.length() > 0)
+    {
+        struct tm tm;
+        strptime(_timestamp.c_str(), PREVALUE_TIME_FORMAT_OUTPUT, &tm);
+        time_t t = mktime(&tm); // Time in Localtime (looks like timezone is not used by strptime)
+
+        struct tm * ptm;
+        ptm = gmtime ( &t );
+        time_t utc = mktime(ptm);
+        utc = 2*t - utc;        // Take care of timezone (looks difficult, but is easy: t = t + (t - utc), weil t-utc = timezone)
+
+        sprintf(nowTimestamp,"%ld000000000", (long) utc);           // UTC
+
+        payload = _influxDBMeasurement + " " + _key + "=" + _content + " " + nowTimestamp;
+//        payload = _influxDBMeasurement + " " + _key + "=774 " + nowTimestamp;
+    }
+    else
+    {
+        payload = _influxDBMeasurement + " " + _key + "=" + _content;
+    }
+
+    payload.shrink_to_fit();
+
+    LogFile.WriteToFile(ESP_LOG_INFO, TAG, "sending line to influxdb:" + payload);
+
+
+    // use the default retention policy of the database
+    std::string apiURI = _influxDBURI + "/api/v2/write?bucket=" + _influxDBDatabase + "/";
+    apiURI.shrink_to_fit();
+    http_config.url = apiURI.c_str();
+
+    LogFile.WriteToFile(ESP_LOG_INFO, TAG, "API URI: " + apiURI);
+
+    esp_http_client_handle_t http_client = esp_http_client_init(&http_config);
+    LogFile.WriteToFile(ESP_LOG_DEBUG, TAG, "client is initialized");
+
+    esp_http_client_set_header(http_client, "Content-Type", "text/plain");
+    LogFile.WriteToFile(ESP_LOG_DEBUG, TAG, "header is set");
+
+    ESP_ERROR_CHECK(esp_http_client_set_post_field(http_client, payload.c_str(), payload.length()));
+    LogFile.WriteToFile(ESP_LOG_DEBUG, TAG, "post payload is set");
+
+    esp_err_t err = ESP_ERROR_CHECK_WITHOUT_ABORT(esp_http_client_perform(http_client));
+
+    if( err == ESP_OK ) {
+      LogFile.WriteToFile(ESP_LOG_DEBUG, TAG, "HTTP request was performed");
+      int status_code = esp_http_client_get_status_code(http_client);
+      LogFile.WriteToFile(ESP_LOG_DEBUG, TAG, "HTTP status code" + std::to_string(status_code));
+    } else {
+      LogFile.WriteToFile(ESP_LOG_DEBUG, TAG, "HTTP request failed");
+    }
+    esp_http_client_cleanup(http_client);
+}
+
+
+
+/*
+void InfluxDBPublish(std::string _key, std::string _content, std::string _timestamp) {
+    char response_buffer[MAX_HTTP_OUTPUT_BUFFER] = {0};
+    esp_http_client_config_t http_config = {
+       .user_agent = "ESP32 Meter reader",
+       .method = HTTP_METHOD_POST,
+       .event_handler = http_event_handler,
+       .buffer_size = MAX_HTTP_OUTPUT_BUFFER,
+       .user_data = response_buffer
+    };
+
+    if (_influxDBUser.length() && _influxDBPassword.length()){
+       http_config.username = _influxDBUser.c_str();
+       http_config.password = _influxDBPassword.c_str();
+       http_config.auth_type = HTTP_AUTH_TYPE_BASIC;
+    }
+
+    LogFile.WriteToFile(ESP_LOG_DEBUG, TAG, "InfluxDBPublish - Key: " + _key + ", Content: " + _content + ", Timestamp: " + _timestamp);
+
+    // Format:     #define PREVALUE_TIME_FORMAT_OUTPUT "%Y-%m-%dT%H:%M:%S%z"
+    struct tm tm;
+    strptime(_timestamp.c_str(), PREVALUE_TIME_FORMAT_OUTPUT, &tm);
+    time_t t = mktime(&tm);  // t is now your desired time_t
+
+    struct tm * ptm;
+    ptm = gmtime ( &t );
+    time_t utc = mktime(ptm);
+
+//    time_t now;
+//    time(&now);
+    char nowTimestamp[21];
+    // pad with zeroes to get nanoseconds
+//    sprintf(nowTimestamp,"%ld000000000", (long) now);
+//    sprintf(nowTimestamp,"%ld000000000", (long) t);           // Localtime
+    sprintf(nowTimestamp,"%ld000000000", (long) utc);           // UTC
+    
+
+//    LogFile.WriteToFile(ESP_LOG_DEBUG, TAG, "Test Time Conversion - t: " + std::to_string(t) + ", utc: " + std::to_string(utc));
+//    LogFile.WriteToFile(ESP_LOG_DEBUG, TAG, "Test Time Conversion - now: " + std::to_string(now) + ", timestamp: " + std::to_string(t)  + "(correct time not used yet)");
+
+    std::string payload = _influxDBMeasurement + " " + _key + "=" + _content + " " + nowTimestamp;
+    payload.shrink_to_fit();
+
+    LogFile.WriteToFile(ESP_LOG_INFO, TAG, "sending line to influxdb:" + payload);
+
+
+    // use the default retention policy of the database
+    std::string apiURI = _influxDBURI + "/api/v2/write?bucket=" + _influxDBDatabase + "/";
+    apiURI.shrink_to_fit();
+    http_config.url = apiURI.c_str();
+
+    LogFile.WriteToFile(ESP_LOG_INFO, TAG, "API URI: " + apiURI);
+
+    esp_http_client_handle_t http_client = esp_http_client_init(&http_config);
+    LogFile.WriteToFile(ESP_LOG_DEBUG, TAG, "client is initialized");
+
+    esp_http_client_set_header(http_client, "Content-Type", "text/plain");
+    LogFile.WriteToFile(ESP_LOG_DEBUG, TAG, "header is set");
+
+    ESP_ERROR_CHECK(esp_http_client_set_post_field(http_client, payload.c_str(), payload.length()));
+    LogFile.WriteToFile(ESP_LOG_DEBUG, TAG, "post payload is set");
+
+    esp_err_t err = ESP_ERROR_CHECK_WITHOUT_ABORT(esp_http_client_perform(http_client));
+
+    if( err == ESP_OK ) {
+      LogFile.WriteToFile(ESP_LOG_DEBUG, TAG, "HTTP request was performed");
+      int status_code = esp_http_client_get_status_code(http_client);
+      LogFile.WriteToFile(ESP_LOG_DEBUG, TAG, "HTTP status code" + std::to_string(status_code));
+    } else {
+      LogFile.WriteToFile(ESP_LOG_DEBUG, TAG, "HTTP request failed");
+    }
+    esp_http_client_cleanup(http_client);
+}
+*/
+
+
+void InfluxDBInit(std::string _uri, std::string _database, std::string _measurement, std::string _user, std::string _password){
+    _influxDBURI = _uri;
+    _influxDBDatabase = _database;
+    _influxDBMeasurement = _measurement;
+    _influxDBUser = _user;
+    _influxDBPassword = _password;
+ 
+}
+
+void InfluxDBdestroy() {
+}
+
+#endif //ENABLE_INFLUXDB

code/components/jomjol_influxdb/interface_influxdb._h_old

@@ -0,0 +1,24 @@
+#ifdef ENABLE_INFLUXDB
+
+#pragma once
+#ifndef INTERFACE_INFLUXDB_H
+#define INTERFACE_INFLUXDB_H
+
+#include <string>
+#include <map>
+#include <functional>
+
+// Interface to InfluxDB v1.x
+void InfluxDBInit(std::string _influxDBURI, std::string _database, std::string _measurement, std::string _user, std::string _password);
+void InfluxDBPublish(std::string _key, std::string _content, std::string _timestamp);
+
+// Interface to InfluxDB v2.x
+void InfluxDB_V2_Init(std::string _uri, std::string _database, std::string _measurement, std::string _org, std::string _token);
+void InfluxDB_V2_Publish(std::string _key, std::string _content, std::string _timestamp);
+
+
+
+void InfluxDBdestroy();
+
+#endif //INTERFACE_INFLUXDB_H
+#endif //ENABLE_INFLUXDB