tudattr/5g-iana-uc6-aio
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
master
5g-iana-uc6-aio/aggregator-node/logs/2024-07-15.log
Tuan-Dat Tran 931652c494 Initial commit 
Signed-off-by: Tuan-Dat Tran <tuan-dat.tran@tudattr.dev>
2024-12-31 13:36:22 +01:00

152 lines
12 KiB
Plaintext
Raw Permalink Blame History

nxw@5g-iana-manager:~$ kc logs $(kc get pods --all-namespaces | grep agg | awk '{ print $2 } ') -n $(kc get pods --all-namespaces | grep agg | awk '{ print $1 } ') -f
2024-07-15 15:48:16.807096: I tensorflow/tsl/cuda/cudart_stub.cc:28] Could not find cuda drivers on your machine, GPU will not be used.
2024-07-15 15:48:16.828642: E tensorflow/compiler/xla/stream_executor/cuda/cuda_dnn.cc:9342] Unable to register cuDNN factory: Attempting to register factory for plugin cuDNN when one has already been registered
2024-07-15 15:48:16.828672: E tensorflow/compiler/xla/stream_executor/cuda/cuda_fft.cc:609] Unable to register cuFFT factory: Attempting to register factory for plugin cuFFT when one has already been registered
2024-07-15 15:48:16.828715: E tensorflow/compiler/xla/stream_executor/cuda/cuda_blas.cc:1518] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registe
2024-07-15 15:48:16.833761: I tensorflow/tsl/cuda/cudart_stub.cc:28] Could not find cuda drivers on your machine, GPU will not be used.
2024-07-15 15:48:16.833925: I tensorflow/core/platform/cpu_feature_guard.cc:182] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: AVX2 FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
2024-07-15 15:48:17.538321: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT
/federated-example/src/server.py:169: DeprecationWarning: setDaemon() is deprecated, set the daemon attribute instead
flask_thread.setDaemon(True)
* Serving Flask app 'server'
* Debug mode: off
WARNING: This is a development server. Do not use it in a production deployment. Use a production WSGI server instead.
* Running on all addresses (0.0.0.0)
* Running on http://127.0.0.1:5000
* Running on http://10.1.68.68:5000
Press CTRL+C to quit
10.1.3.0 - - [15/Jul/2024 15:49:00] "POST /upload_kpi04 HTTP/1.1" 200 -
10.1.3.0 - - [15/Jul/2024 15:49:01] "POST /check_connection HTTP/1.1" 200 -
10.1.3.0 - - [15/Jul/2024 15:49:07] "POST /upload_kpi04 HTTP/1.1" 200 -
10.1.3.0 - - [15/Jul/2024 15:49:08] "POST /check_connection HTTP/1.1" 200 -
10.1.3.0 - - [15/Jul/2024 15:49:12] "POST /config_server HTTP/1.1" 200 -
10.1.3.0 - - [15/Jul/2024 15:49:12] "GET /select_clients HTTP/1.1" 200 -
INFO flwr 2024-07-15 15:49:12,445 | app.py:162 | Starting Flower server, config: ServerConfig(num_rounds=5, round_timeout=None)
INFO flwr 2024-07-15 15:49:12,449 | app.py:175 | Flower ECE: gRPC server running (5 rounds), SSL is disabled
INFO flwr 2024-07-15 15:49:12,449 | server.py:89 | Initializing global parameters
INFO flwr 2024-07-15 15:49:12,450 | server.py:272 | Using initial parameters provided by strategy
INFO flwr 2024-07-15 15:49:12,450 | server.py:91 | Evaluating initial parameters
Parameters loaded
Inializing Model
Model loaded
Model Compiled
(2003, 400, 3)
(2003, 1, 3)
63/63 [==============================] - 2s 23ms/step - loss: 0.0739 - quantile_metric: 0.1243 - mean_absolute_error: 0.5655
63/63 [==============================] - 2s 22ms/step
INFO flwr 2024-07-15 15:49:16,180 | server.py:94 | initial parameters (loss, other metrics): 0.07388024777173996, {'accuracy': 0.5655196309089661}
INFO flwr 2024-07-15 15:49:16,180 | server.py:104 | FL starting
10.1.3.0 - - [15/Jul/2024 15:49:35] "POST /upload_kpi04 HTTP/1.1" 200 -
10.1.3.0 - - [15/Jul/2024 15:49:36] "POST /check_connection HTTP/1.1" 200 -
DEBUG flwr 2024-07-15 15:49:41,146 | server.py:222 | fit_round 1: strategy sampled 2 clients (out of 2)
10.1.3.0 - - [15/Jul/2024 15:49:41] "POST /upload_kpi01 HTTP/1.1" 200 -
10.1.3.0 - - [15/Jul/2024 15:49:41] "POST /upload_kpi01 HTTP/1.1" 200 -
10.1.3.0 - - [15/Jul/2024 15:50:21] "POST /upload_kpi04 HTTP/1.1" 200 -
10.1.3.0 - - [15/Jul/2024 15:50:21] "POST /check_connection HTTP/1.1" 200 -
10.1.3.0 - - [15/Jul/2024 15:50:42] "GET /select_clients HTTP/1.1" 200 -
10.1.3.0 - - [15/Jul/2024 15:51:06] "POST /upload_kpi05 HTTP/1.1" 200 -
10.1.3.0 - - [15/Jul/2024 15:51:12] "POST /upload_kpi05 HTTP/1.1" 200 -
DEBUG flwr 2024-07-15 15:51:12,130 | server.py:236 | fit_round 1 received 2 results and 0 failures
WARNING flwr 2024-07-15 15:51:12,131 | fedavg.py:242 | No fit_metrics_aggregation_fn provided
2 clients connected.
WARNING: 2 clients are needed but only 3 client IDs are received. The training will wait for another list with enough eligible clients.
(2003, 400, 3)
(2003, 1, 3)
63/63 [==============================] - 1s 23ms/step - loss: 0.1734 - quantile_metric: 0.1908 - mean_absolute_error: 2.4910
63/63 [==============================] - 1s 22ms/step
INFO flwr 2024-07-15 15:51:15,075 | server.py:125 | fit progress: (1, 0.1733752340078354, {'accuracy': 2.490957498550415}, 118.89502924995031)
DEBUG flwr 2024-07-15 15:51:15,149 | server.py:173 | evaluate_round 1: strategy sampled 3 clients (out of 3)
DEBUG flwr 2024-07-15 15:51:26,920 | server.py:187 | evaluate_round 1 received 3 results and 0 failures
WARNING flwr 2024-07-15 15:51:26,920 | fedavg.py:273 | No evaluate_metrics_aggregation_fn provided
DEBUG flwr 2024-07-15 15:51:26,974 | server.py:222 | fit_round 2: strategy sampled 3 clients (out of 3)
10.1.3.0 - - [15/Jul/2024 15:51:27] "POST /upload_kpi02 HTTP/1.1" 200 -
10.1.3.0 - - [15/Jul/2024 15:51:27] "POST /upload_kpi02 HTTP/1.1" 200 -
10.1.3.0 - - [15/Jul/2024 15:51:27] "POST /upload_kpi01 HTTP/1.1" 200 -
10.1.3.0 - - [15/Jul/2024 15:51:27] "POST /upload_kpi01 HTTP/1.1" 200 -
10.1.3.0 - - [15/Jul/2024 15:51:27] "POST /upload_kpi01 HTTP/1.1" 200 -
10.1.3.0 - - [15/Jul/2024 15:52:05] "POST /upload_kpi05 HTTP/1.1" 200 -
10.1.3.0 - - [15/Jul/2024 15:52:42] "POST /upload_kpi05 HTTP/1.1" 200 -
10.1.3.0 - - [15/Jul/2024 15:52:47] "POST /upload_kpi05 HTTP/1.1" 200 -
DEBUG flwr 2024-07-15 15:52:47,347 | server.py:236 | fit_round 2 received 3 results and 0 failures
2 clients connected.
2 clients connected.
(2003, 400, 3)
(2003, 1, 3)
63/63 [==============================] - 1s 21ms/step - loss: 0.0874 - quantile_metric: 0.2492 - mean_absolute_error: 0.2591
63/63 [==============================] - 1s 21ms/step
INFO flwr 2024-07-15 15:52:50,161 | server.py:125 | fit progress: (2, 0.08735799789428711, {'accuracy': 0.2590666115283966}, 213.98151048796717)
DEBUG flwr 2024-07-15 15:52:50,221 | server.py:173 | evaluate_round 2: strategy sampled 3 clients (out of 3)
DEBUG flwr 2024-07-15 15:52:59,542 | server.py:187 | evaluate_round 2 received 3 results and 0 failures
DEBUG flwr 2024-07-15 15:52:59,589 | server.py:222 | fit_round 3: strategy sampled 3 clients (out of 3)
10.1.3.0 - - [15/Jul/2024 15:52:59] "POST /upload_kpi02 HTTP/1.1" 200 -
10.1.3.0 - - [15/Jul/2024 15:52:59] "POST /upload_kpi02 HTTP/1.1" 200 -
10.1.3.0 - - [15/Jul/2024 15:52:59] "POST /upload_kpi02 HTTP/1.1" 200 -
10.1.3.0 - - [15/Jul/2024 15:52:59] "POST /upload_kpi01 HTTP/1.1" 200 -
10.1.3.0 - - [15/Jul/2024 15:52:59] "POST /upload_kpi01 HTTP/1.1" 200 -
10.1.3.0 - - [15/Jul/2024 15:52:59] "POST /upload_kpi01 HTTP/1.1" 200 -
10.1.3.0 - - [15/Jul/2024 15:53:34] "POST /upload_kpi05 HTTP/1.1" 200 -
10.1.3.0 - - [15/Jul/2024 15:53:36] "POST /upload_kpi04 HTTP/1.1" 200 -
10.1.3.0 - - [15/Jul/2024 15:53:36] "POST /check_connection HTTP/1.1" 200 -
10.1.3.0 - - [15/Jul/2024 15:54:12] "POST /upload_kpi05 HTTP/1.1" 200 -
DEBUG flwr 2024-07-15 15:54:13,045 | server.py:236 | fit_round 3 received 2 results and 1 failures
2 clients connected.
2 clients connected.
(2003, 400, 3)
(2003, 1, 3)
63/63 [==============================] - 1s 22ms/step - loss: 0.0654 - quantile_metric: 0.1364 - mean_absolute_error: 0.9301
63/63 [==============================] - 1s 22ms/step
INFO flwr 2024-07-15 15:54:15,922 | server.py:125 | fit progress: (3, 0.06537292897701263, {'accuracy': 0.9301236867904663}, 299.7421916149906)
DEBUG flwr 2024-07-15 15:54:15,981 | server.py:173 | evaluate_round 3: strategy sampled 3 clients (out of 3)
DEBUG flwr 2024-07-15 15:54:28,262 | server.py:187 | evaluate_round 3 received 3 results and 0 failures
DEBUG flwr 2024-07-15 15:54:28,314 | server.py:222 | fit_round 4: strategy sampled 3 clients (out of 3)
10.1.3.0 - - [15/Jul/2024 15:54:28] "POST /upload_kpi01 HTTP/1.1" 200 -
10.1.3.0 - - [15/Jul/2024 15:54:28] "POST /upload_kpi02 HTTP/1.1" 200 -
10.1.3.0 - - [15/Jul/2024 15:54:28] "POST /upload_kpi02 HTTP/1.1" 200 -
10.1.3.0 - - [15/Jul/2024 15:54:28] "POST /upload_kpi01 HTTP/1.1" 200 -
10.1.3.0 - - [15/Jul/2024 15:54:28] "POST /upload_kpi01 HTTP/1.1" 200 -
10.1.3.0 - - [15/Jul/2024 15:55:03] "POST /upload_kpi05 HTTP/1.1" 200 -
10.1.3.0 - - [15/Jul/2024 15:55:40] "POST /upload_kpi05 HTTP/1.1" 200 -
10.1.3.0 - - [15/Jul/2024 15:55:53] "POST /upload_kpi05 HTTP/1.1" 200 -
DEBUG flwr 2024-07-15 15:55:53,632 | server.py:236 | fit_round 4 received 3 results and 0 failures
2 clients connected.
2 clients connected.
(2003, 400, 3)
(2003, 1, 3)
63/63 [==============================] - 1s 22ms/step - loss: 0.1268 - quantile_metric: 0.3151 - mean_absolute_error: 0.3247
63/63 [==============================] - 1s 22ms/step
INFO flwr 2024-07-15 15:55:56,563 | server.py:125 | fit progress: (4, 0.12679509818553925, {'accuracy': 0.3247184455394745}, 400.3833388419589)
DEBUG flwr 2024-07-15 15:55:56,646 | server.py:173 | evaluate_round 4: strategy sampled 3 clients (out of 3)
DEBUG flwr 2024-07-15 15:56:06,016 | server.py:187 | evaluate_round 4 received 3 results and 0 failures
DEBUG flwr 2024-07-15 15:56:06,066 | server.py:222 | fit_round 5: strategy sampled 3 clients (out of 3)
10.1.3.0 - - [15/Jul/2024 15:56:06] "POST /upload_kpi02 HTTP/1.1" 200 -
10.1.3.0 - - [15/Jul/2024 15:56:06] "POST /upload_kpi02 HTTP/1.1" 200 -
10.1.3.0 - - [15/Jul/2024 15:56:06] "POST /upload_kpi01 HTTP/1.1" 200 -
10.1.3.0 - - [15/Jul/2024 15:56:06] "POST /upload_kpi01 HTTP/1.1" 200 -
10.1.3.0 - - [15/Jul/2024 15:56:06] "POST /upload_kpi02 HTTP/1.1" 200 -
10.1.3.0 - - [15/Jul/2024 15:56:06] "POST /upload_kpi01 HTTP/1.1" 200 -
10.1.3.0 - - [15/Jul/2024 15:56:41] "POST /upload_kpi05 HTTP/1.1" 200 -
10.1.3.0 - - [15/Jul/2024 15:57:17] "POST /upload_kpi05 HTTP/1.1" 200 -
10.1.3.0 - - [15/Jul/2024 15:57:25] "POST /upload_kpi05 HTTP/1.1" 200 -
DEBUG flwr 2024-07-15 15:57:25,615 | server.py:236 | fit_round 5 received 3 results and 0 failures
2 clients connected.
2 clients connected.
(2003, 400, 3)
(2003, 1, 3)
63/63 [==============================] - 1s 22ms/step - loss: 0.0718 - quantile_metric: 0.1710 - mean_absolute_error: 0.3574
63/63 [==============================] - 1s 22ms/step
INFO flwr 2024-07-15 15:57:28,518 | server.py:125 | fit progress: (5, 0.0717623308300972, {'accuracy': 0.35737916827201843}, 492.3376815340016)
DEBUG flwr 2024-07-15 15:57:28,599 | server.py:173 | evaluate_round 5: strategy sampled 3 clients (out of 3)
DEBUG flwr 2024-07-15 15:57:37,732 | server.py:187 | evaluate_round 5 received 3 results and 0 failures
INFO flwr 2024-07-15 15:57:37,732 | server.py:153 | FL finished in 501.5518533719587
INFO flwr 2024-07-15 15:57:37,732 | app.py:225 | app_fit: losses_distributed [(1, 0.22432586054007211), (2, 0.05442244683702787), (3, 0.06365528702735901), (4, 0.05708811432123184), (5, 0.04476702958345413)]
INFO flwr 2024-07-15 15:57:37,732 | app.py:226 | app_fit: metrics_distributed_fit {}
INFO flwr 2024-07-15 15:57:37,732 | app.py:227 | app_fit: metrics_distributed {}
INFO flwr 2024-07-15 15:57:37,732 | app.py:228 | app_fit: losses_centralized [(0, 0.07388024777173996), (1, 0.1733752340078354), (2, 0.08735799789428711), (3, 0.06537292897701263), (4, 0.12679509818553925), (5, 0.0717623308300972)]
INFO flwr 2024-07-15 15:57:37,732 | app.py:229 | app_fit: metrics_centralized {'accuracy': [(0, 0.5655196309089661), (1, 2.490957498550415), (2, 0.2590666115283966), (3, 0.9301236867904663), (4, 0.3247184455394745), (5, 0.35737916827201843)]}
2 clients connected.
10.1.3.0 - - [15/Jul/2024 15:58:02] "POST /upload_kpi04 HTTP/1.1" 200 -
10.1.3.0 - - [15/Jul/2024 15:58:02] "POST /check_connection HTTP/1.1" 200 -
10.1.3.0 - - [15/Jul/2024 15:58:13] "POST /upload_kpi04 HTTP/1.1" 200 -
10.1.3.0 - - [15/Jul/2024 15:58:14] "POST /check_connection HTTP/1.1" 200 -
Reference in New Issue View Git Blame Copy Permalink