From 5a8c7f0248e9739bc9c6df71501c2079f091f252 Mon Sep 17 00:00:00 2001 From: Tuan-Dat Tran Date: Sat, 28 Feb 2026 11:30:58 +0100 Subject: [PATCH] feat(proxmox): add hosts config Signed-off-by: Tuan-Dat Tran --- issues/001_fix_vault_security_issue.md | 74 ++ issues/002_replace_dict2items_filter.md | 57 ++ issues/003_add_granular_tags.md | 105 +++ issues/004_add_error_handling.md | 125 +++ issues/005_add_performance_optimizations.md | 119 +++ .../docs/plans/2026-02-24-edge-vps-role.md | 715 ++++++++++++++++++ roles/proxmox/tasks/04_configure_hosts.yaml | 15 + roles/proxmox/tasks/05_setup_node.yaml | 3 + roles/proxmox/tasks/42_download_isos.yaml | 3 +- vars/group_vars/proxmox/secrets_vm.yaml | 130 ++-- 10 files changed, 1280 insertions(+), 66 deletions(-) create mode 100644 issues/001_fix_vault_security_issue.md create mode 100644 issues/002_replace_dict2items_filter.md create mode 100644 issues/003_add_granular_tags.md create mode 100644 issues/004_add_error_handling.md create mode 100644 issues/005_add_performance_optimizations.md create mode 100644 roles/edge_vps/docs/plans/2026-02-24-edge-vps-role.md create mode 100644 roles/proxmox/tasks/04_configure_hosts.yaml diff --git a/issues/001_fix_vault_security_issue.md b/issues/001_fix_vault_security_issue.md new file mode 100644 index 0000000..8d9cd49 --- /dev/null +++ b/issues/001_fix_vault_security_issue.md @@ -0,0 +1,74 @@ +# Issue: Fix Vault Security Risk in Proxmox Role + +**Status**: Open +**Priority**: High +**Component**: proxmox/15_create_secret.yaml +**Assignee**: Junior Dev + +## Description +The current vault handling in `roles/proxmox/tasks/15_create_secret.yaml` uses insecure shell commands to decrypt/encrypt vault files, creating temporary plaintext files that pose a security risk. + +## Current Problematic Code +```yaml +- name: Decrypt vm vault file + ansible.builtin.shell: cd ../; ansible-vault decrypt "./playbooks/{{ proxmox_vault_file }}" + no_log: true + +- name: Encrypt vm vault file + ansible.builtin.shell: cd ../; ansible-vault encrypt "./playbooks/{{ proxmox_vault_file }}" + no_log: true +``` + +## Required Changes + +### Step 1: Replace shell commands with Ansible vault module +Replace the shell-based decryption/encryption with `ansible.builtin.ansible_vault` module. + +### Step 2: Remove temporary plaintext file operations +Eliminate the need for temporary plaintext files by using in-memory operations. + +### Step 3: Add proper error handling +Include error handling for vault operations (missing files, decryption failures). + +## Implementation Steps + +1. **Read the current vault file securely**: + ```yaml + - name: Load vault content securely + ansible.builtin.include_vars: + file: "{{ proxmox_vault_file }}" + name: vault_data + no_log: true + ``` + +2. **Use ansible_vault module for operations**: + ```yaml + - name: Update vault data securely + ansible.builtin.set_fact: + new_vault_data: "{{ vault_data | combine({vm_name_secret: cipassword}) }}" + when: not variable_exists + no_log: true + ``` + +3. **Write encrypted vault directly**: + ```yaml + - name: Write encrypted vault + ansible.builtin.copy: + content: "{{ new_vault_data | ansible.builtin.ansible_vault.encrypt('vault_password') }}" + dest: "{{ proxmox_vault_file }}" + mode: "0600" + when: not variable_exists + no_log: true + ``` + +## Testing Requirements +- Test with existing vault files +- Verify no plaintext files are created during operation +- Confirm vault can be decrypted properly after updates + +## Acceptance Criteria +- [ ] No shell commands used for vault operations +- [ ] No temporary plaintext files created +- [ ] All vault operations use Ansible built-in modules +- [ ] Existing functionality preserved +- [ ] Proper error handling implemented diff --git a/issues/002_replace_dict2items_filter.md b/issues/002_replace_dict2items_filter.md new file mode 100644 index 0000000..87241f0 --- /dev/null +++ b/issues/002_replace_dict2items_filter.md @@ -0,0 +1,57 @@ +# Issue: Replace Deprecated dict2items Filter + +**Status**: Open +**Priority**: Medium +**Component**: proxmox/40_prepare_vm_creation.yaml +**Assignee**: Junior Dev + +## Description +The task `roles/proxmox/tasks/40_prepare_vm_creation.yaml` uses the deprecated `dict2items` filter which may be removed in future Ansible versions. + +## Current Problematic Code +```yaml +- name: Download Cloud Init Isos + ansible.builtin.include_tasks: 42_download_isos.yaml + loop: "{{ proxmox_cloud_init_images | dict2items | map(attribute='value') }}" + loop_control: + loop_var: distro +``` + +## Required Changes + +### Step 1: Replace dict2items with modern Ansible practices +Use `dict` filter or direct dictionary iteration instead of deprecated filter. + +### Step 2: Update variable references +Ensure the loop variable structure matches the new iteration method. + +## Implementation Steps + +### Option A: Use dict filter (recommended) +```yaml +- name: Download Cloud Init Isos + ansible.builtin.include_tasks: 42_download_isos.yaml + loop: "{{ proxmox_cloud_init_images | dict | map(attribute='value') }}" + loop_control: + loop_var: distro +``` + +### Option B: Direct dictionary iteration +```yaml +- name: Download Cloud Init Isos + ansible.builtin.include_tasks: 42_download_isos.yaml + loop: "{{ proxmox_cloud_init_images.values() | list }}" + loop_control: + loop_var: distro +``` + +## Testing Requirements +- Verify all cloud init images are still downloaded correctly +- Test with different dictionary structures +- Confirm no regression in functionality + +## Acceptance Criteria +- [ ] Deprecated `dict2items` filter removed +- [ ] All cloud init images download successfully +- [ ] No changes to existing functionality +- [ ] Code works with current and future Ansible versions diff --git a/issues/003_add_granular_tags.md b/issues/003_add_granular_tags.md new file mode 100644 index 0000000..c82f75b --- /dev/null +++ b/issues/003_add_granular_tags.md @@ -0,0 +1,105 @@ +# Issue: Add Granular Tags for Better Control + +**Status**: Open +**Priority**: Medium +**Component**: proxmox/tasks/main.yaml +**Assignee**: Junior Dev + +## Description +The Proxmox role lacks granular tags, making it difficult to run specific parts of the role independently. Currently only has high-level `proxmox` tag. + +## Current Limitation +```yaml +# Current tag structure +roles: + - role: proxmox + tags: + - proxmox +``` + +## Required Changes + +### Step 1: Add tags to main task includes +Add specific tags to each major task group in `roles/proxmox/tasks/main.yaml`. + +### Step 2: Update playbook to use new tags +Ensure playbooks can leverage the new tag structure. + +## Implementation Steps + +### Update roles/proxmox/tasks/main.yaml +```yaml +- name: Prepare Machines + ansible.builtin.include_tasks: 00_setup_machines.yaml + tags: + - proxmox:setup + - proxmox + +- name: Create VM vault + ansible.builtin.include_tasks: 10_create_secrets.yaml + when: is_localhost + tags: + - proxmox:vault + - proxmox + +- name: Prime node for VM + ansible.builtin.include_tasks: 40_prepare_vm_creation.yaml + when: is_proxmox_node + tags: + - proxmox:prepare + - proxmox + +- name: Create VMs + ansible.builtin.include_tasks: 50_create_vms.yaml + when: is_localhost + tags: + - proxmox:vms + - proxmox + +- name: Create LXC containers + ansible.builtin.include_tasks: 60_create_containers.yaml + when: is_localhost + tags: + - proxmox:containers + - proxmox +``` + +### Update individual task files +Add appropriate tags to tasks within each included file: + +```yaml +# Example for 04_configure_hosts.yaml +- name: Configure /etc/hosts with Proxmox cluster nodes + ansible.builtin.blockinfile: + # ... existing content ... + tags: + - proxmox:setup + - proxmox:network +``` + +## Usage Examples + +After implementation, users can run specific parts: + +```bash +# Run only setup tasks +ansible-playbook playbooks/proxmox.yaml --tags "proxmox:setup" + +# Run only VM creation +ansible-playbook playbooks/proxmox.yaml --tags "proxmox:vms" + +# Run setup and preparation +ansible-playbook playbooks/proxmox.yaml --tags "proxmox:setup,proxmox:prepare" +``` + +## Testing Requirements +- Verify each tag group runs the correct subset of tasks +- Test tag combinations work properly +- Ensure backward compatibility with existing `proxmox` tag + +## Acceptance Criteria +- [ ] Granular tags added to all major task groups +- [ ] Each functional area has its own tag +- [ ] Original `proxmox` tag still works for backward compatibility +- [ ] Documentation updated with tag usage examples +- [ ] All tags tested and working diff --git a/issues/004_add_error_handling.md b/issues/004_add_error_handling.md new file mode 100644 index 0000000..6d1cb47 --- /dev/null +++ b/issues/004_add_error_handling.md @@ -0,0 +1,125 @@ +# Issue: Add Comprehensive Error Handling + +**Status**: Open +**Priority**: High +**Component**: proxmox/tasks +**Assignee**: Junior Dev + +## Description +The Proxmox role lacks comprehensive error handling, particularly for critical operations like API calls, vault operations, and file manipulations. + +## Current Issues +- No error handling for Proxmox API failures +- No validation of VM/LXC configurations before creation +- No retries for network operations +- No cleanup on failure + +## Required Changes + +### Step 1: Add validation tasks +Validate configurations before attempting creation. + +### Step 2: Add error handling blocks +Use `block/rescue/always` for critical operations. + +### Step 3: Add retries for network operations +Use `retries` and `delay` for API calls. + +## Implementation Steps + +### Example 1: VM Creation with Error Handling +```yaml +- name: Create VM with error handling + block: + - name: Validate VM configuration + ansible.builtin.assert: + that: + - vm.vmid is defined + - vm.vmid | int > 0 + - vm.node is defined + - vm.cores is defined and vm.cores | int > 0 + - vm.memory is defined and vm.memory | int > 0 + msg: "Invalid VM configuration for {{ vm.name }}" + + - name: Create VM + community.proxmox.proxmox_kvm: + # ... existing parameters ... + register: vm_creation_result + retries: 3 + delay: 10 + until: vm_creation_result is not failed + + rescue: + - name: Handle VM creation failure + ansible.builtin.debug: + msg: "Failed to create VM {{ vm.name }}: {{ ansible_failed_result.msg }}" + + - name: Cleanup partial resources + # Add cleanup tasks here + when: cleanup_partial_resources | default(true) + + always: + - name: Log VM creation attempt + ansible.builtin.debug: + msg: "VM creation attempt for {{ vm.name }} completed with status: {{ vm_creation_result is defined and vm_creation_result.changed | ternary('success', 'failed') }}" +``` + +### Example 2: API Call with Retries +```yaml +- name: Check Proxmox API availability + ansible.builtin.uri: + url: "https://{{ proxmox_api_host }}:8006/api2/json/version" + validate_certs: no + return_content: yes + register: api_check + retries: 5 + delay: 5 + until: api_check.status == 200 + ignore_errors: yes + +- name: Fail if API unavailable + ansible.builtin.fail: + msg: "Proxmox API unavailable at {{ proxmox_api_host }}" + when: api_check is failed +``` + +### Example 3: File Operation Error Handling +```yaml +- name: Manage vault file safely + block: + - name: Backup existing vault + ansible.builtin.copy: + src: "{{ proxmox_vault_file }}" + dest: "{{ proxmox_vault_file }}.backup" + remote_src: yes + when: vault_file_exists.stat.exists + + - name: Perform vault operations + # ... vault operations ... + + rescue: + - name: Restore vault from backup + ansible.builtin.copy: + src: "{{ proxmox_vault_file }}.backup" + dest: "{{ proxmox_vault_file }}" + remote_src: yes + when: vault_file_exists.stat.exists + + - name: Fail with error details + ansible.builtin.fail: + msg: "Vault operation failed: {{ ansible_failed_result.msg }}" +``` + +## Testing Requirements +- Test error scenarios (invalid configs, API unavailable) +- Verify cleanup works on failure +- Confirm retries work for transient failures +- Validate error messages are helpful + +## Acceptance Criteria +- [ ] All critical operations have error handling +- [ ] Validation added for configurations +- [ ] Retry logic implemented for network operations +- [ ] Cleanup procedures in place for failures +- [ ] Helpful error messages provided +- [ ] No silent failures diff --git a/issues/005_add_performance_optimizations.md b/issues/005_add_performance_optimizations.md new file mode 100644 index 0000000..ada747f --- /dev/null +++ b/issues/005_add_performance_optimizations.md @@ -0,0 +1,119 @@ +# Issue: Add Performance Optimizations + +**Status**: Open +**Priority**: Medium +**Component**: proxmox/tasks +**Assignee**: Junior Dev + +## Description +The Proxmox role could benefit from performance optimizations, particularly for image downloads and repeated operations. + +## Current Performance Issues +- Sequential image downloads (no parallelization) +- No caching of repeated operations +- No async operations for long-running tasks +- Inefficient fact gathering + +## Required Changes + +### Step 1: Add parallel downloads +Use async for image downloads to run concurrently. + +### Step 2: Implement caching +Add fact caching for repeated operations. + +### Step 3: Add conditional execution +Skip tasks when results are already present. + +## Implementation Steps + +### Example 1: Parallel Image Downloads +```yaml +- name: Download Cloud Init Isos in parallel + ansible.builtin.include_tasks: 42_download_isos.yaml + loop: "{{ proxmox_cloud_init_images | dict | map(attribute='value') }}" + loop_control: + loop_var: distro + async: 3600 # 1 hour timeout + poll: 0 + register: download_tasks + +- name: Check download status + ansible.builtin.async_status: + jid: "{{ item.ansible_job_id }}" + register: download_results + until: download_results.finished + retries: 30 + delay: 10 + loop: "{{ download_tasks.results }}" + loop_control: + loop_var: item +``` + +### Example 2: Add Fact Caching +```yaml +# In ansible.cfg or playbook +[defaults] +fact_caching = jsonfile +fact_caching_connection = /tmp/ansible_facts +fact_caching_timeout = 86400 + +# In tasks +- name: Gather facts with caching + ansible.builtin.setup: + cacheable: yes +``` + +### Example 3: Conditional Task Execution +```yaml +- name: Check if image already exists + ansible.builtin.stat: + path: "{{ proxmox_dirs.isos }}/{{ distro.name }}" + register: image_stat + changed_when: false + +- name: Download image only if missing + ansible.builtin.get_url: + url: "{{ distro.url }}" + dest: "{{ proxmox_dirs.isos }}/{{ distro.name }}" + mode: "0644" + when: not image_stat.stat.exists + register: download_result + +- name: Skip conversion if raw image exists + ansible.builtin.stat: + path: "{{ proxmox_dirs.isos }}/{{ raw_image_name }}" + register: raw_image_stat + changed_when: false + +- name: Convert to raw only if needed + ansible.builtin.command: + cmd: "qemu-img convert -O raw {{ proxmox_dirs.isos }}/{{ distro.name }} {{ proxmox_dirs.isos }}/{{ raw_image_name }}" + when: + - download_result is changed or not raw_image_stat.stat.exists + - image_stat.stat.exists +``` + +### Example 4: Batch VM Operations +```yaml +- name: Create VMs in batches + ansible.builtin.include_tasks: 55_create_vm.yaml + loop: "{{ vms | batch(3) | flatten }}" + loop_control: + loop_var: "vm" + throttle: 3 +``` + +## Testing Requirements +- Measure performance before and after changes +- Verify parallel operations don't cause conflicts +- Test caching works correctly +- Confirm conditional execution skips appropriately + +## Acceptance Criteria +- [ ] Image downloads run in parallel +- [ ] Fact caching implemented and working +- [ ] Tasks skip when results already exist +- [ ] Performance metrics show improvement +- [ ] No race conditions in parallel operations +- [ ] Documentation updated with performance notes diff --git a/roles/edge_vps/docs/plans/2026-02-24-edge-vps-role.md b/roles/edge_vps/docs/plans/2026-02-24-edge-vps-role.md new file mode 100644 index 0000000..5803011 --- /dev/null +++ b/roles/edge_vps/docs/plans/2026-02-24-edge-vps-role.md @@ -0,0 +1,715 @@ +# Edge VPS Ansible Role Implementation Plan + +> **For Claude:** REQUIRED SUB-SKILL: Use superpowers:executing-plans to implement this plan task-by-task. + +**Goal:** Create a modular Ansible role for deploying edge VPS infrastructure components (WireGuard, Traefik, Pangolin, Elastic Agent). + +**Architecture:** Modular task-based role following existing patterns in the repository. Each component has its own numbered task file. Configs are templated with secrets from ansible-vault encrypted group_vars. + +**Tech Stack:** Ansible, Jinja2 templates, Docker Compose, WireGuard, Traefik, Pangolin, Elastic Fleet Agent + +--- + +### Task 1: Create Role Directory Structure + +**Files:** +- Create: `roles/edge_vps/tasks/main.yaml` +- Create: `roles/edge_vps/handlers/main.yaml` +- Create: `roles/edge_vps/defaults/main.yaml` +- Create: `roles/edge_vps/templates/` directory structure + +**Step 1: Create directory structure** + +Run: +```bash +mkdir -p tasks handlers defaults templates/wireguard templates/traefik templates/pangolin templates/elastic-agent +``` + +**Step 2: Create defaults/main.yaml** + +```yaml +--- +edge_vps_config_base: /root/config +edge_vps_wireguard_config_dir: /etc/wireguard +edge_vps_wireguard_interface: wg0 +edge_vps_wireguard_address: "10.133.7.1/24" +edge_vps_wireguard_port: 61975 +edge_vps_traefik_config_dir: "{{ edge_vps_config_base }}/traefik" +edge_vps_traefik_logs_dir: "{{ edge_vps_traefik_config_dir }}/logs" +edge_vps_pangolin_config_dir: "{{ edge_vps_config_base }}/pangolin" +edge_vps_elastic_config_dir: "{{ edge_vps_config_base }}/elastic-agent" +edge_vps_elastic_state_dir: /var/lib/elastic-agent/elastic-system/elastic-agent/state +``` + +**Step 3: Create handlers/main.yaml** + +```yaml +--- +- name: Restart wireguard + ansible.builtin.systemd: + name: "wg-quick@{{ edge_vps_wireguard_interface }}" + state: restarted + listen: restart wireguard + +- name: Restart traefik + ansible.builtin.command: + cmd: docker compose restart + chdir: "{{ edge_vps_traefik_config_dir }}" + listen: restart traefik +``` + +**Step 4: Commit** + +```bash +git add defaults/main.yaml handlers/main.yaml +git commit -m "feat(edge_vps): add role structure and handlers" +``` + +--- + +### Task 2: Create Directory Setup Task + +**Files:** +- Create: `roles/edge_vps/tasks/10_directories.yaml` + +**Step 1: Create 10_directories.yaml** + +```yaml +--- +- name: Create config base directory + ansible.builtin.file: + path: "{{ edge_vps_config_base }}" + state: directory + mode: "0755" + +- name: Create Traefik directories + ansible.builtin.file: + path: "{{ item }}" + state: directory + mode: "0755" + loop: + - "{{ edge_vps_traefik_config_dir }}" + - "{{ edge_vps_traefik_logs_dir }}" + +- name: Create Pangolin config directory + ansible.builtin.file: + path: "{{ edge_vps_pangolin_config_dir }}" + state: directory + mode: "0755" + +- name: Create Elastic Agent directories + ansible.builtin.file: + path: "{{ item }}" + state: directory + mode: "0755" + loop: + - "{{ edge_vps_elastic_config_dir }}" + - "{{ edge_vps_elastic_state_dir }}" +``` + +**Step 2: Commit** + +```bash +git add tasks/10_directories.yaml +git commit -m "feat(edge_vps): add directory setup task" +``` + +--- + +### Task 3: Create WireGuard Task and Template + +**Files:** +- Create: `roles/edge_vps/tasks/20_wireguard.yaml` +- Create: `roles/edge_vps/templates/wireguard/wg0.conf.j2` + +**Step 1: Create templates/wireguard/wg0.conf.j2** + +```jinja2 +[Interface] +Address = {{ edge_vps_wireguard_address }} +ListenPort = {{ edge_vps_wireguard_port }} +PrivateKey = {{ vault_edge_vps.wireguard.private_key }} + +PostUp = sysctl -w net.ipv4.ip_forward=1 +PostUp = iptables -A FORWARD -i {{ edge_vps_wireguard_interface }} -j ACCEPT +PostUp = iptables -A FORWARD -o {{ edge_vps_wireguard_interface }} -j ACCEPT +{% for route in edge_vps_wireguard_routes | default([]) %} +PostUp = ip route add {{ route }} via {{ route.gateway }} dev {{ edge_vps_wireguard_interface }} +{% endfor %} +PostDown = iptables -D FORWARD -i {{ edge_vps_wireguard_interface }} -j ACCEPT +PostDown = iptables -D FORWARD -o {{ edge_vps_wireguard_interface }} -j ACCEPT +{% for route in edge_vps_wireguard_routes | default([]) %} +PostDown = ip route del {{ route }} via {{ route.gateway }} dev {{ edge_vps_wireguard_interface }} +{% endfor %} + +{% for peer in vault_edge_vps.wireguard.peers %} +[Peer] +# {{ peer.name }} +PublicKey = {{ peer.public_key }} +PresharedKey = {{ peer.preshared_key }} +AllowedIPs = {{ peer.allowed_ips }} + +{% endfor %} +``` + +**Step 2: Create tasks/20_wireguard.yaml** + +```yaml +--- +- name: Install WireGuard + ansible.builtin.apt: + name: wireguard + state: present + update_cache: true + +- name: Deploy WireGuard config + ansible.builtin.template: + src: wireguard/wg0.conf.j2 + dest: "{{ edge_vps_wireguard_config_dir }}/{{ edge_vps_wireguard_interface }}.conf" + mode: "0600" + notify: restart wireguard + +- name: Enable WireGuard + ansible.builtin.systemd: + name: "wg-quick@{{ edge_vps_wireguard_interface }}" + enabled: true + state: started +``` + +**Step 3: Commit** + +```bash +git add tasks/20_wireguard.yaml templates/wireguard/wg0.conf.j2 +git commit -m "feat(edge_vps): add WireGuard setup task and template" +``` + +--- + +### Task 4: Create Traefik Task and Template + +**Files:** +- Create: `roles/edge_vps/tasks/30_traefik.yaml` +- Create: `roles/edge_vps/templates/traefik/traefik_config.yml.j2` + +**Step 1: Create templates/traefik/traefik_config.yml.j2** + +```jinja2 +api: + insecure: true + dashboard: true + +providers: + http: + endpoint: "http://pangolin:3001/api/v1/traefik-config" + pollInterval: "5s" + file: + filename: "/etc/traefik/dynamic_config.yml" + +experimental: + plugins: + badger: + moduleName: "github.com/fosrl/badger" + version: "v1.2.1" + +log: + level: "INFO" + format: "common" + maxSize: 100 + maxBackups: 3 + maxAge: 3 + compress: true + +certificatesResolvers: + letsencrypt: + acme: + dnsChallenge: + provider: "cloudflare" + email: "{{ edge_vps_acme_email }}" + storage: "/letsencrypt/acme.json" + caServer: "https://acme-v02.api.letsencrypt.org/directory" + +entryPoints: + web: + address: ":80" + websecure: + address: ":443" + transport: + respondingTimeouts: + readTimeout: "30m" + http: + tls: + certResolver: "letsencrypt" + tcp-6443: + address: ":6443/tcp" + +serversTransport: + insecureSkipVerify: true + +ping: + entryPoint: "web" + +accessLog: + filePath: "/var/log/traefik/access.log" + format: common +``` + +**Step 2: Create tasks/30_traefik.yaml** + +```yaml +--- +- name: Deploy Traefik config + ansible.builtin.template: + src: traefik/traefik_config.yml.j2 + dest: "{{ edge_vps_traefik_config_dir }}/traefik_config.yml" + mode: "0644" + notify: restart traefik + +- name: Deploy Cloudflare credentials for ACME + ansible.builtin.copy: + content: | + CF_DNS_API_TOKEN={{ vault_edge_vps.traefik.cloudflare_api_token }} + dest: "{{ edge_vps_traefik_config_dir }}/cloudflare.env" + mode: "0600" + no_log: true +``` + +**Step 3: Commit** + +```bash +git add tasks/30_traefik.yaml templates/traefik/traefik_config.yml.j2 +git commit -m "feat(edge_vps): add Traefik setup task and template" +``` + +--- + +### Task 5: Create Pangolin Task and Templates + +**Files:** +- Create: `roles/edge_vps/tasks/40_pangolin.yaml` +- Create: `roles/edge_vps/templates/pangolin/config.yml.j2` +- Create: `roles/edge_vps/templates/pangolin/docker-compose.yml.j2` + +**Step 1: Create templates/pangolin/config.yml.j2** + +```jinja2 +gerbil: + start_port: 51820 + base_endpoint: "{{ edge_vps_pangolin_base_endpoint }}" + +app: + dashboard_url: "{{ edge_vps_pangolin_dashboard_url }}" + log_level: "info" + telemetry: + anonymous_usage: true + +domains: + domain1: + base_domain: "{{ edge_vps_pangolin_base_domain }}" + +server: + secret: "{{ vault_edge_vps.pangolin.server_secret }}" + cors: + origins: ["{{ edge_vps_pangolin_dashboard_url }}"] + methods: ["GET", "POST", "PUT", "DELETE", "PATCH"] + allowed_headers: ["X-CSRF-Token", "Content-Type"] + credentials: false + maxmind_db_path: "./config/GeoLite2-Country.mmdb" + +flags: + require_email_verification: false + disable_signup_without_invite: true + disable_user_create_org: false + allow_raw_resources: true +``` + +**Step 2: Create templates/pangolin/docker-compose.yml.j2** + +```yaml +services: + pangolin: + image: fosrl/pangolin:latest + container_name: pangolin + restart: unless-stopped + ports: + - "3001:3001" + - "443:443" + - "80:80" + volumes: + - ./config.yml:/app/config/config.yml:ro + - ./letsencrypt:/letsencrypt + depends_on: + - gerbil + + gerbil: + image: fosrl/gerbil:latest + container_name: gerbil + restart: unless-stopped + network_mode: host + cap_add: + - NET_ADMIN + - SYS_MODULE + volumes: + - /lib/modules:/lib/modules +``` + +**Step 3: Create tasks/40_pangolin.yaml** + +```yaml +--- +- name: Deploy Pangolin config + ansible.builtin.template: + src: pangolin/config.yml.j2 + dest: "{{ edge_vps_pangolin_config_dir }}/config.yml" + mode: "0644" + notify: restart pangolin + +- name: Deploy Pangolin docker-compose + ansible.builtin.template: + src: pangolin/docker-compose.yml.j2 + dest: "{{ edge_vps_pangolin_config_dir }}/docker-compose.yml" + mode: "0644" + +- name: Create letsencrypt directory for Pangolin + ansible.builtin.file: + path: "{{ edge_vps_pangolin_config_dir }}/letsencrypt" + state: directory + mode: "0755" + +- name: Start Pangolin + community.docker.docker_compose_v2: + project_src: "{{ edge_vps_pangolin_config_dir }}" + state: present +``` + +**Step 4: Commit** + +```bash +git add tasks/40_pangolin.yaml templates/pangolin/ +git commit -m "feat(edge_vps): add Pangolin setup task and templates" +``` + +--- + +### Task 6: Create Elastic Agent Task and Templates + +**Files:** +- Create: `roles/edge_vps/tasks/50_elastic_agent.yaml` +- Create: `roles/edge_vps/templates/elastic-agent/docker-compose.yml.j2` +- Create: `roles/edge_vps/templates/elastic-agent/elastic-agent.yml.j2` + +**Step 1: Create templates/elastic-agent/elastic-agent.yml.j2** + +```yaml +fleet: + enabled: true +``` + +**Step 2: Create templates/elastic-agent/docker-compose.yml.j2** + +```yaml +services: + elastic-agent: + image: docker.elastic.co/elastic-agent/elastic-agent:{{ edge_vps_elastic_version }} + container_name: elastic-agent + restart: always + network_mode: host + dns: + - {{ edge_vps_elastic_dns_server }} + dns_search: + - elastic-system.svc.cluster.local + - svc.cluster.local + - cluster.local + user: "0:0" + privileged: true + entrypoint: ["/usr/bin/env", "bash", "-c"] + command: + - | + set -e + if [[ -f /mnt/elastic-internal/elasticsearch-association/elastic-system/elasticsearch/certs/ca.crt ]]; then + if [[ -f /usr/bin/update-ca-trust ]]; then + cp /mnt/elastic-internal/elasticsearch-association/elastic-system/elasticsearch/certs/ca.crt /etc/pki/ca-trust/source/anchors/ + /usr/bin/update-ca-trust + elif [[ -f /usr/sbin/update-ca-certificates ]]; then + cp /mnt/elastic-internal/elasticsearch-association/elastic-system/elasticsearch/certs/ca.crt /usr/local/share/ca-certificates/ + /usr/sbin/update-ca-certificates + fi + fi + exec /usr/bin/tini -- /usr/local/bin/docker-entrypoint -e -c /etc/agent/elastic-agent.yml + environment: + - FLEET_CA=/mnt/elastic-internal/fleetserver-association/elastic-system/fleet-server/certs/ca.crt + - FLEET_ENROLL=true + - FLEET_ENROLLMENT_TOKEN={{ vault_edge_vps.elastic.fleet_enrollment_token }} + - FLEET_URL={{ edge_vps_elastic_fleet_url }} + - STATE_PATH=/usr/share/elastic-agent/state + - CONFIG_PATH=/usr/share/elastic-agent/state + - NODE_NAME={{ inventory_hostname }} + volumes: + - {{ edge_vps_elastic_state_dir }}:/usr/share/elastic-agent/state + - ./elastic-agent.yml:/etc/agent/elastic-agent.yml:ro + - ./elasticsearch-ca.crt:/mnt/elastic-internal/elasticsearch-association/elastic-system/elasticsearch/certs/ca.crt:ro + - ./fleet-ca.crt:/mnt/elastic-internal/fleetserver-association/elastic-system/fleet-server/certs/ca.crt:ro + - {{ edge_vps_traefik_logs_dir }}:/var/log/traefik:ro +``` + +**Step 3: Create tasks/50_elastic_agent.yaml** + +```yaml +--- +- name: Deploy Elastic Agent config + ansible.builtin.template: + src: elastic-agent/elastic-agent.yml.j2 + dest: "{{ edge_vps_elastic_config_dir }}/elastic-agent.yml" + mode: "0644" + +- name: Deploy Elastic Agent docker-compose + ansible.builtin.template: + src: elastic-agent/docker-compose.yml.j2 + dest: "{{ edge_vps_elastic_config_dir }}/docker-compose.yml" + mode: "0644" + +- name: Deploy Elasticsearch CA certificate + ansible.builtin.copy: + src: elastic-agent/elasticsearch-ca.crt + dest: "{{ edge_vps_elastic_config_dir }}/elasticsearch-ca.crt" + mode: "0644" + +- name: Deploy Fleet CA certificate + ansible.builtin.copy: + src: elastic-agent/fleet-ca.crt + dest: "{{ edge_vps_elastic_config_dir }}/fleet-ca.crt" + mode: "0644" + +- name: Start Elastic Agent + community.docker.docker_compose_v2: + project_src: "{{ edge_vps_elastic_config_dir }}" + state: present +``` + +**Step 4: Commit** + +```bash +git add tasks/50_elastic_agent.yaml templates/elastic-agent/ +git commit -m "feat(edge_vps): add Elastic Agent setup task and templates" +``` + +--- + +### Task 7: Create Main Task Orchestrator + +**Files:** +- Create: `roles/edge_vps/tasks/main.yaml` + +**Step 1: Create tasks/main.yaml** + +```yaml +--- +- name: Setup directories + ansible.builtin.include_tasks: 10_directories.yaml + +- name: Setup WireGuard + ansible.builtin.include_tasks: 20_wireguard.yaml + +- name: Setup Traefik + ansible.builtin.include_tasks: 30_traefik.yaml + +- name: Setup Pangolin + ansible.builtin.include_tasks: 40_pangolin.yaml + +- name: Setup Elastic Agent + ansible.builtin.include_tasks: 50_elastic_agent.yaml +``` + +**Step 2: Commit** + +```bash +git add tasks/main.yaml +git commit -m "feat(edge_vps): add main task orchestrator" +``` + +--- + +### Task 8: Create Inventory Variables + +**Files:** +- Create: `vars/group_vars/vps/vars.yaml` +- Create: `vars/group_vars/vps/secrets.yaml` + +**Step 1: Create vars/group_vars/vps/vars.yaml** + +```yaml +edge_vps_wireguard_address: "10.133.7.1/24" +edge_vps_wireguard_port: 61975 +edge_vps_wireguard_routes: + - network: "10.43.0.0/16" + gateway: "10.133.7.4" + +edge_vps_pangolin_dashboard_url: "https://pangolin.seyshiro.de" +edge_vps_pangolin_base_endpoint: "pangolin.seyshiro.de" +edge_vps_pangolin_base_domain: "seyshiro.de" + +edge_vps_acme_email: "me+acme@tudattr.dev" + +edge_vps_elastic_version: "9.2.2" +edge_vps_elastic_dns_server: "10.43.0.10" +edge_vps_elastic_fleet_url: "https://fleet-server-agent-http.elastic-system.svc:8220" +``` + +**Step 2: Create vars/group_vars/vps/secrets.yaml (template)** + +```yaml +vault_edge_vps: + wireguard: + private_key: "YOUR_WIREGUARD_PRIVATE_KEY" + peers: + - name: lilcrow + public_key: "PEER_PUBLIC_KEY" + preshared_key: "PEER_PRESHARED_KEY" + allowed_ips: "10.133.7.2/32" + - name: homelab + public_key: "PEER_PUBLIC_KEY" + preshared_key: "PEER_PRESHARED_KEY" + allowed_ips: "10.133.7.3/32" + - name: k3s + public_key: "PEER_PUBLIC_KEY" + preshared_key: "PEER_PRESHARED_KEY" + allowed_ips: "10.133.7.4/32, 10.43.0.0/16" + pangolin: + server_secret: "YOUR_PANGOLIN_SERVER_SECRET" + traefik: + cloudflare_api_token: "YOUR_CLOUDFLARE_API_TOKEN" + elastic: + fleet_enrollment_token: "YOUR_FLEET_ENROLLMENT_TOKEN" +``` + +**Step 3: Encrypt secrets file** + +Run: +```bash +ansible-vault encrypt vars/group_vars/vps/secrets.yaml +``` + +**Step 4: Commit** + +```bash +git add vars/group_vars/vps/ +git commit -m "feat(edge_vps): add inventory variables for VPS group" +``` + +--- + +### Task 9: Update README + +**Files:** +- Modify: `roles/edge_vps/README.md` + +**Step 1: Update README.md** + +```markdown +# Edge VPS + +Configures edge VPS instances with WireGuard VPN, Traefik reverse proxy, Pangolin, and Elastic Fleet Agent. + +## Requirements + +- Docker and Docker Compose installed +- Ansible community.docker collection + +## Role Variables + +### WireGuard + +| Variable | Default | Description | +|----------|---------|-------------| +| `edge_vps_wireguard_address` | `10.133.7.1/24` | WireGuard interface address | +| `edge_vps_wireguard_port` | `61975` | WireGuard listen port | +| `edge_vps_wireguard_interface` | `wg0` | WireGuard interface name | +| `edge_vps_wireguard_routes` | `[]` | List of routes to add (network, gateway) | + +### Traefik + +| Variable | Default | Description | +|----------|---------|-------------| +| `edge_vps_traefik_config_dir` | `/root/config/traefik` | Traefik config directory | +| `edge_vps_acme_email` | - | Email for Let's Encrypt | + +### Pangolin + +| Variable | Default | Description | +|----------|---------|-------------| +| `edge_vps_pangolin_dashboard_url` | - | Pangolin dashboard URL | +| `edge_vps_pangolin_base_endpoint` | - | Pangolin base endpoint | +| `edge_vps_pangolin_base_domain` | - | Base domain for Pangolin | + +### Elastic Agent + +| Variable | Default | Description | +|----------|---------|-------------| +| `edge_vps_elastic_version` | `9.2.2` | Elastic Agent version | +| `edge_vps_elastic_fleet_url` | - | Fleet server URL | +| `edge_vps_elastic_dns_server` | `10.43.0.10` | DNS server for agent | + +## Secrets + +Store secrets in `vars/group_vars/vps/secrets.yaml` (ansible-vault encrypted): + +```yaml +vault_edge_vps: + wireguard: + private_key: "..." + peers: [...] + pangolin: + server_secret: "..." + traefik: + cloudflare_api_token: "..." + elastic: + fleet_enrollment_token: "..." +``` + +## Dependencies + +None. + +## Example Playbook + +```yaml +- hosts: vps + roles: + - role: edge_vps +``` + +## License + +MIT +``` + +**Step 2: Commit** + +```bash +git add README.md +git commit -m "docs(edge_vps): update README with role documentation" +``` + +--- + +### Task 10: Move Certificate Files + +**Files:** +- Move: `files/agent/agent/elasticsearch-ca.crt` → `files/elastic-agent/` +- Move: `files/agent/agent/fleet-ca.crt` → `files/elastic-agent/` + +**Step 1: Move certificate files** + +Run: +```bash +mkdir -p files/elastic-agent +mv files/agent/agent/elasticsearch-ca.crt files/elastic-agent/ +mv files/agent/agent/fleet-ca.crt files/elastic-agent/ +rm -rf files/agent +``` + +**Step 2: Commit** + +```bash +git add files/ +git commit -m "refactor(edge_vps): reorganize certificate files" +``` diff --git a/roles/proxmox/tasks/04_configure_hosts.yaml b/roles/proxmox/tasks/04_configure_hosts.yaml new file mode 100644 index 0000000..00fea31 --- /dev/null +++ b/roles/proxmox/tasks/04_configure_hosts.yaml @@ -0,0 +1,15 @@ +--- +- name: Configure /etc/hosts with Proxmox cluster nodes + ansible.builtin.blockinfile: + path: /etc/hosts + block: | + # Proxmox Cluster Nodes + 192.168.20.12 aya01.seyshiro.de aya01 + 192.168.20.14 lulu.seyshiro.de lulu + 192.168.20.28 inko01.seyshiro.de inko01 + 192.168.20.10 naruto01.seyshiro.de naruto01 + 192.168.20.9 mii01.seyshiro.de mii01 + marker: "# {mark} ANSIBLE MANAGED BLOCK - PROXMOX CLUSTER NODES" + create: true + mode: "644" + when: is_proxmox_node | bool diff --git a/roles/proxmox/tasks/05_setup_node.yaml b/roles/proxmox/tasks/05_setup_node.yaml index 1c18463..48dfd2b 100644 --- a/roles/proxmox/tasks/05_setup_node.yaml +++ b/roles/proxmox/tasks/05_setup_node.yaml @@ -6,5 +6,8 @@ state: present loop: "{{ proxmox_node_dependencies }}" +- name: Configure hosts file for cluster nodes + ansible.builtin.include_tasks: 04_configure_hosts.yaml + - name: Ensure Harware Acceleration on node ansible.builtin.include_tasks: 06_hardware_acceleration.yaml diff --git a/roles/proxmox/tasks/42_download_isos.yaml b/roles/proxmox/tasks/42_download_isos.yaml index 38189ce..619f17a 100644 --- a/roles/proxmox/tasks/42_download_isos.yaml +++ b/roles/proxmox/tasks/42_download_isos.yaml @@ -10,6 +10,7 @@ dest: "{{ proxmox_dirs.isos }}/{{ distro.name }}" mode: "0644" when: not image_stat.stat.exists + register: download_result - name: Set raw image file name fact ansible.builtin.set_fact: @@ -24,5 +25,5 @@ ansible.builtin.command: cmd: "qemu-img convert -O raw {{ proxmox_dirs.isos }}/{{ distro.name }} {{ proxmox_dirs.isos }}/{{ raw_image_name }}" when: - - download_result is changed or not raw_image_stat.stat.exists + - (download_result is defined and download_result is changed) or not raw_image_stat.stat.exists - image_stat.stat.exists diff --git a/vars/group_vars/proxmox/secrets_vm.yaml b/vars/group_vars/proxmox/secrets_vm.yaml index ac4aff7..fa03b67 100644 --- a/vars/group_vars/proxmox/secrets_vm.yaml +++ b/vars/group_vars/proxmox/secrets_vm.yaml @@ -1,66 +1,66 @@ $ANSIBLE_VAULT;1.1;AES256 -38376362633961306438343561623064353761616565636134623630363864373866643232666465 -3830396166373030623732383836366431363338666133360a633065643865323132616133376366 -31666466613663353431393039386131623837353862336632303832643464366439313734626435 -6365663762313763650a386133396161366134326230383065613432636366626133643732373737 -63643132623337666131333533346261303933366439393766393533623831666536393632656365 -38383465333139666264623632323939396536363863303932316261303135373330326631353565 -62636662613134313836663238316433663865363332646538643930353862356465373362663430 -35333936336665353238636438656530356434353134653461343661616162616333646562393964 -32363965323366363162353238303430343261356237613735616433303635613161653366656439 -39643833633663616665646233356532313030383535636164653539613533623666356561653736 -37383137633830306233633835613864353561306537373238323034663035363535623431316534 -63663234303838373630646536633563633136363730663832393738326163366634613164653532 -63363264333661646133343431356533306564636465363363653035653965313430363665653265 -30646333666630363136306636623262653361383664393162663463666365643735343835373365 -30663633346531386263303432353662323563633636633465373538313434356535383033366663 -37666336363863646432396562316130303661343462313435373936623636633061393030663139 -32323233306236626635656133366230393030366563383835616238393336643364303563643430 -30333731393962373738336331323639346662646539386561623834313638313636623161313236 -62353335633933313131613130313164626238356134653733386334663461326265666437626366 -65333262343164323966333232626635626339323634383735356536353733363933373935636461 -62646130373431326662663163336361393762346630306363633761396664653633396664626530 -38343135333433356135386539313439383738653561653536633936613338373765366139363134 -35366265656164386335366466393066386232663562316665383363323164316337336630343234 -35623138653735326531643063633062353137663763376532663731313537623337356339633532 -64353239346634626539613032303962333662613765643639313266323462346239623736313863 -31306262626161393862633038363061636362303864616566323065663964323563353034383362 -32373665306664303036646565633830613130353531666264646162366538323366636330663737 -37393238346262363039356536643765346165623666613331356636613630396361346566346633 -37613465663631323530393366363766383136336337656163336431653935613765383634633462 -35656264313739313630343238306439323030346465366337333562373132313564653333353461 -37393635313965633064376239343139343663363633613437373632366139396539363265643731 -30393333393236313033353364316535366664613439623163376163386362376161666334393864 -33303365313564636337613239326233616331623166386562366438626135343961356330643861 -37633239353064366262373130383635653037363037633035663738313739313739613136346332 -33373834623365303036313436373037343763633762363833383865666434363533653632373663 -62343565376631346632663265343335616563356632386166383238663663376632646539383339 -61653533363765356365323139383037643363393539393933643164386362363164396535366231 -66613533353864303766376261326139616538353237383235366261383331653435623637396536 -65353063656561383066666134313039383166386238333438356161646562303866313238623337 -35616166623433333130316565333738613163643166373661316338653236363962616337326633 -34653533373636363464346362643166666532656636363432356261633537633535616562313036 -30326232306561646438316533646636623566313963393563323366626566393936316466303635 -38343439313437653835623538346532633936343662666161353765353366383637613964356466 -34323063633132643135393537393061653261316635643838636262323837613134333936383038 -66373538383735656263633066653566663631643062333139396233363764326230653032353264 -62326264666261346265373062316630326636336132666661383765643637383565363433656464 -37663231326334393734646230646263333137313432343763383662383165373037663838306137 -38363262326165366165313230653265616333663062666134356561356236656561333433323935 -39633337303763383435373532333838656335396662336139343931303431363933306562623635 -66376430306165336233343931653231393633623530663133346161636435646236663465303065 -61303035373937613433396465353732396364393231663331346237373939636233333639316130 -61653739613737303362303263333366383437613537633964663932373035326439313239373439 -64383935383661616164616462363462326661373338323864373634663737313261346632663464 -33656330383133326136373331363161333065323533303762356532656264616632323165323166 -66376339343065633165326662343330306662666164316435383264363833663664613338336535 -36396238653361626666306234373564303037633264306261306133663665373939363865396236 -33353037666162376339366563623832653434396237613064386335323837373636613462363034 -65323663636563366161356665356562313165663262653663636266623661343538666239663230 -33333837396132303033646432373633613135633062353930376232653261333036376338386632 -37383132656361383339663833306163636661373339306138383936306137653961306135363036 -62306162623465646131653966306533646166363665353966623132623765613862353665656538 -66623931353032366635666138356365356364663931636435396363623061366131623166363466 -63356234383338373834353666643036396561643261363236646435333466326464636335386664 -64363965386439393236616135636437386432353361353632333363323536313334313462313934 -6531 +37386662376562356339383165393430626530616631666337396134623666336639623534313631 +3062343366656334613538333031343537356639346136610a616165393631306363323735623131 +30383033386538306461363338353034373430393038316566333062626464353661356534303838 +3736326364323461620a346565613539633163316564353434646433336539373762653631336236 +66643831363037373963373134326238303933303732323332336639346333663366626633333734 +64396465356262346137393166636332336139386439653966313337393232636165646431343838 +66366535363936373330393963643063363765383939656334356631633336373062633066313662 +36333231633166396337626538626331366138356236386464316439363561303938353830353336 +38323539366666376465323661643634616663626363306430636233323535313630643330613131 +65303764653762383761323736306638396637653861653462656262373765663066323162643434 +66373066663762336566366430386239633163383365333031396163383864353232326636613566 +35316362643763646463663963333261363230336439373237303662353437383165303836633436 +63376536393733353739633765333639343639616534616563396234306430323838333066616138 +36333461336637663835323761313566353635306336363939393463363030613666363562326564 +66306163326332373235393934353637303466633165313063616433646163653931333630333932 +33373831616331373934316364306238343434666236356163326662636466663662323937633737 +34333063303835613364666334353035383837373261373332626338366631656662643164353836 +64396534393066323164663366653236656163383736383530393536313661376635393836336233 +32646630623934343530396561366166656536353461643935366136616631363464396237616336 +61353738373763323261613131306532386330613238636163663531643562366638303061323464 +39333637323438666566386562643634306266393961383763376464363339646264306330373266 +62346538356530643332633661643065623934363631633765333163666666663336333733393933 +31383134653737653661613638336533623430373661333732316638356264393166396664376339 +65663663396163643865353334343064623833323066663166373338323062343534373664323462 +39316633653662616365323339303138616333303630626163383966386161623535663636316639 +63363561303534653265636133396232633234643033663363656234396161616464663838373931 +37383134656430646665396464653535343964396634613361373235313664386236383365356462 +38323865383261653133303437313833646164393262666436383732633932353032303165653163 +38383165373731633538373232323831313036653366356234393638666562633234323665376539 +36663536633839383434323538666235306363653038383931633333356538333735623837313737 +61306134393537623934313034313330356133313736623563386366633433376635643532383136 +66633362393530323836643235336465316663663863663864346363343965306238383035306663 +31363131633435326264326434616635376465396332663464646365303135353033383361643064 +38363863663131613636316335333361393064343230613039613639316430666138323462373163 +36346335333461346563363337313163336362663235623237376164313131343233316635373235 +62326232396534653736633063633065376630643437383862386365313266303338636331326338 +35313966303064303165313338626337643963333034316434373133613030643463653765303335 +33346134663163323263313466323064313932626535396239303864313937373833643335626132 +37643031653761323833313333303564663238623133353164326237626566663434323262363032 +30333062356565383866613235356538323432333334336564323462346465343030336637356539 +30663231613065383331373433383964653939653364356163333731343639666365313633383863 +66383330636339383738336165396237636431353062333064363831626162376164343537623337 +31643365363736316563373364356336343432663364633264366139333435623436663737363135 +38343439353964303062346165373262326137643431626165333831616564313330626534356536 +64653161323065373939303766313239613366636533303836316663333263316337366562636437 +65636133663235613636353263363263613464636434663139623764363866316235383332336237 +66323436313165373261306637393836646661323462613833343131323633396364323834343430 +62646139656330613863623932643335353663633639383064366137613431643063643061666336 +61643131393831343166313533343133313633353963383365636464353137393765363662316231 +61366633343937393833373132376338653462613564303334386462386665306466626162373936 +64623462336531346463333763373066343539623264363162303030636464656461313138393938 +63626465376535393235386363336535613838376230626331663038653837623237636535623730 +38313134623531333035306134393262663435323063656239383639353563303735656462363334 +33323632383031306533623431353666383739346635353930626335323864636666353530386430 +62346536656535313433356262316436336163396665653762333861633631326362303066376533 +30323431636637306535653561353332653761646532383364613334636430353536623834663161 +37383331326238316533373032353666653264376632623664333463656130613065383139633663 +64383330396561383133363339303434663834333361316562323631383838303035386433316361 +32353364373339326361636130646163633634616364643639313839303765313034616230663633 +34643566613936386331323138633437396465376164333832373763383264366461373933666563 +33316130343130646535306233306633323732323534663436616131613332663930643164343831 +36623364313230663764633361356634616464633962353938363465646435373266356164326231 +37336433666464366665396238313662333431363032653133353230623361636362373733383931 +62646634336462636330626434313434653839356131353232326163616634363035643538303339 +3832