Compare commits
28 Commits
fbd5fa5faa
...
master
| Author | SHA1 | Date | |
|---|---|---|---|
|
a31a07bd16
|
|||
|
54a951b96a
|
|||
|
e1379bc480
|
|||
|
037e0cab9b
|
|||
|
2655869814
|
|||
|
0e96b5030d
|
|||
|
a217c79e7d
|
|||
|
6a16ebf084
|
|||
|
2617aa2bd2
|
|||
|
b686e4da4d
|
|||
|
439c239ac8
|
|||
|
acf599f905
|
|||
|
eae4f5e27b
|
|||
|
4fbe9bd5de
|
|||
|
dcc4970b20
|
|||
|
2eac1362b5
|
|||
|
e3d8479397
|
|||
|
91c5eab236
|
|||
|
ca7f089fe6
|
|||
|
479e256b1e
|
|||
|
11e5b5752e
|
|||
|
392938d0fb
|
|||
|
2cc059104e
|
|||
|
d09a26b73a
|
|||
|
097676f569
|
|||
|
e878661cb3
|
|||
|
cb50c1c515
|
|||
|
33de71a087
|
4
.crushignore
Normal file
4
.crushignore
Normal file
@@ -0,0 +1,4 @@
|
|||||||
|
ansible-vault-password-file
|
||||||
|
|
||||||
|
*secrets.yml
|
||||||
|
*secrets.toml
|
||||||
2
.gitignore
vendored
2
.gitignore
vendored
@@ -7,3 +7,5 @@
|
|||||||
/ansible-vault-password-file
|
/ansible-vault-password-file
|
||||||
/temp
|
/temp
|
||||||
*.retry
|
*.retry
|
||||||
|
|
||||||
|
__pycache__
|
||||||
|
|||||||
69
AGENTS.md
Normal file
69
AGENTS.md
Normal file
@@ -0,0 +1,69 @@
|
|||||||
|
# AGENTS GUIDE
|
||||||
|
|
||||||
|
## Overview
|
||||||
|
Ansible-based server automation for personal services. Playbooks provision Dockerized apps (e.g., gitea, authelia, homepage, miniflux, wakapi, memos) via per-app users, Caddy proxy, and Yandex Docker Registry. Secrets are managed with Ansible Vault.
|
||||||
|
|
||||||
|
## Project Layout
|
||||||
|
- Playbooks: `playbook-*.yml` (per service), `playbook-all-*.yml` for grouped actions.
|
||||||
|
- Inventory: `production.yml` (ungrouped host `server`).
|
||||||
|
- Variables: `vars/*.yml` (app configs, images), secrets in `vars/secrets.yml` (vault-encrypted).
|
||||||
|
- Roles: custom roles under `roles/` (e.g., `eget`, `owner`, `secrets`) plus galaxy roles fetched to `galaxy.roles/`.
|
||||||
|
- Files/templates: service docker-compose and backup templates under `files/`, shared templates under `templates/`.
|
||||||
|
- Scripts: helper Python scripts in `scripts/` (SMTP utilities) and `files/backups/backup-all.py`.
|
||||||
|
- CI: `.gitea/workflows/lint.yml` runs yamllint and ansible-lint.
|
||||||
|
- Hooks: `lefthook.yml` references local hooks in `/home/av/projects/private/git-hooks` (gitleaks, vault check).
|
||||||
|
- Formatting: `.editorconfig` enforces LF, trailing newline, 4-space indent; YAML/Jinja use 2-space indent.
|
||||||
|
|
||||||
|
## Setup
|
||||||
|
- Copy vault password sample: `cp ansible-vault-password-file.dist ansible-vault-password-file` (needed for ansible and CI).
|
||||||
|
- Install galaxy roles: `ansible-galaxy role install --role-file requirements.yml --force` (or `task install-roles`).
|
||||||
|
- Ensure `yq`, `task`, `ansible` installed per README requirements.
|
||||||
|
|
||||||
|
## Tasks (taskfile)
|
||||||
|
- `task install-roles` — install galaxy roles into `galaxy.roles/`.
|
||||||
|
- `task ssh` — SSH to target using inventory (`production.yml`).
|
||||||
|
- `task btop` — run `btop` on remote.
|
||||||
|
- `task encrypt|decrypt -- <files>` — ansible-vault helpers.
|
||||||
|
- Authelia helpers:
|
||||||
|
- `task authelia-cli -- <args>` — run authelia CLI in docker.
|
||||||
|
- `task authelia-validate-config` — render `files/authelia/configuration.template.yml` with secrets and validate via authelia docker image.
|
||||||
|
- `task authelia-gen-random-string LEN=64` — generate random string.
|
||||||
|
- `task authelia-gen-secret-and-hash LEN=72` — generate hashed secret.
|
||||||
|
- `task format-py-files` — run Black via docker (pyfound/black).
|
||||||
|
|
||||||
|
## Ansible Usage
|
||||||
|
- Inventory: `production.yml` with `server` host. `ansible.cfg` points to `./ansible-vault-password-file` and `./galaxy.roles` for roles path.
|
||||||
|
- Typical deploy example (from README): `ansible-playbook -i production.yml --diff playbook-gitea.yml`.
|
||||||
|
- Per-app playbooks: `playbook-<app>.yml`; grouped runs: `playbook-all-setup.yml`, `playbook-all-applications.yml`, `playbook-upgrade.yml`, etc.
|
||||||
|
- Secrets: encrypted `vars/secrets.yml`; additional `files/<app>/secrets.yml` used for templating (e.g., Authelia). Respect `.crushignore` ignoring vault files.
|
||||||
|
- Templates: many `docker-compose.template.yml` and `*.template.sh` files under `files/*` plus shared `templates/env.j2`. Use `vars/*.yml` to supply values.
|
||||||
|
- Custom roles:
|
||||||
|
- `roles/eget`: installs `eget` tool; see defaults/vars for version/source.
|
||||||
|
- `roles/owner`: manages user/group and env template.
|
||||||
|
- `roles/secrets`: manages vault-related items.
|
||||||
|
|
||||||
|
## Linting & CI
|
||||||
|
- Local lint configs: `.yamllint.yml`, `.ansible-lint.yml` (excludes `.ansible/`, `.gitea/`, `galaxy.roles/`, `Taskfile.yml`).
|
||||||
|
- CI (.gitea/workflows/lint.yml) installs `yamllint` and `ansible-lint` and runs `yamllint .` then `ansible-lint .`; creates dummy vault file if missing.
|
||||||
|
- Pre-commit via lefthook (local hooks path): runs `gitleaks git --staged` and secret-file vault check script.
|
||||||
|
|
||||||
|
## Coding/Templating Conventions
|
||||||
|
- Indentation: 2 spaces for YAML/Jinja (`.editorconfig`), 4 spaces default elsewhere.
|
||||||
|
- End-of-line: LF; ensure final newline.
|
||||||
|
- Template suffixes `.template.yml`, `.yml.j2`, `.template.sh` are rendered via Ansible `template` module.
|
||||||
|
- Avoid committing real secrets; `.crushignore` excludes `ansible-vault-password-file` and `*secrets.yml`.
|
||||||
|
- Service directories under `files/` hold docker-compose and backup templates; ensure per-app users and registry settings align with `vars/*.yml`.
|
||||||
|
|
||||||
|
## Testing/Validation
|
||||||
|
- YAML lint: `yamllint .` (CI default).
|
||||||
|
- Ansible lint: `ansible-lint .` (CI default).
|
||||||
|
- Authelia config validation: `task authelia-validate-config` (renders with secrets then validates via docker).
|
||||||
|
- Black formatting for Python helpers: `task format-py-files`.
|
||||||
|
- Python types validation with mypy: `mypy <file.py>`.
|
||||||
|
|
||||||
|
## Operational Notes
|
||||||
|
- Deployments rely on `production.yml` inventory and per-app playbooks; run with `--diff` for visibility.
|
||||||
|
- Yandex Docker Registry auth helper: `files/yandex-docker-registry-auth.sh`.
|
||||||
|
- Backups: templates and scripts under `files/backups/` per service; `backup-all.py` orchestrates.
|
||||||
|
- Home network/DNS reference in README (Yandex domains).
|
||||||
|
- Ensure `ansible-vault-password-file` present for vault operations and CI.
|
||||||
@@ -4,18 +4,28 @@ Backup script for all applications
|
|||||||
Automatically discovers and runs backup scripts for all users,
|
Automatically discovers and runs backup scripts for all users,
|
||||||
then creates restic backups and sends notifications.
|
then creates restic backups and sends notifications.
|
||||||
"""
|
"""
|
||||||
|
import itertools
|
||||||
import os
|
import os
|
||||||
import sys
|
import sys
|
||||||
import subprocess
|
import subprocess
|
||||||
import logging
|
import logging
|
||||||
import pwd
|
import pwd
|
||||||
|
from abc import ABC
|
||||||
|
from dataclasses import dataclass
|
||||||
from pathlib import Path
|
from pathlib import Path
|
||||||
from typing import List, Tuple, Optional
|
from typing import Dict, List, Optional, Any
|
||||||
import requests
|
import requests
|
||||||
import configparser
|
import tomllib
|
||||||
import itertools
|
|
||||||
|
|
||||||
|
# Default config path
|
||||||
|
CONFIG_PATH = Path("/etc/backup/config.toml")
|
||||||
|
|
||||||
|
# File name to store directories and files to back up
|
||||||
|
BACKUP_TARGETS_FILE = "backup-targets"
|
||||||
|
|
||||||
|
# Default directory fo backups (relative to app dir)
|
||||||
|
# Used when backup-targets file not exists
|
||||||
|
BACKUP_DEFAULT_DIR = "backups"
|
||||||
|
|
||||||
# Configure logging
|
# Configure logging
|
||||||
logging.basicConfig(
|
logging.basicConfig(
|
||||||
@@ -28,44 +38,193 @@ logging.basicConfig(
|
|||||||
)
|
)
|
||||||
logger = logging.getLogger(__name__)
|
logger = logging.getLogger(__name__)
|
||||||
|
|
||||||
config = configparser.ConfigParser()
|
|
||||||
config.read("/etc/backup/config.ini")
|
|
||||||
|
|
||||||
RESTIC_REPOSITORY = config.get("restic", "RESTIC_REPOSITORY")
|
@dataclass
|
||||||
RESTIC_PASSWORD = config.get("restic", "RESTIC_PASSWORD")
|
class Config:
|
||||||
AWS_ACCESS_KEY_ID = config.get("restic", "AWS_ACCESS_KEY_ID")
|
host_name: str
|
||||||
AWS_SECRET_ACCESS_KEY = config.get("restic", "AWS_SECRET_ACCESS_KEY")
|
roots: List[Path]
|
||||||
AWS_DEFAULT_REGION = config.get("restic", "AWS_DEFAULT_REGION")
|
|
||||||
TELEGRAM_BOT_TOKEN = config.get("telegram", "TELEGRAM_BOT_TOKEN")
|
|
||||||
TELEGRAM_CHAT_ID = config.get("telegram", "TELEGRAM_CHAT_ID")
|
@dataclass
|
||||||
NOTIFICATIONS_NAME = config.get("telegram", "NOTIFICATIONS_NAME")
|
class Application:
|
||||||
|
path: Path
|
||||||
|
owner: str
|
||||||
|
|
||||||
|
|
||||||
|
class Storage(ABC):
|
||||||
|
def backup(self, backup_dirs: List[str]) -> bool:
|
||||||
|
"""Backup directories"""
|
||||||
|
raise NotImplementedError()
|
||||||
|
|
||||||
|
|
||||||
|
class ResticStorage(Storage):
|
||||||
|
TYPE_NAME = "restic"
|
||||||
|
|
||||||
|
def __init__(self, name: str, params: Dict[str, Any]):
|
||||||
|
self.name = name
|
||||||
|
self.restic_repository = str(params.get("restic_repository", ""))
|
||||||
|
self.restic_password = str(params.get("restic_password", ""))
|
||||||
|
self.aws_access_key_id = str(params.get("aws_access_key_id", ""))
|
||||||
|
self.aws_secret_access_key = str(params.get("aws_secret_access_key", ""))
|
||||||
|
self.aws_default_region = str(params.get("aws_default_region", ""))
|
||||||
|
|
||||||
|
if not all(
|
||||||
|
[
|
||||||
|
self.restic_repository,
|
||||||
|
self.restic_password,
|
||||||
|
self.aws_access_key_id,
|
||||||
|
self.aws_secret_access_key,
|
||||||
|
self.aws_default_region,
|
||||||
|
]
|
||||||
|
):
|
||||||
|
raise ValueError(
|
||||||
|
f"Missing storage configuration values for backend ResticStorage: '{self.name}'"
|
||||||
|
)
|
||||||
|
|
||||||
|
def backup(self, backup_dirs: List[str]) -> bool:
|
||||||
|
if not backup_dirs:
|
||||||
|
logger.warning("No backup directories found")
|
||||||
|
return True
|
||||||
|
try:
|
||||||
|
return self.__backup_internal(backup_dirs)
|
||||||
|
except Exception as exc: # noqa: BLE001
|
||||||
|
logger.error("Restic backup process failed: %s", exc)
|
||||||
|
return False
|
||||||
|
|
||||||
|
def __backup_internal(self, backup_dirs: List[str]) -> bool:
|
||||||
|
logger.info("Starting restic backup")
|
||||||
|
logger.info("Destination: %s", self.restic_repository)
|
||||||
|
|
||||||
|
env = os.environ.copy()
|
||||||
|
env.update(
|
||||||
|
{
|
||||||
|
"RESTIC_REPOSITORY": self.restic_repository,
|
||||||
|
"RESTIC_PASSWORD": self.restic_password,
|
||||||
|
"AWS_ACCESS_KEY_ID": self.aws_access_key_id,
|
||||||
|
"AWS_SECRET_ACCESS_KEY": self.aws_secret_access_key,
|
||||||
|
"AWS_DEFAULT_REGION": self.aws_default_region,
|
||||||
|
}
|
||||||
|
)
|
||||||
|
|
||||||
|
backup_cmd = ["restic", "backup", "--verbose"] + backup_dirs
|
||||||
|
result = subprocess.run(backup_cmd, env=env, capture_output=True, text=True)
|
||||||
|
|
||||||
|
if result.returncode != 0:
|
||||||
|
logger.error("Restic backup failed: %s", result.stderr)
|
||||||
|
return False
|
||||||
|
|
||||||
|
logger.info("Restic backup completed successfully")
|
||||||
|
|
||||||
|
check_cmd = ["restic", "check"]
|
||||||
|
result = subprocess.run(check_cmd, env=env, capture_output=True, text=True)
|
||||||
|
|
||||||
|
if result.returncode != 0:
|
||||||
|
logger.error("Restic check failed: %s", result.stderr)
|
||||||
|
return False
|
||||||
|
|
||||||
|
logger.info("Restic check completed successfully")
|
||||||
|
|
||||||
|
forget_cmd = [
|
||||||
|
"restic",
|
||||||
|
"forget",
|
||||||
|
"--compact",
|
||||||
|
"--prune",
|
||||||
|
"--keep-daily",
|
||||||
|
"90",
|
||||||
|
"--keep-monthly",
|
||||||
|
"36",
|
||||||
|
]
|
||||||
|
result = subprocess.run(forget_cmd, env=env, capture_output=True, text=True)
|
||||||
|
|
||||||
|
if result.returncode != 0:
|
||||||
|
logger.error("Restic forget/prune failed: %s", result.stderr)
|
||||||
|
return False
|
||||||
|
|
||||||
|
logger.info("Restic forget/prune completed successfully")
|
||||||
|
|
||||||
|
result = subprocess.run(check_cmd, env=env, capture_output=True, text=True)
|
||||||
|
|
||||||
|
if result.returncode != 0:
|
||||||
|
logger.error("Final restic check failed: %s", result.stderr)
|
||||||
|
return False
|
||||||
|
|
||||||
|
logger.info("Final restic check completed successfully")
|
||||||
|
return True
|
||||||
|
|
||||||
|
|
||||||
|
class Notifier(ABC):
|
||||||
|
def send(self, html_message: str):
|
||||||
|
raise NotImplementedError()
|
||||||
|
|
||||||
|
|
||||||
|
class TelegramNotifier(Notifier):
|
||||||
|
TYPE_NAME = "telegram"
|
||||||
|
|
||||||
|
def __init__(self, name: str, params: Dict[str, Any]):
|
||||||
|
self.name = name
|
||||||
|
self.telegram_bot_token = str(params.get("telegram_bot_token", ""))
|
||||||
|
self.telegram_chat_id = str(params.get("telegram_chat_id", ""))
|
||||||
|
if not all(
|
||||||
|
[
|
||||||
|
self.telegram_bot_token,
|
||||||
|
self.telegram_chat_id,
|
||||||
|
]
|
||||||
|
):
|
||||||
|
raise ValueError(
|
||||||
|
f"Missing notification configuration values for backend {name}"
|
||||||
|
)
|
||||||
|
|
||||||
|
def send(self, html_message: str):
|
||||||
|
url = f"https://api.telegram.org/bot{self.telegram_bot_token}/sendMessage"
|
||||||
|
data = {
|
||||||
|
"chat_id": self.telegram_chat_id,
|
||||||
|
"parse_mode": "HTML",
|
||||||
|
"text": html_message,
|
||||||
|
}
|
||||||
|
|
||||||
|
response = requests.post(url, data=data, timeout=30)
|
||||||
|
|
||||||
|
if response.status_code == 200:
|
||||||
|
logger.info("Telegram notification sent successfully")
|
||||||
|
else:
|
||||||
|
logger.error(
|
||||||
|
f"Failed to send Telegram notification: {response.status_code} - {response.text}"
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
class BackupManager:
|
class BackupManager:
|
||||||
def __init__(self):
|
def __init__(
|
||||||
self.errors = []
|
self,
|
||||||
self.warnings = []
|
config: Config,
|
||||||
self.successful_backups = []
|
roots: List[Path],
|
||||||
|
storages: List[Storage],
|
||||||
|
notifiers: List[Notifier],
|
||||||
|
):
|
||||||
|
self.errors: List[str] = []
|
||||||
|
self.warnings: List[str] = []
|
||||||
|
self.successful_backups: List[str] = []
|
||||||
|
self.config = config
|
||||||
|
self.roots: List[Path] = roots
|
||||||
|
self.storages = storages
|
||||||
|
self.notifiers = notifiers
|
||||||
|
|
||||||
def get_application_directories(self) -> List[Tuple[str, str]]:
|
def find_applications(self) -> List[Application]:
|
||||||
"""Get all home directories and their owners"""
|
"""Get all application directories and their owners."""
|
||||||
app_dirs = []
|
applications: List[Application] = []
|
||||||
applications_path = Path("/mnt/applications")
|
source_dirs = itertools.chain(*(root.iterdir() for root in self.roots))
|
||||||
source_dirs = applications_path.iterdir()
|
|
||||||
|
|
||||||
for app_dir in source_dirs:
|
for app_dir in source_dirs:
|
||||||
if app_dir == "lost+found":
|
if "lost+found" in str(app_dir):
|
||||||
continue
|
continue
|
||||||
if app_dir.is_dir():
|
if app_dir.is_dir():
|
||||||
try:
|
try:
|
||||||
# Get the owner of the directory
|
|
||||||
stat_info = app_dir.stat()
|
stat_info = app_dir.stat()
|
||||||
owner = pwd.getpwuid(stat_info.st_uid).pw_name
|
owner = pwd.getpwuid(stat_info.st_uid).pw_name
|
||||||
app_dirs.append((str(app_dir), owner))
|
applications.append(Application(path=app_dir, owner=owner))
|
||||||
except (KeyError, OSError) as e:
|
except (KeyError, OSError) as e:
|
||||||
logger.warning(f"Could not get owner for {app_dir}: {e}")
|
logger.warning(f"Could not get owner for {app_dir}: {e}")
|
||||||
|
|
||||||
return app_dirs
|
return applications
|
||||||
|
|
||||||
def find_backup_script(self, app_dir: str) -> Optional[str]:
|
def find_backup_script(self, app_dir: str) -> Optional[str]:
|
||||||
"""Find backup script in user's home directory"""
|
"""Find backup script in user's home directory"""
|
||||||
@@ -126,113 +285,72 @@ class BackupManager:
|
|||||||
return False
|
return False
|
||||||
|
|
||||||
def get_backup_directories(self) -> List[str]:
|
def get_backup_directories(self) -> List[str]:
|
||||||
"""Get all backup directories that exist"""
|
"""Collect backup targets according to backup-targets rules"""
|
||||||
backup_dirs = []
|
backup_dirs: List[str] = []
|
||||||
app_dirs = self.get_application_directories()
|
applications = self.find_applications()
|
||||||
|
|
||||||
for app_dir, _ in app_dirs:
|
def parse_targets_file(targets_file: Path) -> List[str]:
|
||||||
backup_path = os.path.join(app_dir, "backups")
|
"""Parse backup-targets file, skipping comments and empty lines."""
|
||||||
if os.path.exists(backup_path) and os.path.isdir(backup_path):
|
targets: List[str] = []
|
||||||
backup_dirs.append(backup_path)
|
try:
|
||||||
|
for raw_line in targets_file.read_text(encoding="utf-8").splitlines():
|
||||||
|
line = raw_line.strip()
|
||||||
|
if not line or line.startswith("#"):
|
||||||
|
continue
|
||||||
|
targets.append(line)
|
||||||
|
except OSError as e:
|
||||||
|
warning_msg = f"Could not read backup targets file {targets_file}: {e}"
|
||||||
|
logger.warning(warning_msg)
|
||||||
|
self.warnings.append(warning_msg)
|
||||||
|
return targets
|
||||||
|
|
||||||
|
for app in applications:
|
||||||
|
app_dir = app.path
|
||||||
|
targets_file = app_dir / BACKUP_TARGETS_FILE
|
||||||
|
resolved_targets: List[Path] = []
|
||||||
|
|
||||||
|
if targets_file.exists():
|
||||||
|
# Read custom targets defined by the application.
|
||||||
|
for target_line in parse_targets_file(targets_file):
|
||||||
|
target_path = Path(target_line)
|
||||||
|
if not target_path.is_absolute():
|
||||||
|
target_path = (app_dir / target_path).resolve()
|
||||||
|
else:
|
||||||
|
target_path = target_path.resolve()
|
||||||
|
if target_path.exists():
|
||||||
|
resolved_targets.append(target_path)
|
||||||
|
else:
|
||||||
|
warning_msg = (
|
||||||
|
f"Backup target does not exist for {app_dir}: {target_path}"
|
||||||
|
)
|
||||||
|
logger.warning(warning_msg)
|
||||||
|
self.warnings.append(warning_msg)
|
||||||
|
else:
|
||||||
|
# Fallback to default backups directory when no list is provided.
|
||||||
|
default_target = (app_dir / BACKUP_DEFAULT_DIR).resolve()
|
||||||
|
if default_target.exists():
|
||||||
|
resolved_targets.append(default_target)
|
||||||
|
else:
|
||||||
|
warning_msg = f"Default backup path does not exist for {app_dir}: {default_target}"
|
||||||
|
logger.warning(warning_msg)
|
||||||
|
self.warnings.append(warning_msg)
|
||||||
|
|
||||||
|
for target in resolved_targets:
|
||||||
|
target_str = str(target)
|
||||||
|
if target_str not in backup_dirs:
|
||||||
|
backup_dirs.append(target_str)
|
||||||
|
|
||||||
return backup_dirs
|
return backup_dirs
|
||||||
|
|
||||||
def run_restic_backup(self, backup_dirs: List[str]) -> bool:
|
def send_notification(self, success: bool) -> None:
|
||||||
"""Run restic backup for all backup directories"""
|
"""Send notification to Notifiers"""
|
||||||
if not backup_dirs:
|
|
||||||
logger.warning("No backup directories found")
|
|
||||||
return True
|
|
||||||
|
|
||||||
try:
|
|
||||||
logger.info("Starting restic backup")
|
|
||||||
logger.info("Destination: %s", RESTIC_REPOSITORY)
|
|
||||||
|
|
||||||
# Set environment variables for restic
|
|
||||||
env = os.environ.copy()
|
|
||||||
env.update(
|
|
||||||
{
|
|
||||||
"RESTIC_REPOSITORY": RESTIC_REPOSITORY,
|
|
||||||
"RESTIC_PASSWORD": RESTIC_PASSWORD,
|
|
||||||
"AWS_ACCESS_KEY_ID": AWS_ACCESS_KEY_ID,
|
|
||||||
"AWS_SECRET_ACCESS_KEY": AWS_SECRET_ACCESS_KEY,
|
|
||||||
"AWS_DEFAULT_REGION": AWS_DEFAULT_REGION,
|
|
||||||
}
|
|
||||||
)
|
|
||||||
|
|
||||||
# Run backup
|
|
||||||
backup_cmd = ["restic", "backup", "--verbose"] + backup_dirs
|
|
||||||
result = subprocess.run(backup_cmd, env=env, capture_output=True, text=True)
|
|
||||||
|
|
||||||
if result.returncode != 0:
|
|
||||||
error_msg = f"Restic backup failed: {result.stderr}"
|
|
||||||
logger.error(error_msg)
|
|
||||||
self.errors.append(f"Restic backup: {error_msg}")
|
|
||||||
return False
|
|
||||||
|
|
||||||
logger.info("Restic backup completed successfully")
|
|
||||||
|
|
||||||
# Run check
|
|
||||||
check_cmd = ["restic", "check"]
|
|
||||||
result = subprocess.run(check_cmd, env=env, capture_output=True, text=True)
|
|
||||||
|
|
||||||
if result.returncode != 0:
|
|
||||||
error_msg = f"Restic check failed: {result.stderr}"
|
|
||||||
logger.error(error_msg)
|
|
||||||
self.errors.append(f"Restic check: {error_msg}")
|
|
||||||
return False
|
|
||||||
|
|
||||||
logger.info("Restic check completed successfully")
|
|
||||||
|
|
||||||
# Run forget and prune
|
|
||||||
forget_cmd = [
|
|
||||||
"restic",
|
|
||||||
"forget",
|
|
||||||
"--compact",
|
|
||||||
"--prune",
|
|
||||||
"--keep-daily",
|
|
||||||
"90",
|
|
||||||
"--keep-monthly",
|
|
||||||
"36",
|
|
||||||
]
|
|
||||||
result = subprocess.run(forget_cmd, env=env, capture_output=True, text=True)
|
|
||||||
|
|
||||||
if result.returncode != 0:
|
|
||||||
error_msg = f"Restic forget/prune failed: {result.stderr}"
|
|
||||||
logger.error(error_msg)
|
|
||||||
self.errors.append(f"Restic forget/prune: {error_msg}")
|
|
||||||
return False
|
|
||||||
|
|
||||||
logger.info("Restic forget/prune completed successfully")
|
|
||||||
|
|
||||||
# Final check
|
|
||||||
result = subprocess.run(check_cmd, env=env, capture_output=True, text=True)
|
|
||||||
|
|
||||||
if result.returncode != 0:
|
|
||||||
error_msg = f"Final restic check failed: {result.stderr}"
|
|
||||||
logger.error(error_msg)
|
|
||||||
self.errors.append(f"Final restic check: {error_msg}")
|
|
||||||
return False
|
|
||||||
|
|
||||||
logger.info("Final restic check completed successfully")
|
|
||||||
return True
|
|
||||||
|
|
||||||
except Exception as e:
|
|
||||||
error_msg = f"Restic backup process failed: {str(e)}"
|
|
||||||
logger.error(error_msg)
|
|
||||||
self.errors.append(f"Restic: {error_msg}")
|
|
||||||
return False
|
|
||||||
|
|
||||||
def send_telegram_notification(self, success: bool) -> None:
|
|
||||||
"""Send notification to Telegram"""
|
|
||||||
try:
|
|
||||||
if success and not self.errors:
|
if success and not self.errors:
|
||||||
message = f"<b>{NOTIFICATIONS_NAME}</b>: бекап успешно завершен!"
|
message = f"<b>{self.config.host_name}</b>: бекап успешно завершен!"
|
||||||
if self.successful_backups:
|
if self.successful_backups:
|
||||||
message += (
|
message += f"\n\nУспешные бекапы: {', '.join(self.successful_backups)}"
|
||||||
f"\n\nУспешные бекапы: {', '.join(self.successful_backups)}"
|
|
||||||
)
|
|
||||||
else:
|
else:
|
||||||
message = f"<b>{NOTIFICATIONS_NAME}</b>: бекап завершен с ошибками!"
|
message = f"<b>{self.config.host_name}</b>: бекап завершен с ошибками!"
|
||||||
|
|
||||||
if self.successful_backups:
|
if self.successful_backups:
|
||||||
message += (
|
message += (
|
||||||
@@ -245,31 +363,24 @@ class BackupManager:
|
|||||||
if self.errors:
|
if self.errors:
|
||||||
message += f"\n\n❌ Ошибки:\n" + "\n".join(self.errors)
|
message += f"\n\n❌ Ошибки:\n" + "\n".join(self.errors)
|
||||||
|
|
||||||
url = f"https://api.telegram.org/bot{TELEGRAM_BOT_TOKEN}/sendMessage"
|
for notificator in self.notifiers:
|
||||||
data = {"chat_id": TELEGRAM_CHAT_ID, "parse_mode": "HTML", "text": message}
|
try:
|
||||||
|
notificator.send(message)
|
||||||
response = requests.post(url, data=data, timeout=30)
|
|
||||||
|
|
||||||
if response.status_code == 200:
|
|
||||||
logger.info("Telegram notification sent successfully")
|
|
||||||
else:
|
|
||||||
logger.error(
|
|
||||||
f"Failed to send Telegram notification: {response.status_code} - {response.text}"
|
|
||||||
)
|
|
||||||
|
|
||||||
except Exception as e:
|
except Exception as e:
|
||||||
logger.error(f"Failed to send Telegram notification: {str(e)}")
|
logger.error(f"Failed to send notification: {str(e)}")
|
||||||
|
|
||||||
def run_backup_process(self) -> bool:
|
def run_backup_process(self) -> bool:
|
||||||
"""Main backup process"""
|
"""Main backup process"""
|
||||||
logger.info("Starting backup process")
|
logger.info("Starting backup process")
|
||||||
|
|
||||||
# Get all home directories
|
# Get all home directories
|
||||||
app_dirs = self.get_application_directories()
|
applications = self.find_applications()
|
||||||
logger.info(f"Found {len(app_dirs)} application directories")
|
logger.info(f"Found {len(applications)} application directories")
|
||||||
|
|
||||||
# Process each user's backup
|
# Process each user's backup
|
||||||
for app_dir, username in app_dirs:
|
for app in applications:
|
||||||
|
app_dir = str(app.path)
|
||||||
|
username = app.owner
|
||||||
logger.info(f"Processing backup for app: {app_dir} (user {username})")
|
logger.info(f"Processing backup for app: {app_dir} (user {username})")
|
||||||
|
|
||||||
# Find backup script
|
# Find backup script
|
||||||
@@ -289,14 +400,18 @@ class BackupManager:
|
|||||||
backup_dirs = self.get_backup_directories()
|
backup_dirs = self.get_backup_directories()
|
||||||
logger.info(f"Found backup directories: {backup_dirs}")
|
logger.info(f"Found backup directories: {backup_dirs}")
|
||||||
|
|
||||||
# Run restic backup
|
overall_success = True
|
||||||
restic_success = self.run_restic_backup(backup_dirs)
|
|
||||||
|
for storage in self.storages:
|
||||||
|
backup_result = storage.backup(backup_dirs)
|
||||||
|
if not backup_result:
|
||||||
|
self.errors.append("Restic backup failed")
|
||||||
|
|
||||||
# Determine overall success
|
# Determine overall success
|
||||||
overall_success = restic_success and len(self.errors) == 0
|
overall_success = overall_success and backup_result
|
||||||
|
|
||||||
# Send notification
|
# Send notification
|
||||||
self.send_telegram_notification(overall_success)
|
self.send_notification(overall_success)
|
||||||
|
|
||||||
logger.info("Backup process completed")
|
logger.info("Backup process completed")
|
||||||
|
|
||||||
@@ -311,9 +426,53 @@ class BackupManager:
|
|||||||
return True
|
return True
|
||||||
|
|
||||||
|
|
||||||
|
def initialize(config_path: Path) -> BackupManager:
|
||||||
|
try:
|
||||||
|
with config_path.open("rb") as config_file:
|
||||||
|
raw_config = tomllib.load(config_file)
|
||||||
|
except OSError as e:
|
||||||
|
logger.error(f"Failed to read config file {config_path}: {e}")
|
||||||
|
raise
|
||||||
|
|
||||||
|
host_name = str(raw_config.get("host_name", "unknown"))
|
||||||
|
|
||||||
|
roots_raw = raw_config.get("roots") or []
|
||||||
|
if not isinstance(roots_raw, list) or not roots_raw:
|
||||||
|
raise ValueError("roots must be a non-empty list of paths in config.toml")
|
||||||
|
roots = [Path(root) for root in roots_raw]
|
||||||
|
|
||||||
|
storage_raw = raw_config.get("storage") or {}
|
||||||
|
storages: List[Storage] = []
|
||||||
|
for name, params in storage_raw.items():
|
||||||
|
if not isinstance(params, dict):
|
||||||
|
raise ValueError(f"Storage config for {name} must be a table")
|
||||||
|
storage_type = params.get("type", "")
|
||||||
|
if storage_type == ResticStorage.TYPE_NAME:
|
||||||
|
storages.append(ResticStorage(name, params))
|
||||||
|
if not storages:
|
||||||
|
raise ValueError("At least one storage backend must be configured")
|
||||||
|
|
||||||
|
notifications_raw = raw_config.get("notifier") or {}
|
||||||
|
notifiers: List[Notifier] = []
|
||||||
|
for name, params in notifications_raw.items():
|
||||||
|
if not isinstance(params, dict):
|
||||||
|
raise ValueError(f"Notificator config for {name} must be a table")
|
||||||
|
notifier_type = params.get("type", "")
|
||||||
|
if notifier_type == TelegramNotifier.TYPE_NAME:
|
||||||
|
notifiers.append(TelegramNotifier(name, params))
|
||||||
|
if not notifiers:
|
||||||
|
raise ValueError("At least one notification backend must be configured")
|
||||||
|
|
||||||
|
config = Config(host_name=host_name, roots=roots)
|
||||||
|
|
||||||
|
return BackupManager(
|
||||||
|
config=config, roots=roots, storages=storages, notifiers=notifiers
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
def main():
|
def main():
|
||||||
try:
|
try:
|
||||||
backup_manager = BackupManager()
|
backup_manager = initialize(CONFIG_PATH)
|
||||||
success = backup_manager.run_backup_process()
|
success = backup_manager.run_backup_process()
|
||||||
if not success:
|
if not success:
|
||||||
sys.exit(1)
|
sys.exit(1)
|
||||||
|
|||||||
@@ -1,11 +0,0 @@
|
|||||||
[restic]
|
|
||||||
RESTIC_REPOSITORY={{ restic_repository }}
|
|
||||||
RESTIC_PASSWORD={{ restic_password }}
|
|
||||||
AWS_ACCESS_KEY_ID={{ restic_s3_access_key }}
|
|
||||||
AWS_SECRET_ACCESS_KEY={{ restic_s3_access_secret }}
|
|
||||||
AWS_DEFAULT_REGION={{ restic_s3_region }}
|
|
||||||
|
|
||||||
[telegram]
|
|
||||||
TELEGRAM_BOT_TOKEN={{ notifications_tg_bot_token }}
|
|
||||||
TELEGRAM_CHAT_ID={{ notifications_tg_chat_id }}
|
|
||||||
NOTIFICATIONS_NAME={{ notifications_name }}
|
|
||||||
18
files/backups/config.template.toml
Normal file
18
files/backups/config.template.toml
Normal file
@@ -0,0 +1,18 @@
|
|||||||
|
host_name = "{{ notifications_name }}"
|
||||||
|
|
||||||
|
roots = [
|
||||||
|
"{{ application_dir }}"
|
||||||
|
]
|
||||||
|
|
||||||
|
[storage.yandex_cloud_s3]
|
||||||
|
type = "restic"
|
||||||
|
restic_repository = "{{ restic_repository }}"
|
||||||
|
restic_password = "{{ restic_password }}"
|
||||||
|
aws_access_key_id = "{{ restic_s3_access_key }}"
|
||||||
|
aws_secret_access_key = "{{ restic_s3_access_secret }}"
|
||||||
|
aws_default_region = "{{ restic_s3_region }}"
|
||||||
|
|
||||||
|
[notifier.server_notifications_channel]
|
||||||
|
type = "telegram"
|
||||||
|
telegram_bot_token = "{{ notifications_tg_bot_token }}"
|
||||||
|
telegram_chat_id = "{{ notifications_tg_chat_id }}"
|
||||||
@@ -1,7 +1,7 @@
|
|||||||
services:
|
services:
|
||||||
|
|
||||||
gitea_app:
|
gitea_app:
|
||||||
image: gitea/gitea:1.25.2
|
image: gitea/gitea:1.25.3
|
||||||
restart: unless-stopped
|
restart: unless-stopped
|
||||||
container_name: gitea_app
|
container_name: gitea_app
|
||||||
ports:
|
ports:
|
||||||
|
|||||||
@@ -16,9 +16,9 @@ services:
|
|||||||
- "{{ (data_dir, 'gramps_users') | path_join }}:/app/users" # persist user database
|
- "{{ (data_dir, 'gramps_users') | path_join }}:/app/users" # persist user database
|
||||||
- "{{ (data_dir, 'gramps_index') | path_join }}:/app/indexdir" # persist search index
|
- "{{ (data_dir, 'gramps_index') | path_join }}:/app/indexdir" # persist search index
|
||||||
- "{{ (data_dir, 'gramps_secret') | path_join }}:/app/secret" # persist flask secret
|
- "{{ (data_dir, 'gramps_secret') | path_join }}:/app/secret" # persist flask secret
|
||||||
- "{{ (data_dir, 'gramps_media') | path_join }}:/app/media" # persist media files
|
|
||||||
- "{{ (cache_dir, 'gramps_thumb_cache') | path_join }}:/app/thumbnail_cache" # persist thumbnails
|
- "{{ (cache_dir, 'gramps_thumb_cache') | path_join }}:/app/thumbnail_cache" # persist thumbnails
|
||||||
- "{{ (cache_dir, 'gramps_cache') | path_join }}:/app/cache" # persist export and report caches
|
- "{{ (cache_dir, 'gramps_cache') | path_join }}:/app/cache" # persist export and report caches
|
||||||
|
- "{{ media_dir }}:/app/media" # persist media files
|
||||||
environment:
|
environment:
|
||||||
GRAMPSWEB_TREE: "Gramps" # will create a new tree if not exists
|
GRAMPSWEB_TREE: "Gramps" # will create a new tree if not exists
|
||||||
GRAMPSWEB_SECRET_KEY: "{{ gramps_secret_key }}"
|
GRAMPSWEB_SECRET_KEY: "{{ gramps_secret_key }}"
|
||||||
@@ -37,12 +37,8 @@ services:
|
|||||||
GRAMPSWEB_EMAIL_USE_TLS: "false"
|
GRAMPSWEB_EMAIL_USE_TLS: "false"
|
||||||
GRAMPSWEB_DEFAULT_FROM_EMAIL: "gramps@vakhrushev.me"
|
GRAMPSWEB_DEFAULT_FROM_EMAIL: "gramps@vakhrushev.me"
|
||||||
|
|
||||||
# media storage at s3
|
# media storage
|
||||||
GRAMPSWEB_MEDIA_BASE_DIR: "s3://av-gramps-media-storage"
|
GRAMPSWEB_MEDIA_BASE_DIR: "/app/media"
|
||||||
AWS_ENDPOINT_URL: "{{ gramps_s3_endpoint }}"
|
|
||||||
AWS_ACCESS_KEY_ID: "{{ gramps_s3_access_key_id }}"
|
|
||||||
AWS_SECRET_ACCESS_KEY: "{{ gramps_s3_secret_access_key }}"
|
|
||||||
AWS_DEFAULT_REGION: "{{ gramps_s3_region }}"
|
|
||||||
|
|
||||||
gramps_celery:
|
gramps_celery:
|
||||||
<<: *gramps_app # YAML merge key copying the entire grampsweb service config
|
<<: *gramps_app # YAML merge key copying the entire grampsweb service config
|
||||||
|
|||||||
@@ -23,6 +23,3 @@ models:
|
|||||||
undo:
|
undo:
|
||||||
type: sqlite
|
type: sqlite
|
||||||
path: "{{ (data_dir, 'gramps_db/59a0f3d6-1c3d-4410-8c1d-1c9c6689659f/undo.db') | path_join }}"
|
path: "{{ (data_dir, 'gramps_db/59a0f3d6-1c3d-4410-8c1d-1c9c6689659f/undo.db') | path_join }}"
|
||||||
archive:
|
|
||||||
includes:
|
|
||||||
- "{{ data_dir }}"
|
|
||||||
|
|||||||
65
files/gramps/gramps_rename.py
Executable file
65
files/gramps/gramps_rename.py
Executable file
@@ -0,0 +1,65 @@
|
|||||||
|
#!/usr/bin/env python3.12
|
||||||
|
|
||||||
|
import argparse
|
||||||
|
import sys
|
||||||
|
from pathlib import Path
|
||||||
|
|
||||||
|
|
||||||
|
def parse_args() -> argparse.Namespace:
|
||||||
|
parser = argparse.ArgumentParser(
|
||||||
|
description="Rename Gramps document files by appending extensions from a list."
|
||||||
|
)
|
||||||
|
parser.add_argument("directory", type=Path, help="Directory containing hashed files")
|
||||||
|
parser.add_argument("names_file", type=Path, help="Text file with target names")
|
||||||
|
return parser.parse_args()
|
||||||
|
|
||||||
|
|
||||||
|
def read_names(path: Path) -> list[str]:
|
||||||
|
if not path.is_file():
|
||||||
|
raise FileNotFoundError(f"Names file not found: {path}")
|
||||||
|
|
||||||
|
names = []
|
||||||
|
for line in path.read_text(encoding="utf-8").splitlines():
|
||||||
|
name = line.strip()
|
||||||
|
if name:
|
||||||
|
names.append(name)
|
||||||
|
return names
|
||||||
|
|
||||||
|
|
||||||
|
def rename_files(directory: Path, names: list[str]) -> None:
|
||||||
|
if not directory.is_dir():
|
||||||
|
raise NotADirectoryError(f"Directory not found: {directory}")
|
||||||
|
|
||||||
|
for name in names:
|
||||||
|
hash_part, dot, _ = name.partition(".")
|
||||||
|
if not dot:
|
||||||
|
print(f"Skipping invalid entry (missing extension): {name}", file=sys.stderr)
|
||||||
|
continue
|
||||||
|
|
||||||
|
source = directory / hash_part
|
||||||
|
target = directory / name
|
||||||
|
|
||||||
|
if target.exists():
|
||||||
|
print(f"Target already exists, skipping: {target}", file=sys.stderr)
|
||||||
|
continue
|
||||||
|
|
||||||
|
if not source.exists():
|
||||||
|
print(f"Source not found, skipping: {source}", file=sys.stderr)
|
||||||
|
continue
|
||||||
|
|
||||||
|
source.rename(target)
|
||||||
|
print(f"Renamed {source.name} -> {target.name}")
|
||||||
|
|
||||||
|
|
||||||
|
def main() -> None:
|
||||||
|
args = parse_args()
|
||||||
|
try:
|
||||||
|
names = read_names(args.names_file)
|
||||||
|
rename_files(args.directory, names)
|
||||||
|
except Exception as exc: # noqa: BLE001
|
||||||
|
print(str(exc), file=sys.stderr)
|
||||||
|
sys.exit(1)
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
main()
|
||||||
@@ -2,7 +2,7 @@
|
|||||||
|
|
||||||
models:
|
models:
|
||||||
|
|
||||||
gramps:
|
memos:
|
||||||
compress_with:
|
compress_with:
|
||||||
type: 'tgz'
|
type: 'tgz'
|
||||||
storages:
|
storages:
|
||||||
@@ -14,8 +14,3 @@ models:
|
|||||||
users:
|
users:
|
||||||
type: sqlite
|
type: sqlite
|
||||||
path: "{{ (data_dir, 'memos_prod.db') | path_join }}"
|
path: "{{ (data_dir, 'memos_prod.db') | path_join }}"
|
||||||
archive:
|
|
||||||
includes:
|
|
||||||
- "{{ data_dir }}"
|
|
||||||
excludes:
|
|
||||||
- "{{ (data_dir, '.thumbnail_cache') | path_join }}"
|
|
||||||
|
|||||||
@@ -1,7 +1,7 @@
|
|||||||
services:
|
services:
|
||||||
|
|
||||||
netdata:
|
netdata:
|
||||||
image: netdata/netdata:v2.8.2
|
image: netdata/netdata:v2.8.4
|
||||||
container_name: netdata
|
container_name: netdata
|
||||||
restart: unless-stopped
|
restart: unless-stopped
|
||||||
cap_add:
|
cap_add:
|
||||||
|
|||||||
11
lefthook.yml
11
lefthook.yml
@@ -1,6 +1,8 @@
|
|||||||
# Refer for explanation to following link:
|
# Refer for explanation to following link:
|
||||||
# https://lefthook.dev/configuration/
|
# https://lefthook.dev/configuration/
|
||||||
|
|
||||||
|
glob_matcher: doublestar
|
||||||
|
|
||||||
templates:
|
templates:
|
||||||
av-hooks-dir: "/home/av/projects/private/git-hooks"
|
av-hooks-dir: "/home/av/projects/private/git-hooks"
|
||||||
|
|
||||||
@@ -12,3 +14,12 @@ pre-commit:
|
|||||||
|
|
||||||
- name: "check secret files"
|
- name: "check secret files"
|
||||||
run: "python3 {av-hooks-dir}/pre-commit/check-secrets-encrypted-with-ansible-vault.py"
|
run: "python3 {av-hooks-dir}/pre-commit/check-secrets-encrypted-with-ansible-vault.py"
|
||||||
|
|
||||||
|
- name: "format python"
|
||||||
|
glob: "**/*.py"
|
||||||
|
run: "black --quiet {staged_files}"
|
||||||
|
stage_fixed: true
|
||||||
|
|
||||||
|
- name: "mypy"
|
||||||
|
glob: "**/*.py"
|
||||||
|
run: "mypy {staged_files}"
|
||||||
|
|||||||
@@ -7,7 +7,7 @@
|
|||||||
|
|
||||||
vars:
|
vars:
|
||||||
backup_config_dir: "/etc/backup"
|
backup_config_dir: "/etc/backup"
|
||||||
backup_config_file: "{{ (backup_config_dir, 'config.ini') | path_join }}"
|
backup_config_file: "{{ (backup_config_dir, 'config.toml') | path_join }}"
|
||||||
|
|
||||||
restic_shell_script: "{{ (bin_prefix, 'restic-shell.sh') | path_join }}"
|
restic_shell_script: "{{ (bin_prefix, 'restic-shell.sh') | path_join }}"
|
||||||
backup_all_script: "{{ (bin_prefix, 'backup-all.py') | path_join }}"
|
backup_all_script: "{{ (bin_prefix, 'backup-all.py') | path_join }}"
|
||||||
@@ -23,7 +23,7 @@
|
|||||||
|
|
||||||
- name: "Create backup config file"
|
- name: "Create backup config file"
|
||||||
ansible.builtin.template:
|
ansible.builtin.template:
|
||||||
src: "files/backups/config.template.ini"
|
src: "files/backups/config.template.toml"
|
||||||
dest: "{{ backup_config_file }}"
|
dest: "{{ backup_config_file }}"
|
||||||
owner: root
|
owner: root
|
||||||
group: root
|
group: root
|
||||||
|
|||||||
@@ -31,3 +31,10 @@
|
|||||||
community.docker.docker_network:
|
community.docker.docker_network:
|
||||||
name: "monitoring_network"
|
name: "monitoring_network"
|
||||||
driver: "bridge"
|
driver: "bridge"
|
||||||
|
|
||||||
|
- name: "Schedule docker image prune"
|
||||||
|
ansible.builtin.cron:
|
||||||
|
name: "docker image prune"
|
||||||
|
minute: "0"
|
||||||
|
hour: "3"
|
||||||
|
job: "/usr/bin/docker image prune -af"
|
||||||
|
|||||||
@@ -12,6 +12,7 @@
|
|||||||
app_owner_gid: 1010
|
app_owner_gid: 1010
|
||||||
base_dir: "{{ (application_dir, app_name) | path_join }}"
|
base_dir: "{{ (application_dir, app_name) | path_join }}"
|
||||||
data_dir: "{{ (base_dir, 'data') | path_join }}"
|
data_dir: "{{ (base_dir, 'data') | path_join }}"
|
||||||
|
media_dir: "{{ (base_dir, 'media') | path_join }}"
|
||||||
cache_dir: "{{ (base_dir, 'cache') | path_join }}"
|
cache_dir: "{{ (base_dir, 'cache') | path_join }}"
|
||||||
backups_dir: "{{ (base_dir, 'backups') | path_join }}"
|
backups_dir: "{{ (base_dir, 'backups') | path_join }}"
|
||||||
gobackup_config: "{{ (base_dir, 'gobackup.yml') | path_join }}"
|
gobackup_config: "{{ (base_dir, 'gobackup.yml') | path_join }}"
|
||||||
@@ -36,6 +37,7 @@
|
|||||||
loop:
|
loop:
|
||||||
- "{{ base_dir }}"
|
- "{{ base_dir }}"
|
||||||
- "{{ data_dir }}"
|
- "{{ data_dir }}"
|
||||||
|
- "{{ media_dir }}"
|
||||||
- "{{ cache_dir }}"
|
- "{{ cache_dir }}"
|
||||||
- "{{ backups_dir }}"
|
- "{{ backups_dir }}"
|
||||||
|
|
||||||
@@ -55,6 +57,27 @@
|
|||||||
group: "{{ app_user }}"
|
group: "{{ app_user }}"
|
||||||
mode: "0750"
|
mode: "0750"
|
||||||
|
|
||||||
|
- name: "Create backup targets file"
|
||||||
|
ansible.builtin.lineinfile:
|
||||||
|
path: "{{ base_dir }}/backup-targets"
|
||||||
|
line: "{{ item }}"
|
||||||
|
create: true
|
||||||
|
owner: "{{ app_user }}"
|
||||||
|
group: "{{ app_user }}"
|
||||||
|
mode: "0750"
|
||||||
|
loop:
|
||||||
|
- "{{ data_dir }}"
|
||||||
|
- "{{ media_dir }}"
|
||||||
|
- "{{ backups_dir }}"
|
||||||
|
|
||||||
|
- name: "Copy rename script"
|
||||||
|
ansible.builtin.copy:
|
||||||
|
src: "files/{{ app_name }}/gramps_rename.py"
|
||||||
|
dest: "{{ base_dir }}/gramps_rename.py"
|
||||||
|
owner: "{{ app_user }}"
|
||||||
|
group: "{{ app_user }}"
|
||||||
|
mode: "0750"
|
||||||
|
|
||||||
- name: "Copy docker compose file"
|
- name: "Copy docker compose file"
|
||||||
ansible.builtin.template:
|
ansible.builtin.template:
|
||||||
src: "./files/{{ app_name }}/docker-compose.template.yml"
|
src: "./files/{{ app_name }}/docker-compose.template.yml"
|
||||||
|
|||||||
@@ -53,6 +53,18 @@
|
|||||||
group: "{{ app_user }}"
|
group: "{{ app_user }}"
|
||||||
mode: "0750"
|
mode: "0750"
|
||||||
|
|
||||||
|
- name: "Create backup targets file"
|
||||||
|
ansible.builtin.lineinfile:
|
||||||
|
path: "{{ base_dir }}/backup-targets"
|
||||||
|
line: "{{ item }}"
|
||||||
|
create: true
|
||||||
|
owner: "{{ app_user }}"
|
||||||
|
group: "{{ app_user }}"
|
||||||
|
mode: "0750"
|
||||||
|
loop:
|
||||||
|
- "{{ data_dir }}"
|
||||||
|
- "{{ backups_dir }}"
|
||||||
|
|
||||||
- name: "Copy docker compose file"
|
- name: "Copy docker compose file"
|
||||||
ansible.builtin.template:
|
ansible.builtin.template:
|
||||||
src: "./files/{{ app_name }}/docker-compose.template.yml"
|
src: "./files/{{ app_name }}/docker-compose.template.yml"
|
||||||
|
|||||||
@@ -51,13 +51,29 @@
|
|||||||
group: "{{ app_user }}"
|
group: "{{ app_user }}"
|
||||||
mode: "0640"
|
mode: "0640"
|
||||||
|
|
||||||
- name: "Copy backup script"
|
# - name: "Copy backup script"
|
||||||
ansible.builtin.template:
|
# ansible.builtin.template:
|
||||||
src: "files/{{ app_name }}/backup.template.sh"
|
# src: "files/{{ app_name }}/backup.template.sh"
|
||||||
|
# dest: "{{ base_dir }}/backup.sh"
|
||||||
|
# owner: "{{ app_user }}"
|
||||||
|
# group: "{{ app_user }}"
|
||||||
|
# mode: "0750"
|
||||||
|
|
||||||
|
- name: "Disable backup script"
|
||||||
|
ansible.builtin.file:
|
||||||
dest: "{{ base_dir }}/backup.sh"
|
dest: "{{ base_dir }}/backup.sh"
|
||||||
|
state: absent
|
||||||
|
|
||||||
|
- name: "Create backup targets file"
|
||||||
|
ansible.builtin.lineinfile:
|
||||||
|
path: "{{ base_dir }}/backup-targets"
|
||||||
|
line: "{{ item }}"
|
||||||
|
create: true
|
||||||
owner: "{{ app_user }}"
|
owner: "{{ app_user }}"
|
||||||
group: "{{ app_user }}"
|
group: "{{ app_user }}"
|
||||||
mode: "0750"
|
mode: "0750"
|
||||||
|
loop:
|
||||||
|
- "{{ data_dir }}"
|
||||||
|
|
||||||
- name: "Copy docker compose file"
|
- name: "Copy docker compose file"
|
||||||
ansible.builtin.template:
|
ansible.builtin.template:
|
||||||
|
|||||||
321
vars/secrets.yml
321
vars/secrets.yml
@@ -1,161 +1,162 @@
|
|||||||
$ANSIBLE_VAULT;1.1;AES256
|
$ANSIBLE_VAULT;1.1;AES256
|
||||||
62373337383866623263326333653134323532613565633430346266396463376439633833656138
|
66333736306664316131646233393639356436323832386638353237393761386631303639396432
|
||||||
3162643934363462303436653264323862353161633963360a633861646264336133613134336336
|
6266616434663638306637356435393564303633613332640a663366336135373061396239653065
|
||||||
39333431643666323730356163626236333430313136626134333539333330646237333339306238
|
32343339313734346261363461633735383538613033656138383835653661633334316533613738
|
||||||
3936353539653730340a613564656330663336336132356431386264666662333864663064396332
|
3730656661393466370a653863353234323739346630323534333334306432646636383664313865
|
||||||
64393163346538396366366333623431333533666563346163613633393635646531383739323435
|
39666133623032393638633239653665376336626638303334626164376663626666393439346661
|
||||||
36636533383563383265383439323539396531623466323435303164643038653734623133343739
|
32363364363066653933313135666566346138353163333639353062336331623938353131646334
|
||||||
34636430643030323031303434346636623764663239653535323035396634613036386230323032
|
32326466323030643364356530653862633332363338333633393730663536353363663165303838
|
||||||
62623131643639393836346664393734653931346630666539656661626566353730333635393261
|
66643331366231613733316139333233393636626162643861346437613062643133353035353566
|
||||||
35386339363732386137643937373730303765643234326433336366376135656637376637323862
|
38323062653135393364376265346361613163346337636634383932666137636133363864643264
|
||||||
35323230333736366238333734386136383534393661383037346235643561386336393438633965
|
39646634363037383662376166633335636365303734646465623833313766656235333835396263
|
||||||
34646535363835383134353232313839663763303037653766346138363531613162616539353531
|
63326237306337396465343065313263663466613336303934373337623364613839323931623934
|
||||||
65663832633239613236626366623733323365303534616463333435393763656162343764323138
|
35353665363339646365353139643032366164653133616562303930306264343633343439373830
|
||||||
37613530636331316665376634623233653935653933613930333130313563313061333435353639
|
66356635343363363361313633353134313837653163346537656465623832633561323833663763
|
||||||
30636361393562653764613065316563666134336333343433336335353138656239326166636533
|
32323537303931343934376165663466356466333533643237653038373866626439356334663539
|
||||||
31386539646636333438643366336666623764656335343066616666343737356266356335373733
|
36613336326633306331323763393230653236306337353139353135316566336661623464316237
|
||||||
31653234643234313462333138353639323362363834353162663064363938353134313232623533
|
37363930666539373837336466333265333866633733363032323862333239643764333766366134
|
||||||
37363732343935623132303664653436313730316534623630326362376233393233373531316162
|
33396330613463633261626334393262343031663131313737623233323535303965333864366631
|
||||||
63326537316266613339376239376561613136633830633837393035363337343034643230303630
|
65333535343062393434396361313433386463613630353765663363633431393831316233313162
|
||||||
61343039333839313164616536306465646262626332383665666139626534613938303739643566
|
39656333306265386437663863343638323262663734623830336562636366333534373735313932
|
||||||
63343731333835666466626136633761366536366230623364356632653837633934303739623466
|
61653739616635643661306165363364316262386434643033646430366235303334326236373262
|
||||||
39396535313963643535353236373338383431356331393232303336623962646366316132343263
|
34396261653934633337646338333364326237663739623965313039303134393238653634616431
|
||||||
34613162323861643632393736383836376161643036323165393837623734386663383436393062
|
39376539323238353033323333643337613935616335336562626361383164616432633331313536
|
||||||
32383130613063643036656365646332363865646662346431393062323961343039343935643965
|
63373633633966373237393131663064613332316536373163303163653638343264323934393365
|
||||||
62343033646632333337613164353263353834623430336334356666323761353964306133393439
|
62356636643434646465313734363936346435663237393064663138643135303237633965393965
|
||||||
61323438346662623435356537623064373832346338613733613061623164623466363561623466
|
31353564616136366237666139646561363364353738633862346262616637663362333137386133
|
||||||
34393430313838623435663935333638323231663138346539653139663437643066663465666231
|
63383662363866306366326232643462376532313632336662666666386562356436616236663430
|
||||||
31303232363863383839623037636333626166326135393439316662306435306130353639386563
|
37636661656237646431616331633833306266313434326461353561643835646236346539323337
|
||||||
65316638376534326363306562353037653765333834663361353864383265643466396265336264
|
30303065313165306233646135363766613931383433313165366662636137326162633363306431
|
||||||
32353865633131373061323637623736353162366131633161643963623432623133336237366164
|
31626564323566313036363666386338366462613164663735363333313830306238323365363737
|
||||||
63393465656464646635376438363563663461626438336339386564316463343230323964633430
|
65333137363631623139333964316464376662643661643038326162643630323938626332633130
|
||||||
37343563656462333638303638356235326635376138626333613636623830363366343435373464
|
62626334666131373731343432303461656537303062396638326134356166663936663364656139
|
||||||
34623465646437663735333036366566303962356266666363366132643764383638386230643635
|
32333864653231636235663165323936626135313838663866373132376437656236363235353533
|
||||||
30353838633863313037616165666265393930326333613761366566633230373761313162363135
|
39346532626666656233343433383434613238653833626436643566653462346330316565366564
|
||||||
64386461343263646331333565396662636133356361616563396563626232313962303862353338
|
33623831643265623966326638656462306161636366303266333734396433653861393363333332
|
||||||
36383238633134383865663434343062636465623631363461303439666565373339616136613336
|
61616463613931393835613463353039646164383435373330626134376339623861396266623430
|
||||||
32663732333263353666323766333338396532643134316362613930396134363430313363616663
|
30316365356337356531363263623362633332313536373333653964356534623861613232653932
|
||||||
34313766303433333936343936386138313939373734316463376531623562633834626561643134
|
35613064336131313632656663303631363664366163343362643365663932356138376235313466
|
||||||
64356530643637313066656363653165396562616363373637373261316633643331653833623137
|
30353333376461623435363931356665306333623736313562333836336563616137633863346635
|
||||||
33626162366636633932343135353961366631633866343564613834353237326132643963353466
|
37356137306361383134663535626130646135363661353438343961653766333538353330346236
|
||||||
65633033616238313532666366313566373261643531333830306232373439323737313330303662
|
64326639383864396336343062663965343964396162386266363639643962666431336237303864
|
||||||
36613164633264366133623566653034363739633435646634383832343138303931626636373037
|
66663861346139363335663362333032613637336266323439366566373136643133383361323061
|
||||||
36326464353336383964643439393864666238623939393134326134613938303466343536613933
|
37633134323933613665366665383962373935646137616139353661336665613661623834303465
|
||||||
63623961656435656363363365633137383563373265636564613861666331623739623532666632
|
30323264333137636261393535376438363134663734313662383533313130623365386335323662
|
||||||
62646432643432393565396135613364653263386562386262396262333430656335313337646539
|
63366139643238613632326165313835363964336237383936333737646239363365623030666364
|
||||||
62353335303836376433616639613531623432626434306162666134656637616562356635363931
|
64656432653164643565376666373262333839353139666561623731343234326637316636333765
|
||||||
31303063313634336335336233353265346563646530376639663032326239393363626136346466
|
63306331343538386433376566326239376232363434343838653864393562383063663566333263
|
||||||
38616630666432363866316137653730623066373263656331313665643439646636653362393130
|
35326333326462316134383139303534343263646530363266663933353834353138303435646339
|
||||||
36666130616435626135306639303766656430316536376431303337316631316462623564666237
|
33643361363062373735663430346636333431363736373463326439353437356530373935633962
|
||||||
34303262663231303437323266643262363434313232346432363030306661333536353732643761
|
63333031376163656565663536366230333731613833396266383465333461386161373337323863
|
||||||
37333962393436323463633762393135326264323239373838613032353938366238393138393332
|
36313032613534346230636566383930656330656133376431376462656536386263343831393862
|
||||||
34643162303335666330613263386438653835396135626166636435306238643463623061646432
|
32363365623061653837303736636664663361663862656562393661623330386435646336373531
|
||||||
30393564643665303764356334363966623939323138336332646437343265343662333836326566
|
35363037356638653831386261646235613337363066343632653632306631633138633235356139
|
||||||
64663464313730353162306534663731376266373062343534633230663635623535626533613636
|
39663936333262643061646330316538373862353030626632396336393030643239316634343730
|
||||||
65313961326665333234336663303236393232373334623532333863313766613162616666393732
|
35363564373865376533333466666439646633313932656665383930623531633038316363636332
|
||||||
33313937326164396639663835653461333033616463373034643830356361656234393330393233
|
37363466643835353132333532646163316636303662646234613038333334626365623964653235
|
||||||
64313934653133636630323331353262646335363666366339303630303761393262363761663530
|
36393739353164306666313537633538383934363330373235353262616165623132613330303735
|
||||||
66373638336336386432613939343835636463303838383239333361616538643538633738333834
|
65323362666263653937376562633833653264613439303236373466646362316462386632373038
|
||||||
66356234383437336234663964613531306165616636616235386363623738616631646164313538
|
35353462633730666430623638626439626364616335643964623933663233376433353233633235
|
||||||
30313034346264373033633431636331336562316538666639653462313438383038373034663036
|
34616233636335323365646538326639383033643832323139633064616635353331376230663738
|
||||||
30643361323230313761633439326236663531316339303765663032653835643735386536333062
|
35316665616230633335366233363332353432633937653335316662306166383337633262633634
|
||||||
63623338613364343137663136666234376264626663323565343762343834633330306463353135
|
35303766653733376362623436383663663437393461643266376530613533383038373035356234
|
||||||
35643330373139653839313739376134323831383538666532633561323365363534633063653136
|
32343962623930336566363164616361386134376362383138653963366339326431653832393664
|
||||||
32303661303831373633306164393533303336373766643463383635313035376261363461363464
|
66633566343666326339323536343738396532623735343034646130386332323466346365383135
|
||||||
62313539646333633838303635326335326330383261326330333533396364393931643138366562
|
33373732363833316138343938633262373066613930343162663163323331373466366631626237
|
||||||
38343961316432396230616337313164626631616330343861306438396230383466356133643161
|
34623062353139353734653366313363323634346564663861613233366537663732363134643638
|
||||||
36353364303739326265306430653036633134343166383430663264643466653430663936353462
|
65623964396665303765343430623636376637666364363835346462613763306135326463353631
|
||||||
32386330626330393734343938623664386430323438333965346536396639613365663461616162
|
33333834333464383437336531616662643461666138383435636530373761646564626438313433
|
||||||
66633534363737366334633239613663356437366234306437363733376237343132353133326661
|
64643037626637343433366631343266373436343864396663353039333231646462333932353436
|
||||||
63633362356361653264363435303030313962323462313137656532306537616330613532663665
|
33643837616639666363353237313137353133346636383231623634326335386537363030613537
|
||||||
64376131343937636138333332353464633831323361623263323766333034386535313136613831
|
33306333363638643533373237633833663333393566656466313832383636663031663433636663
|
||||||
62363662616661643563326465653664393032666162303364643234303362333164653738616636
|
37313034353734343966613934663530643537366562623137373331623632653466333839376331
|
||||||
34613766316565333136663831396166643163333334366461616333376365383530396638313931
|
61623866633937666365633763613138346365393934383163623730373134373531616138363733
|
||||||
35356264623739663935316165376363633338313164613766663137323261303935613230316661
|
65616230303337396561666462653866333438353463316235303331643834653165363033626537
|
||||||
66376231333539386139353537356163396639663965666334643739333434393134313230356131
|
37386636653337363666393163393031333461383331613965303262616530363133373732313362
|
||||||
65393331393836343962343235356138366634633563613838333139666565363831346166396630
|
33343033373963656333363064303035666663646536373764323833653037316231316430333634
|
||||||
30663630376231646665386137326361336132313535336263653338343430613830346363303239
|
65383236663035326636336535636462323438363165633437333337613439326433626466396365
|
||||||
36623539326161666564633435303439313763356536643737386432633363636162623232656137
|
38393362633266663962646565336539333239346133323434646632643537613435643434623631
|
||||||
66346665623433333131323339636538663363303035633866656139393566313037613439366630
|
63636536393936343165393966663438343261333966363639323462663566613437393838323036
|
||||||
32653465656664343130653937373436336236626566386638393136393536356331356637646461
|
33393163323034336533333632646230343138336333353236376136383664646466626666356234
|
||||||
64336265326438616439336432333432656336393866666666323132323165383039323761313230
|
61653466363933333331613539636431393934393235376433326665643263663638306463393837
|
||||||
33386162376563653638656563616232323132643534626537323161623661303937386138343034
|
34396339383536636461366230383938386339303334393038343239363361666565336237326465
|
||||||
31316230646166316263323262623638393336333739326638316264313835653734373232633934
|
63623231663861303436353533663661656431646165646662383065386362636633333631643335
|
||||||
64376330363035343830353337386632373863346438326332643962303061306138383539363061
|
38373464373432626538616234623638303734626237393566326463633765316365653837303433
|
||||||
62386636366430643835336131663139306138653030383963383034326466393263663230356634
|
34376438323439633237313733343836343733373930643138636333366166353666373966323231
|
||||||
63376437306135396137383633376636633464663365303165643131316339313131303836306136
|
63613466306365386137336538613837613264633735393937343166303736396162303230623430
|
||||||
66633163623162636362636138326530373364316234306565323635303166636132383330316662
|
33363233663761333063363134393935333530343133303138373934666236393732396330643438
|
||||||
65353839333434646166346662366565623061396330666166636263393962373031623632623036
|
61343231666363323834346132376135303339313766383365363837313163393636333563376436
|
||||||
30316537616362636634316166396265633165383232383136643235666330313931326138633135
|
64333832393361626663376161343033383763373537396264646239303736333263303739326436
|
||||||
66376662376230646336343432353634383737623835323038613235323761383830663339363432
|
31393665363734323364643032393031313135623563383639316163616535323938343765356264
|
||||||
64613734343662306230396165636366356139303539313537356631343837323864653736336539
|
35616433396236396662393930646431643063323163343366313030383766323733333865616235
|
||||||
64346135376633326634623839643565623339313361626532313566666263333232316562363537
|
65623737663763373363613030326665613333636366393464383139626462346333323162333537
|
||||||
63633732313336336637313931303037626433646233353361633332343333623639386131663962
|
32386635636532346539623534346364623532656435653638643962353836653330316638633661
|
||||||
33363731656538353134363436383066333564336138633462386331626130363937643166303161
|
38363330656161393766383237663737663835646565363966373563663832313263633238316165
|
||||||
62303266666238626562666432383934393435643836333235626531663334343331613766303237
|
64643536336232396436636261303866376132366261373132333631646166393764636130326335
|
||||||
62613936616136663064353762363666303464633562353061393364303563313831636432326237
|
39613037633239626666396262386533343764623233626335373139663937613433356134633330
|
||||||
30356464653063663535333037393364363164636334653136323065663162323835336462396433
|
32656138383563343734616536303265306661386362343939626332623763393136323138653937
|
||||||
39626532383433363835303862613534303535386131343361363664653831656463316437333035
|
39646537366461353364616538663361663465626634356338323166373837323339343061323261
|
||||||
62333636303236336163363935643361663864623739316461353737383034306261353637363233
|
33393765653034643166313639323962306265383336663265623832393038386334643661336663
|
||||||
32616533333933633730356365666531336135376430613462343662343930663332613961393433
|
66643330306239383736636330393266346537326436663432653031303765393930633761343363
|
||||||
65633865346366396134633962303665633766313432383165363636616335323465303162623639
|
61353163323830656538373366323230353830633534326133613861356132623938663565393130
|
||||||
30356232636165303264366239323932633338613065383537383235373030633736663637383135
|
63333764613761663162633035353036663736656138623234313833326337316438323835313465
|
||||||
31393365336362326366393861313032303430663231303030653539643263353035316564393130
|
31653136356638623965363266636461646465313065666137313931623063303337393963353437
|
||||||
31643031363537343936313538393837323634613630326239616435313732396330646562376332
|
39633166326662303834303261353965363235626535643564613064306636366461666263383833
|
||||||
33643765326538376537376237366263396465323064393830383262386336363838333433373530
|
66363966636538373435323365636630383561336533356631646563646536383333393864613164
|
||||||
34656466333862313336633361343034663238336139356663653835313838363835613034633830
|
38393638623263666239623835613933393564323132373630623734363334663735633438663137
|
||||||
62356535613135663435653566353233623635646235313264353731666432313763666638653739
|
35613563356161316632323562626639633931643762643232636136653731323337333236326363
|
||||||
31393865326632303937373036646432626665356435363762386566653937626237386333656237
|
36633338386230626665373666643831626132323866653430393531336466386463663865393962
|
||||||
37633664326330323665616633666462636139663565633036643935363534626263376137386137
|
37363633343632653963306531393537633762313565303631643064363531363839643363356139
|
||||||
63666162346663366366633565346531623162656534303366373965363131303638343134366438
|
33666162306331393237333966643735643731633331373839356637636462633836633762653039
|
||||||
32373364316565303039386536363833643738623465373566356134386363323066393931646163
|
32633630303437633130613366386633346637623266663361656365633737346432343263646536
|
||||||
31316235653965343936346366393838313836396563346165623630363835313738663565393032
|
35376433386464383533323231353830666363363437623033643238363438303565616137316461
|
||||||
30316233313235353165373561316534333065376466356239373964663963376138643930353263
|
64636233643365663666656431613966396561366236383363303135613433626466366437396465
|
||||||
30326363353434373230383765653933313135373363613439643432343338616564336264326439
|
37303562633336636635363133626238643430663132626331316437626532356237383730626665
|
||||||
39373939373765613531306533363362656164313731383530313462366537633866373238623032
|
38373633323531346366313631376231353966626637636262333936343066336639396262646436
|
||||||
30313438306466343337326631346466346535623139363961626239343861306138343861326566
|
31616165626466623066336138653966323334326236663439643561663863323130653631643231
|
||||||
62323931353637646231303361633162636537373838663235653966663235653236613535623135
|
31396335333066326166383337353033376335376664653162396466346638653531346239353936
|
||||||
32353038616532376636633064353739633064343639333530643562663436633431616237333062
|
33663862323135623432333036336162353061363537363463366435643461356562323935653962
|
||||||
39663061623533393932303537343562663962623734336631376532613731373864373531623937
|
62333130616466353032316462356233393037326438303064656235323966306535346132663232
|
||||||
64636162613066613734313433383330323764393265336239303266356533656166303238623732
|
63393238646534316433643638646536313934666361323061306561396132306431633932313031
|
||||||
33313531393934383639303163633766323961316532386161343665336431333138316438613362
|
31356430313463613331363064363265646361353430646563396238326330323038666264653935
|
||||||
32323634633062373535656566303130643366366265623063346532666566386263666436633837
|
63306635323439326665306135643730386239356661303232353737383532353363303163313437
|
||||||
31343361383735343236316337613762336338353461313965663438616538396530366234313135
|
38623064383035373239623966336563626130636335383833383366343130393939316535333466
|
||||||
61393337626430313537663738303462356133336564353865663532656534333538306131343930
|
65353062396632373330663365373761656161653837396666376531376663313466393230646461
|
||||||
61313961363164653566663930346332353737333432333239343264653134386361623761663331
|
62306465363166663962316634663763626666306631363731633834366433363630383833393361
|
||||||
32646265336533613234326231623466373130376139643763373833316637613136616631333538
|
36393932326138643432666437373463343061313135613163343034373464356666333134366664
|
||||||
39393934343131323535363136393566353632383935366239323661623434323431656435373137
|
66343461393733343130636266623661323066623332633930326361626436366237323664336637
|
||||||
36323139303335656139666437316238383932633965663861363363316337383938633262303830
|
61316135613330663835353139613039613264383838313337373432623133363735333732376566
|
||||||
62626339626638333430653531653039316235393436623237633764383038306235326637313530
|
38633763336438663737653736326466646237313130623934396531306430363461653962336132
|
||||||
31656132306333343830396531363936656537306233616564633733663762663731333362316165
|
64313437323766613632376439336234393165646338396334373037323237633737393866303231
|
||||||
62636331356636386632376338396463396164396334633831313733396136656637303031373330
|
33646338333532613432393234316235653466633639306465363062656335353034666665623631
|
||||||
37663432666263666461386437323536636439366664323730633065396332326464363634386361
|
66386437616462633336636566393537323831313566326637386466616331396464653438396562
|
||||||
62666564343562643433363437633431333062333330346666666432333335643763323361333832
|
33343132626463393364386133323163393336313065316433613133663961633033613232343337
|
||||||
30636134343362656562636663363864616439643364333135386438643132616436653739393161
|
37313334386533366465353461633930643662326235366139306335656163313864636161623239
|
||||||
39303966393261626536646137633935653932613339393764633632313136383366303863333832
|
38336238663630313836363739623334633130616433393536303431333735643565326265613561
|
||||||
61343062336263323436333633343833383864376464626262363933663664643733626632623234
|
30613762396162633964336165666137626333643735323330646266666563313935623230643262
|
||||||
61313936633335363361663930613436613233653962646135303961616161333561313434383736
|
30383635663035653437333339303730346366333765353739663231643433353764363966353435
|
||||||
37323030623832323935313334376362376439646238393030333963376536326531363832353161
|
63663466343864363033646663376261363562636630643038613365333936653165633733623134
|
||||||
61666263376465356564333462376131393434623036653634343030326538373564303637396466
|
36626135313330303463336535663235323536613661353332653139336165613261316638666563
|
||||||
35326630616435306361366136363864636437316366333463386534366366663838653734653633
|
63346131306536353630363236613864393935333431333864333464353664366134313861633463
|
||||||
66646132323165396335316231613932316333366564653538313834383461323561313231643533
|
34336364663030376264663961396235643438653730396661643032623762623965623737326430
|
||||||
35316433383837643561653433656239613764366337323064663262663333306166383935316135
|
34333236316335373863663232356136333431666233353138353139313466343762356632396561
|
||||||
64326131656239666562376538613735353235663336346633333864333637636230646237376435
|
37356162303636343833316432353831373532326436363538333466333636306666626465666333
|
||||||
38666534303038656237386563336136666333383764633564313330353634626336313435323233
|
36653065363538653266643831326234356534303133343564646564323762653238303161336131
|
||||||
33373632633265396537623065333461656130313063643063366236336363633564613366353932
|
35396235343837333331396535646564656433343766323765333465356134323536636639623962
|
||||||
32623832653639323966616634616139653837626165626437636538363937396234373933383366
|
65633233376136316334653335623666313666376661326362663338633862643963353536616563
|
||||||
64386630636235666464396631316434386138313864343230313462616236626134313738383933
|
66323939643332663665306536356566383164356561303935313432373264353738643131653831
|
||||||
64626161636662323133326466636334666339353565613263656334643838373530353263646539
|
31346231633037616139373330316231363938386536303432386638323139326132663539663738
|
||||||
33353565323939336164363131346438376637653835343839333833373461353030383734643863
|
64636231663538333932363332663137323764346336633336363965616535373030663363363861
|
||||||
39376565643437376434666539613364363836343238353365313564613164356664353363623534
|
65363632653831353761306231663562623732353433656637353966303033636666613739306230
|
||||||
62383066643762363134343765643862353038646639303562626239653630653963633432306635
|
64363465623031386431663565623966643836383532626134366363656265646563343635383538
|
||||||
35303561306134653566626434323336386239363439623731333761616366313032663664656432
|
32346166363832626664613335383731616135376635336266326531663731373633303131373965
|
||||||
66363932656564343565346436316438383534613239373337323766346131343364666331393461
|
66663666333739373033616363313132643938656335353164343764383933636265656665663433
|
||||||
38616266663239623763626364653639353435643133353034336632343235626235313565303931
|
65386235636531653236333465316331353938323331623733623731303462306566333365383134
|
||||||
34653438383365353335633061616331633539343636313439303738636239633934393362353739
|
31356337343732613764633731306335366136346633393265666331636465323566656337323838
|
||||||
63613761636465393564363039363235663830373936376236623330313265306630643065396639
|
37393031316630623466393138636666303065623430656336386430363962336539616262633030
|
||||||
61393535616532396232613562316436303066303732386262396664346334393439643738396537
|
62616365623765366236303539353166306630393362386430363736313438346266366336323839
|
||||||
32613834306466623066653765333165356231633065376462343236323730343936396562393263
|
34376462376633366463636339646337623965633663363862393261373535636230613532386464
|
||||||
62663434316466666532313036316539393237653835623432663237376332326138643038643065
|
34393564383130306138366564366666646132363732653036353135656132656339353932383530
|
||||||
34326565653561353465373939616564653833333161303430623339323062333136616664666236
|
61636234393165326631303731633062333735306635303566373838393164306538303664313266
|
||||||
30366330666330333362
|
33346337613836636337316638323631396235643363333836323135303738366235326465376630
|
||||||
|
3363
|
||||||
|
|||||||
Reference in New Issue
Block a user