214 lines
6.2 KiB
Markdown
214 lines
6.2 KiB
Markdown
# NetBird GitOps PoC
|
|
|
|
Proof-of-concept for managing NetBird VPN configuration via Infrastructure as Code (IaC) with GitOps workflow using Terraform.
|
|
|
|
## Project Status: POC Complete
|
|
|
|
**Start date:** 2026-02-15
|
|
**Status:** Core functionality working, remaining pain points documented
|
|
|
|
### What Works
|
|
|
|
- [x] NetBird self-hosted instance deployed (`netbird-poc.networkmonitor.cc`)
|
|
- [x] Gitea CI/CD server deployed (`gitea-poc.networkmonitor.cc`)
|
|
- [x] Gitea Actions runner for CI/CD
|
|
- [x] Terraform implementation - creates groups, policies, setup keys
|
|
- [x] CI/CD pipeline - PR shows plan, merge-to-main applies changes
|
|
|
|
### Remaining Pain Points
|
|
|
|
See [PAIN_POINTS.md](./PAIN_POINTS.md) for detailed analysis of:
|
|
- Peer naming automation (no link between setup keys and enrolled peers)
|
|
- Per-user vs per-role setup keys
|
|
- Secure key distribution to operators
|
|
|
|
---
|
|
|
|
## Architecture
|
|
|
|
```
|
|
+-------------------+ PR/Merge +-------------------+
|
|
| Engineer | ----------------> | Gitea |
|
|
| (edits .tf) | | (gitea-poc.*) |
|
|
+-------------------+ +-------------------+
|
|
|
|
|
| CI/CD
|
|
v
|
|
+-------------------+
|
|
| Terraform |
|
|
| (in Actions) |
|
|
+-------------------+
|
|
|
|
|
| API calls
|
|
v
|
|
+-------------------+ Enroll +-------------------+
|
|
| Operators | ----------------> | NetBird |
|
|
| (use setup keys) | | (netbird-poc.*) |
|
|
+-------------------+ +-------------------+
|
|
```
|
|
|
|
## Directory Structure
|
|
|
|
```
|
|
netbird-gitops-poc/
|
|
├── ansible/ # Deployment playbooks
|
|
│ ├── caddy/ # Shared reverse proxy
|
|
│ ├── gitea/ # Standalone Gitea (no OAuth)
|
|
│ ├── gitea-runner/ # Gitea Actions runner
|
|
│ └── netbird/ # NetBird with embedded IdP
|
|
├── terraform/ # Terraform configuration (Gitea repo content)
|
|
│ ├── .gitea/workflows/ # CI/CD workflow
|
|
│ │ └── terraform.yml
|
|
│ ├── main.tf # Provider config
|
|
│ ├── variables.tf # Input variables
|
|
│ ├── groups.tf # Group resources
|
|
│ ├── policies.tf # Policy resources
|
|
│ ├── setup_keys.tf # Setup key resources
|
|
│ ├── outputs.tf # Output values
|
|
│ ├── terraform.tfstate # State (committed for POC)
|
|
│ ├── terraform.tfvars # Secrets (gitignored)
|
|
│ └── terraform.tfvars.example
|
|
├── README.md
|
|
└── PAIN_POINTS.md
|
|
```
|
|
|
|
## Quick Start
|
|
|
|
### Prerequisites
|
|
|
|
- VPS with Docker
|
|
- DNS records pointing to VPS
|
|
- Ansible installed locally
|
|
- Terraform installed locally (for initial setup)
|
|
|
|
### 1. Deploy Infrastructure
|
|
|
|
```bash
|
|
# 1. NetBird (generates secrets, needs vault password)
|
|
cd ansible/netbird
|
|
./generate-vault.sh
|
|
ansible-vault encrypt group_vars/vault.yml
|
|
ansible-playbook -i poc-inventory.yml playbook-ssl.yml --ask-vault-pass
|
|
|
|
# 2. Gitea
|
|
cd ../gitea
|
|
ansible-playbook -i poc-inventory.yml playbook.yml
|
|
|
|
# 3. Caddy (reverse proxy for both)
|
|
cd ../caddy
|
|
ansible-playbook -i poc-inventory.yml playbook.yml
|
|
|
|
# 4. Gitea Runner (get token from Gitea Admin -> Actions -> Runners)
|
|
cd ../gitea-runner
|
|
ansible-playbook -i poc-inventory.yml playbook.yml -e vault_gitea_runner_token=<TOKEN>
|
|
```
|
|
|
|
### 2. Initial Terraform Setup (Local)
|
|
|
|
```bash
|
|
cd terraform
|
|
|
|
# Create tfvars with your NetBird PAT
|
|
cp terraform.tfvars.example terraform.tfvars
|
|
# Edit terraform.tfvars with actual token
|
|
|
|
# Initialize and apply
|
|
terraform init
|
|
terraform apply
|
|
```
|
|
|
|
### 3. Push to Gitea
|
|
|
|
```bash
|
|
cd terraform
|
|
git init
|
|
git add .
|
|
git commit -m "Initial Terraform config"
|
|
git remote add origin git@gitea-poc.networkmonitor.cc:admin/netbird-iac.git
|
|
git push -u origin main
|
|
```
|
|
|
|
### 4. Configure Gitea Secrets
|
|
|
|
In Gitea repository Settings -> Actions -> Secrets:
|
|
- `NETBIRD_TOKEN`: Your NetBird PAT
|
|
|
|
### 5. Make Changes via GitOps
|
|
|
|
Edit Terraform files locally, push to create PR:
|
|
|
|
```hcl
|
|
# groups.tf - add a new group
|
|
resource "netbird_group" "new_team" {
|
|
name = "new-team"
|
|
}
|
|
```
|
|
|
|
```bash
|
|
git checkout -b add-new-team
|
|
git add groups.tf
|
|
git commit -m "Add new-team group"
|
|
git push -u origin add-new-team
|
|
# Create PR in Gitea -> CI runs terraform plan
|
|
# Merge PR -> CI runs terraform apply
|
|
```
|
|
|
|
---
|
|
|
|
## CI/CD Workflow
|
|
|
|
The `.gitea/workflows/terraform.yml` workflow:
|
|
|
|
| Event | Action |
|
|
|-------|--------|
|
|
| Pull Request | `terraform plan` (preview changes) |
|
|
| Push to main | `terraform apply` (apply changes) |
|
|
| After apply | Commit updated state file |
|
|
|
|
**State Management:** State is committed to git (acceptable for single-operator POC). For production, use a remote backend.
|
|
|
|
---
|
|
|
|
## Key Discoveries
|
|
|
|
### NetBird API Behavior
|
|
|
|
1. **Peer IDs are not predictable** - Generated server-side at enrollment time
|
|
2. **No setup key -> peer link** - NetBird doesn't record which setup key enrolled a peer
|
|
3. **Peers self-enroll** - Cannot create peers via API (WireGuard keypair generated locally)
|
|
4. **Terraform URL format** - Use `https://domain.com` NOT `https://domain.com/api`
|
|
|
|
---
|
|
|
|
## Credentials Reference (POC Only)
|
|
|
|
| Service | Credential | Location |
|
|
|---------|------------|----------|
|
|
| NetBird PAT | `nbp_T3yD...` | Dashboard -> Team -> Service Users |
|
|
| Gitea | admin user | Created during setup |
|
|
| VPS | root | `observability-poc.networkmonitor.cc` |
|
|
|
|
**Warning:** Rotate all credentials before any production use.
|
|
|
|
---
|
|
|
|
## Cleanup
|
|
|
|
```bash
|
|
# Destroy Terraform resources
|
|
cd terraform
|
|
terraform destroy
|
|
|
|
# Stop VPS services
|
|
ssh root@observability-poc.networkmonitor.cc
|
|
cd /opt/caddy && docker compose down
|
|
cd /opt/gitea && docker compose down
|
|
cd /opt/netbird && docker compose down
|
|
```
|
|
|
|
---
|
|
|
|
## Next Steps
|
|
|
|
See [PAIN_POINTS.md](./PAIN_POINTS.md) for remaining challenges to address before production use.
|