volker.raschek/helm-gitea
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

[Breaking] Add HA-support; switch to Deployment (#437)

Browse Source 
# Changes

A big shoutout to @luhahn for all his work in #205 which served as the base for this PR.

## Documentation

- [x] After thinking for some time about it, I still prefer the distinct option (as started in #350), i.e. having a standalone "HA" doc under `docs/ha-setup.md` to not have a very long README (which is already quite long).
      Most of the information below should go into it with more details and explanations behind all of the individual components.

## Chart deps

~~- Adds `meilisearch` as a chart dependency for a HA-ready issue indexer. Only works with >= Gitea 1.20~~
~~- Adds `redis` as a chart dependency for a HA-ready session and queue store.~~
- Adds `redis-cluster` as a chart dependency for a HA-ready session and queue store (alternative to `redis`). Only works with >= Gitea 1.19.2.
- Removes `memcached` instead of `redis-cluster`
- Add `postgresql-ha` as default DB dep in favor of `postgres`

## Adds smart HA chart logic

The goal is to set smart config values that result in a HA-ready Gitea deployment if `replicaCount` > 1.

- If `replicaCount` > 1,
  - `gitea.config.session.PROVIDER` is automatically set to `redis-cluster`
  - `gitea.config.indexer.REPO_INDEXER_ENABLED` is automatically set to `false` unless the value is `elasticsearch` or `meilisearch`
  - `redis-cluster` is used for `[queue]` and `[cache]` and `[session]`mode or not

Configuration of external instances of `meilisearch` and `minio` are documented in a new markdown doc.

## Deployment vs Statefulset

Given all the discussions about this lately (#428), I think we could use both.
In the end, we do not have the requirement for a sequential pod scale up/scale down as it would happen in statefulsets.
On the other side, we do not have actual stateless pods as we are attaching a RWX to the deployment.
Yet I think because we do not have a leader-election requirement, spawning the pods as a deployment makes "Rolling Updates" easier and also signals users that there is no "leader election" logic and each pod can just be "destroyed" at anytime without causing interruption.

Hence I think we should be able to switch from a statefulset to a deployment, even in the single-replica case.

This change also brought up a templating/linting issue: the definition of `.Values.gitea.config.server.SSH_LISTEN_PORT` in `ssh-svc.yaml` just "luckily" worked so far due to naming-related lint processing. Due to the change from "statefulset" to "deployment", the processing queue changed and caused a failure complaining about `config.server.SSH_LISTEN_PORT` not being defined yet.
The only way I could see to fix this was to "properly" define the value in `values.yaml` instead of conditionally definining it in `helpers.tpl`. Maybe there's a better way?

## Chart PVC Creation

I've adapted the automated PVC creation from another chart to be able to provide the `storageClassName` as I couldn't get dynamic provisioning for EFS going with the current implementation.
In addition the naming and approach within the Gitea chart for PV creation is a bit unusual and aligning it might be beneficial.

A semi-unrelated change which will result in a breaking change for existing users but this PR includes a lot of breaking changes already, so including another one might not make it much worse...

- New `persistence.mount`: whether to mount an existing PVC (via `persistence.existingClaim`
- New `persistence.create`: whether to create a new PVC

## Testing

As this PR does a lot of things, we need proper testing.
The helm chart can be installed from the Git branch via `helm-git` as follows:

```
helm repo add gitea-charts git+https://gitea.com/gitea/helm-chart@/?ref=deployment
helm install gitea --version 0.0.0
```
It is **highly recommended** to test the chart in a dedicated namespace.

I've tested this myself with both `redis` and `redis-cluster` and it seemed to work fine.
I just did some basic operations though and we should do more niche testing before merging.

Examplary `values.yml` for testing (only needs a valid RWX storage class):

<details>

<summary>values.yaml</summary>

```yml
image:
  tag: "dev"
  PullPolicy: "Always"
  rootless: true

replicaCount: 2

persistence:
  enabled: true
  accessModes:
    - ReadWriteMany
  storageClass: FIXME

redis-cluster:
  enabled: false
  global:
    redis:
      password: gitea

gitea:
  config:
    indexer:
      ISSUE_INDEXER_ENABLED: true
      REPO_INDEXER_ENABLED: false
```
</details>

## Preferred setup

The preferred HA setup with respect to performance and stability might currently be as follows:

- Repos: RWX (e.g. EFS or Azurefiles NFS)
- Issue indexer: Meilisearch (HA)
- Session and cache: Redis Cluster (HA)
- Attachments/Avatars: Minio (HA)

This will result in a ~ 10-pod HA setup overall.
All pods have very low resource requests.

fix #98

Co-authored-by: pat-s <pat-s@noreply.gitea.io>
Reviewed-on: https://gitea.com/gitea/helm-chart/pulls/437
Co-authored-by: pat-s <patrick.schratz@gmail.com>
Co-committed-by: pat-s <patrick.schratz@gmail.com>
This commit is contained in:
pat-s
2023-07-17 19:09:42 +00:00
committed by pat-s
parent f66a192d45
commit 8e27bb9bae
17 changed files with 675 additions and 245 deletions
Download Patch File Download Diff File Expand all files Collapse all files

26
templates/gitea/config.yaml
View File

@ -16,6 +16,32 @@ metadata:
{{- include "gitea.labels" . | nindent 4 }}
type: Opaque
stringData:
assertions: |
{{- /* multiple replicas assertions */ -}}
{{- if gt .Values.replicaCount 1.0 -}}
{{- if .Values.gitea.config.cron.GIT_GC_REPOS -}}
{{- if .Values.gitea.config.cron.GIT_GC_REPOS.enabled -}}
{{- fail "Invoking the garbage collector via CRON is not yet supported when running with multiple replicas. Please set 'GIT_GC_REPOS.enabled = false'." -}}
{{- end }}
{{- end }}
{{- if eq (first .Values.persistence.accessModes) "ReadWriteOnce" -}}
{{- fail "When using multiple replicas, a RWX file system is required and gitea.persistence.accessModes[0] must be set to ReadWriteMany." -}}
{{- end }}
{{- if eq (get .Values.gitea.config.indexer "ISSUE_INDEXER_TYPE") "bleve" -}}
{{- fail "When using multiple replicas, the issue indexer (gitea.config.indexer.ISSUE_INDEXER_TYPE) must be set to a HA-ready provider such as 'meilisearch', 'elasticsearch' or 'db' (if the DB is HA-ready)." -}}
{{- end }}
{{- if .Values.gitea.config.indexer.REPO_INDEXER_TYPE -}}
{{- if eq (get .Values.gitea.config.indexer "REPO_INDEXER_TYPE") "bleve" -}}
{{- if .Values.gitea.config.indexer.REPO_INDEXER_ENABLED -}}
{{- if eq (get .Values.gitea.config.indexer "REPO_INDEXER_ENABLED") "true" -}}
{{- fail "When using multiple replicas, the repo indexer (gitea.config.indexer.REPO_INDEXER_TYPE) must be set to 'meilisearch' or 'elasticsearch' or disabled." -}}
{{- end }}
{{- end }}
{{- end }}
{{- end }}
{{- end }}
config_environment.sh: |-
#!/usr/bin/env bash
set -euo pipefail

76
templates/gitea/statefulset.yaml → templates/gitea/deployment.yaml
View File

@ -1,20 +1,27 @@
apiVersion: apps/v1
kind: StatefulSet
kind: Deployment
metadata:
name: {{ include "gitea.fullname" . }}
annotations:
{{- if .Values.statefulset.annotations }}
{{- toYaml .Values.statefulset.annotations | nindent 4 }}
{{- if .Values.deployment.annotations }}
{{- toYaml .Values.deployment.annotations | nindent 4 }}
{{- end }}
labels:
{{- include "gitea.labels" . | nindent 4 }}
spec:
replicas: {{ .Values.replicaCount }}
strategy:
type: {{ .Values.strategy.type }}
{{- if eq .Values.strategy.type "RollingUpdate" }}
rollingUpdate:
maxUnavailable: {{ .Values.strategy.rollingUpdate.maxUnavailable }}
maxSurge: {{ .Values.strategy.rollingUpdate.maxSurge }}
{{- end }}
selector:
matchLabels:
{{- include "gitea.selectorLabels" . | nindent 6 }}
{{- if .Values.statefulset.labels }}
{{- toYaml .Values.statefulset.labels | nindent 6 }}
{{- if .Values.deployment.labels }}
{{- toYaml .Values.deployment.labels | nindent 6 }}
{{- end }}
serviceName: {{ include "gitea.fullname" . }}
template:
@ -32,8 +39,8 @@ spec:
{{- end }}
labels:
{{- include "gitea.labels" . | nindent 8 }}
{{- if .Values.statefulset.labels }}
{{- toYaml .Values.statefulset.labels | nindent 8 }}
{{- if .Values.deployment.labels }}
{{- toYaml .Values.deployment.labels | nindent 8 }}
{{- end }}
spec:
{{- if .Values.schedulerName }}
@ -62,8 +69,8 @@ spec:
value: /data
- name: GITEA_TEMP
value: /tmp/gitea
{{- if .Values.statefulset.env }}
{{- toYaml .Values.statefulset.env | nindent 12 }}
{{- if .Values.deployment.env }}
{{- toYaml .Values.deployment.env | nindent 12 }}
{{- end }}
{{- if .Values.signing.enabled }}
- name: GNUPGHOME
@ -97,8 +104,8 @@ spec:
value: /data
- name: GITEA_TEMP
value: /tmp/gitea
{{- if .Values.statefulset.env }}
{{- toYaml .Values.statefulset.env | nindent 12 }}
{{- if .Values.deployment.env }}
{{- toYaml .Values.deployment.env | nindent 12 }}
{{- end }}
{{- if .Values.gitea.additionalConfigFromEnvs }}
{{- toYaml .Values.gitea.additionalConfigFromEnvs | nindent 12 }}
@ -234,8 +241,8 @@ spec:
- name: GITEA_ADMIN_PASSWORD
value: {{ .Values.gitea.admin.password | quote }}
{{- end }}
{{- if .Values.statefulset.env }}
{{- toYaml .Values.statefulset.env | nindent 12 }}
{{- if .Values.deployment.env }}
{{- toYaml .Values.deployment.env | nindent 12 }}
{{- end }}
volumeMounts:
- name: init
@ -250,7 +257,7 @@ spec:
{{- include "gitea.init-additional-mounts" . | nindent 12 }}
resources:
{{- toYaml .Values.initContainers.resources | nindent 12 }}
terminationGracePeriodSeconds: {{ .Values.statefulset.terminationGracePeriodSeconds }}
terminationGracePeriodSeconds: {{ .Values.deployment.terminationGracePeriodSeconds }}
containers:
- name: {{ .Chart.Name }}
image: "{{ include "gitea.image" . }}"
@ -283,8 +290,8 @@ spec:
- name: GNUPGHOME
value: {{ .Values.signing.gpgHome }}
{{- end }}
{{- if .Values.statefulset.env }}
{{- toYaml .Values.statefulset.env | nindent 12 }}
{{- if .Values.deployment.env }}
{{- toYaml .Values.deployment.env | nindent 12 }}
{{- end }}
ports:
- name: ssh
@ -340,6 +347,10 @@ spec:
affinity:
{{- toYaml . | nindent 8 }}
{{- end }}
{{- with .Values.topologySpreadConstraints }}
topologySpreadConstraints:
{{- toYaml . | nindent 8 }}
{{- end }}
{{- with .Values.tolerations }}
tolerations:
{{- toYaml . | nindent 8 }}
@ -378,38 +389,13 @@ spec:
path: private.asc
defaultMode: 0100
{{- end }}
{{- if and .Values.persistence.enabled .Values.persistence.existingClaim }}
{{- if .Values.persistence.enabled }}
{{- if .Values.persistence.mount }}
- name: data
persistentVolumeClaim:
{{- with .Values.persistence.existingClaim }}
claimName: {{ tpl . $ }}
{{- end }}
claimName: {{ .Values.persistence.claimName }}
{{- end }}
{{- else if not .Values.persistence.enabled }}
- name: data
emptyDir: {}
{{- else if and .Values.persistence.enabled (not .Values.persistence.existingClaim) }}
volumeClaimTemplates:
- metadata:
name: data
{{- with .Values.persistence.annotations }}
annotations:
{{- range $key, $value := . }}
{{ $key }}: {{ $value }}
{{- end }}
{{- end }}
{{- with .Values.persistence.labels }}
labels:
{{- range $key, $value := . }}
{{ $key }}: {{ $value }}
{{- end }}
{{- end }}
spec:
accessModes:
{{- range .Values.persistence.accessModes }}
- {{ . | quote }}
{{- end }}
{{- include "gitea.persistence.storageClass" . | indent 8 }}
resources:
requests:
storage: {{ .Values.persistence.size | quote }}
{{- end }}

21
templates/gitea/init.yaml
View File

@ -61,6 +61,27 @@ stringData:
echo "Gitea migrate might fail due to database connection...This init-container will try again in a few seconds"
exit 1
}
{{- if include "redis.servicename" . }}
function test_redis_connection() {
local RETRY=0
local MAX=30
echo 'Wait for redis to become avialable...'
until [ "${RETRY}" -ge "${MAX}" ]; do
nc -vz -w2 {{ include "redis.servicename" . }} {{ include "redis.port" . }} && break
RETRY=$[${RETRY}+1]
echo "...not ready yet (${RETRY}/${MAX})"
done
if [ "${RETRY}" -ge "${MAX}" ]; then
echo "Redis not reachable after '${MAX}' attempts!"
exit 1
fi
}
test_redis_connection
{{- end }}
{{- if or .Values.gitea.admin.existingSecret (and .Values.gitea.admin.username .Values.gitea.admin.password) }}

17
templates/gitea/poddisruptionbudget.yaml Normal file
View File

@ -0,0 +1,17 @@
{{- if .Values.podDisruptionBudget -}}
{{- if .Capabilities.APIVersions.Has "policy/v1" }}
apiVersion: policy/v1
{{- else }}
apiVersion: policy/v1beta1
{{- end }}
kind: PodDisruptionBudget
metadata:
name: {{ include "gitea.fullname" . }}
labels:
{{- include "gitea.labels" . | nindent 4 }}
spec:
selector:
matchLabels:
{{- include "gitea.selectorLabels" . | nindent 6 }}
{{- toYaml .Values.podDisruptionBudget | nindent 2 }}
{{- end -}}

24
templates/gitea/pvc.yaml Normal file
View File

@ -0,0 +1,24 @@
{{- if and .Values.persistence.enabled .Values.persistence.create }}
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
name: {{ .Values.persistence.claimName }}
namespace: {{ $.Release.Namespace }}
annotations:
{{ .Values.persistence.annotations | toYaml | indent 4}}
spec:
accessModes:
{{- if gt .Values.replicaCount 1.0 }}
- ReadWriteMany
{{- else }}
{{- .Values.persistence.accessModes | toYaml | nindent 4 }}
{{- end }}
volumeMode: Filesystem
{{- if .Values.persistence.storageClass }}
storageClassName: {{ .Values.persistence.storageClass }}
{{- end }}
volumeName: ""
resources:
requests:
storage: {{ .Values.persistence.size }}
{{- end }}

2
templates/gitea/ssh-svc.yaml
View File

@ -39,7 +39,9 @@ spec:
ports:
- name: ssh
port: {{ .Values.service.ssh.port }}
{{- if .Values.gitea.config.server.SSH_LISTEN_PORT }}
targetPort: {{ .Values.gitea.config.server.SSH_LISTEN_PORT }}
{{- end }}
protocol: TCP
{{- if .Values.service.ssh.nodePort }}
nodePort: {{ .Values.service.ssh.nodePort }}