Debian system maintenance

Background

I have been maintaining various flavors of Linux/Unix servers since around 1996. Keeping them up to date has always required disciplined practices and good hygiene. Every operating system requires some sort of maintenance over time, and different distributions have different life cycles. Here are examples of the ones I use most often:

  • Debian stable is released every 2 years.
  • Ubuntu LTS is released every 2 years.
  • OpenBSD is released every 6 months.
  • Gentoo is a rolling release.
  • VoidLinux is a rolling release.

Debian lifecycle

The current Debian release lifecycle provides about two years of full support for a new version. For instance, consider Debian 11 (“bullseye”): when Debian 12 (“bookworm”) is released, you typically have about a year to upgrade. If you remain on Debian 11 beyond that time, you enter the LTS (Long-Term Support) phase, which is managed by volunteers rather than by the primary Debian Security Team. While this LTS community support is valuable, it's less than ideal compared to standard support.

Automatic patching

During the lifecycle of a Debian distribution, I use the unattended-upgrades package to automatically apply security and package updates. In most cases, unattended-upgrades can also restart any affected services (if the package includes a restart trigger), helping ensure patches take immediate effect. Of course, kernel vulnerabilities often still require a system reboot for the new kernel to load.

To confirm that this process works effectively, I rely on Nanitor for monitoring and auditing. As the CTO of Nanitor, I find it invaluable for verifying upgrades and ensuring systems remain secure and compliant without having to manually check each server.

Upgrading Debian

Whenever a new major version is released, I use a simple Ansible playbook. Because I run PostgreSQL, I also want to upgrade it to the version shipped with the new Debian release. For example, when I upgraded from Debian 11 (bullseye) to Debian 12 (bookworm), I moved my PostgreSQL cluster from version 13 to 15. Below is the playbook I created.

main.yml - This file controls the flow, checks requirements, and executes the OS and PostgreSQL upgrade tasks:

---
- name: Upgrade Debian 11 to Debian 12.
  hosts: all
  gather_facts: true
  become: true
  become_user: root
  vars:
    old_deb: "bullseye"
    new_deb: "bookworm"
    debian_frontend: "noninteractive"
  tasks:
    - name: Gather the package facts
      ansible.builtin.package_facts:
        manager: auto

    - name: Ensure Debian 11 (Bullseye) is installed
      ansible.builtin.fail:
        msg: "This playbook requires Debian 12 (Bookworm)."
      when: >
        ansible_distribution != 'Debian' or
        ansible_distribution_version is version('11', '<') or
        ansible_distribution_version is version('12', '>=')

    - name: Upgrade the OS
      ansible.builtin.import_tasks: os.yml

    - name: Upgrade postgresql
      ansible.builtin.import_tasks: postgresql.yml

os.yml - This file handles tasks to upgrade Debian 11 to Debian 12:

---
- name: Check if we need to upgrade if old deb repositories are being used.
  ansible.builtin.shell: "grep -r '{{ old_deb }}' /etc/apt/sources.list* | wc -l"
  register: old_deb_count
  changed_when: false
  ignore_errors: true

- name: Set fact based on presence of 'old_deb'
  ansible.builtin.set_fact:
    old_deb_present: "{{ old_deb_count.stdout != '0' }}"

- name: Find repository files
  ansible.builtin.find:
    paths:
      - /etc/apt
      - /etc/apt/sources.list.d
    use_regex: false
    patterns: '*.list'
  register: source_files

- name: Update distribution name from olddeb to newdeb in sources list
  ansible.builtin.replace:
    path: "{{ item.path }}"
    regexp: "{{ old_deb }}"
    replace: "{{ new_deb }}"
    backup: true
  loop: "{{ source_files.files }}"
  failed_when: false
  when: old_deb_present

- name: Change non-free to non-free-firmware
  ansible.builtin.replace:
    path: "{{ item.path }}"
    regexp: "non-free$"
    replace: "non-free-firmware"
    backup: true
  loop: "{{ source_files.files }}"
  failed_when: false
  when: old_deb_present

- name: Update and upgrade all packages to new distribution
  ansible.builtin.apt:
    update_cache: true
    upgrade: 'dist'
    force_apt_get: true
    allow_downgrade: true
    autoremove: true
    allow_change_held_packages: true
    dpkg_options: 'force-confdef,force-confold'
  environment:
    DEBIAN_FRONTEND: "{{ debian_frontend }}"
  when: old_deb_present

- name: Ensure essential packages are installed
  ansible.builtin.apt:
    name:
      - libpcre3
      - curl
      - wget
      - dnsutils
      - net-tools
      - vim
    state: present
    update_cache: true
  when: old_deb_present

postgresql.yml - This file handles tasks to upgrade PostgreSQL 13 to 15:

---
- name: Install PostgreSQL 15
  ansible.builtin.apt:
    name:
      - postgresql-15
      - postgresql-contrib
    state: present

- name: Stop the PostgreSQL service
  ansible.builtin.systemd:
    name: postgresql
    state: stopped

- name: Stop old PostgreSQL cluster
  ansible.builtin.command: pg_dropcluster --stop 15 main
  changed_when: true

- name: Upgrade PostgreSQL cluster from 13 to 15
  ansible.builtin.command: pg_upgradecluster -m upgrade 13 main
  changed_when: true

- name: Start the new PostgreSQL service
  ansible.builtin.systemd:
    name: postgresql
    state: started

- name: Verify new PostgreSQL cluster
  ansible.builtin.command: sudo -u postgres psql -l
  register: psql_check
  changed_when: true
  failed_when: "'template' not in psql_check.stdout"

- name: Drop old PostgreSQL cluster
  ansible.builtin.command: pg_dropcluster 13 main
  changed_when: true
  when: psql_check is succeeded

- name: Remove old PostgreSQL version
  ansible.builtin.apt:
    name: postgresql-13
    state: absent

This demonstrates how straightforward it can be to upgrade to a new Debian release as long as you rely on official packages and avoid custom software that bypasses the package manager.

Conclusion

Debian's stable release cycle, combined with regular updates and careful upgrades, keeps servers secure and reliable. Tools like unattended-upgrades simplify patching by automatically updating packages—often restarting services that need it so you don't have to intervene constantly. Meanwhile, solutions such as Nanitor provide a clear audit trail of updates and configurations, ensuring that you have visibility and confidence in your patching and upgrade processes. With a bit of planning and a clear playbook, moving from one Debian release to the next can be quite painless, helping you maintain a healthy and stable environment.