Filesystem Management in a greenway using Ansible and Event-Driven Automation(2)

In the last article, we have setup PCP on the managed Linux servers. In this article, we’ll setup the environment to run Ansible Rulebooks and implement them.

Implement Ansible Rulebook

Note that you need to install ansible-rulebook and other necessary packages. You can find all of necessary information in the official site here: https://ansible.readthedocs.io/projects/rulebook/en/stable/installation.html

Ready to implement ansible rulebooks? Here, you can start with the below code.

---
- name: Listen for RHEL Performance Co-Pilot events
  hosts: all
  sources:
    - ansible.eda.webhook:
        host: 0.0.0.0
        port: 5000

  rules:
    - name: Respond to PMIE rule filesys.filling
      # Ansible cannot login to the managed node when it is 100% full filesystem. 
      # Therefore, launch this rule only when the file system is not 100% full.
      condition: "event.payload.pcp.pmie.message is not match('100%',ignorecase=true) and event.payload.pcp.pmie.rule == 'File system is filling up'" 
      action:
        run_playbook:
          name: playbooks/manage_filesystems/filesys.filling.yml

    - name: Launch PMIE rule filesys.filling when the managed node file system is 100% full
      condition: "event.payload.pcp.pmie.message is match('100%',ignorecase=true) and event.payload.pcp.pmie.rule == 'File system is filling up'" 
      action:
        run_playbook:
          name: playbooks/manage_filesystems/notify_filesystem_usage.yaml

    - name: Display any contents of event.payload variable
      condition: event.payload is defined
      action:
        debug:
          msg: "Received: {{ event.payload }}"

When this rulebook is activated, Ansible will start listening for webhooks from servers running PCP on port 5000.

When the managed server meets certain PCP rule conditions, such as the filesystem being full(filesys.filling), PCP issues a webhook.

The below JSON data is an example of the issued data by PCP when the server meets filesys.filling.

"pcp": {
    "pmie": {
        "rule": "File system is filling up",
        "hostname": "192.168.122.182",
        "message": "99%used[/dev/mapper/rhel-root]@192.168.122.182"
        }
    }
}

In this rulebook, if Ansible received this data from PCP, it will start Ansible Playbook playbooks/manage_filesystems/filesys.filling.yml to delete unnecessary files/directories and to return the servers to a healthy state.

Here is a sample playbook to maintain the servers in a healthy state 🙂

---
- name: EDA response to PMIE rule file.filling
  hosts: "{{ ansible_eda.event.payload.pcp.pmie.hostname }}"
  become: true
  gather_facts: false #The managed node is in a low performance at the moment, avoid unnecessary tasks.
  tasks: 
    - ansible.builtin.include_vars: ./webex.yml

    - name: Display ansible_eda.event.payload.pcp.pmie.message variable
      debug:
        msg: "ansible_eda.event.payload.pcp.pmie.message value: {{ ansible_eda.event.payload.pcp.pmie.message }}"
   
    - name: Display ansible_eda.event.payload.pcp.pmie.rule variable
      debug:
        msg: "ansible_eda.event.payload.pcp.pmie.rule value: {{ ansible_eda.event.payload.pcp.pmie.rule }}"

    - name: df -h result before delete files
      command: df -h
      register: before_delete_files_df_result
    
    - name: Find files unused for 7 days
      command: find /tmp -type f -atime +7
      register: unused_files

    - name: Delete unused files for 7 days
      ansible.builtin.file:
        path: "{{ unused_files.stdout }}"
        state: absent

    - name: df -h result after delete files
      command: df -h
      register: after_delete_files_df_result

    - name: Cisco Webex Teams - Text Message to a Room
      community.general.cisco_webex:
        recipient_type: roomId
        recipient_id: "{{ room_id }}"
        msg_type: markdown
        personal_token: "{{ token }}"
        msg: "{{ lookup('ansible.builtin.template', './templates/chatMsg.j2') }}"

This playbook automates identifying and cleaning up unused files when a filesystem is filling up and then notifying the appropriate team through Cisco Webex what was performed by a playbook.

Conclusion

In this article, we have been exploring how to maintain Linux filesystems automatically.

By utilizing this solution, filesystems will remain clean, ensuring that unnecessary files and directories are not stored on the servers. This will not only avoid troubles but also contribute to reducing carbon footprints.