6. Personalized Configuration

6.1 Deploying using InfiniBand

TIP

When both the management node and the compute nodes support InfiniBand cards, you can benefit from their high bandwidth and use InfiniBand for deployment. PODsys supports deploying the Ubuntu system via InfiniBand cards.

• Modify the config.yaml file to set the manager_nic to an InfiniBand card.
• Modify the iplist.txt file to change the Ethernet card information in columns 3-5 to InfiniBand card information. Ensure that the IP address in column 3 is consistent with that in column 6.
• Execute the following script on the existing OS of the compute nodes:

Script

#!/bin/bash

# Function to enable PXE and UEFI using mlxconfig tool
configure_infiniband_with_mlx() {
    INFINIBAND_SYS_PATH="/sys/class/infiniband/"
    
    # Iterate through all InfiniBand devices
    for device in $(ls ${INFINIBAND_SYS_PATH}); do
        echo "Enabling PXE and UEFI on InfiniBand device: $device"
        
        # Enable PXE
        mlxconfig -y -d $device set EXP_ROM_PXE_ENABLE=1
        
        # Enable UEFI for x86 architecture
        mlxconfig -y -d $device set EXP_ROM_UEFI_x86_ENABLE=1
    done
}

# Function to enable PXE and UEFI using mstconfig tool
configure_infiniband_with_mst() {
    PCI_DEVICES_PATH="/sys/bus/pci/devices"
    
    # Loop through all PCI devices
    for bus_dev_func in $(ls "${PCI_DEVICES_PATH}"); do
        # Check if the device is an InfiniBand device
        if [[ -e "${PCI_DEVICES_PATH}/${bus_dev_func}/infiniband" ]]; then
            echo "Enabling PXE and UEFI on PCI device: $bus_dev_func"
            
            # Enable PXE
            mstconfig -y -d "$bus_dev_func" set EXP_ROM_PXE_ENABLE=1
            
            # Enable UEFI for x86 architecture
            mstconfig -y -d "$bus_dev_func" set EXP_ROM_UEFI_x86_ENABLE=1
        fi
    done
}

# Main function to enable PXE and UEFI on Mellanox InfiniBand devices
enable_mellanox_pxe() {
    if which mlxconfig > /dev/null 2>&1; then
        configure_infiniband_with_mlx
    elif which mstconfig > /dev/null 2>&1; then
        configure_infiniband_with_mst
    else
        echo "Error: Neither mlxconfig nor mstconfig is available on this system."
        exit 1
    fi
}

# Check if the script is run as root user
if [ "$(id -u)" != "0" ]; then 
    echo "This script must be run with root privileges. Please use sudo or switch to the root user."
    exit 1 
fi

# Execute the function to enable PXE and UEFI
enable_mellanox_pxe

DANGER

you must first enable the UEFI PXE ROM of the card you wish to PXE boot from because it is disabled by default. So, for a Bare Metal Server, direct deployment of InfiniBand may not be possible.

6.2 Using Network Interface Bonding to Accelerate Deployment

Using network interface bonding to increase the management node's upload bandwidth can improve deployment speed.

1. On the management node, install the ifenslave.

sudo apt install ifenslave

2. load bonding module。

# load bonding module
sudo modprobe bonding
# 
sudo lsmod | grep bonding

3. Configure network interface bonding. For example, configure a bond in mode 6 (balance-alb) using two interfaces named eth0 and eth1. Modifying /etc/netplan/00-installer-config.yaml:

/etc/netplan/00-installer-config.yaml

network:
  version: 2
  ethernets:
    eth0:
      dhcp4: false
    eth1:
      dhcp4: false
  bonds:
    bond0:
      addresses: [192.168.1.88/24]
      gateway4: 192.168.1.1
      nameservers:
        addresses:
          - 114.114.114.114
      interfaces:
        - eth0
        - eth1
      parameters:
        mode: balance-alb

Then run:

sudo netplan apply

4. Check the Bonding Status:

# Check the Bonding Status:
cat /proc/net/bonding/bond0 
sudo ethtool bond0

To view the configuration and status information of the bonded network interface bond0, including speed, duplex mode, link status, and hardware features.

5. Delete Bond

Comment out the added content in the /etc/netplan/00-installer-config.yaml file.

sudo ifconfig bond0 down
sudo rmmod bonding

Check:

ifconfig bond0