Lightweight Directory Access Protocol (LDAP) is a protocol used for accessing and managing directory services over a network. It is commonly used for centralised authentication and authorisation in enterprise environments. By integrating LDAP with a PBS cluster, user authentication and management can be centralised, making it easier to handle user accounts and permissions across multiple nodes in the cluster. In the setup, LDAP has two types of nodes LDAP server - Head node (

This is the fifth part of an eight part series  on how to setup an HPC cluster on AWS. This document explains how to setup an OpenPBS job scheduler in an AWS cluster. OpenPBS (Portable Batch System) is an open-source workload management and job scheduling system designed for HPC environments. It efficiently allocates compute resources across clusters by queuing, scheduling, and monitoring batch jobs submitted by users. OpenPBS and PBS will be used interchangeably in this doc

This is the fourth part of an eight part series  on how to setup an HPC cluster on AWS. This document explains how to setup a BeeGFS files system in the AWS cluster. BeeGFS is a high-performance parallel file system designed for scalable storage in HPC and other data-intensive environments. It stripes data across multiple storage servers, enabling fast, concurrent access for many clients, which significantly boosts I/O performance. The first crucial step in installing BeeGFS

This is the third part of an eight part series  on how to setup an HPC cluster on AWS. This document explains how to setup a passwordless SSH across all the nodes in the cluster Passwordless SSH is required in HPC clusters to allow nodes to communicate and execute tasks automatically, such as job scheduling, data transfers, and parallel computations, without repeated password prompts, ensuring efficient and seamless operation. Create the SSH keys On each node, ensure the ~/

This is the second part of an eight part series  on how to setup an HPC cluster on AWS. This part explain the initial setup to install the packages we will need for the different components of the HPC cluster. The first thing we have to do is disable Security-Enhanced Linux (SELinux). It’s a security module built into the Linux kernel that provides mandatory access control (MAC), which is more strict than the usual discretionary access control (DAC) that standard Linux uses.

This is the first part of an eight part series on how to setup an HPC cluster on AWS. This document describes the components of the launched VMs and outlines the key design elements of the cluster The cluster will have seven virtual machines (VM) One head / control node One login node Three compute nodes Two storage nodes All the VM will have the OS Rocky Linux 9.6 (Blue Onyx) AMI: ami-0f2425d4cce4e97dd Instance Type: t3.2xlarge When the VMs are created, an SSH key, terrafor

This is a eight part series on how to setup an HPC cluster on AWS. The main design elements the cluster will be as follows: The cluster will have seven virtual machines (VM) One head / control node One login node Three compute nodes Two storage nodes All the VM will have the OS Rocky Linux 9.6 (Blue Onyx) AMI: ami-0f2425d4cce4e97dd Instance Type: t3.2xlarge When the VMs are created, an SSH key, terraform-user is already added This makes it easy to log in to the VM from the l

Download and install Ollama. There are multiple LLMs available for Ollama. In this case, we will be using Codellama, which can use text...

Some HPC machines today use singularity containers for their machine learning workflows. Once configured Singularity containers can be...

Dask is a popular Python library designed for scalable computing with dynamic task scheduling. A key strength of Dask lies in its...

The usual workflow in developing an HPC application is to develop the code in local machines and then run the completed application in an...

VSCode is a very popular tool to manage large projects in C/C++. One of the main advantages of VSCode is we can automate workflows that...

Debugging a Parallel program is not straightforward as debugging a sequential program because it involves multiple processes with...