Our users are often unaware of their home directory disk usage and get surprised where they are over their assigned disk quota and cannot write anymore. The script, that follows, is placed in the system /etc/profile.d directory with the name homequota.sh. It checks disk usage and displays information about disk space available under user’s quota. Read the rest of this entry »
It is often that one wants to look at distribution of undetermined indices in Illumina Hiseq experiments to spot problems with experiment’s sample sheet or with the library itself. These indices are stored in fastq file(s) in the undetermined_indices folder. Below is the Bash script that processes these fastq files (fastq files can be in gzip’ed and/or uncompressed forms), and prints distribution of indices to stdout in a form of comma-separated values. If the script is named “index_stats”, one calls it from the undetermined_indices folder as:
index_stats > lane2_undetermined.csv
Latest versions of GNU sort program support parallel sorting. The script as presented below, runs GNU sort with 8 threads. It also utilizes parallel unpigz with 8 cores/processors to uncompress fastq.gz files, if unpigz is present. Otherwise, it uses single core gunzip command.
Read the rest of this entry »
Backing up large filesystems with very large numbers of files and directories is challenging. I routinely deal with 20-40TB filesystems with millions, sometimes tens of millions, of files. Filesystem with Unix home directories in a place with large number of users is an example of such a filesystem. The problem is more of backup speed than capacity.
Some commercial backup packages can do a better job here, but if you rely on free backup solutions, the situation is a bit more challenging. I have tried quite a few open-source backup programs, and performance of most of them is not acceptable in this type of environment. However, I have identified a few approaches which performance comes as acceptable, barely acceptable but acceptable from my perspective. Read the rest of this entry »
When one runs non-RHEL linux on modern Dell servers, disk monitoring is a bit of a challenge. Dell’s OpenManage is a fine tool, but it cannot be run on non-RHEL versions of linux. So, how can one monitor health of hard drives when OpenManage is not an option?
An actual approach depends on the storage controller in use on the server.
Read the rest of this entry »
One of great features of GNU tar is that it can create multi-volume archives. This translates to ability of using multiple tapes when tar is used with a tape changer/library device. The only trick is to automate process of changing tapes. Such automation can be achieved by providing a script changing tape via “-F” argument to tar. Here, there is the script that I use with my changer (16x LTO5 tapes). Read the rest of this entry »
Linux provides a great variety of tools to provide a wealth of information about hardware on which it is running.
Dmidecode is probably my favorite tool. It provides not only description of hardware but also other useful information such as serial numbers and BIOS version. To illustrate power of dmidecode, I was asked whether memory can be upgraded on one of our Dell servers. This includes finding out how many memory slots are available and how many are occupied at the present. All of this information can be obtained at once with dmidecode:
dmidecode -t memory | grep Size
Size: 16384 MB Size: 16384 MB Size: 16384 MB Size: 16384 MB Size: No Module Installed Size: No Module Installed Size: No Module Installed Size: No Module Installed Size: 16384 MB Size: 16384 MB Size: 16384 MB Size: 16384 MB Size: No Module Installed Size: No Module Installed Size: No Module Installed Size: No Module Installed
As the above shows, there are 16 memory slots on the server, with 8 of them holding 16GB DIMMs, and 8 being empty and available for upgrades. Read the rest of this entry »
When attempting to install Fedora 17 on Sun Fire X4200 server, the following message displayed on the attached monitor:
Cannot Display This Video ModeI tried a couple of different Dell monitors (both about 4-5 years old) with the same result. Fedora installation was available in text mode only. Unfortunately some of the options, such as custom disk partitioning, are available only during graphical (GUI) installation mode.
The following combination of boot parameters have done the trick to display GUI installation:
nomodeset xdriver=vesa resolution=800x600
It is my experience that email is frequently used (or rather abused) to exchange documents of business sensitive nature.
As all should know email is an insecure way of communicating, and unless documents are fully encrypted, it should not be used to exchange/share documents of any sensitive nature.
A much sensible alternative is to build and to use a secure document exchange portal to facilitate exchange of files between business partners. Such portal can be created with commercial software packages. However, I recommend that you take a serious look at WebDAV protocol. WebDAV can be utilized to build a simple but secure and highly usable file portal, and if you create it with Apache and mod_dav open-source packages, such portal will also be very inexpensive. Read the rest of this entry »