Generate Files with Random Content and Size in Bash

Occasionally you need to generate a bunch of random files with random content, usually for testing compression, user quotas or miscellaneous stuff.

Here’s one way, using the bash shell and a few handy linux utilities.

  1. The bash $RANDOM function. It generates a random number between 0 – 32767.
  2. Linux DD utility, to output files.
  3. /dev/(h|s)da, your hard drive in linux.
  4. All wrapped in a bash while loop.

So lets start. First define a bash variable with the number of files we wish to create, lets say 10.

no_of_files=10

Then we’ll assign the a bash variable for the counter.

counter=1

As for the dd command, this creates a file with random content from your hard disk. (Mine’s /dev/sda) which is 1KB in size. The count switch tells dd to repeat 1024 bytes 1 time, thus the 1Kb file size. skip makes dd skip an x amount of bytes before reading further. Since this requires raw access to your hard drive, you’ll have to run this as root unfortunately. :(

dd bs=1024 count=1 skip=0 if=/dev/sda of=random-file

So now imagine, if we let bash assign the count and skip random numbers, we get random file contents. (Light bulbs flashing eh?)

Of course, all of it will be written to a single file called random-file in my case. To add just slight amount of variety, we can add the counter variable that we will use in our while loop as an extension. The dd command will now be:-

 dd bs=1024 count=$($RANDOM) skip=$($RANDOM) if=/dev/sda of=random-file.$counter

Finally, we will wrap it up in a bash while loop like this:-

no_of_files=10;
counter=1;
while [[ $counter -le $no_of_files ]];
 do echo Creating file no $counter;
  dd bs=1024 count=$RANDOM skip=$RANDOM if=/dev/sda of=random-file.$counter;
  let "counter += 1";
 done

When you run it, you will get output like this:-

Creating file no 1
16614+0 records in
16614+0 records out
17012736 bytes (17 MB) copied, 0.29308 s, 58.0 MB/s
Creating file no 2
14456+0 records in
14456+0 records out
14802944 bytes (15 MB) copied, 0.100101 s, 148 MB/s
.................
Creating file no 10
25224+0 records in
25224+0 records out
25829376 bytes (26 MB) copied, 0.492113 s, 52.5 MB/s

when you do a directory listing, you’ll see this:-

[root@atreides rd-test]# ls -lh
total 226M
-rw-r--r-- 1 root root 17M 2009-07-29 00:25 random-file.1
-rw-r--r-- 1 root root 25M 2009-07-29 00:25 random-file.10
-rw-r--r-- 1 root root 15M 2009-07-29 00:25 random-file.2
-rw-r--r-- 1 root root 20M 2009-07-29 00:25 random-file.3
-rw-r--r-- 1 root root 21M 2009-07-29 00:25 random-file.4
-rw-r--r-- 1 root root 30M 2009-07-29 00:25 random-file.5
-rw-r--r-- 1 root root 22M 2009-07-29 00:25 random-file.6
-rw-r--r-- 1 root root 27M 2009-07-29 00:25 random-file.7
-rw-r--r-- 1 root root 25M 2009-07-29 00:25 random-file.8
-rw-r--r-- 1 root root 29M 2009-07-29 00:25 random-file.9

For more info, refer to the $RANDOM function from the Advanced Bash Scripting Guide.

How to Enable Flash in Google Chrome for Fedora 11 i686

So you’ve installed the latest development builds of Chromium for Fedora Linux. The only thing lacking is that youtube is not loading. If not, what are you waiting for? Refer to one of my previous posts.

First make sure you install the Adobe yum repository.

As root:-

rpm -ivh http://linuxdownload.adobe.com/adobe-release/adobe-release-i386-1.0-1.noarch.rpm

Then install the flash plugin.

yum install flash-plugin.i386

The flash plugin will be installed in /usr/lib/flash-plugin/libflashplayer.so

Chrome plugins are supposed to be located in /usr/lib/chromium-browser/plugins. So lets create a symbolic link, so that whenever your Adobe flash plugin is updated, your Chrome automatically gets the latest version.

n -sv /usr/lib/mozilla/plugins/libflashplayer.so /usr/lib/chromium-browser/plugins/libflashplayer.so

Now start your Chrome browser from the command line, with the --enable-plugins switch. This will enable the flash plugin.

Here’s a screenshot of me watching a video from youtube on chrome:-

Flash Player in Google Chrome for Linux

Flash Player in Google Chrome for Linux

Say goodbye to geek productivity once this is done :D

Temporarily Disable a Yum Repository

Sometimes, you may run into problems updating Fedora via yum. What you could do is temporarily disable the offending repository, and update everything else while the errors are being fixed upstream.

  1. List out your enabled yum repositories.
    $ yum repolist

    You would get something like this:

    [raja@atreides ~]$ yum repolist
    Loaded plugins: fastestmirror, presto, refresh-packagekit
    repo id                   repo name                              status
    adobe-linux-i386          Adobe Systems Incorporated             enabled:     17
    chromium                  Chromium Test Packages                 enabled:      7
    fedora                    Fedora 11 - i386                       enabled: 13,289
    rpmfusion-free            RPM Fusion for Fedora 11 - Free        enabled:    377
    rpmfusion-free-updates    RPM Fusion for Fedora 11 - Free - Upda enabled:    210
    rpmfusion-nonfree         RPM Fusion for Fedora 11 - Nonfree     enabled:    110
    rpmfusion-nonfree-updates RPM Fusion for Fedora 11 - Nonfree - U enabled:    115
    updates                   Fedora 11 - i386 - Updates             enabled:  3,451
    repolist: 17,576
  2. The names on the left are the repo ids. To disable a particular one, for example rpmfusion-nonfree-updates while doing updates, run your yum like this:-
    # yum upgrade --disablerepo=rpmfusion-free-updates
  3. This will help you upgrade the rest of the packages in the mean time.

Configure a Caching-Only Name Server in a Chroot Environment for Fedora 11

Having a caching only name-server on your local Machine speeds up your browsing. Here’s how to set up a slightly more secure caching server using ISC Bind in Fedora 11.

  1. Install bind and bind-chroot packages
    # yum install bind bind-chroot
  2. Edit your /etc/sysconfig/named file.
    # vim /etc/sysconfig/named

    Add the following line:

    ROOTDIR="/var/named/chroot"

  3. Edit your /etc/named.conf file.
    # vim /etc/named.conf
  4. Change the following line:
    listen-on port 53 { 127.0.0.1; };

    to

    listen-on port 53 { any; };

    This allows the bind daemon to listen on all your network IPs, not just your loopback(127.0.0.1) address.

  5. Change this line:
    allow-query     { localhost; };

    to

    allow-query     { 192.168.0.0/24; };

    You now allow all the machines in your home LAN to use your DNS server.

  6. Make sure it starts at boot time.
    # chkconfig named on

    Restart your DNS server.

    # service named restart
  7. Make sure its listening on the correct ports.
    # netstat -ntupl | grep named

    In my case, the DNS server IP is 192.168.0.10. So, as seen here, the line udp 0 0 192.168.0.10:53 0.0.0.0:* 2851/named shows it is listening correctly.

  8. Then test your server from another machine in your network. Most probably another linux box or laptop.
    # dig @192.168.0.10 google.com

    The dig command, with the ‘@’ instructs it to get the IP address for google.com from your newly set up server. On my machine, it looked like this:-

    [root@atreides ~]# dig @192.168.0.10 google.com
     
    ; < <>> DiG 9.6.1-RedHat-9.6.1-2.fc11 < <>> @192.168.0.10 google.com
    ; (1 server found)
    ;; global options: +cmd
    ;; Got answer:
    ;; ->>HEADER< <- opcode: QUERY, status: NOERROR, id: 6515
    ;; flags: qr rd ra; QUERY: 1, ANSWER: 3, AUTHORITY: 4, ADDITIONAL: 0
     
    ;; QUESTION SECTION:
    ;google.com.			IN	A
     
    ;; ANSWER SECTION:
    google.com.		300	IN	A	74.125.127.100
    google.com.		300	IN	A	74.125.45.100
    google.com.		300	IN	A	74.125.67.100
     
    ;; AUTHORITY SECTION:
    google.com.		171853	IN	NS	ns3.google.com.
    google.com.		171853	IN	NS	ns1.google.com.
    google.com.		171853	IN	NS	ns2.google.com.
    google.com.		171853	IN	NS	ns4.google.com.
     
    ;; Query time: 82 msec
    ;; SERVER: 192.168.0.10#53(192.168.0.10)
    ;; WHEN: Sat Jul 18 20:14:59 2009
    ;; MSG SIZE  rcvd: 148

    Note the SERVER: line. that shows you the answer for the query came from my DNS server (192.168.0.10).

  9. Finally, set up your /etc/resolv.conf accordingly.

    On the server:

    nameserver 127.0.0.1

    And on all your other machines:

    nameserver 192.168.0.10

Change Your MAC Address in Linux

Some times it necessary to change your network MAC address on your linux box.

Unlike Windows, which requires some registry editing or even specialized tools, its just a command line away in Linux.

For example, lets change the MAC address of your first netwrok card, usually called eth0.

Let’s swith to being the root user and view the current MAC address:-

ifconfig eth0

You should get something similiar:-

[root@rhel-5new ~]# ifconfig eth0
eth0      Link encap:Ethernet  HWaddr 08:00:27:2C:D2:B5
          inet addr:192.168.0.108  Bcast:192.168.0.255  Mask:255.255.255.0
          inet6 addr: fe80::a00:27ff:fe2c:d2b5/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:20545 errors:0 dropped:0 overruns:0 frame:0
          TX packets:18348 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:10083277 (9.6 MiB)  TX bytes:7489363 (7.1 MiB)
          Interrupt:11 Base address:0xd020

The MAC address is listed next to the HWAddr column, which is 08:00:27:2C:D2:B5 in this case.

We’ll change it. First shutdown the network interface.

ifconfig eth0 down

Now change it to say, 08:00:27:2C:D2:B4.

ifconfig eth0 hw ether 00:00:27:2c:d2:b4

Bring the interface back up.

ifconfig eth0 up

Finally, lets have a look at the new mac address.

[root@rhel-5new ~]# ifconfig eth0
eth0      Link encap:Ethernet  HWaddr 00:00:27:2C:D2:B4
          inet addr:192.168.0.108  Bcast:192.168.0.255  Mask:255.255.255.0
          inet6 addr: fe80::200:27ff:fe2c:d2b4/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:20957 errors:0 dropped:0 overruns:0 frame:0
          TX packets:18642 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:10146055 (9.6 MiB)  TX bytes:7520724 (7.1 MiB)
          Interrupt:11 Base address:0xd020

You can see that the HWaddr field now reports 00:00:27:2C:D2:B4.

Find a Cheap VPS Hosting Provider

With VPS hosting becoming a more popular option among the techies, I’ve been tempted to get one.
Among the reasons you’ve like to get a VPS include:-

  • You need root access
  • You wish to install a distro of your choice
  • Want more capabilities than offered by standard shared hosting
  • You have Ruby On Rails or Python Based Web Apps you want to host
  • Because you can :D

This diagram, taken from Google Trends shows VPS Hosting search to be picking up since late 2004. I think is times pefectly with the release of Ruby On Rails to the masses.

VPS Hosting Search Trends Based on Google Trends

VPS Hosting Search Trends Based on Google Trends

Anyway, I found this site with cheap VPS hosting, some going for as low as US$3 a month. The reliability of these providers maybe questionable, and I don’t think they can compare to the big guns.

Here’s the link, do comment on your experiences.

*sigh* Now to prepare for trackbacks from bots tracking the word *VPS* :P

Check Your Spelling in Linux using the Command Line

You can do a spell check on from the command line in Linux.

First, make sure you have aspell installed. In Fedora, just yum install it.

$ yum install aspell

You’ll also need a dictionary, from which aspell can search for words.
A yum search aspell will give you a list.

[raja@atreides visilon]$ yum search aspell
-----------------------------------------------------------
aspell-af.i586 : Afrikaans dictionaries for Aspell
aspell-bg.i586 : Bulgarian dictionaries for Aspell
aspell-br.i586 : Breton dictionaries for Aspell
aspell-ca.i586 : Catalan dictionaries for Aspell
aspell-cs.i586 : Czech dictionaries for Aspell
aspell-cy.i586 : Welsh dictionaries for Aspell
aspell-da.i586 : Danish dictionaries for Aspell
aspell-de.i586 : German dictionaries for Aspell
aspell-el.i586 : Greek dictionaries for Aspell
aspell-en.i586 : English dictionaries for Aspell
aspell-es.i586 : Spanish dictionaries for Aspell
aspell-fr.i586 : French dictionaries for Aspell
aspell-ga.i586 : Irish dictionaries for Aspell
aspell-gl.i586 : Galician dictionaries for Aspell
aspell-hr.i586 : Croatian dictionaries for Aspell
aspell-id.i586 : Indonesian dictionaries for Aspell
aspell-is.i586 : Icelandic dictionaries for Aspell
aspell-it.i586 : Italian dictionaries for Aspell
aspell-no.i586 : Norwegian dictionaries for Aspell
aspell-pl.i586 : Polish dictionaries for Aspell
aspell-pt.i586 : Portuguese dictionaries for Aspell
aspell-ru.i586 : Russian dictionaries for Aspell
aspell-sl.i586 : Slovenian dictionaries for Aspell
aspell-sr.i586 : Serbian dictionaries for Aspell
aspell-sv.i586 : Swedish dictionaries for Aspell
aspell-fo.i586 : Faeroese dictionaries for Aspell
aspell-gd.i586 : Gaelic dictionaries for Aspell
aspell-nl.i586 : Dutch dictionaries for Aspell
aspell-ar.i586 : Arabic dictionary for Aspell
aspell-bn.i586 : GNU Aspell Bengali Dictionary Package
aspell-gu.i586 : GNU Aspell Gujarati Dictionary Package
aspell-he.i586 : Hebrew dictionary for Aspell
aspell-hi.i586 : GNU Aspell Hindi Dictionary Package
aspell-ml.i586 : GNU Aspell Malayalam Dictionary Package
aspell-mr.i586 : GNU Aspell Marathi Dictionary Package
aspell-or.i586 : GNU Aspell Oriya Dictionary Package
aspell-pa.i586 : GNU Aspell Punjabi Dictionary Package
aspell-sk.i586 : Slovak dictionaries for Aspell
aspell-ta.i586 : GNU Aspell Tamil Dictionary Package
aspell-te.i586 : GNU Aspell Telugu Dictionary Package

That’s a lot of languages. I’ll just install the English dictionaries.

yum install aspell-en.i586

Now the fun part. Checking your spelling via command line.
Just type aspell -a and it will give you a prompt. I’ll intentionally type a misspelled word and see what it returns:-

[raja@atreides visilon]$ aspell -a
@(#) International Ispell Version 3.1.20 (but really Aspell 0.60.6)
psychatrist
& psychatrist 5 0: psychiatrist, psychiatrists, psychiatrist's, psychiatry's, psychiatric

I spelt ‘psychiatrist’ wrongly, and it gave me back a few suggested corrections. (Ok I admit, I didn’t know how to spell it in the first place. :P )

Lets see what happens when you give it a correctly spelled word:-

[raja@atreides visilon]$ aspell -a
@(#) International Ispell Version 3.1.20 (but really Aspell 0.60.6)
disapprove
*

Yes, it returns an asterisk(*), showing you’re spelling is correct.
Press Ctrl + D when you’re done, to bring you back to the command line.

If you have a text file, you could check that using aspell as well. For example, here’s some sample text:-

Mary had a little lamb. It's fleede was whote as snow.

We’ll save it as mary.txt and run aspell like this to check the file:

aspell check mary.txt

You’ll get a interface like this, highlighting every misspelled word, with the suggested actions.

Aspell in action

Aspell in action

When done with the checking, aspell will exit, saving the previous copy with a .bak extension. :)

Stop Vim from Highlighting Search Results

You can search to words in vim using the ‘/’ key when you’re in command mode. Unfortunately, new admins in their desperation to find a word may inadvertently specify a term too generic.

Vim provides search highlighting, but sometimes, the results are just plain fugly. For example, when searching for the word module in a httpd.conf file, this is what could happen:

Vim showing Highlights

Vim showing Highlights

Yeah, not really the the most visually pleasing. To temporarily turn of highlighting, just type

:nohlsearch

in vim command mode.

The next time you search for something, or press the ‘n’ key, it will be automatically turned back on.

Police report against Al-Islam

Sigh. Never thought I’d bog about this.

Can’t people just live together?

Henotheism (Greek εἷς θεός heis theos “one god”) is a term coined by Max Müller, to mean worshiping a single god while accepting the existence or possible existence of other deities.

Fully explained here, in the one & only Wikipedia. :)

How to Mass Rename Files in Linux

When entrusted with the chore of renaming multiple file, the convenience of a script shines. After all, we ain’t robots designed to do just one thing. Today, I’ll show one method of renaming files, using a for loop in bash.

First, the task we are going to do:

We have a list of files:

[raja@atreides test]$ ls
data_file_1   data_file_2  data_file_4  data_file_6  data_file_8
data_file_10  data_file_3  data_file_5  data_file_7  data_file_9

We have been instructed to replace the underscores (_) in the file names with hyphens (-).

A menial task not really suited for humans. We’ll start building a script for this, one piece at a time.

  1. Check which type of shell you are using.
    echo $SHELL

    As I am using bash scripting, make sure the result is /bin/bash

  2. List out all the file that you wish to rename
    $ ls data_file_*

    It should give you output like this:-

    [raja@atreides test]$ ls data_file_*
    data_file_1   data_file_2  data_file_4  data_file_6  data_file_8
    data_file_10  data_file_3  data_file_5  data_file_7  data_file_9
  3. Lets display each one of this in a for loop.
    for file_name in `ls data_file_*`; do echo "The file:" $file_name; done

    What this for loop does is take each of the file names given by the ls command and store it in the $file_name bash variable. You’ll see this:-

    [raja@atreides test]$ for file_name in `ls data_file_*`; do echo "The file:" $file_name; done
    The file: data_file_1
    The file: data_file_10
    The file: data_file_2
    The file: data_file_3
    The file: data_file_4
    The file: data_file_5
    The file: data_file_6
    The file: data_file_7
    The file: data_file_8
    The file: data_file_9
  4. Now that we’ve stored the file names in variables, we can use the sed command to edit it, replacing the underscores with a hyphen. I’ll echo the file name and pipe it to sed, to make sure it replaces the file name, and not modify the files itself.

    I used the sed replace command, s with a global option behind, the g, to make sure it replaces all occurrences of the underscore. Without the ‘g’, sed will only replace the first underscore and ignore the rest.

    for file_name in `ls data_file_*`; do echo "The file:" $file_name; echo $file_name | sed 's/_/-/g' ; done

    When you run this command in your console, you will see this output:

    [raja@atreides test]$ for file_name in `ls data_file_*`; do echo "The file:" $file_name; echo $file_name | sed 's/_/-/g' ; done
    The file: data_file_1
    data-file-1
    The file: data_file_10
    data-file-10
    The file: data_file_2
    data-file-2
    The file: data_file_3
    data-file-3
    The file: data_file_4
    data-file-4
    The file: data_file_5
    data-file-5
    The file: data_file_6
    data-file-6
    The file: data_file_7
    data-file-7
    The file: data_file_8
    data-file-8
    The file: data_file_9
    data-file-9
  5. Of course, all this script is doing right now is editing the file name in the $file_name variable itself. Even though you can see what the modified file name will look like, we have not actually renamed the files on your hard disk yet. (This can be a good method of testing your sed-fu, before you really do it ;) )
  6. In order to actually rename the file, I’ll put the results of sed’s transformation in a variable called $new_file_name.I’ll also print this out to the terminal, so you can see the changes that are about to take place.
    for file_name in `ls data_file_*`; do echo "The file:" $file_name; new_file_name=$(echo $file_name | sed 's/_/-/g'); echo "New name:" $new_file_name ; done

    You’ll then see this being output on the terminal:

    [raja@atreides test]$ for file_name in `ls data_file_*`; do echo "The file:" $file_name; new_file_name=$(echo $file_name | sed 's/_/-/g'); echo "New name:" $new_file_name ; done
    The file: data_file_1
    New name: data-file-1
    The file: data_file_10
    New name: data-file-10
    The file: data_file_2
    New name: data-file-2
    The file: data_file_3
    New name: data-file-3
    The file: data_file_4
    New name: data-file-4
    The file: data_file_5
    New name: data-file-5
    The file: data_file_6
    New name: data-file-6
    The file: data_file_7
    New name: data-file-7
    The file: data_file_8
    New name: data-file-8
    The file: data_file_9
    New name: data-file-9
  7. Now that we’ve got both the old AND new file names, nicely stored in variables, its time to actually rename them using the mv command. The syntax for renaming is mv current_file_name new_file_name. I think you’re starting to see where these variables come in handy. :D
    for file_name in `ls data_file_*`; do echo "The file:" $file_name; new_file_name=$(echo $file_name | sed 's/_/-/g'); echo "New name:" $new_file_name ; mv -v $file_name $new_file_name; done

    I’ve added a -v to the mv, so it displays what it is changing. As I always say, a little more verbosity is always good, especially on the terminal. Your final output will be something like this:-

    [raja@atreides test]$ for file_name in `ls data_file_*`; do echo "The file:" $file_name; new_file_name=$(echo $file_name | sed 's/_/-/g'); echo "New name:" $new_file_name ; mv -v $file_name $new_file_name; done
    The file: data_file_1
    New name: data-file-1
    `data_file_1' -> `data-file-1'
    The file: data_file_10
    New name: data-file-10
    `data_file_10' -> `data-file-10'
    The file: data_file_2
    New name: data-file-2
    `data_file_2' -> `data-file-2'
    The file: data_file_3
    New name: data-file-3
    `data_file_3' -> `data-file-3'
    The file: data_file_4
    New name: data-file-4
    `data_file_4' -> `data-file-4'
    The file: data_file_5
    New name: data-file-5
    `data_file_5' -> `data-file-5'
    The file: data_file_6
    New name: data-file-6
    `data_file_6' -> `data-file-6'
    The file: data_file_7
    New name: data-file-7
    `data_file_7' -> `data-file-7'
    The file: data_file_8
    New name: data-file-8
    `data_file_8' -> `data-file-8'
    The file: data_file_9
    New name: data-file-9
    `data_file_9' -> `data-file-9'

    An ls will reveal that all files have really been changed this time.

    [raja@atreides test]$ ls
    data-file-1   data-file-2  data-file-4  data-file-6  data-file-8
    data-file-10  data-file-3  data-file-5  data-file-7  data-file-9

    This should give you an idea on how to mass rename files. Of course there are probably better ways. Please do comment if you know some.