Posts tagged script

Generate Files with Random Content and Size in Bash

Jul 29th

Occasionally you need to generate a bunch of random files with random content, usually for testing compression, user quotas or miscellaneous stuff.

Here’s one way, using the bash shell and a few handy linux utilities.

The bash $RANDOM function. It generates a random number between 0 – 32767.
Linux DD utility, to output files.
/dev/(h|s)da, your hard drive in linux.
All wrapped in a bash while loop.

So lets start. First define a bash variable with the number of files we wish to create, lets say 10.

no_of_files=10

Then we’ll assign the a bash variable for the counter.

counter=1

As for the dd command, this creates a file with random content from your hard disk. (Mine’s /dev/sda) which is 1KB in size. The count switch tells dd to repeat 1024 bytes 1 time, thus the 1Kb file size. skip makes dd skip an x amount of bytes before reading further. Since this requires raw access to your hard drive, you’ll have to run this as root unfortunately.

dd bs=1024 count=1 skip=0 if=/dev/sda of=random-file

So now imagine, if we let bash assign the count and skip random numbers, we get random file contents. (Light bulbs flashing eh?)

Of course, all of it will be written to a single file called random-file in my case. To add just slight amount of variety, we can add the counter variable that we will use in our while loop as an extension. The dd command will now be:-

 dd bs=1024 count=$($RANDOM) skip=$($RANDOM) if=/dev/sda of=random-file.$counter

Finally, we will wrap it up in a bash while loop like this:-

no_of_files=10;
counter=1;
while [[ $counter -le $no_of_files ]];
 do echo Creating file no $counter;
  dd bs=1024 count=$RANDOM skip=$RANDOM if=/dev/sda of=random-file.$counter;
  let "counter += 1";
 done

When you run it, you will get output like this:-

Creating file no 1
16614+0 records in
16614+0 records out
17012736 bytes (17 MB) copied, 0.29308 s, 58.0 MB/s
Creating file no 2
14456+0 records in
14456+0 records out
14802944 bytes (15 MB) copied, 0.100101 s, 148 MB/s
.................
Creating file no 10
25224+0 records in
25224+0 records out
25829376 bytes (26 MB) copied, 0.492113 s, 52.5 MB/s

when you do a directory listing, you’ll see this:-

[root@atreides rd-test]# ls -lh
total 226M
-rw-r--r-- 1 root root 17M 2009-07-29 00:25 random-file.1
-rw-r--r-- 1 root root 25M 2009-07-29 00:25 random-file.10
-rw-r--r-- 1 root root 15M 2009-07-29 00:25 random-file.2
-rw-r--r-- 1 root root 20M 2009-07-29 00:25 random-file.3
-rw-r--r-- 1 root root 21M 2009-07-29 00:25 random-file.4
-rw-r--r-- 1 root root 30M 2009-07-29 00:25 random-file.5
-rw-r--r-- 1 root root 22M 2009-07-29 00:25 random-file.6
-rw-r--r-- 1 root root 27M 2009-07-29 00:25 random-file.7
-rw-r--r-- 1 root root 25M 2009-07-29 00:25 random-file.8
-rw-r--r-- 1 root root 29M 2009-07-29 00:25 random-file.9

For more info, refer to the $RANDOM function from the Advanced Bash Scripting Guide.

bash, dd, files, generate, How To, Linux, random, script

How to Mass Rename Files in Linux

Jul 12th

Posted by raja in How To

2 comments

When entrusted with the chore of renaming multiple file, the convenience of a script shines. After all, we ain’t robots designed to do just one thing. Today, I’ll show one method of renaming files, using a for loop in bash.

First, the task we are going to do:

We have a list of files:

[raja@atreides test]$ ls
data_file_1   data_file_2  data_file_4  data_file_6  data_file_8
data_file_10  data_file_3  data_file_5  data_file_7  data_file_9

We have been instructed to replace the underscores (_) in the file names with hyphens (-).

A menial task not really suited for humans. We’ll start building a script for this, one piece at a time.

Check which type of shell you are using.
```
echo $SHELL
```
As I am using bash scripting, make sure the result is /bin/bash

List out all the file that you wish to rename

$ ls data_file_*

It should give you output like this:-

[raja@atreides test]$ ls data_file_*
data_file_1   data_file_2  data_file_4  data_file_6  data_file_8
data_file_10  data_file_3  data_file_5  data_file_7  data_file_9

Lets display each one of this in a for loop.

for file_name in `ls data_file_*`; do echo "The file:" $file_name; done

What this for loop does is take each of the file names given by the ls command and store it in the $file_name bash variable. You’ll see this:-

[raja@atreides test]$ for file_name in `ls data_file_*`; do echo "The file:" $file_name; done
The file: data_file_1
The file: data_file_10
The file: data_file_2
The file: data_file_3
The file: data_file_4
The file: data_file_5
The file: data_file_6
The file: data_file_7
The file: data_file_8
The file: data_file_9

Now that we’ve stored the file names in variables, we can use the sed command to edit it, replacing the underscores with a hyphen. I’ll echo the file name and pipe it to sed, to make sure it replaces the file name, and not modify the files itself.

I used the sed replace command, s with a global option behind, the g, to make sure it replaces all occurrences of the underscore. Without the ‘g’, sed will only replace the first underscore and ignore the rest.

for file_name in `ls data_file_*`; do echo "The file:" $file_name; echo $file_name | sed 's/_/-/g' ; done

When you run this command in your console, you will see this output:

[raja@atreides test]$ for file_name in `ls data_file_*`; do echo "The file:" $file_name; echo $file_name | sed 's/_/-/g' ; done
The file: data_file_1
data-file-1
The file: data_file_10
data-file-10
The file: data_file_2
data-file-2
The file: data_file_3
data-file-3
The file: data_file_4
data-file-4
The file: data_file_5
data-file-5
The file: data_file_6
data-file-6
The file: data_file_7
data-file-7
The file: data_file_8
data-file-8
The file: data_file_9
data-file-9

Of course, all this script is doing right now is editing the file name in the $file_name variable itself. Even though you can see what the modified file name will look like, we have not actually renamed the files on your hard disk yet. (This can be a good method of testing your sed-fu, before you really do it )

In order to actually rename the file, I’ll put the results of sed’s transformation in a variable called $new_file_name.I’ll also print this out to the terminal, so you can see the changes that are about to take place.

for file_name in `ls data_file_*`; do echo "The file:" $file_name; new_file_name=$(echo $file_name | sed 's/_/-/g'); echo "New name:" $new_file_name ; done

You’ll then see this being output on the terminal:

[raja@atreides test]$ for file_name in `ls data_file_*`; do echo "The file:" $file_name; new_file_name=$(echo $file_name | sed 's/_/-/g'); echo "New name:" $new_file_name ; done
The file: data_file_1
New name: data-file-1
The file: data_file_10
New name: data-file-10
The file: data_file_2
New name: data-file-2
The file: data_file_3
New name: data-file-3
The file: data_file_4
New name: data-file-4
The file: data_file_5
New name: data-file-5
The file: data_file_6
New name: data-file-6
The file: data_file_7
New name: data-file-7
The file: data_file_8
New name: data-file-8
The file: data_file_9
New name: data-file-9

Now that we’ve got both the old AND new file names, nicely stored in variables, its time to actually rename them using the mv command. The syntax for renaming is mv current_file_name new_file_name. I think you’re starting to see where these variables come in handy.

for file_name in `ls data_file_*`; do echo "The file:" $file_name; new_file_name=$(echo $file_name | sed 's/_/-/g'); echo "New name:" $new_file_name ; mv -v $file_name $new_file_name; done

I’ve added a -v to the mv, so it displays what it is changing. As I always say, a little more verbosity is always good, especially on the terminal. Your final output will be something like this:-

[raja@atreides test]$ for file_name in `ls data_file_*`; do echo "The file:" $file_name; new_file_name=$(echo $file_name | sed 's/_/-/g'); echo "New name:" $new_file_name ; mv -v $file_name $new_file_name; done
The file: data_file_1
New name: data-file-1
`data_file_1' -> `data-file-1'
The file: data_file_10
New name: data-file-10
`data_file_10' -> `data-file-10'
The file: data_file_2
New name: data-file-2
`data_file_2' -> `data-file-2'
The file: data_file_3
New name: data-file-3
`data_file_3' -> `data-file-3'
The file: data_file_4
New name: data-file-4
`data_file_4' -> `data-file-4'
The file: data_file_5
New name: data-file-5
`data_file_5' -> `data-file-5'
The file: data_file_6
New name: data-file-6
`data_file_6' -> `data-file-6'
The file: data_file_7
New name: data-file-7
`data_file_7' -> `data-file-7'
The file: data_file_8
New name: data-file-8
`data_file_8' -> `data-file-8'
The file: data_file_9
New name: data-file-9
`data_file_9' -> `data-file-9'

An ls will reveal that all files have really been changed this time.

[raja@atreides test]$ ls
data-file-1   data-file-2  data-file-4  data-file-6  data-file-8
data-file-10  data-file-3  data-file-5  data-file-7  data-file-9

This should give you an idea on how to mass rename files. Of course there are probably better ways. Please do comment if you know some.

bash, cli, files, How To, Linux, mass rename, rename, script, scripting, sed

Posts tagged script

Generate Files with Random Content and Size in Bash

How to Mass Rename Files in Linux

My Twits

Categories

Recent Posts

Ads