Linux Bash Scripting

[ViLLaN]

Hey Guys..

Im gonna request that we have a linux forum.. But until that happens, Im gonna post any scripts that I write that might be of assistance to anyone (and I encourage you all to do the same).. I script most things I do on a daily basis to limit having to do anything more than once, and I figure the effort should help out someone ;P

Also, if anyone wants to add to the script, suggested differents ways of accomplishing things.. Feel free to add it.

[ViLLaN]

Script for doing iinet disconnection count.. Simply go to the iinet toolbox, copy the information about disconnections to a text file called loglist.txt and run the script in the same directory.. (If anyone know how to automate retrieval of the information from the web site, Id love to add it).

---------------------------------------
#!/bin/bash
themonth=$(date +%B)
theday=1
currentday=$(date +%d)

echo -= Daily Disconnections =-
while (($theday <= $currentday)); do
daycount=$(gawk -F" " '{ print $4" "$5}' loglist.txt | grep "$theday $themonth" -c)
if (($theday < 10))
then
echo 0$theday $themonth = $daycount
else
echo $theday $themonth = $daycount
fi
theday=$((theday+1))
done


-----------------------------------

Output should look something like this -

-= Daily Disconnections =-
01 Feb = 3
02 Feb = 10
03 Feb = 3
04 Feb = 1
05 Feb = 3
06 Feb = 1
07 Feb = 0
08 Feb = 2
09 Feb = 2
10 Feb = 0
11 Feb = 1
12 Feb = 9
13 Feb = 1
14 Feb = 1
15 Feb = 1
16 Feb = 0

[ViLLaN]

Quick script for checking all those core dump files on your machine and finding out what caused the crash. Just run the script in the same directory as the core files.

---------------------------

#!/bin/bash
FILES=$(ls *core*)

for file in $FILES
do
echo -= $file =-
gdb --core=$file --batch
echo
done

-----------------------------

Output should look something like this (shown here with two core files) -

-= old.core =-
Core was generated by `/usr/raptor/bin/httpd -restart-listen-cookie=4,7/5000/1,8/8081/1,9/80/1,10/443/'.
Program terminated with signal 11, Segmentation fault.
#0 0x40127438 in ?? ()

-= old.core.20989 =-
Core was generated by `/usr/raptor/bin/httpd -restart-listen-cookie=4,7/5000/1,8/8081/1,9/80/1,10/443/'.
Program terminated with signal 11, Segmentation fault.
#0 0x40127441 in ?? ()

[ViLLaN]

Script I wrote for searching the data in all the core dump files in a directory for a certain word, and showing a number of lines before and after that word. Could be easily modified for many other uses (for example I have a version that checks IRC logs for bans / kicks daily and emails me a report..).. Ill post that one later.

------------------------------------------------

#!/bin/bash
FILES=$(ls *core*)

for file in $FILES
do
echo -= $file =-
xxd $file | grep raptor -n -B2 -A2
echo
done

----------------------------------------------------

Output will look something like -

-= old.core =-
84-0000530: 0514 0000 0100 0000 b20c 0000 be00 0000 ................
85-0000540: 6874 7470 6400 0000 0000 0000 0000 0000 httpd...........
86:0000550: 2f75 7372 2f72 6170 746f 722f 6269 6e2f /usr/raptor/bin/
87-0000560: 6874 7470 6420 2d72 6573 7461 7274 2d6c httpd -restart-l
88-0000570: 6973 7465 6e2d 636f 6f6b 6965 3d34 2c37 isten-cookie=4,7
--
201449-0312e80: 0000 0000 9c6e 0140 1000 0000 0100 0000 .....n.@........
201450-0312e90: 0100 0000 0100 0000 0200 0000 2f75 7372 ............/usr
201451:0312ea0: 2f72 6170 746f 722f 6269 6e2f 0000 0000 /raptor/bin/....
201452-0312eb0: 786e 0140 dc37 0140 0000 0000 d46e 0140 xn.@.7.@.....n.@
201453-0312ec0: 0000 0000 0200 0000 0200 0000 0200 0000 ................
--
407008-0635df0: 382f 3830 3831 2f31 2c39 2f38 302f 312c 8/8081/1,9/80/1,
407009-0635e00: 3130 2f34 3433 2f31 0050 5744 3d2f 7573 10/443/1.PWD=/us
407010:0635e10: 722f 7261 7074 6f72 2f62 696e 0042 4f4f r/raptor/bin.BOO
407011-0635e20: 545f 4649 4c45 3d2f 626f 6f74 2f73 796d T_FILE=/boot/sym
407012-0635e30: 616e 7465 632f 766d 6c69 6e75 782e 677a antec/vmlinux.gz

[ViLLaN]

This script I use on directories that have more files in it than rf can handle in one go.. But be VERY careful with it.. Run this script in a directory, and everything in or below it will be destroyed.

------------------------------

#!/bin/bash
FILES=$(ls)

for file in $FILES
do
rm -rf $file
done

-----------------------------

Im not going to show you the output ;P haha

[ViLLaN]

For anyone thats interested.. Alot of these scripts could actually be done from one command line if you ever needed to..

Example -

"for file in $(ls *core*); do gdb --core=$file --batch;echo; done"

:P

[Kaygo]

I'm going to steal a line from KaneH for here...

"You're all nerds and I have no idea whats going on."

Anyway...

I'll try and remember to find a spot for this forum, the place is getting kinda bloated though :P

[ViLLaN]

:P

[SilverShaft]

If anyone know how to automate retrieval of the information from the web site, Id love to add it

use wget to access a url and retrieve the data.

consult the man pages for more info can't quite remember atm cause I ain't use it for like 2 years

[SilverShaft]

Ill try and dig up a couple of my old useful scripts and dust off my bash scriptin hat to help out

I can give you some perl scripts also that could come in handy, perl binaries need to be present and in path to use however just in case your not workin from your own personal machine.

[BeStRaFe]

linux form would rock

[ViLLaN]

Originally posted by SilverShaft@Feb 16 2005, 06:38 AM
use wget to access a url and retrieve the data.

consult the man pages for more info can't quite remember atm cause I ain't use it for like 2 years
72726


Hey SilverShaft..

Ive actually been trying to do it with wget.. Its made harder by the fact that the information I need is output from a CGI script, thats started by choosing an option from a drop down box.. I havent found a way to directly connect to the URL :(

[ViLLaN]

Originally posted by SilverShaft@Feb 16 2005, 06:42 AM
Ill try and dig up a couple of my old useful scripts and dust off my bash scriptin hat to help out

I can give you some perl scripts also that could come in handy, perl binaries need to be present and in path to use however just in case your not workin from your own personal machine.
72727


Any scripting is welcome.. Im actually quite interested in perl as well :P

[SilverShaft]

errm from memory I think wget will only be able to retrieve info from static content providers or GET retrieval if you orchestrate your own URL.

I may be wrong about this however as I said you need to read up a bit

If its a POST retrieval and wget doesn't work I can write you a perl script to get the info

[KaneH]

what kaygo said

[MaJ]

very kew Vill

cheers.

[ViLLaN]

I wrote this code today, due to a problem I ran into.. Being that my home connection is on a dynamic IP, and I run MRTG to monitor net traffic at home, the changing of the external IP address of my router stops MRTG from reporting properly.. So I wrote code that runs every 5 minutes (along with the no-ip update client that I use) and makes sure that the mrtg cfg file is up to date, if it isnt, it changes the settings so they are correct and re runs it..

-------------------------------------------------------------

#!/bin/bash
newip=$(host vi11an.com | gawk -F" " '{ print $4}')
oldip=$(cat /etc/oldip.txt)
newmodip=$(echo $newip | gawk -F"." '{ print $1"-"$2"-"$3"-"$4 }')
oldmodip=$(echo $oldip | gawk -F"." '{ print $1"-"$2"-"$3"-"$4 }')

echo Old IP = $oldip
echo New IP = $newip

if [ "$newip" != "$oldip" ]
then
sed -e 's/$oldip/$newip/' -e 's/$oldmodip/$newmodip/' /etc/mrtg/mrtg.cfg > /etc/mrtg/mrtg2.cfg
mv -f /etc/mrtg/mrtg.cfg /etc/mrtg/mrtg.backup
mv -f /etc/mrtg/mrtg2.cfg /etc/mrtg/mrtg.cfg
env LANG=C mrtg /etc/mrtg/mrtg.cfg
indexmaker --output /var/www/mrtg/index.html --columns=1 /etc/mrtg/mrtg.cfg --prefix=data/
echo $newip > /etc/oldip.txt
echo "Config files updated and refreshed"
else
echo "IP's match, no need to update"
fi

----------------------------------

Output should look like one of the following -

If the IP hasnt changed -

Old IP = 203.206.51.56
New IP = 203.206.51.56
IP's match, no need to update

If they have -

Old IP = 203.206.51.56
New IP = 203.206.51.57
Config files updated and refreshed

[coogle]

Make up some Overall Subdirectories.

Ie Web Scripts
File Scripts etc,

Then put in descriptive name for each new topic/script.. will make easy searching after. :)

I think I have to try the dark side soon.

[ViLLaN]

Might be an idea.. I'll do it when I get home ;P

[ViLLaN]

Haha.. Gg maj :)

[ViLLaN]

SGS (Symantec Gateway Security) - Config Backup script I wrote for a customer -

#!/bin/bash
#dateadd = The suffix to the config when it is saved. Generally the date is the easiest way to identify.
#numadd = The start of a counter used for checking whether previous backups allready exist.
#backupdirectory = The directory you have created to store your backups, no slahes need be included.
#cfgfilename = The filename user for the backup.
#password = The password that the cfg is saved with.
#
#Please note that you need to add a cronjob in the format of "15 0 * * Fri /cfgbackup/backup.sh" or something similar.

PATH=/usr/raptor/bin:/sbin:/bin:/usr/sbin:/usr/bin:/usr/local/bin:/root/bin
dateadd=$(date +%d%b%y)
numadd=1
backupdirectory=cfgbackup
cfgfilename=cfgbackup
password=$dateadd

if [ -a /$backupdirectory/$cfgfilename.$dateadd ]
then
while [ -a /$backupdirectory/$cfgfilename.$dateadd.$numadd ]
do
numadd=$(($numadd+1))
done
saveconfig -save /$backupdirectory/$cfgfilename.$dateadd.$numadd -key
$password
else
saveconfig -save /$backupdirectory/$cfgfilename.$dateadd -key $password
fi

[ViLLaN]

Script for checking status of IDLE IRC character from command line :)

-------------------------------------

#!/bin/bash

wget http://ausgame.com/irpg/db.php

if [ $1 > /dev/null ]
then
grep "$1" db.php -A31 | gawk -F"td" '{ print $2 }' | sed -e 's/ nowrap>//' -e 's/>//' -e 's/<\///' > temp.txt
else
wget http://ausgame.com/irpg/players.php
echo "Please choose from the list of players below"
echo "--------------------------------------------"
grep "player=" players.php | gawk -F"=" '{ print $5 }' | sed -e 's/"/-/' | column -t -s - | gawk -F" " '{ print $1 }'
rm -f *players*
rm -f db.php*
exit 0
fi

rm -f db.php*

{
read a1
read a2
read a3
read a4
read a5
read a6
read a7
read a8
read a9
read a10
read a11
read a12
read a13
read a14
read a15
read a16
read a17
read a18
read a19
read a20
read a21
read a22
read a23
read a24
read a25
read a26
read a27
read a28
read a29
read a30
read a31
read a32
} < temp.txt

echo
echo "Idle Channel Stats"
echo "------------------"
echo -e User "\t\t" : $a1
echo -e Level "\t\t" : $a2
echo -e Admin "\t\t" : $a3
echo -e Class "\t\t" : $a4
echo -e TTL "\t\t" : $a5
echo -e Nick!User@Host "\t" : $a6
echo -e Online "\t\t" : $a7
echo -e Total Time Idled"" : $a8
echo
{
echo -e X Pos "/" : $a9 "/" Amulet "/" : $a21
echo -e Y Pos "/" : $a10 "/" Charm "/" : $a22
echo -e Mesg Pen. "/" : $a11 "/" Helm "/" : $a23
echo -e Nick Pen. "/" : $a12 "/" Boots "/" : $a24
echo -e Part Pen. "/" : $a13 "/" Gloves "/" : $a25
echo -e Kick Pen. "/" : $a14 "/" Ring "/" : $a26
echo -e Quit Pen. "/" : $a15 "/" Leggings "/" : $a27
echo -e Quest Pen. "/" : $a16 "/" Shield "/" : $a28
echo -e LOGOUT Pen. "/" : $a17 "/" Tunic "/" : $a29
echo -e Total Pen. "/" : $a18 "/" Weapon "/" : $a30
echo -e Acct. Created "/" : $a19 "/" Sum "/" : $a31
echo -e Last Login "/" : $a20 "/" Alignment "/" : $a32
} | column -t -s /

rm -f temp.txt

exit 0

[MaJ]

Originally posted by ViLLaN@Mar 4 2005, 04:15 PM
Haha.. Gg maj :)
74644


lol i didnt write it.. i just used it :)

[ViLLaN]

Hmm.. This is a group of scripts that Im currently working on.. Im going to tidy them up, this is just a rough copy :P

Ive been trying to write a set of scripts to gather information from the iinet toolbox on the website.. I now have the scripts working and just need to tidy them up ;P

SCRIPT 1 - Gathers the data from the web site. Dumps it as a text file called temp.txt
----------------------------------------------------------

#!/bin/bash
curl -o temp.html -o temp2.html -d "username=username&password=password&submit=Log+In&action=login" https://toolbox.iinet.net.au https://toolbox.iinet.net.au/cgi-bin/chisto...me=&domain=none (https://toolbox.iinet.net.au/cgi-bin/chistory.cgi?service_oid=enteroid&pusername=username&tusername=&domain=none)

sleep 20

lynx --dump temp2.html > temp.txt
cat temp.txt

rm -f temp.html
rm -f temp2.html

------------------------------------------------------------

SCRIPT 2 - Looks at the output from that file, and creates another script for reading the file (ive got plans for this data, which you will see some point in the future). This is why Im creating this weird script.

-------------------------------------------------------------

#!/bin/bash
linecount=$(cat temp.txt -n | tail -n1 | gawk -F" " '{print $1}')
headingline=$(grep "Logged In" temp.txt -n | gawk -F":" '{ print $1 }')
startline=$(($headingline + 2))
totalline=$(grep "Total time online" temp.txt -n | gawk -F":" '{ print $1 }')
endline=$(($totalline - 1))

rm -f temp.sh
echo "#/bin/bash" > temp.sh
echo "{" >> temp.sh

for i in $(seq $linecount)
do
echo "read a$i" >> temp.sh
done
echo "} < temp.txt" >> temp.sh
echo >> temp.sh

for i in $(seq $startline $endline)
do
echo "echo \$a$i" >> temp.sh
done

chmod +x temp.sh

-----------------------------------------------------

SCRIPT 3 - This is actually created by the second script.. Just including it here so you can see what to expect. Ive cut it down to make it fit ;P

-----------------------------------------------------

{
read a1
read a2
read a3
read a4
read a5
read a6
read a57
read a58
read a59
read a69
read a70
} < temp.txt

echo $a23
echo $a24
echo $a25
echo $a26
echo $a27
echo $a28
echo $a35
echo $a36
echo $a37
echo $a38
echo $a39

[ViLLaN]

Updated.. Complete script for downloading the disconnections from the iinet toolboxes and counting them..

--------------------------------------

#!/bin/bash

# First part of the script downloads the information from the iinet toolbox
# and saves it to two tempoary files temp.html and temp2.html.

username=
password=
serviceoid=

curl -o temp.html -o temp2.html -d "username=$username&password=$password&submit=Log+In&action=login" https://toolbox.iinet.net.au https://toolbox.iinet.net.au/cgi-bin/chisto...me=&domain=none (https://toolbox.iinet.net.au/cgi-bin/chistory.cgi?service_oid=$serviceoid&pusername=$username&tusername=&domain=none)


sleep 10

# temp2.html is the file that contains the disconnection information and
# we use lynx to convert the HTML to a straight text file.

lynx --dump temp2.html > temp.txt

# We then remove the two tempoary html files that we no longer require.

rm -f temp.html
rm -f temp2.html

# Second part of the script creates a temporary shell script that crops
# the temp.txt file, and outputs simple the data we need.

# The variables created below, use various methods to determine where the
# usefull data starts and ends in the file, and then outputs only that
# section of the file.

linecount=$(cat temp.txt -n | tail -n1 | gawk -F" " '{print $1}')
headingline=$(grep "Logged In" temp.txt -n | gawk -F":" '{ print $1 }')
startline=$(($headingline + 2))
totalline=$(grep "Total time online" temp.txt -n | gawk -F":" '{ print $1 }')
endline=$(($totalline - 1))

# Below we remove the temp.sh file allready there (in case left over from an
# earlier run), and start creating a new one.

echo "#/bin/bash" > temp.sh
echo "{" >> temp.sh

# We create lines in the tempoary script to read in every line of the file
# into a variable.

for i in $(seq $linecount)
do
echo "read a$i" >> temp.sh
done
echo "} < temp.txt" >> temp.sh
echo >> temp.sh

# We add lines to the script so that only the lines we require are echod.

for i in $(seq $startline $endline)
do
echo "echo \$a$i" >> temp.sh
done

# We make the temp.sh file executable.

chmod +x temp.sh

# We run the temp.sh file and it outputs the cropped data we require.

./temp.sh > loglist.txt

# We setup new variables which include the day and the month.

themonth=$(date +%B)
month=$(gawk -F" " '{ print $5}' loglist.txt | uniq | head -n1)

# We then create a program flow control to decide whether you are looking
# at data from the current month or a previous month.

if [ "$month" != "$themonth" ]
then
theday=1
currentday=$(date +%e)
echo
echo -= Daily Disconnections =-
for i in $(gawk -F" " '{ print $4}' loglist.txt | uniq | tac); do
daycount=$(gawk -F" " '{ print $3" "$4" "$5}' loglist.txt | grep " $i $month" -c)
if (($i < 10))
then
echo 0$i $month = $daycount
else
echo $i $month = $daycount
fi
theday=$((theday+1))
done
else
theday=1
currentday=$(date +%e)
echo
echo -= Daily Disconnections =-
while (($theday <= $currentday)); do
daycount=$(gawk -F" " '{ print $3" "$4" "$5}' loglist.txt | grep " $theday $themonth" -c)
if (($theday < 10))
then
echo 0$theday $themonth = $daycount
else
echo $theday $themonth = $daycount
fi
theday=$((theday+1))
done
fi

# Finally remove all the temp files.

rm -f temp.txt
rm -f loglist.txt
rm -f temp.sh

[ViLLaN]

Output should look something like this -

-----------------------------------------

[root@loki run]# ./run.sh
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 10709 0 10709 0 0 1849 0 --:--:-- 0:00:05 --:--:-- 2762
100 11179 0 11179 0 0 2164 0 --:--:-- 0:00:05 --:--:-- 2685

-= Daily Disconnections =-
01 Mar = 0
02 Mar = 0
03 Mar = 5
04 Mar = 0
05 Mar = 0
06 Mar = 2
07 Mar = 1
08 Mar = 0
09 Mar = 0
10 Mar = 0
11 Mar = 5
12 Mar = 2
13 Mar = 6
14 Mar = 6
15 Mar = 1
16 Mar = 1
[root@loki run]#

[MaJ]

thats awesome dude.

where do you find the time?

[ViLLaN]

I enjoy doing it, the time flyes :) hah

[AlCaTrAzz]

lol, feel like coding some stuff for the PD2004 website? it's PHP/SQL

[ViLLaN]

Sorry.. Perl Im learning, PHP I dont know :)

[HeXADeC]

heres a variation on linecount=
my motivation was to try and get the linecount using only shell tricks, haven't been able to do it without cat yet.

OLDIFS=$IFS;IFS=$'\n'
let linecount=0
for line in $(cat temp.txt); do
let linecount++
done
IFS=$OLDIFS

[HeXADeC]

this is better

OLDIFS=$IFS
IFS=$'\n'
let linecount=0
while read line
do
let linecount++
done < temp.txt
IFS=$OLDIFS

[ViLLaN]

You champ, that was the section of the script, I knew there had to be a better way of doing it :)

Ill give that a go straight away.

Thanks again!

[HeXADeC]

double check it with wc ,
the first snipplet didn't cooperate all the time.