Remove all installed Ruby Gems.

Posted in Bash Scripts, Linux on June 18th, 2013 by admin – Be the first to comment

This one liner will remove all of the installed gems on your systems, including all versions.

for gem in `gem list | awk '{print $1}'` ; do gem uninstall $gem -aix 2> /dev/null ; done

OpenVAS XenServer virtual appliance.

Posted in Security on October 15th, 2012 by admin – Be the first to comment

Requirements

  • Citrix XenServer 6.x (tested), should work with 5.x
  • Requires 38GB space available.

Highlights

  • CentOS 5.8 minimal (latest patches applied as of 10/14/2012) with xen kernel.
  • OpenVAS preconfigured with NVT feed.
  • Greenbone security web management (default admin user configured).

You will need to:

  • Edit network interface information.
  • Set hostname(default is: audit)
  • Change root password (default is:  rambo)
  • Change web interface admin password (default is:  admin)
Versions(via atomic repository)

openvas-administrator   –  1.2.1-2
openvas-cli   –  1.1.5-3
openvas-glib2   –  2.22.5-2
openvas-libraries   –  5.0.3-4
openvas-manager   –  3.0.3-3
openvas-scanner   –  3.3.1-2
greenbone-security-assistant  -  3.0.3-3

Download:
Archive (800Mb):  audit-openvas-xva.tar.gz

 

Mongodb Backup Script.

Posted in mongodb on May 25th, 2012 by admin – Be the first to comment

At work, I recently went on the hunt to improve our existing mongo database backups, it was setup using some script found online that really had no logging, notifications or error handling. Basically, to know whether the backup was successful or not, you would have to logon to verify the dump file existed.

I looked for an existing solution that had all the features one would want in a backup script (such as compression, archiving, exporting, performance throttling etc), but came up empty. I ultimately built an in house solution (mongosback) that’s been working very well, I’ve made it slightly more user friendly for public release.

Highlights

  • Performance throttling (low, medium, high)
  • Integrated compression (fast, medium, best)
  • Archiving (keep X days, automatically purges)
  • Backup exports (upload to FTP/SCP after)
  • In-depth logging, e-mail notifications and many configurable options.

Download on Github – Mongosback

Linux get external IP from command line.

Posted in Linux on April 19th, 2012 by admin – Be the first to comment

Since most of the ‘what is my ip’ address sites have been making constant changes to prevent wget and curl access, essentially so they can drive traffic to their site for higher ad revenue, I needed an alternative.

So without further adieu, feel free to curl, wget the following address to get your external IP address from the CLI.

curl vigeek.net/extip.php 

Making the case for LWN.net

Posted in Linux on February 23rd, 2012 by admin – Be the first to comment

As Linux continues to grow in popularity, I come across more and more users, this is always a good thing because I enjoy to discuss Linux naturally.  However, I’m consistently running into people who strive for great information, yet aren’t familiar with LWN.  I’ve always felt the need to stay as informed as possible on the direction of Linux at the lowest levels, over the years it’s been a very useful asset knowing the caveats of different kernels such as bugs, limitations and new tunable parameters (via /sys /proc sysctl etc.).  I used to religiously read the Linux kernel mailing lists, until it grew and grew until the point where sifting through the shear amount of dialoge can be quite a challenge.

After this frustration, a few years ago I setup a LWN.net account which is primary written by the well known Linux contributor Jonathan Corbet with a variety of contributors.  There are many free elements to the site, however to take full advantage, you’re going to have to pony up a little bit of money, which goes to support the site, the contributions range from $3.50 per month to $50.00 per month.

LWN aggregates and sorts some of the most important Linux information into one location, they also produce some of the most influential Linux content on the web, many LWN discussions have led to changes made in the Linux Kernel.  There is the weekly addition which covers important topics of the week including security alerts, kernel updates, distribution news, development updates (new software releases) also included is Q&A sessions, uniquely generated content and more.

I may sound like a pitchman, but LWN has served me very well over the years and it may not be for everyone, but it’s certainly worth a shot :)

Pidinfo – Statistics for individual process.

Posted in Bash Scripts, Linux on November 24th, 2011 by admin – Be the first to comment

There are so many methods to obtain information about a particular process on Linux, however many tools have one specific purpose whether it’s reporting memory, IO statistics on an individual basis.  Alternatively, there is also the /proc file system which contains a wealth of information, it too is spread out.  I put together ‘pidinfo’ to be a simple tool to grab information from various places and resources and then compiling them into organized results.  It’s generates information very quickly and provides insight into a particular process by providing statistics such as cpu usage, memory usage, IO statistics, state, open file descriptors, executor, startup command and so fourth.

I will continue to update it to add more functionality(Download):  https://github.com/vigeek/nixutils

Samba: Unable to find suitable server

Posted in Linux on September 23rd, 2011 by admin – Be the first to comment

Come across this error on occasion when joining a Linux server to a Windows domain using Samba/Winbind, has always been solvable using one of the two methods.

  • Edit /etc/hosts and create a host entry for the main IP address of the server you’re attempting to join to a domain, localhost is configured by default.
  • Specify the server name, instead of the domain.  For example if the domain is company.domain, use the full server name that is hosting the domain, i.e server.company.domain.  This can be specified in the join parameters:  net rpc join -S server.whatever.domain -U username

OFBiz/Opentaps service wrapper/startup script.

Posted in Bash Scripts on February 23rd, 2011 by admin – Be the first to comment

I was recently tasked with putting together a service wrapper for opentaps, unfortunately the official service wrapper and third party solutions are quite basic and generally just execute startofbiz.sh without anyway to legitimately determine whether the service is correctly running and responding to web requests. Executing the startofbiz.sh shell script simply reports “starting ofbiz”, it’s up to you from that point to determine whether or not the start succeeded, I wanted something more robust than this. This company in particular generates significant income on their website and heavily utilizes opentaps, some of the users executing this script would be junior level administrators therefore it was import for this script to take a lot of the guess work on what opentaps/ofbiz is doing during the start/stop phase.

The Script
Note: There are two versions of this script (I’ll be posting the other soon), one wraps around the existing stop/start scripts for opentaps/ofbiz. The other completely replaces it. Both achieve the same result.

Details: We essentially use CURL to determine whether the service is running, CURL will make a header request, if the header request delivers the desired results, it’s online and responding. Else, it’s offline. Depending on your server performance you may need to adjust the connect-timeout, but in virtually every case the default (2) should suffice.

Features:

  • debugging (optional): Features a simple ‘debug’ mode that after start, opens the console.log in more.
  • cache clear (optional): Allows the script to drop the Linux memory cache after stopping opentaps/ofbiz.
  • runuser: Allows you to run the service as any user (including root, however not recommended).
  • chkconfig: chkconfig-able (CentOS/RHEL) easily add service to start-up services. (i.e: chkconfig otaps on)
  • emailing (on): On failure or error, the script will send an e-mail with information regarding the failure.
  • error handling: Has built in error handling for most functions, will add to this shortly with traps.
  • configurable: Script can easily be edited, most values are definable.
  • distribution independent: Designed to work on any Linux distribution (tested on: CentOS/RHEL/Debian).

Functions:

  1. stats: The ‘stats’ option reports some information regarding a running opentaps/ofbiz instance. (active connections, memory usage, cpu use, runtime)
  2. start: Starts opentaps/ofbiz, utilizes curl to determine when the application is fully operational.
  3. stop: Stops opentaps/ofbiz utilizes curl to determine when the application is offline.
  4. status: Reports whether opentaps/ofbiz is running and reports the pid.
  5. restart: Simply calls the stop and start functions, allows for proper restarting.

Installation

(Assumes you’re using RHEL/CentOS)

  1. Copy the code listed below.  Or download script: otaps.sh
  2. Change into /etc/init.d (cd /etc/init.d)
  3. Open up new file in nano/vi  (nano otaps.sh)
  4. Paste all code, then save file.
  5. Edit the configurable options at the top of the script.
  6. Make file executable (chmod ug+x test.sh)
  7. Ensure it’s working properly (service otaps) should report the script usage.
  8. To enable startup/shutdown for current runlevel:  chkconfig otaps on
  9. Test & Enjoy (please report bugs/suggestions if any)
#!/bin/sh
# chkconfig: - 95 20
# description: Open Taps / OFbiz service wrapper/control script.
# Tested on:  CentOS/RHEL/Fedora
# Author:  Russ Thompson @ viGeek.net
# Objective:  Method to bring up opentaps/ofbiz instance and ensure it's fully operational via curl.

# Functions
# start - starts opentaps/ofbiz
# stop - stops opentaps/ofbiz
# restart - restarts opentaps.
# status - gets opentaps/ofbiz run status.  Aslo reports connections.
# stats - Gets number of active connections, memory usage, CPU usage, runtime etc.

# Source function library
. /etc/rc.d/init.d/functions

# Define runuser (root not recommended).  Ensure open taps/ofbiz path chowned to this user.
RUSER="svc-user"

# Set pidfile (not used in this version)
PFILE="/var/run/otaps.pid"

# OpenTaps/OFBiz installation path.
OPATH="/usr/local/opentaps"

# E-mail parameters (we e-mail on errors, failed starts etc)
EADDY="rthompson@reddit.com"
ESUB="$(hostname -s) E-mail subject message"
EFILE="tmpfile.txt"

# Host IP Address & PORT (We use this to curl to ensure instance is fully started).  Must be seperated by colon.
HIPP="69.178.69.69:8080"

# Define JAVA path (not needed if variables are exported for user).
export PATH=/opt/jdk1.6.0/bin:$PATH
export JAVA_HOME="/opt/jdk1.6.0/"
JAVA=$JAVA_HOME/bin/java

# Counter (Must be 0)
COUNTER="0"

# DEBUG (simply opens console.log after start, uses more)
DEBUG="0"

# Drop Linux memory cache after stop? May help with application performance after multiple restarts.
# Not recommended in shared environments, use at your own risk, test first.
DCACHE="0"

mail_send () {
/bin/mail -s "$ESUB" "$EADDY" < $EFILE
rm -f $EFILE
}

start() {
 if [[ -z "$(curl --connect-timeout 2 -m 1 --silent -f -I http://$HIPP | grep Apache-Coyote)" ]] ; then
# Opentaps/OFBiz is not running.  Let's bring it up.
        cd $OPATH ; if [ ! $PWD == $OPATH ] ; then echo -e "Failed to change directories on line $LINENO" | tee -a $EFILE ; mail_send ; exit 1 ; fi
        echo -e "\033[1mOpentaps/OFBiz service is currently down, bringing online...\033[0m"
        runuser -m $RUSER ./startofbiz.sh >> /dev/null
          while [[ -z "$(curl --connect-timeout 2 -m 1 --silent -f -I http://$HIPP | grep Apache-Coyote)" ]] ; do
          sleep 2
          let COUNTER+=1
             if [ $COUNTER == 30 ] ;then
                 # It has taken one minute to bring up, suspect failure, send e-mail.
                 echo -e "$(hostname) OFBiz/Opentaps failed to start in 60 seconds" | tee -a $EFILE
                 mail_send
                 exit 1
             fi
    echo -e "caught $COUNTER" >> test1.txt
          echo -n "."
          done
          echo -e "\033[0;32mOK\033[0m"
      if [ $DEBUG -eq "1" ] ; then cat /usr/local/opentaps/runtime/logs/console.log | more ; fi
 else
 echo -e "\033[1mOpentaps/OFBiz is already running....\033[0m"
 fi
}

stop() {
 if [[ -n "$(curl --connect-timeout 2 -m 1 --silent -f -I http://$HIPP | grep Apache-Coyote)" ]] ; then
    # Opentaps/OFBiz is running.... Let's stop it...
        cd $OPATH ; if [ ! $PWD == $OPATH ] ; then echo -e "Failed to change directories on line $LINENO" | tee -a $EFILE ; mail_send ; exit 1 ; fi
        echo -e "\033[1mOpentaps/OFBiz service is currently online, taking offline...\033[0m"
        runuser -m $RUSER ./stopofbiz.sh >> /dev/null
          while [[ -n "$(curl --connect-timeout 2 -m 1 --silent -f -I http://$HIPP | grep Apache-Coyote)" ]] ; do
            echo -n "."
            sleep 2
            let COUNTER+=1
               if [ $COUNTER == 30 ] ;then
              # It has taken one minute to bring up, suspect failure, send e-mail.
                   echo -e "$(hostname) OFBiz/Opentaps failed to stop in 60 seconds" | tee -a $EFILE
                   mail_send
                   exit 1
                  fi
          done
          echo -e "Result:  \033[0;32mOK\033[0m"
      if [ $DCACHE -eq "1" ] ; then sync; echo 3 > /proc/sys/vm/drop_caches | more ; fi
 else
   echo -e "\033[1mOpentaps/OFBiz is already stopped, please start....\033[0m"
 fi
}

ostatus() {
  STAT=$(ps -ef |grep "ofbiz.jar" | grep -v "grep" | awk '{ print $2}')
  if [[ -n $STAT ]] ; then
    echo -e "\033[1mOFBiz/Opentaps is running with pid:\033[0m \033[0;32m $STAT \033[0m"
    echo -e "\033[1mCurrent connection total:\033[0m" $(netstat -an |grep $(echo '$HIPP' | cut -d ":" -f2) | wc -l)
  else
    echo -e "\033[4;31mOFBiz/Opentaps service is OFFLINE\033[0m"
  fi
}

ostats() {
  STAT=$(ps -ef |grep "ofbiz.jar" | grep -v "grep" | awk '{ print $2}')
  if [[ -n $STAT ]] ; then
    echo -e "\033[1mOFBiz/Putting together open taps report for pid:\033[0m \033[0;32m $STAT \033[0m"
    echo -e "\033[1mCurrent connection total:\033[0m $(netstat -an |grep $(echo '$HIPP' | cut -d ":" -f2) | wc -l)"
    echo -e "\033[1mTotal process $STAT memory usage:\033[0m $(ps aux | grep $STAT | grep -v "grep" | awk '{ s += $6 } END { print s/1024, "Mb"}')"
    echo -e "\033[1mCurrent CPU utilization (percent):\033[0m  $(ps -eo %mem,cmd|sort -k2 -r | grep ofbiz | cut -d " " -f1)"
    echo -e "\033[1mThe process started on:\033[0m $(ls -ld /proc/$STAT | awk '{print $6,$7,$8}')"
  else
    echo -e "\033[4;31mOFBiz/Opentaps service is OFFLINE\033[0m"
  fi
}


case $1 in

start)
  start
;;
stop)
  stop
;;
restart)
  stop
  start
;;
status)
  ostatus
;;
stats)
  ostats
;;
  *)
    echo $"Usage: $0 {start|stop|restart|status|stats}"
    exit 1

# Cleanup
esac
exit 0

MySQL – Specified key was too long; max key length is 1000 bytes

Posted in mySQL on February 22nd, 2011 by admin – Be the first to comment

Chances are if you’ve worked with mySQL for a while, you’ve probably ran into Error #1071 or ‘Specified key was too long; max key length is 1000 bytes’, There seems to be a high degree of confusion regarding this error message, I’ve typically ran into this error while exporting/importing databases from different systems. i.e I’ve taken a development mySQL dump and tried importing it into our staging mySQL server.

This error is typically a relatively easy fix and unfortunately this resolution is not easy to find.

Stop mySQL

service mysqld stop or /etc/init.d/mysqld stop

Change into default mySQL data path (usually: /var/lib/mysql).

cd /var/lib/mysql

Delete both ib_logfiles (ib_logfile0 & ib_logfile1 etc).

rm -f ib_logfile*

Now simply start up mySQL and the issue should be resolved.

mySQL Slow Query Killer script.

Posted in Perl Scripts on February 22nd, 2011 by admin – 1 Comment

You’re probably wondering, why the hell do I need this? In my case our development team releases a patch that has some SQL changes that inevitably slow our mySQL cluster down to a crawl. I was given the task of finding a temporary solution while the developers worked on a fix. The solution would be, we go through the process list and kill queries that have been hung for X. Instead of reinventing the wheel, I came across a solid Perl script from ‘spe’ which achieves this task, I added some additional functionality and updated some existing functions.

This script will simply go through the process list, identify slow running queries and then kill them. The threshold and alert options are configurable.

#!/usr/bin/perl -w
# spe - 10/2006
# Russ Thompson 2010 @ viGeek.net (Added kill count, alert options, adjustable threshold, console and emailing functions)
# If leaving exposed passwords please wrap in SHcrypt

use diagnostics -verbose;
enable  diagnostics;
use DBI;
use MIME::Lite;

my $user = "theuser";
my $password = "LOLOLOL";
my $mysqladmin = "/usr/bin/mysqladmin";
my $sql = "SHOW FULL PROCESSLIST";
my $killedAQuery = 0;
my $dbhost=`/bin/hostname`;

# Enable E-mail notifications
my $enote = "0";

# Enable console output
my $ecout = "1";

# Set enable variables
my $emailalertfrom = "you\@who.com";
my $emailalertto = "you\@who.com";

# Set integer value for kill counter
my $count = 0;
# Set threshold on what defines a slow query (in seconds)
my $definq = 15;

# Set log file
my $file="slow-query.txt";

# Open our log
open FILE, ">$file" or die "unable to open $file $!";

while (1) {
        $db_handle = 0;
        while ($db_handle == 0) {
                $db_handle = DBI->connect("dbi:mysql:database=mysql;hostname=127.0.0.1:port=3306;user=".$user.";password=".$password);
                if ($db_handle == 0) {
                        sleep(1);
                }
        }

        $statement = $db_handle->prepare($sql)
            or die "Couldn't prepare query '$sql': $DBI::errstr\n";

        $statement->execute()
            or die "Couldn't execute query '$sql': $DBI::errstr\n";
        while (($row_ref = $statement->fetchrow_hashref()) && ($killedAQuery == 0))
        {
                if ($row_ref->{Command} eq "Query") {
                        if ($row_ref->{Time} >= $definq) {
                                @args = ($mysqladmin, "-u".$user, "-p".$password, "kill", $row_ref->{Id});
                                $returnCode = system(@args);
                                # Include console output and kill counts.
                                if($ecout == 1){
                                print ("Killing row ID:  $row_ref->{Id}\n");
                                $count+=1;
                                print("Total killed: $count\n");
                                }
                                $emailMessage = "A slow query as been detected (more than $row_ref->{Time} seconds). SQLkiller will try to kill this request.\nThe query is:\n$row_ref->{Info}\n\n";
                        print FILE "A slow query as been detected (more than $row_ref->{Time} seconds). SQLkiller will try to kill this request.\nThe query is:\n$row_ref->{Info}\n\n";                
       
                if ($returnCode != 0) {
                                        $emailMessage .= "Result: The SQL request cannot be killed. This SQL request is probably a fake slow query due to an another SQL request. The problematic
request is the first killed successfully\n";
                                }
                                else {
                                        $emailMessage .= "Result: The SQL request has been killed successfully\n";
                                }
                                # Establish and send e-mail
                                if($enote == 1){
                                my $msg = new MIME::Lite
                                        From    =>$emailalertfrom,
                                        To      =>$emailalertto,
                                        Subject =>'[ SQLkiller ] A query has been killed on. '.$dbhost,
                                        Type    =>'TEXT',
                                        Data    =>$emailMessage;
                                $msg -> send;
                                }
                               
                                $killedAQuery = 1;
                        }
                }
        }
        $statement->finish();
        $db_handle->disconnect();
        if ($killedAQuery == 0) {
                sleep(5);
        }
        else {
                $killedAQuery = 0;
                #sleep(1);
        }
}