silef.de

Recursive Duplicate Finder

sebastian — Fri, 14 Jan 2011 08:55:06 +0000

Recently I needed quickly a tool which compares two directories file by file and delete duplicate copies on one side. So it checks the source directory with the copy directory file by file. Each file is compared by inode, size and md5sum. If the source file and the copy file have the same md5 hash value (but not the same inode) the file copy could be deleted via the -d parameter or moved with the -m parameter.
At the beginning I found fslint which was to fancy for this task. And at the end of this blog post I found fdupes which seems to do exactly what I needed. I love YAGN programming

#!/bin/sh
#
# dupremover - Recursive Duplicate Remover
#
# (c) 2011 by Sebastian Felis
#
# You are free to use, modify, restribute without warranty while keeping it
# free of charge. It would be nice to get an feedback note if you are using it.
#
# History
# 2011-01-14 Initial Version
#
 
SCRIPT="$0"
SRC=
COPY=
LOGLEVEL=0 # 2 DEBUG, 1 VERBOSE, 0 INFO, -1 WARN, -2 ERR
DELETE=0
DELETE_HARDLINK=0
MOVE=0
MOVEDST=".dupremover"
TRY=0
 
help() {
  echo "$(basename "$SCRIPT") [OPTIONS] SRC COPY
 
Compares files of SRC directory with COPY directory recursivly via md5sum and
deletes or moves duplicated files of the COPY directory.
 
\t-d|--delete
\t\tDelete the copy file
\t-m|--move
\t\tMoves the copy file to SRC/$MOVEDST
\t-t|--try
\t\tTry run. Do not perform any action
\t-H|--delete-hardlink
\t\tDelete multiple hardlink file
\t-L|--log-level LEVEL
\t\tSet log level: 2: DEBUG, 1: VERBOSE, 0: INFO, -1: WARN, -2: ERR
"
}
 
while [ -n "$1" ]; do
  case "$1" in
    -d|--delete)
      DELETE=1
      ;;
    -m|--move)
      MOVE=1
      ;;
    -t|--try)
      TRY=1
      ;;
    -H|--delete-hardlink)
      DELETE_HARDLINK=1
      ;;
    -L|--log-level)
      shift
      LOGLEVEL=$1
      ;;
    -h|--help)
      help
      exit 0
      ;;
    *)
      if [ -z "$SRC" ]; then
        SRC=$1
      elif [ -z "$COPY" ]; then
        COPY=$1
      fi
      ;;
  esac
  shift
done
 
if [ -z "$COPY" ]; then
  help
  exit 1
fi
 
log() {
  if [ $1 -eq 2 ]; then
    LEVEL=DEBUG
  elif [ $1 -eq 1 ]; then
    LEVEL=VERBOSE
  elif [ $1 -eq 0 ]; then
    LEVEL=INFO
  elif [ $1 -eq -1 ]; then
    LEVEL=WARN
  elif [ $1 -eq -2 ]; then
    LEVEL=ERR
  else
    LEVEL=UNKNOWN
  fi
  if [ $1 -le $LOGLEVEL ]; then
    echo $LEVEL: $2
  fi
}
 
action() {
  local FILE="$1"
  local DIR="$2"
  if [ $MOVE -ne 0 ]; then
    if [ ! -f "$FILE" ]; then
      log 1 "Skip moving of directory $FILE"
      return 0
    fi
    local DST="$SRC/$MOVEDST$DIR"
    log 0 "Move duplicate $FILE to $DST"
    if [ $TRY -eq 0 ]; then
      mkdir -p "$DST"
      mv "$FILE" "$DST"
      if [ $? -ne 0 ]; then
        log -2 "Could not move $FILE to $DST"
      fi
    fi
  elif [ $DELETE -ne 0 ]; then
    log 0 "Delete duplicate $FILE"
    if [ $TRY -eq 0 ]; then
      rm -rf "$FILE" &gt; /dev/null
      if [ $? -ne 0 ]; then
        log -2 "Could not delete $FILE"
      fi
    fi
  else
    log 0 "Do nothing for duplicate $FILE"
  fi
}
 
checkemptydir() {
  local DIR="$1"
  if [ ! -d "$DIR" ]; then
    log -1 "$DIR is not a directory"
    exit
  fi
 
  COUNT=$(ls -a "$DIR" | wc -l)
  if [ $COUNT -eq 2 ]; then
    action "$DIR"
  fi
}
 
compare() {
  local DIR="$1"
  if [ ! -d "$SRC$DIR" ]; then
    log -2 "Skip non directory $_DIR"
  fi
  ls -a "$SRC$DIR" | while read NAME; do
    if [ "$NAME" = "." -o "$NAME" = ".." ]; then
      continue
    fi
    SRCFILE="$SRC$DIR/$NAME"
    COPYFILE="$COPY$DIR/$NAME"
    if [ ! -e "$COPYFILE" ]; then
      log 1 "File $NAME only in source $SRC$DIR"
      continue
    fi
    if [ -d "$SRCFILE" ]; then
      log 1 "Read directory $SRCFILE"
      compare "$DIR/$NAME"
      checkemptydir "$COPYFILE"
      continue
    fi
    if [ $(stat -c%i "$SRCFILE") -eq $(stat -c%i "$COPYFILE") ]; then
      if [ $(ls -ld "$SRCFILE" | awk '{print $2}') -gt 1 -a $DELETE_HARDLINK -ne 0 ]; then
        log -1 "File $COPYFILE is duplicated hardlink"
      else
        log 0 "Skip file $DIR/$NAME with same inode. Use -H to delete hardlinks"
        continue
      fi
    fi
    SRCSIZE=$(stat -c%s "$SRCFILE")
    COPYSIZE=$(stat -c%s "$COPYFILE")
    if [ $SRCSIZE -ne $COPYSIZE ]; then
      log 2 "File size mismatch of {$SRC|$COPY}$DIR/$NAME $SRCSIZE != $COPYSIZE"
      continue
    fi
    SRCMD5=$(md5sum "$SRCFILE" | awk '{print $1}')
    COPYMD5=$(md5sum "$COPYFILE" | awk '{print $1}')
    if [ "$SRCMD5" != "$COPYMD5" ]; then
      log 2 "MD5 mismatch of {$SRC|$COPY}$DIR/$NAME"
      continue
    fi
    log 1 "{$SRC|$COPY}$DIR/$NAME:$SRCSIZE are equal"
    action "$COPYFILE" "$DIR"
  done
  # Get files only in the copy directory
  ls -a "$COPY$DIR" | while read NAME; do
    if [ "$NAME" = "." -o "$NAME" = ".." ]; then
      continue
    fi
    SRCFILE="$SRC$DIR/$NAME"
    COPYFILE="$COPY$DIR/$NAME"
    if [ ! -e "$SRCFILE" ]; then
      log 1 "File $NAME only in copy $COPY$DIR"
      continue
    fi
  done
}
 
compare

Coping mp3 to iPod with Rhythmbox on Ubuntu 10.10

sebastian — Sun, 07 Nov 2010 14:50:06 +0000

Recently I was unable to copy some audio files to my iPod Touch using Rhythmbox. The copy seems to be OK and I was able to playback the songs of the iPod using Rhythmbox. However, the playback from the iPod did not work. The play sign in the right top corner on the iPod was blinking and the iPod tried some next songs without playing anything. Didn’t it work some days ago?

After depressing searches I found the solution here. You need to install some extra (or simply all) gstreamer plugins with

sudo apt-get install gstreamer0.10-plugins*

and everything works fine again with Ubuntu 10.10, Rhythmbox, mp3s and the iPod again. I love open source!

Top 20 Travel Advice

sebastian — Sat, 11 Sep 2010 10:04:34 +0000

Between 2008 and 2010 my girlfriend and I did a world trip through Asia, Australia, India, Africa, and South America. Here here the summarize of our Top 20 Travel Advice

First of all: DO IT!
The most important things are your passport and your credit card – everything else is optional and replaceable!
All preparation you need for a (long term) travel is a good health insurance and the first flight ticket including required visa to your first destination country. Everything else will change during to your trip…
Do not book trips from home! They are much cheaper if you buy them local
Before you leave your country, leave the half of your luggage at home and double you money on your credit card
You need about three weeks to get in touch with one country (How much is a dollar worth, how to travel, where to sleep, where to eat)
Take some days off! Traveling and sightseeing is demanding
Monkey see – monkey do ! Adapt your behavior in the foreign country – if they cross the street on red – do it as well!
Before you enter a country get to know the exchange rate
Ask you travel fellows for advices – they might know better places than Lonely Planet
Eat in places where Moms eat with their kids. If it is good for them it is good for you
Drink water always from sealed water bottles – only in some countries you can drink water direct from the pipe
Treat locals always with respect
According beggars: Money is not everything. Having a nice and funny conversation helps a lot – especially for child beggars
Take care of your belongings in crowded places like main sights, carnival or festivals. (Did you backup your pictures before?)
Get to know the local money to detect false money (especially Chinese Yen)
Relax: The first taxi drive is mostly overpriced until you know the price (You cant know until you know)
If you do not know the price – ask different places and walk around (also helpful for bargain)
There will be bad situations – but they are temporary. The worst and coldest 9 hour bus ride on the dirtiest bump road at night has its end
Wear a money belt underneath your trousers containing the credit card and the passport
Have a fake money pocket with the daily money and some unimportant documents. Just in case you get stolen – the thief is happy with your fake wallet
Reduce money transactions as much as possible. A friend back home can do it for you. It is safer
Use a partially loaded credit card and reloaded it from time to time (by a friend back home). If your credit card gets lost not all money might be gone
For medical care: Keep your own hygienic clove and injection needle with you – you never know how bad world hospitals are
Pay always after the service (especially in Sieam Reap, Cambodia)
Label you important belongs with your contact details. Sometimes you just forget or lose your items. There are good people out there and want to contact you – at least give them a change
On bus rides: Keep the important stuff always in your hand luggage with you. On sights: leave most things in your hostel
You do not need an notebook. There are plenty cheap Internet Cafes out there and you have one item less which you have to take care of
Backup and duplicate your digital memories as much as possible to minimize data lost
Not a high end camera itself takes awesome pictures – you take the pictures. So get to know how to take pictures well
Bind the camera belt around your wrist while taking pictures against thieves
For media backup of your photos/videos: Backup your data! Make two DVD copies and ship one to your friend at home. If your friend confirms the shipping, you can trash your 2nd copy. If the shipping gets lost, create another copy and repeat these steps (BTW the shipping is faster than uploading your photos in the world and your friend can upload the pictures for you)

KlarMobil XSSurfstick W14 mit Ubuntu 10.04

sebastian — Fri, 25 Jun 2010 18:23:41 +0000

Hier möchte ich kurz beschreiben, wie man mit dem Surfstick von KlarMobil (Tarif Internet Flat500) unter Linux/Ubuntu mobil ins Internet kommt. KlarMobil liefert den UMTS Stick XSStick W14 von 4G Systems mit – doch Ubuntu bzw Linux steht weder bei KlarMobil noch bei 4G Systems auf der Supportliste. Die beschriebene Lösung benutzt usb-modeswitch und wvdial unter Ubuntu 10.04. Sie benötigt weder Windows noch Mac OSX und ist recht zuverlässig.

Die Foreneinträge Baustelle XS_Stick_W14 und mobilcom-debitelstick-t-mobile beschreiben das Problem recht ausführlich, auf denen diese Kurzbeschreibung basiert.

Man installiert die benötigten Pakete usb_modeswitch, usb-modeswitch-data und wvdial.

$ sudo apt-get install usb-modeswitch usb-modeswitch-data wvdial

Man erstellt /etc/usb-modeswitch.conf

########################################################
# 4GSystems XS Stick W10/W14
#
# USB-ID im Datentraegermodus
DefaultVendor=  0x1c9e
DefaultProduct= 0xf000
# USB-ID im Modemmodus
TargetVendor=   0x1c9e
TargetProduct=  0x9603
MessageContent="55534243123456788000000080000606f50402527000000000000000000000"

Dazu erstellt man /etc/wvdial.conf und gibt die eigene PIN der SIM Karte unter [Dialer pin] an.

[Dialer Defaults]
Init1 = AT
Baud = 460800
Phone = *99***1#
Modem = /dev/ttyUSB2
Modem Type = Analog Modem
ISDN = 0
 
[Dialer pin]
; Die eigene PIN eingeben
Init3 = AT+CPIN="1234"
 
[Dialer reset]
Init3 = ATZ
 
[Dialer umts]
Stupid Mode = on
Carrier Check = on
Init2 = ATQ0 V1 E1 S0=0 &amp;C1 &amp;D2 +FCLASS=0
Init3 = ATS0=0
Init4 = AT+COPS?
Init5 = AT+CGDCONT=1,"IP","internet.mobilcom"
ISDN = off
Password = "klarmobil"
Username = "klarmobil"
Dial Attempts = 3

Nun ist alles konfiguriert. Nach jedem Systemstart führt man das USB Modeswitch aus, um den Stick zu erkennen. (Bei mir auf Ubuntu 10.04 läuft der Stick ohne weiteren Änderungen wie in den oben genannten Foreneinträgen beschrieben wird. Man kann USB Modeswitch mit udev automatisieren, siehe dazu die Foreneinträge.):

$ sudo usb_modeswitch

Jedes mal, wenn man den erneut Stick W14 einsteckt (bzw. wenn der Stick wieder Strom bekommt), muss der UMTS Stick W14 per PIN wieder freigeschalten werden:

$ sudo wvdial pin

Danach kann man per wvdial sich ins Netz einloggen mit:

$ sudo wvdial umts

Dabei wird eine PPP Verbindung aufgebaut und man kann endlich surfen, nachdem die DNS Einträge übermittelt wurden. Unter Umständen muss man sich öfters einwählen, bis man einen vollständigen Verbindungsaufbau hat. Die Verbindung wird durch ein STRG+C beendet.

Bei mir ist der Verbindungsaufbau recht zuverlässig, auch wenn die Qualität stark variiert. Viel Spaß beim mobilen surfen!

[Update] Laut einem Kommentar ist modeswitch nicht notwendig. Siehe Kommentare.

Convert PalmDB to iCalendar

sebastian — Mon, 18 Aug 2008 13:08:05 +0000

Reasently I hat the problem to convert a palm DB to iCalendar format. But with jpilot and some scripting tricks it was possible to import the final iCalendar into SunBird.

Export the your palm calendar with iCalendar format to palm.ics

cat palm.ics | perl -n -e 's/\n/BBRR/g; print;'| \
perl -n -e 's/BBRR //g; print'|perl -n -e 's/BBRR/\n/g; print;'| \
iconv -c -f utf8 -t utf8 > palm4sunbird.ics

Import palm4sunbird.ics in SunBird.

The command converts the multi-line descriptions to single line descriptions (If you know a shortcut for this, please tell me). Afterwards the encoding is fixed to UTF8. With this it was possible to import a palm DB with over 10000 entries. Cheers.

Convert Image to DIN A4 PDF

sebastian — Mon, 18 Aug 2008 12:52:01 +0000

Following commands shows how to convert an image to an A4 pdf using ImageMagick’s convert and GhostScript.

$ convert input.jpg output.ps
$ gs -dNOPAUSE -dBATCH -sDEVICE=pdfwrite -sPAPERSIZE=a4 -sOutputFile=output.pdf output.ps
$ rm output.ps

Süddeutschlandtour

sebastian — Mon, 24 Mar 2008 20:35:43 +0000

Größere Kartenansicht

Diamond Lane

sebastian — Wed, 21 Nov 2007 19:17:16 +0000

Als ich vor kurzem in den Staaten war, habe ich mich beim Anflug auf den Flughafen gewundert, warum die linke Spur der Autobahn so leer ist. Als ich später mit dem Taxi zum Hotel fuhr, erklärte mir der Fahrer, dass es eine besondere “Car Pool” Regel gibt: Werktags dürfen auf der linken Fahrspur (gut gekennzeichnet mit einem Diamantensymbol) nur Autos mit mehreren Insassen fahren. Dh. auch besetzte Taxis. Und so fuhren wir ohne viel Stau Richtung Hotel während auf der rechten Seite die einsamen Fahrer zu ihrem Ziel dahin krochen. Eine schöne Regel (siehe auch: Wikipedia).

Flash on Ubuntu/Gusty AMD64

sebastian — Sun, 21 Oct 2007 09:40:55 +0000

OK, finally this was easy. Install the package flashplugin-nonfree and restart firefox…

$ sudo apt-get install flashplugin-nonfree

up2date

sebastian — Wed, 03 Oct 2007 15:10:31 +0000

Man muss ja schließlich mit der Zeit gehen! Somit habe ich auch meine Homepage auf ein Blog umgestellt.