Sorting and publishing images (2015, OSC15)
odp presentation (only titles)
Images setup
Cameras
We wont speak about the camera setup, but about the workflow setup when you are back to your openSUSE desktop.
Many people use to connect the camera to the computer with an usb cable. I don't like it. First you have to power on the camera, so you have to have a charged battery, then connecting any device to any other makes them at risk to break by electrical or mechanical problem.
I use to remove the memory card from the camera and insert it in the desktop reader. Most modern camera uses SD cards and SD card readers are available everywhere.
As soon as possible, I protect the camera in his bag, after all the camera is the most expensive device I have.
Image format
I use to make mostly informative pictures, like what you can see in news papers or web sites. So it's pretty usefull to have many images of the same event, nobody knowing what will be important some years in the future. Then I have no time to nitpick the content of the image and never use raw format, but always jpg. I wont say how to use The GIMP, I use it only to remove sensor dust if any (rare).
Scanners
The goal is to make jpg from films shots. Films and diapos scanners of good quality where never cheap but could be sold back after usage - I had to scan around 20.000 images for 6 month worth work. The second hand purchases becomes rare and questionable, the hardware being more and more old. I hope any good photographer have already scanned his images, for the others a cheap scanner may be enough.
If speed is not a problem, the Epson photo scanners are pretty good, but very slow. I use still mine when by chance I find a forgotten film. For color diapos, it's nearly mandatory to use "ICE", so much durst and scratches are a problem. Professional offers are up-priced.
Storage
SD cards are so cheap, nowaday than I even now keep them and use a new one when the old one is full. 32 Gb SD card (I often buy micro SD variant, nearly the same price and can be read by smartphones) are less than $20. My go with me Sony RX100 M camera have 20Mpix shots and can pack more than 3000 photos as jpg and 1000 as raw, which should be enough for some time.
I will say at first that it's mandatory to store the images exactly like there are coming from the camera. You never know what you may feel in ten years :-). Never delete anything there. I have a special tree labelled "originals" for this purpose, and an other labelled "edits" for the edited ones.
I use date sorted system. Originals are stored by creation date, Editeds are sometimes spread also by subject (added to the date). Use the computer friendly date as YYYYMMDD so sorting is natural. Make YYYYMMDD-subject if necessary.
I like write only media, because erasing them is much more difficult, but with the increasing photo size it's more difficult to use them. That said I have a Blu-Ray writer and this is enough for photo. You anyway have to use also Hard drives. A good backup/archive system may have up to 4 copies of the same file. One on the SD card (may be lost or stolen with the camera), the one working copy on the computer (the disk may broke or you may erase it by mistake), an archive copy and an extra just in case you inadvertently erase one of the other. Just in case your house burns, you have to keep one copy elsewhere (the internet is a good place).
Two months ago, having to format a SD card... I formatted a 4Tb USB drive... I was happy to have other copies. My bad, I was a bit tired...
openSUSE makes all this a snap, I simply use extensively Dolphin / Kde. I use K3b for BD writing, BUT I dislike "wodim" the default burning tools that where not updates for ages and do not even know about Blu-Ray. They can burn one, but not fix it properly, and after a long time stops with an error. Do not panic, the disk is still usable in the writer. But go to the Build service, search "cdrecord" and installs the Joerg Schilling one, this one is much faster and works cleanly with BD.
First editing: normalizing and anonymising
After having copied the photos in the "originals" folder (often under subfolders like "originals/photos/2010-2019/20150318-san-diego"), I make an other copy to the same tree but under "edits".
After that I use a first script to "normalize" and "anonymize" file names. Most of these photos will be soon or later on the web and I don't want them to be too easily stolen by robots. Some source filenames are weird, specially with smartphones or scanners. being french I often have diacritics characters in them, or white spaces. All this is BAD! Also the photo numbers are often sequential, so I add a random number in them. I do not remove the original name because it's sometime meaningful.
DigiKam
Sorting and Indexing
Working essentially with Linux (openSUSE), I tested many products and for now keep using digiKam, the Kde image processing program.
So, I import images in digiKam (the "edits" tree). In fact I sort all this in folders and have different collections depending of the date. When time passes, old folders get archived and only current year is kept on the main hard disk. So I have to use the digiKam capacity to see if the collection is always online or on removable media.
I have a 4Tb usb disk connected, but I try to keep it switched off for safety reason when I don't really need it.
DigiKam uses ordinary Kde folders, so any new album created in digiKam is simply a folder and any folder created in Kde is also an album if it is located in the right place.
So after opening digiKam, I remove all the bad shots and add tags. I try to add as many tags as I can afford to make. It's long and tedious, but very useful after then.
DigiKam allows to search tag list by typing letters, keeping already used tags in the album etc. My work is not always consistent (I'm not perfect), so the tags are not always consistent. So I sometimes end up with several tags for the same content, it's not very important.
What is very important is to keep the tags inside the photos. It's always possible with jpg, but not always with raw images, depending of the make. Databases are not reliable when time is a matter of years. including tags in the photo makes it very easy to send the tag in the same time when you give a photo, but do not use offending tags, you never know if you will think of it (to remove it) in occasion. For example do not write in it your street address, but only a way to contact you by mail or web site.
I never found an usable way to sync the tags between my digiKam collection and my online (Piwigo) one, so when I inadvertently forget a tag, I have to add it both on the local collection and online by hand.
Tagging mean noting what and who is on the image: name of the people, make of the car... But it's also useful to sort out photos by subject.
For example during a travel you go to openSUSE conference in Dan Haag. The same day you see nice 3D printer and make photos of the guys in the meeting room. Chance is you dont want to make public the guy faces (too much drinks :-), but you want to share the printer images, so in fine the latter is better stored in an other folder. You can create in digiKam a folder 20150502-osc-drink (will stay private) and 20150502-osc-printers (will be public).
Editing
Edit one image with The GIMP or Photoshop may take as long as a full day, I can't afford to spend a so long time for the 2000+ images I took in USA in Feb 2015...
That said, I know I have the bad habit to make images slightly rotated left by one degree, so I very often have to edit the images. DigiKam include such an editor, also available outside of digiKam, "showPhoto".
So I work like this: in digiKam I select the images that have the same subject (10 images of Los Angeles from Griffith, for example), then press F4. This opens all the images in showPhoto. I then can edit all these images without quitting showPhoto.
I first try automatic color correction. it's often a bit better than the camera one, if so I keep it. Then I use "free rotation" to fix the rotation problem, and use the mouse to make a better cropping. I do not care or the 3:2 or 4:3 or 1:1 format, I keep what I think gives the better look on screen.
All this can be done in a matter of minutes (for the 10 images). Very handy.
Spreading the web
Main idea
Photos are not usually for your own use only, you certainly want to show then to the world, or at least to part of it. You of course can use a public (proprietary) system like Flicker or Instagram, but better use your own.
I use most of the time two of them: Owncloud and Piwigo. The first one is pretty simple, as it allows automatic upload and download of documents, including images, without any further action. Handy for uploading smartphone photos and to share them with wife or children. The second is a full web gallery, with indexing and tags and allows you to share thousands of images.
People that are listening to me are not birds of the year, they may already know they can have a free web site from they Internet access provider or loan for very cheap an online server if they want to manage it themselves. At least in France. If you get DSL, you can even have a personal server at home, at the expense of very slow connections.
Remember 1Gb data needs around 2 hours to upload from a correct DSL connection with 1Mb upload bandwidth.
Owncloud
I use Owncloud mainly in two cases: uploading the smartphone photos (it's named "instant uploading"), or sharing photos in forums. Uploading a number of images needs to have them all connected to Owncloud and this is not convenient for so many relatively huge files. I don't care to have all my photo collection on my smartphone, my 32Gb SD card wont be enough.
Smartphone is perfect to add a photo to forums, and Owncloud allow me not to have to wire connect this device to the computer, I have all these images at hand at home when writing the page.
When I need to share some photo in forums, it's often images done only in this purpose, I just paste in the made for Owncloud folder. I then have only to clic right on the image line in the web interface an tick "share with link". I can even give a password if necessary.
From digiKam
DigiKam allows you to share your image to most known web services directly from digiKam, including creating your own html web site. I used this feature for a long time and stopped only when I had to manage too many passwords, .htaccess is not very handy, and I couldn't use any tag or search feature. I do not use digiKam with Piwigo, though, because I think there are more convenient ways. Uploading photos can be very long and I don't care keeping large applications like digiKam running when not necessary, and I want to master exactly where my photos are stored in the server, met only to be able to change gallery software at anytime.
Piwigo
A bit like digiKam, Piwigo uses ordinary folders to store photos, but the folders are located on the server, not your home. Like many php scripts, Piwigo have to be installed on the server, but it's mostly a question of untaring an archive to an Apache accessible folder. Piwigo needs a mysql database, but this is available nearly everywhere. Piwigo do not write to the images, so the tags or comments written from the web to Piwigo are not backed up in the image, so why you have absolutely to use digiKam in this purpose.
Once the install done, you can start using Piwigo. You can use Piwigo own way to upload the images, but the simpler way is to make this directly from openSUSE. I just open Dolphin, split the file windows, open fish://root@my.server in the right part, the image folder on he left and copy paste. Once the uploading done, I have to go in Piwigo to the administrator page and hit "sync" to have all the images synced to mysql.
The images are stored in a subfolder of the "gallery" folder of Piwigo. After that and the images synced, Piwigo is in charge of displaying them in the good order, resize them for display, etc.
Initially I wrote a script (see below) to reduce the image size for uploading, but now I have a 2Tb disk online and can't fill it, so I now keep uploading the full image size. It's a bit long, but manageable. If yous space size is limited, HD size, that is 1920x180 pixels is enough for most uses, including printing up to A4 (this is nearly 2Mpix).
Scripts
Scripts are simply copied to ~/bin to be in the path. Better make them executables, but it's not mandatory (add "sh " in front of the name to run them).
Normalizing the images names
#!/bin/sh # use: start in the given folder #launch "sh normalize-randomize-photo.sh" #you get usable filenames for I in *.jpeg ; do rename -- "jpeg" "jpg" "$I" ; done for I in *.JPG ; do rename -- "JPG" "jpg" "$I" ; done for I in *.jpg ; do ran=-$RANDOM ; rename -- "." "$ran." "$I" done detox -s utf_8 *
You have of course to run the script in the photo folder.
Detox (http://detox.sourceforge.net/ - you install it from the openSUSE Build Service) is a smart utility that removes from the filename most weird characters. The script makes all extensions lowcase then add the random number.
Use quotes around filenames allow blank in them, using detox at the end remove the problem of files having the same name because after the random number none are identicals.
Reducing images, "piwi.sh"
#!/bin/sh # use: go to the image folder # run "sh piwi.sh" # and you get the files ready and have only to upload them for I in *.jpg ; do convert -resize 1920x1920 $I ; done
Can't be simpler :-). Needs Imagemagick, but who care to manage images without it! Beware this script replaces the images by the smaller one, do not execute in the main image folder. See below for a fix...
Notice vertical images are not cut to 1024, too small in my idea. Image are simply cut to have they larger size at 1920.
Preparing folders
Older versions of Piwigo expected images to have special folders and thumbnails, this may be of some use sometimes, so I write this here (now you only have to upload edited images and let the server do the job):
mkdir thumbnail mkdir pwg_high for I in *.jpg ; do convert -resize 1800x1800 $I pwg_high/$I ; convert -resize 800x800 pwg_high/$I $I ; convert -resize 128x128 $I thumbnail/TN-$I ; done
This is a bit more complicated. I first create the necessary folders for thumbnails and full sized images. Then I copy 1800 pix images to high, 128 pix images to thumb and resize standard to 800 - at this moment, HD screens where not frequent :-). You can use similar syntax to copy files to an other folder in place of overwriting the images like the last script.