Keep the RAW if you edit in DNG?

For awhile now I’ve been shooting RAW images on my camera, and converting to Adobe’s DNG files.  DNG has offered the ability to edit my photos better, a well understood reason, but also to keep metadata, like tags, and gps information, stored in the file.  Features that are not normally associated with vendor RAW files.  This allows me to keep everything in one place for a picture, rather than having multiple files (raw + xmp being the adobe convention).

As I create the DNG files, I also archived the original RAW photos in case something is wrong with the DNG, or I need the original file.

In the years that I’ve been editing the DNG format, I have not once needed to go back to the RAW file for anything.  With that knowledge in hand, I’ve declared that 2015 is the year I abandon keeping the original RAW files around after I create DNG versions.

This idea came up based on my output from a recent photo workshop.  When Neeley and I shot pictures during our London/Paris workshop, we ended up with 120gigs of raw image files (+120 gigs of DNG files), from 3 weeks of shooting. Giving us a total usage of 250gigs of data a month if we were to travel and shoot continuously. Or about a TB of data every 3 months.  I also started shooting video recently, and that eats up even more storage.

Chopping out the original raw files in this equation results in a storage, and organization savings, and a time savings from any backups we decide to do.  The time it takes to back up 120gigs of photos “to the cloud” while travelling is an entertain exercise left to the reader.

So, 2015, the year I abandoned keeping the RAW, and just use DNG files.  This will also be the year I determine if that was a smart decisions.

What do you do?

-Matt

What’s In My Camera Bag?

Neeley and I recently got back from teaching a photography workshop in Paris and London.  And I remembered that I had yet to followup with my previous post about what to shoot with, graphically.

I carry an olive Domke F2 bag.  I’m on my second F2,  though more due to color choices, then condition.  My previous black F2 is approaching 10 years old, and probably has another 5 in it.

The Domke F2 is a medium-sized bag that hold quite a bit of stuff, with minimal padding, and protection.  Though it’s more then enough protection to prevent damage from bumps/bangs of tossing things in the overhead bin, or putting it down by your feet.  They are shooters bags, designed to be carried and used, not for shipping gear.

The newer F2 bags come with an adjustable 4 slot padded insert for lenses, while the older F2 bags had a non-adjustable 4-slot insert.  Both are movable in the bag, and removable if you want.   After spending a couple of weeks fiddling with the insert, I’ve come to the layout that I use now.  A Canon 5D Mark 2 on it’s side, back facing the back of the bag, right hand of the camera facing up (with a hand strap), lens forward.  This allows me to have any normal or wide lens mounted on the camera and easy access in and out of the bag with it (though I have to flip the lens hood around).  I currently have a Canon 24-105 F4L mounted on the camera.

I place a Canon 17-40 F4L lens in one of the padded slots near my body to keep the weight closer to me.  I can replace this wide angle lens with a couple primes, or a 70-200 depending on what I’m planning on taking pictures of.

The additional padded slots carry my Gopro 3+, along with waterproof and skeleton back, and the wifi remote.  I replace the Xshot stick with a Manfroto Pixi tripod depending on what I’m looking to do with the gopro.  I have a medium-sized USB battery pack for my phone or GoPro if I run out of power, with cables.  I also found myself just carrying a spare phone, gopro, and gps battery in a small ziplock.  I have a small cotton rag I use to wipe things down with, dry things off, etc.  This doubles as added padding if needed.

On the front outside pockets, I have a Moleskin notebook and Space Pen for old fashion note taking if needed.  I also stuff in a LensPen, ThinkTank PocketRocket CF/SD card holder with spare cards, and some lip balm.  I stuff a package of business cards and a couple bogo-ties into one of the pockets.  Lastly I turn on my GPS logger, and leave it in a pocket all day.  See my older geotagging post for some more information.

This leaves both end pockets on the Domke empty, allowing me to put water bottles, sunglasses, flash-units, or other temporary items in (subway cards, small gifts, etc).

Weight can be heavy to light depending on the lens loadout that I use for the day, if you want a three lens zoom setup (wide, normal, telephoto) things can get heavy for all day carry on the shoulder.  It’s important that you take the camera out and carry that independently if the bag is heavy, or learn to shoot with less stuff.

This is how I’ve been shooting for about ten years now, slightly different gear in the bag, but this bag, and this layout.  During our Paris and London workshop I also shot with a much lighter Olympus OMD EM1 micro 4/3 camera.  I’m still debating using this bag as my primary if I were to switch to something like that for most of my camera work.

How do you carry your gear?

Technical Tips – Photo Preprocessing Workflow with Geotagging

Workflow workflow workflow.

A photo processing workflow is a process that you follow after you have taken the pictures. It should be straight forward and allow you spend time working on the photos, not the organization of the photos. My main goal was to create a workflow that was easy to reproduce, something I could do while traveling, and would leave me with files that I could identify and use years later with the minimal of work. All of this pre-processing is done before I actually do any editing or reviewing of pictures I’ve taken. That stage comes after doing this prep work.

Included below is the workflow I use to prep my photos for actual editing. I do a small number of things across both linux and windows boxes to prep my photos after taking them. I’ve attempted to convert all the tools used below for use in a single windows environment. Most of these tools are PERL programs, so installing a perl interpreter will allow you to run the original source. Links to all the tools are included at the bottom of the article.

A straight forward file system format was the first stage. So I keep all my images stored in individual directories, by date. A directory for every year, and a sub directory for every “shoot”. Where a shoot can be as short as a day, or as long as about a week. Within this primary shoot directory are additional subdirectories used to store different data. I create a “RAW” directory to store straight from the camera files, a “DNG” directory where I store Digital Negatives (DNG) versions ,a “JPG” directory for JPG images that come straight from the camera , a “GPS” directory to store gps tracking info, in GPX format for right now, and a “WEB” directory containing post processed ready for posting images for websites. In addition to these directories I will create a “notes.txt” file and include information about the shoot, timezones, mistakes, and general notes about the images. The end results is something that looks like this:

2009/
2009/2009-06-02 Half Moon Bay/
2009/2009-06-02 Half Moon Bay/DNG/
2009/2009-06-02 Half Moon Bay/DNG/20090602-1405-32162.dng
2009/2009-06-02 Half Moon Bay/DNG/20090602-1412-32163.dng
2009/2009-06-02 Half Moon Bay/GPS/
2009/2009-06-02 Half Moon Bay/GPS/20090602.gpx
2009/2009-06-02 Half Moon Bay/JPG/
2009/2009-06-02 Half Moon Bay/RAW/
2009/2009-06-02 Half Moon Bay/RAW/_MG_2162.cr2
2009/2009-06-02 Half Moon Bay/RAW/_MG_2163.cr2
2009/2009-06-02 Half Moon Bay/WEB/
2009/2009-06-02 Half Moon Bay/WEB/20090602-1405-32162.jpg
2009/2009-06-02 Half Moon Bay/WEB/20090602-1412-32163.jpg
2009/2009-06-02 Half Moon Bay/WEB/index.html

I’ve chosen to use DNG files over RAW files, an open format for my negatives, that also supports metadata. I keep the originals in a separate directory, as opposed to embedding them, so that I have completely unaltered copies. Keeping both a DNG and a RAW doubles my storage requirements for pictures. With the current cost of storage being so low, I keep both. Keeping all your files organized is important in this situation, as having too many files in random locations is as good as not having them at all.

I also chose to use DNG so that all metadata gets embedded into the DNG file. As I add GPS info, keywords, or IPTC data, it all gets written back out into the file. This keeps all the custom data in the actual file, and not located in a XMP sidecar, or similar. I do this so that if in the future I need to switch software tools, the new tool should be able to read all the metadata out of the files directly.

So how do I do all this?

  • Turn on the GPS logger, and then go out and shoot pictures.
  • At the end of the day I create the primary directory I’m going to work in, with the format “YYYY/YYYY-MM-DD – Subject Info”, and create subdirectories in there for “RAW”, “DNG”, and “GPS” data.
  • Download all the raw pictures from my camera, and copy them in the RAW directory. I do this with a simple USB card reader, and copy by hand. Sooner or later I’ll find something I like to automate this step.
  • Convert all the RAW files to Adobe DNG format, using the GUI Abobe DNG converter, putting the resulting files into the DNG directory.
  • Optionally download my GPS data, in GPX format, placing it in the GPS directory. I use a Qstarz BT1000P gps logger that tracks every 10 seconds. I leave my Camera set to GMT time and adjust everything in post processing. This helps solve the problem of remembering to update the camera clock when traveling.
  • Use the gpsPhoto tool to insert basic location tags and gps coords. The command to do this:
    gpsPhoto --dir . --gpsdir ../gps/ --timeoffset 0 --geoinfo geonames --overwrite-geotagged --city auto --sublocation auto --state auto --country autoThis will automatically take the GPX file from the GPS directory, write the latitude/longitude coordinates into the DNG, and then make best guesses at location, and write that into the IPTC section of the DNG.
  • Correct the time information in the photo to match the local time. I use the ExifTool program. In this example, I adjust for local California time from GMT in the camera with the following command:
    exiftool -AllDates-=8:00 -overwrite_original *dng
  • Exiftool again to rename the file to something useful. I use a format that has the date and time in it:
    exiftool "-filename<${CreateDate}-$filenumber.%e" -d "%Y%m%d-%H%M" *dngOn my Canon 20D, this produces files that look like this – “20090705-1202-327-2721.dng”. I run a small custom program which merges the last numbers into a single 5 digit number, and the file becomes – “20090705-1202-32721.dng”. This results in the Date-Time-Img# filename that is unique and “relatively” compact in length. Source/binary for this is provided below.
  • Append boilerplate metadata. Using Adobe Bridge, I apply a metadata template that has my contact and copyright data to all the DNG files. Menu item “Tools”, then “Replace Metadata”. I prebuilt a template to include my name, email address, and basic copyright.

After all this is done, we now have a directory filled with DNG files ready for processing. It takes about 10-20 minutes to do the above steps and I try to do it daily when traveling to make sure I have gps tags and locations on everything.

Later, anywhere from later that day, to weeks later, I will go through and in multiple passes cull down the pile of shots to 1/10 or similar number of shots that I want to do more changes to. This is done in simple two stage system. Initially scanning through all the pictures and marking interesting ones with one star in Bridge if they look good, or I want to edit them. This usually cuts out 50-75%, and I try to spend less than 10 seconds judging an image. The idea is to cut out all the obvious ones you are not going to spend any time on. After the first quick pass, I make a second pass just on the ones that are marked with one star. This pass is longer, and I’ll spend time playing around with an image to determine which shot I like. Photos that I decide to keep, I’ll mark with a second star. I may have done some minor work with camera raw at this point (cropping, basic levels).

Following this two pass method, I then go through and crop/level all the two star photos in camera raw or lightroom. Most of the time this is enough editing and I post the resulting images online. In my next blog entry I’ll update how I use “TravelGen” to create static travel log pages with my images.

Software:
ExifTool is the swiss army knife of metadata modifications. Download the zip file, unzip, and copy the executable into “c:\windows”.

gpsPhoto. This is optional and allows you to add GPS coordinates to your pictures. It needs a one line change to recognize DNG files. This change is included the binary version I compile. Win32 Executable that you can use if you are running on a windows environment.

ImageMagick. Download the the Q16-windows-dll version. Run the installer, and make sure to install the activestate libraries (this is unchecked by default, so will require you to select it). My experience is you need to reboot or the libraries don’t get recognized by other programs properly.

TravelGen. This is a tool I created that generates static web pages of thumbnails and text. It was originally developed so that I could build web pages with image thumbnails and have a travel log mixed in, and make it fast to load (completely static). It provides a number of simple customizations, and I still use it today. I’ll document this in my next Technical Tip page.