Archiving Digital: What the Hell am I Going to Do?

IMG_2509

I need your help. I have no idea what to do.

Sitting at home with my wife. We are both in the office. I look over and see my Firmtek SATA drive system I use to backup my wealth of digital files. Staring at this machine I realize I’ve had it in play for at least five years, probably closer to seven or eight. A two-bay system partnered with matching sets of 2 terabyte drives and coupled to a Mac Pro tower of the same age. At one point a state-of-the-art system. But now?

“Man, I need to transfer all that data,” I said to my wife. “How are you going to do that?” she asked. I sat wondering, thinking for a brief second that this little issue was no big deal. I figured something would pop into my mind. A mega-drive? A cloud solution? Something. Anything. “I have no idea,” I answered. “I actually have no f%$#$%$ idea.”

Days go by. A friend faced with a similar situation plugs in a drive from 2005. Dead. Tries another from 2007. Dead. Pays for salvation, but is only partially saved.

An event in LA. Filled with high-end still shooters and filmmakers. I begin my informal survey. “No idea,” is the answer I get. “Well, how are you doing it?” Each and every person has a different solution, each as flawed as the next. There are admissions of guilt. Lost images. Lack of concern with anything really lasting over time. The changing face of archiving and even how it’s viewed by the generations. What there isn’t is an answer. Not blaming these folks. We are all in the same sinking ship, but that does not change the reality that I need a solution. A few feel offended by the question, quickly offering the standard “Just use the cloud,” answer, but when I explain the AMOUNT of data I have they look confused, guilty even and walk off. (This has been happening since about 2002.)

Yes, the amount of data. You see I’ve been doing this for quite some time. I’ve been shooting jobs for clients, since the late 1990’s, and having to reflect on this gave me even greater fear. Yes, fear. You see the SATA solution is the least of my issues. There was digital life before the Mac Pro and SATA were a reality. Long before. I have boxes, crates, stacks of 4x DVD’s burned in triplicate from hundreds of prior jobs. Even years ago, a short time after these jobs were completed and burned, which is what EVERYONE did at the time, these discs were already beginning to fail. Yes, it’s true. Regardless of what the hype machine says about portable media. The idea of this being a long-term solution is a myth because the odds of finding a disc made of the right material, A PLAYER THAT IS MANUFACTURED AND SUPPORTED FOR MORE THAN A TEN YEAR PERIOD is also a myth. Just look at history. Kodak Photo CD anyone? Laser CD anyone? 4x DVD anyone? All of these portable media platforms were undone by the parts surrounding them. Great, you have a disc made of gold, awesome, but what about the player? Who makes it? How are they doing? Where will THEY be in five years? Ten?

It gets worse. Lets’ say I have 50 terabytes of data, which is not a stretch. Remember, I’m not a “Just save the JPG” guy. No, I need/want those RAW files. My cabinet is FULL of drives, and then I have all those discs, at least the ones that still work. How would I even begin to get that on a cloud and how much would that cost? How many years would it take to upload to the cloud? And is the cloud really responsible for my work? Ever read the service agreement from online services? Typically within the first page you are signing something agreeing to the fact they are not responsible for your data. Okay……….

So what if a mega drive exists? How great would it be if I could go buy a 50-terabyte drive, times three, and just slowly migrate all my data to those and let them sit? What do I do in five years? Ten? I do it all over again. All of you should know, whatever you create with digital today you will be handling and migrating for the rest of your life, on about this same schedule I’m guessing.

This is NOT a popular topic. In fact through the years when I’ve brought up these questions at industry events I’m met with anger, denial and defensiveness. “What are you going to do?” people ask. “Just keep buying drives.” “Hard drives are cheap now.” Jesus, is THIS really the solution we need? Why does our industry spend the bulk of their time and revenue on promoting the FRONT end of photography while ignoring the back end?

And here is the final reality. I’m getting off easy. I’ve been shooting film, at least in part, all these years. How about you guys and gals shooting all digital and compiling even more data than me? I’ve been to some of your studios. I’ve seen the daisy chain of drives. I’ve seen the fans blowing to keep things cool. I’ve heard the anger from lost drives, images and any hope of salvation. I’ve even seen this digital frailty change your mentality. I’ve seen the client contracts about NOT being responsible for their work, for their archive, for their history. Sound familiar?

So I sit here today no further along than I was a week ago. What to do? My 50 terabytes. Heck, even if I had 20 terabytes, what difference does it really make. This problem is snowballing every single moment of every single day. My industry says “Someone will figure something out,” while they promote the seventeen new cameras….from this week.

We live in a time when I’ve been accused of being egotistical because I’m even THINKING of making an archive. Yes, true story. Well people I could give a s%$# about the now. I don’t care about the now because so few people are actually paying attention. I AM interested in the future, the distant future, but I’m realizing the reality may be that NOTHING I create digitally will be around.

Just so you know, this is NOT the first time I’ve made this inquiry. Back in the early 2000’s I knew I had the same problem and I spent one year researching, asking “experts” at archiving houses what they were doing and I received the EXACT same answers I’m hearing today. “We don’t know.” “We don’t throw away ANY equipment and we can’t fire anyone who knows how these old machines work.” “Don’t worry about it, someone will figure something out.” I wish this was a sad dream, but it’s not. I know for a fact a lot of those around me have already come to terms with NOT having anything more than a few images left over. I think we are all so overloaded by imagery and the disease of social media that it has someone allowed people to think, “Well, I can see these things in social so even if I don’t have these images any longer, there is NO WAY I can remove it from social,” and because of this digital Coolaid their fate is sealed.

This is a REST OF MY LIFE situation people. And yes, I know you can say “Well, things could go wrong with film.” True. Fire, flood, etc. Exactly. But Jesus, I’ll take my chances with that over realizing every five years for the rest of my life I’m going to have to figure out a way to migrate my entire digital archive. That is just plain depressing.

If ANYONE has a solution I would love to hear it. But if you are going to say “use the cloud,” or “just keep buying drives,” then save your energy. I’ve heard it all before.

34 Responses to “Archiving Digital: What the Hell am I Going to Do?”

  1. Andreas says:

    Wow this just stressed me right out. eBay – Leica -film…searching now.

  2. Robert Catto says:

    My solution, for what it’s worth, is to keep everything live & accessible on the largest affordable drives available at any given moment, and replace them every 2-3 years – by which point I find there is now a single drive large enough to hold many of the previous generation. My current 6TB Thunderbolt drives replaced 2TB ones, and so on.

    Everything is daisychained & active, so a failure can be detected instantly and recovered from the off-site duplicate (updated weekly). No offline ‘archive’ drives to fail when you’re not looking, only to be discovered years too late when the duplicates have also failed.

    And, I use the best consumer / prosumer drives readily available at the time; no complex RAID 5 systems that can’t be replaced same-day in the event of a failure, as happened a few years ago – the motherboard couldn’t be replaced 3 years from construction, so the RAID couldn’t be rebuilt – but I had just that week transferred all the data to a new single drive the size of a three-year-old RAID array. Such is progress.

    And when it comes time to migrate to new new platform / drives, it’s not a matter of finding 50,000 DVDs, as so many people backed up to over the past decade; half of those have silently failed already, but even if they hadn’t, who has time to import them all & re-burn them to some new format?

    It’s less than ideal, but it’s the best option I can see at the present moment…

    • Smogranch says:

      Robert
      Thank you. What you wrote is reason enough for me to be concerned. I mean that in a good way. What you do won’t work for me, just too much data. A 6tb drive does nothing for me. I’d need to buy 24 of them to even migrate what I have now. And your RAID description is right on the money as well. There is no solution.

  3. Robert Catto says:

    Sorry, didn’t mean to freak you out further! Perhaps this is a business opportunity, then? Create the solution for everyone! Digital Smogranch – massive storage solutions…!

  4. Mike says:

    ‘They’ are so going to hate you, but not me: I’ve been thinking along the same lines, digital capture makes me nervous. But let’s remember that we all share digital photographs across digital media. Without it we would be back to the photographic stone age and I’d be alone in my non-photographic world.

    The first thing that would help is if the industry decided on one universal file format. DNG would be fine by me but ANY format would be better than continually adding to the list with every new camera model. That way we would at least have a chance at finding a machine to look at our ten-year-old photographs.

    But let’s face it, digital files are not photographs. Photograph that rely on a plug in order to be seen are not photographs; they are ephemera. Here, gone. If you are shooting digital – and this goes for film too in many ways – the best way to guarantee that your work stands even a chance of being accessible in the decades to come is to make prints or books. The irony is that for man,y the prints or books will be made with digital machines. Another thought that comes to mind is to have digital negatives made of your very best work. That way you can make traditional silver halide prints or copy them to the latest digital devices that are sure to come along. Analogue storage of digital media, another irony. Attempting to do this for 50 terabytes of digital capture? Not going to happen, Daniel, sorry.

    Mike.

  5. Short answer: I have no f?!$&&!! Clue.

    I’m going to remind everyone who’s reading this. Film has been around for over 100 years. The negatives, are still in your hand, because a chemical process burned content on a material that was always in the real world.
    This is an off topic, but I think about this from time to time. I shudder to think that Chris Hondros died for .jpegs. I know his images are in the machine, but they are ALL digital
    I will stay, for as long as I can with the time tested (just fix properly) silver halide method. The digital archive issue is a classic example of putting the cart before the horse I’ve ever seen.

    Just think. And don’t forget to breathe.

  6. Brendan says:

    When you hear of something that works let us know.

    I’ve tried to step off the bigger, faster, sooner, brighter train but keep getting unstuck. I look at new cameras and the files are bigger, requiring faster processors and bigger storage. I hear people doing multiple layers on 50- 60Mb Tiffs and then complaining about the resulting file size and slow pace of their PS tools.

    I shoot film 99% of the time, scan it and end up with a 40Mb Tiff, a 1Mb Tiff and two flavours of JPEG (so I can post to different sites) and that’s more data than I can handle or remember.

    My only workaround to the data problem so far is to just edit much, much harder.

    And then print (multiple copies)!

  7. Harold says:

    I composed a lengthy answer but then deleted it. So.
    I’m going with this.

    I went to my first computer conference in 1977, I am still surprised any of this stuff works.

  8. Andrew says:

    Just use the cloud! No wait…hear me out. Obviously it depends on your budget but the cloud storage game has changed in the past year. There’s competition now and prices have dropped big time (e.g. Amazon Glacier at $10/TB/month). Most providers have an import/export service so you can send devices/disks for them to upload, mitigating the ‘eternity to upload’ problem.

    Sure, relying on a company to look after your archive is a risk, but realistically they have a multi-$billion incentive to provide reliable service. And even then, the big players (Microsoft, Amazon, Google) are big enough that there will be PLENTY of warning if it looks like they’re in trouble. And if you’re still paranoid after all that, you can use two at once.

    The best part is that once you’re set up it’s only going to get better, faster and less expensive! Still not convinced? There’s hope….IBM & Fujifilm recently announced they’re able to get 150TB on a single tape cartridge. But that’s a few years away.

    Good luck with it all!

    • Smogranch says:

      Andrew,
      So I’m spending $400 per month if I don’t shoot anymore images. Can’t do it. I’m glad cloud is getting better. We need it and it should be the right solution. I think.

    • Andrew says:

      You want a robust archive solution for FIFTY+ TB of data….no matter what you eventually do there will be a significant cost.

      I don’t know, of course, but I would guess that the rate at which cloud storage costs decline will outpace the rate at which you generate data (unless you’re shooting lots of video?).

      You could roll-your-own using a mid-range NAS system from someone like Qnap or Synology – they have systems with redundant storage that looks just like one giant disk. And you can add new or replace aging disks as needed relatively transparently without changing the configuration. This is what I do at home but I’m a hobbyist with <2Tb. But for 50Tb the capital outlay will still be several thousand anyway… And you're stuck with one-site storage subject to natural disasters and obsolescence.

      A few years from now we may well have cheap 100Tb disks. There is a solution right now, but it's all about risk tolerance and budget, not so much technology.

    • Smogranch says:

      I’ll wait….

  9. Tal says:

    thank you for bringing this not-so-popular topic up front. it has to be addressed by all branches of the imaging industry. it’s just too big of a problem.
    I can humbly suggest my perspective. I just let go. as years go by and digital files get stacked, i keep diluting the pile and keep only finished products and only the ones that will help me push my creative cart further down the road. i don’t need them all. what for? even when producing negatives, why bother keeping those you know you’ll never want to look at again? all the outtakes, the blurs, the errors, the lighting tests. why keep them at all?
    i think the problem is too big to be solved by means of hardware. maybe it’s time to just lower our expectations and find a more realistic solution?

    • Smogranch says:

      Tal,
      Man, I’m glad you wrote in, but I’m really on the opposite side here. Why keep all the outtakes, the blurs? CAuse that is where the magic happens. I worked in Sicily for five years on the same project. I left it along for another five years and went back and found SO much I had missed the first, second and third times I did my edit. I found hidden stories, images that worked with other projects of mine, themes I didn’t even see. And then I had an agent look at them, a rep, and she found ANOTHER entirely different edit and association with my other work. Then again with a gallery own in LA who in 2 mins saw something, a theme, in my work I had never seen. I would never throw these away. Digital is the same. Ask Dirk Halstead about throwing away images……
      Again, glad you commented here, not wanting to be a jerk, just saying I can’t toss things out. I’ve done that once in my life, with negs and have regretted it ever since.

  10. John (aka Wish I Were Riding) says:

    Ugh. (I’m mainly just subscribing to the comments)

    The company I work for has an all flash product that holds 40TB, though too expensive for individual photogs, and still does not solve the problem of obsolete over time.

  11. Reiner says:

    Hi Dan,
    From a pure hobbyistic perspective: I shoot film for the keepers, I shoot some digital phone pics for fun and diary loggings allong the analog flow, logging them in iPhoto+1 copy. Once a year a print purge on fuji chrystal photopaper for hardcopying the keepers, putting them after that in books and I’m done. Also still shooting e6 trannies for photographing the travelings which brings me the old style way to the fact I always will be able to look at them just by using a bulb(!) or sunlight and some magnifier glass. I don’t need more. I don’t really care after that. So I count on luck to have them around when I will go to dust and bones. Pics on sensors and lousy big drives now. Big spacy cupboards filled with celluloid then. Nothing new. Once combined, camera and pc made a golden couple,… for pc makers I mean, not for us, shooters. It’s the money they’re after, not your beautiful pictures. Not my beautiful memories. Last month another machine was added: the instax print machine combined with a lens called instax 90. The little marvelous instant prints are given away to family and friends. They are my cloud. I hope they will keep them near to their heart… Keep strong.

  12. Mike says:

    “Ask Dirk Halstead about throwing away images……”. Ah yes, Monica.

  13. What a mess, huh? At least the files from digital cameras keep getting larger and larger with every new release! “Disk space is cheap.” I want to punch people in the mouth for saying that. Hard drives are arguably inexpensive relatively speaking. That’s only part of the picture. You need a system for organizing, cataloging, retrieving – basically knowing what is where and how to get it when you need it. That takes time and time isn’t cheap. One copy of the data isn’t enough. You need at least two. One copy needs to be far away. Where I live I’ve seen tornadoes and fire practically wipe towns off the map. That backup does you no good if it’s sitting next to your computer. Managing data is expensive and it doesn’t get better or easier with time.

    Sorry, I’m probably no help whatsoever.

    Wait, I will rant about something. Is it really necessary that we always capture huge raw files or have we been duped – maybe duped ourselves. I started thinking about it when I got my first Fuji X camera. The JPGs are darn good and I can do little, if anything, with raw files to make them that much better – to warrant storing 3-4x more data. Careful shooting can produces great files in-camera these days. Even when I do shoot raw, which I do for sports in crappy indoor light with my Canon gear, I rarely capture the full size raw file. I’ll capture mRaw and get a substantially smaller file. I don’t see a difference in prints I do at 13×19″. I have life size banners hanging on the area concourse that were printed from those lesser raw files – cropped from those 10-12mp files even. I’ve learned to capture with the end product in mind. Something else to ponder if you’re shooting a lot of huge raw files.

  14. Patrick says:

    I don’t comment much, but I was thinking about this subject the other day. After considering buying a digital camera, the thought came to mind that I’d have to store all those digital photos. That ended that. Copying files is one of the most depressing and boring activities ever… In my own archives, there has still never been a comprehensive ordering of all files into searchable structure. So many tiffs without thumbnails! In the end, I’m willing to let my digital files die because know the negatives are sitting in an envelope on the shelf.

    Nothing is more soul-sucking than staring at a computer screen all day. Apple would tell you that the ipad is the meaning of life… right. There was a lab that shut down where I live that used to do film negatives of digital photos. Perhaps that’s the solution, although I couldn’t tell you where to get that done cheaply. My dream is to have a giant 35mm positive print reel transparency with all my photos on it. THAT’s a backup!

  15. Alex says:

    Get a large server like these: http://www.broadberry.co.uk/servers/storage/50Tb.html

    Once you have all your data migrated from your obsolete media (assuming it all works?), you won’t have to go through the painful part of the process again. In a few years time you’ll be ready to decommission this storage system and just buy the new one. Transferring the files should be as easy as copying from one server to the next.

    The issues you’ll need to be aware of are obsolescence of the connection type between your computer and the storage system (e.g.: nobody uses E-IDE anymore, FireWire has pretty much gone the way of the dodo, Ethernet is still going strong, and ThunderBolt looks set to hang around for a while, then there are Fiber Channel and other specialist technologies).

    There’s no magic solution: in the old days you never had the option of transferring negatives 100% accurately onto new stock. Your old film just perished. In the new days, you have the option of transferring your images 100% accurately onto new stock.

    The issues in the future will be: “oh, Canon raw format? Really? I don’t know any tools that still work with that.”

  16. rose says:

    during my school years I used most of my time to write poetry and short stories, I lost most of my earliest writings due to harddrive malfunction. so I decided to print everything I had left, hole punch it and put it in a folder. These days its all about having it all in digital format, but after loosing stuff I wrote when I was 10 years old, a time I’ll never get back, I knew digital was not safe. Even now, with the “cloud”, Its still not safe; and certainly not guaranteed to last for 20+ years to come.

    Making a physical copy of my writing eased my mind, I begun to do the same with my photos. I don’t know much about filming, I’ve never thought about how people store their digital files up until I read these posts (I move form hard drive to hard drive once every 5 years). But although I’m ignorant on the subject I believe what I did with my writing holds true with everything else. The safest way to store files is physically. I don’t know how that’s done with film but if ABC can hold onto their oldest of Dr Who episodes, you should be able to keep your films safe too.

    creating physical copies may cost money but it wont be like spending $400/month, it’ll be a one off expense which ‘should’ keep your legacy safe for a lifetime, (try not to burn the house down! yikes!).

    That’s the best advice I can give given my knowledge on the subject. Find out how and where you can make physical copies.

    • Smogranch says:

      Rose,
      Thank you. Agree. I have options, not anything perfect, but such is life. Film is easy, but can also disappear with flood, fire, earthquake, riot, mudslide, or spontaneous human combustion…….digital..not so easy, but I’m SURE things will get better.

  17. Hannah Kozak says:

    I sometimes feel as if storing raw files of each and every one of my digital images is like the hoarder keeping piles of newspapers in the corner of each room. “Maybe someday I will need to look at those old papers.” Having said that, I too am baffled about what to do with all the digital files I have created since I went kicking and screaming into digital in 2004 (and I was late going into the game, I was forced in for work).

    So what I have been doing is editing all my photo essays and printing out a series of each. Then, I have a hard, physical copy of what I truly love. Because after all, a picture isn’t a picture until there is a photograph.

    As for the rest, I am keeping multiple external drives with all my data. Every few years, I switch them out, because eventually they will crash.

    The Cloud is not a place where I would trust my soul’s work. It’s just too easy for companies to fold and your data be lost in the ethers.

    I started to make photos when I was around 10 years old. To this day, I can easily and swiftly find any negative by going into my boxes that are all numbered and dated. It’s an old system, but it works.

    And, I have returned to film again. The entire process slows me down like the old days. And then, there’s the magic of waiting for my film to be returned. I have digital scans made and again, keep the edits of what I truly love. The rest, on externals. And so it shall be.

    I have had multiple hard drives crash and cried in the back room of the Apple store. Now, if a drive crashes, it doesn’t matter, because I have at least 3 sets of my data. I don’t know if that helps you but it’s what I have figured out to do.

    I think basically, once someone goes digital, they just have to accept we are going to have to keep updating our storage until the end of our days.

  18. maarten says:

    Hi Daniel,

    Welcome to the backup hell :(. I’m an software engineer used to high-performance environments with failover provisions and rigorous backup regimes, but at home I have an embarrassing mess of separate hard-drives and usb-drives.
    There are NAS solutions like Synology, Western Digital, QNap, etc. Don’t bother about RAID, you need a lot more drives and when configured the wrong way it can ruin your data. These NAS solutions are not cheap for let’s say 50TB, but you have full control. Cloud solutions might be cheaper, but you’re never sure what’s happening with your files.
    I’d go for a NAS solution, maybe not starting with the full 50TB at once.

  19. John (aka Wish I Were Riding) says:

    This is probably a very naive question… but why do you feel the need to keep backups of your client jobs? Unless they also contain personal work, why aren’t all the project collateral passed to the client for them to worry about (or not)?

    • Smogranch says:

      John,

      I would do whatever possible to never get myself into those shoots. Shooting things you don’t feel the need to keep leads to a miserable career. I’ve done it, everyone else I’ve known has done it’s not really a way to create a lasting career. It has become VERY routine these days with so many people throwing their hat in the photography ring, but I’ve made some important work on assignment and can’t imagine throwing these things away.

  20. Reiner says:

    http://www.fujifilm.com/support/photo_rescue/10.html

    This only speaks in favor of the Print. It can survive a lot, really a lot, as long it only has to take the flood.

Leave a Reply