The Online Photographer

Check out our new site at!

Friday, January 12, 2007

To Delete or Not: Computer Expert

by Dave New

The problem with in-camera deletion comes from the fact that given various combinations of reads/deletions/stores, there is an almost infinite combination of card file system state.

A good software tester can usually group a large number of those possible test cases using techniques like "equivalence classes," but that can be an inexact science (it depends on knowing sometimes too much about the internals of the filesystem code implementation), and it still typically leaves a lot of "unturned stones."

Consider, also, that actually testing a large number of those combinations may be prohibitive, because the card read/write speed becomes a factor. Most regression test suites aren't designed to take more than a day or so at the most to run. Executives that want products to ship don't have the patience for more testing, it seems.

The upshot is that a lot of potential end-cases probably don't get tested very well, except by poor consumers who end up losing images in the field when they hit the untested combination and uncover a bug in the file system software.

The two most buggy pieces of software traditionally are memory allocators and file systems. They are hard to get exactly right, and most companies that have what looks like good code in those areas will practically set up armed guards to keep newbie programmers from attempting to "improve" the code.

Not even Microsoft's file systems have been completely bug-free. Windows' FAT file system had a long-standing bug that surfaced when you filled a disk and still had pending data in the write buffers: it used to silently close the truncated file and discard the remaining data without reporting the data loss error to the application layer.

If you must delete, delete only the last image (or images, deleting the last recorded one first and working backward), rather than creating "holes" in the file system by pulling images out of the middle of a batch. Also, as a rule, I don't completely fill my cards.

Then, after transferring my images to two backups, I format the card in the camera. Doing a "delete all" in camera doesn't do any good—a format is the only reliable way to reset the file system to a given known state.

No sense tempting fate.

Posted by: DAVE NEW

ADDENDUM by Dave New: Most folks don't download through the USB cable, mainly because it is usually a lot slower than mounting the card directly in a high-speed USB or Firewire reader, or using a PC-Card reader, mounted in a PCMCIA slot.

That said, the rule of thumb should be to only let the camera format, write, and/or delete images.

It's particularly dangerous to format a card on the computer, and then write to it with another device. Likewise, it is dangerous to format or write files on a device, and then use the computer to delete those files.

When you format a card in the PC, it doesn't know the purpose that it is to be used for, and among other things, it doesn't set up any of the required "DCIM" folders, etc., that are used by camera devices.

The safest way to treat your cards is to assume they are "read-only" for all computer-based operations, and "write-only" for camera-based operations.

So, here is a summary of "safe" card usage:

1) Format the card in the camera you intend to use it in (not on the PC, nor in any other camera—you can delay doing this until you are about to use the card, so there will be no confusion as to which camera it has been formatted for).

2) Take pictures. Best practice is not to edit (delete) pictures in-camera in the field. There have been several good reasons given for this previously, some having to do with avoiding possible file system bugs, others having to do with accidental deletion (operator error).

3) To be really safe, don't try to squeeze the last shot out of a card. This is just pure paranoia based on past experience with file system bugs, but it certainly doesn't hurt to be conservative in this case.

4) Remove the card from the camera, and mount it in a card reader device. This really speeds up the download process, and conserves the camera battery. If the camera battery runs down during a USB cable transfer, you run the risk of corrupting images. Even if you ignore this advice, and download your images via a direct USB-to-camera connection, don't be tempted to manipulate images on the card from your PC.

5) Do not modify the card file system from the PC, period. Treat the card as read-only while it is mounted on the PC, and only copy your images from the card to the PC, preferably to two different locations for safekeeping.

6) Remove the card from the PC reader (I place mine upside down in my card case, to identify them as ready to reuse, but delay formatting them until I actually place them in the camera for use).


Blogger Matthew Miller said...

There's really no excuse for having dataloss bugs in a FAT filesystem at this point -- there's plenty of perfectly good and well-tested open source-licensed (including BSD license) implementations.

(And FAT is what all cameras use.)

7:02 AM  
Blogger Robert Roaldi said...

In my previous life, I wrote software on mainframes, minis, and micro-embedded systems from 1978 until about 5 years ago. In the old days, bugs were considered defects that had to be fixed before releasing the software to clients.

Now, people shrug them off and accept that they have to reformat disks or reboot their computers or reinstall software now and then. At my later jobs, I was considered a crank by younger colleagues for complaining about this state of affairs.

It reminds me that when salesmen ran companies like GM and Ford, we ended up with Vegas and Pintos. Eventually, others like Toyota come along and showed us how it's supposed to be done. It IS possible to make good products. It's not a mystery.

But as a photographer, I have to agree with the poster. We have to protect ourselves against the second-rate programs we buy, no question, so I reformat my cards in-camera. What's my other choice?

End of rant. :)

10:23 AM  
Blogger Dave New said...

You'd think so, but I came across a buggy FAT implementation in a VxWorks-based device not too long ago.

Everyone seems to want/need to re-invent the wheel, and since Microsoft never publically published a FAT specfication, all of the various implementations have been reverse-engineered.

At least FAT is simple compared to something like NTFS, which also has never been publically documented by Microsoft. There are open-source implementations of NTFS, but a lot of them default to Read-Only, to guard against the possibility of lingering bugs that might destroy data on a user's C drive, if mounted, for instance, under Knoppix (a Linux distribution which can be booted and run from CD/DVD on a PC).

11:08 AM  
Blogger Michael Canyes said...

I am inclined to agree with Matthew Miller; file systems should just work at this point. However, if I am shooting for $$$ I don't delete, if I am on vacation I do get rid of the stuff I know for sure is bad. I have never lost the contents of a memory card.

One other thing worth mentioning: don't run out and use a brand new card for something important. Usually, electronics will fail pretty quickly, or last a long time. Sort of a bell-shaped curve thing. A new card should be filled and formatted a few times to see if it has any lurking issues. Try to do a low-level format if you can, and ALWAYS format in the camera.

12:01 PM  
Blogger RJ said...

IMO the in-camera format is not a format but a quick format which isn't much different from delete all.

12:05 PM  
Blogger Dave New said...

It depends on how the "delete all" is implemented by the camera firmware. Some cameras will just implement it as a short-cut to the "format" command, and others will just delete all the files, like a "delete *.*" command does on a DOS file system.

It's true that "format" does not normally do a low-level format -- in most cases that is not what you want, anyway. But, "format" will almost always zero out the file allocation table (FAT), along with placing an empty, intialized root directory on the media. The data is still on the media (and possibly even intact "orphaned" subdirectories), but there is no longer a chain FAT entries that account for the space usage and location of the data on the media. It is similar to what happens if you do a "quick format" to a floppy, instead of a complete low-level format.

It's usually not necessary to overwrite the media with a given data pattern and check for hard errors, which is what a low-level format usually does.

After that, a low-level and high-level format do the same thing -- initialize the file system structures on the media to a known good (empty) state.

Photo rescue utilities depend on the fact that "format" rarely changes the actual image data on the media -- just erases all the file pointers. Then the rescue utility walks through all the data sectors, looking for likely candidates for groups of sectors that form a valid image (the JPEG/RAW file formats have a recognizeable structure in their data fields), and offer it to you to name and recover.

This works because usually deletion of a file in a DOS filesystem involves just replacing the initial character of the filename entry with a binary zero, and erasing the file's sector usage in the FAT, thus freeing those sectors for re-use. Also, FAT usage on FLASH media tends to walk up through increasing sectors, not re-using previously freed ones until all the others have been used (note this is not the same as wear-leveling, which is performed "under the covers" by the FLASH controller, so there is a transparent logical-to-physical mapping going on that the file system is not concerned with).

This means that even if you have deleted images early in your shoot, as long as you have not filled the FLASH, or deleted/re-used more than the remainder of the FLASH, chances are good that you could recover that early deletion with a rescue utiliy.

Not something I'd depend on, mind you, but it might be good to know, the day you realize that you blew the wrong image away early in the shoot, and subsequently shot several more before realizing your mistake.

1:50 PM  
Blogger Thomas Passin said...

What I've been doing is to transfer the files to the computer via USB, leaving the card in the camera. Once I'm sure they're transferred, I select all the files and delete them from the computer.

This is quick, simple, and natural. So far, with two cameras and three cards and thousands of images, no problems.

7:09 PM  
Blogger scouter573 said...

I look at it from the other point of view - the cost of an extra memory card or two is minimal. For all the reasons above: Fill 'em up in the field, delete on the computer, then reformat in the camera. Simple. Cheap. Safe.

9:57 PM  
Anonymous Anonymous said...

Why format in camera, and which camera if like me you have a few?

There is no logic to this thinking, if you want to format a storage device do it with a platform that has had the greater real world testing, OS X or Windows by your preference. Cameras are recording devices and tested hard as such, the format software code is a necessary design requirement, not directly a part of the products primary function.

Delete in camera, it works but is it worth the risk of losing more than you anticipate? For a professional, the value of the images is greater then the media and the recording device is not a good editing tool; No, I don't delete.

There is an issue with data cards that I have not seen addressed, how much storage is available: 1Gb card = SanDisk 976Mb or Kingston 944Mb, regardless of where the format is done. This would suggest the card is controlling the format and not the platform?

If you do delete there is a difference to the filesystem on the card from the type of files your camera creates. If you shoot raw, the files will (in my experience) all be the the same size, however jpegs will vary in size by as much as 60%. If you delete a raw file it will leave an empty hole in the filesystem the right size for the next raw file, no big deal. With jpeg (or raw and jpeg) however when you delete a photo and then save a new one, which may be much larger so the file system is then placing the photo in multiple storage locations on the media. This could cause performance and integrity issues, your going to be very unlucky if you experience the failure. but you don't want to be a victim of Murphy's Law...

11:50 PM  
Blogger RAGS said...

I also plug the camera in, let the usb open an explorer window the drag and drop the files to the computer. And always as part of that process select them on the computer and make them read only.

If the card is full or close to full, then after making at least one DVD backup, I select all in the explorer window and delete them off the CF card. Haven't had a failure or problem over three years and many thousands of images on two cameras that share the cards.

Well, did have a problem with my daughter's camera, a cheap one, where she had been deleting images in camera. Finally she came to me and we found her card unreadable and wouldn't reformat.

I'm not sure formatting in the camera is a good idea at all, unless that is the only camera you have that uses those cards and the card is no longer readable. If the card is readable through the USB connection, FAT is in place and ok. FAT is relatively simple, and as long as there are no holes, close to fool proof.

I suppose one of these days I'll have a card fail, I just hope I find out when trying to do an image preview in the field and not when I get home and try to download.

12:14 PM  
Blogger Thomas Passin said...

Dave New said -

"When you format a card in the PC, it doesn't know the purpose that it is to be used for, and among other things, it doesn't set up any of the required "DCIM" folders, etc., that are used by camera devices."

All three of my cameras have recreated these folders when I have deleted them. So, at least with my cameras (one Canon Digital Rebel, a Minolta A2, and a Minolta 7D), This doesn't seem to be an issue.

6:25 PM  
Blogger megaperls said...

You may want to add to "eject" the card on the computer to make sure everything is closed cleanly. Not good to rip out a disk still in use.

8:36 PM  
Anonymous Anonymous said...

The update is interesting, however lacks any technical foundation. Using the word "dangerous" is emotive and without justification. The cards are smart devices and operate in the same way as an ATA (IDE) hard disk, this means there are two data structures, the device's and then the format for the operating environment. It is the latter format you are discussing, it provides the structure to save files and then retrieve them, this is a standard and well understood disk format FAT or FAT32.

If the FAT structure was bad, millions of folk would be losing their data, that is fortunately not happening, problems will always surface but after 30 years of operation and development it can be described as safe! The folder structure is created in camera as soon as you insert a new card, this will be part of the power on test the camera completes when you insert a new card.

Your points:

1) Format in the camera it is to be used in - you can delay doing this until you are about to use the card

- Why, it is a standard card in a well tested device.
- Besides my subjects don't wait while I format the card's

2) Don't delete in the camera.

- Quite right, regardless of the reason, don't delete when it matters that the photos are safe.

3) don't fill the card up.

- Try it when you have some time to see what the camera reports, then you'll know what to expect.

4) remove the card from the camera

- Because performance is better, and don't edit on the card because performance is lousy.
- Battery failure on the camera during the copy is only reading the device so it no more going to damage the images than a power failure with a card reader.

5) Don't write on the PC

- covered above, the computer is a safer place to format.

Much of the advice is personal preference, the process you feel works best for you. However it is best that you decide that process on solid information, not from me (over 30 years professional in computers and professional photographer) but from the people who write the spec's links below.

So, what is dangerous to your cards, static - keep them in cases safe from sparks off nylon clothes etc. and water.

happy shooting


1, Compact Flash Org
2, Microsoft - FAT32 spec

3:18 AM  
Blogger Olaf Ulrich said...

Dave New said:
> Do not modify the card file
> system from the PC, period.

Since the original file system has been written to the card in the factory (and not in the camera), I don't see why I shouldn't modify it outside the camera.

Actually, the first thing I do with most new memory cards is modfying the card's file system from a PC. That's because a) the manufacturer often has written the wrong file system onto the card, and b) the camera cannot change it.

With CF cards up to and including 4 GB, I want the card's file system to be FAT16. But usually the manufacturer has formatted it FAT32 which in my camera (Konica-Minolta Dynax 7D) is slower than FAT16. So I have no choice but re-formatting it from the PC.

Of course, the first thing to do after a low-level format is re-formatting the card inside the camera. The format performed by the camera always is just a quick-format ... but it shouldn't be omitted.

I've never run into problems re-formatting my cards this way. Of course, if you format your card in order just to delete the contents then you'd better do it per quick-format from the camera. Low-level formatting from the PC makes sense (and is required) only when you want to change the card's format from FAT32 to FAT16 or vice versa.

-- Olaf

9:21 AM  
Blogger Dave New said...

OK, we can debate this to death, but I'll attempt to summarize a few things:

1) thomas passin pointed out that most cameras know how to supply missing files/folders from a PC-based format. That's true in most cases, but I was trying to state a rule that would always work. In particular, in spite of the idea that there is supposedly a standard for digital imaging device use of FLASH media, there is a certain amount of wiggle room within the specification. Thus, a card that is formatted in one imaging device might not play correctly in another, particularly if images have already been recorded, because of assumptions made by the device manufacturer about file/folder naming, etc. So, cross-formatting may not be an issue in your case, but that doesn't mean it will hold true for everyone.

2) megaperls correctly pointed out that any removeable media on a PC must be 'ejected' or otherwise made safe, before removing. I assumed (probably incorrectly) that everyone knew what I meant when I said to remove the media.

3) victor called me out correctly on the use of the word "dangerous". My intention was avoiding getting further involved in the minutiae of file systems, etc., and to lay down some simple guidelines that should help everyone stay out of trouble.

victor spake "If the FAT structure was bad, millions of folk would be losing their data, that is fortunately not happening, problems will always surface but after 30 years of operation and development it can be described as safe!"

The FAT structure may have been around for 30 years, but various implementations of it have not. FAT may even be well understood, but that does not prevent one from writing bad code to support it. I can provide recent bad examples of FAT implementations, if you desire. Ask anyone that has suffered through WindRiver's VxWorks' implementation, for instance.

If you are pressed for time, by all means, don't wait until you need the card to format in-camera. That was a guideline for those with multiple cameras that may wish to freely mix their cards between cameras, to help prevent compatibility problems.

I warned folks about possible data loss during a USB cable transfer if the camera battery runs down, mainly because I've seen this in more than one camera manual.

Like victor, I too have over 30 years experience in professional software development, essentially all of it in embedded systems. I'm not certain what it proves, though. Maybe that we are both too old to know any better? 8-)

4) olaf makes the point that sometimes you 'gotta just format on the PC', and I agree. These kinds of things have to be handled on a case-by-case basis, but after the card is put into regular service, you should not have to re-format the card on the PC. And olaf's advice to re-format the card on your camera before using it, is spot on.

4:11 PM  
Blogger Prophoto said...

Hi everyone, its not often I have time to speak my mind... but here it is a new perspective on this thread...

The advice given about formatting in the camera all the time, is good advice and should be good for most situations. however I have a workflow used on occasion, which contradicts the advice. its ok to delete all images with downloader pro, I have done it 1,000's of times in several years time, and never had an issue with many cameras and 30 different cf cards. I currently use downloader pro to upload images to the hard drive and then only when they are successfully transferred on the hard drive I use the software to delete all the images on the card. this is useful where we reuse the same cards or when you use a photo assistant to do the downloading. you can easily run out of cards, even if you thought you had enough, and get stuck at a shoot very quickly. but most importantly, if someone tells you they downloaded the images and you believe them and then format the card in the camera, if that assistant was wrong or gave you the wrong card, which can easily happen if someone distracts him while he is working, you just erased all your images for good, because you are reshooting on the card. This can easily happen if you are never backing up at all and keep shooting with many cards in your camera bag, all it takes is one mistake, even if you are working by yourself, all you have to do is have some distract you while you are putting a just filled up card in your exposed images pouch, a sexy model could ask you a question, or request to see images, and you could put it back in the camera, and then forget you did that, and format it, or you could put it in the unexposed section by mistake… or it could just take one quick second… one mistake and grab a card you thought was already backed up, but you never backed it up.... when you format in the camera you have no way of knowing if the data was ever backed up, and you have so much going on, and several people rushing you to take photos asap...

My point is downloader pro offers a great service by only deleting the images that are successfully backed up, (yes it checks they are on the hard drive before deleting the files on the cf card) this way when I receive an empty card from my photo assistant I am 100% sure the images are safely backed up, especially because they can now go to two hard drives at once! it is my clue that its all backed up and safe to shoot on, and 100% more assuring than some surfer dude giving me the thumbs up!

In fact if I get a card where the photo assistant forgot to delete the files I make it policy to go to the computer and try to re-download the photos, just to be sure they were saved. I caught a photo assistant once where he would have had me erase photos as he handed me a card that he never backed up... on other occasions it wouldn’t have been a big deal he just forgot to delete the files after the download to the hard drive.

With this workflow I could easily reformat in the camera just to be safe once I see the card is empty from the photo assistant deleting the images with one simple safe click in downloader pro. That simple step should make most everyone ok with my workflow.

here is my big beef.... I know many of you will now agree with me that perhaps you can see that this is a viable workflow especially if the card is formatted in the camera before shooting. its just nice to have a way to know you are not erasing the images permanently by mistake when you use the cameras format feature. so now that you all agree that this could be a work flow that may be acceptable in some cases ( when you want to use faster or more reliable cards more often, or if your in a jam and never backed up many of your other cards yet and need an emergency workflow to keep the client happy that just asked you to take more photos when you normally would not be ready... or if you just want to review images as you shoot, or if you are saving money on cards) as you can see there’s lots of excuses for this workflow, we can plan ahead to use your workflow, but we can't always have enough cards unless we went completely wild buying cards like crazy, and then you end up using untested cards that may prove to be unreliable, definitely not a good workflow!

well here is the problem, the maker of this very nice program image ingester, believes your workflow is gospel for everyone. it would be nice if he could offer at least an extra feature where you have to dig deep into the options area and key in a password to make a safe delete feature where it checks the images are backed up first. It would be a feature that only experienced users can enable if they prefer, or if they pay for the feature. that’s all... a simple request that only makes the software better for some people that choose to use this buried feature, the rest will continue to use the software the same way, never finding out how to unlock this feature.... enough said, now please can some people that agree with me tell Marc from image ingester... he is a nice guy, but he seems set in his ways that no one needs this feature due to this authors workflow, he completely believes that no one needs any other workflow. it may be good for many people but as you can see I prefer my work flow for some assignments where a computer is handy and I am counting on a photo assistant... other times especially on assignment in the field your workflow works great.

Evan Lauber

Fashion Photographer

PS: it is also possible for images that the camera saves on your memory card to be corrupt immediately during a shoot. This happened to me the other day. the camera wouldn't show the last 50 images taken. I only luckily noticed this because I hit the preview button in the camera. The missing images were most likely due to a new defective cf cards, which worked fine the first three times it was used. the images had to be recovered with image recovery software. The authors of image ingester and downloader pro have both agreed to look into adding a feature that will look for breaks in file number sequences, alerting the user that there may be missing files not downloaded, a situation which although rare did happen to me and can re-occur and cause photos that could have been restored to be permanently erased. The user will be alerted at the end of the download, so the user knows to save the card avoiding permanent loss of the images if it was formatted in the camera and reused. The missing images can be recovered at a later time with image recovery software. This would be a great new feature which will save many missing photos from never being deleted. (currently if there are missing files which could be retrieved many users never know about this situation, and the images get formatted in the camera.)

2:56 PM  
Blogger prentz said...

As a lab owner that sees tons of memory cards- and their problems- I suggest exactly what Dave has said here. Almost every problem we find with memory cards relates to NOT formatting- when new or before reuse. And yes, we've seen the destruction of files from camera batteries dying mid-transfer (or someone knocking the camera loose) to the point that even my recovery software couldn't bring them back! Go ahead and argue, but it works! By the way, any good suggestions on recovery software?

1:48 AM  

Post a Comment

<< Home