Using the UDF as a successor of FAT for USB sticks

30 01 | 2013

Using the UDF as a successor of FAT for USB sticks

Written by Tanguy

Classified in : Homepage, Debian, Command line

FAT

USB sticks are traditionally formatted with FAT 32, because this file system is implemented by almost every operating system and device. Unfortunately, it sucks, as it cannot use more than 2 TiB, store files larger than 2 GiB or store symbolic links for instance. In a word, it is an obsolete and deficient file system.

exFAT

Good news: someone addressed that problem. Bad new: that someone is Microsoft. So as you could expect, exFAT, the extended FAT, is a stinking proprietary, secret and patented file system. There are free implementations of that shit, but it is safer to stay away from it.

UDF to the rescue!

Good news: there is one file system that is implemented almost everywhere as well, and which does not suffer from such limitations. UDF, the Universal Disk Format, is an ISO standard originally designed for DVDs, but it is perfectly usable for USB sticks. It also supports POSIX permissions, with one killer feature for removable media: a file can belong to no specific person or group.

So, to use it, assuming you USB stick is /dev/sdc:

$ dd if=/dev/zero of=/dev/sdc bs=1M count=1
$ mkudffs -b 512 --media-type=hd /dev/sdc

The initial dd is to erase the existing partition table or file system information, to prevent you USB stick from being detected as a FAT after it has been formatted with UDF.

The -b 512 is to force a file system block size equal to the USB stick's physical block size, as required by the UDF specification. Adapt it if you have the luck of having a USB stick with an more appropriate block size.

After that, you USB stick will be usable for reading and writing with GNU/Linux and the other free operating systems of course, but also with current versions of Windows (read-only with the outdated version XP) and with MacOS.

Tags : USB, stick, file, system, FAT, exFAT, sucks, UDF, DVD, ISO, POSIX, dd

28 comments

wednesday 30 january 2013 à 14:19 Mikael Eriksson said : #1

Careful with UDF on linux, when the filesystem is around 70-80GB full, you will not be able to write more to it, even if there is lots of free space. It will still work correctly in windows though.

wednesday 30 january 2013 à 16:33 Tanguy said : #2

Note that UDF is usable for a whole medium, or for a partition. You always have to zero the beginning of the existing data, because UDF does not seem to put its identification data at the same location as FAT, so a device previously formatted with FAT, then re-formatted with UDF can still be detected as a FAT.

wednesday 30 january 2013 à 16:35 Tanguy said : #3

@Mikael Eriksson : Well, this is good to know, I never faced such a situation because I never used devices that large with UDF. I should try that.

wednesday 30 january 2013 à 18:28 Camelek said : #4

Tanguy you are genius! :D
I will try that when I only get bigger pendrive :]

wednesday 30 january 2013 à 22:32 Olzvoi said : #5

Is it really necessary to give the option -b 512 to mkudffs? The man page reads "If omitted, mkudffs block-size is 2048". I tested it leaving the option - it seems to be working fine with a SD card.

wednesday 30 january 2013 à 22:40 Tanguy said : #6

@Olzvoi : Yes, it is necessary. You can use a random file system block size and still be able to mount your file system under Linux, but the UDF specification clearly requires it to match the device's physical block size. Doing so would 1. violate the spec. and 2. render your file system unusable with some operating systems that assume you respect it without checking. That includes Windows systems, and may include other systems as well, because I am not even sure there is a block size field at all in the file system header so Linux's support for random block size may just be guesswork.

thursday 31 january 2013 à 06:02 Adam said : #7

I have read that USB flash drives often wear-level sectors used for the FAT differently than the rest of the drive, so using non-FAT filesystems may cause premature failure. Have you considered this?

thursday 31 january 2013 à 08:19 Tanguy said : #8

@Adam : I have not, but I frankly doubt manufacturers would do such an optimization in such low-cost and generic devices. My guess would be that they just have a plain flash translation layer, but is there a way to test that?

thursday 31 january 2013 à 15:56 DePingus said : #9

How can we find out what block size our USB device has? Also, thanks.

thursday 31 january 2013 à 16:22 Tanguy said : #10

@DePingus : I use parted print for that, but there may be other ways of getting it.

thursday 31 january 2013 à 18:40 S P Arif Sahari Wibowo said : #11

About wear levelling, I believe USB flash drives have no specific logic on wear levelling for FAT. It simply will replace the block more frequently because it used more often. Indeed in FAT / FAT32 formatting one must be careful to make sure the FAT align properly with the flash' block.
Since UDF is logging file system (any write, even update, will be written in the previously unused block), I believe it will be by nature do better wear levelling than FAT / FAT32.

thursday 31 january 2013 à 18:43 S P Arif Sahari Wibowo said : #12

Is there anyway to make UDF formatted drive bootable? The fact that UDF does not use the first sector seems to be promising, but I haven't found any writing that show that this is possible.

thursday 31 january 2013 à 20:23 Tanguy said : #13

@S P Arif Sahari Wibowo : I believe it is the opposite. Usual file systems such as FAT leave some space at the beginning to put some executable code, but I am not sure UDF also does. Anyway, modern boot loaders do not require to be at the beginning of a partition, so what you can do, asuming you boot from a BIOS, is: create an MS-DOS partition table, create a partition, format it with UDF, then install GRUB there. I usually install GRUB in a rather manual way and I can assure you that it does work with UDF, but the automated GRUB installer should work too.

thursday 31 january 2013 à 20:55 ludovico said : #14

any cons to using udf also for external hard disks?

thursday 31 january 2013 à 21:31 Tanguy said : #15

@ludovico : Not that I know of, but then, I have never tried it since I do not have an external hard drive… I would be interested to know the result if you try it.

friday 01 february 2013 à 04:15 JanKusanagi said : #16

Interesting idea, thanks!

friday 01 february 2013 à 05:05 Martin said : #17

Can I use UDF for the SD card on my Android (CyanogenMod9) telephone? FAT works, but file name limitations (no question mark) s*ck. ext-2/3/4 didn't work.

friday 01 february 2013 à 08:28 Tanguy said : #18

@Martin : Someone told me it was not supported, but you should try it by yourself, and if it does not work, ask Google to implement it, which should be easy for them since Linux already supports it.

friday 01 february 2013 à 10:01 Folknology said : #19

Hmm I wonder if there are any open embedded (non linux) implementations that would be handy and could be much less of a PITA than FAT.

regards
Al

friday 01 february 2013 à 10:12 Tanguy said : #20

@Folknology : Hardware or rather, firmware implementations of UDF certainly exist, because it is the standard file system of DVDs. There is no reason why these implementations would not be usable for USB sticks as well, but I doubt hardware manufacturers ever thought of that possibility.

friday 01 february 2013 à 10:29 Folknology said : #21

Interesting thread here about possibility of using UDF on AVRs : http://www.avrfreaks.net/index.php?name=PNphpBB2&file=viewtopic&t=101200&start=all&postdays=0&postorder=asc

friday 01 february 2013 à 20:12 Eligius said : #22

Is there a way to identify the UDF-formatted device by label or UUID? I've tried to format using the various --*id= parameters to no avail. However, Windows *does* recognise the LVID and shows it as a label.

This would not bother me but pendrives have no fixed /dev/sd? block and therefore I can't put the flash drive in /etc/fstab.

blkid does not show anything usable.

saturday 02 february 2013 à 21:18 nom said : #23

Max filesize of FAT32 is (almost) 4GB.
Max volume size when using 4K sectors is 16TB

monday 04 february 2013 à 14:44 Sellers said : #24

Just for those MacOSX users:
(use sudo if needbe)

diskutil unmount /dev/disk{x}s{x} (where x is the disk and slice number, usually 1 for both)

newfs_udf -b 512 /dev/disk1 (defaults should work for media type and other options)

diskutil mount /dev/disk{x} (don't need partition anymore)

thursday 07 february 2013 à 03:21 MoritzJT said : #25

Hi there, I stumbled upon your article while searching for the best way to format my 1TB external USB harddrive. Would you recommend me the UDF way, (Needs to run on Mac, Linux, Windows and Symbian) exFAT was my prior choice but got broken too many times. And I think Symbian only supports UDF, FAT, FAT32 and exFAT.

Sorry if I sound totally clueless - I am!

Cheers

sunday 10 february 2013 à 20:52 Tanguy said : #26

@MoritzJT : Yes, I would, this is the whole point of this article! Just try it…

sunday 17 february 2013 à 09:31 Adam said : #27

@Tanguy: I think you may be wrong. See here:

https://lwn.net/Articles/428584/

"In contrast, the more common SD cards and USB flash drives are very sensitive to specific access patterns and can show very high latencies for writes unless they are used with the preformatted FAT32 file layout."

"the FAT file table area (from 4MB to 8MB) is managed differently, enabling small writes to be done efficiently there."

"The FAT area in the second segment is a bit slower when reading because it uses a block remapping algorithm."

"Drives that are hardwired to linear-access optimized segments are basically useless for ext3 and most other Linux filesystems because of this, because they keep small data structures like inodes and block bitmaps in front of the actual data and need to seek back to these in order to write new small files."

"This approach is normally used only in the first few segments on the drive, which contain the file allocation table in FAT32 preformatted drives. Some drives are also able to use this mode when they detect access patterns that match writes to a FAT32 style directory entry."

"Obviously, any such optimizations don't normally do the right thing when a different filesystem is used on the drive than it was intended for, but there is some potential for optimization, e.g. by ensuring that the ext3 journal uses the blocks that are designed to hold the FAT."

"...but some brands can only have one or two segments open at a time, which causes them to constantly go through garbage collection when used with most of the common filesystems other than FAT32."

"When a drive reserves the segments specifically to hold the FAT, these will always be open to allow updating it while writing streaming data to other segments."

Seems like LWN did their homework. As much as I'd love to avoid using FAT on flash devices, it sounds to me like it's still the wisest choice as far as media longevity is concerned.

sunday 17 february 2013 à 09:33 Adam said : #28

And for more: https://encrypted.google.com/search?q=usb+flash+fat+wear+leveling

Rss feed of the article's comments

Tanguy Ortolo