30 01 | 2013

Using the UDF as a successor of FAT for USB sticks

Written by Tanguy

Classified in : Homepage, Debian, Command line

USB Stick

FAT

USB sticks are traditionally formatted with FAT 32, because this file system is implemented by almost every operating system and device. Unfortunately, it sucks, as it cannot use more than 2 TiB, store files larger than 2 GiB or store symbolic links for instance. In a word, it is an obsolete and deficient file system.

exFAT

Good news: someone addressed that problem. Bad new: that someone is Microsoft. So as you could expect, exFAT, the extended FAT, is a stinking proprietary, secret and patented file system. There are free implementations of that shit, but it is safer to stay away from it.

UDF to the rescue!

Good news: there is one file system that is implemented almost everywhere as well, and which does not suffer from such limitations. UDF, the Universal Disk Format, is an ISO standard originally designed for DVDs, but it is perfectly usable for USB sticks. It also supports POSIX permissions, with one killer feature for removable media: a file can belong to no specific person or group.

So, to use it, assuming you USB stick is /dev/sdc:

$ dd if=/dev/zero of=/dev/sdc bs=1M count=1
$ mkudffs -b 512 --media-type=hd /dev/sdc

The initial dd is to erase the existing partition table or file system information, to prevent you USB stick from being detected as a FAT after it has been formatted with UDF.

The -b 512 is to force a file system block size equal to the USB stick's physical block size, as required by the UDF specification. Adapt it if you have the luck of having a USB stick with an more appropriate block size.

After that, your USB stick will be usable for reading and writing with GNU/Linux and the other free operating systems of course, but also with current versions of Windows (read-only with the outdated version XP) and with MacOS.

54 comments

wednesday 30 january 2013 à 14:19 Mikael Eriksson said : #1

Careful with UDF on linux, when the filesystem is around 70-80GB full, you will not be able to write more to it, even if there is lots of free space. It will still work correctly in windows though.

wednesday 30 january 2013 à 16:33 Tanguy said : #2

Note that UDF is usable for a whole medium, or for a partition. You always have to zero the beginning of the existing data, because UDF does not seem to put its identification data at the same location as FAT, so a device previously formatted with FAT, then re-formatted with UDF can still be detected as a FAT.

wednesday 30 january 2013 à 16:35 Tanguy said : #3

@Mikael Eriksson : Well, this is good to know, I never faced such a situation because I never used devices that large with UDF. I should try that.

wednesday 30 january 2013 à 18:28 Camelek said : #4

Tanguy you are genius! :D
I will try that when I only get bigger pendrive :]

wednesday 30 january 2013 à 22:32 Olzvoi said : #5

Is it really necessary to give the option -b 512 to mkudffs? The man page reads "If omitted, mkudffs block-size is 2048". I tested it leaving the option - it seems to be working fine with a SD card.

wednesday 30 january 2013 à 22:40 Tanguy said : #6

@Olzvoi : Yes, it is necessary. You can use a random file system block size and still be able to mount your file system under Linux, but the UDF specification clearly requires it to match the device's physical block size. Doing so would 1. violate the spec. and 2. render your file system unusable with some operating systems that assume you respect it without checking. That includes Windows systems, and may include other systems as well, because I am not even sure there is a block size field at all in the file system header so Linux's support for random block size may just be guesswork.

thursday 31 january 2013 à 06:02 Adam said : #7

I have read that USB flash drives often wear-level sectors used for the FAT differently than the rest of the drive, so using non-FAT filesystems may cause premature failure. Have you considered this?

thursday 31 january 2013 à 08:19 Tanguy said : #8

@Adam : I have not, but I frankly doubt manufacturers would do such an optimization in such low-cost and generic devices. My guess would be that they just have a plain flash translation layer, but is there a way to test that?

thursday 31 january 2013 à 15:56 DePingus said : #9

How can we find out what block size our USB device has? Also, thanks.

thursday 31 january 2013 à 16:22 Tanguy said : #10

@DePingus : I use parted print for that, but there may be other ways of getting it.

thursday 31 january 2013 à 18:40 S P Arif Sahari Wibowo said : #11

About wear levelling, I believe USB flash drives have no specific logic on wear levelling for FAT. It simply will replace the block more frequently because it used more often. Indeed in FAT / FAT32 formatting one must be careful to make sure the FAT align properly with the flash' block.
Since UDF is logging file system (any write, even update, will be written in the previously unused block), I believe it will be by nature do better wear levelling than FAT / FAT32.

thursday 31 january 2013 à 18:43 S P Arif Sahari Wibowo said : #12

Is there anyway to make UDF formatted drive bootable? The fact that UDF does not use the first sector seems to be promising, but I haven't found any writing that show that this is possible.

thursday 31 january 2013 à 20:23 Tanguy said : #13

@S P Arif Sahari Wibowo : I believe it is the opposite. Usual file systems such as FAT leave some space at the beginning to put some executable code, but I am not sure UDF also does. Anyway, modern boot loaders do not require to be at the beginning of a partition, so what you can do, asuming you boot from a BIOS, is: create an MS-DOS partition table, create a partition, format it with UDF, then install GRUB there. I usually install GRUB in a rather manual way and I can assure you that it does work with UDF, but the automated GRUB installer should work too.

thursday 31 january 2013 à 20:55 ludovico said : #14

any cons to using udf also for external hard disks?

thursday 31 january 2013 à 21:31 Tanguy said : #15

@ludovico : Not that I know of, but then, I have never tried it since I do not have an external hard drive… I would be interested to know the result if you try it.

friday 01 february 2013 à 04:15 JanKusanagi said : #16

Interesting idea, thanks!

friday 01 february 2013 à 05:05 Martin said : #17

Can I use UDF for the SD card on my Android (CyanogenMod9) telephone? FAT works, but file name limitations (no question mark) s*ck. ext-2/3/4 didn't work.

friday 01 february 2013 à 08:28 Tanguy said : #18

@Martin : Someone told me it was not supported, but you should try it by yourself, and if it does not work, ask Google to implement it, which should be easy for them since Linux already supports it.

friday 01 february 2013 à 10:01 Folknology said : #19

Hmm I wonder if there are any open embedded (non linux) implementations that would be handy and could be much less of a PITA than FAT.

regards
Al

friday 01 february 2013 à 10:12 Tanguy said : #20

@Folknology : Hardware or rather, firmware implementations of UDF certainly exist, because it is the standard file system of DVDs. There is no reason why these implementations would not be usable for USB sticks as well, but I doubt hardware manufacturers ever thought of that possibility.

friday 01 february 2013 à 10:29 Folknology said : #21

Interesting thread here about possibility of using UDF on AVRs : http://www.avrfreaks.net/index.php?name=PNphpBB2&file=viewtopic&t=101200&start=all&postdays=0&postorder=asc

friday 01 february 2013 à 20:12 Eligius said : #22

Is there a way to identify the UDF-formatted device by label or UUID? I've tried to format using the various --*id= parameters to no avail. However, Windows *does* recognise the LVID and shows it as a label.

This would not bother me but pendrives have no fixed /dev/sd? block and therefore I can't put the flash drive in /etc/fstab.

blkid does not show anything usable.

saturday 02 february 2013 à 21:18 nom said : #23

Max filesize of FAT32 is (almost) 4GB.
Max volume size when using 4K sectors is 16TB

monday 04 february 2013 à 14:44 Sellers said : #24

Just for those MacOSX users:
(use sudo if needbe)

diskutil unmount /dev/disk{x}s{x} (where x is the disk and slice number, usually 1 for both)

newfs_udf -b 512 /dev/disk1 (defaults should work for media type and other options)

diskutil mount /dev/disk{x} (don't need partition anymore)

thursday 07 february 2013 à 03:21 MoritzJT said : #25

Hi there, I stumbled upon your article while searching for the best way to format my 1TB external USB harddrive. Would you recommend me the UDF way, (Needs to run on Mac, Linux, Windows and Symbian) exFAT was my prior choice but got broken too many times. And I think Symbian only supports UDF, FAT, FAT32 and exFAT.

Sorry if I sound totally clueless - I am!

Cheers

sunday 10 february 2013 à 20:52 Tanguy said : #26

@MoritzJT : Yes, I would, this is the whole point of this article! Just try it…

sunday 17 february 2013 à 09:31 Adam said : #27

@Tanguy: I think you may be wrong. See here:

https://lwn.net/Articles/428584/

"In contrast, the more common SD cards and USB flash drives are very sensitive to specific access patterns and can show very high latencies for writes unless they are used with the preformatted FAT32 file layout."

"the FAT file table area (from 4MB to 8MB) is managed differently, enabling small writes to be done efficiently there."

"The FAT area in the second segment is a bit slower when reading because it uses a block remapping algorithm."

"Drives that are hardwired to linear-access optimized segments are basically useless for ext3 and most other Linux filesystems because of this, because they keep small data structures like inodes and block bitmaps in front of the actual data and need to seek back to these in order to write new small files."

"This approach is normally used only in the first few segments on the drive, which contain the file allocation table in FAT32 preformatted drives. Some drives are also able to use this mode when they detect access patterns that match writes to a FAT32 style directory entry."

"Obviously, any such optimizations don't normally do the right thing when a different filesystem is used on the drive than it was intended for, but there is some potential for optimization, e.g. by ensuring that the ext3 journal uses the blocks that are designed to hold the FAT."

"...but some brands can only have one or two segments open at a time, which causes them to constantly go through garbage collection when used with most of the common filesystems other than FAT32."

"When a drive reserves the segments specifically to hold the FAT, these will always be open to allow updating it while writing streaming data to other segments."

Seems like LWN did their homework. As much as I'd love to avoid using FAT on flash devices, it sounds to me like it's still the wisest choice as far as media longevity is concerned.

sunday 17 february 2013 à 09:33 Adam said : #28

And for more: https://encrypted.google.com/search?q=usb+flash+fat+wear+leveling

wednesday 29 may 2013 à 23:56 udby said : #29

1. Can a USB media with UDF FS, boot with Legacy and UEFI BIOS with Linux kernel? If so how ? mkudffs doesn't seem to have all the reqd. options to do this.
2. or, Can I have one FAT32 partition and a second partition that is UDF. I can't seem to do this, as parted or fdisk doesn't understand UDF.

thursday 30 may 2013 à 08:31 Tanguy said : #30

@udby : This is technically feasible, but, as with any dual-bootable (UEFI + BIOS) media, it will require manual partitioning and GRUB installation. You will need an MS-DOS or GPT partitionning with one EFI System Partition, a BIOS Boot Partition if you use GPT, and your data partition. You will then have to generate GRUB images for BIOS and for UEFI with the appropriate modules (part_msdos or part_gpt, udf and biosdisk for the BIOS image) and with custom prefix paths (see below), to install the BIOS image to the drive with grub-setup, to format the EFI System Partition in FAT 32 and to put the UEFI image into it at the right path, and finally to format your data partition to UDF and put the data for the two GRUB versions there (each of them in a distinct directory, hence the custom prefix paths for the images), in addition to your data of course. In fact, using UDF here does not add any complexity, only dual-booting does.

I am currently preparing a USB drive that is dual-bootable like that, so I may write an article about that.

tuesday 04 june 2013 à 14:49 Rpnpif said : #31

With Debian Linux, you should install the udftools packages from the official repository.

Cheers.

sunday 16 june 2013 à 09:45 Subodh said : #32

Thaks for the great info. I did not see anything further, though, on the problem writing on linux to drives 70-80GB full. If that is a real problem with no fix, using this method on linux with something like 1TB external drive may not be recommended. No?

monday 17 june 2013 à 10:08 Tanguy said : #33

@Subodh : I do not know, never tried it…

wednesday 19 june 2013 à 14:38 antistress said : #34

"The -b 512 is to force a file system block size equal to the USB stick's physical block size, as required by the UDF specification. Adapt it if you have the luck of having a USB stick with an more appropriate block size."

How can I know the USB stick's physical block size ?
Here [1] a are differents answears but i'm a bit lost

Also, I've read that blocksize=512 mau be mandatory for Windows compatibilty [2]

BTW it would have been great to be able to use a GUI for that. I've checked Palimpset 3.0 on Debian Sid but even with the udftools packages installed, UDF is not listed.

[1] http://unix.stackexchange.com/questions/52215/determine-the-size-of-a-block-device
[2] http://forum.ubuntu-fr.org/viewtopic.php?id=1131391

wednesday 19 june 2013 à 14:46 Tanguy said : #35

@antistress : I think the utility “blockdev” is exactly for that.

saturday 10 august 2013 à 01:05 Tom Maneiro said : #36

I've tried the whole UDF enchilada for my external drives... and while it makes a lot of sense for USB sticks (even if you consider that you lose write access under XP which on some markets isn't going away anytime soon), it's absolutely a miserable option for large HDDs, and if you care about data integrity at all, UDF is not your winner. Why?

The lack of fully-functional filesystem check tools.

- Linux DOES NOT HAVE a fsck for UDF (there is a "fsck.udf" in udftools, but it's nothing than a stub)
- Windows chkdsk does support UDF on those versions that can write to UDF. Sadly, it doesn't even recognize the filesystem on really large drives (say, anything over 1TB?) - it tells that your drive is "RAW" even if Windows mounts it just fine.
- Solaris DOES have a functional fsck for UDF... that has the same problems as the Windows one (fail to support large drives). Or at least the one in OpenIndiana has this problem.
- Not sure about Mac, don't really care.
- There is a multiplatform UDF "compliance checker" by Philips, but it only checks for UDF spec compliance (and gets very vocal at the minor out-of-spec value), it doesn't actually fix your filesystem.

Why data integrity matters? Because at the first filesystem error, funky things will happen:
- Linux will keep happily writting data to the drive, until something goes wrong and your filesystem gets reduced to a heap of molten scrap
- Windows WILL REFUSE TO WRITE ANYTHING to the drive, until it gets cleared by chkdsk. Oh, you're using a 1TB partition? Then no chkdsk for you, and therefore your drive ends read-only forever.

Sadly I ended with NTFS for my external USB affairs, at least drivers are mature enough for Linux (ntfs-3g), and as a bonus these can do POSIX and symlinks (not NTFS native symlinks, but something called "Interix symlinks", which won't work under Windows unless if you install SUA).

monday 16 september 2013 à 02:02 Sergey said : #37

For those who do not know how to determine the Linux path to the USB drive in the /dev tree, you can use the following command:

sudo lshw -C disk

saturday 28 september 2013 à 16:37 MH said : #38

The best two references about reformatting and wear levelling problems:
http://wiki.laptop.org/go/How_to_Damage_a_FLASH_Storage_Device
https://wiki.linaro.org/Flash%20memory

See also:
https://www.sdcard.org/downloads/formatter_4/

Note: this issue applies to UDF and NTFS and else.

sunday 10 november 2013 à 22:53 David Balažic said : #39

There is UDF read/write software for XP and older Windows, mostly targeted to be used with optical media, but they might work with USB sticks too. Look for Nero InCD, Sonic DLA, DirectCD and similar.

sunday 22 december 2013 à 15:56 Dohn Joe said : #40

This idea is sheer GENIUS! Most DVD/DivX players with a USB/SD slot have UDF built in to play optical discs and so can read it!!

sunday 22 december 2013 à 17:49 Jakub Sadowski said : #41

This is a fantastic recommendation, I can't believe I hadn't heard of it years ago.

I only have one question: how do I set the permissions to "No Specific User" and "No Specific Group"? I've tried the (seemingly standard) linux UID/GID of 65534 (nouser/nogroup) but this doesn't work...

friday 10 january 2014 à 15:14 Jonas Platte said : #42

I tried to format my 320GB external hard drive with UDF, but after a while of copying, the write speed dropped to less than 10MB/s (I have no clue why).
It is tremendous that I can't have my drive formatted with a free filesystem that is readable and writeable in Windows + Linux and supports files bigger than FAT32 does.

tuesday 18 february 2014 à 14:07 phlebire said : #43

Hello,

I am looking for a FS that could be as portable as possible (Windows, Max & Linux essentially) and have a sense of security (it seems that I would be looking for a journalism-style FS).

So FAT won't work since they don't have a journalism system, but NTFS isn't as portable. I felt like this UDF idea would be great but I might deal with large amount of data therefore what Tom Maneiro has said has pretty much scared me off:



"I've tried the whole UDF enchilada for my external drives... and while it makes a lot of sense for USB sticks (even if you consider that you lose write access under XP which on some markets isn't going away anytime soon), it's absolutely a miserable option for large HDDs, and if you care about data integrity at all, UDF is not your winner. Why?

The lack of fully-functional filesystem check tools.

- Linux DOES NOT HAVE a fsck for UDF (there is a "fsck.udf" in udftools, but it's nothing than a stub)
- Windows chkdsk does support UDF on those versions that can write to UDF. Sadly, it doesn't even recognize the filesystem on really large drives (say, anything over 1TB?) - it tells that your drive is "RAW" even if Windows mounts it just fine.
- Solaris DOES have a functional fsck for UDF... that has the same problems as the Windows one (fail to support large drives). Or at least the one in OpenIndiana has this problem.
- Not sure about Mac, don't really care.
- There is a multiplatform UDF "compliance checker" by Philips, but it only checks for UDF spec compliance (and gets very vocal at the minor out-of-spec value), it doesn't actually fix your filesystem.

Why data integrity matters? Because at the first filesystem error, funky things will happen:
- Linux will keep happily writting data to the drive, until something goes wrong and your filesystem gets reduced to a heap of molten scrap
- Windows WILL REFUSE TO WRITE ANYTHING to the drive, until it gets cleared by chkdsk. Oh, you're using a 1TB partition? Then no chkdsk for you, and therefore your drive ends read-only forever.

Sadly I ended with NTFS for my external USB affairs, at least drivers are mature enough for Linux (ntfs-3g), and as a bonus these can do POSIX and symlinks (not NTFS native symlinks, but something called "Interix symlinks", which won't work under Windows unless if you install SUA)."




Could anyone give me a hint to what I should do? Maybe your personnal experience and/or expertise on the matter (i'm pretty new to file systems)

Thanks!

tuesday 18 february 2014 à 14:37 Tanguy said : #44

@phlebire : You may want to try Ext3, I think there are drivers to use it with platforms other than Linux. It may be useful to have one small FAT partition providing this driver, and a big Ext3 partition covering the remaining space, but that layout may cause problems as Windows support for multiple partitions on external drives is terrible.

tuesday 18 february 2014 à 17:22 phlebire said : #45

Thanks Tanguy for answering so fast, but how will an Ext3 file system will be better than the other two?
* portability (it seems it's the same as NTFS (using the correct drivers))
* Ease of implementation (I will probably install linux on a NIOS II and be transfering data through SATA to the HDD)
* Performance (??)
* Integrity (seems it also has a journalism feature)
What you have suggested concerning partitionning my device is a good idea, but at first I might simply try to see if it works on my Linux device.

tuesday 18 february 2014 à 17:35 Tanguy said : #46

@phlebire : Well, portability: Ext3 is an open format, while NTFS is a closed one, so its implementations on platforms other than Windows are reverse-engineered. Ease of implementation: not sure I understand what you mean by that. Performance: that would be difficult to compare, but Ext3 is usually considered good enough. Integrity: it uses a journal indeed, which secures all writing. By the way, for flash-based USB stick, it is usually better to disable it, which means using Ext2 (Ext3 is just Ext2 with a journal).

wednesday 19 february 2014 à 10:05 phlebire said : #47

Great, thanks for all this :-)

saturday 22 february 2014 à 23:15 GreatEmerald said : #48

This is a pretty nice article. One thing that could also be mentioned, though, is that you can use partitions as well (type 06, FAT16). Some OSs like it better that way from what I've read. And you don't need to zero out the whole drive, just the first and last parts of it is enough (GPartEd has a quick format option like that – saves time).

Another thing to note is that in order to make use of POSIX permissions, you have to make sure you mount the volume without umask/fmask/dmask and uid/gid options, as they override the permissions (and, for example, openSUSE 13.1 KDE automounter mounts these with uid=1000,gid=100,umask=77). I think you need to make a udev rule to override these automounter settings.

tuesday 25 february 2014 à 21:54 Thomas Lundqvist said : #49

I can confirm the problem in Linux with writing. I have 69 GB allocated and around 800 GB free and get a disk full. It is a known bug fixed in later kernels: https://bugzilla.kernel.org/show_bug.cgi?id=53021.

wednesday 05 march 2014 à 00:55 Yulian Kuncheff said : #50

I ported an old Perl tool that I found to Node.js so you can create specially formatted UDF drives that will work great on Windows, Linux, and OSX. I haven't checked for the Linux 69gig limit, but I will look into it.

It can be found here: https://github.com/Daegalus/udfhd

And can be installed with npm using: npm install -g udfhd

tuesday 27 may 2014 à 01:49 tuxicoman said : #51

Ca sonne bien cette solution.

Tu as toujours de bon retours après plusieurs échanges Windows/Linux ?

tuesday 27 may 2014 à 09:50 Tanguy said : #52

@tuxicoman : Non, je n'ai jamais utilisé ça pour faire beaucoup d'échanges, surtout que je n'ai pas de Windows sous la main…

sunday 01 june 2014 à 19:35 tuxicoman said : #53

Ok, merci.

wednesday 16 july 2014 à 09:30 Steve said : #54

Given the prevalence of FAT-optimized cheap sticks, I'm slowly thinking about reviving UMSDOS, in a more VFAT compatible manner.

Sort of UVFAT, which would degrade very gracefully on non-POSIX OS.

Write a comment

What is the first letter of the word zlsp? : 

Archives