Tanguy Ortolo

wednesday 30 january 2013 à 14:19 Mikael Eriksson said : #1

Careful with UDF on linux, when the filesystem is around 70-80GB full, you will not be able to write more to it, even if there is lots of free space. It will still work correctly in windows though.

wednesday 30 january 2013 à 18:28 Camelek said : #4

Tanguy you are genius! :D
I will try that when I only get bigger pendrive :]

wednesday 30 january 2013 à 22:32 Olzvoi said : #5

Is it really necessary to give the option -b 512 to mkudffs? The man page reads "If omitted, mkudffs block-size is 2048". I tested it leaving the option - it seems to be working fine with a SD card.

thursday 31 january 2013 à 06:02 Adam said : #7

I have read that USB flash drives often wear-level sectors used for the FAT differently than the rest of the drive, so using non-FAT filesystems may cause premature failure. Have you considered this?

thursday 31 january 2013 à 15:56 DePingus said : #9

How can we find out what block size our USB device has? Also, thanks.

thursday 31 january 2013 à 18:40 S P Arif Sahari Wibowo said : #11

About wear levelling, I believe USB flash drives have no specific logic on wear levelling for FAT. It simply will replace the block more frequently because it used more often. Indeed in FAT / FAT32 formatting one must be careful to make sure the FAT align properly with the flash' block.
Since UDF is logging file system (any write, even update, will be written in the previously unused block), I believe it will be by nature do better wear levelling than FAT / FAT32.

thursday 31 january 2013 à 18:43 S P Arif Sahari Wibowo said : #12

Is there anyway to make UDF formatted drive bootable? The fact that UDF does not use the first sector seems to be promising, but I haven't found any writing that show that this is possible.

thursday 31 january 2013 à 20:55 ludovico said : #14

any cons to using udf also for external hard disks?

friday 01 february 2013 à 04:15 JanKusanagi said : #16

Interesting idea, thanks!

friday 01 february 2013 à 05:05 Martin said : #17

Can I use UDF for the SD card on my Android (CyanogenMod9) telephone? FAT works, but file name limitations (no question mark) s*ck. ext-2/3/4 didn't work.

friday 01 february 2013 à 10:01 Folknology said : #19

Hmm I wonder if there are any open embedded (non linux) implementations that would be handy and could be much less of a PITA than FAT.

regards
Al

friday 01 february 2013 à 10:29 Folknology said : #21

Interesting thread here about possibility of using UDF on AVRs : http://www.avrfreaks.net/index.php?name=PNphpBB2&file=viewtopic&t=101200&start=all&postdays=0&postorder=asc

friday 01 february 2013 à 20:12 Eligius said : #22

Is there a way to identify the UDF-formatted device by label or UUID? I've tried to format using the various --*id= parameters to no avail. However, Windows *does* recognise the LVID and shows it as a label.

This would not bother me but pendrives have no fixed /dev/sd? block and therefore I can't put the flash drive in /etc/fstab.

blkid does not show anything usable.

saturday 02 february 2013 à 21:18 nom said : #23

Max filesize of FAT32 is (almost) 4GB.
Max volume size when using 4K sectors is 16TB

monday 04 february 2013 à 14:44 Sellers said : #24

Just for those MacOSX users:
(use sudo if needbe)

diskutil unmount /dev/disk{x}s{x} (where x is the disk and slice number, usually 1 for both)

newfs_udf -b 512 /dev/disk1 (defaults should work for media type and other options)

diskutil mount /dev/disk{x} (don't need partition anymore)

thursday 07 february 2013 à 03:21 MoritzJT said : #25

Hi there, I stumbled upon your article while searching for the best way to format my 1TB external USB harddrive. Would you recommend me the UDF way, (Needs to run on Mac, Linux, Windows and Symbian) exFAT was my prior choice but got broken too many times. And I think Symbian only supports UDF, FAT, FAT32 and exFAT.

Sorry if I sound totally clueless - I am!

Cheers

sunday 17 february 2013 à 09:31 Adam said : #27

@Tanguy: I think you may be wrong. See here:

https://lwn.net/Articles/428584/

"In contrast, the more common SD cards and USB flash drives are very sensitive to specific access patterns and can show very high latencies for writes unless they are used with the preformatted FAT32 file layout."

"the FAT file table area (from 4MB to 8MB) is managed differently, enabling small writes to be done efficiently there."

"The FAT area in the second segment is a bit slower when reading because it uses a block remapping algorithm."

"Drives that are hardwired to linear-access optimized segments are basically useless for ext3 and most other Linux filesystems because of this, because they keep small data structures like inodes and block bitmaps in front of the actual data and need to seek back to these in order to write new small files."

"This approach is normally used only in the first few segments on the drive, which contain the file allocation table in FAT32 preformatted drives. Some drives are also able to use this mode when they detect access patterns that match writes to a FAT32 style directory entry."

"Obviously, any such optimizations don't normally do the right thing when a different filesystem is used on the drive than it was intended for, but there is some potential for optimization, e.g. by ensuring that the ext3 journal uses the blocks that are designed to hold the FAT."

"...but some brands can only have one or two segments open at a time, which causes them to constantly go through garbage collection when used with most of the common filesystems other than FAT32."

"When a drive reserves the segments specifically to hold the FAT, these will always be open to allow updating it while writing streaming data to other segments."

Seems like LWN did their homework. As much as I'd love to avoid using FAT on flash devices, it sounds to me like it's still the wisest choice as far as media longevity is concerned.

sunday 17 february 2013 à 09:33 Adam said : #28

And for more: https://encrypted.google.com/search?q=usb+flash+fat+wear+leveling

wednesday 29 may 2013 à 23:56 udby said : #29

1. Can a USB media with UDF FS, boot with Legacy and UEFI BIOS with Linux kernel? If so how ? mkudffs doesn't seem to have all the reqd. options to do this.
2. or, Can I have one FAT32 partition and a second partition that is UDF. I can't seem to do this, as parted or fdisk doesn't understand UDF.

tuesday 04 june 2013 à 14:49 Rpnpif said : #31

With Debian Linux, you should install the udftools packages from the official repository.

Cheers.

sunday 16 june 2013 à 09:45 Subodh said : #32

Thaks for the great info. I did not see anything further, though, on the problem writing on linux to drives 70-80GB full. If that is a real problem with no fix, using this method on linux with something like 1TB external drive may not be recommended. No?

wednesday 19 june 2013 à 14:38 antistress said : #34

"The -b 512 is to force a file system block size equal to the USB stick's physical block size, as required by the UDF specification. Adapt it if you have the luck of having a USB stick with an more appropriate block size."

How can I know the USB stick's physical block size ?
Here [1] a are differents answears but i'm a bit lost

Also, I've read that blocksize=512 mau be mandatory for Windows compatibilty [2]

BTW it would have been great to be able to use a GUI for that. I've checked Palimpset 3.0 on Debian Sid but even with the udftools packages installed, UDF is not listed.

[1] http://unix.stackexchange.com/questions/52215/determine-the-size-of-a-block-device
[2] http://forum.ubuntu-fr.org/viewtopic.php?id=1131391

saturday 10 august 2013 à 01:05 Tom Maneiro said : #36

I've tried the whole UDF enchilada for my external drives... and while it makes a lot of sense for USB sticks (even if you consider that you lose write access under XP which on some markets isn't going away anytime soon), it's absolutely a miserable option for large HDDs, and if you care about data integrity at all, UDF is not your winner. Why?

The lack of fully-functional filesystem check tools.

- Linux DOES NOT HAVE a fsck for UDF (there is a "fsck.udf" in udftools, but it's nothing than a stub)
- Windows chkdsk does support UDF on those versions that can write to UDF. Sadly, it doesn't even recognize the filesystem on really large drives (say, anything over 1TB?) - it tells that your drive is "RAW" even if Windows mounts it just fine.
- Solaris DOES have a functional fsck for UDF... that has the same problems as the Windows one (fail to support large drives). Or at least the one in OpenIndiana has this problem.
- Not sure about Mac, don't really care.
- There is a multiplatform UDF "compliance checker" by Philips, but it only checks for UDF spec compliance (and gets very vocal at the minor out-of-spec value), it doesn't actually fix your filesystem.

Why data integrity matters? Because at the first filesystem error, funky things will happen:
- Linux will keep happily writting data to the drive, until something goes wrong and your filesystem gets reduced to a heap of molten scrap
- Windows WILL REFUSE TO WRITE ANYTHING to the drive, until it gets cleared by chkdsk. Oh, you're using a 1TB partition? Then no chkdsk for you, and therefore your drive ends read-only forever.

Sadly I ended with NTFS for my external USB affairs, at least drivers are mature enough for Linux (ntfs-3g), and as a bonus these can do POSIX and symlinks (not NTFS native symlinks, but something called "Interix symlinks", which won't work under Windows unless if you install SUA).

monday 16 september 2013 à 02:02 Sergey said : #37

For those who do not know how to determine the Linux path to the USB drive in the /dev tree, you can use the following command:

sudo lshw -C disk

saturday 28 september 2013 à 16:37 MH said : #38

The best two references about reformatting and wear levelling problems:
http://wiki.laptop.org/go/How_to_Damage_a_FLASH_Storage_Device
https://wiki.linaro.org/Flash%20memory

See also:
https://www.sdcard.org/downloads/formatter_4/

Note: this issue applies to UDF and NTFS and else.

sunday 10 november 2013 à 22:53 David Balažic said : #39

There is UDF read/write software for XP and older Windows, mostly targeted to be used with optical media, but they might work with USB sticks too. Look for Nero InCD, Sonic DLA, DirectCD and similar.

sunday 22 december 2013 à 15:56 Dohn Joe said : #40

This idea is sheer GENIUS! Most DVD/DivX players with a USB/SD slot have UDF built in to play optical discs and so can read it!!

sunday 22 december 2013 à 17:49 Jakub Sadowski said : #41

This is a fantastic recommendation, I can't believe I hadn't heard of it years ago.

I only have one question: how do I set the permissions to "No Specific User" and "No Specific Group"? I've tried the (seemingly standard) linux UID/GID of 65534 (nouser/nogroup) but this doesn't work...

friday 10 january 2014 à 15:14 Jonas Platte said : #42

I tried to format my 320GB external hard drive with UDF, but after a while of copying, the write speed dropped to less than 10MB/s (I have no clue why).
It is tremendous that I can't have my drive formatted with a free filesystem that is readable and writeable in Windows + Linux and supports files bigger than FAT32 does.

tuesday 18 february 2014 à 14:07 phlebire said : #43

Hello,

I am looking for a FS that could be as portable as possible (Windows, Max & Linux essentially) and have a sense of security (it seems that I would be looking for a journalism-style FS).

So FAT won't work since they don't have a journalism system, but NTFS isn't as portable. I felt like this UDF idea would be great but I might deal with large amount of data therefore what Tom Maneiro has said has pretty much scared me off:



"I've tried the whole UDF enchilada for my external drives... and while it makes a lot of sense for USB sticks (even if you consider that you lose write access under XP which on some markets isn't going away anytime soon), it's absolutely a miserable option for large HDDs, and if you care about data integrity at all, UDF is not your winner. Why?

The lack of fully-functional filesystem check tools.

- Linux DOES NOT HAVE a fsck for UDF (there is a "fsck.udf" in udftools, but it's nothing than a stub)
- Windows chkdsk does support UDF on those versions that can write to UDF. Sadly, it doesn't even recognize the filesystem on really large drives (say, anything over 1TB?) - it tells that your drive is "RAW" even if Windows mounts it just fine.
- Solaris DOES have a functional fsck for UDF... that has the same problems as the Windows one (fail to support large drives). Or at least the one in OpenIndiana has this problem.
- Not sure about Mac, don't really care.
- There is a multiplatform UDF "compliance checker" by Philips, but it only checks for UDF spec compliance (and gets very vocal at the minor out-of-spec value), it doesn't actually fix your filesystem.

Why data integrity matters? Because at the first filesystem error, funky things will happen:
- Linux will keep happily writting data to the drive, until something goes wrong and your filesystem gets reduced to a heap of molten scrap
- Windows WILL REFUSE TO WRITE ANYTHING to the drive, until it gets cleared by chkdsk. Oh, you're using a 1TB partition? Then no chkdsk for you, and therefore your drive ends read-only forever.

Sadly I ended with NTFS for my external USB affairs, at least drivers are mature enough for Linux (ntfs-3g), and as a bonus these can do POSIX and symlinks (not NTFS native symlinks, but something called "Interix symlinks", which won't work under Windows unless if you install SUA)."




Could anyone give me a hint to what I should do? Maybe your personnal experience and/or expertise on the matter (i'm pretty new to file systems)

Thanks!

tuesday 18 february 2014 à 17:22 phlebire said : #45

Thanks Tanguy for answering so fast, but how will an Ext3 file system will be better than the other two?
* portability (it seems it's the same as NTFS (using the correct drivers))
* Ease of implementation (I will probably install linux on a NIOS II and be transfering data through SATA to the HDD)
* Performance (??)
* Integrity (seems it also has a journalism feature)
What you have suggested concerning partitionning my device is a good idea, but at first I might simply try to see if it works on my Linux device.

wednesday 19 february 2014 à 10:05 phlebire said : #47

Great, thanks for all this :-)

saturday 22 february 2014 à 23:15 GreatEmerald said : #48

This is a pretty nice article. One thing that could also be mentioned, though, is that you can use partitions as well (type 06, FAT16). Some OSs like it better that way from what I've read. And you don't need to zero out the whole drive, just the first and last parts of it is enough (GPartEd has a quick format option like that – saves time).

Another thing to note is that in order to make use of POSIX permissions, you have to make sure you mount the volume without umask/fmask/dmask and uid/gid options, as they override the permissions (and, for example, openSUSE 13.1 KDE automounter mounts these with uid=1000,gid=100,umask=77). I think you need to make a udev rule to override these automounter settings.

tuesday 25 february 2014 à 21:54 Thomas Lundqvist said : #49

I can confirm the problem in Linux with writing. I have 69 GB allocated and around 800 GB free and get a disk full. It is a known bug fixed in later kernels: https://bugzilla.kernel.org/show_bug.cgi?id=53021.

wednesday 05 march 2014 à 00:55 Yulian Kuncheff said : #50

I ported an old Perl tool that I found to Node.js so you can create specially formatted UDF drives that will work great on Windows, Linux, and OSX. I haven't checked for the Linux 69gig limit, but I will look into it.

It can be found here: https://github.com/Daegalus/udfhd

And can be installed with npm using: npm install -g udfhd

tuesday 27 may 2014 à 01:49 tuxicoman said : #51

Ca sonne bien cette solution.

Tu as toujours de bon retours après plusieurs échanges Windows/Linux ?

sunday 01 june 2014 à 19:35 tuxicoman said : #53

Ok, merci.

wednesday 16 july 2014 à 09:30 Steve said : #54

Given the prevalence of FAT-optimized cheap sticks, I'm slowly thinking about reviving UMSDOS, in a more VFAT compatible manner.

Sort of UVFAT, which would degrade very gracefully on non-POSIX OS.

wednesday 01 october 2014 à 20:54 Neal Murphy said : #55

BTW, Debian Wheezy has the wipefs(8) utility (part of the util-linux pkg); it removes filesystem signatures from a partition.

Run 'wipefs /dev/sdd1' to see existing signatures.

Run 'wipefs -a /dev/sdd1' to remove all sigs. Then run mkudffs to make the UDF FS.

tuesday 14 october 2014 à 17:41 Inyavic said : #56

According to what Tom Maneiro said, windows chkdsk tool can't check and fix filesystem errors on UDF filesystem and this is very bad. I'm going back to NTFS already.

wednesday 14 january 2015 à 13:42 Guyberton said : #57

People (@Jakub) have been asking about the possibility of using the feature of "no specific user" and "no specific group", a feature that would be desirable on removable media so as not to "hardcode" specific IDs into the permission system. See the comments below this post: http://unix.stackexchange.com/a/38323/16279

In a nutshell, the mount command for UDF allows options to supply a UID and/or GID upon mounting. If these options are *NOT* supplied the filesystem behaves in a way that supports POSIX file permissions just like ext4 etc.
If you do supply these UID/GID values e.g. your own UID, then the filesystem effectively nullifies the permissions system and behave somewhat like FAT32 in this respect. In this way your user "owns" all the files and directories on the filesystem, which is probably what you want for removable flash media.

The GNU "policymount" (pmount) tool will implement this style of mounting of UDF filesystems, supplying by default, the UID/GID of the user that issues the pmount command.

Another thing I have noticed about UDF filesystems is the lack of a UUID.
ls -lF /dev/disk/by-uuid
does not show a reference to the UDF device.

But, UDF does support labels so you can use the labels in the mount command and in /etc/fstab

E.g.
mount LABEL=MyFlashDrive
if you add a corresponding entry in /etc/fstab

Hope this helps!

wednesday 14 january 2015 à 13:54 Guyberton said : #58

Now if only someone would write a decent implementation of fsck for UDF filesystems on a GNU OS!

saturday 24 january 2015 à 20:15 mirabilos said : #59

UDF support is not widespread enough.

I tend to partition “media” sticks with a small FAT filesystem in the front (as NT seems to only recognise the first partition) and a larger ext2fs (not ext3/ext4!) in the back, this works with most OSes (even all BSDs, Atari MiNT, etc).

friday 24 april 2015 à 20:36 brice said : #60

I got it to work nicely between 2 linux machines. MacOS X could not even mount it. Windows 8 could mount it but not write. Am I the only one? Is there a solutions?

wednesday 09 december 2015 à 11:44 arnd said : #61

Generally you'd want the block size to be as large as possible for today's flash devices, as the internal effective page size tends to be around 64KB and smaller writes will be slow.

Try "-b 4096", which is the maximum that UDF supports.

wednesday 09 december 2015 à 21:51 GreatEmerald said : #63

Indeed. When I tried to create a UDF volume with 4096B (and using an actual external HD for it), Windows would interpret it as 512B anyway and either refuse to read it or read it incorrectly (which is way worse).

monday 08 february 2016 à 15:07 hiccup said : #64

I have a pen drive which is already in udf format is there any way to make it into ntfs format. I tried convert in CMD and from device manager also.But it is always recognising as cd drive and not allowing to celete or write in it.

monday 08 february 2016 à 15:09 cheflo said : #65

I got the error `Trying to change type with multiple extend` when using the `mkudfffs`-command here. Using `$ sudo mkudffs -b 512 --media-type=hd --utf8 --lvid=DriveLabel --vid=DriveLabel --fsid=DriveLabel /dev/sdi` instead worked just fine. That command is from http://duncanlock.net/blog/2013/05/13/using-udf-as-an-improved-filesystem-for-usb-flash-drives/

tuesday 23 february 2016 à 22:37 vanilla said : #67

Playing with format-udf.sh 1.1 from github, and with both usb and disk-images on mac-osx-10.11 and linux. In osx man page says fsck_udf cannot do repair, but it seems to work:

bash-3.2$ sudo fsck_udf -D /dev/disk3
Device name = /dev/disk3
/dev/disk3 has a block size of 512, and 2049024 sectors
startSector = 0, endSector = 2049023
** Checking Hierarchy
** Filesystem is clean

Even more fun on osx is the udf support for sparse files. For example you want to use ddrescue to rescue a failing drive or partition on osx box. Without a filesystem that supports sparse files, you had to have at least as much free space as entire size of disk/partition you want to rescue.

Osx has file image called sparse-bundle, so I create a sparse bundle bigger than item I want to rescue. Then format the sparse-bundle to udf. I can then run ddrescue --sparse -f /dev/disk1 /Volumes/udf/disk1.dmg

If the original /dev/disk1 has lots of un-used space, then the sparse-bundle will only grow to the size needed. Then can try repairs/mounting of the dmg created on the udf volume.

Not sure if I would trust udf for mission critical items until we have a way to repair the file-system, tho. So I agree 100% with #58: "Now if only someone would write a decent implementation of fsck for UDF filesystems on a GNU OS!" But the 10-year old code is unmaintained since 2013 at:
https://sourceforge.net/projects/linux-udf/

tuesday 22 march 2016 à 23:52 granger said : #69

What a great idea, from which blog eight years ago did you steal it?

wednesday 27 april 2016 à 22:33 Rimas said : #71

Sadly, what Tom Maneiro wrote three years ago still stands. Linux still doesn't have a working fsck tool for UDF, and on Windows (at least on my Windows 10) "checking for errors" simply freezes on a UDF-formatted drive.

I have a 64 gigabyte USB3 flash drive, which I formatted as UDF with intentions to keep various source-code repositories on it for easier access from both Windows and Linux. And today I noticed that Windows claims it's full. Again. For the third time or so this year. And like I said, "checking for errors", while shows the progress window pretending to do something, doesn't really seem to do anything – even when I press Cancel, nothing happens. Which means I'm gonna have to reformat the USB stick again (or try fsck'ing it from a virtualized OpenSolaris). I guess I'll just go back to VFAT or NTFS this time...

wednesday 27 april 2016 à 22:50 GreatEmerald said : #72

@Rimas: That's odd, because at least when I tried on Windows 10, the checker tool succeeded. In fact, when I had the UDF volume formatted with a sector size > 512B, it seemed to reformat it back to 512B (possibly because there were still leftovers of previous formatting and Windows assumes 512B always).
Did you make sure to format it in 512B from the get-go? Else, it might also be some bug in the Windows checker... Or in Windows in general, if it misdetects the drive size.

P.S. Smagu matyti dar vieną lietuvį :)

thursday 12 may 2016 à 18:00 Guybertron said : #73

Here's a fairly complete introduction to the details of UDF "Wenguang's Introduction to Universal Disk Format": https://sites.google.com/site/udfintro/

saturday 04 february 2017 à 11:29 Abhishek said : #74

I have a pendrive formatted as a UDF drive.
And I need to format it back to FAT 32.
Need some help please.

tuesday 07 february 2017 à 12:53 Abhishek said : #76

Don't mind but a step to step procedure would be really helpful.
Btw,
I've got a Windows system.
Try suggesting with that.

wednesday 08 february 2017 à 09:33 Abhishek said : #78

Fine.
Let's go with another one.
Just tell me a step by step procedure that I can cope up with.
I'm new to this.

monday 27 december 2021 à 10:00 x3mghost said : #79

Why not use exfat cuz that is supported by both linux, macosx and windows and has no limitations on any of them when it comes to read/write and you do not need to do anything due to the support for most OS's we all use. I use ventoy to create a multiboot usb drive@64GB with 10 linux's, 3 windows and a password-changer for windows+hdd clone.iso and this works flawlessly on all..!! Like the option udf if it were to be used with all in a better way, but it seems difficult..? Best of luck, guys..

sunday 18 september 2022 à 19:16 Randy said : #80

@monday this post was made in 2013 before exFAT was properly supported on Linux and Mac.