USB sticks are traditionally formatted with FAT 32, because this file system is implemented by almost every operating system and device. Unfortunately, it sucks, as it cannot use more than 2 TiB, store files larger than 2 GiB or store symbolic links for instance. In a word, it is an obsolete and deficient file system.
Good news: someone addressed that problem. Bad new: that someone is Microsoft. So as you could expect, exFAT, the extended FAT, is a stinking proprietary, secret and patented file system. There are free implementations of that shit, but it is safer to stay away from it.
Installing an operating system
When you want to install a new operating system, the first thing to do is to get an installer medium, which usually involves two steps: downloading an image, and writing it to a medium. There are two major types of media usable for that purpose: optical discs and USB sticks. There used to be a third one, floppy disks, but it is no longer used, and it was conceptually very similar to USB sticks.