User Rating: 3 / 5

Star ActiveStar ActiveStar ActiveStar InactiveStar Inactive
 

Scenery

it's a common situation that we have a modern operative system (i.e. Linux) but with "limited" space and for compatibility purposes (i.e. old operative systems, uefi/recovery environments, game console systems) we still hold very large amounts of space (i.e. partitions, external hard disks) in FAT systems (typically FAT32) and then we want (or need) to use them to store big files (bigger that 4 GB, i.e. backups, system images) but we can't because of the FAT system file size limitations.

Workaround

a way to approach this scenery in Linux is by making use of a combination of Logical Volume Management (LVM), loop devices and modern file systems (i.e. xfs). Basically all that is need to be done to overcome the limitation is to use the FAT system space to create another modern file system inside it which doesn't have that file size limitation.

a proceeding to do that is to create a batch of files on our FAT system, loop those files, create physical volumes on those loop files, create a volume group containing all those physical volumes and then use all that volume group space to create a logical volume. finally once we accomplished that we'll have a block device that we can format with any file system we want and that (virtually for our purpose) doesn't have a file size restriction to work with it like with any other storage space (i.e. partition, image).

now that you know how to work with files bigger than 4 GB in a FAT system a example of how to create such a structure in Linux is the following script:

    #!/usr/bin/sh
    for i in {10..12}; do
     dd if=/dev/zero of=$2/$i bs=4096 count=1048575
     losetup /dev/loop$i $2/$i
     pvcreate /dev/loop$i
    done
    vgcreate $1 /dev/loop{10..12}
    lvcreate -l 100%FREE -n $1 $1
    mkfs.xfs /dev/mapper/$1-$1

saving that in a text file named create.sh and then doing something like sh create.sh backup /mnt/fat32/backup in a root terminal will create a roughly 12 GB (3 physical volumes/4 GB limitation per physical volume) /dev/mapper/backup-backup loop block device inside the /mnt/fat32/backup path (obviously replace that path by one that makes sense in your system) and formats it as xfs.

as noted, that's just a example, it doesn't pretend to be a generalized solution, so feel free to customized as you see fit. to create a bigger one for example simply extend the loop, typing {10..60} instead of {10..12} for example will create a roughly 200 GB space (which I believe it should be around the max anyone may need for a scenery like this one). also it's not even necessary to zero initialize the physical volumes however I like to do it for safety measures (to ensure that the space is properly writable) and is also important to take in consideration the FAT fragmentation so try to allocate a contiguous bock of clusters whenever possible (by doing this when the volume is empty for example and before add other files) other way it will work but the performance will heavily degradate. you may also want to play with cluster/extend/block sizes to fit your actual disk/data properties (though if "unknown" defaults should be ok).

once you have created your block device (following that script, manually typing the commands or with any other method) you just work with it like with any other block device, so after that you just mount/umount it when and where ever you want as usual however since is a custom schema is advisable to better detach it manually in order to attempt to prevent any potential data loss/corruption. following the example, the following command (after being umounted and run as root) will actually detach the device to remove it from your system or to reboot for example:

    dmsetup remove backup-backup && sync && losetup -d /dev/loop{10..12}

now once you attach again (after being detached) your extarnal hd or your system just rebooted to access this custom storage space you can do it with the following command (following the actual example once the FAT system is mounted and run as root):

    for i in {10..12}; do losetup /dev/loop$i /mnt/fat32/bakup/$i; done && sync && lvscan

now you're ready again to work with this device (/dev/mapper/backup-backup in the example) as usual and break that infamous FAT system file size limitation. this procedure also serves as a viable way to have other kind of file systems (i.e. Linux ext) to work with (i.e. symbolic links, permissions) inside a shared FAT system.

finally this article writes about Linux and I can't tell for sure but generally speaking I believe that similar solutions should be available for other operative systems.

Add comment


Security code
Refresh