Linux-kernel mailing list archive 2000-32,: Re: NTFS-like streams?

# Re: NTFS-like streams?

Cesar Eduardo Barros (cesarb@nitnet.com.br)
Sat, 12 Aug 2000 16:10:36 -0300

- Messages sorted by: [ date ][ thread ][ subject ][ author ]
- Next message: Linus Torvalds: "Re: USB initialisation"
- Previous message: Linus Torvalds: "Re: [BUG] PCMCIA CardBus problems"
- Maybe in reply to: Christopher Vickery: "NTFS-like streams?"
- Next in thread: Rares Marian: "Re: NTFS-like streams?"

--tThc/1wpZn/ma/RB
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline

This discussion is looking like an year-old discussion we had in lkml before.

I think that a streams-like structure can be created in userspace, and made a
specification for a "libalbod" to manage it. But I buried the project after I
noticed it was useless.

So, here's the design for libalbod I made long ago. I'm not going to implement
it, so feel free to make a sourceforge project out of it.

    --
    Cesar Eduardo Barros
    cesarb@nitnet.com.br
    cesarb@dcc.ufrj.br

    --tThc/1wpZn/ma/RB
    Content-Type: text/plain; charset=us-ascii
    Content-Disposition: attachment; filename="albod_ref.txt"

    Here is my proposal for a working implementation of albods.

    The current requeriments is:

    1. It must be backwards compatible without the need for LD_PRELOAD hacks, but
    should be able to use them

    2. For some file formats (like mp3), the data part is the only important one;
    for some others (e.g. a new albod-only fileformat) the albod should be traeted
    as a single unit for non-albod-aware programs

    3. The albod must be represented as a flat directory; however, we have to think
    hard about albods-within-albods and compatibility issues. Also, it must have a
    'packed' representation and should be able to convert between them on the fly.
    The packed representation should be readable by standard unix tools. Also, to
    confuse a bit more, it can have subdirs which can pack/unpack on the fly like
    itself (so it can still be treated like a flat directory).

    [FIXME: any other ideas?]

    So I planned this structure:
    - Albod is filename.albod, e.g., my_song.mp3.albod
    - Data part, if using compatibility mode, is my_song.mp3
    - Dot files within the albod are reserved
    - Applications should agree to what should be the name of their subparts

    The structure is this:
    my_song.mp3.albod/
    	.options
    	data
    	icon.xpm
    my_song.mp3 -> hardlink to my_song.mp3.albod/data

    Hard links generate maximum compatibility, but causes some randomness if they
    are deleted. My current idea is to ignore it to avoid confusion and creating an
    utility to fix the albod manually.

    Here are the explanations for the files:
    my_song.mp3.albod/	-	Treated as a file called my_song.mp3 by albod
    				aware apps
    	.options	-	Contains (without quotes, one per line):
    				MUST BE THE FIRST ELEMENT WHEN PACKED!
    				FIRST LINE IS ALWAYS "ALBOD=6518F161"
    				(file magic)!

    				"compat=data" if the compatibility link should
    					be a hard link to the 'data' part
    				It is used for already existing formats like
    				.mp3

    				"compat=dir" if the compatibility link should
    					be a symlink to the whole albod dir (we
    					can't hardlink directories)
    				Used when you want to treat the albod as a dir
    				when using old apps

    				"compat=albod" if the compatibility link should
    					be a file which is the whole albod in
    					packed form (treated as compat=none if
    					there is no LD_PRELOAD hack running)
    				Used when you want to be able to 'cp' a whole
    				albod. This is the best idea for new apps.

    				"compat=none" means there is no compatibility
    					link and that non-albod apps should not
    					be doing anything with this albod
    				Used when you're picky and thinks legacy apps
    				should not be used in your shiny new file. Also
    				used when the LD_PRELOAD hack is not available.

    				DEFAULT: compat=none

    				"pack=always" means it should be unpacked on
    					the fly by the lib and should not be
    					stored as a normal dir
    				This uses less disk space and is slower. This
    				also makes compat=albod work without LD_PRELOAD
    				hacks.

    				"pack=auto" means it can be stored as a dir
    					and will be autounpacked to a dir
    					(replacing the packed file)
    				This uses less disk space but requires a lot of
    				it while unpacking. This is used so that you
    				can copy an albod as a file by some random app
    				and it will be unpacked to its normal form when
    				some albod-aware lib touches it.
    				The autounpack option controls the way it is
    				done.

    				"pack=never" means it _should_ be stored as a
    					dir. A program needs override flags to
    					force packing anyways. It can read it
    					as a file if it wants, but when created
    					from file will fully unpack.
    				With pack=auto, a cron job/disk packing daemon/
    				user requested pack-all-my-dirs-because-I-need-
    				more-space can repack the albod. This option
    				is not intended to be used except in some very
    				special cases.

    				DEFAULT: pack=auto

    				"autounpack=full" means that when albod is
    					opened it is fully unpacked if it was
    					packed. The original file is replaced
    					by the dir.
    				This avoids the loss of disk space caused by
    				constant unpacking and is faster.

    				"autounpack=auto" means that only the
    					acessed files will be unpacked. After
    					all are acessed (and unpacked), the
    					file will be replaced by the dir.
    				Faster startup than autounpack=full, but uses
    				a lot of disk space before it gets fully
    				unpacked.

    				"autounpack=partial"
    					Like autounpack=auto, but files will be
    					repacked after being closed. The dir
    					will never apear unless explicitly
    					requested.
    				Faster than pack=always, and uses less space
    				than autounpack=auto. Opens the whole albod
    				faster than autounpack=full.

    				"autounpack=never"
    					Like autounpack=partial, but do not
    					allow the dir to appear even if
    					requested.
    				Used when you want the faster speed of
    				pack=auto but do not want to allow it to appear
    				as a dir.

    				DEFAULT: autounpack=full

    				"version=0"
    					If version is _not_ zero, the program
    					must report incompatible file formats.
    				Used for forward compatibility.

    				DEFAULT: version=0

    				"mimetype=application/octet-stream"
    					MIME type for this albod's 'data' part.
    					If it is "application/x-albod" it means
    					this albod's data is understood only
    					by albod-aware apps.

    				DEFAULT: if compat=data
    					mimetype=application/octet-stream
    					 else
    					mimetype=application/x-albod

    				Note that compat=data won't make sense with
    				pack=always or autounpack=never. Also
    				autounpack makes no sense with pack != auto.
    	.lock		-	Subdir with one item per item in the root of
    				the albod. If it is present here, it has the
    				FQDN:PID of the program using this item
    				(part/dotfile).
    				This subdir won't appear in packed albods, and
    				is the only item that can't be locked (for
    				obvious reasons).
    	data		-	The main data part. Required if compat=data or
    				mimetype is not albod.
    				Optional if compat != data and
    				mimetype==application/x-albod or x-albod-subdir.
    	data.*		-	Application-specific parts start with data. .
    	icon.xpm	-	An example of a non-application-specific part.
    				A icon. Completely optional.
    .my_song.mp3.albod	-	A packed albod. It is a normal file, not a dir.
    .my_song.mp3.albod.tmp/	-	The partially-unpacked albod dir
    .my_song.mp3.albod.lock	-	This locks the whole albod. Used when messing
    				with the directory/packed file structure.
    				It contains FQDN:PID.
    my_song.mp3		-	This is the compatibility link to:
    				my_song.mp3.albod{,.tmp}/data (hardlink); or
    				my_song.mp3.aldob{,.tmp}/ (symlink); or
    				my_song.mp3.albod (symlink to file); or
    				nothing (it isn't there).

    If any of the content parts (data, data.* and the non-app-specific parts) is a
    subdir, it is treated like an albod (the .options file is inherited with the
    following changes: compat=none, and mimetype for compat=none is
    application/x-albod-sudir. These can be overriden).

    When an albod-aware wants to open an albod, it calls the albod lib with the
    filename without the extension. This means:
    1. If my_song.mp3.albod exists, it does its part (magic unpacks/whatever) and
       returns the whole albod;
    2. If it doesn't but my_song.mp3 exists, it creates a 'virtual albod' in memory
       with only the 'data' part (the default values for the options file are right
       in this case, since compat=data and mimetype=application/octet-stream). If
       the albod is changed (not the locks or the data), a real albod is created
       (holding the main albod lock, creating the dir and hardlinking the data
       part, creating the changed file, getting the lock on it and releasing the
       albod global lock)

    The packed albod is just an ar file. If the albod has other albods within it,
    they will be packed first. If needed, a dir called .my_song.mp3.albod.pack and
    its lockfile (.my_song.mp3.albod.pack.lock) will be created to hold temporary
    files.

    Albods _must_ have .options files. If it doesn't or its magic is invalid it is
    treated as a normal dir/file.

    Files within an albod are treated like files outside an albod for albod-aware
    apps, that is:
    - If my_song.mp3.albod exists, treat it as my_song.mp3 and ignore the real
      my_song.mp3 if it exists
    - If my_song.mp3 exists but my_song.mp3.albod doesn't, treat it like a default
      albod (like when opening it)
    - dotfiles are ignored (this is to be consistent with what happens inside an
      albod).
    - Normal dirs are treated like subdir albods (compat=none, mimetype
      x-albod-subdir).

    [FIXME: s/exists/exists and is valid/ above]

    [FIXME: explain this mess better]

    --tThc/1wpZn/ma/RB--

    -
    To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
    the body of a message to majordomo@vger.rutgers.edu
    Please read the FAQ at http://www.tux.org/lkml/

- Next message: Linus Torvalds: "Re: USB initialisation"
- Previous message: Linus Torvalds: "Re: [BUG] PCMCIA CardBus problems"
- Maybe in reply to: Christopher Vickery: "NTFS-like streams?"
- Next in thread: Rares Marian: "Re: NTFS-like streams?"