Linux-kernel mailing list archive 1999-25,: file streams a la mac, ntfs, etc.

# file streams a la mac, ntfs, etc.

Mike Touloumtzis (miket@geoworks.com)
Thu, 24 Jun 1999 00:18:44 -0700

- Messages sorted by: [ date ][ thread ][ subject ][ author ]
- Next message: Thierry Danis: "2.2.10-ac4 / knfsd 1.4.1 poor parallel reading performance"
- Previous message: Andrew Hilborne: "Re: analysis: securelevels vs securebits with capabilities"

Some thoughts about the 'streams within files' problem:

-- I basically agree with Ted.
-- You probably don't care since you don't know who I am and I haven't
contributed to the kernel.
-- I hope you read this anyway :-)

If an implementation of a feature like this is to go into the kernel,
it should at least strive to be a reasonable abstraction of the
problem rather than an imitation of another system's semantics.

The best abstraction that has emerged here is to allow a directory to
be opened as a file containing all of the bytes of its component
files. This is at least reasonably in keeping with the Unix 'file as
a stream of bytes' precedent.

Problems with this:

-- Not supported by well established tools.
-- Not supported by well established documentation.
-- Linux only feature (for the time being at least), and only for some
filesystems.
-- Made complex by existing Unix metadata (update times, perms, etc).
-- Works well for sequential read, not so well for random access and
memory mapped files.
-- Should it recurse into directories?

These costs are not worth it IMHO. The kernel already does the things
it should do to implement its end of the problem: it allows fast,
hierarchical access to garbage collected data streams (reiserfs takes
it further and allows efficient small streams). The "data hiding"
problem is a UI problem (well, not totally, but you have to admit
that's the main reason the Mac folks went for it).

The Userspace Alternative
-------------------------

Something that people might have been overlooking: what are the
consumers of a feature like this? It's not 'ls' or 'cp'. It's
applications (like MS Word) that use complex, structured, proprietary
data formats. Such applications either:

-- are written as monolithic 'suites' that can be modified by a
central entity to adhere to these conventions, or

-- are written using a modern desktop environment (Gnome, KDE,
probably modern MS office) that provides centralized utility
libraries for doing just this sort of thing.

So the right approach IMHO is to work on lib-gnome-datablob rather
than to hack data streams support into the kernel and try to get
developers to take advantage of a Linux-only feature.

Some people here want the huge, fancy office apps to see the files as
what they really are (directories) and the lean, simple command line
tools to be duped. This is the wrong way around; if desktop end users
working with structured documents have enough expertise to use the
command line tools, they can figure out the directory thing. Rather
than trying to fool thousands of simple, straightforward tools, effort
should be focused on an agreed-upon format for the higher-level tools.

For those who feel that sticking entirely userspace conventions
between the kernel's view of files and the user's is a bad thing: I
draw your attention to the highly successful dotfile convention.

Speed
-----

In terms of speed: applications that both

(a) allow editing of complex blobs containing multiple data streams
(b) require the resulting blobs to be highly user-friendly in format

tend to fit the 'office suite' usage profile: people open blobs for
long periods of time and manipulate them at human speed rather than
with back-to-back command line/batch processing type operations.
Given that, such an app could just store the blob in tar-like format
all the time and untar it as needed. This can be plenty fast enough
for typical office automation users:

sarcastro:~$ du -c /usr/doc/HOWTO/*.gz | grep total
2953 total
sarcastro:~$ mkdir test
sarcastro:~$ cp /usr/doc/HOWTO/*.gz test/
* do a bunch of stuff to flush cache *
sarcastro:~$ time tar cf test.tar test/
real 0m0.381s
user 0m0.040s
sys 0m0.340s
sarcastro:~$ rm -r test
sarcastro:~$ time tar xf test.tar
real 0m0.423s
user 0m0.020s
sys 0m0.350s

(and this isn't a fast machine either)

This model uses filesystems for what they do very well today. Using
the right extensions for the blob file and directory (extensions on
directories are underused) would actually make it easier for the app
to do recovery of partially edited files if it has been killed, and
would help file managers do the right thing.

For power users who want more speed, a 'store in directory format'
checkbox in the apps would solve their problems. And if a shared
desktop-environment library is used to implement this convention, the
file manager app could have 'pack blob/unpack blob' menu options.

Don't forget that people using office apps are often using them on
desktop systems that are essentially single-user. Trading off a
little speed for the increased user friendliness to the average
desktop user is a pretty reasonable thing.

miket

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.rutgers.edu
Please read the FAQ at http://www.tux.org/lkml/

- Next message: Thierry Danis: "2.2.10-ac4 / knfsd 1.4.1 poor parallel reading performance"
- Previous message: Andrew Hilborne: "Re: analysis: securelevels vs securebits with capabilities"

ions"
- Previous message: Andrew Hilborne: "Re: analysis: securelevels vs securebits with capabilities"