Linux-kernel mailing list archive 1999-37,: Re: ext2 file sizes

# Re: ext2 file sizes

Richard B. Johnson (root@chaos.analogic.com)
Thu, 16 Sep 1999 09:06:04 -0400 (EDT)

- Messages sorted by: [ date ][ thread ][ subject ][ author ]
- Next message: ebi4: "Re: State of USB drivers"
- Previous message: Brian Geisel: "Symbios DAC960 and Quad Xeon"

On Thu, 16 Sep 1999, Clem Taylor wrote:

> "Richard B. Johnson" wrote:
> > On Wed, 15 Sep 1999, Blankenship, Keith wrote:
> > > I am having some difficulty with the ext2 file systems. I need to generate a
> > > file that will be > 5 Gigabytes, and there appears to be a file size cap at
> > > approximately 2 Gig. I am running what appears to be a version 2.0.36
> > > Kernel. Is there anything I can adjust, or do to increase the maximum file
> > > size? Or is there a newer kernel that may work?
> >
> > Note that on a 32-bit machine, toff_t, used as fpos_t, for file offsets,
> > i.e., lseek, is unsigned 32 bits. You will not be able to access such a
> > file on a 32-bit machine. There has been some work on changing this
> > to 'long long' (__kernel_loff_t, in ../asm/posix_types.h although you may
> > need a new 'C' runtime library to take advantage of this.
> >
> > What on earth are you doing with a single _file_ of that size? If you
> > need a gob of storage for data acquisition, you might be better off
> > with a dedicated raw device. Since you access it in blocks, rather than
> > bytes, you won't have a size limitation for a few more years.
>
> I work with compressed video and my average file size is 3-6 gigs,
> increasing
> all the time (higher bitrates). A while back I looked into using Linux for
> a
> video application and gave up and went with Solaris and Irix because of
> the lack
> of large file support. I was hoping that 2.4 would have large file support
> (and
> not just on 64 bit platforms), but that isn't looking promising based on
> comments here. Going to a 64 bit off_t (on 32 bit platforms) is a major
> change
> and will have a serious impact in user space. Just look at all the pain
> Solaris
> and Irix had in making the move.
>
> --Clem
>

Here, we acquire great gobs of data (CAT Scan Data), typically 4 gb per
run. I run it off to a dedicated SCSI. After it's converted into useful
images, the images are saved in a conventional database on a file-system.

This seems to be a real good approach because, even with smaller files
the ext2 file-system has a problem keeping up with the data-rate when
buffers get filled and must be flushed to disk. Our data rate is
2880*1440 = 4,147,200 longwords/second. We throw away the high byte
so we have a sustained data rate of 12,441,600 bytes/second. Good SCSI
controllers (BusLogic) keep up with this fine. The ext2 file-system keeps
up with this until its buffers get full. Then it's too bad.

Writes to a dedicated SCSI solved all our problems. I can sustain writes
at these speeds forever (until the drive is full).

Cheers,
Dick Johnson
**** FILE SYSTEM WAS MODIFIED ****
Penguin : Linux version 2.2.4 on an i686 machine (400.59 BogoMips).
Warning : It's hard to remain at the trailing edge of technology.

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.rutgers.edu
Please read the FAQ at http://www.tux.org/lkml/

- Next message: ebi4: "Re: State of USB drivers"
- Previous message: Brian Geisel: "Symbios DAC960 and Quad Xeon"