lseek: Value too large for defined data type

guru · External Since: Jul 20, 2007 Posts: 9

Hi All,

I have written 21GB file and updating the file by seeking 1K and
writing 100MB of data consecutively 10240 times. when after writing
2GB(2100 times) of file it was giving

lseek: Value too large for defined data type error.

What is got is the value passed to 'Offset' argument is too large to
handle.

The last return value of seek is '2099270656'

Is there any way to overcome this.

Thanks & Regads
Gururaja

Tim Southerwood · External Since: Apr 23, 2007 Posts: 113

guru wrote:

> Hi All,
>
> I have written 21GB file and updating the file by seeking 1K and
> writing 100MB of data consecutively 10240 times. when after writing
> 2GB(2100 times) of file it was giving
>
> lseek: Value too large for defined data type error.
>
> What is got is the value passed to 'Offset' argument is too large to
> handle.
>
> The last return value of seek is '2099270656'
>
> Is there any way to overcome this.
>
>
> Thanks & Regads
> Gururaja

Yes - it is a standard problem, with a standard set of solutions.

off_t is normally defined as a 32 bit signed.

Try man lseek64

In summary from that page, you can either explicitly use lseek64() and
off64_t (not portable where lseek64 is not defined but you don't need >2GB
support)

or leave everything as "off_t" and "lseek()" but do a

#define _FILE_OFFSET_BITS 64

higher up in the source (or via the Makefile or whatever)

and that will cast lseek to lseek64 and off_t to off64_t on systems that
suppport it.

HTH

Tim

David Schwartz · External Since: Jun 01, 2007 Posts: 87

On Jul 20, 7:49 pm, phil-news-nos... RemoveThis @ipal.net wrote:

> One of these days we need to fix all this ... basically migrate to purely
> 64-bit file offset platforms. It should "just work" without any changes
> needed at that point, if they do it right (which is not a given). If old
> programs compiled on legacy 32-bit file offset platforms break, so be it.

It is all fixed. It does just work. No changes are needed.

All that is needed is '-D_FILE_OFFSET_BITS=64' and complaining about
that is like arguing that PTHREADS support doesn't "just work" because
you have to specify '-pthread' on the command line when you compile/
link.

DS

phil-news-nospam · External Since: Nov 16, 2006 Posts: 329

On Fri, 20 Jul 2007 11:42:46 +0100 Tim Southerwood <ts RemoveThis @dionic.net> wrote:
| guru wrote:
|
|> Hi All,
|>
|> I have written 21GB file and updating the file by seeking 1K and
|> writing 100MB of data consecutively 10240 times. when after writing
|> 2GB(2100 times) of file it was giving
|>
|> lseek: Value too large for defined data type error.
|>
|> What is got is the value passed to 'Offset' argument is too large to
|> handle.
|>
|> The last return value of seek is '2099270656'
|>
|> Is there any way to overcome this.
|>
|>
|> Thanks & Regads
|> Gururaja
|
| Yes - it is a standard problem, with a standard set of solutions.
|
| off_t is normally defined as a 32 bit signed.
|
| Try man lseek64
|
| In summary from that page, you can either explicitly use lseek64() and
| off64_t (not portable where lseek64 is not defined but you don't need >2GB
| support)
|
| or leave everything as "off_t" and "lseek()" but do a
|
| #define _FILE_OFFSET_BITS 64
|
| higher up in the source (or via the Makefile or whatever)
|
| and that will cast lseek to lseek64 and off_t to off64_t on systems that
| suppport it.

One of these days we need to fix all this ... basically migrate to purely
64-bit file offset platforms. It should "just work" without any changes
needed at that point, if they do it right (which is not a given). If old
programs compiled on legacy 32-bit file offset platforms break, so be it.

--
|---------------------------------------/----------------------------------|
| Phil Howard KA9WGN (ka9wgn.ham.org) / Do not send to the address below |
| first name lower case at ipal.net / spamtrap-2007-07-20-2147 RemoveThis @ipal.net |
|------------------------------------/-------------------------------------|

"spacecriter · External Since: Jul 21, 2007 Posts: 1

guru wrote:
> Hi All,
>
> I have written 21GB file and updating the file by seeking 1K and
> writing 100MB of data consecutively 10240 times. when after writing
> 2GB(2100 times) of file it was giving
>
> lseek: Value too large for defined data type error.
>
> What is got is the value passed to 'Offset' argument is too large to
> handle.
>
> The last return value of seek is '2099270656'
>
> Is there any way to overcome this.
>

As others have said, add -D_FILE_OFFSET_BITS=64 to the compiler command line

I have found that it is a good idea to also add -D_LARGEFILE64_SOURCE to the
command line as well.

--
Bill C.

guru · External Since: Jul 20, 2007 Posts: 9

I am unable to seek. again same error it is giving.

First i will write 21GB then run the below program

The program

#define _GNU_SOURCE
#include<stdio.h>
#include<stdlib.h>
#include<string.h>
#include <fcntl.h>
#include<sys/types.h>
#include<sys/stat.h>
main()
{

int file = 0;
char *buff = NULL;
int bytewr,count=0,loopcount,choice;
long long seekOffset = 0;

buff = (char *)malloc(1024 * 1024);
if(buff == NULL)
{
printf("Memory Allocation failed\n");
return -1;
}
else
{
printf("Memory allocation success\n");
}

file = open("/HDR1/test1.txt", O_LARGEFILE|O_RDWR);
if(file == -1)
{
printf("Error opening file\n");
return -1;
}
else
{
printf("File opening success\n");
}
/*1MB seek*/
seekOffset = lseek(file,(1024*1024),SEEK_CUR);
if(seekOffset == -1)
{
printf("Seek Fail\n");
return -1;
}
else
{
printf("Seek Success=%d\n",seekOffset);
}

memset(buff,0x44,(1024*1024));
loopcount = 10240; /*do it for this many times*/
{
count = 0;
do {
bytewr = write(file, buff,(1024*1024));
if(bytewr == (1024*1024))
{
printf("Count: %d Write Success\n",count); }
else
{
printf("Write fail\n");
return -1;
}
if((count%100) == 0)
{
/*Seek afterevery
100MB*/
seekOffset = lseek(file,(1024),SEEK_CUR);
if(seekOffset == -1)
{
printf("Seek Fail\n");
return -1;
}
else
{
printf("Seek Success=%Ld\n",seekOffset);
}
}

}while(count++ < loopcount);
close(file);
free(buff);
}

while compiling i have to give

gcc -D_FILE_OFFSET_BITS=64 -D_LARGEFILE64_SOURCE append.c -o append

after 2100 loop, it is failing.

is there any problem with application.

Regards
Gururaja

spacecriter (Bill C) wrote:
> guru wrote:
> > Hi All,
> >
> > I have written 21GB file and updating the file by seeking 1K and
> > writing 100MB of data consecutively 10240 times. when after writing
> > 2GB(2100 times) of file it was giving
> >
> > lseek: Value too large for defined data type error.
> >
> > What is got is the value passed to 'Offset' argument is too large to
> > handle.
> >
> > The last return value of seek is '2099270656'
> >
> > Is there any way to overcome this.
> >
>
> As others have said, add -D_FILE_OFFSET_BITS=64 to the compiler command line
>
> I have found that it is a good idea to also add -D_LARGEFILE64_SOURCE to the
> command line as well.
>
> --
> Bill C.

guru · External Since: Jul 20, 2007 Posts: 9

I am unable to seek. again same error it is giving.

First i will write 21GB then run the below program

The program

#define _GNU_SOURCE
#include<stdio.h>
#include<stdlib.h>
#include<string.h>
#include <fcntl.h>
#include<sys/types.h>
#include<sys/stat.h>
main()
{

int file = 0;
char *buff = NULL;
int bytewr,count=0,loopcount,choice;
long long seekOffset = 0;

buff = (char *)malloc(1024 * 1024);
if(buff == NULL)
{
printf("Memory Allocation failed\n");
return -1;
}
else
{
printf("Memory allocation success\n");
}

file = open("/HDR1/test1.txt", O_LARGEFILE|O_RDWR);
if(file == -1)
{
printf("Error opening file\n");
return -1;
}
else
{
printf("File opening success\n");
}
/*1MB seek*/
seekOffset = lseek(file,(1024*1024),SEEK_CUR);
if(seekOffset == -1)
{
printf("Seek Fail\n");
return -1;
}
else
{
printf("Seek Success=%d\n",seekOffset);
}

memset(buff,0x44,(1024*1024));
loopcount = 10240; /*do it for this many times*/
{
count = 0;
do {
bytewr = write(file, buff,(1024*1024));
if(bytewr == (1024*1024))
{
printf("Count: %d Write Success\n",count); }
else
{
printf("Write fail\n");
return -1;
}
if((count%100) == 0)
{
/*Seek afterevery
100MB*/
seekOffset = lseek(file,(1024),SEEK_CUR);
if(seekOffset == -1)
{
printf("Seek Fail\n");
return -1;
}
else
{
printf("Seek Success=%Ld\n",seekOffset);
}
}

}while(count++ < loopcount);
close(file);
free(buff);
}

while compiling i have to give

gcc -D_FILE_OFFSET_BITS=64 -D_LARGEFILE64_SOURCE append.c -o append

after 2100 loop, it is failing.

is there any problem with application.

Regards
Gururaja

spacecriter (Bill C) wrote:
> guru wrote:
> > Hi All,
> >
> > I have written 21GB file and updating the file by seeking 1K and
> > writing 100MB of data consecutively 10240 times. when after writing
> > 2GB(2100 times) of file it was giving
> >
> > lseek: Value too large for defined data type error.
> >
> > What is got is the value passed to 'Offset' argument is too large to
> > handle.
> >
> > The last return value of seek is '2099270656'
> >
> > Is there any way to overcome this.
> >
>
> As others have said, add -D_FILE_OFFSET_BITS=64 to the compiler command line
>
> I have found that it is a good idea to also add -D_LARGEFILE64_SOURCE to the
> command line as well.
>
> --
> Bill C.

Tim Southerwood · External Since: Apr 23, 2007 Posts: 113

guru wrote:

> I am unable to seek. again same error it is giving.
>
> First i will write 21GB then run the below program
>
> The program
>
> #define _GNU_SOURCE
> #include<stdio.h>
> #include<stdlib.h>
> #include<string.h>
> #include <fcntl.h>
> #include<sys/types.h>
> #include<sys/stat.h>
> main()
> {
>
> int file = 0;
> char *buff = NULL;
> int bytewr,count=0,loopcount,choice;
> long long seekOffset = 0;
>
> buff = (char *)malloc(1024 * 1024);
> if(buff == NULL)
> {
> printf("Memory Allocation failed\n");
> return -1;
> }
> else
> {
> printf("Memory allocation success\n");
> }
>
> file = open("/HDR1/test1.txt", O_LARGEFILE|O_RDWR);
> if(file == -1)
> {
> printf("Error opening file\n");
> return -1;
> }
> else
> {
> printf("File opening success\n");
> }
> /*1MB seek*/
> seekOffset = lseek(file,(1024*1024),SEEK_CUR);
> if(seekOffset == -1)
> {
> printf("Seek Fail\n");
> return -1;
> }
> else
> {
> printf("Seek Success=%d\n",seekOffset);
> }
>
> memset(buff,0x44,(1024*1024));
> loopcount = 10240; /*do it for this many times*/
> {
> count = 0;
> do {
> bytewr = write(file, buff,(1024*1024));
> if(bytewr == (1024*1024))
> {
> printf("Count: %d Write Success\n",count); }
> else
> {
> printf("Write fail\n");
> return -1;
> }
> if((count%100) == 0)
> {
> /*Seek afterevery
> 100MB*/
> seekOffset = lseek(file,(1024),SEEK_CUR);
> if(seekOffset == -1)
> {
> printf("Seek Fail\n");
> return -1;
> }
> else
> {
> printf("Seek Success=%Ld\n",seekOffset);
> }
> }
>
> }while(count++ < loopcount);
> close(file);
> free(buff);
> }
>
> while compiling i have to give
>
> gcc -D_FILE_OFFSET_BITS=64 -D_LARGEFILE64_SOURCE append.c -o append
>
> after 2100 loop, it is failing.
>
> is there any problem with application.
>
> Regards
> Gururaja

Hi,

It works if you apply this patch:

% diff -u append.c.orig append.c
--- append.c.orig 2007-07-24 09:48:17.857169930 +0100
+++ append.c 2007-07-24 10:25:21.523034930 +0100
@@ -11,7 +11,7 @@
int file = 0;
char *buff = NULL;
int bytewr,count=0,loopcount,choice;
- long long seekOffset = 0;
+ off_t seekOffset = 0;

buff = (char *)malloc(1024 * 1024);
if(buff == NULL)
@@ -24,7 +24,7 @@
printf("Memory allocation success\n");
}

- file = open("/tmp/test1.txt", O_LARGEFILE|O_RDWR);
+ file = open("/tmp/test1.txt", O_RDWR);
if(file == -1)
{
printf("Error opening file\n");
@@ -35,7 +35,7 @@
printf("File opening success\n");
}
/*1MB seek*/
- seekOffset = lseek(file,(1024*1024),SEEK_CUR);
+ seekOffset = lseek(file,(off_t)(1024*1024),SEEK_CUR);
if(seekOffset == -1)
{
printf("Seek Fail\n");
@@ -63,7 +63,7 @@
if((count%100) == 0)
{
/*Seek afterevery 100MB*/
- seekOffset = lseek(file,(1024),SEEK_CUR);
+ seekOffset = lseek(file,(off_t)(1024),SEEK_CUR);
if(seekOffset == -1)
{
printf("Seek Fail\n");

##########################################################

The first line is not correct, though it doesn't break due to "long long"
being good enough in this case, but you really need to follow the header
definitions faithfully otherwise random bad things will happen in the
future. lseek returns an off_t, so that's what you should use.

The second line replacement is just that O_LARGEFILE is not needed, it is
brought in by the -D options.

The latter two lines indicate that you have an implicit problem with the
compiler casting your int's to the correct this - off_t (which will be
off64_t by them due to the -D options.

Explicitly casting fixes it quite nicely. Personally I might explicitly cast
the "== -1" to "== (off_t) -1" just for clarity, but I suspect the compiler
is able to infer that reliably from seekOffset and do the right thing.

HTH

Tim

David Schwartz · External Since: Jun 01, 2007 Posts: 87

On Jul 24, 11:29 pm, guru <guru.nav....RemoveThis@gmail.com> wrote:

> But if I dont include this header file, it should give warning?.

If you turn the appropriate warning on.

> and
> How, including of this header file affecting the return value of
> lseek.

It tells the compiler what type 'lseek' returns. With the wrong type,
it could be looking in the wrong place completely (the register
instead of the stack, the wrong register, and so on) so it's not
surprising you get garbage.

DS

guru · External Since: Jul 20, 2007 Posts: 9

Hi,

I got it. I am able to run this program.
But i am seeing difference in output of the lseek. After seeking 1024
bytes, i am printing the value returned by the lseek. it is always
printing 1024 bytes only.
But according to lseek, it should return number of bytes starting from
the beginning of the file.

What is making it to return always 1024 but not the actual value.

Thanks
Gururaj

Tim Southerwood wrote:
> guru wrote:
>
> > I am unable to seek. again same error it is giving.
> >
> > First i will write 21GB then run the below program
> >
> > The program
> >
> > #define _GNU_SOURCE
> > #include<stdio.h>
> > #include<stdlib.h>
> > #include<string.h>
> > #include <fcntl.h>
> > #include<sys/types.h>
> > #include<sys/stat.h>
> > main()
> > {
> >
> > int file = 0;
> > char *buff = NULL;
> > int bytewr,count=0,loopcount,choice;
> > long long seekOffset = 0;
> >
> > buff = (char *)malloc(1024 * 1024);
> > if(buff == NULL)
> > {
> > printf("Memory Allocation failed\n");
> > return -1;
> > }
> > else
> > {
> > printf("Memory allocation success\n");
> > }
> >
> > file = open("/HDR1/test1.txt", O_LARGEFILE|O_RDWR);
> > if(file == -1)
> > {
> > printf("Error opening file\n");
> > return -1;
> > }
> > else
> > {
> > printf("File opening success\n");
> > }
> > /*1MB seek*/
> > seekOffset = lseek(file,(1024*1024),SEEK_CUR);
> > if(seekOffset == -1)
> > {
> > printf("Seek Fail\n");
> > return -1;
> > }
> > else
> > {
> > printf("Seek Success=%d\n",seekOffset);
> > }
> >
> > memset(buff,0x44,(1024*1024));
> > loopcount = 10240; /*do it for this many times*/
> > {
> > count = 0;
> > do {
> > bytewr = write(file, buff,(1024*1024));
> > if(bytewr == (1024*1024))
> > {
> > printf("Count: %d Write Success\n",count); }
> > else
> > {
> > printf("Write fail\n");
> > return -1;
> > }
> > if((count%100) == 0)
> > {
> > /*Seek afterevery
> > 100MB*/
> > seekOffset = lseek(file,(1024),SEEK_CUR);
> > if(seekOffset == -1)
> > {
> > printf("Seek Fail\n");
> > return -1;
> > }
> > else
> > {
> > printf("Seek Success=%Ld\n",seekOffset);
> > }
> > }
> >
> > }while(count++ < loopcount);
> > close(file);
> > free(buff);
> > }
> >
> > while compiling i have to give
> >
> > gcc -D_FILE_OFFSET_BITS=64 -D_LARGEFILE64_SOURCE append.c -o append
> >
> > after 2100 loop, it is failing.
> >
> > is there any problem with application.
> >
> > Regards
> > Gururaja
>
> Hi,
>
> It works if you apply this patch:
>
> % diff -u append.c.orig append.c
> --- append.c.orig 2007-07-24 09:48:17.857169930 +0100
> +++ append.c 2007-07-24 10:25:21.523034930 +0100
> @@ -11,7 +11,7 @@
> int file = 0;
> char *buff = NULL;
> int bytewr,count=0,loopcount,choice;
> - long long seekOffset = 0;
> + off_t seekOffset = 0;
>
> buff = (char *)malloc(1024 * 1024);
> if(buff == NULL)
> @@ -24,7 +24,7 @@
> printf("Memory allocation success\n");
> }
>
> - file = open("/tmp/test1.txt", O_LARGEFILE|O_RDWR);
> + file = open("/tmp/test1.txt", O_RDWR);
> if(file == -1)
> {
> printf("Error opening file\n");
> @@ -35,7 +35,7 @@
> printf("File opening success\n");
> }
> /*1MB seek*/
> - seekOffset = lseek(file,(1024*1024),SEEK_CUR);
> + seekOffset = lseek(file,(off_t)(1024*1024),SEEK_CUR);
> if(seekOffset == -1)
> {
> printf("Seek Fail\n");
> @@ -63,7 +63,7 @@
> if((count%100) == 0)
> {
> /*Seek afterevery 100MB*/
> - seekOffset = lseek(file,(1024),SEEK_CUR);
> + seekOffset = lseek(file,(off_t)(1024),SEEK_CUR);
> if(seekOffset == -1)
> {
> printf("Seek Fail\n");
>
> ##########################################################
>
> The first line is not correct, though it doesn't break due to "long long"
> being good enough in this case, but you really need to follow the header
> definitions faithfully otherwise random bad things will happen in the
> future. lseek returns an off_t, so that's what you should use.
>
> The second line replacement is just that O_LARGEFILE is not needed, it is
> brought in by the -D options.
>
> The latter two lines indicate that you have an implicit problem with the
> compiler casting your int's to the correct this - off_t (which will be
> off64_t by them due to the -D options.
>
> Explicitly casting fixes it quite nicely. Personally I might explicitly cast
> the "== -1" to "== (off_t) -1" just for clarity, but I suspect the compiler
> is able to infer that reliably from seekOffset and do the right thing.
>
> HTH
>
> Tim

guru · External Since: Jul 20, 2007 Posts: 9

Hi,

I got the problem. I was not included the header file
#include<unistd.h>. after including this i will returning correct
value. so it is running perfectly.

But one more clarification.

But if I dont include this header file, it should give warning?. and
How, including of this header file affecting the return value of
lseek.

Thanks for the help. i really learnt a lot from this.
-Gururaja

guru wrote:
> Hi,
>
> I got it. I am able to run this program.
> But i am seeing difference in output of the lseek. After seeking 1024
> bytes, i am printing the value returned by the lseek. it is always
> printing 1024 bytes only.
> But according to lseek, it should return number of bytes starting from
> the beginning of the file.
>
> What is making it to return always 1024 but not the actual value.
>
> Thanks
> Gururaj
>
>
>
> Tim Southerwood wrote:
> > guru wrote:
> >
> > > I am unable to seek. again same error it is giving.
> > >
> > > First i will write 21GB then run the below program
> > >
> > > The program
> > >
> > > #define _GNU_SOURCE
> > > #include<stdio.h>
> > > #include<stdlib.h>
> > > #include<string.h>
> > > #include <fcntl.h>
> > > #include<sys/types.h>
> > > #include<sys/stat.h>
> > > main()
> > > {
> > >
> > > int file = 0;
> > > char *buff = NULL;
> > > int bytewr,count=0,loopcount,choice;
> > > long long seekOffset = 0;
> > >
> > > buff = (char *)malloc(1024 * 1024);
> > > if(buff == NULL)
> > > {
> > > printf("Memory Allocation failed\n");
> > > return -1;
> > > }
> > > else
> > > {
> > > printf("Memory allocation success\n");
> > > }
> > >
> > > file = open("/HDR1/test1.txt", O_LARGEFILE|O_RDWR);
> > > if(file == -1)
> > > {
> > > printf("Error opening file\n");
> > > return -1;
> > > }
> > > else
> > > {
> > > printf("File opening success\n");
> > > }
> > > /*1MB seek*/
> > > seekOffset = lseek(file,(1024*1024),SEEK_CUR);
> > > if(seekOffset == -1)
> > > {
> > > printf("Seek Fail\n");
> > > return -1;
> > > }
> > > else
> > > {
> > > printf("Seek Success=%d\n",seekOffset);
> > > }
> > >
> > > memset(buff,0x44,(1024*1024));
> > > loopcount = 10240; /*do it for this many times*/
> > > {
> > > count = 0;
> > > do {
> > > bytewr = write(file, buff,(1024*1024));
> > > if(bytewr == (1024*1024))
> > > {
> > > printf("Count: %d Write Success\n",count); }
> > > else
> > > {
> > > printf("Write fail\n");
> > > return -1;
> > > }
> > > if((count%100) == 0)
> > > {
> > > /*Seek afterevery
> > > 100MB*/
> > > seekOffset = lseek(file,(1024),SEEK_CUR);
> > > if(seekOffset == -1)
> > > {
> > > printf("Seek Fail\n");
> > > return -1;
> > > }
> > > else
> > > {
> > > printf("Seek Success=%Ld\n",seekOffset);
> > > }
> > > }
> > >
> > > }while(count++ < loopcount);
> > > close(file);
> > > free(buff);
> > > }
> > >
> > > while compiling i have to give
> > >
> > > gcc -D_FILE_OFFSET_BITS=64 -D_LARGEFILE64_SOURCE append.c -o append
> > >
> > > after 2100 loop, it is failing.
> > >
> > > is there any problem with application.
> > >
> > > Regards
> > > Gururaja
> >
> > Hi,
> >
> > It works if you apply this patch:
> >
> > % diff -u append.c.orig append.c
> > --- append.c.orig 2007-07-24 09:48:17.857169930 +0100
> > +++ append.c 2007-07-24 10:25:21.523034930 +0100
> > @@ -11,7 +11,7 @@
> > int file = 0;
> > char *buff = NULL;
> > int bytewr,count=0,loopcount,choice;
> > - long long seekOffset = 0;
> > + off_t seekOffset = 0;
> >
> > buff = (char *)malloc(1024 * 1024);
> > if(buff == NULL)
> > @@ -24,7 +24,7 @@
> > printf("Memory allocation success\n");
> > }
> >
> > - file = open("/tmp/test1.txt", O_LARGEFILE|O_RDWR);
> > + file = open("/tmp/test1.txt", O_RDWR);
> > if(file == -1)
> > {
> > printf("Error opening file\n");
> > @@ -35,7 +35,7 @@
> > printf("File opening success\n");
> > }
> > /*1MB seek*/
> > - seekOffset = lseek(file,(1024*1024),SEEK_CUR);
> > + seekOffset = lseek(file,(off_t)(1024*1024),SEEK_CUR);
> > if(seekOffset == -1)
> > {
> > printf("Seek Fail\n");
> > @@ -63,7 +63,7 @@
> > if((count%100) == 0)
> > {
> > /*Seek afterevery 100MB*/
> > - seekOffset = lseek(file,(1024),SEEK_CUR);
> > + seekOffset = lseek(file,(off_t)(1024),SEEK_CUR);
> > if(seekOffset == -1)
> > {
> > printf("Seek Fail\n");
> >
> > ##########################################################
> >
> > The first line is not correct, though it doesn't break due to "long long"
> > being good enough in this case, but you really need to follow the header
> > definitions faithfully otherwise random bad things will happen in the
> > future. lseek returns an off_t, so that's what you should use.
> >
> > The second line replacement is just that O_LARGEFILE is not needed, it is
> > brought in by the -D options.
> >
> > The latter two lines indicate that you have an implicit problem with the
> > compiler casting your int's to the correct this - off_t (which will be
> > off64_t by them due to the -D options.
> >
> > Explicitly casting fixes it quite nicely. Personally I might explicitly cast
> > the "== -1" to "== (off_t) -1" just for clarity, but I suspect the compiler
> > is able to infer that reliably from seekOffset and do the right thing.
> >
> > HTH
> >
> > Tim

phil-news-nospam · External Since: Nov 16, 2006 Posts: 329

On Sat, 21 Jul 2007 01:06:34 -0700 David Schwartz <davids.DeleteThis@webmaster.com> wrote:
| On Jul 20, 7:49 pm, phil-news-nos....DeleteThis@ipal.net wrote:
|
|> One of these days we need to fix all this ... basically migrate to purely
|> 64-bit file offset platforms. It should "just work" without any changes
|> needed at that point, if they do it right (which is not a given). If old
|> programs compiled on legacy 32-bit file offset platforms break, so be it.
|
| It is all fixed. It does just work. No changes are needed.
|
| All that is needed is '-D_FILE_OFFSET_BITS=64' and complaining about
| that is like arguing that PTHREADS support doesn't "just work" because
| you have to specify '-pthread' on the command line when you compile/
| link.

"No changes are needed." and "All that is needed is ..." are contrary.
Adding the '-D_FILE_OFFSET_BITS=64' is a change.

If the nature of pthreads requires the compiler to generate different code,
then sure, pthreads usage needs to be specified. If there was a reason to
never use anything but pthreads, then I'd say the compiler should default
to doing what pthreads needs.

These two things are not analogous. In the case of file offset, there is
virtually no need for any application to not be using 64 bit offsets. It
should be the default. Cases where a program needs to _not_ use them would
be very small embedded code and such. But we still have plenty of need for
non-pthreads code that making pthreads the default does not make sense.

--
|---------------------------------------/----------------------------------|
| Phil Howard KA9WGN (ka9wgn.ham.org) / Do not send to the address below |
| first name lower case at ipal.net / spamtrap-2007-07-26-0748.DeleteThis@ipal.net |
|------------------------------------/-------------------------------------|

phil-news-nospam · External Since: Nov 16, 2006 Posts: 329

On Sat, 21 Jul 2007 08:11:12 -0500 "spacecriter \(Bill C\)" <spacecriter DeleteThis @sysmatrix.net.yercrank> wrote:
|
| guru wrote:
|> Hi All,
|>
|> I have written 21GB file and updating the file by seeking 1K and
|> writing 100MB of data consecutively 10240 times. when after writing
|> 2GB(2100 times) of file it was giving
|>
|> lseek: Value too large for defined data type error.
|>
|> What is got is the value passed to 'Offset' argument is too large to
|> handle.
|>
|> The last return value of seek is '2099270656'
|>
|> Is there any way to overcome this.
|>
|
| As others have said, add -D_FILE_OFFSET_BITS=64 to the compiler command line
|
| I have found that it is a good idea to also add -D_LARGEFILE64_SOURCE to the
| command line as well.

The big list of interesting symbols from /usr/include/features.h:

/* These are defined by the user (or the compiler)
to specify the desired environment:

__STRICT_ANSI__ ISO Standard C.
_ISOC99_SOURCE Extensions to ISO C89 from ISO C99.
_POSIX_SOURCE IEEE Std 1003.1.
_POSIX_C_SOURCE If ==1, like _POSIX_SOURCE; if >=2 add IEEE Std 1003.2;
if >=199309L, add IEEE Std 1003.1b-1993;
if >=199506L, add IEEE Std 1003.1c-1995;
if >=200112L, all of IEEE 1003.1-2004
_XOPEN_SOURCE Includes POSIX and XPG things. Set to 500 if
Single Unix conformance is wanted, to 600 for the
upcoming sixth revision.
_XOPEN_SOURCE_EXTENDED XPG things and X/Open Unix extensions.
_LARGEFILE_SOURCE Some more functions for correct standard I/O.
_LARGEFILE64_SOURCE Additional functionality from LFS for large files.
_FILE_OFFSET_BITS=N Select default filesystem interface.
_BSD_SOURCE ISO C, POSIX, and 4.3BSD things.
_SVID_SOURCE ISO C, POSIX, and SVID things.
_ATFILE_SOURCE Additional *at interfaces.
_GNU_SOURCE All of the above, plus GNU extensions.
_REENTRANT Select additionally reentrant object.
_THREAD_SAFE Same as _REENTRANT, often used by other systems.
_FORTIFY_SOURCE If set to numeric value > 0 additional security
measures are defined, according to level.

The `-ansi' switch to the GNU C compiler defines __STRICT_ANSI__.
If none of these are defined, the default is to have _SVID_SOURCE,
_BSD_SOURCE, and _POSIX_SOURCE set to one and _POSIX_C_SOURCE set to
199506L. If more than one of these are defined, they accumulate.
For example __STRICT_ANSI__, _POSIX_SOURCE and _POSIX_C_SOURCE
together give you ISO C, 1003.1, and 1003.2, but nothing else.

These are defined by this file and are used by the
header files to decide what to declare or define:

__USE_ISOC99 Define ISO C99 things.
__USE_POSIX Define IEEE Std 1003.1 things.
__USE_POSIX2 Define IEEE Std 1003.2 things.
__USE_POSIX199309 Define IEEE Std 1003.1, and .1b things.
__USE_POSIX199506 Define IEEE Std 1003.1, .1b, .1c and .1i things.
__USE_XOPEN Define XPG things.
__USE_XOPEN_EXTENDED Define X/Open Unix things.
__USE_UNIX98 Define Single Unix V2 things.
__USE_XOPEN2K Define XPG6 things.
__USE_LARGEFILE Define correct standard I/O things.
__USE_LARGEFILE64 Define LFS things with separate names.
__USE_FILE_OFFSET64 Define 64bit interface as default.
__USE_BSD Define 4.3BSD things.
__USE_SVID Define SVID things.
__USE_MISC Define things common to BSD and System V Unix.
__USE_ATFILE Define *at interfaces and AT_* constants for them.
__USE_GNU Define GNU extensions.
__USE_REENTRANT Define reentrant/thread-safe *_r functions.
__USE_FORTIFY_LEVEL Additional security measures used, according to level.
__FAVOR_BSD Favor 4.3BSD things in cases of conflict.

The macros `__GNU_LIBRARY__', `__GLIBC__', and `__GLIBC_MINOR__' are
defined by this file unconditionally. `__GNU_LIBRARY__' is provided
only for compatibility. All new code should use the other symbols
to test for features.

All macros listed above as possibly being defined by this file are
explicitly undefined if they are not explicitly defined.
Feature-test macros that are not defined by the user or compiler
but are implied by the other feature-test macros defined (or by the
lack of any definitions) are defined by the file. */

--
|---------------------------------------/----------------------------------|
| Phil Howard KA9WGN (ka9wgn.ham.org) / Do not send to the address below |
| first name lower case at ipal.net / spamtrap-2007-07-26-0756 DeleteThis @ipal.net |
|------------------------------------/-------------------------------------|

phil-news-nospam · External Since: Nov 16, 2006 Posts: 329

On Tue, 24 Jul 2007 10:31:05 +0100 Tim Southerwood <ts DeleteThis @dionic.net> wrote:

| The first line is not correct, though it doesn't break due to "long long"
| being good enough in this case, but you really need to follow the header
| definitions faithfully otherwise random bad things will happen in the
| future. lseek returns an off_t, so that's what you should use.

Like when we add "long long long long" to be 128 bits and change file
offset bits to be 128 Smile

--
|---------------------------------------/----------------------------------|
| Phil Howard KA9WGN (ka9wgn.ham.org) / Do not send to the address below |
| first name lower case at ipal.net / spamtrap-2007-07-26-0801 DeleteThis @ipal.net |
|------------------------------------/-------------------------------------|

David Schwartz · External Since: Jun 01, 2007 Posts: 87

On Jul 26, 5:54 am, phil-news-nos....TakeThisOut@ipal.net wrote:

> These two things are not analogous. In the case of file offset, there is
> virtually no need for any application to not be using 64 bit offsets. It
> should be the default. Cases where a program needs to _not_ use them would
> be very small embedded code and such. But we still have plenty of need for
> non-pthreads code that making pthreads the default does not make sense.

Actually, I disagree on both counts. With respect to the first claim,
there *is* code that uses 32 bit offsets internally. It will not work
properly with 64 bit offsets to the OS. There is a need for these
applications to not be using 64 bit offsets. With respect to the
second claim, even non-pthreads code has to use libraries, and those
libraries are almost always also used by pthreads code.

DS