Originally posted by: SystemAdmin

What environment/linker/compiler settings do I need,
to compile and link with libstdc++,
such that new ifstream(filename) will read a file larger than 2G?

So far, I've been able to compile and link code with libstdc++-3.3.2
on aix-5.3 which successfully reads and writes files less than 2G.

These same executables exhibit a handful of failure modes with files >2G .
o (new ifstream(file-larger-than-2G)).read() throws an io exception
unable to read.
o (new ofstream(file-to-be-larger-than-2G).write() throws no exception
but the file is truncated at 2G.
Some of the source code has been built on linux-2.6, and when tested, had
no problems handling files >2G.

Does anyone have an example build script and source code which uses
libstdc++ with files larger than 2G?

Can anyone point me at online documentation?

What am I missing?

TIA,
#AIX-Forum

Originally posted by: SystemAdmin

Are you able to create and process file greater than 2GB by any method other than the binaries you've compiled yourself? In AIX filesystems have to be created with a special option to support so-called large files larger than 2GB. If you haven't done this it would cause the error you're seeing. You can check this with "smitty chjfs".

HTH

Jim Lane
#AIX-Forum

Originally posted by: SystemAdmin

The system header files define system calls and data structures according to the directive _LARGE_FILES.
Defining this at compile time should allow your C programs to handle large files; with C++ I have no experience.
a search for _LARGE_FILES on the IBM site should give you more info.

Regards
#AIX-Forum

Originally posted by: SystemAdmin

Thank you for your time and expertise.

> The system header files define system calls and data
> structures according to the directive _LARGE_FILES.
> Defining this at compile time should allow your C
> programs to handle large files; with C++ I have no
> experience.
> a search for _LARGE_FILES on the IBM site should give
> you more info.

So far my searches have turned up nothing useful for C++.
Most report workarounds for C-only code.

Do you have a pointer to where macros, like _LARGE_FILES are documented?

So far, I've used _LARGE_FILES with mixed success in C,
but its use with C++ has so far been unsuccessful.
I suspect that there are related macros required.

Does anyone know what macros are required for compile with
g++-3.3.2 on aix-5.3 and support for files >2G?

TIA,

#AIX-Forum

Originally posted by: SystemAdmin

Hello,

best starting point should be the pseries and aix infocenter:
http://publib.boulder.ibm.com/infocenter/pseries/v5r3/index.jsp

use the site search with "_LARGE_FILES"
Regards
#AIX-Forum

Originally posted by: SystemAdmin

Joachim,

> best starting point should be the pseries and aix infocenter:
> http://publib.boulder.ibm.com/infocenter/pseries/v5r3/index.jsp
> use the site search with "_LARGE_FILES"

Thank you.

Starting from your pointers yesterday (via ... NetCFD bugfix, ... maix64 ...,
X64, ... bbigtoc, ...),
I have found settings that allowed me to successfully compile two test
programs that have successfully built, written, and read large files.

Though not fully tested yet, my current configuration uses:
g++ -maix64 -static ...
ar -X64 ...
g++ -maix64 -static -Xlinker -bbigtoc

Though I have found no detailed documentation on maix64,
it reportedly works around a bug in the aix-5.2 and aix-5.3 systems
related to large files. Note that -maix64 is being used instead of
-mpowerpc64 and/or -mcpu=603 which so far fail.

If successful, I'll report the details I learn.

Hopeful,
drb
#AIX-Forum

Originally posted by: SystemAdmin

Jim,

Thank you for your response.

> Are you able to create and process file greater than
> 2GB by any method other than the binaries you've
> compiled yourself?

Limited. Programs I've written have been able to generate files >2G,
when I explicitly used open64, fopen64, and other hand coded
modifications in strict C.

Am I correct, that if Large File Enabled was unset, not even open64
and fopen64 could create 2G files?
Am I correct that the above indicates our systems do have
Large File Enabled set?

I've not succeeded in using c++ ifstream or ofstream successfully.
We need to build libraries from ansi c++ and g++ for interoperability
with software used on other platforms.

I've encountered LARGE_FILE_API and other macros in the aix /usr/include
header files, but been unable to learn their intended use.

Does anyone know where these macros are documented?
Does anyone know how they can be set to use c++ ifstream/ofstream
successfully with files >2G?
Does anyone know if correct setting of these macros has effect?

We are limited to g++-3.3.2 by the fact that we have seen no evidence
of any successful installation of gcc-4 on aix-5.3 . (Though all the
other packages downloaded from http://www-03.ibm.com/servers/aix/products/aixos/linux/download.html
have proven useful, the g++-4 rpm package has never successfully
compiled much more than a hello world program for us, nor for several
others who've reported problems on this forum
"error: gcc-4.0.0-1.aix5.3.ppc.rpm cannot be installed",
"g++-4.0.0 for aix5.3L and stdc++")

Is there anyone successfully using g++-4 (not just gcc-4) on aix-5.3?
Where can we get a working g++-4 rpm for aix-5.3?
Where can we get a working g++-4 src rpm for aix-5.3?
What aix-5.3 spec file/environment was used to compile a working g++-4?

> In AIX filesystems have to be
> created with a special option to support so-called
> large files larger than 2GB. If you haven't done this
> it would cause the error you're seeing. You can check
> this with "smitty chjfs".

On one aix-5.3 system, "smitty chjfs" leaves the "Large Files Enabled"
field blank. I've not yet learned whether this means large files are
disabled or smitty is unable to report. On a second aix-5.3 system,
the field is absent from smitty's report and the log reports
**** ERROR ****
1800-083 Data error: could not find matching ring
field value for corresponding Command_to_Discover
value. sm_cmd_opt15.disc_field_name:
"VIX".
The current actual value will be used as the
default value for the following dialog. Use
local problem reporting procedures.

TIA,
#AIX-Forum

Originally posted by: SystemAdmin

You should expect to see the value "true" opposite the field "Large File Enabled". If you don't then files greater than 2GB cannot be created in that filesystem by any means. I believe your problem is with the filesystem not with your code.

HTH

Jim Lane
#AIX-Forum

Originally posted by: SystemAdmin

you can check with lsfs -q if your jfs filesystem is "large-file enabled", see below "bf" means "big file", I guess. jfs2 (Enhanced JFS) does not have the 2GB limitation and therfore no "bf" or "large-file enable" option.

 <ol> <li>lsfs -q /jfs/</li> </ol> Name            Nodename   Mount Pt VFS   Size    Options    Auto Accounting


 /dev/lv00       --         /jfs                   jfs   4194304 rw         yes  no


 (lv size: 4194304, fs size: 4194304, frag size: 4096, nbpi: 4096, compress: no, bf: true, ag: 64)

Regards
Joachim Gann
#AIX-Forum

Originally posted by: SystemAdmin

Joachim,

> you can check with lsfs -q if your jfs filesystem is
> "large-file enabled", see below "bf" means "big
> file",
> I guess. jfs2 (Enhanced JFS) does not have the
> 2GB limitation and therfore no "bf" or "large-file
> enable" option.

This was very useful.
Our build environment reports that it is not large-file enabled.
One of our run environments reports that it is large-file enabled,
and another reports that it is jfs2.

Now my job is how to upgrade our build environment.
#AIX-Forum

Originally posted by: SystemAdmin

You should also verify that there are no security limits on the machine that are limiting the size of the files you can create. /etc/security/limits is used to specify the maximum allowed file size (see below).

You may be able to change these limits through smitty, although I have never done so. You will need to reboot the machine for the change to take effect.

1. cat /etc/security/limits

*

• Sizes are in multiples of 512 byte blocks, CPU time is in seconds

*

• fsize - soft file size in blocks
• core - soft core file size in blocks
• cpu - soft per process CPU time limit in seconds
• data - soft data segment size in blocks
• stack - soft stack segment size in blocks
• rss - soft real memory usage in blocks
• nofiles - soft file descriptor limit
• fsize_hard - hard file size in blocks
• core_hard - hard core file size in blocks
• cpu_hard - hard per process CPU time limit in seconds
• data_hard - hard data segment size in blocks
• stack_hard - hard stack segment size in blocks
• rss_hard - hard real memory usage in blocks
• nofiles_hard - hard file descriptor limit

*

• The following table contains the default hard values if the
• hard values are not explicitly defined:

*

• Attribute Value
• ========== ============
• fsize_hard set to fsize
• cpu_hard set to cpu
• core_hard -1
• data_hard -1
• stack_hard 8388608
• rss_hard -1
• nofiles_hard -1

*

• NOTE: A value of -1 implies "unlimited"

*

default:
fsize = 2097151
core = 2097151
cpu = -1
data = 262144
rss = 65536
stack = 65536
nofiles = 2000
#AIX-Forum

Originally posted by: SystemAdmin

OS: aix-5.3
Compiler: gcc-c++-3.3.2

Symptoms (one or more):
o ifstream(string name) reports random mode or intermittent failure to read
files larger than 2G
o ofstream.write() reports no error, but file on disk is truncated at 2G.
o ofstream.write() reports unable to read, and file on disk has mode 0
o linker reports undefined global symbol(s) ofstream::... even though
nm and dump report the symbols exist in provided objects and libraries

Workaround:
Use the following compile, archive, and link options.

CFLAGS=-maix64 -static
CXXFLAGS=-maix64 -static
ARFLAGS=-X64
LDFLAGS=-bbigtoc

maix64 replaces mpowerpc, mpowerpc64, mcpu=<model> and other machine flags.
Though not well documented, this appears to workaround reported large
file problems in the aix-5.2 and aix-5.3 releases.

static - shared library use is either unsupported or requires further options
not covered here

X64 - objects produced by maix64 are not acceptable to ar(1) without this
notification

bigtoc - maix64 expands pointer sizes, and may exceed string space allocation
bigtoc directs use of a potentially growing string pointer
If gcc linker alias is used remember to use gcc -Xlinker -bbigtoc
to pass the option through to the linker.

Hopefully helpful,
#AIX-Forum