AIX 7200-05-06

aws-cli/1.29.41 Python/3.9.17 AIX/2 botocore/1.31.41

When I run the following command to download a file from IBM CoS it fails after downloading around 1.4GB but this seems to change every time I try to dowbnload ir,

# aws --endpoint-url https://s3.XX-YY.cloud-object-storage.appdomain.cloud s3 cp s3://ZZZ-cloud-backups/ZZZ/copyLIVE/copyLIVE_15-12-23.zip /usr2ex/backups/zips/

download failed: s3://ZZZ-cloud-backups/ZZZ/copyLIVE/copyLIVE_15-12-23.zip to ../../usr2ex/backups/zips/copyLIVE_15-12-23.zip

There is sufficent space in the local file system for the zip file I'm downloading, which is 6GB in size.

I've spent quite a while Googling this and there are plenty of hits but nothing definitive that help me.

If I run the aws command with --debug i get a LOT of output here is what it shows when failing:

2023-12-18 15:45:11,213 - ThreadPoolExecutor-2_0 - s3transfer.tasks - DEBUG - IOWriteTask(transfer_id=0, {'offset': 1982070784}) about to wait for the following futures []
2023-12-18 15:45:11,213 - ThreadPoolExecutor-2_0 - s3transfer.tasks - DEBUG - IOWriteTask(transfer_id=0, {'offset': 1982070784}) done waiting for dependent futures
2023-12-18 15:45:11,213 - ThreadPoolExecutor-2_0 - s3transfer.utils - DEBUG - Releasing acquire 0/None
2023-12-18 15:45:11,213 - ThreadPoolExecutor-2_0 - s3transfer.tasks - DEBUG - IOWriteTask(transfer_id=0, {'offset': 1960312832}) about to wait for the following futures []
2023-12-18 15:45:11,213 - ThreadPoolExecutor-2_0 - s3transfer.tasks - DEBUG - IOWriteTask(transfer_id=0, {'offset': 1960312832}) done waiting for dependent futures
2023-12-18 15:45:11,213 - ThreadPoolExecutor-2_0 - s3transfer.utils - DEBUG - Releasing acquire 0/None
2023-12-18 15:45:11,213 - ThreadPoolExecutor-2_0 - s3transfer.tasks - DEBUG - IOWriteTask(transfer_id=0, {'offset': 1975779328}) about to wait for the following futures []
2023-12-18 15:45:11,213 - ThreadPoolExecutor-2_0 - s3transfer.tasks - DEBUG - IOWriteTask(transfer_id=0, {'offset': 1975779328}) done waiting for dependent futures
2023-12-18 15:45:11,214 - ThreadPoolExecutor-2_0 - s3transfer.utils - DEBUG - Releasing acquire 0/None
2023-12-18 15:45:11,214 - ThreadPoolExecutor-2_0 - s3transfer.tasks - DEBUG - IOWriteTask(transfer_id=0, {'offset': 1960574976}) about to wait for the following futures []
2023-12-18 15:45:11,214 - ThreadPoolExecutor-2_0 - s3transfer.tasks - DEBUG - IOWriteTask(transfer_id=0, {'offset': 1960574976}) done waiting for dependent futures
2023-12-18 15:45:11,214 - ThreadPoolExecutor-2_0 - s3transfer.utils - DEBUG - Releasing acquire 0/None
2023-12-18 15:45:11,214 - ThreadPoolExecutor-2_0 - s3transfer.tasks - DEBUG - IOWriteTask(transfer_id=0, {'offset': 1967915008}) about to wait for the following futures []
2023-12-18 15:45:11,214 - ThreadPoolExecutor-2_0 - s3transfer.tasks - DEBUG - IOWriteTask(transfer_id=0, {'offset': 1967915008}) done waiting for dependent futures
2023-12-18 15:45:11,214 - ThreadPoolExecutor-2_0 - s3transfer.utils - DEBUG - Releasing acquire 0/None
2023-12-18 15:45:11,214 - ThreadPoolExecutor-2_0 - s3transfer.tasks - DEBUG - IOWriteTask(transfer_id=0, {'offset': 1968177152}) about to wait for the following futures []
2023-12-18 15:45:11,215 - ThreadPoolExecutor-2_0 - s3transfer.tasks - DEBUG - IOWriteTask(transfer_id=0, {'offset': 1968177152}) done waiting for dependent futures
2023-12-18 15:45:11,215 - ThreadPoolExecutor-2_0 - s3transfer.utils - DEBUG - Releasing acquire 0/None
2023-12-18 15:45:11,215 - ThreadPoolExecutor-2_0 - s3transfer.tasks - DEBUG - IOWriteTask(transfer_id=0, {'offset': 1936457728}) about to wait for the following futures []
2023-12-18 15:45:11,215 - ThreadPoolExecutor-2_0 - s3transfer.tasks - DEBUG - IOWriteTask(transfer_id=0, {'offset': 1936457728}) done waiting for dependent futures
2023-12-18 15:45:11,215 - ThreadPoolExecutor-2_0 - s3transfer.utils - DEBUG - Releasing acquire 0/None
2023-12-18 15:45:11,215 - ThreadPoolExecutor-2_0 - s3transfer.tasks - DEBUG - IOWriteTask(transfer_id=0, {'offset': 2004877312}) about to wait for the following futures []
2023-12-18 15:45:11,215 - ThreadPoolExecutor-2_0 - s3transfer.tasks - DEBUG - IOWriteTask(transfer_id=0, {'offset': 2004877312}) done waiting for dependent futures
2023-12-18 15:45:11,216 - ThreadPoolExecutor-2_0 - s3transfer.utils - DEBUG - Releasing acquire 0/None
2023-12-18 15:45:11,216 - ThreadPoolExecutor-2_0 - s3transfer.tasks - DEBUG - IOWriteTask(transfer_id=0, {'offset': 1997275136}) about to wait for the following futures []
2023-12-18 15:45:11,216 - ThreadPoolExecutor-2_0 - s3transfer.tasks - DEBUG - IOWriteTask(transfer_id=0, {'offset': 1997275136}) done waiting for dependent futures
2023-12-18 15:45:11,216 - ThreadPoolExecutor-2_0 - s3transfer.utils - DEBUG - Releasing acquire 0/None
2023-12-18 15:45:11,216 - ThreadPoolExecutor-2_0 - s3transfer.tasks - DEBUG - IOWriteTask(transfer_id=0, {'offset': 1989935104}) about to wait for the following futures []
2023-12-18 15:45:11,216 - ThreadPoolExecutor-2_0 - s3transfer.tasks - DEBUG - IOWriteTask(transfer_id=0, {'offset': 1989935104}) done waiting for dependent futures
2023-12-18 15:45:11,216 - ThreadPoolExecutor-2_0 - s3transfer.utils - DEBUG - Releasing acquire 0/None
2023-12-18 15:45:11,216 - ThreadPoolExecutor-2_0 - s3transfer.tasks - DEBUG - IOWriteTask(transfer_id=0, {'offset': 1936719872}) about to wait for the following futures []
2023-12-18 15:45:11,217 - ThreadPoolExecutor-2_0 - s3transfer.tasks - DEBUG - IOWriteTask(transfer_id=0, {'offset': 1936719872}) done waiting for dependent futures
2023-12-18 15:45:11,217 - ThreadPoolExecutor-2_0 - s3transfer.utils - DEBUG - Releasing acquire 0/None
2023-12-18 15:45:11,217 - ThreadPoolExecutor-2_0 - s3transfer.tasks - DEBUG - IOWriteTask(transfer_id=0, {'offset': 1952710656}) about to wait for the following futures []
2023-12-18 15:45:11,217 - ThreadPoolExecutor-2_0 - s3transfer.tasks - DEBUG - IOWriteTask(transfer_id=0, {'offset': 1952710656}) done waiting for dependent futures
2023-12-18 15:45:11,217 - ThreadPoolExecutor-2_0 - s3transfer.utils - DEBUG - Releasing acquire 0/None
2023-12-18 15:45:11,217 - ThreadPoolExecutor-2_0 - s3transfer.tasks - DEBUG - IOWriteTask(transfer_id=0, {'offset': 1944584192}) about to wait for the following futures []
2023-12-18 15:45:11,217 - ThreadPoolExecutor-2_0 - s3transfer.tasks - DEBUG - IOWriteTask(transfer_id=0, {'offset': 1944584192}) done waiting for dependent futures
2023-12-18 15:45:11,217 - ThreadPoolExecutor-2_0 - s3transfer.utils - DEBUG - Releasing acquire 0/None
2023-12-18 15:45:11,218 - ThreadPoolExecutor-2_0 - s3transfer.tasks - DEBUG - IOWriteTask(transfer_id=0, {'offset': 1952972800}) about to wait for the following futures []
2023-12-18 15:45:11,218 - ThreadPoolExecutor-2_0 - s3transfer.tasks - DEBUG - IOWriteTask(transfer_id=0, {'offset': 1952972800}) done waiting for dependent futures
2023-12-18 15:45:11,218 - ThreadPoolExecutor-2_0 - s3transfer.utils - DEBUG - Releasing acquire 0/None
2023-12-18 15:45:11,218 - ThreadPoolExecutor-2_0 - s3transfer.tasks - DEBUG - IOWriteTask(transfer_id=0, {'offset': 1982332928}) about to wait for the following futures []
2023-12-18 15:45:11,218 - ThreadPoolExecutor-2_0 - s3transfer.tasks - DEBUG - IOWriteTask(transfer_id=0, {'offset': 1982332928}) done waiting for dependent futures
2023-12-18 15:45:11,218 - ThreadPoolExecutor-2_0 - s3transfer.utils - DEBUG - Releasing acquire 0/None
2023-12-18 15:45:11,218 - ThreadPoolExecutor-2_0 - s3transfer.tasks - DEBUG - IOWriteTask(transfer_id=0, {'offset': 1944846336}) about to wait for the following futures []
2023-12-18 15:45:11,218 - ThreadPoolExecutor-2_0 - s3transfer.tasks - DEBUG - IOWriteTask(transfer_id=0, {'offset': 1944846336}) done waiting for dependent futures
2023-12-18 15:45:11,219 - ThreadPoolExecutor-2_0 - s3transfer.utils - DEBUG - Releasing acquire 0/None
2023-12-18 15:45:11,219 - ThreadPoolExecutor-2_0 - s3transfer.tasks - DEBUG - IOWriteTask(transfer_id=0, {'offset': 1976041472}) about to wait for the following futures []
2023-12-18 15:45:11,219 - ThreadPoolExecutor-2_0 - s3transfer.tasks - DEBUG - IOWriteTask(transfer_id=0, {'offset': 1976041472}) done waiting for dependent futures
2023-12-18 15:45:11,219 - ThreadPoolExecutor-2_0 - s3transfer.utils - DEBUG - Releasing acquire 0/None
2023-12-18 15:45:11,219 - ThreadPoolExecutor-2_0 - s3transfer.tasks - DEBUG - IOWriteTask(transfer_id=0, {'offset': 1960837120}) about to wait for the following futures []
2023-12-18 15:45:11,219 - ThreadPoolExecutor-2_0 - s3transfer.tasks - DEBUG - IOWriteTask(transfer_id=0, {'offset': 1960837120}) done waiting for dependent futures
2023-12-18 15:45:11,219 - ThreadPoolExecutor-2_0 - s3transfer.utils - DEBUG - Releasing acquire 0/None
2023-12-18 15:45:11,219 - ThreadPoolExecutor-2_0 - s3transfer.tasks - DEBUG - IOWriteTask(transfer_id=0, {'offset': 1976303616}) about to wait for the following futures []
2023-12-18 15:45:11,220 - ThreadPoolExecutor-2_0 - s3transfer.tasks - DEBUG - IOWriteTask(transfer_id=0, {'offset': 1976303616}) done waiting for dependent futures
2023-12-18 15:45:11,220 - ThreadPoolExecutor-2_0 - s3transfer.utils - DEBUG - Releasing acquire 0/None
2023-12-18 15:45:11,220 - ThreadPoolExecutor-2_0 - s3transfer.tasks - DEBUG - IOWriteTask(transfer_id=0, {'offset': 1961099264}) about to wait for the following futures []
2023-12-18 15:45:11,220 - ThreadPoolExecutor-2_0 - s3transfer.tasks - DEBUG - IOWriteTask(transfer_id=0, {'offset': 1961099264}) done waiting for dependent futures
2023-12-18 15:45:11,220 - ThreadPoolExecutor-2_0 - s3transfer.utils - DEBUG - Releasing acquire 0/None
2023-12-18 15:45:11,220 - ThreadPoolExecutor-2_0 - s3transfer.tasks - DEBUG - IOWriteTask(transfer_id=0, {'offset': 1968439296}) about to wait for the following futures []
2023-12-18 15:45:11,220 - ThreadPoolExecutor-2_0 - s3transfer.tasks - DEBUG - IOWriteTask(transfer_id=0, {'offset': 1968439296}) done waiting for dependent futures
2023-12-18 15:45:11,220 - ThreadPoolExecutor-2_0 - s3transfer.utils - DEBUG - Releasing acquire 0/None
2023-12-18 15:45:11,221 - ThreadPoolExecutor-2_0 - s3transfer.tasks - DEBUG - IORenameFileTask(transfer_id=0, {'final_filename': '/usr2ex/backups/zips/copyLIVE_15-12-23.zip'}) about to wait for the following futures []
2023-12-18 15:45:11,221 - ThreadPoolExecutor-2_0 - s3transfer.tasks - DEBUG - IORenameFileTask(transfer_id=0, {'final_filename': '/usr2ex/backups/zips/copyLIVE_15-12-23.zip'}) done waiting for dependent futures
2023-12-18 15:45:14,250 - ThreadPoolExecutor-2_0 - s3transfer.utils - DEBUG - Releasing acquire 0/None
download failed: s3://ZZZ-cloud-backups/ZZZ/copyLIVE/copyLIVE_15-12-23.zip to ../../usr2ex/backups/zips/copyLIVE_15-12-23.zip
2023-12-18 15:45:14,268 - Thread-1 - awscli.customizations.s3.results - DEBUG - Shutdown request received in result processing thread, shutting down result thread.

Any thoughts on what is causing this to fail?

Sorry,

core file size              (blocks, -c) 0
data seg size               (kbytes, -d) 131072
file size                   (blocks, -f) unlimited
max memory size             (kbytes, -m) 32768
open files                          (-n) 10000
pipe size                (512 bytes, -p) 64
stack size                  (kbytes, -s) 32768
cpu time                   (seconds, -t) unlimited
max user processes                  (-u) unlimited
virtual memory              (kbytes, -v) unlimited

I've attached the SSH log file with --debug set on the aws command.

I noticed towards the end it has a memory error:

Traceback (most recent call last):
  File "/opt/freeware/lib/python3.9/site-packages/s3transfer/tasks.py", line 139, in __call__
    return self._execute_main(kwargs)
  File "/opt/freeware/lib/python3.9/site-packages/s3transfer/tasks.py", line 162, in _execute_main
    return_value = self._main(**kwargs)
  File "/opt/freeware/lib/python3.9/site-packages/s3transfer/download.py", line 583, in _main
    for chunk in chunks:
  File "/opt/freeware/lib/python3.9/site-packages/s3transfer/download.py", line 728, in __next__
    chunk = self._body.read(self._chunksize)
  File "/opt/freeware/lib/python3.9/site-packages/s3transfer/utils.py", line 599, in read
    value = self._stream.read(*args, **kwargs)
  File "/opt/freeware/lib/python3.9/site-packages/botocore/response.py", line 99, in read
    chunk = self._raw_stream.read(amt)
  File "/opt/freeware/lib/python3.9/site-packages/urllib3/response.py", line 567, in read
    data = self._fp_read(amt) if not fp_closed else b""
  File "/opt/freeware/lib/python3.9/site-packages/urllib3/response.py", line 533, in _fp_read
    return self._fp.read(amt) if amt is not None else self._fp.read()
  File "/opt/freeware/lib64/python3.9/http/client.py", line 464, in read
    return memoryview(b)[:n].tobytes()
MemoryError

Any guidance on this?

Attachment(s)

awsclierror.log   4.73 MB 1 version

I have no idea on AWS S3 CLI, but I see that the errors start after it downloaded 717 MiB. Giving your limits I don't expect that you have limits problem but still:

• check that you have enough space in the filesystem or in temporary filesystem. I don't know where AWS CLI makes temporary files, but I don't believe or better to say I hope that they don't save 717MB in RAM.
• Set all limits to unlimited for the user and try the operation. If it helps, then you definitely have limits problem.
• Try to download differnt files. Does it work if you download 512MB file? Or 2GB file?

Smaller files work fine.

I have just downloaded an bash script from Github, https://github.com/paulhammond/s3simple, which uses curl to download from an S3 bucket. I have run this multiple times using different files and it work without any issues.

Looks to me like the aws-cli issue is python related.