Smaller files work fine.
Looks to me like the aws-cli issue is python related.
Original Message:
Sent: Tue December 19, 2023 06:04 AM
From: Andrey Klyachkin
Subject: awscli crashes when downloading from IBM Cloud Object Storage
I have no idea on AWS S3 CLI, but I see that the errors start after it downloaded 717 MiB. Giving your limits I don't expect that you have limits problem but still:
- check that you have enough space in the filesystem or in temporary filesystem. I don't know where AWS CLI makes temporary files, but I don't believe or better to say I hope that they don't save 717MB in RAM.
- Set all limits to unlimited for the user and try the operation. If it helps, then you definitely have limits problem.
- Try to download differnt files. Does it work if you download 512MB file? Or 2GB file?
------------------------------
Andrey Klyachkin
https://www.power-devops.com
Original Message:
Sent: Tue December 19, 2023 04:34 AM
From: Glenn Robinson
Subject: awscli crashes when downloading from IBM Cloud Object Storage
I've attached the SSH log file with --debug set on the aws command.
I noticed towards the end it has a memory error:
Traceback (most recent call last):
File "/opt/freeware/lib/python3.9/site-packages/s3transfer/tasks.py", line 139, in __call__
return self._execute_main(kwargs)
File "/opt/freeware/lib/python3.9/site-packages/s3transfer/tasks.py", line 162, in _execute_main
return_value = self._main(**kwargs)
File "/opt/freeware/lib/python3.9/site-packages/s3transfer/download.py", line 583, in _main
for chunk in chunks:
File "/opt/freeware/lib/python3.9/site-packages/s3transfer/download.py", line 728, in __next__
chunk = self._body.read(self._chunksize)
File "/opt/freeware/lib/python3.9/site-packages/s3transfer/utils.py", line 599, in read
value = self._stream.read(*args, **kwargs)
File "/opt/freeware/lib/python3.9/site-packages/botocore/response.py", line 99, in read
chunk = self._raw_stream.read(amt)
File "/opt/freeware/lib/python3.9/site-packages/urllib3/response.py", line 567, in read
data = self._fp_read(amt) if not fp_closed else b""
File "/opt/freeware/lib/python3.9/site-packages/urllib3/response.py", line 533, in _fp_read
return self._fp.read(amt) if amt is not None else self._fp.read()
File "/opt/freeware/lib64/python3.9/http/client.py", line 464, in read
return memoryview(b)[:n].tobytes()
MemoryError
Any guidance on this?
------------------------------
Glenn Robinson
Original Message:
Sent: Mon December 18, 2023 11:03 AM
From: Glenn Robinson
Subject: awscli crashes when downloading from IBM Cloud Object Storage
Sorry,
core file size (blocks, -c) 0
data seg size (kbytes, -d) 131072
file size (blocks, -f) unlimited
max memory size (kbytes, -m) 32768
open files (-n) 10000
pipe size (512 bytes, -p) 64
stack size (kbytes, -s) 32768
cpu time (seconds, -t) unlimited
max user processes (-u) unlimited
virtual memory (kbytes, -v) unlimited
------------------------------
Glenn Robinson
Original Message:
Sent: Mon December 18, 2023 10:58 AM
From: Stephen Ulmer
Subject: awscli crashes when downloading from IBM Cloud Object Storage
You did not mention checking the ulimit(s) in the shell from which you are running the download. :)
--
Stephen L. Ulmer
Enterprise Architect
Mainline Information Systems
Original Message:
Sent: 12/18/2023 10:48:00 AM
From: Glenn Robinson
Subject: awscli crashes when downloading from IBM Cloud Object Storage
AIX 7200-05-06
aws-cli/1.29.41 Python/3.9.17 AIX/2 botocore/1.31.41
When I run the following command to download a file from IBM CoS it fails after downloading around 1.4GB but this seems to change every time I try to dowbnload ir,
# aws --endpoint-url https://s3.XX-YY.cloud-object-storage.appdomain.cloud s3 cp s3://ZZZ-cloud-backups/ZZZ/copyLIVE/copyLIVE_15-12-23.zip /usr2ex/backups/zips/
download failed: s3://ZZZ-cloud-backups/ZZZ/copyLIVE/copyLIVE_15-12-23.zip to ../../usr2ex/backups/zips/copyLIVE_15-12-23.zip
There is sufficent space in the local file system for the zip file I'm downloading, which is 6GB in size.
I've spent quite a while Googling this and there are plenty of hits but nothing definitive that help me.
If I run the aws command with --debug i get a LOT of output here is what it shows when failing:
2023-12-18 15:45:11,213 - ThreadPoolExecutor-2_0 - s3transfer.tasks - DEBUG - IOWriteTask(transfer_id=0, {'offset': 1982070784}) about to wait for the following futures []
2023-12-18 15:45:11,213 - ThreadPoolExecutor-2_0 - s3transfer.tasks - DEBUG - IOWriteTask(transfer_id=0, {'offset': 1982070784}) done waiting for dependent futures
2023-12-18 15:45:11,213 - ThreadPoolExecutor-2_0 - s3transfer.utils - DEBUG - Releasing acquire 0/None
2023-12-18 15:45:11,213 - ThreadPoolExecutor-2_0 - s3transfer.tasks - DEBUG - IOWriteTask(transfer_id=0, {'offset': 1960312832}) about to wait for the following futures []
2023-12-18 15:45:11,213 - ThreadPoolExecutor-2_0 - s3transfer.tasks - DEBUG - IOWriteTask(transfer_id=0, {'offset': 1960312832}) done waiting for dependent futures
2023-12-18 15:45:11,213 - ThreadPoolExecutor-2_0 - s3transfer.utils - DEBUG - Releasing acquire 0/None
2023-12-18 15:45:11,213 - ThreadPoolExecutor-2_0 - s3transfer.tasks - DEBUG - IOWriteTask(transfer_id=0, {'offset': 1975779328}) about to wait for the following futures []
2023-12-18 15:45:11,213 - ThreadPoolExecutor-2_0 - s3transfer.tasks - DEBUG - IOWriteTask(transfer_id=0, {'offset': 1975779328}) done waiting for dependent futures
2023-12-18 15:45:11,214 - ThreadPoolExecutor-2_0 - s3transfer.utils - DEBUG - Releasing acquire 0/None
2023-12-18 15:45:11,214 - ThreadPoolExecutor-2_0 - s3transfer.tasks - DEBUG - IOWriteTask(transfer_id=0, {'offset': 1960574976}) about to wait for the following futures []
2023-12-18 15:45:11,214 - ThreadPoolExecutor-2_0 - s3transfer.tasks - DEBUG - IOWriteTask(transfer_id=0, {'offset': 1960574976}) done waiting for dependent futures
2023-12-18 15:45:11,214 - ThreadPoolExecutor-2_0 - s3transfer.utils - DEBUG - Releasing acquire 0/None
2023-12-18 15:45:11,214 - ThreadPoolExecutor-2_0 - s3transfer.tasks - DEBUG - IOWriteTask(transfer_id=0, {'offset': 1967915008}) about to wait for the following futures []
2023-12-18 15:45:11,214 - ThreadPoolExecutor-2_0 - s3transfer.tasks - DEBUG - IOWriteTask(transfer_id=0, {'offset': 1967915008}) done waiting for dependent futures
2023-12-18 15:45:11,214 - ThreadPoolExecutor-2_0 - s3transfer.utils - DEBUG - Releasing acquire 0/None
2023-12-18 15:45:11,214 - ThreadPoolExecutor-2_0 - s3transfer.tasks - DEBUG - IOWriteTask(transfer_id=0, {'offset': 1968177152}) about to wait for the following futures []
2023-12-18 15:45:11,215 - ThreadPoolExecutor-2_0 - s3transfer.tasks - DEBUG - IOWriteTask(transfer_id=0, {'offset': 1968177152}) done waiting for dependent futures
2023-12-18 15:45:11,215 - ThreadPoolExecutor-2_0 - s3transfer.utils - DEBUG - Releasing acquire 0/None
2023-12-18 15:45:11,215 - ThreadPoolExecutor-2_0 - s3transfer.tasks - DEBUG - IOWriteTask(transfer_id=0, {'offset': 1936457728}) about to wait for the following futures []
2023-12-18 15:45:11,215 - ThreadPoolExecutor-2_0 - s3transfer.tasks - DEBUG - IOWriteTask(transfer_id=0, {'offset': 1936457728}) done waiting for dependent futures
2023-12-18 15:45:11,215 - ThreadPoolExecutor-2_0 - s3transfer.utils - DEBUG - Releasing acquire 0/None
2023-12-18 15:45:11,215 - ThreadPoolExecutor-2_0 - s3transfer.tasks - DEBUG - IOWriteTask(transfer_id=0, {'offset': 2004877312}) about to wait for the following futures []
2023-12-18 15:45:11,215 - ThreadPoolExecutor-2_0 - s3transfer.tasks - DEBUG - IOWriteTask(transfer_id=0, {'offset': 2004877312}) done waiting for dependent futures
2023-12-18 15:45:11,216 - ThreadPoolExecutor-2_0 - s3transfer.utils - DEBUG - Releasing acquire 0/None
2023-12-18 15:45:11,216 - ThreadPoolExecutor-2_0 - s3transfer.tasks - DEBUG - IOWriteTask(transfer_id=0, {'offset': 1997275136}) about to wait for the following futures []
2023-12-18 15:45:11,216 - ThreadPoolExecutor-2_0 - s3transfer.tasks - DEBUG - IOWriteTask(transfer_id=0, {'offset': 1997275136}) done waiting for dependent futures
2023-12-18 15:45:11,216 - ThreadPoolExecutor-2_0 - s3transfer.utils - DEBUG - Releasing acquire 0/None
2023-12-18 15:45:11,216 - ThreadPoolExecutor-2_0 - s3transfer.tasks - DEBUG - IOWriteTask(transfer_id=0, {'offset': 1989935104}) about to wait for the following futures []
2023-12-18 15:45:11,216 - ThreadPoolExecutor-2_0 - s3transfer.tasks - DEBUG - IOWriteTask(transfer_id=0, {'offset': 1989935104}) done waiting for dependent futures
2023-12-18 15:45:11,216 - ThreadPoolExecutor-2_0 - s3transfer.utils - DEBUG - Releasing acquire 0/None
2023-12-18 15:45:11,216 - ThreadPoolExecutor-2_0 - s3transfer.tasks - DEBUG - IOWriteTask(transfer_id=0, {'offset': 1936719872}) about to wait for the following futures []
2023-12-18 15:45:11,217 - ThreadPoolExecutor-2_0 - s3transfer.tasks - DEBUG - IOWriteTask(transfer_id=0, {'offset': 1936719872}) done waiting for dependent futures
2023-12-18 15:45:11,217 - ThreadPoolExecutor-2_0 - s3transfer.utils - DEBUG - Releasing acquire 0/None
2023-12-18 15:45:11,217 - ThreadPoolExecutor-2_0 - s3transfer.tasks - DEBUG - IOWriteTask(transfer_id=0, {'offset': 1952710656}) about to wait for the following futures []
2023-12-18 15:45:11,217 - ThreadPoolExecutor-2_0 - s3transfer.tasks - DEBUG - IOWriteTask(transfer_id=0, {'offset': 1952710656}) done waiting for dependent futures
2023-12-18 15:45:11,217 - ThreadPoolExecutor-2_0 - s3transfer.utils - DEBUG - Releasing acquire 0/None
2023-12-18 15:45:11,217 - ThreadPoolExecutor-2_0 - s3transfer.tasks - DEBUG - IOWriteTask(transfer_id=0, {'offset': 1944584192}) about to wait for the following futures []
2023-12-18 15:45:11,217 - ThreadPoolExecutor-2_0 - s3transfer.tasks - DEBUG - IOWriteTask(transfer_id=0, {'offset': 1944584192}) done waiting for dependent futures
2023-12-18 15:45:11,217 - ThreadPoolExecutor-2_0 - s3transfer.utils - DEBUG - Releasing acquire 0/None
2023-12-18 15:45:11,218 - ThreadPoolExecutor-2_0 - s3transfer.tasks - DEBUG - IOWriteTask(transfer_id=0, {'offset': 1952972800}) about to wait for the following futures []
2023-12-18 15:45:11,218 - ThreadPoolExecutor-2_0 - s3transfer.tasks - DEBUG - IOWriteTask(transfer_id=0, {'offset': 1952972800}) done waiting for dependent futures
2023-12-18 15:45:11,218 - ThreadPoolExecutor-2_0 - s3transfer.utils - DEBUG - Releasing acquire 0/None
2023-12-18 15:45:11,218 - ThreadPoolExecutor-2_0 - s3transfer.tasks - DEBUG - IOWriteTask(transfer_id=0, {'offset': 1982332928}) about to wait for the following futures []
2023-12-18 15:45:11,218 - ThreadPoolExecutor-2_0 - s3transfer.tasks - DEBUG - IOWriteTask(transfer_id=0, {'offset': 1982332928}) done waiting for dependent futures
2023-12-18 15:45:11,218 - ThreadPoolExecutor-2_0 - s3transfer.utils - DEBUG - Releasing acquire 0/None
2023-12-18 15:45:11,218 - ThreadPoolExecutor-2_0 - s3transfer.tasks - DEBUG - IOWriteTask(transfer_id=0, {'offset': 1944846336}) about to wait for the following futures []
2023-12-18 15:45:11,218 - ThreadPoolExecutor-2_0 - s3transfer.tasks - DEBUG - IOWriteTask(transfer_id=0, {'offset': 1944846336}) done waiting for dependent futures
2023-12-18 15:45:11,219 - ThreadPoolExecutor-2_0 - s3transfer.utils - DEBUG - Releasing acquire 0/None
2023-12-18 15:45:11,219 - ThreadPoolExecutor-2_0 - s3transfer.tasks - DEBUG - IOWriteTask(transfer_id=0, {'offset': 1976041472}) about to wait for the following futures []
2023-12-18 15:45:11,219 - ThreadPoolExecutor-2_0 - s3transfer.tasks - DEBUG - IOWriteTask(transfer_id=0, {'offset': 1976041472}) done waiting for dependent futures
2023-12-18 15:45:11,219 - ThreadPoolExecutor-2_0 - s3transfer.utils - DEBUG - Releasing acquire 0/None
2023-12-18 15:45:11,219 - ThreadPoolExecutor-2_0 - s3transfer.tasks - DEBUG - IOWriteTask(transfer_id=0, {'offset': 1960837120}) about to wait for the following futures []
2023-12-18 15:45:11,219 - ThreadPoolExecutor-2_0 - s3transfer.tasks - DEBUG - IOWriteTask(transfer_id=0, {'offset': 1960837120}) done waiting for dependent futures
2023-12-18 15:45:11,219 - ThreadPoolExecutor-2_0 - s3transfer.utils - DEBUG - Releasing acquire 0/None
2023-12-18 15:45:11,219 - ThreadPoolExecutor-2_0 - s3transfer.tasks - DEBUG - IOWriteTask(transfer_id=0, {'offset': 1976303616}) about to wait for the following futures []
2023-12-18 15:45:11,220 - ThreadPoolExecutor-2_0 - s3transfer.tasks - DEBUG - IOWriteTask(transfer_id=0, {'offset': 1976303616}) done waiting for dependent futures
2023-12-18 15:45:11,220 - ThreadPoolExecutor-2_0 - s3transfer.utils - DEBUG - Releasing acquire 0/None
2023-12-18 15:45:11,220 - ThreadPoolExecutor-2_0 - s3transfer.tasks - DEBUG - IOWriteTask(transfer_id=0, {'offset': 1961099264}) about to wait for the following futures []
2023-12-18 15:45:11,220 - ThreadPoolExecutor-2_0 - s3transfer.tasks - DEBUG - IOWriteTask(transfer_id=0, {'offset': 1961099264}) done waiting for dependent futures
2023-12-18 15:45:11,220 - ThreadPoolExecutor-2_0 - s3transfer.utils - DEBUG - Releasing acquire 0/None
2023-12-18 15:45:11,220 - ThreadPoolExecutor-2_0 - s3transfer.tasks - DEBUG - IOWriteTask(transfer_id=0, {'offset': 1968439296}) about to wait for the following futures []
2023-12-18 15:45:11,220 - ThreadPoolExecutor-2_0 - s3transfer.tasks - DEBUG - IOWriteTask(transfer_id=0, {'offset': 1968439296}) done waiting for dependent futures
2023-12-18 15:45:11,220 - ThreadPoolExecutor-2_0 - s3transfer.utils - DEBUG - Releasing acquire 0/None
2023-12-18 15:45:11,221 - ThreadPoolExecutor-2_0 - s3transfer.tasks - DEBUG - IORenameFileTask(transfer_id=0, {'final_filename': '/usr2ex/backups/zips/copyLIVE_15-12-23.zip'}) about to wait for the following futures []
2023-12-18 15:45:11,221 - ThreadPoolExecutor-2_0 - s3transfer.tasks - DEBUG - IORenameFileTask(transfer_id=0, {'final_filename': '/usr2ex/backups/zips/copyLIVE_15-12-23.zip'}) done waiting for dependent futures
2023-12-18 15:45:14,250 - ThreadPoolExecutor-2_0 - s3transfer.utils - DEBUG - Releasing acquire 0/None
download failed: s3://ZZZ-cloud-backups/ZZZ/copyLIVE/copyLIVE_15-12-23.zip to ../../usr2ex/backups/zips/copyLIVE_15-12-23.zip
2023-12-18 15:45:14,268 - Thread-1 - awscli.customizations.s3.results - DEBUG - Shutdown request received in result processing thread, shutting down result thread.
Any thoughts on what is causing this to fail?
------------------------------
Glenn Robinson
------------------------------
#AIXOpenSource