I have a self-hosted Minio.
Problem: Unable to download .csv and .jpg files. Able to download .txt and .md files.
This is strange and I have been bashing my head in.
My bucket policies are extremely simple, pretty much open to any operations.
No where in my minio setup/settings, I do not have file type restrictions.
I am able to download any file type through the web portal. Unable to download the .csv, .jpg, and .tar files programmatically.
I tried manually uploading the files via the web portal and then running it programmatically, doesnt work. Only works for .md and .txt files.
I tried programmatically uploading and then downloading from the same access key, doesnt work. Only works for .md and .txt files.
I am at my wits end. Been at this for hours, cant take it anymore. Exhausted all of chatgpt help. Stackoverflow is my last attempt - trust me, I would rather be elsewhere.
Code:
class MinioUploader:
def __init__(self):
self.minioClient = Minio(
endpoint="myminio.website.com",
region="us-east-1",
access_key="myaccesskeywhichiwillnotputhere",
secret_key="mysecretketywhichiwillnotputhere",
secure=True
)
def download_all_files(self, bucket_name, local_path="./storage/local_storage/"):
if self.minioClient is None:
logs_sys.error("Minio client is not initialized.")
return
if not self.minioClient.bucket_exists(bucket_name):
logs_sys.error(f"Bucket: {bucket_name} does not exist")
return
local_path = os.path.join(local_path, bucket_name)
if not os.path.exists(local_path):
os.makedirs(local_path)
print(f"local_path: {local_path}")
objects = self.minioClient.list_objects(bucket_name, recursive=True)
for obj in objects:
try:
print(obj.object_name)
# Decode any percent-encoded characters in the object name
decoded_object_name = unquote(obj.object_name)
# Create a safe, absolute file path
safe_local_path = os.path.join(local_path, *decoded_object_name.split('/'))
# Ensure the directory for the file exists
safe_local_path = safe_local_path.replace("\\", "/")
os.makedirs(os.path.dirname(safe_local_path), exist_ok=True)
# Download the object
print(f"Downloading {obj.object_name} to {safe_local_path}")
self.minioClient.fget_object(bucket_name=bucket_name, object_name=obj.object_name, file_path=safe_local_path)
except Exception as e:
logs_sys.error(f"Error: {e}")
print(f"Error: {e}")
Print Out:
D:\TRADING_RESTORE_STRATEGIES>python -u "d:\TRADING_RESTORE_STRATEGIES\storage\minio_s3\minio_client.py"
local_path: ./storage/local_storage/testdownloadbucket
11.md
Downloading 11.md to ./storage/local_storage/testdownloadbucket/11.md
3-7-2024 3-11-49 AM.jpg
Downloading 3-7-2024 3-11-49 AM.jpg to ./storage/local_storage/testdownloadbucket/3-7-2024 3-11-49 AM.jpg
Error: S3 operation failed; code: AccessDenied, message: Access denied, resource: /testdownloadbucket/3-7-2024%203-11-49%20AM.jpg, request_id:
17BACF23766A4250, host_id: dd9025af9251148b658df7ac2e3e8, bucket_name: testdownloadbucket, object_name: 3-
7-2024 3-11-49 AM.jpg
asad.txt
Downloading asad.txt to ./storage/local_storage/testdownloadbucket/asad.txt
new.csv
Downloading new.csv to ./storage/local_storage/testdownloadbucket/new.csv
Error: S3 operation failed; code: AccessDenied, message: Access denied, resource: /testdownloadbucket/new.csv, request_id: 17BACF23839DECB7, h
ost_id: dd9025af9251148b658df7ac2e3e8, bucket_name: testdownloadbucket, object_name: new.csv
something1.csv
Downloading something1.csv to ./storage/local_storage/testdownloadbucket/something1.csv
Error: S3 operation failed; code: AccessDenied, message: Access denied, resource: /testdownloadbucket/something1.csv, request_id: 17BACF238744
C16C, host_id: dd9025af9251148b658df7ac2e3e8, bucket_name: testdownloadbucket, object_name: something1.csv
test.txt
Downloading test.txt to ./storage/local_storage/testdownloadbucket/test.txt
<minio.api.Minio object at 0x00000248C9B0B130>