Knowledge: Multipart Upload

Multipart Upload

Class 40th AWS Multipart Upload June9th

(AWS Recommended ,you are upload the file more than 100 MB ,the will be split chucks, finally join the chuck into single file that is best approach to upload s3 bucket.)

Multipart upload allow users to break a single file into multiple part and upload them independently.
You can upload these object parts independently and in any order. If transmission of any part fails, you can retransmit that part without affecting other parts
After all parts of your object are uploaded, Amazon S3 assembles these parts and create the object.
In general, when your object size reaches 100 MB, you should consider using multipart uploads instead of uploading the object in a single operation.
You can split the file into 1000 Slices maximum. Each file must be 5 mb in size .Incomplete upload part remain in the S3 and AWS Charges for that.

Practical

Step1: Create one bucket "ccitmultipartupload1"

First choose the file 284 MB (split the file) using command, The command you need to run in git hub

$ split -b 50M video.mp4 video_part_

Administrator@DESKTOP-AV2PARO MINGW64 ~/Desktop/mulitpart

$ ls -lrt

total 1175740

-rw-r--r-- 1 Administrator 197121 298403108 Dec 12 11:06 video.mp4

-rw-r--r-- 1 Administrator 197121 607142013 Dec 15 09:04 video_20241215_085947.mp4

-rw-r--r-- 1 Administrator 197121 52428800 Jun 11 00:48 video_part_aa

-rw-r--r-- 1 Administrator 197121 52428800 Jun 11 00:48 video_part_ab

-rw-r--r-- 1 Administrator 197121 52428800 Jun 11 00:48 video_part_ac

-rw-r--r-- 1 Administrator 197121 52428800 Jun 11 00:48 video_part_ad

-rw-r--r-- 1 Administrator 197121 52428800 Jun 11 00:48 video_part_ae

-rw-r--r-- 1 Administrator 197121 36259108 Jun 11 00:48 video_part_af

Step2: Upload the files in the newly created bucket using Cli command

PS C:\Users\Administrator> aws configure

AWS Access Key ID [****************HWK4]: AKIATFBMO7H4BGIOIEW6

AWS Secret Access Key [****************uOLH]: 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

Default region name [eu-west-1]: eu-west-1

Default output format [None]: json

what is the use of this command means upload id will generate the key , based on the key we will join them into single file

PS C:\Users\Administrator\Desktop\mulitpart>aws s3api create-multipart-upload --bucket ccitmultipartupload2 --key video.mp4 --region eu-west-1

{

"ServerSideEncryption": "AES256",

"Bucket": "ccitmultipartupload2",

"Key": "video.mp4",

"UploadId": "XOdv_ADNJnTuS38S6iXGxorl0TFVpDFCGoEJ2FoO9wx0cpkIhrzMZks4phjX.y_1HUCWDBZKJ9LNO6sVEUOBSud6Y047fk3CcKI3mAhbo2r_QbK_T1DYkTUtj7jbs5Jl"

}

See here upload S3 bucket showing empty ,until you joining all part of the file the uploaded file will not shown

Step3:

Part1

PS C:\Users\Administrator\Desktop\mulitpart> aws s3api upload-part --bucket ccitmultipartupload2 --key video.mp4 --part-number 1 --body C:\Users\Administrator\Desktop\mulitpart\Video_part_aa --upload-id XOdv_ADNJnTuS38S6iXGxorl0TFVpDFCGoEJ2FoO9wx0cpkIhrzMZks4phjX.y_1HUCWDBZKJ9LNO6sVEUOBSud6Y047fk3CcKI3mAhbo2r_QbK_T1DYkTUtj7jbs5Jl

{

"ServerSideEncryption": "AES256",

"ETag": "\"f6279bc6c5aa4efa009c0d599cd1b206\"",

Part2

PS C:\Users\Administrator\Desktop\mulitpart> aws s3api upload-part --bucket ccitmultipartupload2 --key video.mp4 --part-number 2 --body C:\Users\Administrator\Desktop\mulitpart\Video_part_ab --upload-id XOdv_ADNJnTuS38S6iXGxorl0TFVpDFCGoEJ2FoO9wx0cpkIhrzMZks4phjX.y_1HUCWDBZKJ9LNO6sVEUOBSud6Y047fk3CcKI3mAhbo2r_QbK_T1DYkTUtj7jbs5Jl

{

"ServerSideEncryption": "AES256",

"ETag": "\"5224ca5d459b8044bb554cfffdc4c122\"",

"ChecksumCRC64NVME": "VVdRkDOJyYg="

}

Part3

PS C:\Users\Administrator\Desktop\mulitpart> aws s3api upload-part --bucket ccitmultipartupload2 --key video.mp4 --part-number 3 --body C:\Users\Administrator\Desktop\mulitpart\Video_part_ac --upload-id XOdv_ADNJnTuS38S6iXGxorl0TFVpDFCGoEJ2FoO9wx0cpkIhrzMZks4phjX.y_1HUCWDBZKJ9LNO6sVEUOBSud6Y047fk3CcKI3mAhbo2r_QbK_T1DYkTUtj7jbs5Jl

{

"ServerSideEncryption": "AES256",

"ETag": "\"e9942afaa877e2a530da4b06d8eae9fb\"",

"ChecksumCRC64NVME": "9touXfRy/cc="

}

Part4

PS C:\Users\Administrator\Desktop\mulitpart> aws s3api upload-part --bucket ccitmultipartupload2 --key video.mp4 --part-number 4 --body C:\Users\Administrator\Desktop\mulitpart\Video_part_ad --upload-id XOdv_ADNJnTuS38S6iXGxorl0TFVpDFCGoEJ2FoO9wx0cpkIhrzMZks4phjX.y_1HUCWDBZKJ9LNO6sVEUOBSud6Y047fk3CcKI3mAhbo2r_QbK_T1DYkTUtj7jbs5Jl

{

"ServerSideEncryption": "AES256",

"ETag": "\"fcdd80466dd85133c87772e41123f3fa\"",

"ChecksumCRC64NVME": "tctu0jHQJ1A="

}

Part5

PS C:\Users\Administrator\Desktop\mulitpart>aws s3api upload-part --bucket ccitmultipartupload2 --key video.mp4 --part-number 5 --body C:\Users\Administrator\Desktop\mulitpart\Video_part_ae --upload-id XOdv_ADNJnTuS38S6iXGxorl0TFVpDFCGoEJ2FoO9wx0cpkIhrzMZks4phjX.y_1HUCWDBZKJ9LNO6sVEUOBSud6Y047fk3CcKI3mAhbo2r_QbK_T1DYkTUtj7jbs5Jl

{

"ServerSideEncryption": "AES256",

"ETag": "\"9318088f5b6055f1f5af8f69bf982b49\"",

"ChecksumCRC64NVME": "yjZxdkyCmm8="

}

Part6

PS C:\Users\Administrator\Desktop\mulitpart> aws s3api upload-part --bucket ccitmultipartupload2 --key video.mp4 --part-number 6 --body C:\Users\Administrator\Desktop\mulitpart\Video_part_af --upload-id XOdv_ADNJnTuS38S6iXGxorl0TFVpDFCGoEJ2FoO9wx0cpkIhrzMZks4phjX.y_1HUCWDBZKJ9LNO6sVEUOBSud6Y047fk3CcKI3mAhbo2r_QbK_T1DYkTUtj7jbs5Jl

{

"ServerSideEncryption": "AES256",

"ETag": "\"d602321594b8219b5d0a857a4e5d34cd\"",

"ChecksumCRC64NVME": "WSsI3fzEWg8="

}

Complete.jon file created in current directory Vedio.mp4 directory

{
  "Parts": [
    { "ETag": "f6279bc6c5aa4efa009c0d599cd1b206", "PartNumber": 1 },
    { "ETag": "5224ca5d459b8044bb554cfffdc4c122", "PartNumber": 2 },
    { "ETag": "e9942afaa877e2a530da4b06d8eae9fb", "PartNumber": 3 },
    { "ETag": "fcdd80466dd85133c87772e41123f3fa", "PartNumber": 4 },
    { "ETag": "9318088f5b6055f1f5af8f69bf982b49", "PartNumber": 5 },
    { "ETag": "d602321594b8219b5d0a857a4e5d34cd", "PartNumber": 6 }
  ]
}

Final step (use the above Etags and UploadId)

-----------------------------------------------------------

aws s3api complete-multipart-upload --bucket ccitmultipartupload2 --key video.mp4 --upload-id XOdv_ADNJnTuS38S6iXGxorl0TFVpDFCGoEJ2FoO9wx0cpkIhrzMZks4phjX.y_1HUCWDBZKJ9LNO6sVEUOBSud6Y047fk3CcKI3mAhbo2r_QbK_T1DYkTUtj7jbs5Jl --multipart-upload "file://complete.json"

PS C:\Users\Administrator\Desktop\mulitpart> aws s3api complete-multipart-upload `

>> --bucket ccitmultipartupload2 `

>> --key video.mp4 `

>> --upload-id XOdv_ADNJnTuS38S6iXGxorl0TFVpDFCGoEJ2FoO9wx0cpkIhrzMZks4phjX.y_1HUCWDBZKJ9LNO6sVEUOBSud6Y047fk3CcKI3mAhbo2r_QbK_T1DYkTUtj7jbs5Jl `

>> --multipart-upload "file://complete.json"

{

"ServerSideEncryption": "AES256",

"Location": "https://ccitmultipartupload2.s3.eu-west-1.amazonaws.com/video.mp4",

"Bucket": "ccitmultipartupload2",

"Key": "video.mp4",

"ETag": "\"f5827b4132193b51faf9d0aa9d1768e1-6\"",

"ChecksumCRC64NVME": "JNuhFkcmQ8c=",

"ChecksumType": "FULL_OBJECT"

}

Full file uploaded successfully..

Using Phyton Program also we can do Split and upload the file to S3 bucket using below code

Step1: Delete the exist split files in the folder ,See here file path hut coded,use using flex using dynamically uploads

import boto3
from botocore.exceptions import NoCredentialsError, PartialCredentialsError

# AWS S3 credentials
ACCESS_KEY = 'AKIATFBMO7H4BGIOIEW6'
SECRET_KEY = '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'
BUCKET_NAME = 'ccitmultipartupload2'
REGION = 'eu-west-1'

def multipart_upload(file_path, bucket_name, key_name):
    """
    Perform a multipart upload to S3.
    
    Args:
        file_path (str): Local path to the file to upload.
        bucket_name (str): Target S3 bucket.
        key_name (str): Target key in the S3 bucket.
    """
    s3_client = boto3.client(
        's3',
        aws_access_key_id=ACCESS_KEY,
        aws_secret_access_key=SECRET_KEY,
        region_name=REGION
    )

    # Create a multipart upload
    response = s3_client.create_multipart_upload(Bucket=bucket_name, Key=key_name)
    upload_id = response['UploadId']
    print(f"Multipart upload initiated with UploadId: {upload_id}")

    try:
        parts = []
        part_number = 1
        chunk_size = 50 * 1024 * 1024  # 50 MB chunks

        # Read the file and upload in chunks
        with open(file_path, 'rb') as file:
            while True:
                data = file.read(chunk_size)
                if not data:
                    break

                print(f"Uploading part {part_number}...")
                part_response = s3_client.upload_part(
                    Bucket=bucket_name,
                    Key=key_name,
                    PartNumber=part_number,
                    UploadId=upload_id,
                    Body=data
                )
                parts.append({'PartNumber': part_number, 'ETag': part_response['ETag']})
                part_number += 1

        # Complete the multipart upload
        print("Completing multipart upload...")
        s3_client.complete_multipart_upload(
            Bucket=bucket_name,
            Key=key_name,
            UploadId=upload_id,
            MultipartUpload={'Parts': parts}
        )
        print("Multipart upload completed successfully!")

    except Exception as e:
        print(f"Error occurred: {e}")
        print("Aborting multipart upload...")
        s3_client.abort_multipart_upload(Bucket=bucket_name, Key=key_name, UploadId=upload_id)
        print("Multipart upload aborted.")

# Example usage
file_path = 'C:/Users/Administrator/Desktop/mulitpart/video.mp4'
key_name = 'video.mp4'
multipart_upload(file_path, BUCKET_NAME, key_name)

PS C:\Users\Administrator\Desktop\mulitpart> python app.py

Multipart upload initiated with UploadId: 2hLmWOuhdZFPHj0z7bSm0fer3amVbXPnstLAeKx01UT.eG6tHuq7_2Ll32EalvWz8W3WKpo8_vSCuoSvZOQR.Hk9AlLDY_hsbKEpFsoeWDPyC0jMkqVCf5RFxTt4T5b.

Uploading part 1...

Uploading part 2...

Uploading part 3...

Uploading part 4...

Uploading part 5...

Uploading part 6...

Completing multipart upload...

Multipart upload completed successfully!

Step2:

Cross Origin resource Sharing (CORS)

Step1:Create one bucket while create bucket uncheck blocking all public access

ccitpublicbucket1

Step2: Usign Bucket policy generate give permission to the bucket

ccitpublicbucket1

and Then Click add statement >Generate policy

Copy the json

{

"Version": "2012-10-17",

"Statement": [

{

"Sid": "Statement1",

"Effect": "Allow",

"Principal": "*",

"Action": "s3:*",

"Resource": "arn:aws:s3:::ccitpublicbucket1/*"

}

]

}

Step3:

Click Save , Now the bucket is public any one can able to access the files in the bucket

Step4: Create one file source.html ,save with below code

<!DOCTYPE html>

<head>

<title>Source Page</title>

</head>

<body>

<h1>Source Page</h1>

<p>This is some data in the source page.</p>

</div>

</body>

</html>

Step4: Create one file Dest.html ,save with below code

<!DOCTYPE html>

<head>

<title>Read Data from Another Page</title>

</head>

<body>

<h1>Data from Another HTML Page</h1>

document.getElementById('loadData').addEventListener('click', () => {

fetch('https://ccitpublicbucket1.s3.eu-west-1.amazonaws.com/Source.html')

.then(response => response.text())

.then(htmlString => {

// Parse the HTML string into a DOM

const parser = new DOMParser();

const doc = parser.parseFromString(htmlString, 'text/html');

// Extract data from the parsed HTML

const data = doc.querySelector('#data').innerHTML;

// Display the extracted data

document.getElementById('result').innerHTML = data;

})

.catch(error => console.error('Error fetching data:', error));

});

</script>

</body>

</html>

Step5: Upload the file to source.html file to ccitpublicbucket1, the end point of the file

copy that https://ccitpublicbucket1.s3.eu-west-1.amazonaws.com/Source.html

Step6: Dest.html fetch(https://ccitpublicbucket1.s3.eu-west-1.amazonaws.com/Source.html)

Change that file and save and then upload the dest.html to same public bucket

Step7:While click the endpoint source html source text

Source.html

Step8:Destnnation html, while click load data coming from text source html.

Step9: Create one more bucket create bucket default only unchecked blocking all public access

ccitprivatebucket1

Step10: Bucket policy Click Save

{

"Version": "2012-10-17",

"Statement": [

{

"Sid": "Statement1",

"Effect": "Allow",

"Principal": "*",

"Action": "s3:*",

"Resource": "arn:aws:s3:::ccitprivatebucket1/*"

}

]

}

Step10: Upload only the dest.html file in the private bucket ccitprivatebucket1

open object url, click Loaddata ,As see getting error

Access to fetch at 'https://ccitpublicbucket1.s3.eu-west-1.amazonaws.com/Source.html' from origin 'https://ccitprivatebucket1.s3.eu-west-1.amazonaws.com' has been blocked by CORS policy: No 'Access-Control-Allow-Origin' header is present on the requested resource.

Origin is S3: ccitpublicbucket1 , This is different origin ccitprivatebucket1,so the reason not allowed

Step11: For Cross origin need to give permission to different origin

[

{

"AllowedHeaders": [

"*"

"AllowedMethods": [

"GET",

"POST",

"PUT"

"AllowedOrigins": [

"https://ccitprivatebucket1.s3.eu-west-1.amazonaws.com"

"ExposeHeaders": [

"x-amz-server-side-encryption",

"x-amz-request-id",

"x-amz-id-2"

"MaxAgeSeconds": 3000

}

]

ccitpublickbucket1>Permission>edit cross-origin give above code click save

See now data come from one origin to another origin ,This is only warning favicon ,just icon of url not provide just warning error.

Cloud Front(Global service)

Content Delivery network :

We have hosted one static website below

Step1: Static website ,in the ccitpublicbucket1 >permission >Bucket policy

{

"Version": "2012-10-17",

"Statement": [

{

"Sid": "Statement1",

"Effect": "Allow",

"Principal": "*",

"Action": "s3:*",

"Resource": "arn:aws:s3:::ccitpublicbucket1/*"

}

]

}

and then in Static website hosting and then save

upload the html static page to the bucket

Step2:Click the endpoint of the file, if are same region it will open quickly

if you are in other region ,Canada ,usa , it will take time due Geographically distance to far

like this situation ,edge caches will help you speed up the website

Step3:For that aws we have cdn, cloud front need enabled ,where your edge location web site will be distributed there .

For ex:- if any access our static page(india) from canada, they edge location already distributed

they will the cache from near location not from India region,so latency will reduce speed the process

Select the ccitpublicbucket1

Step4:Enable Origin Shield,(Caching the website in the edge location, where you required) select No

Origin shield is an additional caching layer that can help reduce the load on your origin and help protect its availability.

Step5: CDN created successfully

--Done

For Ex:-For Understanding ,course.html page edited the text "IT Sector" Save the file upload to public bucket,ccitpublicbucket1

Step1: Course.html just added text "IT SECTOR subbu"

Step2: Now you see the difference endpoint of the bucket join changes are reflected immediately

Step3: CDN or aws cloud front changes are not reflected immediately

Step4: For that you need enable one policy in the distribution

Cache key and origin requests

Cache policy

Caching Optimized (it is default policy)

Click view policy

As you see below 86400 Seconds it will refresh the edge location , if you want refresh immediately

you need create you own policy

>Click create policy given policy name and give the values 30 second ,so edage location will refresh 30 second ,click create policy

Attach the own policy which we have create above, choose ccitpolicy under cache policy click save changes

Private Bucket (without bucket public policy,the policy gave to distribution)

Step6: Create one more CDN using ccitprivatebucket1 this time

You must update the S3 bucket policy
CloudFront will provide you with the policy statement after creating the distribution.

here you can access file from private bucket using cloud front endpoint using policy

WAF enable click Create distribution

Step7:

{

"Version": "2008-10-17",

"Id": "PolicyForCloudFrontPrivateContent",

"Statement": [

{

"Sid": "AllowCloudFrontServicePrincipal",

"Effect": "Allow",

"Principal": {

"Service": "cloudfront.amazonaws.com"

"Action": "s3:GetObject",

"Resource": "arn:aws:s3:::ccitprivatebucket1/*",

"Condition": {

"StringEquals": {

"AWS:SourceArn": "arn:aws:cloudfront::216989104632:distribution/E24UYTF97GKQCF"

}

]

}

Step8: above policy need ,give you ccitprivatebucket1 policy click save

ccitprivatebucket1 >permission >Bucket policy above json clode copy past click save

Step9: Distribution url able to access bucket images

Step10: As you see bucket endpoint getting access denied , we have given Cross origan policy to the bucket for the distribution,so the reason able access bucket object by distribution url

--Thanks

Knowledge

Monday, June 9, 2025

Multipart Upload

upload the html static page to the bucket

Step2:Click the endpoint of the file, if are same region it will open quickly

if you are in other region ,Canada ,usa , it will take time due Geographically distance to far

like this situation ,edge caches will help you speed up the website

No comments:

Post a Comment

Monday, June 9, 2025

Multipart Upload

upload the html static page to the bucket Step2:Click the endpoint of the file, if are same region it will open quickly

if you are in other region ,Canada ,usa , it will take time due Geographically distance to far

like this situation ,edge caches will help you speed up the website

No comments:

Post a Comment

upload the html static page to the bucket

Step2:Click the endpoint of the file, if are same region it will open quickly