ECS：如何使用 s3curl 執行多部分上傳

摘要: 如何使用 s3curl 執行多分段上傳 （MPU）。

本文章適用於 本文章不適用於 本文無關於任何特定產品。 本文未識別所有產品版本。
查看其他資源

說明

本 KB 示範如何使用 S3curl 執行類似的操作，以支援 MPU 的故障排除問題。


#Create 500M 測試檔案 
  
dd if=/dev/urandom of=example bs=500M count=1

#check this file's md5sum value. 
MD5SUM 範例
9c561EAFC918D7BFAE2A1CAC376836C1 範例

#split 檔案分為兩個部分，example.00 和 example.01 
split -n 2 example -d example.

ls -l example*
-rw-r--r-- 1 root root 524288000 Sep  4 02:13 example
-rw-r--r-- 1 root root 262144000 Sep  4 02:14 example.00
-rw-r--r-- 1 root root 262144000 Sep  4 02:14 example.01


# Init Uploads, get the uploadID. mpubucket is the bucket name, examplempu is the object name for our example. 

 

ecssh124:/usr/share/s3curl # ./s3curl.pl  --id walker --post -- "http://10.xxx.xxx.xxx:9020/mpubucket/examplempu?uploads"  |xmllint --format -
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   266  100   266    0     0    647      0 --:--:-- --:--:-- --:--:--   648
<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<InitiateMultipartUploadResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/">
  <Bucket>mpubucket</Bucket>
  <Key>examplempu</Key>
  <UploadId>11c945604d9d4299b52987c289103f62</UploadId>
</InitiateMultipartUploadResult>

#Use Upload 1st part of the file, example.00 , and record the ETag value,

 

ecssh124:/usr/share/s3curl # ./s3curl.pl --debug --put=example.00  --id walker  -- "http://10.32.xxx.xxx:9020/mpubucket/examplempu?uploadId=11c945604d9d4299b52987c289103f62&partNumber=1"
s3curl: Found the url: host=10.xxx.xxx.xxx; port=9020; uri=/mpubucket/examplempu; query=uploadId=11c945604d9d4299b52987c289103f62&partNumber=1;
s3curl: replaced string: partNumber=1
s3curl: replaced string: uploadId=11c945604d9d4299b52987c289103f62
s3curl: ordinary endpoint signing case
s3curl: StringToSign='PUT\n\n\nMon, 04 Sep 2017 02:23:50 +0000\n/mpubucket/examplempu?partNumber=1&uploadId=11c945604d9d4299b52987c289103f62'
s3curl: exec curl -v -H 'Date: Mon, 04 Sep 2017 02:23:50 +0000' -H 'Authorization: AWS "S3 username":"S3 Secretkey"' -L -H 'content-type: ' -T example.00 http://10.xxx.xxx.xxx:9020/mpubucket/examplempu?uploadId=11c945604d9d4299b52987c289103f62&partNumber=1
* Hostname was NOT found in DNS cache
*   Trying 10.32.169.124...
* Connected to 10.xxx.xxx.xxx (10.xxx.xxx.xxx) port 9020 (#0)
> PUT /mpubucket/examplempu?uploadId=11c945604d9d4299b52987c289103f62&partNumber=1 HTTP/1.1
> User-Agent: curl/7.37.0
> Host: 10.xxx.xxx.xxx:9020
> Accept: */*
> Date: Mon, 04 Sep 2017 02:23:50 +0000
> Authorization: AWS "S3 username":"S3 Secretkey"
> Content-Length: 262144000
> Expect: 100-continue
>
< HTTP/1.1 100 Continue
* We are completely uploaded and fine
< HTTP/1.1 200 OK
< Date: Mon, 04 Sep 2017 02:23:50 GMT
* Server ViPR/1.0 is not blacklisted
< Server: ViPR/1.0
< x-amz-request-id: 0a20a97c:15e46a039b0:a79:3c0
< x-amz-id-2: 428b1d2b3b46e11630cc6afb9812c6047e527d1b6b51f3be014e8b6b71482163
< ETag: "d6ccfd257b8eaa2a7208bbdf2277c821"
< Content-Length: 0
<
* Connection #0 to host 10.32.xxx.xxx left intact


##Use Upload 2nd part of the file,example.01, and record the ETag value,
 

ecssh124:/usr/share/s3curl # ./s3curl.pl --debug --put=example.01  --id walker  -- "http://10.32.xxx.xxx:9020/mpubucket/examplempu?uploadId=11c945604d9d4299b52987c289103f62&partNumber=2"
s3curl: Found the url: host=10.32.xxx.xxx; port=9020; uri=/mpubucket/examplempu; query=uploadId=11c945604d9d4299b52987c289103f62&partNumber=2;
s3curl: replaced string: partNumber=2
s3curl: replaced string: uploadId=11c945604d9d4299b52987c289103f62
s3curl: ordinary endpoint signing case
s3curl: StringToSign='PUT\n\n\nMon, 04 Sep 2017 02:24:57 +0000\n/mpubucket/examplempu?partNumber=2&uploadId=11c945604d9d4299b52987c289103f62'
s3curl: exec curl -v -H 'Date: Mon, 04 Sep 2017 02:24:57 +0000' -H 'Authorization: AWS "S3 username":"S3 Secretkey"' -L -H 'content-type: ' -T example.01 http://10.32.xxx.xxx:9020/mpubucket/examplempu?uploadId=11c945604d9d4299b52987c289103f62&partNumber=2
* Hostname was NOT found in DNS cache
*   Trying 10.32.xxx.xxx...
* Connected to 10.32.xxx.xxx (10.32.xxx.xxx) port 9020 (#0)
> PUT /mpubucket/examplempu?uploadId=11c945604d9d4299b52987c289103f62&partNumber=2 HTTP/1.1
> User-Agent: curl/7.37.0
> Host: 10.32.xxx.xxx:9020
> Accept: */*
> Date: Mon, 04 Sep 2017 02:24:57 +0000
> Authorization: AWS "S3 username":"S3 Secretkey"
> Content-Length: 262144000
> Expect: 100-continue
>
< HTTP/1.1 100 Continue
* We are completely uploaded and fine
< HTTP/1.1 200 OK
< Date: Mon, 04 Sep 2017 02:24:57 GMT
* Server ViPR/1.0 is not blacklisted
< Server: ViPR/1.0
< x-amz-request-id: 0a20a97c:15e46a039b0:b02:1
< x-amz-id-2: 428b1d2b3b46e11630cc6afb9812c6047e527d1b6b51f3be014e8b6b71482163
< ETag: "c59d28e23ac4f4b488dbf85199a73fd4"
< Content-Length: 0
<
* Connection #0 to host 10.32.xxx.xxx left intact


#create mpu.xml file, which has the ETag value from above two parts file
 

ecssh124:/usr/share/s3curl # cat mpu.xml
<CompleteMultipartUpload>
  <Part>
    <PartNumber>1</PartNumber>
    <ETag>"d6ccfd257b8eaa2a7208bbdf2277c821"</ETag>
  </Part>
  <Part>
    <PartNumber>2</PartNumber>
    <ETag>"c59d28e23ac4f4b488dbf85199a73fd4"</ETag>
  </Part>
</CompleteMultipartUpload>

#Complete a multi-part upload

 

ecssh124:/usr/share/s3curl # ./s3curl.pl --debug --post mpu.xml  --id walker  -- "http://10.32.xxx.xxx:9020/mpubucket/examplempu?uploadId=11c945604d9d4299b52987c289103f62" |xmllint --format -
s3curl: Found the url: host=10.32.xxx.xxx; port=9020; uri=/mpubucket/examplempu; query=uploadId=11c945604d9d4299b52987c289103f62;
s3curl: replaced string: uploadId=11c945604d9d4299b52987c289103f62
s3curl: ordinary endpoint signing case
s3curl: StringToSign='POST\n\n\nMon, 04 Sep 2017 02:32:32 +0000\n/mpubucket/examplempu?uploadId=11c945604d9d4299b52987c289103f62'
s3curl: exec curl -v -H 'Date: Mon, 04 Sep 2017 02:32:32 +0000' -H 'Authorization: AWS "S3 username":"S3 SecretKey"' -L -H 'content-type: ' -X POST -T mpu.xml http://10.32.169.124:9020/mpubucket/examplempu?uploadId=11c945604d9d4299b52987c289103f62
* Hostname was NOT found in DNS cache
*   Trying 10.32.xxx.xxx...
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0* Connected to 10.32.xxx.xxx (10.32.xxx.xxx) port 9020 (#0)
> POST /mpubucket/examplempu?uploadId=11c945604d9d4299b52987c289103f62 HTTP/1.1
> User-Agent: curl/7.37.0
> Host: 10.32.xxx.xxx:9020
> Accept: */*
> Date: Mon, 04 Sep 2017 02:32:32 +0000
> Authorization: AWS "S3 username":"S3 SecretKey"
> Content-Length: 257
> Expect: 100-continue
>
< HTTP/1.1 100 Continue
} [data not shown]
* We are completely uploaded and fine
< HTTP/1.1 200 OK
< Date: Mon, 04 Sep 2017 02:32:32 GMT
* Server ViPR/1.0 is not blacklisted
< Server: ViPR/1.0
< x-amz-request-id: 0a20a97c:15e46a039b0:b9e:3
< x-amz-id-2: 428b1d2b3b46e11630cc6afb9812c6047e527d1b6b51f3be014e8b6b71482163
< ETag: "b7a3f5201be2a6a4b130018f79d232b3-2"
< Content-Type: application/xml
< Content-Length: 329
<
{ [data not shown]
100   586  100   329  100   257   1257    982 --:--:-- --:--:-- --:--:--  1260
* Connection #0 to host 10.32.xxx.xxx left intact
<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<CompleteMultipartUploadResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/">
  <Location>http://10.32.xxx.xxx:9020/mpubucket/examplempu</Location>
  <Bucket>mpubucket</Bucket>
  <Key>examplempu</Key>
  <ETag>"b7a3f5201be2a6a4b130018f79d232b3-2"</ETag>
</CompleteMultipartUploadResult>


#list this file
 

ecssh124:/usr/share/s3curl # ./s3curl.pl  --id walker  -- "http://10.32.xxx.xxx:9020/mpubucket/" |xmllint --format -
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   583  100   583    0     0   5152      0 --:--:-- --:--:-- --:--:--  5205
<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<ListBucketResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/">
  <Name>mpubucket</Name>
  <Prefix/>
  <Marker/>
  <MaxKeys>1000</MaxKeys>
  <IsTruncated>false</IsTruncated>
  <ServerSideEncryptionEnabled>false</ServerSideEncryptionEnabled>
  <Contents>
    <Key>examplempu</Key>
    <LastModified>2017-09-04T02:32:32Z</LastModified>
    <ETag>"b7a3f5201be2a6a4b130018f79d232b3-2"</ETag>
    <Size>524288000</Size>
    <StorageClass>STANDARD</StorageClass>
    <Owner>
      <ID>walkerobj</ID>
      <DisplayName>walkerobj</DisplayName>
    </Owner>
  </Contents>
</ListBucketResult>


#Download this file, and we can know it is same as original file before the split.
 

ecssh124:/usr/share/s3curl # ./s3curl.pl  --id walker  -- "http://10.32.xxx.xxx:9020/mpubucket/examplempu" >exampledownload
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  500M  100  500M    0     0   316M      0  0:00:01  0:00:01 --:--:--  317M
ecssh124:/usr/share/s3curl # md5sum example exampledownload
9c561eafc918d7bfae2a1cac376836c1  example
9c561eafc918d7bfae2a1cac376836c1  exampledownload

 

受影響的產品

ECS Appliance

產品

ECS Appliance, ECS Appliance Hardware Gen1 U-Series, ECS Appliance Software with Encryption, ECS Appliance Software without Encryption, Elastic Cloud Storage
文章屬性
文章編號: 000014693
文章類型: How To
上次修改時間: 09 9月 2025
版本:  5
向其他 Dell 使用者尋求您問題的答案
支援服務
檢查您的裝置是否在支援服務的涵蓋範圍內。