Deep Studying AMI and AWS Deep Studying Containers are actually enabled with help for SOCI snapshotter and index. Seekable OCI (SOCI) is a expertise that permits environment friendly container picture administration by means of selective file downloading. It makes use of a layer-based indexing system to map file areas inside container photos, permitting containers to start out with solely the required information loaded (lazy loading). This strategy reduces community bandwidth utilization and improves container startup occasions, making it notably priceless for organizations managing massive container photos in cloud environments.<\/p>\n
On this publish, we have a look at learn how to use SOCI on publicly accessible Deep Studying AMIs and Containers, when to make use of the varied SOCI modes offered by the software, and learn how to rapidly and effectively use this software in your workloads right this moment.<\/p>\n
Background<\/h2>\n
As organizations deploy synthetic intelligence (AI) and machine studying (ML) workloads at scale, container startup time has grow to be a bottleneck in manufacturing environments. Whether or not it\u2019s spinning up coaching jobs, serving inference endpoints, or scaling GPU clusters mechanically, the time spent downloading multi-gigabyte container photos instantly impacts price, consumer expertise, and operational effectivity. Conventional container deployment approaches drive groups to obtain total photos earlier than workloads can start. This course of can take a number of minutes to start out up photos generally utilized in manufacturing. Throughout growth, a couple of minutes of wait time is barely noticeable. In manufacturing, those self same minutes add up quick.<\/p>\n
Organizations deploying deep studying infrastructure at scale usually encounter a number of crucial challenges:<\/p>\n
Extended chilly begin occasions. Customary Docker picture pulls of 15\u201320 GB can take 4\u20136 minutes per occasion, delaying coaching jobs and inference endpoints throughout scaling occasions.
\n Wasted compute sources. GPU situations sit idle throughout picture pulls, burning by means of costly compute hours whereas ready for container initialization to complete.
\n Scaling bottlenecks. When demand spikes set off automated scaling, sluggish container startup occasions forestall fast response, resulting in degraded efficiency or dropped requests.
\n Bandwidth constraints. Giant-scale deployments pulling huge photos concurrently can saturate community bandwidth, creating cascading delays throughout the infrastructure.
\n Developer productiveness. Information scientists and ML engineers waste priceless time ready for containers to start out throughout iterative growth and experimentation cycles.<\/p>\n
Container pulling mechanisms<\/h2>\n
When pulling a container to your workloads, AWS Deep Studying AMIs (DLAMI) and Deep Studying Containers provide three choices: the usual Docker pull, SOCI parallel pull, and SOCI lazy loading by means of SOCI index. Consider these as a sliding scale of tradeoffs. Docker pulls are sequential and sluggish. SOCI parallel pull offers quicker startup occasions by chunking downloads at the price of compute sources. SOCI lazy loading offers near-instant container loading however requires information to be fetched on demand. You should use the next information to decide on the fitting mechanism to your workloads:<\/p>\n
The selection between lazy loading and parallel pull modes depends upon the picture, occasion specs, and storage configuration. Lazy loading requires photos to have a SOCI index. With out one, the system falls again to straightforward pulling.
\n Decrease-spec situations ought to use lazy loading to preserve sources, whereas high-spec situations with a number of vCPUs and excessive community bandwidth profit from parallel pull mode. Storage efficiency varies: EBS volumes are bounded by their provisioned IOPS and quantity sort, probably creating bottlenecks throughout unpacking, whereas NVMe occasion retailer delivers most I\/O efficiency at the price of information persistence throughout occasion cease\/begin cycles.<\/p>\n
The next instance exhibits the varied mechanisms primarily based on the vLLM Deep Studying Container:<\/p>\n
![\"Comparison](\"https:\/\/d2908q01vomqb2.cloudfront.net\/f1f836cb4ea6efb2a0b1b99f41ad8b103eff4b59\/2026\/05\/08\/ML-20939-1.jpg\")
<\/p>\n
Deep Studying Container Pull Mechanisms<\/p>\n
Answer structure<\/h2>\n
The next diagram exhibits the structure for utilizing SOCI with DLAMI and Deep Studying Containers.<\/p>\n
![\"Solution](\"https:\/\/d2908q01vomqb2.cloudfront.net\/f1f836cb4ea6efb2a0b1b99f41ad8b103eff4b59\/2026\/05\/08\/ML-20939-2.jpg\")
<\/p>\n
Container startup time comparability with SOCI snapshotter<\/h2>\n
The next benchmarks evaluate commonplace Docker pulls in opposition to SOCI snapshotter in each lazy loading and parallel pull modes.<\/p>\n
Lazy loading mode<\/h3>\n
Lazy loading mode begins containers instantly by fetching solely the required information on demand, with remaining layers loaded within the background as wanted.<\/p>\n
Conditions<\/h4>\n
SOCI index required<\/p>\n
Essential: Lazy loading mode requires the container picture to have a SOCI index saved within the registry. And not using a SOCI index, the snapshotter will fall again to straightforward pull habits, and also you gained\u2019t see any efficiency enchancment. AWS Deep Studying Containers (DLCs) with the -soci tag suffix include SOCI indexes pre-created and pushed to the registry, enabling lazy loading out of the field. For customized photos, you need to create and push SOCI indexes<\/p>\n
Atmosphere<\/h4>\n
Occasion Kind: g5.2xlarge
\n EBS: Dimension 500GiB, IOPS 3000, Throughput 125
\n AMI: Deep Studying Base OSS Nvidia Driver GPU AMI (Ubuntu 24.04) 20260413 (ami-06abbbf2049359343)
\n Docker Picture: public.ecr.aws\/deep-learning-containers\/vllm:0.19.0-gpu-py312-ec2-soci
\n Picture Dimension: 9.72GB (compressed), 32.7GB (disk utilization)
\n Community: Corp<\/p>\n
Begin container with Docker (non-SOCI)<\/h4>\n
We use Docker to start out the inference server instantly. Since no picture exists regionally, Docker pulls and extracts your entire picture earlier than beginning the container.<\/p>\n
Complete time: 6m59.099s.<\/p>\n
\ntime docker run
\n –gpus all
\n -d
\n -v ~\/.cache\/huggingface:\/root\/.cache\/huggingface
\n –env “HUGGING_FACE_HUB_TOKEN=$HUGGING_FACE_HUB_TOKEN”
\n -p 8000:8000
\n –ipc=host
\n public.ecr.aws\/deep-learning-containers\/vllm:0.19.0-gpu-py312-ec2-soci
\n –model mistralai\/Mistral-7B-v0.1
\n# output
\nUnable to search out picture ‘public.ecr.aws\/deep-learning-containers\/vllm:0.19.0-gpu-py312-ec2-soci’ regionally
\n0.19.0-gpu-py312-ec2-soci: Pulling from deep-learning-containers\/vllm
\n340d44d2921c: Pull full
\n….2001a2421bf1: Pull full
\nDigest: sha256:a6344c96a33ef98a32a27f89b41b8c0529d4fbbba248eb57f811725d415f68fc
\nStanding: Downloaded newer picture for public.ecr.aws\/deep-learning-containers\/vllm:0.19.0-gpu-py312-ec2-soci
\ne12d969eb71517d9a6a23b9b11cfa22ddda26a95f6a0f0d8df00cd5c4fdfe912<\/p>\n
actual 6m59.099s
\nconsumer 0m0.391s
\nsys 0m0.452s\n <\/p><\/div>\n
Begin container with SOCI snapshotter (lazy loading)<\/h4>\n
We use nerdctl with SOCI snapshotter to start out the inference container. Though no picture exists regionally, the SOCI-indexed picture permits nerdctl to tug solely the index and essential layers to start out the container, enabling lazy loading of remaining layers. Complete time: 21.125s.<\/p>\n
\ntime sudo nerdctl run
\n –snapshotter soci
\n –gpus all
\n -d
\n -v ~\/.cache\/huggingface:\/root\/.cache\/huggingface
\n –env “HUGGING_FACE_HUB_TOKEN=$HUGGING_FACE_HUB_TOKEN”
\n -p 8000:8000
\n –ipc=host
\n public.ecr.aws\/deep-learning-containers\/vllm:0.19.0-gpu-py312-ec2-soci
\n –model mistralai\/Mistral-7B-v0.1
\n# output
\npublic.ecr.aws\/deep-learning-containers\/vllm:0.19.0-gpu-py312-ec2-soci: resolved |++++++++++++++++++++++++++++++++++++++|
\nindex-sha256:a6344c96a33ef98a32a27f89b41b8c0529d4fbbba248eb57f811725d415f68fc: completed |++++++++++++++++++++++++++++++++++++++|
\nmanifest-sha256:d91ad3b46204eace6de2fb27c46d9600337fa9c124b4c82fe0f335d391017daa: completed |++++++++++++++++++++++++++++++++++++++|
\nconfig-sha256:886ed36d57c44081a74a0ab052f57366d96ab2c0fe39bb3e2f8a46cc20db8ec2: completed |++++++++++++++++++++++++++++++++++++++|
\nelapsed: 10.5s whole: 48.1 Okay (4.6 KiB\/s)
\n189307b7899438415f3df4288b3fbb26bcc4cd43678e88ec3b062bc6330e3e3b<\/p>\n
actual 0m21.125s
\nconsumer 0m0.004s
\nsys 0m0.011s\n <\/p><\/div>\n
Lazy loading abstract<\/h4>\n
Utilizing SOCI snapshotter with lazy loading, the container began in 21.125 seconds, in comparison with 6 minutes 59.099 seconds with commonplace Docker. This enchancment is achieved as a result of SOCI pulls solely the required layers to start out the container, with remaining layers loaded on demand as wanted.<\/p>\n
Parallel pull mode<\/h3>\n
Whereas lazy loading mode begins containers instantly by fetching solely the required information on-demand, parallel pull mode downloads your entire picture earlier than startup however does so with larger concurrency than commonplace Docker pulls. This mode is right once you want the total picture accessible at startup or when operating I\/O-intensive workloads.<\/p>\n
Atmosphere<\/h4>\n
Occasion Kind: g5.4xlarge
\n EBS: 500GiB gp3, 16000 IOPS, 1000 MB\/s Throughput
\n AMI: Deep Studying Base OSS Nvidia Driver GPU AMI (Ubuntu 24.04) 20260413 (ami-06abbbf2049359343)
\n Docker Picture: 763104351884.dkr.ecr.us-east-1.amazonaws.com\/sglang:0.5.10-gpu-py312-cu129-ubuntu24.04-sagemaker
\n Picture Dimension: 19.32GB (compressed), 60.4GB (Disk Utilization)
\n Community: Corp<\/p>\n
Observe: We use a personal ECR picture for this benchmark as a result of public ECR is fronted by Amazon CloudFront, which limits community bandwidth and impacts parallel mode efficiency. Non-public ECR is served instantly from Amazon Easy Storage Service (Amazon S3), offering larger throughput.<\/p>\n
Enabling parallel pull mode<\/h4>\n
The SOCI snapshotter on Deep Studying AMI defaults to lazy loading mode. To allow parallel pull mode, modify the configuration file at \/and so forth\/soci-snapshotter-grpc\/config.toml:<\/p>\n
\n# These are conservative defaults really useful by AWS for ECR
\n[pull_modes.parallel_pull_unpack]
\nallow = true # false(default): lazy loading\/true: parallel mode
\nmax_concurrent_downloads = -1 # limitless world cap throughout all photos
\nmax_concurrent_downloads_per_image = 20 # per-image obtain connections
\nconcurrent_download_chunk_size = “16mb”
\nmax_concurrent_unpacks = -1 # limitless world cap throughout all photos
\nmax_concurrent_unpacks_per_image = 10 # per-image parallel unpack threads
\ndiscard_unpacked_layers = true\n <\/div>\n
Apply the configuration by restarting the service:<\/p>\n
Tip: You may tune max_concurrent_downloads_per_image and max_concurrent_unpacks_per_image primarily based in your occasion sort and community bandwidth. For detailed tuning steering, see Introducing Seekable OCI Parallel Pull Mode for Amazon EKS.<\/p>\n
Verifying parallel mode is energetic<\/h4>\n
Monitor the SOCI snapshotter logs throughout picture pull to substantiate parallel mode is enabled:<\/p>\n
Search for log entries indicating parallel pull\/unpack:<\/p>\n
\n {“layerDigest”:”sha256:e87500e698966458d9dfc34df84602985c9821f39666619792fe6282aa6df5d4″,
\n “stage”:”data”,
\n “msg”:”getting ready snapshot with parallel pull\/unpack”,
\n “time”:”2026-04-16T23:59:08.654819383Z”}\n <\/div>\n
Pull picture with Docker (non-SOCI)<\/h4>\n
Customary Docker pull downloads and extracts layers with restricted concurrency.<\/p>\n
Complete time: 4m 44.163s<\/p>\n
\n 763104351884.dkr.ecr.us-east-1.amazonaws.com\/sglang:0.5.10-gpu-py312-cu129-ubuntu24.04-sagemaker<\/p>\n
Digest: sha256:fd0cf60bbb34a5d30f22595215a633e5d4a7260fc0868aabe3f04b1174b7365d
\nStanding: Downloaded newer picture for
\n 763104351884.dkr.ecr.us-east-1.amazonaws.com\/sglang:0.5.10-gpu-py312-cu129-ubuntu24.04-sagemaker
\n763104351884.dkr.ecr.us-east-1.amazonaws.com\/sglang:0.5.10-gpu-py312-cu129-ubuntu24.04-sagemaker<\/p>\n
actual 4m44.163s
\nconsumer 0m0.339s
\nsys 0m0.423s\n <\/p><\/div>\n
Pull picture with SOCI parallel mode<\/h4>\n
Utilizing nerdctl with SOCI parallel pull mode makes use of elevated concurrency for each downloads and unpacking operations.<\/p>\n
Complete time: 2m 12.846s<\/p>\n
\n 763104351884.dkr.ecr.us-east-1.amazonaws.com\/sglang:0.5.10-gpu-py312-cu129-ubuntu24.04-sagemaker<\/p>\n
763104351884.dkr.ecr.us-east-1.amazonaws.com\/sglang:0.5.10-gpu-py312-cu129-ubuntu24.04-sagemaker:
\n resolved |++++++++++++++++++++++++++++++++++++++|
\nmanifest-sha256:fd0cf60bbb34a5d30f22595215a633e5d4a7260fc0868aabe3f04b1174b7365d:
\n completed |++++++++++++++++++++++++++++++++++++++|
\nconfig-sha256:5e6a53b7478b0631dd3c4222ab6619dae3a3dd32a565921f10b0b03fdc316d46:
\n completed |++++++++++++++++++++++++++++++++++++++|
\nelapsed: 132.8s whole: 89.3 Okay (688.0 B\/s)<\/p>\n
actual 2m12.846s
\nconsumer 0m0.018s
\nsys 0m0.075s\n <\/p><\/div>\n
Parallel pull abstract<\/h4>\n
Utilizing SOCI parallel pull mode diminished picture pull time from 4 minutes 44 seconds to 2 minutes 12 seconds, representing a 2.2x enchancment in pull efficiency.<\/p>\n
Conclusion<\/h2>\n
SOCI snapshotter offers enhancements for each container startup and picture pull operations:<\/p>\n
Lazy loading mode \u2014 Achieved a 20x enchancment in container startup time (from 6+ minutes to ~21 seconds)
\n Parallel pull mode \u2014 Achieved a 2.2x enchancment in picture pull time (from 4 minutes 44 seconds to 2 minutes 12 seconds)<\/p>\n
Select lazy loading mode once you want the quickest attainable container startup, or parallel pull mode once you want the total picture accessible earlier than your workload begins.<\/p>\n
Clear up<\/h2>\n
When you launched EC2 situations to check SOCI snapshotter, terminate them to keep away from incurring ongoing prices. Delete any container photos you pushed to Amazon Elastic Container Registry (Amazon ECR) throughout testing, and take away any SOCI indexes you now not want.<\/p>\n
Getting began with SOCI<\/h2>\n
DLAMI and Deep Studying Containers are publicly accessible right this moment with SOCI snapshotter and SOCI index. For extra data on publicly accessible DLAMI and Deep Studying Containers, you’ll be able to try SOCI Index DLAMI to pick the pictures that help SOCI, and take a look at the Deep Studying Container repository to get extra data on supported photos with SOCI index.<\/p>\n
For detailed configuration steering and greatest practices, discuss with the SOCI documentation and the Deep Studying Container SOCI documentation.<\/p>\n
![\"Ohad](\"https:\/\/d2908q01vomqb2.cloudfront.net\/f1f836cb4ea6efb2a0b1b99f41ad8b103eff4b59\/2026\/05\/08\/ohadkatz-headshot.jpg\")
<\/p>\n<\/p><\/div>\n![\"Yadan](\"https:\/\/d2908q01vomqb2.cloudfront.net\/f1f836cb4ea6efb2a0b1b99f41ad8b103eff4b59\/2026\/05\/08\/weiyadan-headshot.jpg\")
<\/p>\n<\/p><\/div>\n![\"Nick](\"https:\/\/d2908q01vomqb2.cloudfront.net\/f1f836cb4ea6efb2a0b1b99f41ad8b103eff4b59\/2026\/05\/08\/songnick-headshot.jpg\")
<\/p>\n<\/p><\/div>\n